devopsy.kinsta.cloud

Tag: versioning

  • Unused Code: Proving a Negative

    Unused Code: Proving a Negative

    We have a ton of code that is no longer being exercised. We really don’t talk about it much, but it’s out there. From feature flags shutting off code to API versioning making an interface obsolete, it’s just hanging around out there. Like an albatross. Waiting for someone to misuse it. Or, to address the more pedestrian concerns, confusing people new to the codebase. But that code is not contributing a thing.

    Easy example. You are on version 3.2 of your API. Version 2.1 is not called by anything, but the code is still hanging out there. It is not that 2.1 cannot be called, but that it is not currently called. The problem with that is that new devs have to work through what’s current, and since 2.1 and 3.2 probably share 95% of their code bases, what portion of that code base is unused? And the API is sitting there, waiting for someone to penetrate your network and find malicious uses for it.

    Proving a Negative

    But here is the problem: Identifying that a bit of code–be it an API, a method, just a few lines in an if statement, whatever–is never exercised is terribly difficult and, in highly complex environments, borders on the impossible (it is never impossible, assuming IT owns all of the code that could exercise that bit). The old adage about proving a negative absolutely applies. We can–and testing tools in both the functional and security spaces do–generate a list of code that is exercised, or identify code that isn’t being exercised. But not a list of code that is never exercised. That functionality in test tools is super useful, because it helps prioritize issues that are being used a lot and leaves the ones not seeing a ton of traffic for later resolution.

    But I’ll say that in most organizations, that’s no longer far enough. The cruft in your code causes problems, and you need to clean it up. If you’ve worked in IT for any length of time, you’ve been through the “Oh, that bug is because the developer called something no longer used” scenario at least once. And it is a straight-up waste of time.

    So here is my proposal. Test tools generate a list of what isn’t called from systems under test. Pick one–security testing or functional testing–and require it for any bit of code, no matter how small, to go into production. That means that the system knows about all source code and thus knows definitively if a bit of code has been exercised.

    Clean Up as You Go

    Is that a tall order? Yes. Testing isn’t free. But given a scenario where you reduce technical debt by eliminating excess code, I think it will pay for itself. And that is not counting the security posture improvement that reducing the code base implies.

    You’re running the systems that present your organization to the world, so you may as well do some due diligence on yourselves. Keep making cool new stuff and upgrading the old, but clean up as you go with some form of monitoring, this or a more procedural one, so the quality is high and the excess is low.

  • The Case For an API-First SCM

    The Case For an API-First SCM

    Somewhere around a decade ago, about the time DevOps really started to manifest, we began seeing technology companies—including some of the largest ones today—investing heavily in their APIs.  

    API-first companies managed to rapidly gain widespread adoption because of the benefits of being API-driven. Everything from automation and scalability to flexibility, performance, speed and monetization.  Once DevOps thinking made it increasingly clear that automation would solve problems like human error and increase performance and velocity, creating more machine-to-machine interaction became a critical factor in software delivery—because the problem in the technology world was not innovation, but rather integration.

    Twilio is a prime example. The company became famous for its API-driven approach to customer management and even had a famous Silicon Valley billboard trumpeting their success. They changed the game for customer communication for some of the largest companies in the industry, from Netflix to DoorDash to Coca-Cola, by making communication completely invisible to users. These APIs largely served developers by unifying communication on the backend and providing powerful customer-facing capabilities on the business side. But they’re not the only ones.

    API-first SCM

    API-driven companies managed to disrupt entire markets nobody imagined could be disrupted, such as Stripe for payment processing and Slack for immediate communication (their former head of developer relations, Amir Shevat, said that Slack’s developers were literally measured by their API’s active tokens). Companies that harnessed the power of their APIs saw unprecedented growth over the last decade due to the integration power they unleashed. 

    Today, there really is no self-respecting dev-first technology that doesn’t invest heavily in the robustness of its APIs. There are even entire companies today built around API integrations—from marketplaces like Rapid API to security companies like Akita Software to companies like Kong that focus solely on simplifying API gateways.

    The API-First Dev Tool Revolution

    The same transformation that disrupted various business industries is now beginning to take center stage in the technology and dev tools space. This is also part of the developer experience revolution that is the stimulus for all the nifty dev tools in the first place. As developers become the new decision-makers—masters of their own domain—and are no longer being forced to adopt and use technology someone else purchased for them, the experience of using these tools impacts their choice of whether or not to add these tools to their chain. This means that those creating the tools today need to deeply understand the developer’s workflow and optimize their products for this user persona.

    This very same evolution that typical tooling underwent is now impacting age-old tooling in the developer ecosystem—and developer pain and friction are starting to come to the forefront. This includes the need for better and more robust API capabilities, “as-code” (infrastructure-as-code, testing and security) tooling, low-code/no-code and code variations and much more.  

    It’s interesting to see that, even with the cloud-native and DevOps revolution, our most core developer tool—our source code management—hasn’t really changed much in the last 20 years. Git remains ubiquitous, but its design is optimized for developer workflows from two decades ago. It’s long past time for an API-first SCM, and there are a number of use cases that can benefit from better APIs.

    Evolution and Anatomy of APIs to be Git-Native

    Even with the most exciting innovations we’re seeing from Git SaaS tools like GitHub and GitLab—from Co-Pilot to Codespaces, pipelines and more—they’re only as strong as the technology they are built upon. As the world evolves, this fundamental weakness is proving to be a sort of kryptonite.  

    Despite Git being an old-school, monolithic tool with proprietary protocols, Git hosting providers like GitHub, GitLab and Bitbucket have invested a lot into making their services API-driven to cater to tool integration. True, each has its own API, which drove some players integrating with the service (especially in the early days) to integrate via an SDK rather than an API.

    GitHub itself provides a variety of APIs—from the simpler REST through the more elaborate GraphQL. These APIs provide you with the ability to do pretty much anything, but are somewhat cumbersome for some use cases. To deal with the core Git functionality, you are required to directly manipulate objects in the Git database.

    Code Deployment and GitOps

    As engineering organizations grow, the need for greater control of code and infrastructure resources has become a real pain—and this is where GitOps provided the methods and the tools to support automated policy management at scale. This automated policy and governance is largely enabled, machine to machine, via APIs, and many of its controls are built on code and configuration files that are hosted via Git (hence its name: Git Operations).

    When using this methodology, a configuration change is first committed to Git (as the single source of truth that will apply changes in all the relevant places). Only then does this trigger automatic processes to apply these changes universally. In some cases, these automatic processes have to commit files to Git via API.

    Another area where SCM APIs play a key role is versioning and release management.  DevOps teams need to manage the release of new versions of their applications, including creating release notes, tagging releases and creating release branches. These processes, which sometimes take place multiple times every day, include many steps and require accuracy—a comprehensive suite of efficient APIs is critical for such an operation.

    UX Customization

    Today, there’s a huge diversity of stakeholders in modern engineering processes, particularly in the context of game development, but this is also true in any kind of web-based or UI-based engineering, as well. Today, product and design are an integral part of engineering processes, but Git still remains the domain of developers.

    There are times when you’d like to have a unified interface for your asset management that pulls data from multiple APIs and sources—such as Git for source control and Google Drive for large files. The added benefit is to abstract and decouple the underlying technology from the user experience and interface so that users can be free to migrate and move between their technologies of choice. This is true whether at the SCM or the storage level without having to change the interface that the team is accustomed to.

    APIs that efficiently perform the full set of SCM functionality allow developers to use a single backend for managing code versions while still providing end users with the desired experience, regardless of what that might be.

    Versioning-as-a-Service

    SaaS products have made it significantly easier to manage and maintain our services through simple UIs and dashboards. However, this simplicity also has a tradeoff. Have you ever made a simple configuration or code change, quickly regretted doing so and then attempted to revert it via your SaaS console? In most SaaS apps that aren’t managed as code, this happens all the time—from Okta to Datadog and many others. Without the benefits of “as-code” and the version control that comes with it, reverting changes is nearly impossible.

    The most you can achieve is “undoing” a previous configuration—if you can still catch it with a CTRL+Z—simply because engineering didn’t bother (or did not have the resources) to implement versioning. Imagine if this could come as a service for all of your SaaS applications!

    For software developers, API-based version control will unlock the possibility of easily incorporating rich version control capabilities into their product—in the same way they can currently use other capabilities such as database, file storage, etc. 

    The World is Ready for an API-First SCM

    TL;DR: API-first models have delivered some of the greatest innovations that ‘dinosaur’ industries—from payments to communications—needed to move faster and do so securely. It is almost laughable that the technology industry has inherent limitations due to a core technology that has a legacy design. This should not be a technology limitation we should accept.  

    Our source code should be able to be controlled, manipulated and transformed via API to remove human toil from places it isn’t really needed and free people to focus on higher-order problems.  With the widespread adoption of generative AI and smarter machine overlords, we’ve discovered that there are many formerly manual tasks that we can offload to machines. Free SCM APIs to unlock the next generation of innovation—because soon there will be more machines than humans to manage them. APIs are the key to tapping into this power. 

  • Death of the Release: The Move to Continuous Delivery and Experimentation

    Death of the Release: The Move to Continuous Delivery and Experimentation

    As we move from continuous delivery to continuous experimentation, we also must develop new practices

    As software development processes have evolved, we have mostly said goodbye to the idea of defined product versions. Many modern product delivery teams are taking this a step further—even the concept of a “product release” is starting to fade. Instead, products are becoming a fluid, rapidly evolving set of features, assembled uniquely for any given user.

    Twenty-five years ago, the new release of a software product was a notable event. The marketing department would even insist on a catchy version name for the release. Internet Explorer 6, Office 2000, Windows XP, OSX Mountain Lion. This is still the case for a few types of software—operating systems being the most notable—but for most of the software products we use daily the concept of a version has faded away. What version of Google Maps are you using these days? How about Facebook? Twitter? As far as the end user is concerned, there is no version, just “current.”

    A few different forces have moved software products away from release versions. The rise of agile software methodologies and associated practices such as continuous delivery has allowed software teams to tighten up software delivery cycles dramatically. Products that once were released every few months are now updated every few days.

    In parallel, the growing capabilities of the web as a platform has moved many products off of the desktop and into the browser. This has greatly reduced the friction involved in releasing a new version of a product. Pushing a code change to servers you manage is a lot easier than persuading your users to download and install an update.

    Finally, there has been an architectural shift away from large monolithic systems toward service-oriented architectures and, in recent years, microservices. These modern architectures consist of many small, independently deployable units. This again reduces the friction around deploying a code change by reducing the scope of the deployment to one specific service.

    These forces combined have led to software release cycles shortening dramatically, from multiple months or years between each release to a rate of weekly, then daily, then hourly software delivery for some teams. Delivery of a new software release has moved from a marketing event with fanfare to something that is happening almost continuously, even for large products. As far back as 2011, engineers at Amazon were releasing changes to production every 11.6 seconds.

    As teams move from a monolith to microservices there is no longer one deployable unit, but rather an ecosystem of independently deployable services. It no longer makes sense to ask which version of a release is running in an environment. Instead, we must ask which version is running for each individual service.

    And then there’s feature flagging. One can think of a feature-flagged service as existing in a sort of quantum superposition of potential code paths, resolving to a specific set separately for each request. This means that a service’s behavior in an environment can change radically without any code deployment and without any observable change to the release version.

    Now, as we move from continuous delivery to continuous experimentation, we are reaching the point where it is no longer useful to attempt to describe the current running software in terms of a “release,” or even a set of releases. The situation has become a lot more fluid.

    Coping in a Post-Release World

    Modern approaches like microservices and continuous experimentation are a boon to product delivery teams. They allow us to move faster with more safety, thanks to the tight feedback loops they enable. But this new world also brings challenges.

    It can be a lot more difficult to diagnose a bug when you are not sure what code was involved. The load on testers becomes greater as they need to consider the interaction between multiple releases of different services—each with different feature flags which can behave differently from request to request.

    From a user’s perspective, a product’s rapid pace of change can be a little overwhelming. New features arriving frequently is nice, but having the UI change every time you log in can be disorienting. Even worse is when a feature is available one day and then gone the next. Users can also be wary of the idea that they are not getting the same experience as other people, particularly if there is any whiff of unfair business practices.

    Ironically, while this accelerated feature delivery is partly enabled by continuous experimentation practices, that same speed of change can itself interfere with feature experiments. It can be difficult to statistically correlate an uptick in a KPI with a feature experiment when there are other changes happening over the course of the experiment.

    Enhancing the observability of your software—the quality of logging and other instrumentation—is one key way to mitigate some of these challenges. Another is keeping the number of active feature flags reasonably low, which can help on a variety of fronts. It reduces the testing burden of multiple interacting flags, as well as the risk of experiments interfering with one another. Fewer active flags also mean fewer things changing at once for your users.

    Managing all feature flags in a shared system can help in understanding the number of active flags and avoiding interfering experiments. Centralized flag management also means you can batch up a set of feature changes into a single release, which helps avoid overwhelming your users with many small changes. A centralized system makes it easier for testers to see which flags they should expect to be in play within different systems, and how those flags might interact.

    We’re entering an era that will involve a massive explosion in the rate at which we release new experiences to our users. Embracing this next stage in the evolution of product delivery can enable teams to move faster and with greater safety. But entering this brave new world requires us to develop new practices if we are to avoid the pitfalls of a post-release world.

    — Pete Hodgson

  • 7 Principles for Using Microservices to Build an API That Lasts

    7 Principles for Using Microservices to Build an API That Lasts

     

    SparkPost launched the first beta version of our cloud-based email delivery service three years ago. At its introduction, a handful of customers sent a few million emails a month. Now, our API is used by tens of thousands of customers—including Pinterest, Zillow, and Intercom—to send more than 15 billion emails a month. That dramatic growth demonstrates how rapidly our business pivoted from providing on-premises email infrastructure to operating as a fully cloud-based email delivery service.

    It’s a great business story. But how did we manage change at that scale with regard to technology and development management? In this article, I’ll review several of the choices and best practices that have helped us to not simply manage this pace of change, but actually to thrive from it.

    REST Is Best: Be Practical, Not Pedantic

    From the start, we took an “API first” approach. Our email API is our core application, not an afterthought.

    The first, and most important, step we took was deciding to use REST for our API. Our philosophy was to choose the following three elements as our API’s foundation:

    1. HTTP: This covers response codes as well as operators. The operators include POST, GET, PUT and DELETE, and they can be mapped to the basic CRUD (create, read, update, delete) operations.
    2. Resources: These are the entities that the HTTP operators act on.
    3. JSON (JavaScript Object Notation): This is a data-interchange format.

    Those three elements provide everything needed for a practical REST API, including simplicity, portability, interoperability and modifiability. After the API is built, users can easily integrate against it regardless of their programming language, including C#, PHP, Node.js, Java or even cURL in a shell. They can do so without worrying about the underlying technology, including its use of multiple microservices.

    When we created the SparkPost API, we tried not to be too pedantic about following a pure REST model, opting for ease of use instead. Here are two examples that may not follow RESTful best practices:

    1. GET /api/v1/account?include=usage
    2. POST/api/v1/sending-domains/example.domain.com/verify

    The first example uses a query string parameters for GET to filter what comes back in an entity. In the second example, we use the action word “verify” in the endpoint name, which may not be RESTful. We discuss each new use case and do our best to ensure it’s consistent and easy to use.

    Related Content:

    APIs: Automation is Great. So is Usability

    Tips to Optimize the Cloud for Scaling

    Evolving and Managing Change

    We have many developers and teams working on our API’s microservices, with changes delivered on a continuous basis. We automate deployment of a change to production when an engineer (and a second) concludes it has passed our tests. It’s “released” when the product team decides we’re ready to tell customers about the change.

    We decided early on to keep our API consistent in its use of conventions and how changes are managed. We established a governance group that includes engineers representing each team, a member of the product management group, and our CTO. This group establishes and enforces our API conventions, which are thoroughly documented.

    Documenting our conventions reduces inconsistencies and makes it easier to define each new endpoint. Here are a few conventions we’ve established:

    • URL paths are lowercase with hyphens when separating words, and are case sensitive.
    • URL query parameters and JSON fields are also lowercase with underscores, and are case sensitive.
    • Unexpected query parameters and JSON fields in the request body should be ignored.

    Our governance group also sets the ground rules for how changes can be made and what types of changes are allowed. There are a number of good API changes that are beneficial to users and don’t break their integrations, including:

    • A new API resource, endpoint, or operation on an existing resource
    • A new optional parameter or JSON key
    • A new key returned in the JSON response body

    Conversely, a breaking change includes anything that could break a user’s integration, such as:

    • Changing a field’s data type
    • A new required parameter or JSON Key
    • Removal of an existing endpoint or request method
    • A materially different behavior of an existing resource method, such as changing the default for an option from “false” to “true”

    Those kinds of changes will either break users’ integrations or require the addition of a new version, which introduces more overhead.

    Don’t Break Bad When Making Changes – Nearly All the Time

    Breaking changes should be avoided, even if they’re the result of fixing bugs or inconsistencies. It’s usually better to work around such idiosyncrasies rather than risk breaking customers’ integrations. If a change is of the breaking variety, we proceed with extreme caution and seek out alternative ways to achieve our goal. Sometimes that can be accomplished by simply allowing the user to change their behavior through an account setting or an API parameter.

    However, there are times when the benefits to our users outweigh any potential negatives that would be introduced by a change. In those cases, though, we followed these best practices:

    • We received buy-in from our product, support and developer relations teams.
    • We analyzed API logs to see how many users the change might affect.
    • We gave users at least 30 to 60 days of advance warning about the change.
    • We sent an email or published a blog post containing explicit details of the change and why we were making it.
    • We provided guidance in the API documentation.

    One Version to Rule Them All

    After making thousands of changes to our API during the past three years, we’re still on the first version. We decided early on not to version our API beyond the first one because doing so adds a level of unnecessary complexity that can slow down user adoption of our latest and greatest functionality. Versioning an API can also slow down development and testing, complicate monitoring and confuse user documentation.

    In addition, not versioning our API means we can avoid the controversy that tends to swirl around the subject. There are three ways to version an API, and all of them come with potential pitfalls:

    • Put the version in the URL: Easy to do but a bad choice from a semantic perspective because the entity doesn’t change between v1 and v2.
    • Add a custom header: Also easy to do but not semantically correct.
    • Put the version in the accept header: More semantically correct but the most complicated approach.

    Use Client Libraries to Help Non-JavaScript Users

    Some of our users prefer Python, C#, Java or PHP over JavaScript. We make it quick and easy for them to integrate our API into their applications by maintaining client libraries that offer an easy-to-use set of functions for their code.

    Our client libraries have changed over time, and we do version them. We’ve learned that abstraction is hard when wrapping a living, growing API, so we focus on providing a thin layer of abstraction with some syntactic shortcuts that simplify the more complex areas of our API. Doing so lets our users quickly hit any of our API endpoints quickly and with a lot of flexibility—it also lets us “future proof” our API to some extent.

    A ‘Documentation First’ Strategy, Too

    We treat our documentation as code and use it to document our API changes before we write or change a single line of API code. Doing so helps us enforce our conventions, keeps everything consistent and maintains a good customer experience. It also cuts down on support costs.

    We maintain our documentation in GitHub, which makes it easy for technical and non-technical users to contribute changes. We’ve also found that it’s easier to review changes that way. We use API Blueprint Markdown format and Jekyll to generate the HTML docs, along with a nice search service called Algolia. Doing so lets us have full control over the customer experience, including mobile.

    For those who don’t want to “roll their own” documentation, we recommend OpenAPI (previously known as Swagger), Apiary and API Blueprint. It’s important to avoid a tool that isn’t well-suited for REST API documentation. We suggest including a bright orange “Run in Postman” button in the documentation so it’s easy to try an API, along with good examples of success and failure scenarios.

    Listen to Users

    Finally, we recommend that all developers pay attention to what their users have to say. SparkPost has a community Slack channel where thousands of users can easily access members of our product, support, engineering and executive management teams. We also have a dedicated Developer Relations team that’s squarely focused on engaging with the developer community. All of that allows us to listen to our users and incorporate their feedback into our API.

    — Chris McFadden