devopsy.kinsta.cloud

Tag: API First

  • The Case For an API-First SCM

    The Case For an API-First SCM

    Somewhere around a decade ago, about the time DevOps really started to manifest, we began seeing technology companies—including some of the largest ones today—investing heavily in their APIs.  

    API-first companies managed to rapidly gain widespread adoption because of the benefits of being API-driven. Everything from automation and scalability to flexibility, performance, speed and monetization.  Once DevOps thinking made it increasingly clear that automation would solve problems like human error and increase performance and velocity, creating more machine-to-machine interaction became a critical factor in software delivery—because the problem in the technology world was not innovation, but rather integration.

    Twilio is a prime example. The company became famous for its API-driven approach to customer management and even had a famous Silicon Valley billboard trumpeting their success. They changed the game for customer communication for some of the largest companies in the industry, from Netflix to DoorDash to Coca-Cola, by making communication completely invisible to users. These APIs largely served developers by unifying communication on the backend and providing powerful customer-facing capabilities on the business side. But they’re not the only ones.

    API-first SCM

    API-driven companies managed to disrupt entire markets nobody imagined could be disrupted, such as Stripe for payment processing and Slack for immediate communication (their former head of developer relations, Amir Shevat, said that Slack’s developers were literally measured by their API’s active tokens). Companies that harnessed the power of their APIs saw unprecedented growth over the last decade due to the integration power they unleashed. 

    Today, there really is no self-respecting dev-first technology that doesn’t invest heavily in the robustness of its APIs. There are even entire companies today built around API integrations—from marketplaces like Rapid API to security companies like Akita Software to companies like Kong that focus solely on simplifying API gateways.

    The API-First Dev Tool Revolution

    The same transformation that disrupted various business industries is now beginning to take center stage in the technology and dev tools space. This is also part of the developer experience revolution that is the stimulus for all the nifty dev tools in the first place. As developers become the new decision-makers—masters of their own domain—and are no longer being forced to adopt and use technology someone else purchased for them, the experience of using these tools impacts their choice of whether or not to add these tools to their chain. This means that those creating the tools today need to deeply understand the developer’s workflow and optimize their products for this user persona.

    This very same evolution that typical tooling underwent is now impacting age-old tooling in the developer ecosystem—and developer pain and friction are starting to come to the forefront. This includes the need for better and more robust API capabilities, “as-code” (infrastructure-as-code, testing and security) tooling, low-code/no-code and code variations and much more.  

    It’s interesting to see that, even with the cloud-native and DevOps revolution, our most core developer tool—our source code management—hasn’t really changed much in the last 20 years. Git remains ubiquitous, but its design is optimized for developer workflows from two decades ago. It’s long past time for an API-first SCM, and there are a number of use cases that can benefit from better APIs.

    Evolution and Anatomy of APIs to be Git-Native

    Even with the most exciting innovations we’re seeing from Git SaaS tools like GitHub and GitLab—from Co-Pilot to Codespaces, pipelines and more—they’re only as strong as the technology they are built upon. As the world evolves, this fundamental weakness is proving to be a sort of kryptonite.  

    Despite Git being an old-school, monolithic tool with proprietary protocols, Git hosting providers like GitHub, GitLab and Bitbucket have invested a lot into making their services API-driven to cater to tool integration. True, each has its own API, which drove some players integrating with the service (especially in the early days) to integrate via an SDK rather than an API.

    GitHub itself provides a variety of APIs—from the simpler REST through the more elaborate GraphQL. These APIs provide you with the ability to do pretty much anything, but are somewhat cumbersome for some use cases. To deal with the core Git functionality, you are required to directly manipulate objects in the Git database.

    Code Deployment and GitOps

    As engineering organizations grow, the need for greater control of code and infrastructure resources has become a real pain—and this is where GitOps provided the methods and the tools to support automated policy management at scale. This automated policy and governance is largely enabled, machine to machine, via APIs, and many of its controls are built on code and configuration files that are hosted via Git (hence its name: Git Operations).

    When using this methodology, a configuration change is first committed to Git (as the single source of truth that will apply changes in all the relevant places). Only then does this trigger automatic processes to apply these changes universally. In some cases, these automatic processes have to commit files to Git via API.

    Another area where SCM APIs play a key role is versioning and release management.  DevOps teams need to manage the release of new versions of their applications, including creating release notes, tagging releases and creating release branches. These processes, which sometimes take place multiple times every day, include many steps and require accuracy—a comprehensive suite of efficient APIs is critical for such an operation.

    UX Customization

    Today, there’s a huge diversity of stakeholders in modern engineering processes, particularly in the context of game development, but this is also true in any kind of web-based or UI-based engineering, as well. Today, product and design are an integral part of engineering processes, but Git still remains the domain of developers.

    There are times when you’d like to have a unified interface for your asset management that pulls data from multiple APIs and sources—such as Git for source control and Google Drive for large files. The added benefit is to abstract and decouple the underlying technology from the user experience and interface so that users can be free to migrate and move between their technologies of choice. This is true whether at the SCM or the storage level without having to change the interface that the team is accustomed to.

    APIs that efficiently perform the full set of SCM functionality allow developers to use a single backend for managing code versions while still providing end users with the desired experience, regardless of what that might be.

    Versioning-as-a-Service

    SaaS products have made it significantly easier to manage and maintain our services through simple UIs and dashboards. However, this simplicity also has a tradeoff. Have you ever made a simple configuration or code change, quickly regretted doing so and then attempted to revert it via your SaaS console? In most SaaS apps that aren’t managed as code, this happens all the time—from Okta to Datadog and many others. Without the benefits of “as-code” and the version control that comes with it, reverting changes is nearly impossible.

    The most you can achieve is “undoing” a previous configuration—if you can still catch it with a CTRL+Z—simply because engineering didn’t bother (or did not have the resources) to implement versioning. Imagine if this could come as a service for all of your SaaS applications!

    For software developers, API-based version control will unlock the possibility of easily incorporating rich version control capabilities into their product—in the same way they can currently use other capabilities such as database, file storage, etc. 

    The World is Ready for an API-First SCM

    TL;DR: API-first models have delivered some of the greatest innovations that ‘dinosaur’ industries—from payments to communications—needed to move faster and do so securely. It is almost laughable that the technology industry has inherent limitations due to a core technology that has a legacy design. This should not be a technology limitation we should accept.  

    Our source code should be able to be controlled, manipulated and transformed via API to remove human toil from places it isn’t really needed and free people to focus on higher-order problems.  With the widespread adoption of generative AI and smarter machine overlords, we’ve discovered that there are many formerly manual tasks that we can offload to machines. Free SCM APIs to unlock the next generation of innovation—because soon there will be more machines than humans to manage them. APIs are the key to tapping into this power. 

  • Index Tracks Impressive Growth of API-First Startups

    Index Tracks Impressive Growth of API-First Startups

    APIs are the building blocks of modern software, and their use continues to grow around the globe. A full 89% of investment into APIs will increase or stay the same over the next 12 months, Postman’s 2022 State of the API report found. And the evolving API-first approach is ushering in many innovations and startups into the field. This trend is epitomized by the latest research from GGV Capital, which maintains an API-First Index of leading developer-first software companies.

    The API-First Index tracks API-first companies that have raised $50 million in funding or more. And as of Q3 2022, the number of businesses that met this threshold increased to over 60. Just in Q3 alone, $350 million in funding was raised by companies on the index, and, in total, the index represents over $14 billion in funding. Undoubtedly, interest and investment in this space are expanding and will continue to increase in the year ahead. It’s a good area to keep tabs on, as the emphasis on modularity impacts how software is pieced together.

    I recently met with Tiffany Luck, partner at GGV Capitol, who oversees the API-First Index. Below, we’ll explore the index a bit more and get Luck’s viewpoint on why these findings represent a unique moment for API-first technology and business models as a whole.

    The API-First Index

    The API-First Index focuses on companies that serve niche functionally through APIs; also known as API-as-a-product. The list includes an assortment of API-based tools from FinTech, communications, health care, commerce and other sectors.

    For example, Stripe is likely the most well-known API-as-a-product, as it is commonly integrated as a way for businesses to accept payments. There’s also MessageBird, an API for sending notifications across any channel. Other examples include Mapbox, an API for geolocation data, Mux, an API for creating live and on-demand video experiences, Nylas, which offers a popular email API and Alpaca, an API for stock and cryptocurrency trading and many others.

    Currently, the index doesn’t include the entire API platform landscape of tools, which ranges from areas like API management to testing, documentation, monitoring, design, security and observability. There are also middleware integration platforms to consider as well. With these included, the list of well-funded API-related toolsets would surely be even longer.

    Why the Excitement About API-First?

    The number of people deciding to build API-first companies is rapidly increasing. These API-based web components are interchangeable and represent the future of software development. “The world is going toward web and cloud components,” said Luck. The trend represents a global movement with much effort reflected in Europe, Asia and the U.S. Furthermore, API models seemed resilient throughout ongoing economic uncertainty and the patchy VC funding of 2022. “APIs are holding up very well compared to the broader VC landscape,” said Luck.

    So, why is there so much excitement around APIs? Well, people are looking for best-in-class structural blocks that can be used to build common features like geolocation, AI, weather, payments or specific CRM functions. Niche APIs can swiftly deliver particular features without the bulk of a large platform. Another reason is that they’re relatively sticky—API-based delivery models are efficient at building a self-service platform that expands over time, which is a very keen prospect to investors, said Luck.

    Unicorn APIs Solve Hard Problems

    In evaluating API-first companies, Luck looks for those offering B2B solutions to super-hard, shared problems. This equates to reusable functionality that is difficult to build, maintain and scale individually. Furthermore, the solution must solve a need that’s common across companies but that is not core to their products.

    For example, Luck has her eye on a few impressive API-first use cases. Stream, for example, powers in-app user-facing features. Knock offers solutions for handling notifications. But many of the top most-challenging areas include financial services, as this functionality is difficult to build from a regulatory and connectivity perspective. Examples include Pinwheel, which aggregates real-time income and employment data. Method API does something similar but for aggregating consumer debts.

    Luck described APIs as a “modern way of accessing what would otherwise be technically inaccessible.” We’re seeing this action unfold in open banking, as new standards around the globe now mandate that consumer financial data be programmatically accessible, often in the form of an API connection.

    Developer Experience: The New Competitive Advantage

    In a competitive market surging with powerful developer-facing tools, non-functional aspects such as developer experience (DX), security, performance and reliability are becoming more of a focus. According to Luck, developer experience is probably at the top of the pyramid as it represents a catch-all for many of the aforementioned traits. Specifically, developer tools with high-quality DX have excellent documentation and must maintain backward compatibility to avoid breaking changes on the consumer side. “Ultimately, what you’re implementing impacts that customer’s end customer. Customers want zero negative impact and zero disruption,” she said.

    The API-First Index is good for the industry as it helps educate the public on private investing rounds. And as discussion around it increases, it brings awareness and visibility to the benefits of building API-first.

  • We’re Doing More Less Development

    We’re Doing More Less Development

    My roots are in software development. I’ve written tax software, SmartGrid firmware and servers, enterprise integration code for many developed and purchased servers, mapping software, cell phone firmware—the list goes on. The point is development is in my blood. When I look at something that impacts software development, I look at it through a practitioner’s lens as much as an analyst’s. And there is a lot of change going on in the realm of software development.

    For a very long time, there have been options out there that reduce the amount of software we develop. And those options are often popular. There was a time where the most common phrase in enterprise IT was “Develop core systems, purchase support systems,” and the message sank in enough that we don’t need to say it anymore. There was a time when enterprises cobbled together systems as essential as email–some developing their own systems, but most working hard to integrate available offerings. We have completely flipped that equation, and now almost no one would consider spending significant IT resources on email. Not on development, not on integration. There are of course corner cases that do have to–our industry could be defined as a collection of sharp-corner cases we haven’t yet smoothed out–but few, if any of those reading this blog have had to worry about those corners. So, change number one: Purchasing IT solutions is massively successful at reducing our time. From CRM to email to security, we’re using far more tools written by others.

    There have also been code-generation tools forever. While the most well-known ones have historically been UI-focused, 4GLs generally tried to address development with less code. Again, time, failures and iterations have given us useful code-reduction techniques. From pre-formatted ‘if’ statements that we just plug in variables to API generation through (sometimes AI-powered) initial program generation engines that give us code to start a project. The requirements for writing code are less per-project.

    And the big one: Open source. There are entire software markets for which it isn’t worth developing solutions because there are good open source solutions on the market. If it is popular but even vendors cannot afford to make solutions because everyone is using the OSS solution—well, that reduces the amount of code developed. There are several companies whose entire business model is simply offering enterprises service and support on OSS. No development required, even for the “vendor.” Large portions of any given application we develop these days are based on open source that we include to solve one piece of the problem. We’ve had some spectacular hacks and failures based upon that model, but it is undeniably an enabler for more solutions.

    With all of these (and arguably more–like API-first and API standardization) reducing the amount of code we’re developing, you would think developers would be less busy, right?

    DevOps and Agile streamlined our development and shortened delivery timelines for subsets of a given project, but haven’t—and weren’t designed to—reduce the amount of software developed. Indeed, quite the opposite. With faster iterations, more could be tried out.

    But it is wrong to think there is less development effort. There are several reasons for this, but I’m going to focus on the big one. People invested in the company—business leaders, etc—have always had a massive pent-up desire to improve software to increase efficiency, competitiveness, etc. And IT was the bottleneck. Agile/DevOps are by no means perfect, but they offered a pressure release valve for that pent-up demand. And we’ve been at it long enough to know that rapid iterations in response to those demands inspire more new ideas, creating another iteration… So it will be unlikely to stop. Indeed, we all know what technical debt is, but there seems to be a disconnect between knowing intuitively that more code means more debt in the form of maintenance overhead. So developers are doing more because the organization has good ideas that need implementation, even though we have plenty of tools to reduce development effort. Some organizations have reduced developer headcount, but not many. And that might slow, but that’s unlikely. In the coming highly competitive environment, more features and solutions will be desired, not less.

    And you’ll be there, continuing to rock it. Pay attention to ways you can further reduce code effort. This is a long-term process, but the goal should be to reduce the growth rate in staff time. It is unlikely to go down overall–because maintenance overhead plus new projects is a constantly growing number–but slowing growth helps developers and working managers stay sane and take the time to find good hires.

    And just keep doing more less development.

  • Languages and DevOps: API-First

    Languages and DevOps: API-First

    One of the places development has made significant changes is in the land of APIs. These days, a program almost isn’t considered a proper application unless it offers APIs to do most of what can be done via UI (and even command line). This is a big deal for several reasons; first and foremost, the ability to subdivide the work and then test each subdivision. A proper API does one thing and does it well. Testing can validate that this statement is true for any given API. It also allows for distribution of processing. It is far easier to use a load balancer to scale APIs than it is to scale a monolithic application. And finally, it helps with portability, because with API-first, developer users don’t care what platform it’s running on, just that it can be reached via URL and responds as expected.

    Everything is a trade-off, and there are several places that API-first has issues. Integration testing is objectively harder. Loosely coupled applications, in general, are harder to test, and requiring multiple portions of an application be available to run testing becomes onerous, particularly as the layers of APIs start to build up. Second, security is another issue – particularly if you’re offering a public-facing API, but also for internal usage. Once an attacker has access to the network, internal-only APIs are no different to them than external. Smart IT departments design internal APIs with the same controls that external requires. Third is DevOps in general. API-first grew up in the agile/DevOps world, but because there are cascading dependencies, is actually harder to manage DevOps for than most monolithic applications. Finally, documentation – the bane of API-first’s existence. The number of poorly documented APIs vastly outnumbers the astoundingly well-documented ones. Even well-documented APIs, like Google Places, require some thought and testing to use well. Poorly documented ones – my favorite is, “It’s mysite.com/users – everyone knows what’s in there!” – are next to useless without a learning curve.

    With all of that said, what about languages? That greatly depends on what you’re doing. Users don’t care what language it is implemented in – as noted above, they care that the API can be reached via URI, and that it behaves as expected. So you do have options. Indeed, at this point in the evolution of APIs, the language options are so open that you’ll probably be choosing an API definition language instead of a programming language, and just instruct your definition tool to spit out the language of choice.

    Wait, what?

    For those who haven’t followed API development closely over the last few years, it goes something like this. APIs became standardized, then APIs became a big deal, then smart people started to think in terms of code generation á la ancient 4GLs. What if you could use a specification language to describe what you want to do, then pick your language and tell the tool to generate the API? What if you could do this for the entire API hierarchy? That’s where we are. I’m most familiar with OpenAPI and RAML, but there are a large number of specifications out there, and a larger number of tools to develop in them. Before my API friends freak out, of course this is an over-simplification of what happened. I only have 1000 words, max, and my editor would like less. Relax a bit.

    And then you generate.

    I’ll rely on my own experience here, but there are literally dozens of tools available to get you from idea to API interface (doing the work behind the interface is where you come in). With Swagger CodeGen, the tool I’m most familiar with, you specify the API, then generate server interfaces (they call them stubs), and generate client libraries. Finally, you develop what goes on behind the interface on the server side, and (you or others) use the API on the client side. When I say “generate,” I mean “Tell it your favorite language, and click generate.” It spits out the code you can then use to flesh out your API. It does not spit out the info to get the API into your DevOps toolchain, so you’ll have to do the work for that. Which means the more APIs you can generate in a batch, the less DevOps lifting you’ll have to do (more on that if/when we cover microservices). Since most languages can easily be placed into DevOps via their build tools, I don’t want to make this sound too onerous – security and testing will be a bigger problem, by far.

    API security is slowly getting automated and placed right into generated code, but all the usual bits, like certificate management, still have to be handled. The interesting thing, that hasn’t (yet) gotten a ton of attention, is security of clients. It is far harder to verify that the client library is secure when the purpose of that library is to grant access to your API. I expect we’ll eventually have to get there for some applications, but today, it appears to be largely individual orgs figuring out how best to lock down client libraries, and most of the limiting factors end up server-side.

    I’ve generated Python, Java and Ruby APIs via various toolsets, and considering the problem domain is still relatively small, they work fine. And it saves one piece of the puzzle … but does introduce another piece – coordinating client and server updates via DevOps.

    Keep making the apps that run the world, and use API generators to further standardization, spitting out whatever language you need on the backend, and a portable interface for clients.

    You. Rock.

    Whatever your language of choice, in the end, it will likely be your build tool that links it to DevOps. Your biggest concern for DevOps will probably be portability, at least in the near future. Meanwhile, keep rocking it. You’re doing the work that needs doing, even if it isn’t the part of DevOps everyone is currently buzzing about.