The number of DevOps tools keeps growing, as does the number of cloud resources and options to manage those resources. On EC2 alone, AWS offers nearly 400 instance types across storage, networking and operating systems. Complicating this further is that users can choose from machines located in 24 regions and 77 availability zones.
This proliferation is driving a ‘speed paradox’ wherein the faster you grow the more fragmented and complex your ecosystem becomes which, in turn, slows everything down. Most DevOps teams struggle to keep up with the demand for more environments from developers.
The Speed Paradox
The Rise of the Internal Developer Platform (IDP)
One intriguing solution to the speed paradox is the internal developer platform (IDP). As its name suggests, an IDP provides a single place where developers can find all the resources needed to run their environments. An IDP allows developers to speed development processes by offering improvements in:
- Efficiency – Simplifying access to development and testing infrastructure through a self-service experience;
- Consistency – Providing a consistent way to consume infrastructure resources across teams; and
- Visibility – Providing a single place to see all the pipelines, workflows and states associated with each environment.
Gartner sums it nicely: “Product teams often struggle due to disparate tools and disjointed workflows as they accelerate digital transformation. Software engineering leaders leading platform teams must establish internal, self-service developer portals to enable consistency and scale cloud, agile and DevOps initiatives.”
Lessons From the ‘PaaS-t’
The idea of a developer platform isn’t new. Previous-generation PaaS—Google App Engine, Heroku Cloud Foundry, for instance—provided a simple interface to write applications by abstracting cloud resources. But early PaaS failed, perhaps mostly due to the long tail of developer needs. Back in 2013, I observed:
“Every Access project will eventually fail because, while 80% of what the user wants is fast and easy to create and the next 10% is possible with difficulty, ultimately the last 10% is impossible because you can’t get far enough underneath the built-in abstractions and users always want 100% of what they want.”
You can sum up the failures of early PaaS in three points:
- Too opinionated: It is hard (impossible?) to control the way the platforms run their infrastructure resources. You’re dependent on the platform vendor for customization.
- Limited to simple web applications and language choices: Modern SaaS applications are complex and include backend services that don’t fit that architecture.
- Maturity: Early PaaS attempted to abstract cloud when it was a young, moving target.
What’s Changed?
With early PaaS, we traded simplicity for flexibility. Finding the right balance between the two is the heart of the challenge. Kubernetes and Docker, as well as IaC tools like Terraform, Cloud Formation and Azure ARM are a great substrate: They provide a consistent framework to manage applications across a wide range of infrastructure resources among clouds. Maturity and adoption of these tools reduces barriers to build a developer platform. This opens the opportunity to create a custom platform for each organization, allowing teams to control how their platform fits into their organization’s environments. It also offers balance between simplicity and flexibility, because each organization can choose where to draw the line—and even redraw that line without being dependent on an external vendor.
The next-generation platform
Enter Backstage
Backstage is an open source IDP project led by Spotify. Its three main parts are:
- Service Definition—Based on the Kubernetes template format, it provides a service descriptor that fits into a range of services: monitoring, CI/CD and Git repositories
- Backstage service catalog—Basically a web application with a database backend providing a single point of access to manage resources
- Integrated tooling via plugins —A marketplace of integrated resources
Here’s how an IDP based on Backstage can look:
An example of an internal development platform based on backstage
We can use Backstage to organize all development services and resources such as our squad KPI, the development pipeline, API and Git repositories, all in one place.
Turning Backstage Into a Full-Blown PaaS
One of the missing pieces in Backstage is infrastructure provisioning. Adding it turns Backstage into something closer to a full-blown PaaS. Cloudify provides a remote execution engine that integrates with a variety of infrastructure resources and allows users to organize them into a self-service development or production environment. The integration of Cloudify into Backstage includes synchronizing the development environments into the Backstage service catalog and allowing self-provisioning of those services through the Backstage portal or through GitOps.
Integrating Cloudify as a remote execution engine to Backstage
You can watch a demo or get more information on the integration from the Cloudify Backstage Git repo.
The End of ‘Growth At Any Cost’
Last year, Sarah Wang and Marti Cassado analyzed the impact of cloud spending on hypergrowth companies. As tech companies trade away efficiency in pursuit of velocity, they face a non-sustainable growth model where cloud costs surpass 50% of total revenue. Wang and Cassado’s analysis illustrates how this condition directly impacts a company’s profitability—and valuation.
In today’s market upheaval, it’s even easier to see how a “growth at any cost” policy is unsustainable. Today, DevOps teams have to find ways to support growth while controlling costs. This forces organizations to better control how development teams use cloud infrastructure.
IDPs can become a major tool in this effort. Unsurprisingly they’re becoming the next hot DevOps trend. Backstage provides a great framework and ecosystem to shorten the time a team needs to build its own internal platform and Cloudify provides the open, remote execution engine with a single place to organize infrastructure resources into a self-service environment. The integration turns Backstage into a full-blown PaaS.
The current Cloudify-Backstage integration is only an MVP release. Next, we’ll deepen the integration, providing more monitoring within the Backstage portal. Here, you can find more information on our open source contribution to Backstage and offer feedback.
An Open Source Solution to the Speed Paradox
The “speed paradox”—where the faster you grow the more fragmented and complex your ecosystem becomes, which in turn slows everything down—need not grind our DevOps agility goals into the ground. By using open source Cloudify integrated into the Backstage project, DevOps teams can avoid becoming the new bottleneck in software development agility goals.
