Category: Blogs

There has never been a better time to be a software developer. There is a language and framework to solve virtually any challenges we encounter. New tools, architectural patterns and methodologies like DevOps and Agile have changed how we work, architect our applications and organize ourselves. This has enabled teams to be nimbler than ever before. But this has all come with a hidden cost: Sprawling complexity.

Sprawl is a complex web of disparate tools, languages and frameworks used across an organization’s projects. While this diversity allows for specialized solutions tailored to specific problems, it creates a labyrinth of interdependencies and varied skill requirements. This complexity often results in developers constantly navigating through a maze of tools, leading to inefficiencies and a steep learning curve for new team members. As organizations adopt more tools and technologies, the cohesive understanding of the entire software ecosystem diminishes. There are two areas in particular where this pain is felt acutely.

The first area sprawl acutely impacts is the ability to find essential knowledge about systems, which becomes fragmented across tools and resources as sprawl increases. This fragmentation leads to longer incident response times and makes it more difficult for newly hired engineers to ramp up. Knowledge fragmentation also makes it considerably harder for enterprises to measure and manage initiatives to achieve high standards for security and reliability, as well as track migrations to new architectures like containers or clouds.

Sprawl also impacts frequently occurring tasks for developers, such as managing “day two operations” like changing environment variables, adding secrets or modifying a replica count, or creating something new like a microservice or an environment. Given the complexity required to create and operate infrastructure, many organizations have invested in creating automations via infrastructure-as-code and CI/CD systems; however, these automations are often difficult for developers to discover and use. As a result, developers often file tickets with ops teams to complete these tasks, creating a process bottleneck that slows development velocity while burdening ops teams with mundane, repetitive tasks. This is commonly referred to as TicketOps.

All of this has made the developer experience worse than ever. Developers report high levels of cognitive overload and burnout, which can cause unwanted developer attrition and reduced overall productivity. But it doesn’t have to be this way!

Introducing Backstage

In the early days of the cloud, only a handful of enterprises with lofty valuations could afford to solve these challenges by dedicating 50 – 100 engineers to build internal tooling designed to address the challenges of sprawl. Everyone else was left to suffer with subpar developer experiences or partial solutions like stale spreadsheets, manual processes and information that were often out of sync with the underlying system.

Just as the pain was becoming unbearable for most enterprises, Spotify donated the tooling it created to address these challenges internally to the Cloud Native Computing Foundation. As a free, customizable, open-source project focused on these pain points, Backstage quickly captured the attention of enterprises worldwide.

Over 2,000 enterprises have tried Backstage, making the project a tremendous success by promoting a vision for how internal developer portals can alleviate the pain of sprawl. However, a surprising number of enterprises struggle to adopt Backstage. In an interview with the head of engineering for Backstage, it was revealed many companies that have deployed Backstage have achieved only ~10% adoption rates, which is typically equivalent to a proof of concept. Understand why enterprises have struggled to succeed and decide if Backstage is right for you.

Deciding if Backstage is Right for You

Here are key questions to help guide your decision.

Are You Ready to Build a New Product?
Backstage is a framework to build an internal developer portal, but it is not an off-the-shelf tool or a solution. Many adopters have not understood what this means, at least at first.

As a framework to build a tool, Backstage requires dedicated product and engineering resources to configure, customize and maintain. It is not uncommon for organizations to require four to six or more fully dedicated resources to succeed with Backstage. The engineers you dedicate to building this framework into a product will need to be skilled in TypeScript and React, and willing to work with an open source community and product.

Getting up and running with Backstage involves more than 70 steps. It is not intuitive and can result in months of work before you can demonstrate value. Some of the steps and associated time commitments are obvious, such as deploying your instance of Backstage. Other steps may not be obvious to new adopters.

For example, Backstage has historically relied on open-source plugins to enable organizations to connect their tools. The open-source nature of these plugins has been a key selling point of Backstage; however, the plugins have uneven quality and depth and can require additional customization to enable basic functionality like search. In another example, tuning the Backstage data model to your organization is so difficult that many adopters abandon hope and work with the default data model. Upgrading versions and plugins and the effort required to keep the data in Backstage synced with your system contribute to the engineering resource load required to succeed.

The advantage of Backstage is the infinite customization you can accomplish thanks to its open-source code base. You can change anything you want if you write and maintain it. This includes adding custom data sources, customizing the data model, extending plugin functionality and having the UI reflect your corporate brand identity. Again, the trade-off is time and complexity relative to your alternatives. For instance, configure8 and Port offer the flexibility to tailor the portal’s data model to the contours of your organization and process any custom data source for use throughout the porta. Several vendors can support custom logos and color schemes in the portal’s UI (or you can build a more tailored UI on top of their APIs).

Does it Support Your Scenarios of Interest?
Internal developer portals as a product category have evolved well beyond their microservice catalog roots. If you are considering Backstage, it is important to ensure it offers functionality that addresses the needs of your enterprise’s stakeholders.

For example, many organizations leverage functionality called Scorecards to define and measure teams’ compliance with reliability standards, security and architecture. Open source Backstage has a version of this functionality, but it is not intuitive and has a fairly high difficulty level (i.e., writing code, writing data collectors, etc.). The commercial vendor community typically all offer Scorecards. Adopters should consider the breadth and depth of information that can be checked, the ease of creating new scorecards, and the quality of the developer workflow and UI when evaluating Scorecards.

In another example, Backstage enables users to scaffold new services via the Cookiecutter open-source package. Why take a principled dependency on a code generation solution instead of an open approach that enables you to use the best tool for the job? There is a robust landscape of available solutions to generate code and orchestrate workflows. Having a flexible way to directly integrate these reduces complexity.

Backstage also offers functionality that enables developers to see the cloud costs of the code they operate. Upon closer inspection, adopters find this module only enables high-level aggregations of costs over a fixed timeframe. To truly provide developers with the knowledge and functionality they need to improve costs, consider closed-source vendors that offer more granular insights and actions, such as drilling in beyond aggregate costs at the microservice level to see the per resource cost drivers, an individual resource’s cost drivers, config settings and blast radius and weave in self-serve actions so developers can safely make changes where appropriate and permitted.

The resource and environment catalog is another area to consider. Is it necessary to have the environments and resources that power them automatically mapped to each service? Do you want a centralized catalog of your resources across clouds and data centers that you can sort and filter using various keys? Backstage can support some of these scenarios, but not without significant work and at present, the connection to this data is brittle and manual versus automatic. This stands in contrast to certain closed-source vendors, such as configure8 which has extensive functionality.

In a prior section, we discussed the importance of data model flexibility. This is an important evaluation criterion for organizations that want the flexibility to push in and present any data source, to customize the schema of the catalog, and finally to insert custom keys for results filtration and analysis (i.e., does this resource touch personally identifiable information; show me results related to a line of business, etc.). Ensure you understand whether you are poised to succeed in tailoring Backstage’s data model, which has stymied many adopters, and that any commercial vendor you evaluate can easily support customizations.

Another consideration is Backstage’s TechDocs, and whether you want to adopt TechDocs for your entire enterprise or allow teams to use best-of-breed solutions. Whereas Backstage locks you into TechDocs (which can also make a future migration to a new portal solution more difficult once end-user behavior becomes entrenched), commercial vendors generally support open and flexible approaches that enable integration with any documentation solution.

Also, ensure Backstage meets your information security needs. For example, most organizations desire a robust role-based access control model so developers only see permitted services, scorecards and actions in the UI, via search and the API. Backstage does not offer this functionality in the open-source edition. There is a framework to work with permissions included in Backstage but to get real value out of it, you need to budget to license a closed-source RBAC solution from Spotify or build it yourself.

Finally, evaluate the pace of innovation you desire from your internal developer solution. Closed source vendors, two in particular, have demonstrated an extremely rapid pace of innovation, launching many industry firsts. Compare this to the pace of innovation in the open source community as internal developer portals are sticky once installed and tend to be long-term relationships.

Are You Prepared for Success?
Achieving broad internal adoption of an internal developer portal requires more than getting a portal configured, it requires solving the pain points that matter to your organization, demonstrating success, making the organization aware of that success, and creating a feedback loop to overcome challenges. To accomplish this, you must be aware of best practices, create content and forums to engage the organization, and ultimately adopt a product-oriented approach to ensure your platform engineering effort succeeds.

Success in this area often combines budgeting and access to best practices.

The Backstage community is vibrant and active, and you can pick up best practices by engaging with community members. But for enterprises and internal champions who need to succeed in driving the adoption of a new internal developer portal, you may wish to consider alternatives to community support.

This can be accomplished via multiple methods. Consulting firms and individual practitioners specialize in this area and can coach you. Organizations above a certain size often invest in dedicated product management resources to own this function. Finally, certain vendors like configure8 focus on delivering success versus a tool (or framework to build a tool like Backstage) by providing consultative best practices, content and other value-adds packaged with their software to ensure you achieve robust adoption levels.

Finally, to the extent your scenarios of interest involve self-serve actions to alleviate TicketOps, ensure that you have budgeted the appropriate engineering investment to develop the golden paths or blueprints that are vended inside an internal developer portal. Certain vendors like configure8 offer a solutions layer that creates custom golden paths for you. However you procure golden paths, you must balance your investment to ensure you get a great “razor” and all of the “razor blades” you will need to support the unique needs of your various internal customers.

Do You Understand Your Total Cost of Ownership and Alternatives?
One of the considerations of Backstage is understanding its total cost of ownership versus its best alternative.

While Backstage is open source and free, we have noted that a meaningful engineering investment is required for installation, customization, maintenance, internal support and product management. Quantify the fully loaded cost (i.e., salary, taxes, benefits, facility charges, etc.) of each human resource required to succeed with Backstage.

We have noted there may be licensing costs from Spotify for any essential features you do not wish to build yourself. There will be internal hosting costs and budgeting for any consulting costs.

Several enterprises who have evaluated Backstage now realize the total cost of ownership of alternative vendors is much lower and more predictable. Leading vendors offer robust alternatives for catalog and scorecarding, self-serve actions, and collaborative cost management, and two vendors have evolved to provide compelling functionality and high degrees of flexibility. The key is finding a vendor who balances overall functionality and flexibility with ease of use, speed to value, and enterprise-grade support. This is where configure8 has struck a unique balance in addition to a lower cost of ownership relative to Backstage.

Finally, any return on investment analysis must capture the opportunity costs. Here are some common examples:

• Time to market trade-offs. Setting up Backstage for your proof of concept can take over a month, getting to a first releasable version can take 3-6 months, and full adoption can take 6-12 months. During this time, you may generate no or minimal business value. Specialist vendors can deliver success faster. The incremental business value you can generate with a specialist vendor versus Backstage related to ‘speed to value’ represents an opportunity cost you need to quantify.
• Foregone features: Scenarios like Scorecards or RBAC that are important to your organization and you will have to pay for or forgo, you need to factor these incremental costs (or the cost of not having these features) into your analysis.
• Opportunity cost of internal engineering resources: How many engineers are required to support Backstage over and above what’s needed to succeed with a specialist vendor? What business value could these incremental engineers have generated by instead supporting the unique value your enterprise brings to its customers versus an internal tool in a world of compelling alternatives? This becomes another cost of owning Backstage you need to compare v. the cost of buying a solution from a vendor.

Understanding the direct and opportunity costs of your best alternatives will help you put your budget to work in the most efficient way and aid in your internal discussions with finance and other stakeholders.

Carefully Consider Any Clones

Given the initial popularity of Backstage, it is not surprising to see clones emerge built on Backstage. A clone is a vendor who has taken Backstage source code and added some feature crust on top, often a security wrapper, hosting and maintenance, light UI customizations, and potentially filling in a functionality gap like Scorecards. This approach can alleviate some of the burdens of open source Backstage in exchange for licensing costs you incur, it is important to understand if the clone is the best fit for your unique needs given the highly competitive vendor landscape that is now available. An easy way to think about it might be comparing Jenkins versus hosted Jenkins versus Jenkins with a vendor security wrapper relative to what enterprises get with Azure DevOps and GitHub Actions.

Final Thoughts

Building an internal developer portal from an open-source framework can be rewarding for organizations that can achieve value from extensive customization that is impossible elsewhere. For many organizations, however, it is important to properly scope the effort required to succeed with Backstage and compare it to your alternatives in the vendor community. While Backstage does offer certain advantages, DevEx and platform teams are increasingly finding greater success more quickly and at a lower total cost of ownership via specialist vendors like configure8.