devopsy.kinsta.cloud

Author: Bill Doerrfeld

How to Get Developer Productivity Engineering Right

Executives want a clear measurement of developer productivity, but determining it is proving elusive. Because how do you even define software development productivity? Is it the number of points you completed during a sprint? Lines of code or bugs fixed? Features correlated with tangible business outcomes? Or, is it the satisfaction you feel after a day’s job well done? Or is productivity shaped by the social cues you get from your colleagues?

When Brian Houck, principal productivity engineer, Microsoft, asked this question to the thousands of internal developers at Microsoft, he was surprised to see hundreds of different answers across all the above areas. “Developer productivity is many things and means different things to different people,” said Houck.

I recently attended DPE Forum: NYC, which brought together leaders in the burgeoning field of developer productivity engineering (DPE) to share their experiences implementing DPE in practice. Hosted by Gradle on October 3 at the Nasdaq MarketSite building, the event was an eye-opener into the science behind DPE, data-backed internal studies on developer productivity, and what engineering managers are actively doing to improve efficiencies.

Below, I’ll highlight some key takeaways from talks at the DPE Forum to help you get an edge on this emerging practice. We’ll end with some actionable developer productivity tips that you can apply to your own engineering culture.

Reduce Feedback Cycles to Reduce Cognitive Fatigue

According to Hans Dockter, CEO and founder, Gradle Inc., developers see a huge latent potential in optimizing their toolchain. But, although engineering leadership is becoming more open to new DPE initiatives, they tend not to feel as strongly about its potential. Leaders may not see anything wrong with waiting for a ten-minute build, telling developers to “just check your email while you wait.”

Yet, studies have found that the productivity of multitasking teams is far less than that of teams that focus on singular tasks. Furthermore, cognitive control tasks, in which the brain must build new neural pathways, contribute to added mental fatigue. The pervasive ‘multitasking myth’ leads to a superficial perspective that is a blocker to improving the toolchain processes, said Dockter. “When your muscle is exhausted, it’s a biological effect.”

So, how can we reduce cognitive fatigue? Well, Dockter attributes cognitive fatigue and burnout in part to the lack of quick feedback while testing new builds. Code verification failures, flaky tests and failed CI/CD builds can really yank developers out of a ‘flow state.’ And he’s found that when the feedback cycle is longer than ten minutes, you tend to get more task-switching, accelerating cognitive fatigue and decreasing productivity.

The benefit of investing in DPE, he said, is reducing this feedback cycle, thus reducing context switching that contributes to cognitive dissonance. “You cannot have a great developer experience without a great toolchain experience. And that’s what DPE is about.”

Case Study: Decreasing Cycle Time at Netflix

Engineering leaders are actively looking for ways to keep developers in the “inner loop” with fewer interruptions. Netflix, for example, has been investing in ways to improve developer experience around their toolchain, like instigating automated retries for flaky tests. Aubrey Chipman, senior software engineer, Netflix, shared how Netflix Engineering is actively configuring and optimizing their testing processes to reduce this cycle time.

As part of these endeavors, many Netflix projects now use Testcontainers to optimize testing. They also use Gradle’s Develocity for predictive test selection (PTS), which Chipman estimated has saved roughly 30 thousand potential hour savings in a year.

Netflix is also toying with remote test execution (RTE) as a beta offering, which distributes tests across remote agents to execute them faster. However, Chipman acknowledged that parallel executions come with potential limitations, such as network security issues and debugging for different environments.

Build a Systematic Approach to Developer Velocity

Although all teams are special snowflakes, there are definitely common requirements they all share. This is why the idea of an internal developer platform (IDP) has seen greater interest lately. IDP architectures deliver common ‘paved roads’ to internal developers with guardrails to remove friction within the software development life cycle (SDLC).

Amado Gramajo, VP, infrastructure and DevOps engineering, Nasdaq, defined DPE as enhancing developer productivity and happiness through accelerated technologies. And to him, the SDLC really has focused on improving the Ops side in recent years. Now, there is a big opportunity to focus on improving the build and testing aspects of this toolchain.

To distribute common features, Nasdaq adopts a systematic approach to developer velocity, as he called it. They haven’t created a centralized team per se but have developed a shared service available to all developers to help them improve velocity. This global service cuts across otherwise siloed teams, bringing them shared requirements, like resiliency and compliance features. As areas like Kubernetes, SRE and containerization become more normalized, said Gramajo, we need to build the operational controls so things can be automated.

Case Study: Tracking Toolchain Data at GitLab

Some are taking the idea of a systemizing developer velocity quite literally. For instance, Lee Faus, global field CTO, GitLab, described the concept of operating a ‘software factory.’ How can we optimize the supply chain to figure out how to produce artifacts that customers are happy with every single time, just like Oreo cookies?

Well, Faus identified a core issue in webhooks, which he referred to as “the dumpster fire of our toolchain,” he said. Every tool emits different events, which are produced by many actions within the toolchain. However, if this data isn’t properly triggered or received correctly, you could easily have data corruption, causing issues in the testing process.

Therefore, Faus recommended properly handling event data with solutions that enable bi-directionality and avoid creating data silos. Ideally, collecting data related to the software development process should be traversable to help developers pinpoint where problems emerge from. And this root cause analysis should be tied to actionable outcomes for remediation.

Base DPE Descisions on the Data

Although defining productivity is challenging, we have data to back up certain actions leaders can take to increase developers’ perceived productivity and satisfaction. After surveying thousands of internal developers at Microsoft, Houck was able to make some data-backed recommendations.

Formerly Set Aside ‘Focus Time’

The first finding isn’t surprising. Developers like to code. They don’t like things that distract them. Houck’s research found that the amount of time spent coding was positively correlated to developer satisfaction. Engineers who formerly blocked off uninterrupted time to code (especially in the morning) are far more likely to say they are productive. This perceived productivity remains, regardless of whether focus time is chosen or mandated.

Practice Meeting Hygiene

One way to free developers up for more concentrated work is to reduce their time in low-quality meetings. These could be recurring meetings that have outlived their usefulness or ones where the information is not relevant in the first place. Yet, interestingly, not all meetings are bad. For example, Houck’s research shows developers are 2.5x more satisfied with one-on-one syncs with their direct manager.

Allow Flexible Work Arrangements

Houck’s research found that happiness and productivity linearly decline as developers are asked to go into the office more often. The findings are clear—remote workers are more likely to feel more productive than fully on-site workers. Furthermore, the developers who don’t have agency in where they work feel 11% less productive. While there are certainly benefits to in-person brainstorming and team-building, leaders shouldn’t force engineers back into the office for an arbitrary number of days.

Final Thoughts

Measuring developer productivity is a tricky art. Therefore, Houck recommends choosing a few metrics, such as developer satisfaction, the amount of dedicated focus time developers have and the degree of friction within their workflows.

Similarly, Dockter recommended measuring the obstacles and hidden costs in the developer toolchain—obstacles like slow build and testing cycles—that can be readily removed to logically enhance productivity. “The toolchain gives you indications that certain teams are not as mature as others,” he said. Implementing DPE is steady progress, not firefighting, so you’ll probably want to sneak these changes in slowly, he added.

Fitter, happier, more productive. Developer productivity engineering is an emerging practice with a lot of science and opinions surrounding it. But there’s also the human factor to consider. A solid engineering culture will require meeting people halfway and giving them purpose and incentives to retain their interest.

Finally, anyone’s bound to get burned out sometimes with work. At least by minimizing the unnecessary toil in the toolchain, we can encourage a more satisfied developer experience that keeps up high productivity—however you want to define it.

October 10, 2023
Tips For Managing Internal Developer Platforms (IDPs)

Platform engineering has emerged as a discipline to centralize configurations and standardize the DevOps process. As part of this practice, internal developer platforms (IDPs) are a key ingredient to establishing a paved road or “golden path” for developer users. But how do we ensure these IDPs are designed and rolled out correctly?

Well, the IDP concept is still relatively new, but some best practices are beginning to emerge. For example, treating the platform as a product helps encourage proper feedback. Reference architectures can guide initial designs. And testing first iterations with a single team can encourage buy-in around a proof of concept.

I recently chatted with internal developer platform experts, including Luca Galante, product and growth, Humanitec, and Markus Eisele, senior principal product marketing manager, Red Hat, to gather insights on how to best manage IDPs. Below, we’ll consider how to create golden paths that help optimize developer productivity and happiness and consider how these techniques affect the bottom line.

Start With a Reference Architecture

First, it’s a good idea to begin with a reference architecture when creating an IDP. This can help frame the initial infrastructure and its constituent components. For example, McKinsey recently debuted a sample IDP architecture, which has been reinterpreted for Azure and GCP. Platform engineering also maintains a tooling landscape that lists tool options for the various planes.

It’s good to note that an IDP is not a single product, like a platform-as-a-service. “An IDP is not a turnkey solution,” said Galante. “You can’t buy an IDP.” Therefore, there is not a one-size-fits-all enterprise-grade option that covers the entire stack. An IDP will most likely be a combination of open source and proprietary components.

According to Eisele, these new reference architectures are a good step in the right direction. “These IDP reference architectures are developer-centered views about modern application development platforms,” he said. “They shift the perspective from a sheer infrastructure perspective into what matters most for the developer.” And while a one-size-fits-all platform may not exist, the tools to build them do, such as Spotify’s Backstage or Red Hat’s Developer Hub, a platform for building developer portals.

Treat the Platform as a Product

Treating the platform as a product is another commonly shared best practice in platform engineering circles. “This is something that will make or break an IDP initiative,” said Galante. According to Galante, this platform shouldn’t be architected and then taught to developers in a top-down fashion—this can result in lengthy onboarding, broken functionality and ShadowOps.

Instead, successful platform teams take an existing team’s work and restructure it around a product manager to evangelize, said Galante. Another option is to keep your infrastructure team and develop a separate product team to start from scratch. Regardless of the exact organizational makeup, the product mindset is crucial for kickstarting and maintaining IDPs.

Have a Tight Feedback Loop

Part and parcel of retaining a product mindset is conducting research and having a tight feedback loop with stakeholders. For IDPs, this means understanding what developers need and not building on assumptions alone. According to Galante, all stakeholders should be given a voice, including DevOps teams, developer senior backend developers, junior front-end developers and even executives. Listen to these groups to understand their needs to frame the IDP correctly and sell them on the value of a platform. This is where the skillset of a PM is crucial to navigate these kinds of dynamics, he added.

“IDPs are about enabling developers,” said Eisele. While these tools should make them more efficient, they should also help them love their job more despite the cognitive load, he said.

First Attempt With a Lighthouse Team

Before rolling out a standardized internal development platform company-wide, it’s good to validate it in practice with a smaller group. Galante calls this a “lighthouse team.” Ideally, this team is more advanced with their deployment pipelines and is well-versed with cloud-native technologies, he said. This should help establish the right golden path with minimum friction and stress test it under real-world scenarios. After an IDP is validated in practice, rolling it out across other teams will be much easier.

Centralize the Platform Team Itself

Another facet to consider is the organizational makeup. Who will architect and support the IDP? Galante believes the platform team should be centralized instead of having various platform teams alongside each department or division, and such a team should consult various stakeholders throughout the application lifecycle. He also references Team Toplogies as a helpful resource for implementing platform teams.

At the same time, Galante admitted that at scale, large enterprises might spin up multiple platform teams simultaneously. Sometimes, this is done intentionally and could even bring positive competition to improve operations, he said.

Plan For Flexibility

Lastly, the big takeaway is that IDPs are not a silver bullet, and a reference architecture that works great for small agile teams might not transition well into a complex enterprise with much brownfield development. Furthermore, you run the risk of over-specificity, too, which could step on people’s toes.

Therefore, the platform must be flexible enough to adapt to new situations. Galante recommends first looking at your configuration management to see where the most time is wasted and ensuring you standardize the most common configurations.

Setting Golden Paths For IDP Success

Developer productivity practices have emerged as a response to exploding complexity in the last ten years, said Eisele. The old days are gone, and a brand new execution stack is required to architect and deploy applications. “All this led to a point where developers just want to get back to coding,” he said.

The golden path, or paved road as Netflix calls it, is like an approved path for traveling through the software development lifecycle. As Eisele puts it, it’s a form of best practices and a new way to replicate development knowledge across an organization.

“IDPs seem to be natural partners to cloud/standardized developer environments (CDE/SDEs),” said Ivan Burazin, CEO and co-founder at Daytona. He pointed out how Uber’s developer satisfaction increased dramatically upon adopting a platform called DevPod, an integral part of their IDP. He also cites successful CDE implementations at AirBnb as further proof of the power of internal developer platforms.

But to create these golden paths, IDP architects have their work cut out for them. They must juggle the input from multiple stakeholders and simultaneously balance complexity and abstraction effectively. The platform itself must be specific enough to reduce developer cognitive load yet generic enough to be flexible for outliers.

Thankfully, the aforementioned tools and best practices should help efforts toward designing golden paths for your internal developers. The next step will be testing the platform and measuring the success of these initiatives.

October 5, 2023
3 Steps to Secure Your CI/CD Pipelines

More and more software teams are waking up to the fact that their CI/CD pipelines are prone to risk. These pipelines often interface with various toolsets and automation, which may be prone to misconfigurations and abuse. This is not to mention the potential for insecure code being pushed through this funnel and external factors that threaten to circumvent the pipeline.

In recent years, these risks produced some CI/CD tooling breaches—concerns that gained high-level attention recently after a joint CISA and the NSA alert about the importance of securing CI/CD pipelines. “This is something that moves the tectonic plates of the industry,” said Daniel Krivelevich, former founder of Cider Security and current CTO of AppSec at Palo Alto Networks.

I reconnected with Krivelevich, the ‘godfather of CI/CD security,’ to reevaluate the CI/CD threat landscape and understand how IT should respond. According to Krivelevich, we’re now seeing more maturity around CI/CD security, both in terms of recognizing that these risks exist as well as emerging frameworks to help engineering teams respond. He shared a general three-step framework organizations can use to secure the CI/CD pipeline and surrounding areas, which we’ll explore below.

The Evolving Threat Landscape

The traditional approach to application security (AppSec) used to be simpler. Defenders could implement more standard controls to make their code more secure, said Krivelevich. However, throughout the years, the software engineering process has transformed dramatically. It’s become faster, dynamic and reliant upon third-party frameworks. “The entropy of the engineering ecosystem is very high,” he said.

This paradigm shift has introduced a slew of new risks, which requires rethinking since most traditional security solutions deal solely with AppSec. “Software engineering is a big part of what makes the business tick,” said Daniel. Therefore, having a common language and understanding of this area is crucial to respond to evolving risks that threaten business operations.

A Three-Step Framework For CI/CD Security

Krivelevich is involved with creating the OWASP Top 10 CI/CD Security Risks; therefore, he has excellent insight into the types of vulnerabilities that most engineering practices and their CI/CD pipelines are facing. His three-step framework for CI/CD security, which Palo Alto Networks has applied in practice, looks to holistically secure not only the pipeline itself but also internal and external factors.

1. Security in the Pipeline (SIP)

SIP can be thought of as the traditional AppSec problem space. This involves ensuring developers are not shipping anything inherently insecure into production. Therefore, whatever code flows through the CI/CD pipeline must be vetted to ensure it has no security flaws or misconfigurations. Of course, this is easier said than done, given that teams are juggling more technologies and frameworks than ever, each of which has different knobs and requires unique security scanning capabilities.

2. Security of the Pipeline (SOP)

This area has to do with the security of the software delivery chain itself. Attackers may attempt to breach these pipelines by running malicious code within the pipeline or stealing credentials to CI/CD tools to harm the system. Poisoned dependencies, access management holes or insufficient credential hygiene could all be doorways to disrupting a CI/CD process. Therefore, SOP focuses on the security posture of the actual systems and tools that compromise the software delivery chain.

3. Security Around the Pipeline (SAP)

However, even if SIP and SOP are secure, what if hackers can bypass the pipeline altogether? That would render even a perfect set of controls null and void. Therefore, SAP is necessary to ensure that attackers can’t push code directly into production. It’s good to see the bigger picture so that everything that reaches production actually originates from the pipeline.

To deploy SAP effectively, Krivelevich recommended leveraging both detective and preventative measures. You need detective measures to issue alerts when anomalies occur, such as an artifact deployed by a human account instead of a service account. You also need preventative measures that set the proper branch protection rules across all repositories. For instance, this could be a rule that no single developer can publish to production unless approved by two reviewers.

The Role of AI in Software Security and CI/CD

By using both detective and preventative measures, you can detect when settings are disabled or abused and have better configurations across all stages of the software delivery chain. But what other tactics can be used to secure CI/CD? For example, does machine learning have any merit in this area?

Well, Krivelevich described AI as “both a blessing and a curse.” Attackers are already using AI to operate much faster and in a much more agile way. And unfortunately, he viewed the engineering ecosystem as the new path of least resistance. “I expect the power of AI will be used by sophisticated attackers who are motivated to get access to the ‘crown jewels’ by abusing the engineering ecosystem,” he said.

On the defensive side, Daniel doesn’t see a specific opportunity for deploying AI to protect CI/CD. That being said, there’s a lot of potential for security builders to utilize AI to advance their efforts and accelerate the pace at which they detect anomalies, which could help respond more quickly to emerging threats.

Final Thoughts on CI/CD Security

The recent chatter around CI/CD threats appears to be garnering more attention around securing this niche area of agile software development. To thwart CI/CD attacks, it will be necessary to assess the settings of systems, understand their technical natures and analyze them against an attacker’s perspective, Krivelevich explained. Therefore, understanding what realistic attacks can be carried out will help build measures to correct these gaps.

In doing so, the SIP, SOP and SAP descriptions serve as a solid reminder of what needs to be addressed when attempting to secure the software delivery process and the supply chain at large. Code that goes through the pipeline needs to be free from vulnerabilities, and the channel itself must be vetted for gaps that could be abused. We also need to consider the possible existential threats if the entire process can be undermined.

September 26, 2023
Innersourcing Open Source Principles in the Enterprise
Software development has experienced a seismic shift in recent years, with open source becoming the heartbeat of innovation. A full 90% of enterprise IT leaders say their organization is using open source, according to a Red Hat report. Leading this transformation is GitHub, the ubiquitous platform that not only jumpstarted the open source movement but has also been quietly transforming how enterprises approach development at large.

But it’s not just tech companies trumpeting the open source creed—academics are on board, too. A paper recently published by the University of Georgia found a hard-to-dismiss correlation between open source innovation and overall business success. “We find a significantly positive association between GitHub activity and future realized net income, suggesting that open innovation is a strong predictor of future earnings,” according to the report.

I recently met with Chris Reddington, senior manager of developer advocacy at GitHub, to dive into how and why open source is becoming essential for enterprise innovation. According to Reddington, the once-dominant proprietary information mindset is shifting, with open source cultural principles now permeating most companies, a practice also known as innersource. Put simply, open source is now cool in the enterprise.

Bridging the Open Source-Enterprise Divide

GitHub’s close ties to the open source space have allowed them to illuminate the parallels between the open source and enterprise worlds. Reddington noted that, nowadays, open source and enterprise aren’t really separate communities but rather intertwined facets of the same software ecosystem. This movement is represented by the open source program office (OSPO) becoming more commonplace in large organizations.

For an example of innersource in practice, take Porsche, who recently announced the Porsche FOSS Movement to guide internal common values and development principles around open source. We’ve also seen similar large-scale commitments to open source values at Walmart, Spotify, Facebook and others.

Open source clearly isn’t just for startups—large companies are increasingly seeing an advantage in both using open source software and sharing their own open source components. The growing investment and maturation of OSS initiatives are positive strides, as it’s important to the stability of the free open source software (FOSS) ecosystem as a whole that those using software contribute back to its maintenance, especially to core packages the world depends upon.

The Power of Innersource

The study from the University of Georgia brings to light how enterprises increasingly recognize the need to work in sync with their developers and customers to remain competitive. This is a testament to the evolving nature of development—an industry that’s embracing a “team sport” mentality instead of a siloed approach, explained Reddington. And as enterprises align with open source principles, they are discovering the power of innersource, he said. “Innersource is a development methodology where engineers build proprietary software using best practices from large-scale open source projects, like Kubernetes or Microsoft’s Visual Studio Code,” GitHub explained.

Reddington elaborated on two key elements of this approach:
- Breaking down silos: The traditional organization-centric structure of projects is being replaced by a more transparent and collaborative approach. Innersource encourages transparency and asynchronous communication, tearing down barriers that previously inhibited efficient collaboration.
- Engaging responsibly: Reddington emphasized the significance of enterprises giving back to the open source community. Whether through sponsorships, allocating time for engineers to contribute, or open sourcing their projects, companies can foster symbiotic relationships where they benefit from and contribute to the open source realm.
Essentially, by adopting innersource principles, enterprises foster an environment that promotes collaboration and knowledge sharing, thus enabling innovation to flourish across teams and projects.

Balancing Open Innovation and Proprietary Software

One of the most intriguing aspects of this transformation is finding the equilibrium between open innovation and proprietary software. Reddington acknowledged the necessity for businesses to generate profits, which inherently stems from delivering value.

While reinventing the wheel may be tempting, Reddington highlighted the importance of recognizing existing solutions and incorporating them into ongoing projects. This pragmatic approach maximizes efficiency and enables teams to focus on the unique, value-generating aspects of their work.

Organizations should also consider what software of their own could be open sourced. From a business perspective, this most likely won’t be the “crown jewels,” said Reddington, but ancillary components that others could benefit from. For example, had it not been for Google open sourcing Kubernetes, Lyft open sourcing Envoy Proxy or Facebook sharing GraphQL, we would not have these standard components for all to use.

Open Source Movements: Past, Present, and Future

Over the last decade, we’ve seen significant, more institutional maturation around open source projects. Reddington pointed to the Cloud Native Computing Foundation (CNCF) as a standout. He specifically noted Kubernetes-based Event Driven Autoscaler (KEDA) as a cool project to help scale Kubernetes containers based on the number of events needing to be processed.

He also underscored the surge of interest in generative AI, evidenced by projects like GitHub Copilot. It will be curious to see how AI assistants incorporate open source at their core, such as open foundational models and community-supported datasets.

Potential Setbacks To The Open Source and Innersource Movement

In the dynamic realm of enterprise-scale software development, nurturing innovation can be a complex endeavor. Innersource principles are one way to change the attitude toward development, making it more efficient, collaborative and aligned to the current and future developer experience expectations.

Yet, this isn’t to say we haven’t had some setbacks recently that could threaten the progress the OSS community has built. For example, HashiCorp recently abandoned its open source license around Terraform in favor of a business source license (BSL). There is also an increasing number of supply chain-related attacks that threaten common open source libraries.

Gazing Into the Future

As Reddington aptly summarized, we’re still in the early days of this transformative journey. The developer community’s inclination for experimentation will undoubtedly drive the solutions to the most pressing challenges in open source. And with enterprise adoption on the rise, the future promises more cohesive collaboration, broader adoption of innersource principles and continued innovation driven by open source values.

Interestingly, we’re seeing more synergy between open source and enterprise domains, which used to feel like separate realms entirely. And it will be interesting to see how businesses continue to balance transparency and proprietary value into the future. No matter what, the lines between them are blurring, enabling companies to harness the power of innersource for enhanced collaboration and innovation.
September 21, 2023
Should You Measure Developer Productivity?
A recent McKinsey article entitled ‘Yes, you can measure software developer productivity‘ has got the software developer community talking. And the general consensus is they’re not happy.

Most software developers are not enthusiastic about the prospect of having their individual performance metrics monitored. “Anyone who attempts to measure software in some sort of scientific way is just doing the wrong thing,” said YouTube personality ThePrimeTime. Dave Farley of Continuous Delivery even goes as far as to call the McKinsey report “harmful dribble.”

This debate arises as developer productivity engineering (DPE) is gaining traction as a means to improve developer performances and, perhaps, in response to employment landscape shifts like the Great Resignation, which saw developers and DevOps pros leaving their roles. Companies like Spotify, LinkedIn, Netflix and others are also investing in internal developer platforms (IDPs) to increase developer productivity and satisfaction amid a limited talent supply.

Alongside these ambitions, much has been written about how managers can measure developer productivity in terms of qualitative metrics, like developer satisfaction, as well as quantitative metrics, like number of commits or story points, deployment frequency or failure rates. But can tracking such metrics deliver tangible results? Or do they go too far?

Below, we’ll consider whether you can actually measure software developer productivity. But more than that, we’ll evaluate whether this is a meaningful tactic with desirable outcomes.

Ways to Measure Development Productivity: DORA

First, let’s look at some of the standard techniques to measure development productivity. The DORA metrics, produced by the DevOps Research and Assessment (DORA) team at Google after years of research, identified four key metrics:
1. Deployment frequency: How often you ship changes.
2. Lead time for changes: The time between a commit and it reaching production.
3. Change failure rate: The percent of releases that fail; downtime, worsened service or rollbacks.
4. Time to restore service: The average time it takes to respond to an issue, also known as mean-time-to-recovery.
These metrics could be helpful indicators of a software development team’s overall agility and health. They can signal bottlenecks in the CI/CD process or highlight where too much time is spent remediating issues, encouraging positive changes. Solving these root causes could all improve a team’s productivity.

Going Beyond DORA: SPACE

DORA also added reliability to the list. That’s understandable, given the interest in building reliable systems. But things start to get a little more complex when SPACE enters the equation.

Proposed in a 2021 paper by GitHub researchers, SPACE augments DORA with the following attributes:
- Satisfaction and well-being
- Performance
- Activity
- Communication and collaboration
- Efficiency and flow
Metrics like satisfaction or communication and collaboration are pretty subjective. Plus, it can be challenging to gather this data and encourage engineers to speak honestly in surveys. Measuring the activity of individual members, like the number of issued closed or pull requests made, isn’t exactly tied to end value and could even produce a negative psychological effect.

“Consequently, teams can become fixated on them, as seemingly positive changes are relatively easy to achieve,” wrote James Walker. He adds that SPACE still doesn’t tell you why measurements have value—furthermore, it lacks actionable information and might be tricky to implement, given developer pushback against performance analysis systems.

Opportunity-Focused Metrics

The McKinsey report adds to the mix what it calls “opportunity-focused productivity metrics,” which include inner/outer loop time spent, a Developer Velocity Index benchmark, contribution analysis and a talent capability score. (It’s not totally clear how these metrics can be computed). It then organizes all these metrics into a framework that further segments measurements into system, team and individual levels.

While some believe in the framework’s legitimacy, many commentators don’t see the practicality of implementing such a framework in practice. “Introducing the kind of framework that McKinsey is proposing is wrong-headed and certain to backfire,” wrote the PragmaticEngineer.com. “The report is so absurd and naive that it makes no sense to critique it in detail,” wrote Ken Beck.

Things to Avoid With Developer Productivity Metrics

Apparently, developers are very weary of being measured. And given the recent backlash, it seems like overreliance on productivity metrics could negatively impact the engineering culture. So, engineering leadership should be careful when implementing these sorts of tactics. Here are some anti-patterns to consider.

Don’t make it about individual performances. Metrics like DORA have become standards for measuring the efficiency of software teams, but they shouldn’t be applied at the individual level. “Measuring software development in terms of individual developer productivity is a terrible idea,” said Dave Farley. Technology development is more of a collaborative process; performance metrics should reflect how the team operates.

Don’t measure developer productivity in isolation. On that note, performance metrics shouldn’t be judged in isolation. Issues with failures, for example, often depend on contexts outside of an individual developer’s control. At the same time, it can be hard to correlate individual developer contributions to end customer satisfaction.

Don’t create an opportunity to gamify productivity systems. Leadership shouldn’t directly tie a developer’s salary, reputation or position to whether or not they hit certain metrics. If productivity systems carry this much weight, they could become gamefied.

Don’t put in more effort than it’s worth. In tech, where there’s hype, there’s money to be made—and a host of vendors ready to hawk monitoring tools. What’s more, to enact these productivity measurement enhancements, McKinsey noted that “most companies’ tech stacks will require potentially extensive reconfiguration.” Devoting many engineering resources to monitor engineers just seems like a big pill to swallow.

Don’t put too much faith in metrics. Lastly, metrics only show one side of the picture. They are only data points and can’t always depict what it’s really like on the ground. Furthermore, qualitative responses also might be skewed due to reluctance to speak up.

Remember Goodhart’s Law and Focus On Quality

Personally, I believe that measuring things like developer productivity, efficiency, experience and satisfaction does have some merit for some businesses. Most teams are now working asynchronously, and metrics help leaders take the pulse of a team’s performance, even if they are all remote workers.

It’s also nothing new. Leadership has been attempting to measure developer productivity for years. But, these metrics often lack context and come with many caveats to consider. Thus, they shouldn’t be given a huge seat at the table.

I like this ‘law’ from British economist Charles Goodhart, which is colloquially summarized as “when a measure becomes a target, it ceases to be a good measure.” In other words, when you start working for the metrics, the engineering culture and overall code quality might decline. This matters, since a separate study found that increases in code quality tend to be followed by an increase in developer productivity.

So, focus first on establishing code quality and, hopefully, productivity will flow.
September 18, 2023
Tips for Increasing Developer Velocity in the AI Age

These days, everyone is trying to improve release frequencies. Under mounting pressure to deliver more, many software teams are considering new strategies to enhance their developer velocity, like AI-assisted code development and platform engineering. AI has been a boon for automatic code generation, but all the code in the world doesn’t matter if you can’t deploy it into production efficiently.

Another point is that velocity isn’t always measured productively, said Dori Exterman, chief evangelist of Incredibuild and a self-proclaimed ‘productivity freak.’ Too often, managers place value on quantitative elements, such as points per sprint and unrealistic metrics, when they really should be considering the quality of these components and how easy they are to maintain in the long run.

I recently chatted with Exterman and other experts about tactics organizations can use to enhance their development velocity. Below, we’ll consider some helpful methods, from optimizing testing automation to creating internal development platforms (IDPs) and evolving your CI/CD processes. Essentially, optimizing the plumbing will be necessary to match the productivity of instant code generation—otherwise, the vision of continual release will remain stunted.

New AI, Same Plumbing…

No tech article today can mention developer productivity enhancements without a nod to AI. Just take the interest in Copilot, for example. Among GitHub Copilot users, 40% of the code they’re checking in is now AI-generated and unmodified, according to a Microsoft executive. And GitHub reported that 88% of Copilot users feel more productive when using the tool.

In contrast to past hype cycles, the latest AI wave is not just a passing fad—it will have long-lasting effects, said Exterman. However, most of our current DevOps processes were constructed for manual code writing, not for the era of AI-assisted development.

If you quadruple the amount of water you use but use the same plumbing system, you can’t push more water through, said Exterman. Similarly, if you amplify the amount of code you’re producing but don’t evolve your build processes, staging and testing, then DevOps suddenly becomes the new blocker.

Ways to Accelerate Developer Velocity

So, how can we improve development efficiency and productivity in the age of AI? Here are some actionable strategies for overcoming these bottlenecks and improving developer velocity.

Accelerating the Plumbing

Exterman stresses the importance of leveraging AI to enhance the foundational processes of development. This involves using techniques such as distributed computing and smart test selection to streamline build and testing processes. By optimizing these foundational steps, development teams can ensure that their coding efforts translate into efficient and rapid progress.

Optimizing Testing through AI

AI’s potential not only lies in code generation—it extends to optimizing testing processes. Exterman suggests tools like Launchable that allow AI to prioritize tests that matter most. This strategic approach to testing enables teams to focus on high-impact areas while maintaining comprehensive test coverage.

Embracing Test Automation

Exterman underscores the importance of comprehensive test automation. Rapid iteration loops are essential for maintaining velocity, and automation facilitates fast feedback cycles. This approach ensures that each code change undergoes rigorous quality checks, preventing errors from propagating downstream.

Keeping an Eye on Technical Debt

One of the potential downsides of increasing programming velocity is the accumulation of more technical debt. As more development occurs, the temptation to compromise on documentation, unit testing and best practices grows. Exterman emphasizes that velocity must be pursued in tandem with managing technical debt to ensure the long-term health of projects.

Investing in Platform Engineering

Recently, we’ve witnessed a paradigm shift toward platform engineering. Just over half of IT organizations have already embraced platform engineering, according to the 2023 State of DevOps Report. Exterman emphasizes the need for a robust internal developer platform that supports the seamless transition of code from development to staging and production. For instance, tools like Terraform and Earthly play a pivotal role in establishing testable environments and reproducible builds.

“Bring more data services closer to the developers,” said Venkat Ramakrishnan, VP, product management and engineering, Portworx by Pure Storage. “Choose platforms that provide easy-to-deploy services that developers can consume and provide visibility and control into the underlying costs.”

Enabling Observability

Observability, according to Exterman, is also important for gauging efficiency. The ability to continuously monitor and log processes enhances the overall development cycle. One report from Splunk even found that 64% report their investment into observability has exceeded their expectations. Tools from New Relic and Datadog, for example, offer insights that tie quality, delivery and productivity together, enabling teams to make informed decisions.

Evolving CI/CD With Continuous Merge

One method to progressing CI/CD is continuous merge, a new concept that can be implemented with tools like gitStream that aims to decrease the window between a pull request, review and merge.

“CI/CD has done wonders to improve the quality and efficiency of the software development toolchain, but it mainly focuses on things that happen after code has been written and reviewed,” said Ori Keren, CEO and co-founder of LinearB. “Continuous merge is a mindset that extends CI/CD practices further upstream to improve the quality and efficiency of the code review process. We’ve reached a point where code review times frequently cause the most significant delays in the software delivery process; continuous merge is how we fix that.”

Using Cloud-Agnostic Platforms

Cloud-native development has become more popular as organizations seek to design software architectures bred for the natural scalability of the cloud. Simultaneously, both technology and business motivators are resulting in a multi-cloud condition. In this scenario, IT really needs agnostic, open platforms to retain flexibility and reduce bottlenecks.

“Enable developers to build their apps in a cloud-agnostic way so they can scale their apps globally,” said Ramakrishnan. “Much of developer and app momentum is lost when scaling apps. Helping the application teams architect it right by providing cloud-agnostic application platforms like Kubernetes enables developers to move faster, scale faster and reach their customers everywhere.”

Measuring Developer Velocity

The above tips should help improve velocity, but how exactly should you measure success? Well, measuring developer velocity goes beyond simply tracking the number of points completed in a sprint, reiterated Exterman. He emphasizes the importance of focusing on the ability to deliver high-quality products to the market swiftly. He underscores the need for a fast feedback loop for addressing bugs and iterating quickly. AI plays a role here, too, aiding in test writing and enabling continuous delivery.

End Benefits of Improved Developer Velocity

Exterman concludes by highlighting the real-world benefits of increased developer velocity. Faster bug-fixing and release cycles, particularly crucial in industries like gaming, result in improved user experiences and customer satisfaction. He emphasized the significance of aligning key performance indicators (KPIs) and objectives and key results (OKRs) with the aspects that truly matter, promoting a culture of measured and sustainable progress.

September 11, 2023
Internal Developer Platform (IDP) Reference Architectures
If you’ve been following the DevOps space recently, you’ve probably noticed the excitement brewing around platform engineering. Despite uncertainty in other areas of tech, it seems like platform engineering is poised for growth—Gartner even expects that by 2026, 80% of software engineering organizations will have established platform teams.

The platform engineering idea hinges on an internal tooling system known as an internal developer platform (IDP), which platform teams own and operate and is exposed to developers to improve their efficiency, satisfaction and release fluidity. IDPs have been the talk of the town. But what does an IDP actually look like? What sort of components are they usually composed of?

Well, at PlatformCon 2023, McKinsey representatives shared a reference architecture for building out IDPs. This framework was modeled after hundreds of internal platforms and could be a helpful benchmark for other organizations as they begin developing and standardizing their internal developer platforms.

Below, we’ll examine the sample IDP reference architecture and its components. We’ll also have a refresher on the platform engineering concept and offer some expert tips on overcoming potential obstacles while implementing platform engineering and IDPs.

Why Engineer Internal Platforms, Why Now?

First, for those new to the idea, let’s explain what platform engineering is. I’ve previously defined platform engineering as “maintaining an internal self-service platform that serves the needs of developer consumers.” The practice doesn’t necessarily supplant DevOps, rather, the experts I’ve spoken to view it as the next evolution.

This trend is essentially a response to developers overseeing more and more aspects of software life cycle management. Complexity in enterprise-level software delivery workflows has escalated in the last decade, leading to what The State of Platform Engineering Report describes as “increased cognitive load on engineers.” This is negative, as it could lead to shadow operations and burnout.

The fact is, most developers require the same sort of technologies, which could be standardized across the board. For instance, take the consolidation around containers and Kubernetes, infrastructure-as-code and the need to implement similar build configurations and security policies across applications.

By outsourcing the maintenance of tools for shipping and running cloud-native applications to a common platform, you take the burden off developers and free them up to code. Also, by centralizing DevOps tooling, you can enforce more governance and build out internal standards. Which should help reusability and overall deployment agility … at least, that’s the promise of it all. 😉

Sample IDP Reference Architectures

So, let’s turn to some real-world examples to see how this is being implemented. Mike Gatto, senior DevOps engineer, McKinsey, shared an internal platform reference architecture for AWS-specific environments:

McKinsey’s Internal Developer Platform reference architecture

As you can see, the architecture is divided into these subgroups. At a high level, they perform the following functions:
- Developer control plane: This is the version control, where developers publish their code. They provide a workload specification (in McKinsey’s case, a Score file) that codifies specific configurations.
- Integration and delivery: This includes the CI pipeline to build application code into a container and publish it to a registry. This notifies the platform orchestrator to initiate a build. Then, resources are provisioned. This area also includes a CD pipeline and infrastructure control plane.
- Monitoring and logging: This plane includes tools for observability, analytics, and application performance monitoring.
- Security plane: This area includes tools for secrets management, policies, and security vetting.
- Resource plane: This plane encompasses cluster management, database tools, networking, and messaging architectures.
“By separating developer and platform team concerns and allowing them to control the respective areas they’re experienced in separately, without requiring too much overlap,” said Gatto, “we’re making deployments more compliant and better aligned with client initiatives.”

Humanitec has also created internal developer platform reference architectures for GCP and Azure, displayed below.

IDP Reference Architecture for GCP

IDP Reference Architecture for Azure

PlatformEngineering.org has also expanded McKinsey’s original IDP reference architecture with a tooling landscape that adopts the same umbrella subgroupings but includes other pluggable tools.

The Internal Developer Platform tooling landscape

Roadblocks to Developer Platforms

Starting from a blueprint could be a big help to rapidly construct IDPs. Yet, there will still be some potential roadblocks associated with adopting an internal developer experience platform. One of them is creating common platforms that actually cater well to all environments.

“You can not and should not try to please everybody,” said Kaspar von Grünberg, CEO and co-founder of Humanitech. “Platform engineering is an 80/20 game.”

“Platform organizations often face a lot of pressure when it comes to meeting demands for building more services that are both agile and agnostic to cloud and data center environments,” said Venkat Ramakrishnan, VP, product management and engineering, Portworx by Pure Storage. “At the same time, almost all platform engineering teams are understaffed while facing the need to support large application development initiatives and developer teams twice their size.”

Tips to Get Platform Engineering On Track

Reference architectures for various clouds could help cater IDPs to different scenarios. But outside of that, it will take a product mindset approach to strategically accept feedback and iterate the platform. This will involve doubling down on developer experience and adopting cloud-native foundations that can scale.

“Platform engineering teams must consider adopting Kubernetes as it enables them to manage and deploy applications in production at scale, without having to manage applications at the individual level,” added Ramakrishnan. “Additionally, it may not be realistic to expand platform engineering teams alongside growing platforms, so teams should also consider leveraging automation and self-service capabilities to help them better manage projects at scale.”

Is ‘You Build It, You Run It’ Dead?

The idea of ‘you build it, you run it’ is starting to look a little antiquated next to the new and shiny platform model. It’s tough to say, but I’ll be keeping an eye on this area to see if all the effort to standardize the internal engineering approach pays off!
September 5, 2023
Low-Code and AI: Friends or Foes?

The lines are becoming blurred between low-code and new artificial intelligence (AI). Historically, low-code/no-code platforms have introduced software development automation through graphical user interfaces, enabling professional and citizen developers alike to quickly build workflows and generate applications. But the rise of large language models (LLMs) represents an existential threat to the status quo of low-code development.

So, are generative AI and low-code friends or foes? Since new generative AI can produce complex code in milliseconds given natural language prompts, the barrier to software development has never been lower. On the other hand, enterprises already have ingrained low-code practices and operations, which may stick around for a while. With that in mind, there are plenty of ways AI and low-code can coexist.

Below, we’ll consider if AI and low-code are diametrically opposed tools or complementary forces. We’ll see how AI and low-code are merging and consider the benefits of using them in tandem.

Unleashing the Power of AI

LLMs, like OpenAI’s GPT-3.5 and GPT-4, can deliver impressive software generation capabilities, such as creating code snippets and even complete programs given natural language prompts. Because of these remarkable capabilities, it makes sense that low-code platforms should be a bit wary of AI.

Artificial intelligence can actually deliver on some of the original promises of low-code, like software democratization, enhanced agility and filling the widening skills gap. “ChatGPT could allow users to make more significant changes to applications, potentially enabling organizations to achieve far more than with low-code and in half the time,” wrote Romy Hughes on devopsy.kinsta.cloud.

There are also plenty of ways LLMs could accelerate the DevOps side of the cloud-native equation. In addition to code generation, AI could help generate configuration manifests, run code tests and perform cybersecurity analysis. While AI can’t fully replace the nuance of human oversight, it can indeed automate more and more elements of the software development life cycle.

Congruence Between Low-Code and AI

Although it appears likely that AI will replace low-code, there are actually many opportunities for symbiosis between the two concepts. Rather than eradicate low-code platforms entirely, LLMs will likely become more embedded within them. We’ve already seen this occur as low-code providers like Mendix and OutSystems integrated ChatGPT connectors. Microsoft has also embedded ChatGPT into its Power Platform as well as integrated GPT-driven Copilots into various developer environments.

“Low-code and AI on their own are powerful tools to increase enterprise efficiency and productivity,” said Dinesh Varadharajan, the chief product officer at Kissflow. “But there is potential for the combination of both to unlock game-changing automation for almost every industry. The power comes from the congruence between low-code/no-code and AI.”

There is also the opportunity to train bespoke LLMs on the inner workings of specific software development platforms, which could generate fully-built templates upon natural language prompts. This would make the LLM more custom-built and relevant to the platform at hand and ease development fluidity. In addition to integrating existing LLMs, another potential synergy is that low-code solutions could help automate the training of new custom AI models.

Tips to Keep In Mind

As with any technology solution, the choices should be driven by concrete business outcomes. Start by defining the business objectives and let that determine the technological solutions to arrive at the outcomes you desire. In the context of low-code and AI, this might equate to deciding between which models to support.

Next, enterprises must avoid risk when using low-code platforms. And on the AI side of the coin, platforms will have to navigate the data privacy and legal complications of using public LLMs. Like any new technology, AI and low-code will require governance, said Varadharajan. “Organizations should step in to standardize the tools, processes and DevOps cycles. Only certain AI-powered tools should be approved for development where there are agreeable security, privacy and IP protection agreements in place.”

He also explained that a focus on change management would be paramount in reaping the benefits of both strategies. “I suspect that the greatest barrier to adoption and implementation will be change management,” he said. “Enterprise users are accustomed to certain ways of working and can be resistant to change, making them unlikely to embrace AI-powered low-code. Leaders should anticipate this and incorporate change management into their AI/low-code adoption strategy.”

Closing The Semantic Gap

Low-code came on the scene to help citizen developers create applications without requiring advanced technical experience. Yet, low-code still requires knowledge of the development platform as well as specific programming terminologies and formulas. But now, since generative AI behaves in response to natural language, it has the potential to instantly bypass this barrier.

“Generative AI closes this semantic gap,” said Varadharajan. “Generative AI serves as a translator, and users no longer have to understand exactly how the system works.”

As more AI pair programmers come onto the market, expectations are rising within all areas of software development, including low-code platforms, to integrate these advanced models. If leaders can navigate the technical change effectively, the outcomes will likely significantly increase agility and output.

“Ultimately, all developers will end up saving time delegating repetitive tasks to these AI-powered pair programmers,” said Varadharajan.

August 17, 2023
Most Organizations Benefit From a Mature Cloud Strategy
Today’s cloud adoptions are wide and varied. Large enterprises are composed of vast, hybrid multi-cloud strata, often incorporating legacy technology acquired at various times and haphazardly equipped from department to department. But a fractured cloud approach can leave IT wanting in terms of shared automation and standard security policies.

Not all organizations have addressed best practices around common areas like CI/CD, resource optimizations, or common policies for multiple clouds. And many teams are just getting the hang of the new cloud-native stack. But studies show that a mature approach to cloud operations can pay dividends in the long run—and this approach hinges on cloud platform teams.

HashiCorp recently released the findings from its third annual State of Cloud Strategy Survey, which surveyed 1,000 technology practitioners on their cloud adoptions. In short, the report discovered that increased cloud maturity equates to increased overall value. Below, I’ll distill the major findings from the report to better understand how cloud-native operations are maturing.

State of Cloud Maturity

Most organizations are moderately increasing their cloud investments. Overall, 56% increased their cloud spending in the last year. Increases were much higher for higher cloud-maturity organizations and stilted for lower-maturity organizations. The findings demonstrate general confidence in cloud adoption, even amid economic uncertainties.

There is also persistent optimism in multi-cloud strategies. In fact, 73% of companies said a multi-cloud strategy helped advance or achieve their business goals. And to successfully accomplish a multi-cloud strategy, respondents ranked security, uptime and availability, infrastructure scalability and regulatory compliance as top factors.

However, there is still a lot of waste in the cloud. Nearly all organizations (94%) said they have avoidable cloud spending. These expenses most commonly stem from areas like over-provisioning resources, idle or unused resources, cost premiums or a lack of expiration date on cloud resources.

Benefits of Maturing Cloud-Native Operations

Overall, there are many reasons for adopting a multi-cloud approach—the report found reliability, cost reduction, security and governance and digital transformation as top motivators. And on average, high-maturity cloud adopters tend to reap the rewards more often, the report found. Some of these top rewards ranged from a stronger security posture to better visibility and increased automation. Furthermore, half of high cloud-maturity organizations reported successfully saving money using multi-cloud.

Most interestingly, perhaps, is that high-maturity cloud adopters said they are better equipped to attract, motivate and retain talent than lower-maturity cloud adopters (74% compared with 48%). It’s generally easier to pull in talent when operating with attractive cutting-edge technologies that look good on a resume.

Spotlight on Platform Teams

Even then, skills shortages remained the top bottleneck in most organizations’ ability to operationalize multi-cloud and set up platform teams. This can be side-stepped by reskilling or upskilling staff, incorporating more consultants or increasing the budget. But the best strategy is to standardize on a common operating model.

In that vein, investments into the platform engineering concept are rising. In fact, 92% of organizations are adopting, standardizing or scaling platform teams. Low cloud maturity firms tend to lag on platformification, most likely due to lacking the necessary skills to build and maintain internal developer-facing infrastructure, like the tools used to measure and improve developer productivity and satisfaction at LinkedIn. Others are still on the cusp of exploring how to implement it.

The idea of platform engineering is still taking root, so it was interesting to see how organizations described the platform teams’ roles and responsibilities. The results showed platform engineers commonly take the reins on everything from architecting solutions to tooling acquisition and internal evangelism. Here are the five top-cited roles for platform teams:
- Develop and standardize a cloud infrastructure strategy (95%)
- Architect cloud solutions (95%)
- Create and distribute cloud management, operational policies and best practices (94%)
- Determine which cloud vendors and/or technologies are used in production (93%)
- Define and measure site reliability (91%)
The Path to Cloud Maturity

In general, companies are still confident about their cloud trajectories. But, getting to a mature multi-cloud state will take work. One area that needs further scrutiny is cybersecurity, as password or credential leakage is now seen as the most prevalent cloud security threat. Organizations will require better DevSecOps best practices to avoid such leaks. That, and certain open source cloud-native tools could also aid security and compliance efforts.

A mature cloud strategy will also require localizing development efforts through shared platforms and increasing infrastructure automation. On that note, 83% of organizations say automation tools are important or extremely important to operationalize their multi-cloud infrastructure. More automation for cloud-native areas like CI/CD, observability and Kubernetes management could also abstract complexity and fill the skills gap.

As the HashiCorp study clearly showed, those organizations that have matured their cloud strategy are far better off because of it. They tend to understand their workflows better and can more seamlessly introduce new automation. Now, it’s only a matter of getting there …
August 10, 2023
How to Prevent Zombie APIs

There is a growing reliance on web APIs. A full 92% of organizations plan to increase their investments into APIs over the next year, the 2023 Postman State of the API Report found. APIs are powering everything from internal microservices strategies to partner strategies and fully-fledged products.

Yet, this newfound API sprawl has consequences; a looming threat that might rise from the grave to haunt you …

I’m talking, of course, about zombie APIs. Zombie APIs are endpoints that are no longer maintained yet are still active. They may be unused endpoints, old features never officially deprecated or forgotten development or testing environments. Regardless, zombie APIs are a type of technical debt that could pose a legitimate threat if left to rot.

The sudden growth of APIs, along with immature inventory management and a lack of internal communication, can contribute to creating countless shadowy endpoints. Below, I’ll review some of the issues that zombie APIs might cause and consider some actionable steps DevOps teams can take to prevent them.

Understanding Zombie API Risks

Forgotten APIs can cause a number of issues. For example, suppose an insecure testing environment was accidentally left exposed to the public. An attacker using automated endpoint scanning could find authentication and authorization holes to exploit, potentially exfiltrating sensitive data.

It’s also a bad idea to let any piece of technology go unmaintained, given the daily influx of new exploits and vulnerabilities. This point is accentuated by the connected nature of APIs, which may plug into a vast network of databases and allow both read and write privileges to the backend infrastructure.

Insecure APIs can cause a business both economic harm and damage reputation. Just consider the high number of API-related data breaches in recent years, which occurred at companies like Optus, Dropbox, Twitter and Zendesk. It’s even estimated that somewhere between $12 billion to $23 billion was lost in 2022 due to compromises in web APIs.

Zombie APIs are an easy target for these sorts of cyberattacks because they tend to have insecure designs that are easier to crack. They also tend to use immature monitoring systems, which means hackers can act under the radar more easily than with actively used endpoints.

Tips to Avoid Zombie APIs

Keep an Active Inventory of APIs

As they say, you can’t secure what you don’t know about. In fact, OWASP lists improper inventory management as one of the ten top API security risks. Therefore, proper inventory management is a good first step to avoiding zombie APIs. Scan your service portfolio for forgotten endpoints, catalog them and update the catalog regularly.

It’s also a good practice to thoroughly document your services to aid in internal discoverability. If possible, adopting specification-first practices can also ensure the documentation and implementation are based on a single source of truth, thus avoiding mismatched versions and documentation drift.

“The key to preventing zombie APIs is visibility,” said Jeremy Snyder, CEO, FireTail. “Many organizations lack ongoing, up-to-date inventories of all the APIs that are running in either production or staging environments.”

Use Proper Versioning and Life Cycle Strategies

Another method to reduce the chance of zombie APIs is to adopt mature API life cycle management practices. The chosen API versioning strategy should be clearly defined and codified. And versioning changes should be clearly communicated to all stakeholders.

On that note, it’s just as important to plan out a sunsetting, retirement and deprecation strategy from the onset. Then, remove deprecated services when they have reached their end of life! This is not a glamorous task, but it should be baked into someone’s assignments.

“Good cyber hygiene is an often overlooked strategy,” explained Snyder. “A second requirement is to have a policy around versioning and how many versions of an API can exist at any given moment. For instance, that policy may state that only the latest version can exist or the latest plus the previous. However, this relies on having visibility in order to maintain compliance.”

Other areas of proper life cycle management include employing governance around development and deployment and utilizing automated testing in continuous deployment (CI/CD) pipelines. Both of these could help set policies and security requirements, which should also bolster API safety.

Share Knowledge of Internal and Third-Party Services

Tribal knowledge is the antithesis of DevOps culture. As such, it’s a good practice to de-silo and share internal knowledge of API-powered services to limit the propagation of rogue endpoints. This could be done through private network repositories or even an internal Postman Collection or Workspace.

“Internal processes like organizational coordination and communication are critical,” added Snyder. “If teams don’t tell each other about new APIs being developed and launched into production, security teams can’t assess or audit them. Conversely, if app teams don’t know about network infrastructure resources designed to cover various API security aspects, developers don’t know what security controls need to be baked into the API code.”

Visibility is Required to Avoid Zombie APIs

A recent CISO survey revealed that API security was the number one concern for CISOs. For many reasons, web APIs are still low-hanging fruit for attackers. This is underscored by the 400% spike in malicious API attack traffic recorded by Salt Labs earlier this year.

Many of these attacks, we’ve also learned, are occurring over authenticated APIs. This shows that greater accountability and visibility are required to begin to stem the tide of API-related security incidents.

To reach an adequate level of visibility, leaders should consider adopting tooling that helps determine if there are outdated APIs or APIs with multiple live versions, said Snyder. “These tools should help display what security live APIs have and provide recommendations around how to fix them, if not providing the necessary code or modules to secure the APIs directly.”

August 4, 2023