devopsy.kinsta.cloud

Author: Rob Zuber

The Anatomy of a $900,000 Validation Bill

A 50-developer team shipping at an agentic pace will spend roughly $900,000 this year on AI and developer tooling costs. Most of the engineering leaders I’ve talked to saw the compute costs coming. What they didn’t anticipate was where the rest of the bill was hiding.

The culprit is a timing problem. Frontier models price context reads on a sliding scale, and standard validation pipelines take longer than the discount window allows. By the time a failure comes back, the agent re-reads the full codebase at full price, carrying the previous attempt’s output plus build logs. Each retry costs more than the last.

The teams getting ahead of this are moving validation earlier, into the agent’s working environment, so failures get caught while that cache is still warm.

Understanding what’s driving the bill requires a different kind of measurement. The metrics most teams use to gauge delivery health were designed around the same assumption as the tools themselves: that a developer is driving the work. DORA metrics like deployment frequency and change failure rate are useful signals, but they weren’t built to account for what happens when agents are driving the volume. They tell you how fast you’re moving, not what each step costs.

That’s what led us to focus on cost per shipped change (CPSC): the total CI and AI token spend required to get one change from development to production, including all feature branch tries and retries, and the final merge to main. CPSC captures something that per-seat AI pricing and compute invoices don’t show on their own. It reveals the full economic weight of every cycle required to ship.

The operational picture makes clear why this metric matters: teams are generating more code than ever but struggling to get it into production. The most recent State of Software Delivery Pulse Report puts it plainly: main branch success rates stand at 76.7%, meaning nearly one in four attempts to merge code into production fails. When builds do fail, the median team takes well over an hour to recover. You can explore the data yourself to see how these trends are playing out across industries, team sizes, and regions.

The gap between high performers and everyone else is striking. Looking at an anonymized cohort of the 20 organizations with the highest main-branch throughput on CircleCI, we found that the top teams cut their CI compute spend per shipped change by 31% year over year. The median team moved the other way, with CI credit usage per shipped change rising 13% over the same period. The difference traces directly to a metric we call Merge Efficiency Ratio (MER): the number of feature-branch validation cycles required before a change lands on main. The top teams ran far fewer of them.

Here is what the economics look like when you account for AI token costs as well. Take a 50-developer team shipping roughly 3,000 changes per month. At the typical MER, each shipped change requires about five total CI runs, four on feature branches and one on main. For each change, our cost model puts the bill at roughly $13 in input tokens from cold cache reloads and context bloat, $9 in output tokens from agents rewriting code after CI failures, and $3 in CI compute. That comes to approximately $25 per shipped change. Across 3,000 changes, that’s $75,000 per month, or $900,000 per year. Actual costs will vary by model, context size, cache behavior, and retry rate, but the pattern holds: at AI-driven volume, slow feedback loops consume a meaningful share of the delivery budget. For larger codebases requiring 500K input tokens or more, the same pattern crosses $1.5 million.

The expensive part is not catching failures. It is catching them too late.

The teams getting ahead of this have found ways to validate in the inner loop, before the push, so CI handles the work it was actually built for: reproducible environments, integration tests, the final gate before code ships. Tools like Chunk sidecars are built for this. They run a distilled set of checks inside the agent’s environment in under 30 seconds, catching failures while the context cache is warm and the fix is cheap. In our testing, microbuild output was about 5x more token-efficient than standard CI logs, because it returns focused failure context rather than full pipeline output.

Teams that close this gap aren’t just reducing their bills today. They are building delivery infrastructure that holds as the pace of agentic development continues to grow.

July 28, 2026
The Validation Gap Is Costing You More Than You Think

Our latest State of Software Delivery report analyzed more than 28 million CI workflows and found a pattern that should give engineering leaders pause. Average throughput grew 59% year over year. Main branch activity for the median team declined 7%. Teams are generating more code than ever before. Less of it is reaching production.

The cost of poor validation used to show up mostly in developer hours: debugging, blocked deployments, context switching. That cost hasn’t gone away. But there is a second bill now. Every failed build means agent retries. Every slow pipeline is compute burning while an agent waits. Main branch success rates have fallen to a five-year low of 70.8% against a 90% benchmark, and the AI spend attached to every failed cycle is climbing alongside it.

The teams doing well are catching failures earlier and keeping their pipelines healthier. They are running the same tools as everyone else. What they have structured differently is where and when validation happens.

For most of our careers, the inner loop was something that was managed and optimized by the developer. Once AI generated code entered the picture, the inner loop as we knew it could not handle the volume. Today, the inner loop is agentic. It’s where the agent is actively working: writing, iterating, checking before anything is committed or pushed. The outer loop is CI: shared infrastructure, integration, the final gate before shipping. Most teams have invested heavily in the outer loop. Until recently, the inner loop didn’t need much attention. That has changed.

CI was designed around a human pace of development: one engineer, one branch, one push at a time. Agents generate changes in parallel, across multiple tasks, at a volume that makes the push-wait-fix cycle a serious drag on throughput. By the time CI returns feedback, the agent has moved on to the next task. Context is gone. Fixing the failure means starting a new cycle: reloading context, re-examining the change, potentially redoing work that was already completed.

Human review was always the backstop before code reached shared infrastructure. For most teams today, the volume of AI-generated change has simply outpaced what any reviewer can meaningfully assess before it hits CI. The code arrives faster than the review process was designed to handle. Most teams end up making a choice without fully realizing it: either throttle the agents to match available review capacity, or let the volume through and absorb the failures downstream.

Validation needs to happen earlier, while the agent is still working on the change. CI still matters. System-level validation, integration, packaging and deployment belong in the outer loop and always will. But by the time code reaches CI, it should have already passed basic scrutiny. The inner loop is where that confidence gets built, before anything touches shared infrastructure. Getting validation right at this stage is also how teams close the gap between a 70.8% success rate and the 90% benchmark the data points to.

The requirements for inner loop validation are specific to how agents work. Feedback has to arrive within the window the agent is still operating in. Tests need to be scoped to the relevant change, not the full suite. Failures should surface one at a time: an agent given a long list of problems fills its context window and stops being productive. These constraints are different from what CI was designed to satisfy, which is why existing tooling often doesn’t fit.

When the two loops share context, the picture changes. When the inner loop draws on what CI has historically flagged in a codebase, it runs smarter checks before anything is pushed. When CI sees changes that have already been validated locally, it can focus on what actually needs system-level verification. The two stages inform each other and the system gets better at catching the right things over time.

By the time CI returns a failure, is the agent that introduced it still working on that change?

This is where CircleCI has been focusing, building validation that spans both stages and learns from how a codebase actually builds. The infrastructure question is one teams will need to answer if they want to improve their throughput and success rates.

Every model release increases agent velocity and the volume of code flowing into delivery pipelines. Teams that have solved the validation gap absorb that increase and get faster. Teams that haven’t will find the gap between them and the top performers widening with every upgrade.

May 30, 2026
4 Ways to Uplevel Your Software Engineering Team

In my 20 years leading software development teams, I’ve witnessed the industry’s continuous growth alongside technology’s evolution. The only thing we know for certain is that the industry will continue to change and winning businesses will be those that build with change in mind, backed by effective, fast-moving software teams.

This is the year we must focus on fine-tuning our DevOps processes and uplevel our teams, but where do we start?

Below are four ways you can begin to transform your teams into engineering powerhouses, no matter how big or small or where they are in the world.

1. Embrace New Delivery Models

Remember when deployments were a big deal; a highly-anticipated event that often meant late nights? These days, many teams have managed to make deployments a non-event and are deploying any time of day. Those that haven’t adapted risk being left behind. The rate of deployments will only increase, so how teams handle this progression will be a deciding factor for success in an already competitive field.

In 2022, teams should continuously evaluate their procedures as they push the boundaries of their tech stack. Navigating the complexity of the digital era—from massive datasets and machine learning to networks of third-party services—will force both developers and company leaders to embrace flexible tooling and processes to ultimately refine the risk/innovation tradeoff.

2. Prioritize People and Culture

While 2021’s Great Resignation saw many software professionals jumping ship in search of a fresh work environment, in 2022 we expect to see a Great Reshuffle. Realizing their new or current gig still may not be quite the right fit, engineers and developers will start looking for roles that are more in line with their long-term career and personal goals.

Don’t fear this reshuffle. Instead, recognize it as an opportunity to build a strong and stable team. Be honest about what, exactly, the job entails and the hard and soft skills you’re looking for. Honesty throughout the recruitment process leads to a better match between employer and employee and also contributes to resilient company culture.

3. Deploy Low-Code Solutions Strategically

Pressure to get the highest productivity out of software development teams has led to the adoption of no-code and low-code solutions. But that means these tools are being used predominantly by people who already know how to code. The folks with big ideas who don’t actually want to be programmers remain unserved and untapped.

Expect this disparity to soon lead to a split in the category of no-code and low-code solutions—those built specifically for optimizing developer teams, and those that can serve everyday creators. To really get the most out of your team, keep an eye on the low-code space and ensure the solution you use is the one that’s actually best suited for your goals.

4. Build With Change Validation in Mind

The growing adoption of cloud storage, service-oriented architectures, third-party API-based services and open source code (to name a few) makes modern software development monumentally complex.

A way to manage this complexity is by pulling our focus away from the application itself and onto myriad sources of change that affect the application—a concept we can think of as change validation.

Ultimately, change validation builds upon tried-and-true software best practices such as continuous integration, continuous delivery and release orchestration. Cloud-native architectures can be effective and scalable, but you need to have the right tools and systems in place to validate changes to your code at every step.

In the past, the general consensus was that you shouldn’t test in production; you should just release and wait for the bug reports to tell you what’s wrong. But in the world of extensive software supply chains and cloud-native architectures, testing in production is the only way to truly know whether changes to your code are working as intended before releasing software to customers.

In the new year, software developers will be of critical importance as organizations continue to expand on digital services and projects. Embracing new methods of software delivery, prioritizing people and culture and keeping an eye on how to best leverage automation tools like no-code/low-code and change validation will all help to transform your team for the better in 2022. This forward-looking approach and a willingness to reassess and pivot guarantees that your software engineering team will remain agile amidst a rapidly changing tech landscape.

February 7, 2022
How to Build a Team That Embraces Metrics
When we talk about metrics in software delivery, a lot of developers think of execution metrics — things like throughput, delivery and number of deploys. But in reality, those metrics don’t motivate anyone — at least not without connecting them to a bigger picture.

I’ve worked in software for 23 years. I’m a three-time founder and four-time CTO, responsible for leading a 200+ member distributed engineering organization. I’m passionate about metrics because they can tell me a lot about where things are working and where they are not. On the other hand, there’s a risk that operating metrics become the focus, taking attention away from measures of business value.

Getting your developers to embrace metrics isn’t about hand-selecting specific metrics to track. It’s about disseminating the bigger picture and sharing measures of business impact that allow your team to see progress against those larger, shared goals.

When developers understand how their day-to-day work directly impacts the company’s goals and vision, they’re able to understand their impact, too, and will want to do better. When they want to do better, they want execution metrics to get them there.

Clarify Your Goals

If you want your developers to not only track their metrics but be excited to use them, there are a few things you need to do first:
- Get clarity on the goals you’re trying to accomplish as a team and as an organization
- Figure out how you’re executing against those goals
- Find out what’s getting in the way
Developers shouldn’t be obsessing over execution metrics. Rather, you need them to understand what the company is trying to achieve. What is the outcome you’re aiming to deliver to customers and for the business overall?

If your developers have clarity on the company’s mission and vision, they understand how their work directly impacts the customer and the success of the organization, which is highly motivating.

Individual contributors will start asking for access to more information so they can diagnose their own problems and improve their work.

Are we spending too much time on technical investment? Does it feel like we have too many incidents, issues and escalations from customers? Are we far off on our estimates? By answering these questions, your team can determine which metrics to track.

Don’t Make These Mistakes

Using one metric to rule them all. A lot of engineering organizations try to rally entire departments around the same execution metrics. Velocity, or throughput, for example, is a great tool to expose other issues, but consistent velocity is not a guarantee of delivering business value. In fact, targeting consistent velocity is likely to result in tradeoffs you didn’t anticipate, like taking shortcuts to get things out the door.

Treating the metric as the goal. Meeting a metric shouldn’t be the goal; rather, metrics should be viewed as a way to ensure your team is meeting their goals. Organizing around business value is what really helps your team understand the purpose of their work. If you’re just rallying around how many times you can deploy per day, then sure, you can be an amazing software-deploying machine — but you might be deploying changes all day that have nothing to do with the actual goals of the business.

Being a blocker. Developers need autonomy and flexibility to do their jobs well. Give teams a high-level objective and let them determine which metrics they need to push, instead of giving everybody the same metrics and telling them to get better.

If engineers don’t have the autonomy to make their own decisions, then they’re going to feel like they’re just part of the machine. As engineers, and as humans, problem solving is what motivates us, along with the knowledge that our skills are contributing to a larger, shared goal.

A Team That Embraces Metrics: Putting it Into Practice

There are three channels that are essential to communicating business-value metrics to teams. At the company level, executives will introduce their high-level goals at all-hands meetings — where the company is trying to go this year, how you differentiate in the market — the things that really matter to a business.

One level down, during organizational or department meetings, is where to communicate how teams should think about the goals from a product and engineering perspective, while still thinking about them within that overall business context.

It’s also important that this is understood down through the management chain. An engineering manager should be able to explain to their team the overall business goals with local-level reinforcement and how it all ties together.

When you have that knowledge as a team, you know what you’re capable of executing because you understand what you’re trying to achieve for the company and the customer. That’s when developers start asking for more metrics because they want to understand how to get there faster.

If You’re Not Hitting Your Goals, You Should be Asking for the Data

The only real failure is not learning from failure. It should be okay to not hit your numbers. Set a bold target and push yourself, then learn something from your misses. Figure out what you didn’t understand, what got in your way and why.

Estimation is hard. Trying to plan work is hard. But if you know how much you were able to deliver last quarter, you can focus on getting something similar done this quarter instead of assuming this will be the one quarter where there are no interruptions, your initiatives will all be perfectly defined and nobody will get sick.

If you’re not hitting your business goals and you’re not looking at your historical data, then you’re missing your opportunity to improve.

Your Team Embraces Metrics. Now, How Do You Get Them Excited?

The first time I started using continuous deployment (CD), I was truly amazed at how quickly it allowed my team to put new functionality into the hands of our customers. We could instantly find out if they liked it, and if they didn’t, we would build another version tomorrow.

As an engineer, CD is a great way to feel more connected to your customers because you get to put something you built in front of them the same day you complete it. You also become interested in a different form of metrics: usage and adoption.

The connection between your work and the ultimate value to the user is much clearer when the feedback cycle is shorter. CD allows you to get that feedback from customers directly and really see the impact you have on them. That gets people excited.

There’s no Shortcut

You can try to drive an entire organization by execution metrics, but it simply won’t be fulfilling for anyone. If your team is aligned around what they’re trying to achieve, they will recognize the value of execution metrics and they will come looking for them.

There’s no shortcut.

If you start with the reason and you end with the numbers, you know you’re on the right track. You’ll know you’re getting there because you’ll see the numbers moving, your team will be excited by that, and they’ll be motivated to do even better.
May 26, 2021