devopsy.kinsta.cloud

Tag: capacity planning

GitHub’s August Outages Show Growth Is Outpacing Infrastructure

Five incidents in one month is a lot for any platform, but it’s especially notable when that platform hosts most of the world’s code review, CI/CD, and now AI coding agents. GitHub’s August 2026 availability report reads less like a list of unrelated hiccups and more like a pattern: the company’s own infrastructure is straining to keep up with how fast usage of Actions and Copilot has grown.

The report opens with a line that doubles as GitHub’s operating philosophy right now: “Availability, then capacity, then features.” That ordering matters, because most of August’s incidents trace back to capacity limits, not bugs in new features.

The first hit came August 6, when a routine deployment reduced pod capacity in one datacenter. That triggered service mesh saturation and cascading failures across multiple clusters, knocking out Actions (both hosted and self-hosted runners), Copilot’s coding agent, code review, Pages builds, Dependabot, and repository migrations for roughly nine hours. GitHub’s own postmortem put it bluntly: “The affected actions services were running close to their capacity and concurrency limits.” The fix was a rollback, expanded capacity, and a fix for a latent bug that had let runners pick up jobs they couldn’t actually run.

Eleven days later, on August 17, a traffic peak exceeded datacenter load balancer limits. A service-mesh sidecar failed to scale, network flow limits got exhausted, and the shared authentication path degraded. The blast radius was wide: issues, pull requests, both APIs, Actions, Copilot, authentication, and webhooks all took a hit. Peak front-door failure rate reached 56%, and about 29,000 organizations saw errors across roughly 4.8 million requests. GitHub also flagged a compounding factor familiar to anyone who has debugged a distributed system under load: “A latent client retry bug sharply amplified traffic to one internal authentication endpoint.” Retry storms make bad days worse, and this was no exception.

August 20 brought a different kind of failure, tied to Copilot’s cloud agent. An upstream managed database had a regional outage, and a storage configuration issue slowed the failover. Tasks kept running, but status and results visibility lagged for up to 90 minutes at more than 54 organizations over nearly 11 hours.

Mitch Ashley, vice president and practice lead for CIO & Technology Buyers and Software Lifecycle Engineering at The Futurum Group, said that the incident points to a gap most engineering teams haven’t closed yet. “Ordering availability ahead of features is the right call,” Ashley said. “The Copilot agent incident showed why: tasks kept running while status lagged, so teams could not tell what had finished. Engineering leaders should require agent status checks they own, outside any vendor’s UI.”

Then came back-to-back incidents on August 26 and 27. The first was a database saturation problem tied to Actions run starts, Pages deployments, and Copilot code review, with GitHub owning the underlying issue directly: “Our shared infrastructure services have not kept up with our month-over-month actions growth and peak load.” At least 24 organizations saw run-start failures, and 386 organizations felt some impact before manual throttling brought things back under control. The next day, a separate issue hit only Copilot requests routed to the Kimi K3 model, with 63% of those requests failing due to a problem at the upstream model provider. Other models were unaffected, but it’s a useful data point on how much of Copilot’s reliability now depends on providers GitHub doesn’t fully control.

For platform teams, the throughline across all five incidents is the same: growth in Actions usage and AI-assisted workflows is outpacing the shared infrastructure meant to support it.

Ashley sees that as more than an infrastructure problem. “GitHub is bidding to become the surface agents run on, which changes what availability means,” he said. “With Actions and Copilot in the delivery path, an outage there is a production incident for every team downstream.”

GitHub isn’t ignoring the pattern. Alongside the incident writeups, the company detailed parallel infrastructure work: moving MySQL primaries to Azure, cutting roughly a million queries per second from its database load, routing a third of Actions jobs to spare capacity, and expanding pull request isolation to authenticated reads. None of that is glamorous, and none of it will show up in a product announcement. But it’s the plumbing that determines whether a platform this central to software delivery can keep scaling without buckling every few weeks.

That’s the real takeaway for engineering leaders who depend on GitHub as critical infrastructure, not just a code host. Actions and Copilot have become part of the software delivery pipeline itself, which means an outage there isn’t an inconvenience — it’s a production incident for every team whose builds, deployments, and reviews run through it. Teams that haven’t already should be asking what happens to their pipeline when GitHub has a bad day: whether critical deployments have a manual fallback, whether long-running agent tasks have their own status checks outside GitHub’s UI, and whether retry logic on their end could make an incident worse instead of better.

GitHub deserves credit for its transparency here. Five detailed postmortems in one report, each naming a specific root cause, is more candor than most vendors offer. But candor doesn’t replace capacity. The next few months of availability reports will say a lot about whether GitHub’s infrastructure investments are keeping pace with a platform that’s becoming more central to how software gets built.

September 11, 2026
From Reactive to Predictive: Capacity Planning Systems That Actually Work
I used to think capacity planning was about setting up CloudWatch alarms and hoping they’d fire before things broke. Spoiler: that’s not capacity planning—that’s just reactive firefighting with extra steps.

Real capacity planning means knowing you’ll need more database capacity three weeks from now, not three minutes after your site starts timing out. It means being able to confidently tell your team, “We’re good until mid-January, then we need to scale,” instead of scrambling during a production incident.

After years of being on the wrong side of this problem, I finally built a system that works. Here’s what I learned.

The Problem With How Most Teams Do Capacity Planning

Most engineering teams fall into one of two camps:

Camp 1: The Reactive Firefighters
They wait until something breaks, then throw resources at it. CPU hits 90%? Spin up more instances. Database slow? Upgrade the tier. This works until you’re burning budget on over-provisioned resources or, worse, you’re down because you scaled too late.

Camp 2: The Over-Engineers
They build elaborate forecasting models with machine learning, multiple data sources, and dashboards that take longer to understand than the problems they’re solving. These systems are impressive but rarely used because they’re too complex for day-to-day decisions.

I’ve been in both camps. Neither works long-term.

What Actually Makes Capacity Planning Work

After multiple iterations, I’ve learned that effective capacity planning needs three things:
1. Simple, Actionable Metrics
  You don’t need fifty metrics. You need the right five. For most systems, that’s:
- Resource utilization trends (CPU, memory, disk)
- Request rate growth patterns
- Database connection pool usage
- Storage growth rate
- Network throughput trends
The key is tracking these consistently and understanding what “normal” looks like for your system.
1. Historical Context That Matters
  Raw metrics are useless without context. You need to know:
- What happened during your last traffic spike
- How your system behaved during that product launch
- What “seasonal” patterns exist in your usage
I keep a simple log of significant events alongside our metrics—product launches, marketing campaigns, major features. When I look at a utilization spike from three months ago, I can see it coincided with a feature release, not a problem.
1. Forward-Looking Projections Based on Reality
  This is where most systems fail. They either use naive linear projections (useless for growing systems) or overly complex models (unusable for actual decisions).
What works: trend analysis with growth factors that you adjust based on what’s actually happening. If you’re growing 15% month-over-month, project that forward. If a major feature is launching next month, factor in expected load based on similar past events.

Building a System That Your Team Will Actually Use

Here’s the architecture that’s worked for me:

Data Collection Layer
Pull metrics from your existing monitoring (CloudWatch, Datadog, Prometheus, whatever you’re already using). Don’t build a new metrics pipeline. Use what you have.

I built a simple script that runs daily and pulls:
- Average and peak utilization metrics for the past 30 days
- Week-over-week growth rates
- Month-over-month trends
Analysis Layer
This is where the magic happens, but it’s simpler than you think. For each critical resource:
1. Calculate the current growth rate (7-day, 30-day, 90-day averages)
2. Project forward using the most conservative (slowest) growth rate
3. Calculate time-to-threshold (when will you hit 80% capacity?)
4. Flag anything that’ll hit threshold in the next 60 days
I use Python with pandas for this. The entire analysis script is under 200 lines.

Alert and Reporting Layer
Here’s the crucial part: make it impossible to ignore.
- Weekly email summary to the engineering team showing resources projected to need scaling
- Slack alerts when time-to-threshold drops below 30 days
- Monthly capacity review doc that auto-generates and gets dropped into our team drive
The key is automation. If you have to remember to check a dashboard, you won’t.

The Guardrails That Keep It Honest

Early versions of my system gave terrible predictions because I made these mistakes:

Mistake 1: Trusting short-term trends
A three-day spike doesn’t mean you’re growing 300% per week. I added a rule: never project based on less than 14 days of data, and always compare against 30-day and 90-day trends.

Mistake 2: Ignoring known events
If there’s a major feature launch next month, your linear projection is worthless. I maintain a simple calendar of planned events that might impact capacity and manually adjust projections around them.

Mistake 3: Setting thresholds too high
Waiting until you’re at 90% CPU to scale means you’re already feeling pain. I alert at 80% capacity with 60 days runway. That gives us time to plan, provision, and test without panic.

What This Actually Looks Like in Practice

Here’s a real example from last quarter:

Our RDS database was sitting at 52% CPU utilization. Traditional monitoring said we were fine. But my capacity system showed:
- 30-day growth rate: 3.2% per week
- 90-day growth rate: 2.8% per week
- Projected to hit 80% CPU: 47 days
Without this, we would’ve kept ignoring it until we hit 85% and had to do an emergency scaling operation during business hours. Instead, we scheduled a maintenance window, upgraded the instance class, and validated performance; all without impacting users.

That’s what predictive capacity planning actually delivers: boring, planned infrastructure changes instead of exciting 2 AM emergencies.

The ROI You Can’t Ignore

Since implementing this system, we’ve:
- Eliminated emergency scaling incidents (which used to happen every 6-8 weeks)
- Reduced over-provisioning by right-sizing based on actual projections
- Cut time spent in war rooms discussing “do we need to scale?” from hours to minutes
More importantly, our team sleeps better. When you know what’s coming, you can plan for it. When you’re constantly reacting, you burn out.

Starting Your Own Capacity Planning System

If you’re building this from scratch, start small:

Week 1: Pick your critical resources
What are the three things that, if they ran out of capacity, would take your system down? Database? Application servers? Cache layer? Start there.

Week 2: Set up automated data collection
Write a script that pulls utilization metrics for those resources daily. Store them somewhere: S3, a database, even a CSV in git. Just make sure it’s consistent.

Week 3: Build your first projection
Take the last 30 days of data, calculate growth rate, and project forward. It doesn’t need to be fancy. A spreadsheet works.

Week 4: Add alerting
When time-to-threshold drops below 60 days, send an email. That’s it. You now have a working capacity planning system.

Everything else—dashboards, ML models, sophisticated forecasting—can come later if you need it. But this core system will save you from most capacity-related incidents.

The Mindset Shift That Matters Most

The biggest change isn’t technical, it’s cultural. Capacity planning only works when your team believes that preventing problems is more valuable than heroically solving them.

Early in my career, I got praise for staying up all night scaling systems during an outage. Now I get praised for the outages that never happened because we scaled two weeks early.

That shift from reactive hero culture to proactive engineering culture is what makes capacity planning sustainable. The system I’ve described is just the tool that makes that culture possible.

What’s Next

Capacity planning isn’t a project you finish; it’s a practice you refine. As your system grows and changes, your capacity planning needs to evolve with it.

The system I’ve built continues to get better as I learn what matters and what doesn’t. Some quarters I add new metrics. Other times I simplify and remove things that weren’t adding value.

The goal isn’t perfection. It’s having enough visibility and lead time to make good decisions instead of panicked ones.

If you’re tired of reactive firefighting and want to build something better, start with the basics I’ve outlined here.
January 9, 2026
How to Keep Your IT Infrastructure Invisible

IT infrastructure is critical. IT infrastructure enables businesses to run—from day-to-day operational execution to visionary strategic innovation and everything in between. What’s interesting, though, is that people are only aware of it when there’s a problem: HR can’t access the payroll application, customers are complaining about how slow the transactional system is or this month’s cloud bill was double what you expected it to be.

No one should notice a well-orchestrated environment, but once application performance slows or an outage occurs, people start to grumble. The same is true for IT infrastructure.

The Top Four Ways Your IT Infrastructure Can Become Undesirably Visible

Given the complexity of modern hybrid/multi-cloud environments and the applications they support, it’s no surprise that countless things can go wrong—and when they do, your IT team takes the flack.

If your infrastructure is humming along nicely, that’s a testament to the IT infrastructure team’s hard work and expertise. Sadly, they don’t get the recognition because it’s all invisible to the rest of the organization. It’s only when *BAM!* something happens that the team is scrutinized.

While there are many ways this can manifest, it usually falls under one of the following four categories:

• Availability — This may be the most catastrophic because when systems are unavailable, people can’t do whatever it is they need to do.
• Performance — A system might be available, but that doesn’t mean it’s performing to SLA or user expectations. The result can range from minor annoyance to lost revenue and customer churn.
• Cost — Runaway expenditures don’t necessarily affect users, but you do not want to be called to account by the finance team or, worse, executive leadership.
• Compliance — This is another area that may not have a broad impact on user experience, but the repercussions of non-compliance can be financially and reputationally significant.

Common Infrastructure Culprits and How to Address Them

To ensure the necessary availability, performance, cost control and compliance to keep your IT infrastructure under the radar, you must understand what can cause problems in these areas. Here are eight common culprits, along with ways you can address them, so your IT team stays invisible.

Broken or Misunderstood Dependencies
Your infrastructure is a complex web of interdependent applications and services. Every time you make a change, it could have a rippling effect on the categories mentioned above. Automated discovery and service mapping capabilities enable you to easily trace dependencies throughout your infrastructure to avoid unintended consequences.

Cloud Waste
Chances are very high that you’ve got cloud waste. This means that you could be getting billed for a storage block that’s attached to a stopped instance, for compute instances that are not being used, or for unattached load balancers. Finding idle and abandoned resources can keep your monthly cloud bill under control and on budget.

Un-Breakdownable Cloud Costs
Cloud bills are notoriously complex, and while cloud providers do offer tools to help you analyze your cloud spend, those tools often don’t answer all the questions your business might have. In addition to adding clarity to such opaque expenses, you need to be able to parse your cloud spend based on what makes business sense to your organization, not what makes technical sense to your cloud provider.

Poor Capacity Planning
Your infrastructure needs to meet your business needs, which are continuously evolving. As changes happen, whether it’s increased utilization of existing resources or modified/added resources to support new requirements, workloads could push current capacity and impact performance and stability. You need to be able to forecast infrastructure demands against business objectives to minimize potential issues before they happen.

Poor Workload Placement Decisions
Your infrastructure consists of various environments each with its own characteristics. If you migrate a workload without understanding the implications of that particular environment, you could risk performance problems or spiking costs. To prevent this, you need any-to-any migration insights to make better decisions about workload priorities, groups and deployments.

Poor Reservation Planning
Reservations (i.e., reserved instances or programmatic discounts) offer discounts for committing to a particular plan. If you plan well, the savings can be significant. But if you don’t use the capacity you’ve bought, the savings might not materialize. To improve cloud resource planning, you need to be able to evaluate your potential savings in varied scenarios.

Workload Drift
Over time, your workloads evolve due to changing demand, data volumes, etc. As a result, they can outgrow their current configurations, leading to performance issues or rising costs. This is a natural result of business growth, so the goal isn’t to prevent that drift. Instead, you want to optimize capacity and cost on an ongoing basis to keep your workloads right-sized.

Slow Problem Troubleshooting and Remediation
No matter how many precautions you take, something somewhere will inevitably go wrong. And once you become aware of the situation, the clock starts ticking to find the cause and fix it, all of which takes time. The longer the entire process takes, the longer your infrastructure will be visibly and painfully not working as expected. For those cases, you need full-stack visibility with application-centric, AI-based analytics to minimize impact duration as much as possible.

Keeping the engine running behind the curtain can be a daunting task, but with proper planning and management, your IT team will remain visible to those responsible for managing it and invisible to those in the rest of the organization. Of course, the complexity of today’s cloud infrastructure makes that challenging, and there will be IT fire drills that take place, but it’s about how IT leaders react to the situation and resolve it in the quickest way possible, allowing themselves to return to their preferred place in the shadows.

July 7, 2023
The Difference Between Capacity and Scalability Planning

Not only does the tech industry love terms, we love to make them super nuanced. For example SecOps versus DevSecOps, continuous delivery versus continuous deployment, hybrid cloud versus multi-cloud, etc. These terms may seem similar on the surface, but when you dig in they are quite different.

Here are two more: capacity planning versus scalability planning. Both are critical aspects of keeping applications up and running, but two different activities.

The reason we invest time in capacity and scalability planning is simple: We want to make sure that system resources such as compute, storage and memory are not the cause of an application outage. It is a response to expected and unexpected increases in application usage, as well as the steady growth of application adoption.

Here are a few differences between capacity planning and scalability planning.

Long Term Versus Short Term

Capacity planning is more focused on the base resources, as it relates to the long-term growth in application traffic. The goal is to have enough resources to handle the steady state of the application without breaking the bank by over anticipating usage. With capacity planning there is a big assumption that the application requires to always have a pool of resources available.

Automated Versus Static

Scalability could be a manual task, you see utilization creeping up and you respond with more instances. This is probably more common than not. But I don’t believe it’s the overall objective of most teams. Auto-scaling is the goal, where the system knows on its own when it’s time to spin up new resources. The public cloud providers, instrumentation of kubernetes, etc., help support this. Whereas capacity planning is a more deliberate identification of needs, and then statically setting those resource needs within the cloud.

Strategic Versus Responsive

Similar to static versus automated, capacity planning is a more strategic activity whereas scalability planning is a more responsive activity. Usually, scalability comes up during peak usage activities such as black Friday sales or school registrations.

Capacity planning generally calls for more research and leveraging tools such as load balancers and historic data to understand the best path forward. It also should involve the input from the broader team, especially development to discuss the resource usage of new functionality and application architectures.

There is a good possibility that scalability planning could be all that is left with some modern applications. The reason this might be true is resource allocation theoretically can be 100% application based, where the resources are always one step ahead of current utilization. In order to embrace this model resources need to be provisioned 100% automatic, which means that all infrastructure needs to be scripted. As more companies adopt serverless components of their application, this is already inherently true.

Capacity planning is where most operations teams spend their time, but resource utilization is shifting left and scalability planning is becoming more common. Scalability planning has become the responsibility of DevOps engineers, SREs and even developers. The main reason capacity planning will survive the long haul relates to budgets. While the application itself can fully control the infrastructure laid out before it, the IT organization and the organization itself will want to impose an upper limit so there is an indicator of when the cost of running the application on the resources may or may not be an issue. Capacity planning in this case will be purely a cost control mechanism, and negotiated just as any budgets are.

— Chris Riley

January 17, 2020
Capacity planning isn’t dead, but it’s not about supply any more

DevOps, flexible cloud architectures and continuous delivery have all simplified a lot of equations in IT today, but there is one element that’s grown infinitely more complex: capacity planning. With the ability to spin up new environments on a dime through services like Amazon and Rackspace, the bucket of potential IT supply has grown infinitely more flexible. Meanwhile, as DevOps practices helps organizations speed up the continuous delivery cycle of designing, building and running new application instances, the demand for computer power and storage space is constantly changing. This week at Camp DevOps Houston (link), Steve Wilson with VMTurbo will explore what this means for capacity planning in his talk “Is Capacity Planning Dead?”

“In this cloud day and age, it really isn’t about capacity anymore, because capacity becomes a figurative thing with IT borders and the data center really just melting away,” Wilson says. “I can rent if I need to. Really this idea of capacity goes away, and what you really want to understand is where’s the cutoff where I stop buying and I start renting?”

Wilson believes that while capacity planning isn’t necessarily DOA in today’s DevOps world, it does need to be completely turned on its head. Rather than focusing on supply-side issues, planners need to concern themselves much more with demand.

“For eons and eons, we have been really good at managing supply—because it never changed. It was finite. It was like ‘OK, I’m going to put alerts, and I’m going to put thresholds, and I’m going to set the bar, because I know that when my supply runs out, that’s bad,’” Wilson says, explaining that capacity planning was all about moving thresholds to ensure IT never ran out of supply. “But now, if it’s a truly software-defined data center, it’s no longer defined by your infrastructure. So what if I could actually manage demand — not supply.”

The idea, he explains, is to gather a deep and continuous understanding of what demand is in order to buy infrastructure for the average demand and then ‘rent’ cloud infrastructure to soak up the peaks of demand when they present themselves.

“If I understand demand, I can buy for my 50th percentile plus maybe one or two standard deviations, and then rent above that and scale out,” he says. ” then all I really want to do is collapse that line of getting a good understanding of where that financial line is where it is economical for me to extend CapEx internally, and then that tipping point where it’s actually cheaper for me to extend OpEx to handle the peaks.”

With DevOps, he believes it may be easier for organizations to truly understand the demand profile of different applications, due to greater collaboration between developers and operations staff. The trick is to ensure that capacity planners are roped into the design, build, run cycle from the beginning, as they are currently operating in the fringes, frequently using spreadsheets that are outdated as quickly as they are made when an organization is delivering continuously.

“Suddenly, every time you try to make a change, you try to deploy, you’re playing whack a mole, and then you start to break down. One place you’re hot, the other place you’re cold, and you don’t understand the demand workload and you can’t move it around and you’re just managing supply,” he says. “If I can always understand and map demand properly, I can automate the crap out of that, because now I’m basically creating just-in-time compute. If I can run computes super lean, then at peak I can just rent.”

October 28, 2014