devopsy.kinsta.cloud

Tag: downtime

Most Outages Don’t Announce Themselves

Many outages never announce themselves as outages. They show up as rising latency or an error rate that creeps from 2% to 4% over an afternoon while everyone’s busy with something else. The site is up. Nothing has paged. Something is still wrong, and by the time it’s obvious, it’s been wrong for an hour.

Catching the server that falls over is the easy case. The hard one is deciding which of these slow, quiet changes should pull someone out of bed. Most teams get that wrong, and they get it wrong by monitoring too specifically.

Start From a Baseline

A CPU spike on one box out of a hundred running the same workload isn’t worth a phone call. The same spike on your only server might be. It depends entirely on the case, and that’s the part people skip.

The number on its own barely tells you anything. Take a 2% error rate. If it’s been flat at 2% all year, that’s your baseline. It’s what normal looks like for you, and it doesn’t need an alert. It needs one when it moves, and moves by more than your usual variation.

Monitor the Outcome

The same problem shows up in how most teams watch an application: Every part in isolation. A check on each endpoint, each page, each box. All of them can return a clean 200 while the one thing your customer came to do is quietly broken.

Monitoring the outcome works better. Log in the way a user would, and measure whether the login completes, how long it takes, and what error you get when it fails. Or walk the whole path: Open the site, drop a couple of products in the basket, go to checkout, and confirm you reach the payment screen. Now you’re testing what the user actually experiences.

It also tells you what to ignore. If a few of your hundred servers drop out but login still works, send a note to a non-urgent channel and look in the morning. If nothing the user touches is affected, it doesn’t warrant a page.

When Outages Actually Happen

The timing is worth taking seriously. Across 1.8 million outages on thousands of sites, Oh Dear found that about 68% started outside the 9-to-6 workday, on weeknights and weekends. The same data is a caution about averages. The median outage was over in under two minutes, but the mean was close to 22, because a thin tail of multi-hour failures pulls it up.

So design your alerting around the rare long outage, the one that runs for hours, and assume it lands outside working hours, because most do. The common short one usually clears before you’ve even read the page.

Alert Fatigue

Alert fatigue wrecks more teams than any single outage. Monitor every detail and you drown in notifications, and after a while, they stop meaning anything. The on-call engineer glances, dismisses, goes back to work, and one day dismisses the alert that actually mattered along with the rest. It’s the boy who cried wolf.

The fix I trust most is dull: At a hosting company where I ran engineering, the team sat down every Monday and went through the past week’s alerts, urgent and not. For each one, we asked out loud whether it was useful, whether it warranted an action, and whether the severity was right. The point was to bring the off-hours noise down deliberately, and to move the work earlier — to catch the warning sign at 2 in the afternoon and deal with it then, instead of being paged for it at 3 in the morning.

Anomaly detection helps, but only with a duration attached. A 2% baseline that jumps to 10% for thirty seconds and settles again probably isn’t worth waking anyone. See it in the morning, write a short postmortem, tune the alert. A spike that’s already gone is tomorrow’s job. Get that distinction right and you shed a lot of pages without missing anything that counts.

One Question for Your Team

There’s one question worth putting to a team out loud: Would a customer notice this before we do? People in the middle of using your product spot a slow page or a failed checkout fast. If they’re the ones telling you, your monitoring missed it.

July 7, 2026
Survey: IT Teams Spend About a Third of Time Responding to Disruptions

A global survey of 1,700 IT professionals finds on average teams are grappling with 280 hours of median annual downtime a year, which at 12 hours of a 40-hour work week equates to about 30% of their time.

Conducted by Enterprise Technology Research (ETR) on behalf of New Relic, the most common causes of downtime over the past two years have been network failure (35%), third-party or cloud provider services failure (29%) and human error (28%).

The survey also estimates that IT downtime costs businesses $376.66 million annually, with median total annual downtime involving a high-impact outage lasting 77 hours, or three days. A full 62% said high-business-impact outages cost their organization at least $1 million per hour of downtime.

Nearly half of organizations (45%) use more than five observability tools to reduce downtime, with the median average of budget dollars allocated being $1.85 million.

Conducting root cause analysis (RCA) and post-incident reviews (37%), monitoring DevOps Research and Assessment (DORA) metrics (34%), monitoring golden signals (33%), and tracking, reporting and influencing the mean time required to detect and resolve outages (33%) are the top best observability best practices being followed. The median annual value received from observability is $8.15 million. However, only a quarter (25%) said their organization has achieved full-stack observability.

On average, those with full-stack observability experienced 79% less downtime per year than those without and incurred 92% less outage costs per year. They also had a 27% lower annual observability budget and were 51% more likely to learn about interruptions with observability.

Nic Benders, chief technical strategist at New Relic, said the survey makes it clear that organizations need a lot more than access to log data to prevent and minimize disruptions. Log data as an early warning system is itself not very valuable, he added.

In general, the adoption of artificial intelligence (AI) technologies and an increased focus on security, governance, risk and compliance at 41% each are the top drivers of observability spending.

At least half had deployed one or more core observability capabilities, including security monitoring (58%), network monitoring (57%), database monitoring (55%), alerts (55%), dashboards (54%), infrastructure monitoring (54%), log management (51%) and application performance monitoring (APM; 50%). More than a third had deployed digital experience monitoring (DEM) capabilities, including browser monitoring (44%), error tracking (43%), and mobile monitoring (35%).

More than half (51%) of respondents used an open-source solution for at least one observability capability. More than a third (38%) were using Grafana, 23% were using Prometheus and 19% were using OpenTelemetry. Respondents are also relying on open source software for AI monitoring (31%), synthetic monitoring (28%), distributed tracing (28%), application performance monitoring (27%), Kubernetes monitoring (27%) and AIOps capabilities (26%).

Overall, 42% of respondents noted their organization is already applying artificial intelligence (AI) to monitoring, with 29% employing machine learning and 24% having embraced AI for IT operations (AIOps) platforms. An additional third is expected to deploy artificial intelligence for IT operations (AIOps) capabilities (39%), AI monitoring (36%) and machine learning (ML) model monitoring (34%) in the next year.
A total of 40% have also business observability (40%) as part of an effort to correlate business outcomes in real-time with telemetry data. On average, respondents who had deployed business observability experienced spent 25% less time addressing disruptions compared to those who hadn’t, the survey finds.

Finally, the survey finds there is a 2-to-1 preference for a single, consolidated platform over multiple tools they need to integrate. In addition, 41% said they plan to consolidate tools in the next year.

The one clear thing is observability that goes beyond merely monitoring a set of pre-defined metrics remains a work in progress. The challenge, of course, remains to turn all the telemetry data available into actual actionable intelligence.

November 1, 2024
Five Lessons Learned the Hard Way From the Global Blue Screen of Death

As a product management professional, I am trained to understand customer pain, especially for customers who have challenges with downtime due to poor software quality. Last Friday I experienced the problem firsthand though when I arrived at Heathrow Airport early, only to discover that the Microsoft outage had grounded flights across the world. My flight from London was only delayed a couple of hours, but my connecting flight was canceled completely. Since it was the last flight of the day to my hometown, I had to find a hotel room for the night, but all of the rooms within 30 miles of the airport were sold out. All of this is because of a bug in one small file not much larger than this article, impacting millions of people across the world.

Most software releases contain defects. I have never seen production code that didn’t have issues, especially enterprise-level applications. Findings that are released to production are usually cosmetic, or minor bugs that impact a seldom-used feature. Defects like the one that took down the airlines, however, are another situation entirely. Now it’s time to consider what lessons we can learn from this event.

Lesson one: Shift Right is NOT optional. Shift Right is the concept that you should test in production, after the release. You have to wonder if this was done in the case of last week’s outage. Even if testing was performed in the pre-release environments, you can’t be sure that your pre-release environments are the same as production. You must smoke-test in production at a minimum. Better yet, run regression tests regularly. Even when testing in production, testing a single process or component may not be enough. Which leads us to the next lesson.

Lesson two: Integration and end-to-end system testing is a must. Most enterprise-class applications today are built from a complex collection of interconnected services and systems. Each service in isolation can function precisely as expected, but when you test it together with all the other moving pieces, you may find that small delays here and there will break the end-to-end process. These timing-related issues are exacerbated by unusually high volumes, which brings us to the next lesson.

Lesson three: Performance and load testing are important too. Performance and load testing may not seem important under normal circumstances, but when recovering from an outage, there is typically a bow wave of users trying to get on with the delayed work. This could be considered a cascading effect, but the situation often rears its ugly head at month and quarter end for some businesses.

In the case of last week’s incident, a huge number of canceled flights led to an increase in volume on the airline reservation system. A system that may not have been impacted by the actual bug is impacted by an unusual level of activity. Load testing is not an easy job, and it may not be necessary to do it regularly, but you shouldn’t wait until disaster strikes to learn that a 20% above-average volume will render a system unusable.

Lesson four: You don’t have to release to everyone all at once. Two concepts would have helped prevent a global meltdown. The first is “dogfooding” aka eating your own dog food. This is the practice of using your own software before releasing it to others. That way if there are major issues, you find them fast and you contain the suffering in your own organization.

The second concept is called canary releases. Just as miners used to take canaries into the coal mines to warn of a deadly condition, so too can you release code to a small subset of external users for a short period to ensure that a release works outside of your organization. Once again, you contain the negative impact.

Lesson five: The cost of system outages has cascading impacts. It’s not just the lost productivity during the downtime but it’s also the domino effect. For example, downtime in the order entry system can cause challenges with the inventory management system which can delay shipments and impact the top line as well as the bottom line. Lost revenue and profit may cause shareholders and customers to vote with their feet.

Lost productivity is compounded by the need for overtime to catch up. For hourly workers, this is a direct cost to the company. For salaried employees, like developers and testers, this means a negative impact on their job satisfaction and time away from their families. Increased turnover in IT has a cost as well.

The decline in customer satisfaction is another indirect cost of poor quality that is not always tied directly back to an outage. Even if your competitors were impacted by the same incident, your customers may not know that. They were traveling with you that day.

The next time you are negotiating a budget for testing tools and resources, remember that there are not only significant downtime costs, but many indirect costs as well. Sometimes I think the cost of testing tools and personnel should be classified as insurance because it is. The fact that bugs very seldom lead to downtime means the pain is most often not acute. We live with it. Our customers live with it. Until one day it knocks you off your feet.

Don’t wait until the pain is unbearable. Invest in people, processes and tools to ensure that you will never be put in the position of explaining why a bug in one small file impacted the entire world.

July 25, 2024
Observability Leaders: The Mighty Minority Holding Their Own Against Outages
The job of IT teams has become much harder thanks to the new realities of hybrid IT, with workflows occurring across on-premises and cloud-based networks and applications. Yet so much rests on your ability to respond to these challenges, all while managing the expectations of your customers and the health of your bottom line. With the growing complexity of ecosystems comes the need for better visibility, both into specific systems and across your entire stack.

Observability solutions address this by providing visibility across your network, infrastructure, systems, application, database, digital experience and log monitoring.

In other words, observability enhances your ability to observe your digital ecosystem with more transparency, more effective detection and more intelligent insights. Take, for example, the very timely and pressing issue of outages. On average, the typical business experiences nine brownouts or outages every month, lasting roughly twelve hours each and averaging $13.7M annually, according to new research we conducted at SolarWinds.

Over the course of the year, that can add up to nearly 1,300 hours, with outages chipping away at your organization’s productivity, reputation and competitive edge.

It’s not enough to monitor for these outages and address them once they take down your systems. Repeated or prolonged outages signify a gap in the enterprise’s ability to comprehensively monitor and understand its network’s internal state and performance. A robust observability strategy can preemptively detect anomalies and potential issues—before they become full-blown outages.

According to new data from the 2023 IT Trends Survey released this month by SolarWinds, companies who’ve embraced observability tend to not only lead in detecting, remediating and preventing outages—they’re experiencing better business and IT outcomes, too.

The technology professionals surveyed shared how observability is helping them achieve better performance, compliance and resilience in digital environments—and how you can follow their lead.

Observability Drives Better Business Outcomes

The observability leaders surveyed were 300% more likely to say their organization is doing extremely well with growing revenue and 250% more likely to say the same about their organization’s speed of innovation. Interestingly, they also reported a better employee experience: They were 388% more likely to rate their organization as doing “extremely well” with employee experience and also reported fewer and smaller skill gaps on their teams.

So, how does embracing observability actually translate into better business outcomes? According to the data, observability leaders are…
- Investing in top priorities: Data shows organizations using observability solutions to support the priorities most critical to their growth and success: Improve their customer experience (96%), enable faster innovation (71%), reduce time spent solving (71%) and detecting (60%) issues and increase operational efficiency (55%).
- More automated and integrated: Observability leaders embracing automation and investing in tools that provide enhanced efficiency are 214% more likely to say they are doing extremely well with operational efficiency, 750% more likely to say they are doing extremely well with auto-remediation of complex alerts and 300% better at automatically collecting background diagnostic data for IT support staff.
- Ahead on IT: The data found that those ahead of the curve on observability are also leading by huge margins when it comes to monitoring, detecting and resolving issues that could otherwise bring the business to a screeching halt. When it comes to IT, they are 233% better at auto-escalation of tickets, 213% better at auto-remediation of simple alerts and 36% better at settling alert levels based on historical behavior.
How to Reap Observability Benefits in Your Organization

All told, observability adoption is still in its early days. Only 1% of enterprises are already up and running using observability tools on a regular basis, while another third are undergoing the implementation process.

That’s good news for other businesses who are ready to reap the rewards—it’s not too late to be an observability pioneer.

If you’re interested in bringing observability to your own organization, start by addressing a few important factors that can help you sidestep challenges—such as the accelerating pace of change in technology (72%) or insufficient budget (58%)—that might otherwise obfuscate your observability opportunities:
- Make sure your observability solutions are appropriate for multi-cloud environments: Multi-cloud environments have become standard for many companies because of the many upsides, but they can be challenging for IT teams to manage. Strategically accelerate your organization’s cloud migration with an observability solution designed for complex hybrid and multi-cloud environments to save a lot of headaches down the road.
- Give your teams autonomy—and build toward autonomous operations: Observability brings your enterprise a step closer to autonomous operations. Autonomous operations, where IT systems can govern themselves, providing a proactive approach to addressing issues, rather than reactive, won’t replace your human teams but rather free up their time and brainpower to support the business and drive innovation.
- Don’t ignore the database: Databases represent the most difficult ecosystems to observe, tune, manage and scale. They require significant computing power, tend to be memory-intensive and likely make up the bulk of your organization’s cloud spending. Tasks crucial to the health and functionality of your business—including troubleshooting, root cause analysis and remediation—simply cannot happen without full, transparent observability across the numerous resources a database needs to engage to do its job.
Observability helps balance competing priorities and demystify application requirements while increasing your overall DevOps agility, efficiency and speed to market. Observability not only fosters transparency and collaboration but also encourages a system of ongoing review, which benefits the organization, IT teams and—crucially—your customers.
September 22, 2023
Fire at Data Center Causes Chaos | 20% Costlier Cloud

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: A South Korean conflagration leads to a ridiculously long outage, and the price of public cloud is skyrocketing. (more…)

October 20, 2022
Dependability is Crucial for Real-Time Experiences
Millions of people around the world experienced a major disruption after an October 2021 outage saw Facebook, WhatsApp and Instagram go offline. A few days prior, the popular workplace collaboration platform, Slack, also went down for some users. And in the last few weeks of 2021, Amazon’s AWS caused considerable global chaos after significant networking issues in its busiest data center, US East 1, disrupting streaming services such as Disney+ and Netflix.

Downtime can be costly, causing organizations to lose $5,600 per minute and up to $300,000 per hour, on average, according to Gartner. But it’s not only financial and reputational loss that businesses need to worry about. Outages also seriously downgrade—and in the case of real-time experiences, completely disrupt—the customer experience.

At a time when consumers are demanding more live and interactive digital experiences, organizations can ill afford disruption or downtime, as customers quickly move to other services that ‘just work’.

It’s not just the capacity within a real-time system that matters, but also globally distributed failover capabilities when something does go wrong. While organizations could try to build their own failover mechanisms, it’s extremely difficult to failover so services continue seamlessly, regardless of the status of its underlying infrastructure. Simple failover systems not only take longer to resume but typically suffer data loss during the time that a region is disrupted. Businesses need to know that even if, say, AWS suffers an outage, they won’t be affected.

This is why it’s vital that organizations get it right when looking to develop their own real-time infrastructure. Managing tens of thousands if not hundreds of thousands of concurrent connections while synchronizing data in real-time means that if anything does go wrong it’s immediately obvious to users. If tech behemoths like Amazon and Facebook can’t get it right, what chance do organizations with far fewer resources have when building and managing their own real-time infrastructure?

The Steps to Success With Real-Time Infrastructure

Regardless of whether it’s for synchronized multi-user collaboration, audience engagement or instant updates, developers have a hard enough time developing even the most basic of real-time infrastructure, let alone making sure it’s dependable. There are four key elements to achieve dependability: Predictable performance, data integrity, reliability of service and high scalability.

Predictable performance: When we think of performance we usually imagine more computing power or speed. But when it comes to real-time, it’s not just about minimizing latency and bandwidth, it’s about having a predictable system. If developers have global latencies and bandwidth within certain operating boundaries, it provides certainty in uncertain operating conditions. This makes it easier to design, build and scale features around this certainty.

Data integrity: As we all know, the internet can be an unreliable network. Users abruptly disconnect and reconnect all the time, resulting in data that gets lost, is delivered in the wrong sequence or that is sent multiple times. As a result, organizations must ensure their infrastructure can guarantee high integrity without sacrificing anything else.

Reliability of service: The live nature of delivering updates in real-time requires total reliability. Outages can cause distrust and annoyance from users. Organizations could find their reputation taking a hit or customers quickly jumping ship if reliability proves to be a recurring issue. To prevent this, organizations should develop infrastructure that’s fault-tolerant and redundant at both a global and regional level so they can deliver a great experience even if there are multiple failures.

High scalability: More often than not, delivering an experience in real-time can result in unpredictable traffic; for example, a huge number of people wanting sports updates for a World Cup game. If thousands of devices connect over a short period of time but capacity can’t scale up quickly enough, then an outage will occur. To avoid this, developers must use infrastructure that can be elastic and meet changing levels of demand. Otherwise, the user experience is seriously affected and real-time capabilities crash when the network is swamped with connections it can’t cope with.
While organizations often build their own infrastructure with the best of intentions, it’s easy for them to run into trouble. Most lack the internal expertise, resources and time; plus, going it alone often results in a huge technical debt. It’s also a massive undertaking, due to the amount of ongoing complexity and cost involved.

Third-Party Partnerships Enable Dependability

Unless organizations have the resources, finances and expertise of the likes of Amazon or Google, then building your own real-time infrastructure is just setting yourself up for failure. Instead, they should save on development cycles and focus on generating revenue more quickly by offloading their real-time infrastructure needs to a third party. This will mean organizations can design without limitations and focus on solving the challenges that really matter to them while putting in place real-time infrastructure that is as dependable and future-proof as possible.
February 15, 2022
How IT Ops Teams Prepare to Prevent Web Downtime
IT operations teams can anticipate most periods of peak web traffic. The key to preventing outages and issues is to plan ahead. Here at BigPanda, we’ve prepared a checklist of the key factors IT ops teams need to consider to ensure their IT infrastructure is ready.

Here’s is a recommended checklist for IT and ops teams:
- Review monitoring metrics and work with your developers to test applications for stability. Some companies may need additional support to accommodate peak seasons and events.
- Perform regular security tests and ensure necessary measures are taken to protect your systems. Exposure to hackers and the potential for unanticipated load are real threats.
- Prepare for alert storms with an effective correlation platform. For IT ops teams, spikes in workload can cause what is known as alert storms. Reduce risk by utilizing an alert correlation platform that identifies patterns in unstructured alert data to separate signal from noise. This allows you to more effectively identify, triage and take action on issues before they have a chance to affect your customers.
Companies also run a higher risk of downtime during peak traffic times. Seventy-five percent of outages are due to unplanned configuration changes to a system—when IT ops teams find something they think might cause a problem and try to fix it immediately, unintentionally creating a much bigger issue for the web or mobile site.

To avoid unexpected downtime, we recommend companies take the following steps to ensure the availability and reliability of their services:

1. Identify what is mission-critical.

To avoid unexpected downtime, we recommend that IT ops teams tier their services and identify the systems that are mission-critical to the business. Top-tier applications should include those that are linked directly to the success or failure of the business, such as point-of-sale, ticketing or billing.

2. Develop an ironclad failover plan for top-tier systems.

Offering a high level of availability is not something that happens by chance. It must be planned carefully for every aspect of your systems architecture. Top-tier systems should be bolstered by an ironclad failover plan—one that plans carefully for load capacity to handle unexpected spikes.

3. Invest in a best-of-breed monitoring stack.

You can’t protect against what you don’t see coming. In the age of continuous integration and continuous delivery, the only way to ensure that you have an accurate pulse on the health of your IT systems is to implement the best monitoring tool for each layer of your stack (such as systems monitoring, application monitoring, web and user monitoring, logging and error tracking). The industry rapidly is replacing monolithic monitoring architectures with this “best-of-breed” approach to better service increasingly complex and dynamic IT systems.

4. Implement alert correlation to distinguish signal from noise.

More tools—monitoring more moving parts—lead to more noise. It’s a simple fact. To efficiently identify, triage and remedy potential issues before they have the chance to do real damage, IT teams require a way to properly separate the signal—the “real problem”—from the many sources of noise. By implementing an alert correlation solution, IT teams are able to see how alerts from their various monitoring tools are related, allowing them to quickly filter non-critical issues and focus on what matters most.

— Assaf Resnick
January 6, 2017