Tag: alert fatigue

Reducing MTTR: A Practical Guide to Correlating Incidents with AIOps
At 2 a.m., a payment service starts throwing errors. Within minutes, your observability stack fires off 40 alerts: Elevated latency on three services, a spike in 5xx errors, a memory warning on a downstream cache and a handful of dependency timeouts. Somewhere in that noise is the one signal that actually explains what broke. Finding it manually, alert by alert, is how a 5-minute fix turns into a 45-minute outage.

This is the core problem AIOps was built to solve — not more dashboards, not more alerts but correlation. By automatically grouping related signals across metrics, logs and traces, AI-driven correlation collapses an alert storm into a single, prioritized incident with a probable root cause attached.

In this guide, you’ll learn how incident correlation actually works, how to set it up on top of your existing observability stack and how to measure whether it’s actually reducing your mean time to resolution (MTTR). By the end, you’ll have a practical framework for moving your team from manual alert triage to AI-assisted incident response.

Why MTTR Breaks Down Without Correlation

MTTR is typically broken down into four phases: Detection, triage, diagnosis and remediation. Most teams invest heavily in detection — that’s what observability tooling is good at. The bottleneck almost always sits in triage and diagnosis, for a simple reason: One root cause typically produces many alerts, and someone has to manually figure out which alerts belong together before the diagnosis of the actual problem can even begin.

A few patterns show up repeatedly:

Alert storms mask root cause. A single upstream failure can cascade into dozens of downstream alerts, each looking like an independent incident.

Signals live in silos. Metrics, logs and traces often sit in different tools or dashboards, so the engineer on call has to manually stitch together a timeline.

Tribal knowledge bottlenecks diagnosis. The person who ‘just knows’ that a cache-eviction warning usually precedes a checkout failure isn’t always the person on call.

AI-driven correlation directly targets this middle stretch of the incident life cycle — the time between ‘something is wrong’ and ‘here’s what’s actually wrong’ — which is usually where the most time is lost.

Prerequisites

Before you begin, make sure you have:
1. An existing observability stack producing metrics, logs and traces (StackGen’s ObserveNow or a Prometheus/Grafana/Loki/Jaeger-style setup)
2. Alerting already configured on key services, even if it’s currently noisy
3. Admin access to configure integrations and correlation rules
4. At least one recent incident with alert history you can use to validate correlation output
5. Estimated time: 30–45 minutes for initial setup, plus 1–2 weeks of tuning based on real incidents.
Step 1: Connect Your Observability Stack

AI-driven correlation is only as good as the data it can see. If your metrics live in one tool, logs in another and traces in a third with no shared identifiers, correlation has nothing to work with.

Start by making sure your telemetry shares common context — service name, environment and ideally a trace ID or request ID that threads through logs and traces.

# Example: Shared labeling convention across metrics, logs and traces

service: checkout-api

environment: production

trace_id: “{{request.trace_id}}”

If you’re using ObserveNow, connecting metrics, logs and traces into a single pane is largely a matter of pointing your existing Prometheus, Loki and Jaeger (or equivalent) instances at it — StackGen integrates with these directly rather than requiring a rip-and-replace.

Expected Result: A unified view where a single service’s metrics, logs and traces are queryable together, ideally with a shared identifier connecting them.

Troubleshooting: If correlation later feels inaccurate, this is almost always the first place to look — inconsistent service-naming or missing trace propagation breaks the AI’s ability to link signals correctly.

Step 2: Configure AI-Driven Correlation Rules

With unified telemetry in place, the next step is telling the correlation engine what ‘related’ should mean for your environment. Most AIOps platforms, including Aiden AI Copilot, combine two approaches:

Topology-Aware Correlation: Using service dependency maps to group alerts from services that call each other

Pattern-Based Correlation: Using historical incident data to learn which alert combinations tend to co-occur

# Example: Correlation policy

correlation_policy:

time_window: 5m

group_by:

– service_dependency

– shared_trace_id

confidence_threshold: 0.75

The time_window matters more than it looks. Set it too short and you’ll miss slower-cascading failures; set it too long and unrelated incidents start getting merged. Five minutes is a reasonable starting point for most microservice architectures — narrow it down if your services are tightly coupled with fast failure propagation, widen it for slower, more decoupled systems.

Expected Result: Incoming alerts are grouped into incidents rather than arriving as isolated tickets.

Troubleshooting: If everything gets grouped into one mega-incident, either your confidence threshold is too low or your time window is too wide. Tighten both and re-test against a known incident.

Step 3: Test Against a Real Incident

Don’t wait for a live outage to find out if your correlation setup works. Pull the alert history from a recent real incident and replay it against your new configuration.

Ask three questions about the output:
1. Did the correlation engine group the alerts that actually belonged together?
2. Did it surface a root-cause candidate that matches what you now know actually happened?
3. Did it leave out anything that should have been included or wrongly include something unrelated?
This is also where Aiden/AIOps automated root cause analysis is worth validating directly. It should be able to point to the earliest anomaly in the correlated group and rank likely causes, not just group alerts together and leave the analysis to you.

Expected Result: The correlated incident matches (or comes close to) the retrospective understanding of the incident, with the root-cause candidate ranked at or near the top.

Step 4: Monitor, Tune and Automate Remediation

Correlation isn’t a one-time setup. It improves while identifying more of your real incident patterns. Over the first few weeks:
- Review every correlated incident and confirm or correct the grouping.
- Watch for recurring false groupings (two unrelated services that keep getting merged) and adjust topology mappings.
- Track false-negative correlations — alerts that should have been grouped but weren’t.
Once correlation accuracy is consistently high, you can layer-in automated remediation for well-understood incident patterns — for example, automatically restarting a service instance when a specific correlated pattern (memory leak signature + latency spike + no recent deploy) is detected with high confidence. This is where Aiden for SRE typically comes in: Using the correlated, root-caused incident as the trigger for a pre-approved remediation action, rather than waiting for a human to read the alert and act.

Expected Result: A shrinking gap between detection and remediation, with routine incident patterns increasingly self-resolving.

Testing Your Setup

Before trusting correlation in production, run it through a structured validation pass:
- Replay 3–5 historical incidents of varying severity and complexity.
- Compare AI-correlated groupings against what your team determined manually during the actual incident.
- Check root-cause ranking accuracy — was the true cause in the top 1–2 candidates?
- Confirm the time to first correlated alert is meaningfully faster than your team’s manual triage time.
If correlation accuracy is inconsistent across incident types, it’s usually a sign that your topology mapping is incomplete for certain services, which is why it is worth revisiting Step 1 before adding more correlation rules on top of gaps in the data.

Real-World Results

Teams that move from manual alert triage to AI-driven correlation typically see improvement concentrated in the triage and diagnosis phases specifically, since that’s the part of MTTR correlation is designed to compress:
- Alert Volume per Incident: Often reduced from dozens of individual alerts down to a single correlated incident
- Time to Root-Cause Identification: Meaningfully faster when root-cause ranking is accurate, since engineers stop manually cross-referencing dashboards
- On-Call Burden: Fewer distinct pages per incident, since related alerts are grouped rather than firing independently
Your specific numbers will depend heavily on how mature your topology mapping is and how much historical incident data the correlation engine has to learn from — treat the first few weeks as a tuning period, not the final benchmark.

Next Steps

Now that you have AI-driven correlation running, you can:
- Extend correlation rules to cover infrastructure-layer signals (node health, network) alongside application-layer telemetry
- Pilot automated remediation on your highest-confidence, lowest-risk incident patterns first
- Expand from Aiden for SRE into Aiden for DevOps if pipeline and deployment failures are a significant source of your incident volume
August 13, 2026
How We Got Here: Alert Fatigue to Decision Fatigue
Operations teams have been battling alert fatigue for a very long time now. We saw monitoring systems multiply while cloud-native architectures increased system complexity. Every new microservice, dependency, and API introduced another stream of signals for folks to contend with.

As anyone who’s worked in SRE or on-call culture knows, this caused a lot of stress and burnout.

The industry responded by trying to deduplicate alerts, but that alone was not enough. Correlation engines had to mature and SRE practices had to formalize on-call discipline. Observability platforms took the place of monitoring stacks. Teams tuned thresholds.

Alert volume dropped significantly in many environments, but operational toil persists.

Why?

Because we reduced signal overload, sure, but we did not reduce decision load.

The Evolution From Signal Noise to Decision Pressure

Noise was the big problem in the cloud’s early days. Systems generated more alerts than humans could process, and a ton of them were redundant. This meant that the cognitive burden came from sheer volume.

In mature environments, alerts are fewer but more consequential, which changes the cognitive dynamic.

When every alert matters, they all demand judgment. But is this transient or systemic? Is it safe to restart a service or shift traffic? The questions are many, and the time to answer them is short.

This is decision fatigue.

Unlike alert fatigue, decision fatigue does not arise from too many notifications. It actually comes from too many high-stakes judgments, and in systems as complex as yours, ambiguity is constant.

Observability Answers “What,” Not “What Next”

Modern observability can correlate logs, traces, and metrics while surfacing anomalies and pinpointing regressions. It also answers critical questions, such as:
- What changed?
- Where did it change?
- When did it change?
- How does this event correlate with others?
Observability is powerful, but that’s all it is. It watches, rather than acts.

It doesn’t define which remediation steps are appropriate, and it doesn’t encode acceptable risk thresholds.

It also can’t determine when automation should intervene and when a human should stay in control. It does not provide guardrails for rollback if a response makes things worse.

As a result, the human operator remains the final decision maker.

This scales very well in small systems! In large enterprises with thousands of services and global traffic, not so much.

The paradox here is that the more mature an organization becomes in filtering alerts, the more responsibility concentrates on each remaining decision. The work moves from filtering noise to weighing consequences.

The Concealed Scaling Constraint

Enterprise technology leaders usually assume that improved signal quality automatically correlates to reduced operational burden, but the truth is that it redistributes it.

When systems are instrumented well, they surface anomalies with nuance and detect subtle deviations. But each of these is an invitation to make a choice.

The problem is that, without a defined framework for how decisions are made, the burden accumulates in the following ways:
- Engineers debate whether to automate remediation.
- Teams are reluctant to act without full context.
- Escalations multiply because ownership is unclear.
- Fear of unintended impact slows response.
See how cognitive load becomes the new bottleneck? The architecture gap is real; this is not a tooling failure.

Introducing Decision Architecture

If observability was the answer to signal overload, then decision architecture is the answer to decision overload.

Decision architecture is the deliberate design of how operational choices are made, delegated, and constrained within a system. It formalizes boundaries around when and how action occurs.

At a bare minimum, you need:

Defined confidence thresholds: Not every anomaly deserves intervention, and organizations must define what level of confidence justifies automated or semi-automated action. This also forces your team to have a frank but necessary conversation about tolerable risk.

Explicit remediation boundaries: Which classes of incidents are safe to automate? Which require human oversight?

Guardrails and rollback mechanisms: Every automated action should assume the possibility of failure. Safe rollback paths make automation less threatening and more reliable.

Clear ownership models: Decision authority must be explicit. When ambiguity exists about who can act, response drags and escalations soar.

Outcome instrumentation: Measuring alert counts alone is insufficient. Teams must measure the impact of every action.

These elements move organizations from insight to proactivity.

From Observability to Action Systems

It is useful to view operational maturity as a progression from monitoring to observability, but now it’s time to view the next step: action systems. Systems that not only surface insight, but also operate within predefined decision boundaries to respond safely and consistently.

This does not mean you’re removing humans from the loop. Rather, you’re shifting humans from ad hoc decision-makers to decision designers.

Organizations risk stalling at the insight stage without this evolution. They will know more about their systems than ever before, but they’ll still rely on human judgment for every meaningful response. That model just does not scale with system complexity.

Reducing Toil Requires Reducing Cognitive Friction

Operational toil is often framed as repetitive manual work, but that definition becomes more cognitive in mature environments. It is the constant evaluation of edge cases, the repeated validation of safe actions, and the mental overhead of determining whether a signal justifies disruption.

Reducing this toil requires more than better dashboards. It requires reducing cognitive friction. That happens when decisions are constrained, structured, and delegated within safe limits.

When teams design systems with decision architecture in mind, they’ll notice several changes:
- Incident response becomes more consistent.
- Escalations decline because authority is clear.
- Automation expands safely within defined scopes.
- Engineers focus on improving systems, not adjudicating every anomaly.
Alert fatigue is visible and measurable. Decision fatigue is subtler. It hides in hesitation and burnout. But it is no less real.

The Next Decade of Operational Maturity

Enterprises have made significant progress in managing signal overload. We know this because observability is now table stakes for modern systems. The challenge ahead is not collecting more data or visualizing it more elegantly; rather, the challenge is designing how decisions are made at scale.

Organizations that treat observability as the final stage of maturity will, unfortunately, continue to struggle with cognitive bottlenecks. Those who invest in decision architecture will reduce alerts, yes, but also the mental burden associated with acting on them.

We solved alert fatigue, and we should be proud of ourselves for that. Now we must turn our attention to solving decision fatigue and achieving greater IT clarity.
March 9, 2026
On-Call Rotation Best Practices: Reducing Burnout and Improving Response
It’s 2:47 a.m. Your phone buzzes. An alert fires again. You acknowledge it, diagnose the issue half asleep, patch it, write a quick note and crawl back to bed. Three hours later, you’re at your desk like nothing happened.

If that sounds familiar, you’re not alone. On-call duty is one of the most important — and most mismanaged responsibilities in engineering. If done right, it protects your systems and distributes the load fairly. If done wrong, it destroys team morale and drives your best engineers to the door.

According to the 2024 State of Engineering Management Report, 65% of engineers reported experiencing burnout in the past year. On-call stress is a major contributing factor, and it compounds quickly when rotations are poorly designed, alert noise is high and there’s no automation to catch the easy stuff.

This guide covers the on-call best practices that high-performing SRE and platform engineering teams actually use: Rotation models, compensation approaches, alert hygiene, tooling selection and how automation is changing the on-call equation.

What Makes On-Call Unsustainable

Before fixing your on-call rotation, it helps to understand why most rotations break down.

The core problem is rarely the concept of being on-call — it’s the accumulation of bad patterns that make it unbearable. On-call engineers typically allocate 30–40% of their bandwidth during an on-call period to incident responsibilities. When that load spikes beyond sustainable thresholds, or when rotations are unfair, the effects cascade fast.

The following are the most common failure modes:
- Alert Fatigue: Too many low-signal, non-actionable alerts hitting the same pager. The Google SRE Workbook recommends a maximum of 2–3 actionable incidents per shift as a sustainable baseline. If your team is consistently seeing 8–10, you don’t have an on-call problem — you have an alerting problem.
- Unbalanced Rotations: Small teams with outsized coverage responsibilities. When less than five engineers share 24/7 coverage, each person gets paged far more often than they should. This accelerates fatigue and creates fragile single points of knowledge.
- Lack of Tooling and Runbooks: Engineers who wake up at 3 a.m. without documented procedures are forced to wing it under pressure. This leads to longer mean time to resolution (MTTR), more stress and decisions made without context.
- No Separation Between On-Call and Project Work: Google’s SRE philosophy explicitly reserves at least 50% of SRE time for project work. When on-call bleeds into sprint goals and delivery timelines, engineers feel perpetually behind — even when they did everything right.
Choosing the Right On-Call Rotation Model

There is no universal on-call rotation. The right model depends on your team size, geographic footprint and service criticality. Here are the three most common patterns, and when to use each.

1. Weekly Rotational Schedules

The most common model. One engineer carries the primary pager for a defined period — usually one week — with a secondary backup available for escalation. The handoff occurs on a fixed cadence with a structured knowledge transfer.

This model works well for small- to mid-sized teams with a single time zone. The main risk is that weekly shifts can feel long when the alert volume is high. The mitigation is a strict cap on pager load and a clear secondary escalation path, so the primary isn’t carrying the weight alone.

2. Follow-the-Sun

For distributed teams across three or more time zones, follow-the-sun (FTS) distributes coverage, so each regional team owns its daylight hours. With three sites spanning the U.S., Europe and APAC, this model can reduce on-call duration per engineer by as much as 67% because no one works overnight.

The overhead is real: FTS requires reliable handoff procedures, strong documentation and enough engineers in each region to make it viable. However, for teams with global presence, it dramatically reduces fatigue while maintaining 24/7 coverage.

3. Round Robin

Every eligible engineer cycles through on-call responsibilities in a fixed order. This model distributes load evenly and exposes more engineers to incident response, which builds organizational resilience and cross-functional knowledge.

Round robin works best in environments where the alert load is moderate and manageable. It pairs well with shadow rotations, where newer engineers observe experienced peers before carrying the pager independently — a practice that builds confidence and accelerates ramp-up.

Seven On-Call Best Practices That Actually Work

Here is what consistently separates high-functioning on-call programs from ones that churn through engineers.

1. Cap Incident Load Per Shift

Set a hard expectation: If on-call incidents consistently exceed the team’s defined threshold (Google recommends 2–3 actionable incidents per shift), the rotation schedule is not the solution. The alerting stack needs an audit. Every alert in your stack should pass a simple test: Has this required human action in the last 90 days? If the answer is no, route it to a non-paging channel or delete it.

Categorize every alert as actionable (immediate response required), informational (useful context, no action needed) or noise (false positives to eliminate). Pruning the noise category alone often cuts pager load by 30–40%.

2. Standardize the Handoff Process

A clean handoff is the difference between a confident on-call engineer and one who walks into a minefield. Establish a structured weekly transition meeting — 30 minutes is sufficient — where the outgoing and incoming engineers review active incidents, silenced alerts and upcoming risky changes. The incoming engineer summarizes back before the outgoing engineer signs off.

Document this in your runbooks. The handoff summary should also be posted to a shared Slack or Teams channel visible to the entire SRE organization, so context is never trapped in a single person’s head.

3. Build and Maintain Runbooks for Your Top Incidents

Identify your five most common incident types from the last quarter. For each one, write a runbook: Specific commands, specific dashboards, specific escalation contacts. A runbook is a checklist, not a manual. It should answer what do I do right now? — not what is the architectural history of this service?

Runbooks dramatically reduce MTTR, lower the cognitive load on on-call engineers and reduce the fear that makes burnout worse. They’re especially valuable for junior engineers or anyone onboarded into a complex system quickly.

4. Add a Shadow Rotation for new Engineers

Before any engineer carries the pager independently, they should shadow an experienced colleague through real incidents. This shadow layer is added to the rotation schedule explicitly — not as an informal mentorship, but as a structured step in on-call readiness.

Shadow rotations build confidence, surface knowledge gaps early and accelerate the time it takes for new team members to contribute to incident response rather than just observe it.

5. Track Four Key Metrics

Gut feelings about on-call health are unreliable. Four metrics give you the data to make decisions and justify rotation changes or additional headcount to leadership:
- MTTR: From alert to resolution — the primary health metric for your incident response program.
- Alert Volume per Shift: Total alerts fired vs. actionable alerts acted upon.
- On-Call Load Distribution: Are alerts evenly distributed across engineers, or is one person absorbing most of the load?
- Incident Recurrence Rate: Are the same issues recurring? Recurrence signals a remediation gap that automation or better runbooks can address.
Review these metrics on a monthly cadence. They will tell you when your rotation model needs to change before burnout does.

6. Compensate On-Call Fairly

On-call is work. It disrupts sleep, social plans and recovery time. Engineering teams that treat on-call as an informal obligation — without compensation, time back or formal acknowledgment — send a clear message: Your time outside business hours doesn’t matter.

The compensation model varies by organization. Some provide direct pay for on-call shifts, particularly for out-of-hours paging. Others offer compensatory time off after heavy weeks. What matters is consistency and transparency. Engineers are far more willing to participate in on-call when they know the program is fair and the organization recognizes the burden.

7. Conduct Blameless Postmortems

Postmortems are the learning mechanism of on-call programs. When something goes wrong — and it will — the goal is to understand what happened and how to prevent recurrence, not to assign blame. Google’s SRE culture explicitly mandates blameless postmortems: Everyone involved had good intentions, and systemic improvement comes from analyzing processes, not people.

A good postmortem answers: What happened? What was the user impact? What was the timeline? What worked well in the response? What should change? Each postmortem should produce at least one action item with an owner and deadline. Without follow-through, postmortems become theater.

The Tooling Layer: What you Actually Need

On-call tooling has matured significantly. The core stack for an effective on-call program includes:
- Alert Routing and On-Call Scheduling: PagerDuty, OpsGenie or incident.io manage schedules, escalation policies and notification routing. These platforms let you define primary and secondary responders, configure time-based escalation and give engineers visibility into shift loads.
- Incident Management: Centralized platforms where incidents are declared, communicated and tracked through resolution. The best platforms integrate with Slack or Teams so incident management happens where engineers already work, reducing context switching during a live incident.
- Observability and Monitoring: You can’t manage what you can’t see. Your on-call program is only as good as the signals feeding it. Clean dashboards, meaningful SLIs and SLOs and well-tuned alerting thresholds are prerequisites for reducing noise.
- Runbook Automation: The ability to execute pre-defined remediation steps automatically or semi-automatically for known incident patterns. This is where the biggest gains in on-call efficiency come from, and where platforms such as StackGen make a measurable difference.
Reducing On-Call Burden Through Automation

The most durable strategy for on-call burnout prevention is reducing the number of incidents that require human response in the first place.

StackGen applies AI-powered automation to the most repetitive and predictable on-call scenarios — the kinds of incidents that wake engineers up at 3 a.m. for something that could have been handled automatically. By correlating signals across your observability stack, Aiden identifies known incident patterns and executes pre-approved remediation actions without requiring an engineer to acknowledge and investigate.

The result is a reduction in actionable pager load, which directly addresses the root cause of on-call burnout. Engineers are still in the loop for novel incidents that require judgment. But the routine restarts, rollbacks and scaling events that constitute the bulk of alert volume? Those resolve automatically, with full audit trails for postmortem review.

For platform teams managing dozens of services across complex infrastructure, this shift from reactive human response to proactive automated remediation is the difference between an on-call program that’s sustainable and one that costs you engineers, every quarter.

Building a Sustainable On-Call Culture

Tooling and rotation models matter, but on-call culture is what holds everything together. A few practices that high-performing teams consistently apply:
- Treat On-Call Readiness as a Team Responsibility: If one engineer is carrying a disproportionate share of incidents — because they built the system, or they’re the most senior — that’s a knowledge distribution problem. Fix it with pairing, documentation and deliberate rotation design.
- Make Psychological Safety Explicit: Engineers should feel comfortable escalating incidents they can’t resolve without being judged. The cost of a slow escalation is always higher than the cost of asking for help.
- Review Your Rotation Design Quarterly: Teams and services change; what worked six months ago may no longer fit your current team composition, coverage needs or alert volume. Treat your on-call program as a living system that requires the same maintenance as the infrastructure it protects.
- Separate On-Call From Performance Evaluation: Engineers who are afraid that escalations or missed response times will affect their reviews will game the system and burn out trying. On-call incidents are system events, not performance events.
Key Takeaways

On-call best practices come down to a few core commitments: Fair rotation design, clean alerting, structured knowledge transfer and meaningful automation to reduce the load.

Organizations that get this right, don’t just have fewer burnout incidents — they have faster response times, better postmortem follow-through and on-call programs that engineers trust rather than dread.

If your team is ready to reduce the operational burden on your on-call rotation through intelligent automation, explore how StackGen handles the routine incidents so your engineers can focus on the ones that actually require them.
March 6, 2026
The Problem’s Not Your Monitoring Tools, It’s Your Workflow
The real cost of poor observability isn’t just downtime; it’s lost trust, wasted engineering hours, and the strain of constant firefighting. But most teams are still working across fragmented monitoring tools, juggling endless alerts, dashboards, and escalation systems that barely talk to one another, which acts like chaos disguised as control. The result is alert storms without context, slow incident response times, and engineers burned out from reacting instead of improving. As organizations scale across multi-cloud and microservices architectures, this fragmentation becomes unsustainable. But there is a solution.

Recent reports estimate the cost of poor observability data to be approximately $12.9 million per organization in annual losses, with some reports suggesting enterprises lose 20-30% of revenue due to data inefficiencies. Additionally, fixing data quality issues can be expensive; the 1x10x100 rule suggests that fixing a problem at the boardroom level can cost 100 times more than catching it at the ingestion point.

For companies to avoid these costs and be able to redirect engineering resources to innovation, a shift to a more unified approach to observability is required. However, unified observability is more than a technical upgrade; it’s an operational shift toward achieving visibility, precision, and speed. When monitoring, alerting, and response exist within the same ecosystem, teams can move beyond reactive firefighting to proactive reliability. A unified observability platform breaks down silos between data sources and responders, creating a continuous feedback loop in which every alert leads to faster insight, smarter automation, and ultimately, a more resilient digital organization.

The Evolution Toward Unified Observability

The gap between alerting and action can make or break uptime. Legacy monitoring tools and disconnected on-call systems often generate too many uncoordinated alerts, forcing teams to waste time gathering context instead of solving problems. As organizations scale and adopt microservices or hybrid cloud infrastructure, this fragmentation becomes unmanageable, leading to alert fatigue, misrouted incidents, and delayed recovery times.

A unified observability approach aligns all telemetry, including metrics, logs, traces, and user experience data, under one consistent source of truth. Visibility into every service and dependency enables teams to identify not only where issues occur, but also why they happen.

How to Establish a Unified Monitoring and Alerting Workflow

When it comes to unified visibility, Datadog looms the largest, bringing together infrastructure, application performance, logs, and security. The Datadog unified monitoring and alerting workflow automates responses to alerts and security signals, using the Workflow Automation feature to trigger a series of actions across integrated tools like Slack and Jira. Building a cohesive on-call and observability environment within Datadog starts with clarity, integration, and alignment of team processes. Here’s a practical approach used by modern DevOps and SRE teams:
1. Centralize your telemetry
  Begin by consolidating metrics, logs, and traces into Datadog’s unified dashboards. Tag services with key parameters such as env, service, and version to correlate data across applications and infrastructure.
2. Create intelligent alerts
  Use multi-alerts and tagging structures to route notifications by ownership or geography. Properly scoped alerts prevent noise by ensuring that only the right team is engaged when a service degrades or fails.
3. Integrate On-Call and Incident Response
  Datadog On-Call connects directly with alerting and telemetry streams, so responders can act immediately. Define escalation paths and schedules within the platform to notify backup roles when alerts go unacknowledged automatically.
4. Empower mobile and distributed teams
  Datadog’s mobile interface allows engineers to acknowledge, analyze, and even trigger incidents remotely. This ensures incident ownership regardless of location or time zone.
5. Analyze and continuously optimize
  Leverage Datadog’s analytics to measure on-call load, time-to-acknowledge, and alert frequency. These insights make it easier to balance workloads, eliminate redundant alerts, and improve response efficiency.
Faster, Smarter Incident Response

A unified observability architecture allows teams to identify root causes in real time during critical events, without switching context across tools. Datadog’s service maps and distributed tracing make it simple to follow the path of an issue across services, understanding dependencies and pinpointing failure points instantly.

At the same time, built-in AI features such as Watchdog and Bits AI help detect anomalies, summarize incidents, and recommend probable causes automatically. The result is an accelerated response cycle, from detection to resolution to learning, that continuously strengthens an organization’s resilience.

From Detection to Action

Unifying monitoring, alerting, and response within Datadog eliminates barriers between observability and operations. When teams share the same data, dashboards, and workflows, they spend less time coordinating and more time improving reliability. The outcome is measurable, with fewer missed alerts, faster time-to-resolution, and a healthier on-call culture built on collaboration instead of fatigue.
February 3, 2026
Why Privacy-Safe Logging Remains One of the Hardest Problems in DevOps

As cloud-native architectures scale and regulatory pressure intensifies, organizations are finally recognizing that their logging pipelines contain sensitive. Logs fuel observability, debugging, compliance investigations, and incident response, yet they also remain one of the least governed data streams in the enterprise.

Despite years of progress in DevSecOps, true privacy-safe logging, logs that remain operationally useful and compliant with modern privacy requirements, is still rare. The barriers are not only technical but also organizational, cultural, and architectural.

1. Culture Clash: Observability vs. Compliance

Logging sits at the intersection of multiple teams whose priorities often conflict. Developers want rich logs to troubleshoot quickly. SREs need consistent observability across distributed systems. Security demands strict redaction and restricted access. Compliance requires provable guarantees around PII handling.

These priorities rarely align, and breakdowns frequently occur.

2. Tooling Fragmentation Across Cloud-Native Pipelines

Modern logging stacks include collectors, agents, ingestion layers, message buses, storage layers, and SIEM systems. Each stage introduces new formats and places where PII can slip through.

Frequent breakdowns include inconsistent redaction across layers, unstructured logs from older systems, differing policies across teams, and the absence of a unified control plane for redaction.

3. The Ship-First Secure-Later Mentality

Logging is often treated as a secondary concern. Developers add logs freely, teams prioritize features, and redaction rules are added only after incidents.

Consequences include delayed discovery of PII leaks, costly investigations, loss of trust in engineering, and broken traces caused by over-redaction.

4. Skill Gaps in Privacy Engineering

Building a compliant logging platform requires expertise in distributed systems, privacy engineering, detection algorithms, schema governance, and retention models. Few organizations possess this breadth.

Gaps include developers lacking secure logging training, SREs lacking classification skills, compliance misunderstanding operational realities, and tooling teams lacking automation expertise.

5. Alert Fatigue and False Positives

Pattern-based PII detectors generate many false positives, while ML-based detectors require tuning. High alert volume leads to reduced trust and bypassed systems.

6. Lack of Industry Standardization

Logging lacks universal definitions, standard policy languages, consistent compliance benchmarks, and cross-platform schema governance. Most companies must invent their own approach.

Making Privacy-Safe Logging a Reality

Successful organizations follow several principles. They design with privacy in mind, centralize enforcement, automate redaction, provide developer-friendly tools, create shared responsibility models, and continuously validate logging health through audits and scanning.

Conclusion

Privacy-safe logging remains one of the industry’s most persistent challenges. The complexities of modern architectures, evolving regulations, fragmented toolchains, and cultural misalignment have made compliant logging a difficult target. Treating logging as a governed, scalable, privacy-critical data product allows teams to preserve both user trust and operational excellence.

December 10, 2025

AIOps for SRE — Using AI to Reduce On-Call Fatigue and Improve Reliability

Site reliability engineering (SRE) has become an emergent niche practice invented at Google to become a foundation of contemporary enterprise performance worldwide. With the continued growth of microservices, a multi-cloud infrastructure and continuous deployment pipelines adopted by organizations, the operational surface area has increased to the extent that human personnel cannot monitor and manage it in real time. The effectiveness of distributed systems has compounded their complexity, such that traditional monitoring, alerting and incident response are no longer effective.

This complexity brings about a side effect, which is alert fatigue. There is an enormous torrent of sporadic, redundant, intractable and downright fake notifications of on-call engineers. Rotations 24/7, with escalation links excessively burdened, have already overstretched teams and demoralized the staff, promoting burnout. Under this scenario, human judgment has become a bottleneck instead of a defense mechanism since cognitive overload slows the decision-making process and compromises incident response. The practices of SRE, such as maintaining a high rate of innovation and stability simultaneously, are more challenging to maintain as most of the staff is occupied with operational toil.

Here is where AIOps (artificial intelligence for IT operations) comes into the picture. As Gartner defines AIOps as the implementation of artificial intelligence (AI) and machine learning (ML) to optimize the work of IT, it introduces such advanced features as anomaly detection, event correlation, predictive insights and automatic remediation. This is not a substitute for human expertise but an additional feature to the SRE teams: Reducing noise, bringing out actionable indicators and refocusing attention on high-value engineering work.

This article will discuss how AIOps can be used by SREs to alleviate on-call exhaustion and enhance system reliability. It will analyze the problem area, unpack the pragmatic application of AI/ML to the SRE process, emphasize the practical examples and comment on the challenges and prospects.

The Problem Space: On-Call Fatigue in SRE

The core of SRE is the on-call model — a rotating approach in which engineers must respond 24/7 to any incident. The mechanics are simple: Incidents are identified by monitoring systems, escalated by a paging or chat-based alert and resolved based on a documented runbook or a make-it-up investigation. However, this is not a simple step. Escalation trees imply that one unresolved alert can trickle down through several levels of engineers, all of whom were taken out of intensive work or sleep. Eventually, this pattern of upheaval is disastrous for productivity and health.

The high number of false positives and low-value alerts is one of the most urgent issues. Modern observabilities create massive telemetry; without enlightened filtering, providers get inundated with thousands of reports daily. According to a survey by PagerDuty (2021), most incident responders have over 10 alerts per shift, and most said that they could not be acted on. The outcome is alert fatigue, wherein engineers grow used to the notifications, leading to the danger of missing the most critical signals when they are the most needed.

Repeated laboring is also troublesome, with incidents that have no final system solution. Google’s SRE book describes toil as manual, repetitive, automatable labor that increases linearly with the growth of services. The same problem that on-call engineers may experience night after night could be a repetitive memory leak, a one-time API crash, a cranky monitoring threshold or similar issues. Every repetition takes away attention and makes the feeling of futility even stronger.

These situations cause human bottlenecks in peak outages. Manual triage and system correlation overload the responders when several systems fail simultaneously. Critical minutes are wasted, and the question is how to decide whether it was 50 alerts of one root cause or 50 discrete issues.

It is significant in terms of human cost. A 2025 report by Catchpoint lists almost 70% of SREs, indicating that on-call stress impacted burnout and attrition. Chronic sleep disturbance, constantly switching between contexts and psychological stress, damages not only the well-being of an individual but also the work of the team and the reliability on a large scale.

What is AIOps and Why Now?

AIOps was a term Gartner initially popularized. It refers to using ML and sophisticated analytics on the data of IT operations to enhance event correlation, anomaly identification and root cause analysis. As a practitioner, though, AIOps will be less about the use of marketing terms and more about addressing a particular operational truth: Humans are simply incapable of operating on the size and speed of the systems of today. AIOps to SREs constitutes a set of AI- and ML-based methods that achieve impedance matching, indicate patterns that cannot be seen by human eyes and allow proactive operations instead of reactive ones.

AIOps has four main capabilities. Anomaly detection detects log, metric or trace abnormalities, which can be used to indicate incidents. Event correlation broke the events into single incidents to create fewer pages that do not inundate on-call engineers. Predictive analysis uses past data to predict possible outages or performance declines to alert customers before they are affected. Lastly, auto-remediation will undertake available runbooks or coordinate corrective measures without manual intervention and maintain the stability of services as SREs concentrate on high-order engineering.

Figure 1. AIOps main capabilities

There is no coincidence in the timing of the adoption of AIOps. Architectures based on the cloud produce unprecedented amounts of telemetry, whether short-lived containers or microservice connections. With the current ML developments and real-time data processing, it is now possible to analyze these streams in real time, and insights can be revealed within seconds instead of hours. In the meantime, companies feel the ever-increasing pressure of providing greater availability and reliability with no corresponding increase in the number of SREs. This compounding factor continues to drive automation as an essential but not extraneous feature.

Although traditional monitoring is stationary-based with high static thresholds and manual triage requirements, AIOps systems are learned and adaptable. They do not signal at each CPU spike, rather than just the exception, they put them into perspective, coordinate and expand the exception across layers and suggest or take action. In the case of SREs, this shift is a transition to more strategic (mission-based) and intelligence-augmented (to a greater degree) reliability engineering practice instead of reactive firefighting.

Applying AIOps to SRE Practices

The great potential of AIOps to SRE does not lie in its abstract definitions but in its direct assurance of practices in everyday operations. AIOps transform the incident lifecycle by tackling historic causes of toil, including noise, detection delays, slow diagnosis and manual repairs. The five tangible areas where it provides quantifiable value are presented below.

1. Noise Reduction & Event Correlation

The most apparent irritant of SREs is the deluge of alerts provided by monitoring systems. The CPU spike in one microservice will cascade into the downstream latency warning, database connection error and end-user timeouts, generating dozens or even hundreds of unnecessary alerts. Without AIOps, an engineer must manually correlate relationships across pages, meaning no system is designed to handle high stress.

AIOps platforms use a method of clustering and deduplication to reduce these to a single coherent incident. AI can automatically detect links between related events by analyzing the metadata containing the time, topological dependencies and historical co-occurrence. Consequently, the number of alerts is reduced, and they have more context. As a real number, it may imply transforming 1,000 raw events into one actionable event with causal chains attached. In the case of on-call engineers, this directly converts into reducing pages, less exhaustion and less time to respond.

2. Anomaly Detection & Early Warning

The old methods of monitoring use hard-coded thresholds: CPU exceeding 80%? Page the team. However, distributed systems hardly ever act in linear, foreseeable fashions. Technically correct functional alarms may occur in seasonal traffic bursts, short-lived load tests or warming up a cache. AIOps proposes using statistical and ML-based anomaly detection, in which normal behavior is dynamically trained on logs, metrics and traces. As opposed to set thresholds, models identify the delicate differences between what is expected to be the case. This allows it to give early alerts (before service-level objectives (SLOs) are violated).

An example of this is an incremental change in 99th percentile latency that could be undetected by conventional systems until it is enacted by user degradation. Trends can be detected at an earlier hour, and anomaly detection can alert the teams so that they can take the initiative. The alteration of reactive firefighting to preventative action is radical. Anomalies can be surfaced during working hours when they are less disruptive, instigating engineers to wake up at 2 a.m. because a customer-impacting incident transpired.

3. Root Cause Analysis Acceleration

Identifying the actual root cause can be the most time-consuming procedure when an incident has occurred. Dependency tracing through manual methods is a titanic undertaking in microservices, where one user request could pass through dozens of services. It is common to waste minutes or hours of critical time perusing dashboards, cross-checking logs and speculating on possible hypotheses by the engineer.

AIOps systems have the potential to speed up a root-cause analysis with the use of graph-based algorithms and ML models on service associations. Analyzing the data on historical incidents and the current telemetry can allow you to propose the root cause with confidence scores. An example is where the latency warning of several services keeps being correlated with the memory pressure in a particular cluster of the cache. The AI can instantly identify the probable location as the cluster.

This does not eliminate human validation but reduces the time to insight. Engineers do not have to work with a blank slate because a set of hypotheses supported by evidence is provided as a starting point, and the mean time to resolve (MTTR) is significantly minimized.

4. Predictive Incident Management

AIOps has the strongest ability to look ahead and is the most compelling. It can predict system degradation before it occurs by training predictive models using past performance, seasonal usage and infrastructure measurements.

Take the case of an e-commerce platform going into the holiday shopping season. Under modern traffic patterns and resource consumption patterns, AIOps may indicate that a database cluster will become congested in the next two hours. Other scaling actions can be proactively taken, or SREs can be notified to take preventive measures by the system rather than waiting till the inevitable event of an outage occurs. This foresight lens transforms the operations from reactive due to the need to fight fires all the time and the reliability-first strategy. It concerns minimizing downtime and generating trust that should systems be subjected to the rarely observed demand spikes they would not be catastrophic.

5. Automated Runbooks & Self-Healing

Automation is the final promise of AIOps. There are many known solutions to incidents: Restart a service, delete a cache or turn around a certificate. Conventionally, these are recorded on runbooks that the engineers run manually.

AIOps can execute runbooks automatically so that some types of incidents invoke a predefined workflow or script. For instance, when a service starts failing consistently because it has been discovered there is a memory leak, the system can just do a safe restart without alarming an engineer. More sophisticated applications proceed to self-healing systems, where repairing decisions are made dynamically (dependent on incident context). The key here is balance. Complete automation is potentially dangerous, particularly in novel or high-stakes incidents. Most mature organizations use a human-in-the-loop approach, whereby automation is used to deal with routine, complex or uncertain situations under the supervision of engineers. The result is a decrease in unnecessary wakeups and more significant engineering work.

Figure 2. AIOps maturity spectrum from reactive to proactive incident management

Use Cases

1. Walmart’s AIDR — Anomaly Detection for Incident Response at Scale

Walmart’s AI Detect and Respond (AIDR) project offers a remarkable viewpoint on scaling AIOps in a large and heterogeneous operational environment to decrease the on-call load and significantly enhance reliability. The proposed AIDR is an AI-driven anomaly detection platform aimed at maintaining the health of the business and the system in real time across Walmart’s applications, platforms and teams.

During three months of validation, AIDR implemented over 3,000 models supporting more than 25 business, platform and ops teams. In the same period, it reported approximately 63% of the major incidents and an average decrease of more than seven minutes in mean time to detect (MTTD).

Table 1. Results and metrics

Metric	Improvement
Mean time to detect (MTTD)	Reduced by more than seven minutes on various major incidents
Coverage of major incidents	~63% of major incidents covered by AIDR in the test/validation period
False positives and noise	Lowered compared to prior methods (including static thresholds) due to the models + rules hybrid approach

Architecturally, AIDR relies on univariate and multivariate ML models, statistical models and rule-based fixed thresholds. This hybrid technique enables the maintenance of domain-specific constraints (through rules) and uses adaptive models that acquire baseline models and automatically identify variations.

To maintain long-term performance, AIDR incorporates a feedback loop: Feedback on the quality of the alerts (true vs. false positives) is provided by the teams, and drift detection is included in the system to detect instances of poor performance and cases of change in input distribution. The self-onboarding is also present: Different teams possessing different degrees of ML knowledge can input their metrics/signals into AIDR, model them and be deployed.

In terms of output other than raw MTTD, AIDR’s impact was also significant in curtailing noise (many alerts are removed or muted vs. legacy rule-based or hard threshold-based alerts) and amplifying the signal-to-noise ratio of the signal of on-call engineers. It has generated previous identification of occurrences that would otherwise be realized only after customer exposure.

2. Cambia Health Solutions — Alert Correlation & Noise Reduction with BigPanda

Cambia Health Solutions is a nonprofit health services organization with a severe problem of alert overload and manual triage of incidents. Their legacy monitoring systems emitted various alerts, and many emerged as redundant or were not contextualized, causing delayed responses and high operational expenses. To eliminate such problems, Cambia deployed the AIOps platform developed by BigPanda that employs ML to correlate and enrich alerts, converting them into actionable incidents.

Cambia significantly enhanced their network operations centre (NOC) operations by integrating BigPanda:

Less Alert Noise: The high level of correlation generated by BigPanda’s advanced correlation engine eliminated irrelevant alerts, and the noise level was also significantly reduced.
Enhanced Alert Enrichment: Contextual data, including host names and CI/CD pipeline data, was added to alerts, offering insight to the NOC teams.
Automated Incident Management: Close to 83% of the alerts were automatically handled and did not require manual intervention.
Better SLA Compliance: The NOC reached its service level agreements (SLAs) compliance 95% of the time, and critical alerts were recognized within 30 seconds of occurrence.

Such improvements enabled Cambia’s NOC team to prioritize quickly resolving real incidents, increasing system reliability and customer satisfaction.

The table presented below lists the most critical performance indicators of Cambia before the adoption of BigPanda and following it:

Table 2. Metrics and BigPanda

Metric	Before BigPanda	After BigPanda
Alert Noise Reduction	High	Low
Percentage of Automated Alerts	0%	83%
SLA Compliance	70%	95%
Time to Identify Critical Alerts	30 minutes	30 seconds

Challenges and Limitations of AIOps in SRE

Although AIOps has transformative potential in the SRE teams, its adoption is challenged. Data quality has become the inherent bottleneck: ML models and anomaly detection algorithms can only be as good as the data being fed. False positives, missed events or a direct inflection of automated behavior may occur due to inconsistent, incomplete or noisy data. This is why the cliche of garbage in, garbage out applies.

It is also essential to trust and explain. Most SREs are wary of using AI insights, especially when models propose remedial procedures without explicit reasoning. The hidden nature of black-box predictions may destroy trust, and teams may disregard this information or confirm every suggestion by hand, thereby reducing the productivity benefits.

The other practical limitation is integration complexity. Current observability stacks are diverse combinations of monitoring solutions, logging systems and CI/CD pipelines. To ensure data integration, exact similarity in event schemes and end-to-end incident context, retrofitting AIOps must be carefully planned.

There is also organizational and cultural resistance. The teams might feel that AI will diminish human judgment, impacting employment by reducing the number of people employed, slowing the adoption process and creating a clash between engineers and management. Moreover, overheads such as costs and resources to manage pipelines in ML (such as compute, storage and staff) can be substantial, particularly in smaller companies.

The key principles of overcoming these challenges have to do with incremental adoption: Begin with low-risk automation (e.g., alert correlation), keep human-in-the-loop validation, ensure that data are strongly governed and ensure the message about AI is that the latter is an augmentation and not a replacement. A gradual trust-establishment process will make SRE teams understand that productivity and reliability benefits are achieved without losing confidence in operational stability.

Future Directions and Emerging Trends

The AIOps in SRE on the next frontier assumes smart augmentation instead of automation. Among the recent trends is the emergence of SRE AI copilots, which have been developed and implemented within ChatOps. Engineers may use such copilots during a live incident to summarize alerts, prescribe remediation actions or point out impacted services. It is a 24/7 operational partner that can help lessen the cognitive load on engineers.

Generative AI (Gen AI) is also starting to enter incident postmortem, where a summary, timeline and root cause hypothesis are constructed out of telemetry and logs without human intervention. These systems make incident analysis faster, more efficient and error-free since the documentation is at least consistent and fast due to codifying knowledge based on previous incidents.

Continuous learning systems are also becoming popular. These systems are dynamically configured to respond to changing system behavior, continuously learning normal behavior and anomalies to improve their thresholds, correlations and predictive models. This adaptable model will ensure that AIOps will be helpful as systems become more intricate, with model drift minimized and actionable information preserved.

Lastly, AI is also being used in chaos engineering and resilience testing. AI can be used to make SREs proactively monitor services that anticipate the systems to respond to failures and highlight latent vulnerabilities before any actual incident happens. All these trends combined indicate a future in which AI can alleviate the toil and suggest SREs into more resilient self-optimizing systems.

Conclusion

AIOps is one of the paradigm shifts in SRE, which changes how teams handle complex distributed systems. Using AI and ML, SREs can minimize alert noise, speed up anomaly creation and root cause analysis and automate repetitive remediation efforts. These capabilities directly respond to the endemic issues of on-call fatigue, burnout and operations bottlenecks, enabling engineers to devote their time to high-value problem-solving activities rather than solving problems by hand.

The practical applications, such as the alert correlation improvements in Cambia Health Solutions and the AIDR predictive monitoring in Walmart, prove the improvements in reliability, efficiency and well-being among the teams. Nonetheless, some challenges such as data quality, trust, integration complexity and cultural resistance remain. These risks can be avoided through careful and gradual implementation, along with human-in-the-loop validation to achieve the potential benefits of AIOps.

Ultimately, AI augments the future of SRE: Systems that learn, adapt and proactively ensure reliability would allow engineers to be more strategic and tactical in creating a resilient, self-healing infrastructure than they think they can be now, and act as responders instead of architects.

November 7, 2025

When Metrics Overwhelm: How SREs Help Engineers Reclaim Focus
If engineers were polled on their feelings toward observability, many would express skepticism if not outright frustration. While observability promises transparency and faster root cause analysis, its rise has reshaped developer work in ways that few anticipated and not always for the better. What was once DevOps turf now encroaches on core engineering, dragging developers into data overload, endless dashboard hopping, and “war rooms” at 3 a.m..

Modern systems generate vast amounts of logs, traces and metrics, but most observability tooling was designed with operators in mind, rather than those actually writing the code. Platforms often deliver “visibility” at the expense of clarity, requiring developers to sift through irrelevant signals and noisy alerts just to find actionable insight. As Stephen Elliot, IDC analyst, notes, “Developers waste valuable time sifting through excessive logs and metrics to pinpoint the root cause of issues.”

The Contradiction: Reactive Alert Fatigue vs. Real Dev Value

Engineers lament that observability, billed as a paradigm shift, frequently devolves into reactive firefighting. When every service emits hundreds of metrics and “critical” alerts, distinguishing noise from real problems becomes a full-time job and ironically, can obscure the root cause engineers are paid to fix. Developers complain of “alert fatigue” and claim that they’re pulled away from building features to become unintentional support staff.

Case in point: One financial sector developer recalls, “We’d spend more time identifying who needed to be involved than resolving the issue. Logs were scattered across different systems, requiring switching between multiple dashboards. This inefficiency meant prolonged downtime and frustrated customers.”

SREs to the Rescue: Building the Bridge, Not the Barrier

Yet amidst developer complaints, Site Reliability Engineers (SREs) are fighting back and winning. SRE teams are reframing observability as a means for empowerment, not just surveillance. By championing smarter, more integrated observability stacks, SREs reduce the chaos and restore developer focus.

How SREs turn the tide:
- Contextual logging: Instead of generic logs and traces, SREs work with developers to embed meaningful context correlating frontend symptoms with backend causes, dramatically reducing time-to-resolution.
- Refined metrics: SREs audit what’s actually measured, pruning useless alerts and aligning metrics with real business outcomes. One success story involved cutting unneeded error logs by 70%, boosting feature release velocity.
- Distributed tracing: Tracing frameworks like Jaeger and OpenTelemetry help SREs “follow the data,” mapping service-to-service dependencies. Engineers get end-to-end visibility without drowning in dashboards.
Real Use Cases: From War Rooms to Win Rooms

A high-profile e-commerce incident shows SRE impact: During a critical outage, SREs led the charge by leveraging distributed tracing, narrowing the issue to a slow authentication service amid thousands of requests. Developers, once skeptical, credited observability tooling paired with seasoned SRE navigators for cutting downtime from hours to minutes.

At a global media company, SRE-driven observability transformed post-mortem culture. Instead of finger-pointing, collaborative incident reviews highlighted engineering wins — such as rapid root cause identification and improved system resilience—ultimately boosting morale and cross-team trust.

Engineers: From Reluctant Observers to Advocates?

While resistance lingers, smart observability (championed by SREs) is beginning to convert skeptics. Tools like middleware, Grafana, etc., are tailored for real workflow, not just metrics, and are bringing developers closer to high-value debugging and proactive system improvement. Engineer buy-in increases when:
- They see immediate feedback on code changes.
- Alerts are relevant, actionable, and tied to the service they support.
- They collaborate with SREs, gaining context rather than just data dumps.
Key Takeaways
- Observability demands developer involvement, but poor tooling breeds resentment and alert fatigue.
- SRE teams succeed by curating signal over noise, contextualizing data and aligning metrics with business goals.
- Collaborative post-mortems and contextual tracing turn observability from a pain point into a productivity win.
- Engineer advocacy grows when observability improves their workflow, not just system surveillance.
Ready for the next round of observability? Let those building the systems help design how they’re seen.
October 13, 2025
Filter the Firehose

We are tired. Information overload is a problem in the modern world. We hear instantly about events we never would have known about otherwise, or that we would have learned about months after the fact. Today, moments after an event, we have thousands of “professionals” analyzing it for us, a millions-strong army of amateurs telling us they know everything about it and a legion of bots telling us what to think about the event. No matter what the event is. Often “the event” isn’t even newsworthy, and yet the process feeds on itself.

We in IT have it worse. We have a flood of security and application events constantly flooding in and so much data that we call it a ‘data lake.’ More like a data ocean at some organizations. We’re tired. They call it by various names—alert fatigue, data overload, event flood, “the firehose.”

No matter what it’s called, it’s a big problem. And if it is not effectively managed, we will make mistakes. Particularly in security—but also across IT—we cannot afford to allow the flood to distract us or burn us out.

Look into AI. The amount of data, alerts, information, metrics, logs, traces, etc. that we are accumulating—and the rate at which we are doing so—is growing faster than humans can reasonably keep up with—even if there were no staffing constraints. Don’t hesitate any longer; don’t assume you or a vendor NOC are managing this well enough with only eyeballs. AI solutions have matured enough to reliably sift through at least the top layer(s) of noise, reducing the burden without increasing risk. In fact, given the increasing possibility of human error as volumes increase, probably with less risk.

Ask vendors about it; talk to your security vendors about their offerings. A huge number of security vendors have NOCs now that they would love to have you subscribe to. Many of them are highly automated and use AI. Data vendors are behind a bit in this regard (for a lot of reasons) but they are catching up. Not too long ago, the vast majority of work done to normalize disparate datasets was done via trial and error. AI and scripts written from massive experience are lightening that load, at least.

So check it out. Find ways to make things manageable again. We all know things are slipping through the cracks as the volume of items reported and the number of locations they are reported from increases. Don’t wait until you have another emergency; look into the options and get something running.

For those of you already using AI/ML, unless you implemented it very recently, you’ll probably want to validate what you’re using, at what level and how much the AI/ML solution is helping, and tweak or replace. This space is moving too fast for “set it and forget it” today. Maybe in a few years, we can.

With the rare exception of something I use every day and think you might benefit from, I adhere to a policy of not recommending products here, so I can’t point you in the right direction other than to say that your vendors (data, SIEM and security) can offer solutions and/or suggestions. That, and “You shouldn’t wait” are my suggestions.

And keep rocking it. Part of that data/log/alerting growth is all the apps and infrastructure you have helped put in and that are critical to the organization. So protect it with every tool available—and use that free time to work more miracles.

July 20, 2022
SRE’s Guide to Pragmatic Incident Response
In my past experience as an SRE, I learned some valuable lessons about how to respond to and learn from incidents. If you want the TL;DR, I’ll summarize them here:
- Declare and run retros for the small incidents. It’s less stressful, and action items become much more actionable.
- Decrease the time it takes to analyze an incident. You’ll remember more, and will learn more from the incident.
- Alert on pain felt by people – not machines. The only reason we declare incidents at all is because of the people on the other side of those machines.
Now, let’s dive into each of these lessons a little deeper, and explore how they can help you build a better system for pragmatic incident response.

1. Focus on the Small Incidents First

The bad habit of ignoring small issues often leads to bigger issues. You should run retrospectives for small incidents (slowdowns, minor bugs, etc.) because these often have the most actionable takeaways—instead of shooting the moon and creating “rearchitect async pipeline” as a Jira ticket that never happens. Focusing on low-stakes incidents and retrospectives are a great introduction to behavior change across your organization.

Let’s look at an obvious example of how a focus on small incidents can have an outsized impact. Case in point—my apartment. I live in an old candy factory retrofitted into apartment units. We have an elevator (thank God), but some of the buttons don’t light up when you press them. The LED display doesn’t match up with the numbers on the floor buttons. Yes, the elevator goes up and down, but overall, you can tell that things are wrong with it.

One day, I came home and noticed an “Out of Order” notice on the elevator door. Was I surprised? Not at all. Of course an elevator with mislabeled buttons and broken LEDs would stop working!

This isn’t dissimilar from an important software lesson—ignoring small issues often leads to much bigger issues. What starts as an inconvenience can lead to a complete breakdown. This is why you should focus on the small incidents first. I hear a lot of companies say, “We need to fix incident response! Every time we have an incident, it’s just chaos!”

They are talking about unexpected downtime during a high severity level incident and they want to improve it.

I say that’s the wrong type of incident to use to fix your incident response system and process. Focus on the small incidents first. Running retros for small incidents can help you build strong incident response models, because they have the most actionable takeaways and they are the best way to change behavior through repetition and practice.

If you have a high-stakes, 12-hour incident and you run a one-hour retrospective you’re not going to get the results you want. You need to start small. Run retros for bugs that were introduced; maybe a bad data migration that didn’t really impact anything but took up a couple of hours of your day.

Heidi Waterhouse captured this idea really well in this piece on reliability. Every airplane you’ve ever flown on has many tiny problems, Waterhouse said, “… like a sticky luggage latch or a broken seat or a frayed seatbelt. None of these problems alone are cause to ground the plane. However, if enough small problems compound, the plane may no longer meet the requirements for passenger airworthiness and the airline will ground it. A plane with many malfunctioning call buttons may also be poorly maintained in other ways, like faulty checking for turbine blade microfractures or landing gear behavior.”

I couldn’t agree more. Extrapolating to software: The small things are typically indicators of bigger issues and could cause catastrophes down the road.

2. Track Mean Time to Retro (MTTR)

It’s important to think about what you measure in your organization. You should be measuring how you’re improving, and the most important metric here is mean time to retro (MTTR). Everyone should be tracking MTTR. It’s a great statistic to improve incident response in your team because it helps you understand the delay between incidents and retrospectives.

The easiest way to have a bad incident retro is to wait two weeks. It’s better to get into a room quickly and hash out what happened than wait a long time until you’ve got everything perfectly prepared.

Tracking MTTR can help you hold prompt and consistent retrospectives after incidents. Set a timer and make an SLO or SLA for yourself that says, “This is how long we take for retros.”

Retro time will vary depending on the severity of the incident itself. If it’s a SEV1, clear everyone’s schedules, because you need to have retro within 24 hours of the incident. For SEV3 incidents, you have much more leniency.

I also like tracking the ratio of retros to declared incidents. This is a metric that should go up; you should see your ratio of retrospectives to incidents increasing. You can break down that number by severity, as well. If you have a low retro ratio for a SEV1 versus a SEV3 incident, that might be okay at first (remember, start small) but you want those metrics to eventually become equal.

3. Alert on Degraded Experience with the Service, and Not Much Else

The severity of incidents is directly linked to customer pain. We would not declare SEV1s if there weren’t a lot of people feeling a lot of pain.

Alerting on computer vitals is an easy way to create alert fatigue and burnout. As your company starts to scale, you are going to use more CPU; you’re going to use more memory. Tying alerts to computer vitals is not a good sign.

If I run, my heart will beat faster—it’s just doing its job. Paging people at 2:00 a.m. because disc capacity is at 80% even though you won’t run out of space until next month is a good way to lose great teammates. I have worked with people who left their previous companies strictly because they got paged too many times for stuff that didn’t matter.

This is why you need to alert on a degraded experience with the service and not much else. A CPU burning hot at 90% is not necessarily a bad thing, but you need context to decide. Create SLOs that are tied to customer experience and alert on those. People experiencing problems with the service is the only thing you should alert on—for the most part.

One of the best ways I’ve seen to think about this came from Soundcloud developers, who explained that you should alert on symptoms, not causes. My fast heartbeat is not necessarily a problem. But if my elevated heart rate leads to lightheadedness and I fall—that’s a problem. So, I need to be able to alert on something like that. Paging people at 2:00 a.m. because disc capacity is at 80% and you won’t run out until next month is not good. But paging people because you know that disc capacity problems cascade into other, systemic problems is. You can apply the same thought to other potential causes of an outage. Paging alerts that wake you up in the night should only be based on symptoms.
September 7, 2021
How AIOps Makes DevOps Less Noisy
For DevOps engineers, “noise” is the enemy of productivity. In this context, the noise we’re talking about is unnecessary or low-priority alerts and notifications that distract engineers from identifying serious issues—and ultimately can cause alert fatigue syndrome, in which alerting systems are ignored altogether.

Without the application of a well-constructed noise reduction plan, alert noise will end up taking away from the focus of solving the actual problem that may be causing a more critical issue.

So, how can you fight alert noise? You might be tempted to assume that the answer lies in reducing the number of alerting tools you use, but that’s probably unrealistic. Teams today are dealing with complex application stacks and multiple tools and data sources, all of which can produce their own alerts for issues that may be interdependent or isolated. You simply can’t take tools out of your stack.

What you can do, however, is analyze and manage alerts more effectively. Dealing with alert noise ultimately boils down to developing an effective problem analysis strategy. Let’s explore how, with a specific focus on how AIOps factors into keeping alerts manageable.

Reduce the Time in Analyzing Alerts

While reducing noise is important, DevOps teams risk losing valuable time solving critical problems if they also have to create and/or add systems to properly analyze alerts and their sources and priority level. Maintaining this system also would be a source of unnecessary labor that may reduce the alert noise, but also reduces the amount of time that DevOps staff can work on the solution to the root cause— which brings up another consideration. When coming up with a plan, the amount of complexity involved in adding the monitoring tools necessary to cover an entire application and/or service can be equally laborious compared to not having a solution at all. At this point, we may be merely shifting the labor emphasis.

This is Where AIOps Comes In

Now, let us envision a way that a team can gain full insight into a root cause without exposure to alert noise. The team doesn’t need to analyze every single alert, which would include:
- Duplicate alerts.
- Alerts from multiple sources (tools, microservices).
- Informational alerts (warnings).
Instead, the team can let AIOps analyze and classify the stream of alerts and identify critical issues. DevOps no longer would need to spend valuable collaboration time analyzing every link in the toolchain. When a team has to analyze every tool, the team could produce false positives, redirect resources toward the wrong solution, spend too much time trying to identify the root cause and create excessive alert noise. Let us envision a way in which the team does not need to spend excessive time in analysis and can experience more effective collaboration in service of solving the actual problem.

AI and machine learning can relieve this manual process through AIOps. Automated collection of data across the application stack, alert correlation, identifying root cause and decision-making can be leveraged by a team to get to the problem immediately without all of the messy work of manual analysis, sorting and identifying root cause. This level of automation allows for needed human interaction through collaboration across teams, which can result in faster resolution, decreased downtime and effective communication of key events to company staff and end users.

Analyzing to Predict Future Problems

Another benefit of AIOps and the power of automated data collection, analysis, issue identification and decision-making is the ability to predict future events. A DevOps team can leverage these insights into effective preventative measures. The ability of AIOps to produce these insights will help a team find ways to automate and/or improve operations. Reducing alert noise no longer needs to be a goal because it is happening through a smooth interaction between the human team and the AI.

There is a growing need for DevOps teams to utilize the power of AIOps. While human teams are not replaceable, redundant and time-consuming tasks are. The ability of AIOps to effectively traverse a complex toolchain with isolated issues to analyze alert importance, correlate alerts, collect and analyze relevant data, identify root cause and produce holistic actionable insights eliminates inefficiencies in problem-solving and reduces alert noise.

Making DevOps less noisy is a goal with its roots in effective use of information. Alerts serve to inform, but thanks to the increasing complexity of applications, alerts become noise. Leveraging the power of AIOps serves to do a lot of the dirty work for a team so that noise can be reduced—but more importantly, it can make operations run smooth and keep customers happy.

— Phillip Lorenzo
January 14, 2019

Tag: alert fatigue

Why MTTR Breaks Down Without Correlation

Prerequisites

Step 1: Connect Your Observability Stack

Step 2: Configure AI-Driven Correlation Rules

Step 3: Test Against a Real Incident

Step 4: Monitor, Tune and Automate Remediation

Testing Your Setup

Real-World Results

Next Steps

The Evolution From Signal Noise to Decision Pressure

Observability Answers “What,” Not “What Next”

The Concealed Scaling Constraint

Introducing Decision Architecture

From Observability to Action Systems

Reducing Toil Requires Reducing Cognitive Friction

The Next Decade of Operational Maturity

What Makes On-Call Unsustainable

Choosing the Right On-Call Rotation Model

Seven On-Call Best Practices That Actually Work

The Tooling Layer: What you Actually Need

Building a Sustainable On-Call Culture

Key Takeaways

The Evolution Toward Unified Observability

How to Establish a Unified Monitoring and Alerting Workflow

Faster, Smarter Incident Response

From Detection to Action

Making Privacy-Safe Logging a Reality

Conclusion

The Problem Space: On-Call Fatigue in SRE

What is AIOps and Why Now?

Applying AIOps to SRE Practices

Use Cases

Challenges and Limitations of AIOps in SRE

Future Directions and Emerging Trends

Conclusion

The Contradiction: Reactive Alert Fatigue vs. Real Dev Value

SREs to the Rescue: Building the Bridge, Not the Barrier

Real Use Cases: From War Rooms to Win Rooms

Engineers: From Reluctant Observers to Advocates?

Key Takeaways

1. Focus on the Small Incidents First

2. Track Mean Time to Retro (MTTR)

3. Alert on Degraded Experience with the Service, and Not Much Else

Reduce the Time in Analyzing Alerts

This is Where AIOps Comes In

Analyzing to Predict Future Problems