devopsy.kinsta.cloud

Tag: telemetry correlation

Reducing MTTR: A Practical Guide to Correlating Incidents with AIOps
At 2 a.m., a payment service starts throwing errors. Within minutes, your observability stack fires off 40 alerts: Elevated latency on three services, a spike in 5xx errors, a memory warning on a downstream cache and a handful of dependency timeouts. Somewhere in that noise is the one signal that actually explains what broke. Finding it manually, alert by alert, is how a 5-minute fix turns into a 45-minute outage.

This is the core problem AIOps was built to solve — not more dashboards, not more alerts but correlation. By automatically grouping related signals across metrics, logs and traces, AI-driven correlation collapses an alert storm into a single, prioritized incident with a probable root cause attached.

In this guide, you’ll learn how incident correlation actually works, how to set it up on top of your existing observability stack and how to measure whether it’s actually reducing your mean time to resolution (MTTR). By the end, you’ll have a practical framework for moving your team from manual alert triage to AI-assisted incident response.

Why MTTR Breaks Down Without Correlation

MTTR is typically broken down into four phases: Detection, triage, diagnosis and remediation. Most teams invest heavily in detection — that’s what observability tooling is good at. The bottleneck almost always sits in triage and diagnosis, for a simple reason: One root cause typically produces many alerts, and someone has to manually figure out which alerts belong together before the diagnosis of the actual problem can even begin.

A few patterns show up repeatedly:

Alert storms mask root cause. A single upstream failure can cascade into dozens of downstream alerts, each looking like an independent incident.

Signals live in silos. Metrics, logs and traces often sit in different tools or dashboards, so the engineer on call has to manually stitch together a timeline.

Tribal knowledge bottlenecks diagnosis. The person who ‘just knows’ that a cache-eviction warning usually precedes a checkout failure isn’t always the person on call.

AI-driven correlation directly targets this middle stretch of the incident life cycle — the time between ‘something is wrong’ and ‘here’s what’s actually wrong’ — which is usually where the most time is lost.

Prerequisites

Before you begin, make sure you have:
1. An existing observability stack producing metrics, logs and traces (StackGen’s ObserveNow or a Prometheus/Grafana/Loki/Jaeger-style setup)
2. Alerting already configured on key services, even if it’s currently noisy
3. Admin access to configure integrations and correlation rules
4. At least one recent incident with alert history you can use to validate correlation output
5. Estimated time: 30–45 minutes for initial setup, plus 1–2 weeks of tuning based on real incidents.
Step 1: Connect Your Observability Stack

AI-driven correlation is only as good as the data it can see. If your metrics live in one tool, logs in another and traces in a third with no shared identifiers, correlation has nothing to work with.

Start by making sure your telemetry shares common context — service name, environment and ideally a trace ID or request ID that threads through logs and traces.

# Example: Shared labeling convention across metrics, logs and traces

service: checkout-api

environment: production

trace_id: “{{request.trace_id}}”

If you’re using ObserveNow, connecting metrics, logs and traces into a single pane is largely a matter of pointing your existing Prometheus, Loki and Jaeger (or equivalent) instances at it — StackGen integrates with these directly rather than requiring a rip-and-replace.

Expected Result: A unified view where a single service’s metrics, logs and traces are queryable together, ideally with a shared identifier connecting them.

Troubleshooting: If correlation later feels inaccurate, this is almost always the first place to look — inconsistent service-naming or missing trace propagation breaks the AI’s ability to link signals correctly.

Step 2: Configure AI-Driven Correlation Rules

With unified telemetry in place, the next step is telling the correlation engine what ‘related’ should mean for your environment. Most AIOps platforms, including Aiden AI Copilot, combine two approaches:

Topology-Aware Correlation: Using service dependency maps to group alerts from services that call each other

Pattern-Based Correlation: Using historical incident data to learn which alert combinations tend to co-occur

# Example: Correlation policy

correlation_policy:

time_window: 5m

group_by:

– service_dependency

– shared_trace_id

confidence_threshold: 0.75

The time_window matters more than it looks. Set it too short and you’ll miss slower-cascading failures; set it too long and unrelated incidents start getting merged. Five minutes is a reasonable starting point for most microservice architectures — narrow it down if your services are tightly coupled with fast failure propagation, widen it for slower, more decoupled systems.

Expected Result: Incoming alerts are grouped into incidents rather than arriving as isolated tickets.

Troubleshooting: If everything gets grouped into one mega-incident, either your confidence threshold is too low or your time window is too wide. Tighten both and re-test against a known incident.

Step 3: Test Against a Real Incident

Don’t wait for a live outage to find out if your correlation setup works. Pull the alert history from a recent real incident and replay it against your new configuration.

Ask three questions about the output:
1. Did the correlation engine group the alerts that actually belonged together?
2. Did it surface a root-cause candidate that matches what you now know actually happened?
3. Did it leave out anything that should have been included or wrongly include something unrelated?
This is also where Aiden/AIOps automated root cause analysis is worth validating directly. It should be able to point to the earliest anomaly in the correlated group and rank likely causes, not just group alerts together and leave the analysis to you.

Expected Result: The correlated incident matches (or comes close to) the retrospective understanding of the incident, with the root-cause candidate ranked at or near the top.

Step 4: Monitor, Tune and Automate Remediation

Correlation isn’t a one-time setup. It improves while identifying more of your real incident patterns. Over the first few weeks:
- Review every correlated incident and confirm or correct the grouping.
- Watch for recurring false groupings (two unrelated services that keep getting merged) and adjust topology mappings.
- Track false-negative correlations — alerts that should have been grouped but weren’t.
Once correlation accuracy is consistently high, you can layer-in automated remediation for well-understood incident patterns — for example, automatically restarting a service instance when a specific correlated pattern (memory leak signature + latency spike + no recent deploy) is detected with high confidence. This is where Aiden for SRE typically comes in: Using the correlated, root-caused incident as the trigger for a pre-approved remediation action, rather than waiting for a human to read the alert and act.

Expected Result: A shrinking gap between detection and remediation, with routine incident patterns increasingly self-resolving.

Testing Your Setup

Before trusting correlation in production, run it through a structured validation pass:
- Replay 3–5 historical incidents of varying severity and complexity.
- Compare AI-correlated groupings against what your team determined manually during the actual incident.
- Check root-cause ranking accuracy — was the true cause in the top 1–2 candidates?
- Confirm the time to first correlated alert is meaningfully faster than your team’s manual triage time.
If correlation accuracy is inconsistent across incident types, it’s usually a sign that your topology mapping is incomplete for certain services, which is why it is worth revisiting Step 1 before adding more correlation rules on top of gaps in the data.

Real-World Results

Teams that move from manual alert triage to AI-driven correlation typically see improvement concentrated in the triage and diagnosis phases specifically, since that’s the part of MTTR correlation is designed to compress:
- Alert Volume per Incident: Often reduced from dozens of individual alerts down to a single correlated incident
- Time to Root-Cause Identification: Meaningfully faster when root-cause ranking is accurate, since engineers stop manually cross-referencing dashboards
- On-Call Burden: Fewer distinct pages per incident, since related alerts are grouped rather than firing independently
Your specific numbers will depend heavily on how mature your topology mapping is and how much historical incident data the correlation engine has to learn from — treat the first few weeks as a tuning period, not the final benchmark.

Next Steps

Now that you have AI-driven correlation running, you can:
- Extend correlation rules to cover infrastructure-layer signals (node health, network) alongside application-layer telemetry
- Pilot automated remediation on your highest-confidence, lowest-risk incident patterns first
- Expand from Aiden for SRE into Aiden for DevOps if pipeline and deployment failures are a significant source of your incident volume
August 13, 2026
The End of Manual Triage
Manual triage is becoming the slowest and most expensive part of modern incident response. In distributed systems, the problem is no longer detecting that something is wrong; the real problem is turning noisy alerts, scattered telemetry and recent changes into a confident next step fast enough to protect reliability.

That is why the future of SRE is moving away from human-first investigation toward machine-assisted triage with human judgment on top. The teams that win will not be the ones with the most dashboards. They will be the ones that can move from signal to context to action with the least wasted motion.

Why Manual Triage is Breaking Down

Manual triage was tolerable when systems were smaller, dependencies were clearer and a single engineer could hold most of the architecture in their head. That model breaks down in cloud-native environments, where one alert may touch dozens of services, recent deploys, feature flags, CI/CD events and infrastructure state changes at once.

The cost is not just time. It is cognitive overload. On-call engineers often spend the first critical minutes of an incident doing detective work: Opening dashboards, comparing logs, checking traces, looking for recent commits and asking who owns what. By the time they form a usable hypothesis, the business impact may already be growing.

This is why ‘manual triage’ is no longer a badge of operational maturity. In many teams, it is a symptom of missing automation around the very part of incident response that should already be accelerated by telemetry, context and tooling.

What Replaces It

The replacement for manual triage is not blind automation. It is context-aware triage. That means systems that can ingest an alert, correlate logs, metrics, traces, deployment data and known incident history, then produce a structured explanation of what is likely happening and what should happen next.

This is where agentic SRE becomes useful. Instead of forcing engineers to begin every incident from a blank page, agentic workflows can pre-build the first layer of understanding. They can identify the affected service, highlight the most likely blast radius, pull relevant runbooks, compare the issue with recent changes and produce an initial incident summary in seconds.

That does not remove engineers from the loop. It removes the waste at the beginning of the loop. Human expertise still matters for judgment, business trade-offs and unusual failure modes. However, the low-value correlation work no longer has to begin from scratch every single time.

The Triage Gap

Every incident has a triage gap: The time between ‘an alert fired’ and ‘the team understands enough to act safely’. In many organizations, this is where the most preventable delay occurs. Detection may be fast, escalation may be automated, but understanding is still painfully manual.

Closing that gap requires three things:
1. Better telemetry correlation
2. Better operational context
3. Better automation around repetitive diagnosis
Without those, teams keep adding more alerts, more dashboards and more pressure to on-call rotations without solving the root issue. They are optimizing detection while neglecting interpretation.

What Good, Automated Triage Looks Like

A strong automated triage system does not just say that CPU is high or latency has increased. It connects the event to recent deploys, service dependencies, failing queries, noisy hosts, customer-facing symptoms, and prior similar incidents. It provides the responder with a prepared investigation rather than a raw alert stream.
- A good triage workflow usually looks like this:
- Alert arrives from an observability or incident platform.
- The system identifies the affected service and likely dependencies.
- It pulls correlated logs, traces, metrics and deployment history.
- It checks ownership, runbooks and known incident patterns.
- It produces a ranked set of likely causes and safe next actions.
That is a major shift. The engineer is no longer doing first-pass aggregation manually. The engineer is reviewing, validating and deciding based on pre-assembled context.

Why Observability Matters More Now

Automated triage only works if observability is strong enough to support it. If logs are incomplete, traces are missing, ownership is unclear or change history is fragmented, no amount of AI will make triage trustworthy. In fact, weak observability combined with aggressive automation can make incidents harder to understand, not easier.

This is why observability-first design matters. Metrics tell you something is wrong. Traces tell you where latency or failure is moving. Logs provide detail. Deploy data explains what changed. Incident history gives pattern memory. Automated triage becomes powerful only when these signals are stitched into one operational story.

The future of triage is therefore not ‘AI instead of observability’. It is ‘AI on top of observability’. Teams that already have a strong telemetry foundation are in the best position to reduce MTTR because they can automate understanding, not just detection.

Where Agentic Workflows Fit

Agentic workflows sit between raw observability and final remediation. They are the decision-support layer that interprets live signals, consults the right tools and recommends or triggers bounded actions. In triage, that often means pulling context from dashboards, CI/CD systems, ticketing platforms and runbooks, then building a structured response path.

This is why the current generation of AI SRE tools such as Aiden for SRE, Rootly, and Bits of DataDog is getting attention. They are not just adding chat to dashboards. They are helping teams automate the expensive thinking work around correlation, prioritization and first-response structure. That is where the operational leverage really is.

Still, the right model is progressive trust. Start with summarization, context gathering and correlation. Then allow guided recommendations. Then, for well-understood scenarios, let the system trigger low-risk actions inside strict guardrails. The end of manual triage does not mean the end of control.

A Practical Architecture

A modern automated triage architecture can be understood as five layers:
1. Signal Layer: Metrics, logs, traces, alerts and deployment events from tools such as Prometheus, OpenTelemetry, OpenObserve, Datadog, Grafana, Elastic or cloud-native monitoring stacks
2. Context Layer: Service ownership, dependency maps, incident history, runbooks and code or infrastructure changes
3. Reasoning Layer: An AI- or rules-based triage engine that ranks hypotheses and suggests next actions
4. Action Layer: Integrations with Slack, PagerDuty, Jira, GitHub, Kubernetes or internal tooling for guided response
5. Safety Layer: Role-based access, approval workflows, audit logs and action limits to prevent unsafe automation
When these layers work together, the incident channel becomes a decision environment rather than a panic room.

Example Logic Flow

Here is a simple example of what automated triage logic can look like in practice:

This logic matters because it not just mirrors how experienced responders already work but does it faster and more consistently. Instead of searching six systems manually, the triage engine assembles the first draft of reality before the human starts deciding.

What Teams Should Do Next

Most teams do not need to jump straight to autonomous remediation. They need to eliminate the waste in first-response workflows. A smart rollout looks like this:
- Start with alert enrichment and incident summaries.
- Correlate telemetry with recent changes and ownership data.
- Build triage assistants into Slack or incident tooling.
- Add approval-based automation for low-risk steps.
- Improve infrastructure consistency so triage has fewer unknowns.
This matters because the end of manual triage is not a single tool purchase. It is an operating model shift. You are moving from fragmented, person-dependent investigation toward structured, system-assisted reliability work.

Closing Thought

Manual triage is ending not because engineers are no longer needed, but because manual correlation is no longer an efficient use of engineering judgment. In modern systems, the fastest path to better reliability is not more dashboards and more heroics. It is better context assembly, better automation and better infrastructure discipline across the entire life cycle.

That is why the future belongs to teams that combine observability, agentic workflows and infrastructure automation into one coherent reliability model. When alert understanding, change context and infrastructure intent all become easier to inspect and automate, SREs can spend less time hunting and more time solving with tools such as Aiden for SRE. That is the real meaning of the end of manual triage.
July 24, 2026
Co-Developing an AI Native Observability Platform 

As AI capabilities continue to evolve, AI is becoming central to managing the growing complexity of distributed, hybrid enterprise environments, enabling more effective analysis, correlation, and automation across interconnected systems.

Traditional infrastructure and specifically network monitoring approaches, often built around siloed tools and static thresholds, struggle to keep pace with the scale, velocity, and interdependencies of modern systems. Further blurring the boundaries between network, application, and infrastructure domains makes it harder to isolate root causes and maintain operational resilience. In this context, AIOps platforms have emerged as one response to the growing need for integrated observability, automation, and data-driven decision-making.

At AI Field Day, Selector AI presented an AIOps platform, which can be considered a foundation for co-creating more adaptive and data-driven network operations. Rather than positioning it purely as a product choice, it embraces the SaaS approach, considering professional services as part of the offering, coupled with the product features and encouraging a co-development approach towards the platform instance for customers.

Demonstrations from the AI Field Day highlight the capability of full-stack observability with a data-centric approach, where data becomes the core part of the stack. Selector’s strength lies in its data-centric foundation, ingesting diverse, multi-domain sources metrics, logs, configs, alerts, and topology into a unified analytics layer. Unlike model-first tools, it prioritizes raw telemetry correlation via ML before layering AI, creating a “single source of truth” that slashes alert fatigue and supports hybrid/cloud environments without siloed dashboards. A unified data approach combined with the co-development of the specific platform instance, provides a more deterministic way of visualizing the problem, combined with causal analysis, which can help identify root causes more efficiently.

Another strategic feature embedded within the platform is the Network Language Model, built on a fine-tuned Model with vast networking telemetry data, which bridges natural language queries to complex ops tasks. It understands domain-specific terms (e.g., interface connectivity, routing paths) and powers Slack/Teams chats. This capability provides Selector AI advantage to advance beyond basic observability with AI agent workflows that enable autonomous, explainable network operations via its agent framework. These agents leverage Retrieval-Augmented Generation (RAG) with the Network Language Model to process unified telemetry, then trigger actions.

Striking comparison with building an in-house platform offers customization but typically requires significant investment often on the order of 18 to 24 months, multimillion-dollar budgets, and dedicated engineering teams. In addition, ongoing maintenance can increase as telemetry volumes grow and machine learning models require continuous tuning. Over time, internally developed systems may accumulate technical debt, particularly if they struggle to keep pace with evolving data and operational complexity. In contrast, purchasing a platform such as Selector, where organizations can engage in a co-development approach, may reduce initial development effort and accelerate deployment, with integrated capabilities like cross-domain correlation, incident summarization, and extensibility through a partner ecosystem. Another highlight is that the Selection instance is per customer, which means that there is no need for additional overheads.

The role of AI in operations is also evolving. Rather than optimizing capabilities like site reliability, the approach tends to shift the focus toward higher-level validation and decision-making. This creates a collaborative model where human expertise, operational data, and machine intelligence reinforce each other.

Adoption of such platforms often benefits from a phased approach. Initial efforts may focus on a limited proof of value, targeting a small number of critical services to measure improvements in alert reduction and incident response times. Subsequent phases can expand telemetry ingestion, introduce agentic workflows, and automate routine operational tasks, supported by cross-functional governance structures. Over time, organizations may extend capabilities toward predictive operations, capacity planning, and broader automation, while continuously evaluating outcomes against defined performance and cost metrics.

Lastly, co-creation allows for the customization of AI models and analytics to fit unique customer needs. This “mass customization” enables teams to create specific actionable insights rather than relying on generic, “one-size-fits-all” heuristics, according to Selector’s blog.  These elements combine for agentic, self-healing networks, aligning with AI Field Day 8 themes of production-scale inference and infrastructure evolution.

From a strategic perspective, platforms like Selector can be viewed less as a standalone product and more as enablers of operational evolution. The long-term value depends on how effectively organizations integrate them into their workflows, align them with business objectives, and build internal capabilities around them.

May 26, 2026
We Spent 15 Years Automating Infrastructure. Now We’re Automating Decisions

For most of the last 15 years, DevOps has been engaged in a massive automation project. First, it was server provisioning, then configuration management, then infrastructure as code. CI/CD pipelines followed, along with containers, Kubernetes, GitOps and eventually platform engineering. Each wave built on the previous one, steadily pushing infrastructure and operations further away from manual processes and deeper into programmable systems.

The industry became extraordinarily successful at it. Tasks that once required ticket queues, weekend maintenance windows and large operations teams became automated workflows that could execute repeatedly and reliably. Infrastructure stopped being something organizations manually assembled and increasingly became something they declared, versioned and continuously reconciled through software.

What is important, though, is that most of this automation was still fundamentally deterministic. Engineers wrote scripts. Teams defined workflows. Desired states were declared in code. Pipelines executed predefined sequences. Even Kubernetes, for all of its complexity, operates around deterministic reconciliation. The system continuously attempts to move reality toward a declared state.

The complexity of environments exploded, but the operational philosophy remained relatively stable. DevOps automation was largely about making known tasks repeatable, scalable and reliable.

AI changes that equation because it changes the target of automation itself.

For the first time, the industry is moving beyond automating infrastructure tasks and into automating pieces of operational judgment. That is not simply a more advanced form of automation. It is a different category altogether.

We are already seeing AI systems inserted into incident response workflows, observability platforms, vulnerability prioritization systems, remediation tooling and operational coordination layers. AI copilots recommend fixes, correlate telemetry, suggest rollback strategies and increasingly coordinate actions between systems. Vendors position these capabilities as the natural next step in DevOps evolution, and in some ways, they are right. Modern environments are too large, too dynamic and too interconnected for humans alone to manage efficiently at scale.

But there is a major difference between automating tasks and automating judgment.

A provisioning script that deploys the wrong infrastructure is a problem, but usually a bounded one. Engineers trace the workflow, identify the logic failure and correct it. The failure mode is procedural. The system executed the wrong instructions.

An AI system that incorrectly prioritizes a cascading outage, triggers the wrong remediation path or misinterprets operational telemetry introduces a very different kind of risk. The failure is no longer simply executional. It becomes interpretive. The system is not just doing the wrong thing. It is reasoning incorrectly about what the right thing is in the first place.

That distinction matters because the blast radius changes dramatically once automation moves into probabilistic decision-making.

DevOps spent years reducing operational unpredictability. Infrastructure as code reduced configuration drift. CI/CD pipelines reduced deployment inconsistency. GitOps improved reconciliation discipline. Platform engineering standardized developer workflows and operational controls. Much of modern cloud-native architecture is built around creating more predictable operational systems.

AI introduces probabilistic behavior into that environment.

Traditional automation generally behaves the same way under the same conditions unless inputs change. AI systems can produce different conclusions from similar operational scenarios because they rely on inference rather than strictly procedural logic. That does not make them bad or unusable, but it does fundamentally alter how organizations need to think about trust, governance and operational authority.

This is where the conversation around AI operations often becomes far too simplistic. The current industry narrative frequently frames AI as just another efficiency layer, another productivity accelerator similar to earlier automation waves. In reality, the transition underway is much more profound because organizations are beginning to delegate portions of operational reasoning itself to machines.

The implications reach far beyond tooling.

Consider observability. Traditional monitoring systems surfaced metrics and alerts for humans to interpret. Increasingly, AI-driven platforms attempt to identify root causes automatically, correlate dependencies and propose remediation paths. In security operations, AI systems prioritize vulnerabilities and determine which threats deserve immediate attention. In platform engineering, AI systems are starting to route workflows, coordinate actions between tools and make recommendations about deployment or rollback strategies.

At small scale, these capabilities feel helpful and manageable. At enterprise scale, however, organizations are effectively creating operational systems that participate directly in decision-making loops. Once that happens, governance becomes significantly more important than many organizations currently appreciate.

This is one reason platform engineering is evolving again. The first major wave of platform engineering focused primarily on abstraction and standardization. Internal developer platforms simplified Kubernetes complexity, improved developer experience and centralized operational controls. Platform teams became responsible for creating reliable paved roads through increasingly fragmented infrastructure environments.

Now the role is expanding beyond infrastructure abstraction.

Platform teams are increasingly becoming governance teams for autonomous operational systems. They are defining trust boundaries, approval chains, remediation authority and escalation policies for AI-assisted workflows. They are deciding which operational actions can happen autonomously, which require supervision and which should remain entirely human-driven.

In many ways, the platform layer is becoming an orchestration layer for operational judgment itself.

That shift also explains why open orchestration, interoperability and governance frameworks have become central themes across the broader open source and cloud-native ecosystem. Much of the discussion happening around agent interoperability, MCP frameworks and AI infrastructure at events like the Linux Foundation’s Open Source Summit is not really about hype. Underneath it is a growing recognition that operational authority is starting to migrate into AI-driven coordination systems.

Organizations are beginning to understand the risks of allowing critical operational reasoning to become trapped inside opaque proprietary systems. If AI-driven operational coordination becomes a black box, companies may eventually lose visibility into how key infrastructure, security and remediation decisions are actually being made.

That is not just a tooling concern. It becomes an operational sovereignty issue.

The irony here is that DevOps originally emerged in part as a reaction against opaque operational silos. Infrastructure as code and GitOps increased transparency because operational behavior became codified, observable and reviewable. AI systems risk reintroducing opacity into operations if organizations do not establish strong governance around explainability, oversight and accountability.

None of this means AI should be resisted. Modern operational environments almost certainly require some level of AI-assisted coordination going forward. The scale is already exceeding what humans alone can reasonably process in real time. AI will become necessary in many environments simply because telemetry volume, dependency complexity and operational velocity continue to increase faster than teams can manually absorb.

But the industry does need to stop pretending this is simply another incremental automation cycle.

Automating infrastructure changed how systems operated. Automating decisions changes how organizations distribute trust, authority and operational responsibility between humans and machines.

That is a much bigger transition than many people currently realize.

Shimmy’s Take

The DevOps movement spent the last decade and a half proving that infrastructure automation works. That success reshaped the entire technology industry and made modern cloud-native computing possible.

What comes next is fundamentally different.

We are no longer just teaching machines how to execute predefined tasks faster. We are beginning to allow machines to participate in operational reasoning itself. Once that happens, the conversation can no longer focus solely on efficiency and productivity.

It has to focus on governance, trust and control.

The future of DevOps is not about whether we automate more. It is about understanding which decisions can safely be automated, which require human supervision and which should never leave human hands at all.

May 20, 2026
From What Now to What’s Next: How AI Is Closing the Gap Between Detection and Resolution

Modern DevOps toolchains, often beyond full organizational control, automate and streamline the software development life cycle. They enhance collaboration, reduce context-switching and leverage observability to deliver better software faster. As applications now span complex, distributed environments — including microservices, multiple clouds, SaaS APIs and evolving AI agents — pinpointing the source of failures becomes increasingly challenging when disruptions occur.

The core challenge DevOps teams face today is no longer simply detecting whether something is broken; it is now about understanding why it broke and what to do next — and the quicker, the better. The industry shift now underway is from reactive troubleshooting to proactive, prescriptive resolution. Monitoring must evolve from signaling that a problem exists to guiding teams toward the fastest, most effective path to restore normal operations.

A New Kind of Outage

In the past, failures were often straightforward. A server crashed; a deployment introduced a faulty configuration or a database became overloaded. Engineers could trace the issue and apply a fix. However, the systems we operate today behave differently. Failures are increasingly distributed, triggered by services that live outside the organization’s infrastructure. They can be subtle, i.e., rooted not in total outages, but in small latencies that cascade under load — and as systems become more interdependent, identifying the precise source of an issue becomes far more difficult.

For example, in a workflow that depends on an AI-driven service, a request may travel through multiple remote endpoints before a response is returned. If even one of those dependencies slows down or returns unexpected results, the entire experience degrades. The user sees failure, but the root cause may be several layers removed. DevOps teams might detect an increase in error rates, latency, timeouts or customer support complaints, yet still struggle to determine where the problem lies — in their own code, their cloud provider, an external API or somewhere along the network path.

Where Traditional Monitoring Falls Short

Monitoring systems continue to play a critical role in alerting teams to performance changes. The problem is not alerting; it is interpretation. Traditional application performance monitoring (APM) excels at conveying what is happening, but not why. It can reveal that a page load slowed down, or that error rates spiked, or that customers are experiencing degraded service. What it cannot always do is indicate where the fault lies within a chain of dependencies that spans multiple services and continuously evolving systems and technologies.

The result is familiar to most DevOps teams: Lengthy triage sessions, extended war rooms and time-consuming manual investigations. Engineers comb through logs, traces, dashboards and reports, often working through elimination rather than direct diagnosis. Mean time to identify (MTTI) the issue increases, and mean time to resolve (MTTR) the issue stretches even longer. Customers feel the impact in real-time, even if the root cause lies far downstream.

AI Introduces Power (and Fragility)

The rise of AI and LLM-powered automation has introduced both breakthrough potential and new fragility. Traditional software is deterministic; it behaves as programmed. AI systems, in contrast, can change behavior as models are updated or as new data influences inference. These changes are often introduced silently by third-party providers, which means that application performance can shift without any direct change being made by the DevOps team.

This is especially evident in systems powered by agentic AI — autonomous agents that retrieve information, make decisions and interact with other systems. These agents often depend on a growing web of third-party services. When one service becomes slow or unavailable, the entire agent workflow can stall. Additionally, as AI systems often mask underlying complexity, diagnosing these failures requires visibility into dependencies that are not always obvious.

Moving From Reactive to Prescriptive Response

To meet the demands of this environment, DevOps teams need tools and practices that help them move faster from detection to resolution. This means improving context, not just alerts. It also means being able to trace issues across internal systems and external dependencies and leveraging AI not just to automate tasks, but to support decision-making during incidents.

Currently, several monitoring companies are leveraging AI, but in limited ways. Most are adding chatbots that can summarize dashboards or generate reports that look impressive in PowerPoint presentations. But that’s not solving the core problem.

The better companies are using AI to compare current behavior against historical patterns, identifying anomalies that may not be immediately visible. They use AI to correlate telemetry across multiple services to pinpoint where failures are most likely originating.

But the very best companies are taking this further with AI that eases the chaos during incidents; AI that analyzes your monitoring setup and makes intelligent recommendations about coverage gaps; AI that suggests additional tests, and even pre-configures them for you — AI that recommends next steps based on prior successful resolutions, turning incident response from a process of trial and error into one guided by evidence and experience.

This is how AI solves the fundamental problem that keeps SREs, DevOps engineers and IT ops teams awake at night: The gap between detection and resolution. The space between I see the problem and I know what to do about it.

The goal is not to eliminate human judgment — it is to amplify it. Teams still choose the strategy, validate the fix and restore normalcy — but they do so with better information, faster.

Distributed systems and AI-powered applications are not becoming simpler. Dependencies will continue to multiply and outages will continue to occur. The defining difference will be the speed and confidence with which teams can respond. In short, the question is shifting from what now to what’s next. Moreover, the teams prepared to make that shift will define the next generation of operational resilience.

November 26, 2025
A Modern Approach to Multi-Signal Optimization
In modern cloud operations, we’re not just facing a data tsunami; we’re caught in a signal storm. An application’s throughput metric looks flat, but its underlying pod central processing unit (CPU) usage is spiking. A new deployment manifest is applied successfully via kubectl, but user latency is slowly creeping up. These signals are a mix of complementary and conflicting information, creating a confusing picture that paralyzes decision-making.

The problem isn’t the signals themselves, but our approach to them. We consider all metrics equally important, which leads to confusion. Without a framework, a spike in a pod’s CPU usage is given the same weight as a drop in user sign-ups. This lack of structure is the root cause of the analysis paralysis many engineering teams feel today.

This confusion has specific negative impacts. It leads to slow incident response (MTTR) as engineers waste time toggling between Grafana dashboards, Datadog traces and kubectl logs, manually trying to correlate conflicting signals. It also drives costly ‘best guesses’. When a pod begins restarting, the loudest symptom is a resource issue. Therefore, the reflexive action is to scale up behavior that contributes directly to the industry-wide 32% cloud waste statistic. Most importantly, you cannot build reliable automation on a foundation of chaotic, unclassified signals.

To escape the storm, we need a deliberate multi-signal optimization strategy. This strategy’s success hinges on a critical first step: Analyzing and classifying all signals into a logical framework. Only then can we build a solution that is not just fast, but also precise and cost-effective.

Creating Order with Metric Classification

The practical first step in this strategy is to organize the chaos. It’s how you make sense of the noise before you can act on it. We do this by implementing a framework called the signal hierarchy.

Outcome metrics are the ‘what matters’ or ‘north star’ metrics. These define business and user success and are what you are solving for. They are always application-level indicators such as p99 latency, application error rates, cost per transaction or user conversion rates.

Primary metrics are the ‘external drivers’ or ‘causal signals’. These are independent events that act upon your system causing it to react. In the Kubernetes world, these include changes in user traffic from an ingress controller, application programming interface (API) requests per second, a code release or a configuration change or the trigger of a CronJob.

Secondary metrics are the ‘internal symptoms’ or ‘diagnostic signals’. They show how the system’s infrastructure is responding to the primary drivers. These are the classic infrastructure metrics we’re all familiar with: Pod CPU and memory usage, network I/O and disk saturation.

The goal of this classification is to understand the role of each signal. This simple act of categorization allows a fix or a resolution to be found systematically and logically: It prioritizes negative changes in outcomes, looks for causes in primary signals, and uses secondary signals for evidence and diagnosis.

The Strategy in Action: From Classification to Correlation

Let’s walk through how this multi-step strategy works in a real-world Kubernetes environment. Imagine a sudden, sharp increase in the p99 latency for an e-commerce checkout service running as a deployment.

The first step is real-time classification. As signals stream in from Prometheus, the Kubernetes API and CI/CD tools, it should go through a classification system to get classified appropriately. The p99_latency_spike would be tagged as an outcome. A pod_cpu_high metric would be tagged as a secondary. The other two signals are identified: A recent increase in api_requests_per_sec and a deployment_event from ArgoCD are tagged as primary. This classification system, which could be a static classifier or an ML-based classifier, provides an essential context for what comes next.

With signals now classified, we need an approach to correlate them. This process prioritizes the negative outcome (high latency) and immediately looks for a causal primary signal to explain it. Here, it finds the ‘bad deployment’ narrative: The p99_latency_spike (outcome) began shortly after the deployment_event (primary), which was followed by the pod_cpu_high metric (secondary). These three signals complement each other perfectly.

However, what if the signals were conflicting? Imagine the latency is high (outcome), but pod CPU and memory are normal (secondary) and ingress traffic is flat (primary). The lack of complementary infrastructure symptoms is just as important. It tells the engine to correctly rule out a resource bottleneck in the service itself and widen its search for other primary causes such as the failure of a downstream API dependency.

The diagnosis is now based on a coherent, correlated story; not a single, noisy alert. In our first scenario, the system generates a precise operation — kubectl rollout undo — instead of the simplistic and costly kubectl scale deployment. This generated operation is then dispatched through a safety pipeline for verification before execution.

The result is that a precise and effective action was taken. The right action was taken because the system first classified the signals to understand their roles, then correlated them to find the true narrative.

Proactive Classification: Focusing on the Metrics That Matter

The signal hierarchy isn’t just a reactive tool for incident response. Its most profound impact comes from applying it proactively. For any given application, we can perform this analysis ahead of time, creating a focused model of what truly drives its behavior and performance.

Instead of waiting for an issue, we can analyze a service’s telemetry landscape in a steady state. Out of the hundreds of metrics an application performance monitoring (APM) tool may expose for a single service, we can pre-identify the critical few. For our checkout service, this means defining that p99_latency and payment_error_rate are its core outcome metrics, and ingress_traffic and inventory_api_calls are its key primary drivers.

The immediate benefit is a dramatic improvement in efficiency. When an alert on an outcome metric fires, the system’s initial hypothesis formation is no longer a search across hundreds of potential signals. It begins with a much shorter, pre-validated subset of high-impact metrics, making the correlation step exponentially faster. But the true power of this pre-identified model is that it allows us to move beyond rapid reaction into the realm of prediction. By continuously monitoring anomalous patterns in the key primary and secondary drivers, the system can predict imminent issues and take corrective action before the user-facing outcome metric is impacted. This is the shift from firefighting to fire prevention.

This proactive approach also yields a powerful secondary benefit: Optimizing the observability stack itself. By identifying which metrics are truly causal and which are simply redundant or highly correlated symptoms, teams can make informed decisions to reduce their monitoring overhead. This allows you to potentially stop measuring, storing and paying for unnecessary telemetry without losing any meaningful visibility, thereby reducing the cost of your observability platform.

From Strategic Insight to Business Impact

A disciplined multi-signal strategy is more than a technical improvement; it’s a direct driver of business value, moving organizations from costly reaction to profitable proaction. The impact of this strategic approach is clear, measurable and significant.
- Crushing Cloud Waste: By strategically correlating traffic patterns (primary) with resource usage (secondary), we’ve found that over 60% of production workloads can be tuned for cost improvements of 20% or more.
- Unlocking ‘Long Tail’ Savings: For the bottom 50% of an organization’s resources (such as staging and development environments), this strategy consistently yields average cost reductions of 30–40% by understanding usage schedules and dependencies.
- Boosting Performance and Reliability: By automatically correlating negative outcomes to their primary triggers, root cause analysis becomes an instantaneous process. This is why our model leads up to 78% less application downtime.
The key to conquering modern cloud complexity isn’t another dashboard or another tool. It’s adopting a holistic strategy. This strategy must begin with the foundational step of classifying signals to create order from chaos. From this foundation of clarity, you can build powerful, intelligent automation that saves money, protects revenue and — most importantly — allows your engineers to focus on innovation.

Stop chasing individual signals in a storm of data. It’s time to adopt a multi-signal optimization strategy and let an autonomous engine turn that chaos into clear, actionable and profitable intelligence.

KubeCon + CloudNativeCon North America 2025 is taking place in Atlanta, Georgia, from November 10 to 13. Register now.
October 27, 2025