devopsy.kinsta.cloud

Tag: incident response

When Metrics Overwhelm: How SREs Help Engineers Reclaim Focus
If engineers were polled on their feelings toward observability, many would express skepticism if not outright frustration. While observability promises transparency and faster root cause analysis, its rise has reshaped developer work in ways that few anticipated and not always for the better. What was once DevOps turf now encroaches on core engineering, dragging developers into data overload, endless dashboard hopping, and “war rooms” at 3 a.m..

Modern systems generate vast amounts of logs, traces and metrics, but most observability tooling was designed with operators in mind, rather than those actually writing the code. Platforms often deliver “visibility” at the expense of clarity, requiring developers to sift through irrelevant signals and noisy alerts just to find actionable insight. As Stephen Elliot, IDC analyst, notes, “Developers waste valuable time sifting through excessive logs and metrics to pinpoint the root cause of issues.”

The Contradiction: Reactive Alert Fatigue vs. Real Dev Value

Engineers lament that observability, billed as a paradigm shift, frequently devolves into reactive firefighting. When every service emits hundreds of metrics and “critical” alerts, distinguishing noise from real problems becomes a full-time job and ironically, can obscure the root cause engineers are paid to fix. Developers complain of “alert fatigue” and claim that they’re pulled away from building features to become unintentional support staff.

Case in point: One financial sector developer recalls, “We’d spend more time identifying who needed to be involved than resolving the issue. Logs were scattered across different systems, requiring switching between multiple dashboards. This inefficiency meant prolonged downtime and frustrated customers.”

SREs to the Rescue: Building the Bridge, Not the Barrier

Yet amidst developer complaints, Site Reliability Engineers (SREs) are fighting back and winning. SRE teams are reframing observability as a means for empowerment, not just surveillance. By championing smarter, more integrated observability stacks, SREs reduce the chaos and restore developer focus.

How SREs turn the tide:
- Contextual logging: Instead of generic logs and traces, SREs work with developers to embed meaningful context correlating frontend symptoms with backend causes, dramatically reducing time-to-resolution.
- Refined metrics: SREs audit what’s actually measured, pruning useless alerts and aligning metrics with real business outcomes. One success story involved cutting unneeded error logs by 70%, boosting feature release velocity.
- Distributed tracing: Tracing frameworks like Jaeger and OpenTelemetry help SREs “follow the data,” mapping service-to-service dependencies. Engineers get end-to-end visibility without drowning in dashboards.
Real Use Cases: From War Rooms to Win Rooms

A high-profile e-commerce incident shows SRE impact: During a critical outage, SREs led the charge by leveraging distributed tracing, narrowing the issue to a slow authentication service amid thousands of requests. Developers, once skeptical, credited observability tooling paired with seasoned SRE navigators for cutting downtime from hours to minutes.

At a global media company, SRE-driven observability transformed post-mortem culture. Instead of finger-pointing, collaborative incident reviews highlighted engineering wins — such as rapid root cause identification and improved system resilience—ultimately boosting morale and cross-team trust.

Engineers: From Reluctant Observers to Advocates?

While resistance lingers, smart observability (championed by SREs) is beginning to convert skeptics. Tools like middleware, Grafana, etc., are tailored for real workflow, not just metrics, and are bringing developers closer to high-value debugging and proactive system improvement. Engineer buy-in increases when:
- They see immediate feedback on code changes.
- Alerts are relevant, actionable, and tied to the service they support.
- They collaborate with SREs, gaining context rather than just data dumps.
Key Takeaways
- Observability demands developer involvement, but poor tooling breeds resentment and alert fatigue.
- SRE teams succeed by curating signal over noise, contextualizing data and aligning metrics with business goals.
- Collaborative post-mortems and contextual tracing turn observability from a pain point into a productivity win.
- Engineer advocacy grows when observability improves their workflow, not just system surveillance.
Ready for the next round of observability? Let those building the systems help design how they’re seen.
October 13, 2025
From Incidents to Insights: The Power of Blameless Postmortems
In complex systems, failure isn’t a possibility – it’s a certainty. Whether it’s transactions vanishing downstream, a binary storage outage grinding builds to a halt, or a vendor misstep cascading into a platform issue, we have all likely seen firsthand how incidents unfold across a wide range of technical landscapes. Often, the immediate, apparent cause points to an obvious suspect like a surge in user activity or a seemingly overloaded component, only for deeper, blameless analysis to reveal a subtle, underlying systemic flaw that was the true trigger. But what separates high-functioning teams from the rest isn’t whether things break, it’s how they respond. Traditional postmortems often descend into subtle finger-pointing and defensive behavior. Blameless post-mortems flip that script, transforming incidents into structured opportunities for learning, accountability and resilience.

Decoding “Blameless” – Beyond Just Forgiveness

Blameless doesn’t mean avoiding accountability; it means shifting the focus from individual fault to systemic understanding. In mature DevOps cultures, incidents aren’t seen as personal failures but as signals from the system, urging teams to examine how processes, decisions and tools may have contributed.

At the heart of this approach is psychological safety, the confidence that team members can speak openly without fear of judgment. When people feel safe, they’re more likely to share what really happened, including actions they took or things they missed. That transparency is essential for uncovering not just what went wrong, but why it made sense at the time.

Consider a typical scenario: A failed deployment ripples across services and triggers a widespread outage. On the surface, it appears simple –

A developer pushed a misconfigured change that bypassed automated checks and production went down. But when the team steps back and applies a blameless lens, deeper issues emerge: A validation system hadn’t been updated to handle edge cases, code reviewers lacked context due to outdated documentation and alerting thresholds were too broad to catch the issue early.

Instead of assigning blame or applying a quick patch, the team focuses on strengthening the system, improving validation logic, updating docs and tuning alerts.

The result isn’t just a fix, it’s a smarter, more resilient culture that gets stronger with every challenge.

Essential Preparation – Setting the Stage for Productive Learning

A successful post-mortem doesn’t begin with the meeting – it begins with structured, collaborative preparation. Without the right people, reliable data and a clear process, even the most well-intentioned sessions can become reactive or incomplete.

One of the most effective practices is to document first, discuss later. Before asking everyone to join a call, share a structured post-mortem document where stakeholders can contribute asynchronously. This includes timelines from incident responders, observations from impacted teams, customer impact summaries from product owners and supporting data logs, metrics, traces, or dashboard snapshots. This approach not only gives the facilitator a head start on organizing the flow, but also allows participants to come prepared—saving valuable time on the call and leveling the playing field for quieter voices.

Equally important is getting the right mix of perspectives. Involve not just engineers, but product partners, SRE or observability teams and security or compliance if relevant. Broader participation leads to richer root cause analysis and more holistic remediation plans.

Finally, ensure that evidence anchors the conversation. Logs, traces, deployment records and communications history should be accessible in advance. When everyone arrives informed, the post-mortem becomes a focused and inclusive discussion not a scramble to piece together what happened.

Facilitating the Learning – Guiding the Discovery

With preparation complete, the post-mortem meeting shifts from information gathering to insight generation. This is where facilitation truly matters, setting the tone for transparency, curiosity and growth.

The session usually opens with a walkthrough of the incident timeline. But rather than just reading it aloud, a good facilitator encourages teams to pause and reflect at each key moment: When did the first symptom emerge? What was observed? What did we assume at the time? This narrative reconstruction often surfaces hidden dependencies, overlooked alerts, or misaligned expectations.

The conversation must stay focused on systems — not people. Instead of asking “Who missed this step?”, the better question is “How was this step missed—and what in the process allowed that?” Techniques like The 5 Whys or cause-and-effect mapping help the team move from surface symptoms to systemic insight.

Facilitation also means managing the human side of the discussion. Tensions may surface, especially if the impact was broad or public. It’s essential to gently redirect blame-driven comments, invite quieter voices to share and validate emotions without letting them override the objective.

The best post-mortems are those where discomfort leads to discovery, and where every contributor leaves with a clearer view of how the system works—and how it can be made stronger.

From Insights to Action – The Continuous Improvement Loop

A meaningful post-mortem doesn’t stop at identifying what went wrong, it lays the foundation for what must change going forward. But in most real-world teams, change doesn’t happen overnight. Instead, the goal is to develop a clear, actionable plan and assign ownership for driving improvements over time.

Rather than jumping into instant fixes, teams should align on a set of SMART follow-up actions – Specific, Measurable, Achievable, Relevant and Time-bound. These are then delegated to the appropriate team or individual as part of upcoming sprint cycles, backlog grooming, or architectural roadmaps.

Examples include:
- Enhancing validation logic to catch similar edge-case configs
- Updating documentation to improve reviewer context
- Revisiting alert thresholds to shorten time-to-detection
Each action is tracked through established workflows such as Jira or Confluence and assigned a clear owner to ensure accountability.

Just as important is closing the loop. Summarizing key findings and sharing them broadly, whether through wikis, incident dashboards, or team retros, reinforces transparency and fosters trust.

When teams see that insights turn into improvements and that no one is punished for surfacing uncomfortable truths, it creates a culture where learning thrives and systems grow stronger with every challenge.

Real-World Best Practices to Strengthen Your Post-Mortems

To build a high-value post-mortem culture, teams can adopt these field-tested habits:
- Ritualize Pre-Meeting Contribution: Establish an expectation that all roles asynchronously add their perspective to a shared post-mortem doc. This sets a tone of shared responsibility and respect for others’ time.
- Rotate Facilitators: Empower team members across roles – Dev, Ops, Product to occasionally lead post-mortems. This builds empathy and shared accountability and brings a new perspective to the meetings.
- Use a Consistent Template: Capture key elements like incident timeline, contributing factors, customer impact and action items. Structure supports clarity in the long run.
- Track Actions in Real Workflows: Ensure follow-up items are entered into Jira, sprint boards, or team backlogs with clear owners and target dates.
- Revisit Outcomes at regular intervals: keep a recurring sync until the final items are completed, maintain a tracker, check in on progress.
- Close with a Reflection Prompt: Before wrapping up the meeting, ask: “What’s one thing you learned or would approach differently next time?” It reinforces personal insight and collective learning.
- Guard the Blameless Ethos: If blame creeps in, gently redirect toward process gaps and systemic contributors.
Conclusion: The Cultural Dividend of Blamelessness

Blameless post-mortems are more than an operational ritual – they are a reflection of an organization’s maturity and mindset. By shifting the focus from who failed to what the system allowed, teams foster trust, drive continuous improvement and make space for honest reflection.

Over time, this practice builds far more than just resilience. It cultivates a culture where engineers feel safe to surface risks, where issues are seen as opportunities to learn and where accountability strengthens, not stifles collaboration.

The final outcomes are clear: Stronger systems, more cohesive teams and a powerful learning loop that consistently closes.

Ultimately, blameless post-mortems don’t just fix today’s problems. “They build resilient teams ready for tomorrow’s challenges.“
August 6, 2025
Logz.io Leverages AI to Identify Anomalies in Real-Time

Logz.io this week revealed it has added a real-time anomaly detection capability to its observability platform that makes it simpler to correlate the impact IT events will have on business processes.

This capability is based on artificial intelligence (AI) models trained using historical data collected via the Logz.io Open 360 platform.

Logz.io CTO Asaf Yigal said this addition to the Open 360 platform enables DevOps teams to employ machine learning algorithms to identify anomalies in mission-critical processes that warrant further investigation. Armed with insights that go beyond simple IT infrastructure consumption metrics, it becomes possible to significantly reduce mean time to resolution, he added.

Logz.io has been embedding AI models within its platform since last year. Still, this latest capability will enable organizations to further bridge a divide between IT and the business that has long plagued many organizations, noted Yigal.

The ultimate goal is to make it simpler for DevOps teams to instantly identify any anomalous application behavior that represents a risk to the business, including cybersecurity events, he added.

The AI capabilities that Logz.io has been steadily adding to its observability platforms are largely transparent to DevOps teams, said Yigal. Instead, Logz.io is embedding multiple types of AI models into its platform that can be invoked in much the same way any other capability is provided, he added. DevOps teams are not necessarily going to need to build their own models to take advantage of AI to observe application behavior at scale, noted Yigal.

The goal is to surface those capabilities consistently at a level of cost organizations can afford, added Yigal.

It’s not likely AI will replace the need for DevOps teams any time soon, said Yigal. After all, there is a reason AI models are referred to as co-pilots versus an automatic pilot, he noted.

It’s not clear to what degree organizations today can correlate the impact an IT event is going to have on the business. But as organizations continue to realize how dependent they are on applications to drive digital processes, the need to bridge the historic divide between IT and the rest of the business has become more urgent. In fact, many of the product and brand champions that organizations now rely on to manage a business unit often have a background in DevOps.

Of course, business professionals that are steeped in IT are still relatively rare but as advances in AI continue to make it simpler to understand how complex application environments function the number of users of an observability platform will naturally expand beyond IT. The challenge is presenting each stakeholder with a view into the IT environment that meets their specific requirements.

It may take a yet before the divide between IT and the rest of the business is permanently bridged but with each passing year it gets narrower. The challenge and the opportunity now is making sure everyone on both sides of that divide understands the art of the possible today that only a few short years ago would have seemed impossible to achieve.

March 1, 2024
Elastic Previews Unified Query Language for Search Platform

Elastic this week previewed a standard query language that can be used across its portfolio to streamline investigations into IT and cybersecurity events.

Gagan Singh, vice president of product marking for Elastic, said Elasticsearch Query Language (ES|QL) makes it simpler, via a single query, to aggregate data from multiple sources using a piped query syntax that makes it simpler to iteratively interrogate data.

Scheduled to become generally available in 2024, ES|QL replaces the multiple domain-specific languages the company has been providing for each version of its search engine. Instead of having to move data into a central repository, Elastic Seach makes it possible to query data wherever it resides in an organization, noted Singh.

That approach enables, for example, a site reliability engineer (SRE) to discover the root cause of an issue through a series of queries that can be launched from a familiar search interface, he said. It will also foster increased collaboration across IT teams that need to visualize query results, added Singh.

In general, IT teams are struggling with an exponential increase in the amount of data they need to analyze to discover the root chaise of an issue, noted Singh. The Elastic search engine provides a means to analyze that data at the level of scale that is now required, he added.

The preview of ES|QL comes on the heels of a technical preview of a generative artificial intelligence (AI) tool for observability the company has made available. The Elastic AI Assistant for Observability makes use of large language models (LLMs) and the Elasticsearch Relevance Engine (ESRE) to explain application errors, log messages, alerts and recommendations for improving code in plain language. DevOps teams can visualize and interactively chat with telemetry data and then invoke runbooks to remediate issues.

Elastic has also generally made available a Universal Profiling tool to enable DevOps teams to identify bottlenecks that adversely impact code performance.

There is little doubt that fundamental changes to how DevOps and cybersecurity teams interact with telemetry data are on the way. The challenge now will be determining when to rely on a generative AI tool versus diving deeper into that data by launching a series of queries. At the very least, AI should provide DevOps and cybersecurity professionals with a better sense of where to start an investigation.

Ultimately, the goal should be to reduce the level of DevOps and cybersecurity effort that is currently required to succeed. In theory, as it becomes simpler to investigate issues, the number of organizations capable of embracing DevOps best practices should increase as well. That’s crucial because IT environments are only becoming more complex with each passing day. The number of dependencies that exist between services has become too numerous for IT teams to track manually. More challenging still, those services are dynamically being updated, so what was known today about them might no longer be relevant tomorrow.

Hopefully, the amount of time and toil required to address issues will significantly decline in the months ahead as the next generation of tools becomes available. The challenge now is hanging on long enough for them to be battle-tested.

November 10, 2023
IT Service Incidents Are Becoming More Frequent, Survey Says

A survey of 1,000 IT operations, DevOps, site reliability engineering (SRE) and platform engineering professionals in the U.S. conducted by Transposit, a provider of an incident management platform, found more than two-thirds (67%) have seen an increase in the frequency of service incidents that have affected their customers over the past 12 months.

Announced at the Kubecon + CloudNative conference today, the survey found 62% of respondents have also seen an increase in the amount of time it takes to resolve incidents over the course of the last year, with 80% reporting it takes up to six hours on average to resolve incidents from the first alert to mitigating the issue.

Paradoxically, however, 71% of respondents also claim to have automated incident management to their satisfaction, with 59% having a defined incident management process in place. One-third (33%) said only 11-25% of their incident management tasks or workflows are automated.

Transposit CEO Divanny Lamas said despite that level of comfort it’s clear the number of incidents that need to be addressed are continuing to increase as IT environments become more complex. In fact, in the next 12 months, 72% expected their organization to expand its IT stack.

At the same time, costs per incident are rising, with organizations at risk of losing up to $499,999 per hour on average, according to 63% of respondents. Nearly half (47%) said downtime can cost anywhere from $100,000 to $2 million.

A full 85% of respondents are hopeful that generative artificial intelligence (AI) will help further streamline incident management processes, with 80% having already embraced AI to varying degrees. More than half (51%) feel AI is making their job better, with 65% using it to improve the accuracy and quality of data. Just over half (51%) reported faster time to incident resolution, with 50% using it to more quickly and easily identify the root cause of issues, potential threats and vulnerabilities. Just under half (48%) use it to automate repetitive tasks or processes, streamlining their operations effectively.

A full 90% of respondents said integrating generative AI capabilities into incident management tools or platforms decreased the time it takes to create new automations.

Overall, 43% reported current incident management processes are not effective or are only being used by some team members due to confusing documentation (41%), limited access to tools (40%) and reliance on institutional knowledge (40%). Well over a third (37%) reported that only select team members have a comprehensive understanding of the defined incident management process and that they are adhered to consistently.

A full 96% said they believed it would be beneficial if all the tools their organization used to manage incidents were integrated through one tool or platform, the survey finds.

Top challenges cited included not enough buy-in from leadership or management (57%), not enough sharing of knowledge (54%), inadequate documentation of institutional knowledge and existing processes (54%) and a lack of clarity about what to automate (52%).

Respondents reported they expected to embrace technologies including AI or tools that employ machine learning algorithms (60%), automation tools or applications (53%) and communication/collaboration tools or applications (48%).

In addition, over the past year, 62% of respondents have increased their focus on SRE practices, with 58% planning to increase adoption of platform engineering as a methodology for centralizing the management of DevOps workflows.

Ultimately, automation should reduce incident management stress for all concerned, said Lamas. The challenge, as always, is making it simpler to achieve that goal in a way that doesn’t rely on a complex mix of custom scripts that don’t scale, she added.
One way or another, it’s clear that when it comes to incident management, there is still plenty of room for improvement.

November 7, 2023
New Relic Extends AI to Identify Alert Coverage Gaps

New Relic this week added additional artificial intelligence (AI) capabilities to its observability platform that make it possible to detect and resolve alert coverage gaps.

Camden Swita, a senior product manager for New Relic, said it’s not uncommon for DevOps teams to discover an issue that the existing alerts they put in place were not designed to address. New Relic is adding machine learning algorithms at no additional cost that are designed to identify anomalies indicative of those potential issues. The platform then surfaces recommended alerts that DevOps teams can apply or further customize as they see fit, he added.

Those machine learning algorithms ensure DevOps teams are tracking the right thresholds to ensure there are no blind spots that could adversely affect application performance, noted Swita.

New Relic is employing both predictive and generative AI to enable DevOps team to detect and resolve issues faster. The company has been using machine learning algorithms for predictive AI for several years and, more recently, started providing early access to a generative AI tool dubbed Grok.

The overall goal is to make the capabilities of an observability platform more accessible, Swita said. This is accomplished by surfacing issues without requiring a software engineer to craft intricate queries to interrogate metrics, logs and traces, he noted.

The AI capabilities also serve to reduce the amount of alert noise; it’s now possible to more easily understand the root cause of an issue and reduce the total number of alerts that might otherwise be sent, he added.

While there’s no doubt AI will help democratize DevOps by making it possible to write scripts and better manage workflows, it might be a while before DevOps teams trust the recommendations made. AI models typically require some time before they understand how an IT environment functions, and not every initial recommendation may be valid. In time, however, DevOps teams will find themselves increasingly supervising processes that AI models are executing.

In addition, there are competing approaches to building large language models (LLMs) that drive generative AI. For example, it’s not clear whether chaining LLMs together will be more efficient than creating a single large ‘parent’ LLM that has ‘child’ relationships with smaller LLMs, noted Swita.

In the meantime, DevOps teams would be well-advised to list tasks that are likely to be automated by AI models. Many of them will tend to be manual tasks that most software engineers don’t especially relish, so in that regard, there’s naturally a lot of interest in AI. It remains to be seen, however, whether various forms of AI might one day automate an entire DevOps workflow. In fact, most DevOps engineers are going to set a high bar for AI models that, at the very least, will be expected to perform a task as well as humans before it’s trusted completely.

August 24, 2023
Unleashing AI in SRE: A New Dawn for Incident Management

In my recent blog, Revolutionizing the Nine Pillars of SRE with AI-Engineered Tools, I indicated that AI could assist with incident management by automating the detection and triage of incidents and helping to quickly identify the root cause. In this blog, I explain in more detail how AI-engineered tools can be used to improve the SRE incident management pillar of practice.

AI Use Cases for Incident Management

Here are use cases for SREs to use AI to assist with software deployments.

Detection of Incidents: AI can be instrumental in identifying and predicting incidents before they cause severe damage. Tools like Moogsoft and Datadog use AI to analyze system behavior and predict potential failures. However, a challenge could be ensuring these AI models are correctly trained to understand what constitutes an incident. To overcome this, invest time in training these models with historical incident data and continuously fine-tune them.

Communication of Incidents: AI can automate the process of notifying the right teams when an incident occurs based on its severity and impact. Tools like PagerDuty use machine learning to streamline incident communication. A challenge might be setting up correct escalation policies. This can be overcome by carefully analyzing past incidents, understanding their impact and setting up escalation rules accordingly.

Triage of Incidents: AI can assist in categorizing and prioritizing incidents based on their potential impact on the business. IBM’s Watson AIOps uses AI for effective triaging. However, defining the right parameters for triaging could be a challenge. It can be overcome by understanding the nature of incidents, their potential impact on the business and defining the rules for triaging accordingly.

Identify the Root Cause: AI can help quickly identify the root cause of incidents by analyzing patterns in historical data. Tools like RunWhen and Zebrium use machine learning for root cause analysis. A potential challenge is ensuring that the AI model has access to all necessary data for analysis. You can overcome this by setting -p comprehensive logging and monitoring systems that collect all relevant data.

Blameless Postmortems: AI can provide data-driven insights to facilitate blameless postmortems, helping teams understand what went wrong and why. OverOps and Blameless are examples of such tools. The challenge here is ensuring an unbiased analysis. Overcome this by setting up AI models that rely on factual data rather than assumptions.

Documentation of Incidents: AI can help in auto-documenting incidents, capturing important details that could be crucial for future reference. JupiterOne, for instance, uses AI for this purpose. However, capturing all relevant details in documentation can be a challenge. To overcome this, configure the AI tools to collect comprehensive details about the incidents.

Automation of Playbooks: AI can help automate the execution of incident response playbooks, reducing human intervention and response time. Tools such as StackStorm and Blameless Insights are tools that can help here. A potential challenge is ensuring the right playbook is executed for the right incident. Overcome this by training the AI model with historical incident data and the corresponding successful playbook execution.

RoadMap for SREs to Transform Incident Response Using AI

Needs Assessment and Gap Analysis: Start by understanding the current incident management process and identifying areas where AI can improve. Also, analyze the types of incidents you face most frequently and the time taken to resolve them. This will help create a clear problem statement and set expectations from AI implementations.
Tool Selection: Based on the needs assessment, identify the right AI tools. Look at AI tools specializing in incident detection, communication, triage, root cause analysis and automation. Tools such as Moogsoft, PagerDuty, IBM’s Watson AIOps, Zebrium, OverOps, JupiterOne, RunWhen, Blameless and StackStorm might be good starting points.
Model Training and Configuration: Configure the chosen AI tools as per your incident management process. You’ll need to train the models using historical incident data so they accurately reflect your environment.
Integration with Existing Systems: Ensure the chosen AI tools can integrate seamlessly with your existing systems, such as your ITSM tool, monitoring and logging systems and communication channels. This is crucial for the automation of the incident management process.
Pilot Implementation: Start with a pilot implementation on a small scale, maybe for a single service or a part of your infrastructure. This will help you understand the effectiveness of the AI tools and make necessary adjustments before a full-scale rollout.
Measure and Optimize: Regularly measure the performance of the AI tools and their impact on your incident management process. Make necessary optimizations based on feedback and performance metrics.
Full-Scale Rollout: Once you’re confident about the performance of the AI tools and their effectiveness in improving the incident management process, plan for a full-scale rollout.
Continuous Learning and Improvement: The power of AI lies in its ability to learn and improve over time. Keep feeding it new incident data and train the models for better accuracy and effectiveness. Also, keep an eye on new advancements and features in AI tools and update your implementations accordingly.

Summary

Modern incident management is being revolutionized by AI, bringing unprecedented improvements in detection, communication, root cause analysis, postmortem reporting and automation of playbooks. The power of AI lies in its ability to learn and adapt, making it a powerful ally for SRE teams dealing with complex and evolving systems. But the journey to effective AI-based incident management isn’t without challenges. With the right tools like Moogsoft, PagerDuty, IBM’s Watson AIOps, Zebrium, OverOps, JupiterOne and StackStorm and a well-planned implementation roadmap, organizations can successfully overcome these challenges and unlock the full potential of AI in incident management.

The roadmap to AI-based incident management starts with a thorough needs assessment and gap analysis, followed by tool selection and configuration. An integration with existing systems ensures seamless data flow, while a pilot implementation allows for tweaks and adjustments. Measuring and optimizing the performance of AI tools is a crucial step before a full-scale rollout. And the journey doesn’t end there—continuous learning and improvement is key to harnessing the full potential of AI. With the right approach and tools, the transformation to AI-based incident management can significantly reduce downtime, speed up resolution times and improve system reliability—a win-win for both SRE teams and their users.

August 15, 2023
The DevOps Pendulum: Agility Vs. Control

Managing changes to cloud assets is a universal pain felt by many engineering leaders today, despite all of the advancements in tooling and practices like GitOps. This is because, in reality, it is simply impossible to always have everything completely locked down—we don’t live in a utopia with zero incidents. If your engineering organization prevents any changes from being made through the cloud console or to your infrastructure-as-code (IaC) without complying with strict GitOps practices or change management processes via CI/CD, then it’s likely you have very frustrated developers—developers who can’t troubleshoot or debug in real-time and who have little control in a real-world incident.

Engineering, like everything else in life, is all about balance.

The Pendulum Swings

The pandemic created a whole new mindset and practice for managing distributed and high velocity engineering and operations. Overnight, companies that were not built for remote ways of working had to continue their operations in a global, distributed and asynchronous way that they weren’t completely familiar with. This required a new way to think about software delivery—and it accelerated the DevOps practices that support that delivery. Self-service infrastructure removed barriers for developers to ensure continued performance and velocity.

At the same time, your cloud cannot be the wild west with everyone creating bespoke infrastructure. It becomes impossible to manage, and misconfigurations can be risky. Guardrails and policy automation have become a hot topic. Today, with tech markets depressed and cloud costs escalating, there seems to be a growing trend toward locking things down again, even at the risk of frustrating developers.

This begs the question: How can you achieve unencumbered infrastructure for your developers while simultaneously following policies and best practices for compliance, risk and cost? There is a way to find balance.

Like many aspects of security, we’ve learned that when constraints and barriers are too high, users ultimately find ways to bypass them. This is true for operations, as well. While sometimes it might seem easier to lock everything down than design a better and more balanced way to enable developers to move fast, eventually this approach backfires. This is the exact same evolution the application and cloud-native security industry are undergoing right now. All the guardrails and controls applied have created too much friction in development processes and developers ultimately bypass them.

Real-Time Insight is Key

CloudOps can learn a lot from the disruption the security industry is going through today. In the same way that point-in-time security has been rendered completely useless, non-real-time alerting or eventual alerting to infrastructure drift just won’t cut it when managing an ephemeral cloud. What is really needed is the same kind of real-time and continuous scanning of cloud assets and IaC, similar to what we apply to our systems through monitoring and observability. These solutions became an essential backbone of our business to ensure continuous operations and availability of cloud services.

As we embrace IaC and the benefits it brings, everything-as-code unlocks greater agility and visibility, enabling you to automatically remediate without locking everything down. As DevOps says, “fail forward and fail fast.” Rather than focusing on never making a mistake, focus on how to fix it immediately.

The Solution

By providing a continuous comparison of actual cloud assets with their desired state through IaC and GitOps, it’s possible to surface configuration drift and policy violations immediately, much like any other kind of breach or major system failure. Failures and incidents are inevitable. It’s unrealistic, and even dangerous, to build systems with an inherent underlying design that prevents you from changing something at the cloud console at 2:00 A.M. during an outage.

With a locked-down approach, a developer would need to wait for the change management approval during a high-pressure production incident, oftentimes at 3:00 A.M. (because the incident gods of always make them happen in the middle of the night) or on a weekend. We don’t need to look much further than the catastrophic 40-minute and $440 million Knight Capital incident to understand that sometimes time is not on our side and that a delay can have severe consequences.

Cloud drift has become equally troubling as uptime or any other mission-critical aspect of continuous business operations, and that’s why we need to apply the same principles for monitoring drift in real-time as we do to our CPU and load. This will make it possible to be alerted when your cloud and your IaC are no longer aligned and even flag issues that can be prevented in real-time. You can automate ticket creation and escalation and also triage whether this is something that needs to be fixed in real-time with remediation suggestions provided or open a ticket to fix it later. The choice should be yours, not made by an administrator who has a lot less context about the systems and the ultimate business impact.

It is Not ‘One and Done’

Some vendors preach codifying your cloud into IaC and then locking it down so that you’ll never have to do it again. This feels like deja vu; we’ve definitely heard this before. It sounds a lot like the ‘write once, deploy anywhere’ JVM promise that never quite panned out.

By going down this path, you’re resigning yourself to never adding new cloud assets or changing your cloud configurations. If there’s one thing we’ve learned about the cloud, it’s that it’s a constantly moving target. (Read what cloud asset management can learn from the world of finance—another industry in constant flux). If this didn’t work decades ago with the JVM, it certainly won’t work in a dynamic and constantly changing environment like the cloud. That’s a fool’s journey.

Continuous Cloud Up to Code

There are major downsides of locking down agility and velocity. This is hard-earned advice, witnessed firsthand from tools like legacy CMDB and startups who have forgotten that cloud is ephemeral—and possibly the reason why we all use it in the first place! So if our cloud is constantly changing and is a continuously moving target, then shouldn’t our tools be built in just the same way to keep up? Solutions built for the rapidly evolving cloud-native era must continuously look for new, unmanaged and changed assets and ensure their cloud infrastructure always follows policies and regulatory standards.

Once we are able to find the balance between agility and control, we’ll be able to tap into the benefits of speed and safety (as DORA research shows time and again) that are the benchmark of high-performing teams. Automating cloud asset management can provide governance, policies and control while still enabling the velocity and agility engineering organizations today require.

Agility Versus Control

There’s a minority of people who claim they don’t worry about cloud asset configuration drift because they have everything completely locked down. Zero changes can be done at the cloud console or to IaC without going through GitOps for strict change management processes via CI/CD. I bet they have happy developers—not!

Life is All About Balance

Throughout the pandemic, organizations had to make developers happy for fear they would leave. So DevOps went full tilt aiming for self-serve infrastructure for developers to remove barriers to speed and velocity. Now, with tech markets depressed, maybe it’s easier to lock things down even if that risks losing engineers.

As we embrace IaC and its benefits, you need not live in the stone age of locking everything down. How can you achieve unencumbered infrastructure for your developers while following policies and best practices for compliance, risk and cost? There is a way.

The solution? Real-time continuous scanning of your cloud assets and your IaC with continuous comparison can surface configuration drift and policy violations immediately. Emergencies are inevitable. It’s bold/stupid/foolish to think you’ll never need to change something at the cloud console during an emergency or that a developer will wait for change management when production is down at 3:00 A.M. on a weekend. It happens, and you must deal with it.

August 2, 2023
Logz.io Taps AI to Surface Incident Response Recommendations

Logz.io this week added a supervised machine learning capability to its observability platform that reduces mean-time-to-remediation by surfacing recommendations for resolving incidents.

Asaf Yigal, vice president of product for Logz.io, said the Alert Recommendation capability added to the Logz.io Open360 platform uses artificial intelligence (AI) to model the steps a DevOps team needs to complete to resolve an incident.

The goal is to reduce the amount of time required to resolve incidents at a time when IT environments are becoming increasingly complex, he added.

In fact, a recent Logz.io survey found 75% of respondents said it currently takes them hours to resolve production issues, with only 14% satisfied with their current mean-time-to-resolution (MTTR). A total of 41% specifically cited monitoring and observability of Kubernetes environments as a primary challenge.

Alert Recommendation is the latest in a series of investments in AI that Logz.io has made. Previously, Logz.io integrated the ChatGPT generative artificial intelligence (AI) platform to surface links to related information and best practices for resolving IT issues.

In general, AI tools should make it possible to manage IT at a level of scale that eliminates many of the low-level data engineering and analytics tasks that previously required manual effort from a DevOps engineering team. If, for example, the observability platform is generating recommendations to address issues, there may be less of a need to create runbooks that DevOps teams typically create to address a wide range of known issues.

Along with making it simpler to discern the root cause of an issue, AI also makes it possible for less experienced members of a DevOps team to resolve an issue using guidance generated by the observability platform, noted Yidal.

In effect, AI technologies are reducing the cognitive load required to be an effective member of a DevOps team, he added.

One way or another, it’s not so much a question of whether AI will be applied to DevOps as much as it is to the degree. Many of the manual tasks that often conspire to create DevOps bottlenecks should be significantly reduced in the months ahead as more advances are made. The challenge now is determining how best to reallocate DevOps expertise in anticipation of those advances.

Of course, there’s always going to be some sense of trepidation when it comes to AI. However, many of the tasks that are about to become automated tend to be tedious. Many DevOps professionals would just as soon see those tasks become automated in the expectation that more time will become available to take on more complex challenges.

Regardless of the motivation, the way IT is managed is about to change. There may be plenty of instances where AI does not live up to its initial hype, but as AI models are exposed to more data, they will become more accurate. That doesn’t mean, however, there won’t always be a need for a DevOps engineer to ensure those algorithms are behaving as expected.

June 1, 2023
PagerDuty Bets on AIOps and Automation to Simplify IT Management

PagerDuty this week made generally available an artificial intelligence for IT operations (AIOps) platform that leverages the data model embedded in its incident management software to reduce the amount of time required for an AI platform to learn how an IT environment operates.

Jonathan Rende, senior vice president and general manager for PagerDuty, said that approach enables the PagerDuty AIOps platform to begin surfacing valid recommendations sooner than other AIOps platforms. Other platforms first need to be integrated into IT environments by an expensive team of consultants and integrators, he added. The PagerDuty approach makes the benefits of AIOps immediately available to both IT operations teams and application development teams alike, he said.

The arrival of PagerDuty AIOps comes on the heels of the launch of PagerDuty Process Automation, a platform for orchestrating automation across multiple platforms that is integrated within PagerDuty Operations Cloud. That capability makes it possible for organizations to centralize the management of islands of automation that have emerged across the enterprise without having to replace any framework that has already been deployed, said Rende. There are also more than 700 integrations between PagerDuty AIOps and the PagerDuty Operations Cloud platform.

Like other platforms, PagerDuty AIOps relies on machine learning algorithms and an AI model created by PagerDuty to correlate events so that the platform can put them in context. Incident response recommendations are then made, which an IT team can accept and automatically implement or reject. The platform then uses those inputs to surface more accurate recommendations over time, said Rende.

The platform will also automatically group apply defined rules coupled with its machine learning capabilities to aggregate alerts. That capability reduces the overall alert noise that might otherwise be generated, noted Rende.

Organizations are looking to invest in AI and automation more heavily at a time when many of them are reducing the overall size of their IT staff, said Rende. The challenge with AI, however, is that it can take six months or more for a rival AIOps platform to learn enough about an IT environment to begin making recommendations. The PagerDuty approach takes advantage of the data within the PagerDuty incident management platform to train an AI model more quickly, in a way that reduces the time to generate valid recommendations to a matter of days, noted Rende.

PagerDuty has been building that AI model for the last four and a half years with that specific goal in mind, he added.

It’s not clear how quickly organizations are embracing AIOps, but as IT environments continue to evolve, many once-manual tasks are being automated. There is a natural fear of the impact those advances might have on specific roles within an IT organization, but as IT environments get more complex, there will always be a need for IT personnel that can focus on higher-order tasks, said Rende.

Overall, the rate of IT staff turnover should also decline as much of the toil that contributes to burnout is also eliminated.

It may be a while yet before AI becomes pervasively applied to manage IT, but the proverbial genie is not going back into the bottle. The challenge is determining precisely where the line between the man-machine interface with IT environments is today versus where it might be tomorrow.

April 14, 2023