devopsy.kinsta.cloud

Tag: incident management

The Evolution of Incident Management
Have you ever thought about the history of incident management?

If you’re an SRE, you might be so caught up in the day-to-day work of managing reliability and responding to incidents that you never take time to step back and reflect on the evolution of your role and your responsibilities. And that’s a shame because SREs didn’t invent incident management concepts and strategies on their own.

On the contrary, the way SREs think about incident response, structure incident management teams and rank the priority of incidents owes much to incident management strategies developed in the offline world decades ago. To understand fully what it means to be an SRE today, you have to appreciate this deep history of incident management and response.

So, let’s take a look at that history and examine where modern incident response concepts originated.

Historical Problems in Incident Management

Societies have always had incidents, of course. Fires, floods, infrastructure breakdowns and similar crises have been happening for millennia.

For most of history, however, humans lacked an efficient, purposeful way to manage these sorts of incidents. Response efforts were ad hoc, and their effectiveness owed more than a little of their success to sheer luck.

Particular challenges included:
- Lack of effective, consistent communication between stakeholders.
- Varying organizational structures which made it hard to identify leaders, coordinate response efforts and delegate tasks.
- Inconsistent response strategies.
- Differing approaches to assessing the priority of incidents.
Historically, organizations may have been able to handle incidents well enough if the incidents required response from only one, small group. But the more stakeholders involved, the harder it was to respond quickly and effectively.

Putting Out Fires: The Birth of the ICS

Things began to change for the better when stakeholders started thinking about better ways to put out fires—literally.

By the 1960s, fire chiefs in California realized that they were struggling to respond effectively to the wildfires that broke out every summer. Each year brought worse blazes than the last, with more land burned and more buildings lost. The Laguna fire of 1970 brought matters to a head and was the catalyst for a new approach to incident response for fire agencies.

After assessing what was going wrong, the fire chiefs determined that it wasn’t a lack of equipment or personnel. It was poor coordination among the various firefighting agencies that responded to blazes. Lacking a clear chain of command and a systematic approach to firefighting, the agencies struggled to deploy their resources rapidly and, thus, effectively.

To fix the problem, California fire chiefs developed what became known as the Incident Command System, or ICS. The ICS defined a hierarchy for incident response with an incident commander at the top. It also defined several categories of incident response processes including operations, planning, logistics and finance. And it established a consistent set of terms that stakeholders can use to describe their actions during incident response, which makes it easier to communicate clearly.

Although the ICS was initially conceived to fight fires, it became the de facto standard for organizing incident response strategies of all types.

From ICS to NIMS

The history of incident response doesn’t end with the ICS. A new chapter began in the early 2000s when the U.S. federal government developed an even more comprehensive approach to incident management called the National Incident Management System, or NIMS.

NIMS was born in the wake of the September 11, 2001, terrorist attacks, which underlined the importance of efficient communication not just between different agencies of the same type (like fire departments), but of entirely separate organizations. To achieve this, NIMS expanded upon the principles of the ICS.

In addition to adopting most of the incident command principles and practices included in the ICS, NIMS included standards for coordinating the distribution of resources. It also embraced the concept of the emergency operations center, which is in some ways similar to a network operations center in the digital world.

In some respects, NIMS resembled a compliance framework (although to be clear, that’s not what it is). It includes fourteen management principles, similar to compliance controls, that organizations must implement to manage incidents using a NIMS approach.

Incident Management Today

Obviously, putting out forest fires and responding to terrorist attacks is pretty different from dealing with data center failures or a buggy application deployment. ICS and NIMS weren’t designed for site reliability engineering or IT teams specifically.

Still, the influence of ICS and NIMS on the way SREs think is clear enough. Terminology like “incident commander” comes from these frameworks. So do concepts like shared accountability for incident response processes and the importance of involving all stakeholders—not just technical teams—in incident response.

ICS and NIMS may not be acronyms familiar to most SREs. But they should be, because they are the historical sources of the incident management philosophies that form the foundation for SRE work today and offer valuable lessons for any SRE on the job today.
February 8, 2022
Verica Launches Database for Sharing IT Incident Reports

Verica, a provider of an incident management platform, announced this week it is making available a database to make incident reports publicly available via a single repository.

Courtney Nash, senior research analyst at Verica, said the Verica Open Incident Database (VOID) initiative is part of an effort to enable organizations to learn from outages and failures caused by software. Today, reports about such incidents tend to be difficult to find because they are scattered all across the internet, she added.

That lack of transparency makes it difficult to compare and contrast incidents in a way that surfaces meaningful insights, noted Nash.

Launch partners for the VOID include Indeed, NS1, Adaptive Capacity Labs, Security Scorecard and Auxon. Each will provide support to help fund the development, research and dissemination of knowledge generated by the project.

The VOID contains more than just standard company postmortems or status updates, added Nash. Verica is committed to applying metadata to better understand how individuals, companies, media and others reacted during an incident.

One early finding already derived from this initiative is that mean-time-to-resolve (MTTR) is proving to be an inaccurate metric due to the varied distribution of incident data. Other findings include the fact that only one-quarter of incident reports took the time to identify the root cause of an issue, while less than 1% took the time to study incident reports of near-misses.

In general, there’s now more riding on the performance and availability of IT environments than ever. Digital business transformation became all the rage in the wake of the economic downturn brought on by the COVD-19 pandemic. As the economy continues to recover, businesses now expect IT to respond and pivot to changing conditions no matter how sudden.

DevOps teams have, of course, learned to expect the unexpected by embracing modern incident management based on DevOps best practices that enable IT teams to adroitly respond to any outage or sudden degradation in application performance. The most important thing to remember about incident management is that practice makes perfect. While practice runs may never precisely mirror an actual crisis, organizations that have processes that are well-known by their IT incident teams and that practice ‘fire drills’ tend to respond both faster and better. After all, the more muscle memory there is to call on the more resilient the organization will be.

It’s not clear to what degree IT organizations will be willing to share incident reports. However, there is an opportunity for IT professionals to come together that is similar to how engineers from different airlines collaborated to make flying safer for the general public, said Nash. In fact, with so many processes now dependent on software, it’s amazing there have not been more catastrophic events due to software issues. It’s even arguable that it’s only a matter of time before such levels of collaboration are mandated.

In the meantime, the first step toward achieving that goal may simply be sharing a report with colleagues so they can learn from previous mistakes.

October 7, 2021
Choosing an Incident Management Platform

When you’re feeling the stress and pain of manually managing incidents and incident response, making the decision to find an incident management tool is a no-brainer. But how do you choose the one that will work best for you, your team and your business?

You might be asking yourself, “Where do I start? What do I need to know? What questions do I ask? What are the options? How can I be sure we’re choosing the right tool?”

To help identify the most appropriate solution for your organization, here are the top four things to consider.

1. Do you have the culture and processes you need to support an incident management solution?

Different business models require different approaches to incident management. You should be looking for a platform to fit the unique needs of your organization. More importantly, even if you find a solution that seems like a fit, do you have the people, culture and processes necessary to support a new tool? And if not, do you have a plan to put them in place?

Implementing solutions successfully takes planning, so you’ll want to make sure that you and your team can handle the workload.

2. What are your pain points and goals?

You’re looking for an incident management solution for a reason, but do you know exactly which pain points you hope to solve? Your top goal may be to “make it easier,” but every solution is likely to tout that they make things easier. Determine what you’re hoping to accomplish. Do you want to automate processes? Bring your team together more quickly? Track every action from declaration to retro? All of the above? If you prioritize the problems you want to solve, it’ll be easier to check off boxes when you investigate your top choices.

3. Who should be involved in this search?

You may be the one taking the lead on searching for a better way to manage incidents, but knowing who else should be part of the decision-making process will make the search more efficient. You’ll need to know if you have support from your team and from leadership; who’ll authorize the budget for your new tool and who has to sign off on your decision.

And don’t forget other stakeholders outside of your team! Incidents impact all parts of an organization, so consider who needs to be a part of the selection process—from your client success team to your sales team.

4. How will you make a decision?

Finding and implementing an incident management solution requires time and resources. It’s especially important to account for the time it will take to onboard your team and configure your solution, including assigning individual roles, setting up integrations with your existing tools and test-driving your incident management process.

The worst-case scenario is that you purchase a platform but don’t have time to implement and use it. You won’t be happy, nor will your team or executive stakeholders.

So have a clear idea of what you need to make a final decision—whether it’s considering who’s involved, how much budget you need to secure, what your timeline is or how you’ll address any legal or security issues.

August 12, 2021
5 Ways to Reduce DevOps Toil

Over the last several years, DevOps has become a bit of a buzzword. It has become simultaneously a practice, a culture, a team, a job title and a vendor product. You can hire some DevOps, buy some DevOps, adopt DevOps and sprinkle a little bit of DevOps on top for good measure.

But, at its core, DevOps is about service ownership: having a single team with end-to-end ownership of software running in production. You should not have one team building software and another team deploying and operating it. That’s slow. DevOps is literally combining development and operations into a single consolidated team. DevOps means embracing the mantra of “you build it, you own it.”

The primary benefit of DevOps is increased speed and agility of engineering teams. With DevOps, exactly one team is responsible for a given piece of production software. There are no handoffs or knowledge silos. Practices like continuous delivery can emerge. Features can be put in front of your customers’ eyes with a shorter turnaround time. A secondary benefit is increased reliability and security. With “you build it, you own it,” it is no longer someone else’s job to ensure that software is correct or performant or reliable or secure. These responsibilities are “shifted left” onto exactly one team.

So, where do things go off the rails?

Service ownership and DevOps mean giving teams the agency to make whatever changes are necessary to their software. That can and will come at the expense of product functionality at times. So, the first place things go off the rails is when there is not a healthy balance between product needs and engineering needs. If teams are constantly working on new features but do not have the capacity to fix operational issues that are affecting them, product quality will eventually suffer. Ultimately, product-engineering imbalance generally comes from misaligned incentives or poorly defined goals from management.

The second place where things go off the rails is insufficient training. Especially with organizations early in their journey to adopting DevOps, which may have separate development and operations teams, there is a lot of training on new technologies and processes. Developers need to learn about observability, responding to incidents and debugging production applications. Operators and sysadmins need to learn about coding and the principles of software design. These are new skills for both camps and investment in training and mentorship is required.

What are some best practices for keeping DevOps on track?

There are lots of these best practices. It really depends on your starting point as an engineering organization, but here are five important ones:

1. Track Service Ownership

If DevOps is about “you build it, you own it,” you need an authoritative list of a) what’s running in production and b) which team owns it. If a piece of software goes down, you do not want to find out that the last person to work on it was the intern who left three years ago. Many folks start tracking this kind of information in spreadsheets or wikis, but keeping the data up-to-date becomes challenging. SaaS service catalogs have emerged as the easiest way to automatically keep this data accurate, complete and consistent.

2. Adopt Continuous Integration, Continuous Delivery and Feature Flags

Committing small changes multiple times per day is superior to release trains where one large change is committed on an infrequent basis. CI/CD allows for new features and improvements to be tested faster and put in front of customers more quickly. Feature flag tools let you easily test new functionality and product hypotheses for different segments of customers.

3. Embrace Incident Management

Software breaks sometimes. Having a robust process for how to triage and respond to both alerts and incidents is important. Within the context of DevOps, the team that owns a piece of software should be responsible for being on-call to respond to associated incidents. If the software breaks or behaves in some unexpected way, the team is expected to fix it. The team is also responsible for identifying what happened and investing in ways to ensure the same error does not occur again.

4. Treat Everything as Code, Then Automate all the Things

Applications and microservices? That’s code. Infrastructure? That’s code, too. Security policies? Also code. Deployments? You guessed it–code.

Treating everything as code allows for processes like “code review” to emerge beyond just your applications. You can have multiple eyes on every change, which catches bugs sooner and spreads knowledge to others. As well, version control systems (like Git) are a natural audit log, allowing anyone to see what was changed, when, and by whom.

With everything defined as code, the next step is to invest in automation. How many steps does it take for a developer to deploy new code to production? Or to spin up new infrastructure? Or to hook up monitoring or error tracking tools?

Your goal should be “one step:” run a command. It can be a shell script, a Slack bot or a button with a nice web UI. But it should always be one step. If there are more, you’re now in “manual steps” territory, and that’s brittle. You have no guarantees that successive individuals will run whatever sequence of steps in the same order, which can lead to divergence in how various parts of production are set up.

Another example is maintaining a list of a) what’s running in production and b) who owns it. Manually keeping that list up-to-date is impossible as the size of your production infrastructure or engineering team scales. You’re better off investing in service catalog tooling that can automatically integrate and capture this information.

5. Invest in Reducing Toil

Service ownership and DevOps are two-way streets between management and their engineering teams. On one hand, consolidated ownership will lead to faster delivery and more reliable software. That’s good for management. But, on the other hand, teams require actual empowerment and agency to make decisions about their software. Part of that is having capacity on their roadmaps to make improvements to issues that are causing toil.

For example, if a particular service is paging a lot or experiencing load issues, service ownership means giving the owning team the time to make any necessary changes. That will come at the short-term expense of new production functionality.

There’s no free lunch on this last point. Engineering management can’t just combine dev and ops responsibilities and magically expect results. Service ownership is a two-way street. Otherwise, toil will kill your teams in the long term.

July 29, 2021
Why You Should Embrace Incidents and Ditch MTTR

The cliché is that everyone in IT hates incidents, and the natural reaction when assembling incident response metrics is to look for numbers that you can lower over time. Fewer incidents and shorter incident response times must be better, we think. You might already be familiar with the common metrics associated with these goals, including mean time-to-resolution (MTTR) and total incident count.

As it turns out, however, these metrics actually work against your company’s resilience goals. To understand why, let’s look at the industry that birthed most of our modern incident management practices and has learned the most about how to properly apply them: the airline industry.

What the Airlines Can Teach Us

From the 1950s to the 1990s, the airline industry built and refined best practices for incident management that we have since simplified and that have been widely adopted across the tech industry. What’s often termed “modern incident management” simply means the predictable and repeatable approach to handling unexpected disasters and the learnings-driven remediation steps that are prized when building a DevOps or SRE practice.

One of the most important findings to come out of this research is the somewhat surprising fact that incidents weren’t being declared often enough. Instead, incident reports were created and the carefully assembled response and cataloging process was only followed after catastrophic events, which meant earlier opportunities to catch the underlying problem during a less severe incident were being missed.

So why did an industry with such incredibly high stakes take so long to realize that more incidents were actually a good thing? You guessed it: a misguided desire to decrease the incident count. In chasing this goal of “lowering the number,” pilots and mechanics were often reluctant to file incident reports for anything less than the most critical situations, which meant the powerful, learnings-focused response and remediation process was only triggered in response to relatively rare and critical situations.

This is a textbook example of Goodhart’s Law in action. To paraphrase, Charles Goodhart posited that any metric created will ultimately become the goal. In other words, if the incentives are aligned to reduce incidents, no matter how good your response and learning process is, it will only be executed in the most dire and unavoidable circumstances. After discovering this, the industry rapidly moved their focus towards expanding the use of the incident process by working to expand the spectrum of situations that could be classified as incidents with the goal of triggering the formal incident response process more often, distributing learnings for more incidents of lower relative severity and, yes, ultimately resetting the goal to increase the number of incidents.

This approach effectively turned Goodhart’s Law to the airlines’ advantage—any effort to game the metric(s) meant more learning, not less. Filing incidents became directly and heavily incentivized and these metrics were never used as a justification for punitive action. This resulted in the safest decade of commercial aviation on record, with over 12 billion passengers traveling and no fatal accidents.

What IT Can Learn About Successful Incident Response

As technology companies, there is much for us to learn from this exercise in achieving resilience versus chasing “lower numbers.” The primary learning is this: increasing your incident count by incentivizing the use of your incident management process for more non-critical incidents—likely your SEV2s and below—and focusing your metrics and incentives on that goal, is likely the best way to reduce your overall count of catastrophic failures.

For most companies, this means putting MTTR on the back burner and instead focusing on making the incident process itself more accessible throughout your company and across your organizations. It means leaders should talk about impactful and recent postmortems publicly and that postmortem distribution should be a priority; it should also be something that’s measured. Most importantly, it means opening up the incident management process more widely. Ideally, everyone at your company should have access to systems that allow them to file incidents, read incidents in progress and review postmortems from previous incidents.

I like to reference the “big red button” philosophy of factory floors, where the decision-making power to declare an incident is in the hands of every employee. An unsafe situation shouldn’t be up for discussion, debate or approval before being acted on, even if the situation is later determined to be less severe than originally thought. The context of why that incident was filed—why that big red button was pressed—may prevent larger, more catastrophic failures in the future.

So if you’re just getting started building out an incident management practice at your company, or you’ve been chasing MTTR metrics and focusing on pushing your incident count down as a primary goal, consider taking these learnings to heart and creating a positive incident culture with positive metrics meant to encourage and reward incident responders and reporters. Your company’s future depends on it.

July 16, 2021
Best Practices for Cloud Incident Response

Cloud computing is now mainstream, with almost all organizations running at least some resources in the public cloud—whether software-as-a-service (SaaS), platform-as-a-service (PaaS) or infrastructure-as-a-service (IaaS). Security teams have been scrambling to adapt to cloud environments, and with the growing adoption of DevSecOps, they are working together with DevOps teams to secure cloud systems from the ground up.

As your organization discovers the best way to secure its cloud investments, you also need to develop an incident response strategy for the cloud. Even if your cloud security controls are perfect (and they aren’t), attacks will happen. Knowing what to do during an attack and preparing teams for incident response can be the difference between an incident quickly contained and resolved, and a multimillion dollar disaster.

What Is Incident Response?

Incident response enables organizations to make sure they are aware of security incidents and can respond in time to limit the damage to their systems. The objective is to block attacks and prevent similar attacks in the future.

The SANS Institute’s six-step incident response process provides a structured framework for security incidents. These steps are:

Prepare—establish security policies, carry out risk assessments, determine which assets are sensitive and establish an incident response team.
Identify—monitor your systems to detect anomalous activity, identify real security incidents and investigate the severity and type of threats.
Contain—conduct short-term containment procedures to stop the spread of the threat, followed by long-term containment, such as applying temporary fixes and rerunning a clean system.
Eradicate—identify the root cause of the incident, remove malware and implement measures to prevent future attacks.
Recover—restore your production systems and apply measures for preventing further attacks. Test and monitor recovered systems.
Learn—perform retrospective analysis within two weeks of the incident with complete documentation evaluating containment efforts and determine how you can improve the incident response process.

How to Prepare Your Cloud for Incident Response

There are several ways you can prepare your incident response team and your cloud environment for more effective incident response:

Establish response goals—determine incident response objectives in consultation with stakeholders, legal advisors and organizational leaders. Common goals include problem control and mitigation, restoration of affected resources, and storage of data for evidence and attribution.
Use the cloud to respond—ensure your cloud resources include the tools and resources you need to respond to an incident. For example, ensure you have robust cloud-based logging and monitoring systems, and set up cloud-based backup and disaster recovery so you can rapidly restore affected systems.
Determine your requirements—keep copies of logs, snapshots and other evidence in a centralized cloud account. Apply mechanisms to enforce retention policies. Use tags and metadata to maintain visibility and connect logs and cloud resources to organizational units, projects or corporate systems.
Use a redeployment mechanism—if a security anomaly is caused by a misconfiguration, you should be able to remediate it easily by redeploying the resource with the appropriate configuration. Ensure the response mechanisms can be executed multiple times if necessary.
Leverage automation—after identifying recurring problems and incidents, automate as much as possible and break them down programmatically to build response mechanisms for common situations. For example, use the mature auto scaling service on AWS or Microsoft Azure’s infrastructure-as-code (IaC) capabilities. This is much easier on the cloud than it was in an on-premises data center. Ensure you use human response only for unique, new or critical incidents.

Effective Incident Response in the Cloud

Use the following tips to improve your ability to respond to security incidents in a public cloud environment.

Shift Your Focus

Cloud environments require you to monitor different elements than you would in traditional on-premises environments. In the cloud, you should focus on applications, APIs and user roles. Consider how incident responders can operate successfully in a cloud environment, and what tasks they may need to perform. The incident response team must have proper access and visibility into your systems so they can detect, remediate and prevent attacks.

Integrate Alerting and Incident Management Tools

The security team must have direct access to supporting data to triage alerts and classify incidents. To this end, security alerting tools should be integrated with any incident management tools you use, like PagerDuty and Slack, for example. This enables security alerts to directly reach the existing tools and workflows used by your teams. Responders won’t have to alternate between tools to see what is happening.

Build an audit trail to capture the response to each alert, which will provide visibility and accountability and help you refine your response processes. All actions taken in security tools must be visible in the relevant collaboration tool, so you can see who dismissed a specific alert and when, and what annotations they made.

Work with Your Cloud Provider

Cloud providers usually have an incident response team, but you cannot assume the vendor will handle everything during an event. Be aware of the shared responsibility model, where cloud providers are responsible for securing the infrastructure, while the customer is responsible for data and workloads.

Make sure you understand the service agreement for your cloud provider and who is responsible for exactly which element of the response. Find out exactly what alerts you can expect from the vendor’s team and how it can support your own team. Having a clear relationship and establishing points of contact can save critical time during an incident.

Protect Your Logs

The major cloud vendors provide logging capabilities for their environments, including log files or operational metrics to provide insight into service operations. Logging services may be free or paid, ranging from basic access logs to complete audit and configuration logs. Most cloud logging service will allow you to store logs outside the cloud or on-premises—and it is critical to do so.

Logs are a useful resource for incident response investigation, and you must ensure they remain inaccessible to attackers. An attacker might be able to compromise your cloud system or services, but they won’t be able to modify or delete your logs. Logs are a protected source of information that can help you identify the attack timeline, targeted systems and the attacker’s IP address. This provides a reliable starting point for investigations.

Conduct Cyber Range Training

Organizations often rely on exercises to train or test their security and incident response capabilities. Today, cloud environments provide an opportunity to simulate your production network in a protected environment, allowing your security team to practice their response to real attacks on your network within a safe setting.

Tools such as AWS CloudFormation allow you to quickly design and deploy training networks that are identical to your actual network. You can keep costs down by limiting the duration of exercises. These exercises may be the best way to prepare your team to respond to attacks in the real world.

These are the basics of security incident response, preparing a cloud environment for incident response and how teams can effectively react when the inevitable attack occurs. In short, it’s important to:

Focus on what matters – In the cloud, this is APIs, applications, and identity and access management (IAM) systems.
Integrate alerting and incident management tools – The cloud provides ample automation capabilities. Use them to respond automatically to common anomalies.
Work with your cloud provider – You are not alone in the cloud, and teams need to understand exactly which part cloud providers will take in responding to an incident.
Protect your logs – If logs are exposed to tampering, you will have no way to detect, investigate and respond to attacks. Protect them at all costs.
Conduct cyber range training – You’ll never really know what it’s like to respond to an incident until one actually happens. Instead of waiting for a real attack, conduct a “cyber range training” or security drill and see how everyone works together in an attack scenario.

Now, you’ll be better prepared as you move towards a cohesive cloud security strategy for developers, operations, and security teams.

June 29, 2021
PagerDuty Allies with JFrog to Modernize IT Incident Management

PagerDuty has integrated its IT incident management platform with the continuous integration/continuous delivery (CI/CD) platform from JFrog as part of an effort to streamline troubleshooting of application environments.

Steve Gross, senior director of strategic business development at PagerDuty, said the overall goal is to make it easier for IT operations teams and developers to collaboratively discover the root cause of issues by, for example, being able to identify builds within a JFrog pipeline that might be having a negative impact on a production environment.

At the same time, IT teams employing PagerDuty can now also receive notification from JFrog Xray, a tool that discovers both vulnerabilities in open source software and license compliance issues.

Organizations of all sizes are attempting to minimize the impact of an IT incident by providing deeper insights into how applications were constructed. The goal is to reduce the amount of time IT teams currently spend looking for the root cause of a problem.

Ideally, IT teams would discover those issue before they impact end users. The challenge they face is it can take weeks to discover the root cause of an issue that takes a developer a few hours, at most, to fix. Armed with insights from the CI/CD platform, Gross said it becomes easier for IT operations teams to proactively pinpoint the source of an issue in a way that provides the developer with enough insight to act.

That is critical because, more often than not, it’s the last build of application that was deployed in a production environment that is usually the source of an issue, especially when that IT environment was operating normally before that build was promoted into a production environment.

The data collected by PagerDuty from various DevOps platforms is fed into the PagerDuty tools via a Change Events application programming interface (API). Over time PagerDuty will also be able to apply machine learning algorithms and other forms of artificial intelligence (AI) to identify issues long before they reach a meaningful level of disruption, noted Gross.

In general, there’s a movement to modernize IT incident management in a way that aligns with DevOps best practices as part of an effort to identify issues before an application is actually deployed. The hope is that those efforts will reduce the number of “war room” meetings that need to be convened to determine the root cause of a performance or security issue after an application has been deployed.

It may be a while before IT incident management processes are modernized to that level. However, as IT environments become more complex—in part because of the number of microservices-based applications with multiple dependencies that are now being deployed—it’s apparent existing approaches to IT incident management don’t allow IT teams to respond fast enough to an issue. In fact, that’s more challenging than ever now that many of the members of an IT team are working from home more often.

The pressure to modernize IT incident management increases in almost direct proportion to IT teams’ stress levels. The challenge now is to modernize those processes before the members of an IT team are too burned out to appreciate the effort.

June 17, 2021
Three Ways Machine Learning Can Change Incident Management

Incident management for important applications has many connotations, many of them negative. Most are well-deserved. Stress, panic, hurry, urgent, drop-everything, all-hands-on-deck, emergency, to name a few. Typically, there is a critical problem, ranging from complete failure to loss of certain functionality or performance. The problem or outage potentially means loss of revenue, loss of customers, negative social media commentary or bad reviews and more. Time is of the essence, dictating that the primary objective is fixing the incident as quickly as possible. Often, this means that symptoms or partial causes are mainly addressed rather than the underlying root cause.

Without fully addressing the underlying issue, companies virtually guarantee that the same problem – or a similar one – will reoccur. Not identifying root cause often prevents a durable fix. In addition, companies lose the opportunity to proactively improve application code or infrastructure based on real-world experiences and issues. Postmortems may only result in reviews of monitoring and observability solutions and the inevitable updates to alert rules. Most DevOps professionals not only understand but have lived through these frustrations on an ongoing basis. Management, then, often wonders why their systems are so unstable.

Changing the model for incident management has been limited by a combination of the overriding urgency combined with short-staffed, overworked teams. Although AI and machine learning has been positioned as the panacea for nearly every kind of technical ill, this is a clear case where “machines” could fundamentally enhance human efforts to improve a situation. The best troubleshooters exhibit a combination of instinct, experience and patience to carefully sift through reams of data, spotting unusual events and their correlation with bad outcomes. This turns out to be a perfect application for machine learning.

By ingesting all logs and metrics to find meaningful problems or anomalies, systematized machine learning can quickly and comprehensively sift through all data to gain a thorough understanding of the real problem. Using this approach significantly speeds up the root cause analysis. Such an applied solution introduces three game-changers for incident management.

Three Game-Changers for Incident Management

First, incident remediation times can be reduced dramatically and with less of an all-hands-on deck, drop-everything frenzy. Rather than SREs and engineers spending hours digging through dashboards, traces and millions of log lines, an automated approach can accomplish it more quickly, more thoroughly and with less impact on human professionals, allowing them to focus on other responsibilities. This instantly relieves stress and panic and alleviates pressure because incidents can be remedied better and faster, restoring the application to its expected service level. The value is, of course, two-fold: faster time to resolution and greater efficiency or productivity of the teams who would otherwise be involved. Ending or mitigating the frustration professionals have to deal with is an additional bonus.

Second, besides faster MTTR, an automated, machine-learning-driven approach is more comprehensive and gets to the root problem. This means that future incidents may be avoided altogether. This also means a potential end to the alert rule “hamster wheel.” The knee-jerk tendency to continually adjust alert rules and settings to enable better “early warning systems” is far less necessary. Instead of trying to gain an early indication of a similar failure in the future, the cause of such failures can be found and addressed proactively. Eliminating such problems in the first place is far better than trying to evolve alerts to catch unresolved causes. Alerts are still important, but machine learning can shift the basis for them to more root-level issues, and they can be constructed automatically by the system.

Third, automated, machine-learning-based incident management can help proactively catch silent bugs that may not yet directly be part of an incident. It can also inform developers earlier in the cycle. Not terribly long ago, new releases were tested extensively before deploying to production, allowing for the creation of specific test plans, conducting stress tests and potentially catching bugs that might have potentially terrible consequences in the future. Now with faster, non-stop development cycles, testing mainly occurs during production. Machine learning can surface new or unusual errors, event patterns and patterns in metrics. The approach can proactively surface subtle bugs early, before they impact users or cause widespread failure.

Changing the DevOps Life Cycle

Each of these three results significantly changes not only the incident management life cycle, but also DevOps as a whole. Poring over logs and metrics is a perfect application for machine learning and offers the opportunity to fundamentally change incident management, transforming it from a scourge and necessary burden to a competency. An automated system of machine learning applied to incident management advances DevOps rather than encumbers it.

May 21, 2021
Using RPA for DevOps Process Automation

Modern operations teams today are at the cutting edge of automation. DevOps and SRE practices are entirely based on an automation-first philosophy. CI/CD is fully automated and the next generation of automated testing is now being adopted.

However, these same teams are still running their human processes based on hastily thrown-together wiki pages. The increase in remote work over the last year has amplified the problem. Engineering teams that were used to quickly getting the right people physically into a war room now struggle to find the people with the right knowledge to troubleshoot or resolve a problem.

Teams with ample resources have built tools and bots for DevOps process automation and orchestration. These tools can help automate things like getting communications channels set up quickly, with the right people, the moment an incident is identified. Or, they may automate actions across the DevOps toolchain that would otherwise live as static documentation for an on-call engineer to try to decipher and execute. But only a handful of organizations have the resources to build this kind of automation technology in-house.

These higher-level DevOps processes are ripe for innovation similar to robotic process automation (RPA). Companies like UiPath and Automation Anywhere have shown that there’s an enormous market for this in every business unit and industry. Process automation for DevOps needs to be another layer of automation available to any engineering team. Automated remediation of incidents and resolution of service desk tickets has to be the future of this space. However, there are important differences in how automation is optimized for DevOps versus the application of RPA for business functions.

Where RPA Fits on the Automation Spectrum

RPA is great for solving deterministic problems – those that can be resolved in the same way every time you run into them – which are fully automatable. An example of this type of end-to-end automation is any back-office task, which are consistently good candidates for the application of automation. It’s the reason why fields like HR and finance have made some of the greatest progress in automating business processes to date. Every time you close the books, for example, the steps are pretty similar.

Process automation for DevOps and SRE, however, is on the other end of the spectrum. Although teams may benefit from RPA for simplistic, repeatable tasks, many DevOps processes are dynamic and unpredictable, making full end-to-end automation too often fallible. The complexity of modern technology stacks and infrastructure requires processes to be agile and adaptable to change, rather than deterministic. This is where it’s important to enable humans to step into the spectrum of automation. Human-in-the-loop automation – the idea that lets humans be in the cockpit while letting machines provide guidance and support – is the future of DevOps automation.

Applying Process Automation to DevOps

To use automation in a meaningful way, DevOps teams need a mix of what RPA provides with the added element of human judgment. Human-in-the-loop automation can be applied to numerous DevOps processes including incident management, service requests and continuous deployment and delivery.

Rather than automating from end-to-end right from the start, DevOps teams should use incremental automation to strengthen and build trust in automation. This starts by codifying free-form processes so teams can more easily identify predictable steps that can be automated. Then teams can mature to more complex workflows, automating pieces within processes and incorporating human checkpoints for areas that will need judgment or approvals.

As teams incrementally automate, they build up sustainability – in their systems and their roles – in high-stress environments and are able to orchestrate processes across the continuum. The goal is to find the right balance between leveraging machines to make lives easier while creating room for human judgment and adaptability.

Level-Up DevOps Automation

Looking ahead to the next few years, what the RPA movement did for back office tasks will also happen for DevOps and SRE. Technology leaders can take notes from the RPA movement and apply them to eliminating the toil of modern production operations. The issue is that, in DevOps, many problems are unique.

The next evolution of DevOps automation will force us to look at how we can model the real world in a more holistic way, which includes incorporating human judgement and manual execution of non-repeatable tasks while automating repetitive steps.

May 6, 2021
Blips and Downtime: Incident Management 101
Nothing gets the blood pumping like seeing an IT incident alert flash across the screen. Is the network traffic anomaly a potential intrusion? Are we about to have an outage? Is it just a random blip on the radar? The implications of these different potential incidents vary wildly, and each requires a different set of response tactics.

Responding to incidents both large and small is essential for every team. Especially in a remote-first world, it is more important than ever to ensure that all developers are in the same war room to tackle the problem as efficiently and effectively as possible. It’s also critical to have a single view that aggregates data from all of the systems and services used to monitor and respond to incidents. But not every team has invested in a fully fledged incident management strategy.

In fact, it’s not uncommon for incident management to be completely ad hoc. Teams wait for something to happen and then take a “dive and catch” approach to incidents. Some teams have some processes in place, but those plans may be missing key components. In both cases, teams struggle with slower time to resolution and often waste resources in the process of responding to incidents. Critical stakeholders often suffer from a lack of clarity in communication around an incident.

Depending on the type of incident, the consequences can be challenging to overcome. Running afoul of a compliance law, like GDPR, can result in business-ending financial penalties. Incidents that impact uptime can keep an organization from meeting SLA terms and potentially losing customers and revenue. Regardless of the size of your company or customer base, formulating an effective incident response begins with people.

People are the Cornerstone of Incident Response

Companies should assess their business needs and the risk profile of their applications and services. With those barometers in place, leaders can then determine the appropriate staff levels to ensure 24/7/365 incident management coverage. The size and breadth of the team will depend on the size of the company and the structure of response plans, but the team should attempt to cover a few areas.

Ideally, a team has at least three responders.
1. Incident owner. This is the team’s quarterback. They coordinate everything, make sure the team follows runbooks, involves the correct people and removes roadblocks.
2. Incident investigator. The investigator has their hands on the keyboard. They run the investigation, pull and analyze logs, review metrics and generally try to determine how to best mitigate the issue.
3. Incident communicator. Smaller teams may not have the personnel to have a designated communicator in every incident. On those teams, the incident owner will often assume this role, as well. Communicators focus almost exclusively on two tasks: documenting the incident via an “incident timeline” and communicating with stakeholders when appropriate. It’s critical that a complete timeline is built during the incident. It’s nearly impossible to put together a properly documented incident timeline after the fact. Critical details will get lost or forgotten, and your retrospectives and RCAs will be low quality.
Runbooks offer Muscle Memory During Incident Management

The incident response team, no matter how experienced, will need a set of runbooks—a compilation of processes and steps to take when incidents arise. Runbooks will help guide teams as they perform everything from analyzing and mitigating common problems, to post-action investigations. They represent the process portion of incident responses, and act as a guidebook for the team when the pressure is high.

Typically, an alert system will ping the on-call engineer about a metric that doesn’t look right. The runbook will help that on-call person determine if the alert should be categorized as an incident or not. When that person determines the alert indeed represents an incident, they must estimate the severity of the incident. Different teams may use different definitions for severity, but here is an example:

Severity one – Production down or highly degraded.
Severity two – Serious bug impacting customers or system stability.
Severity three – Important bug. Requires tracking until mitigation, but likely not a 24/7 workstream.

Runbooks can also guide teams through analysis of the situation. After a root cause is determined, the book will remind teams to perform an immediate post-action investigation to see if specific processes worked well, if gaps exist, and if communications were sent out to customers or other stakeholders.

Tools are Important; Tying Them all Together is Crucial

People and processes are important for responsive incident management. Without the right tools, even seasoned incident-management veterans may struggle with the complexity and pressure of an incident. Tooling can prevent a lot of mistakes and omissions when the heat is on.

There are numerous platforms that address different components of incident management response. Depending on the size of the organization, teams will need some or all of the following:
- On-call rotation management
- Observability and alerting
- Ticketing system
- Communication system
One of the biggest challenges in incident management is having the ability to tie together the data and actions defined by this broad toolset. If you’re working a severity level one incident, teams will likely rotate people into and out of the incident based on their geographic location and timezone. How does the next person in the rotation come up to speed on what’s been done so far? How do we document a detailed timeline that can be used for an RCA investigation later? How can we provide an up-to-the-minute view of the status of an incident for key stakeholders? Ideally, teams will have a single place where all of this data is collected and managed and where the data can be filtered and exported at the end of the incident.

Effective Incident Management: People, Processes and Tools

Is that traffic spike a sign of impending doom, or just another meaningless bump? Without the proper people, processes and tooling, answering that question becomes very difficult. Unfortunately, the wrong answer can lead to disastrous consequences. Only with the right set of tools and processes can companies develop an incident response system that provides maximum stability and keeps customers happy.
May 4, 2021