devopsy.kinsta.cloud

Tag: incident

  • What SREs Can Learn From the Atlassian Outage of 2022

    What SREs Can Learn From the Atlassian Outage of 2022

    What happens when the tools and services you depend on to drive site reliability engineering turns out to be susceptible to reliability failures of their own? That’s the question teams at about 400 businesses presumably asked themselves in the wake of a major outage in Atlassian Cloud. The incident offers a number of insights for SREs about reliability risks within reliability management software itself—as well as how to work through complex outages efficiently and transparently.

    What Caused the Atlassian Cloud Outage?

    The outage, which began on April 4 and finally resolved April 18, affected about 400 Atlassian Cloud customer accounts. Atlassian Cloud is a hosted suite of popular Atlassian products including Jira and OpsGenie. The outage meant that affected customers could no longer access these tools or the data they managed in them.

    According to Atlassian, the problem was triggered by a faulty application migration process. Engineers wrote a script to deactivate an obsolete version of an application. However, due to what Atlassian called a “communication gap” between teams, the script was written in such a way that it deactivated all Atlassian Cloud products, not just the obsolete application.

    To make matters worse, the script was apparently configured to delete data permanently, rather than mark it for deletion, which was the intention. As a result, data in affected accounts was removed permanently from production environments.

    The Good, the Bad and the Ugly

    The Atlassian Cloud outage may not be the very worst type of incident imaginable—failures like Facebook’s 2021 outage were arguably worse because they affected more people and because service restoration was complicated by physical access issues—but it was still pretty bad. Production data was permanently deleted and hundreds of enterprise customers experienced total service disruptions that have lasted several days and counting.

    Given the seriousness of the incident, it’s tempting to point fingers at Atlassian engineers for letting an incident like this happen in the first place. They seem to have written a script with some serious issues, then presumably deployed it without testing it first—which is exactly the opposite of what you might call an SRE best practice.

    On the other hand, Atlassian deserves credit for responding to the incident efficiently and transparently. Although the company was silent at first, it ultimately shared details about what happened and why even though those details were a bit embarrassing to its engineers.

    Crucially, Atlassian also had backups and failover environments in place, which it used to speed the recovery process. The major reason why the outage lasted so long, the company said, is that restoring data from backups to production requires integrating backup data for individual customers into storage that is shared by multiple customers, a tedious process that Atlassian apparently can’t perform automatically (or doesn’t want to, presumably because it would be too risky to automate).

    Unfortunately for impacted customers, it does not appear that any fallback tools or services were made available while they waited for Atlassian to restore operations. We imagine this poses more than minor issues for teams that rely on tools like Jira to manage projects and OpsGenie to handle incidents. Perhaps those teams stood up alternative tools in the meantime – or perhaps they crossed their fingers, hoping their project and reliability management tools would come back online ASAP. 

    Takeaways for SREs From the Atlassian Outage

    For SREs, then, the key takeaways from this incident would seem to be:

    • Always perform dry runs of migration processes in testing environments before putting them into production. Presumably, if Atlassian engineers had tested their application migration script first, they would have noticed its flaws before it took out live customer environments.
    • Back up, back up, back up—and make sure you have failover environments where you can rebuild failed services based on backups. While this outage is bad, it would be 100 times worse if Atlassian couldn’t restore service based on backups and data had been lost permanently.
    • Ideally, each customer’s data should be stored separately. As we noted above, the fact that Atlassian used shared storage seems to have been a factor in delaying recovery. That said, it’s hard to fault Atlassian too much on this point; it’s not always practical to isolate data for each user due to the cost and administrative complexity of doing so.
    • SRE teams would do well to think about how they’ll respond if their reliability management software itself goes offline. For example, it might be worth extracting and backing up data from your reliability management tools so you can still access it if your tool provider experiences an incident like this.
    • Over-communicate with your customers often and early. In this case, there was initial radio silence that left customers in the dark. Most of this chatter eventually took place in public forums.

    Conclusion

    The Atlassian Cloud outage is notable both for its length and for the fact that, ironically, it took out software that teams use to help prevent these types of issues from happening at their own businesses.

    The good news is that Atlassian had the necessary resources in place to restore service as quickly as possible. A shared data storage architecture led to slow recovery, which is unfortunate; again, it’s hard to blame Atlassian too much for not setting up dedicated storage for each customer.

  • Leading Effective Incident Response Without Interminable Bridge Calls

    Leading Effective Incident Response Without Interminable Bridge Calls

    There are easier ways to manage incident response without creating war rooms and packing IT staff onto bridge calls

    Your phone vibrates at 11 p.m., and you know that can only mean another major incident with one of the business’ critical systems. You get geared up for the war room, dial into the bridge call and start reviewing the major incident report. You do this instinctively because it’s the third major incident in as many weeks, and you wonder if there’s a faster, easier, less stressful way to keep critical systems running, especially during periods of operational uncertainties.

    You’re not the only one recognizing that having too many major incidents is a significant issue. In a recent survey on the future of monitoring and AIOps, 94% of respondents stated that issue resolution is critical to their business, but only 28% are satisfied with their handling of major incidents. Too many organizations feel that their only path toward improving their incident handling capabilities involves redesigning their critical business applications, facing down their technical debt or building out an SRE practice.

    There are easier ways to manage major incidents without creating war rooms and packing IT staff onto bridge calls. When used well, AIOps techniques can address the pain points in resolving major incidents that drove business leaders and CIOs to establish war rooms and bridge calls in the first place. To understand what AIOps capabilities are needed, it helps to review the history of war rooms and bridge calls and why they are inefficient in solving today’s operational challenges.

    Furthermore, IT organizations can take advantage of AIOps platforms much faster than re-architecting applications, addressing technical debt,or hiring more site reliability engineers.

    How Bridge Calls Became the Status Quo in Incident Response

    No one wants unreliable or poor performing business systems. It’s why IT teams have become extremely proficient at recovering from incidents, especially the ones that are easy to diagnose and easy to resolve. Problems such as web servers going down, databases running out of storage or services stuck in deadlocks are relatively easy to diagnose with today’s monitoring tools. In fact, over the last few years, many IT Ops groups have used tools to automate the recovery from these common issues.

    But more complex issues are harder to solve. Issues such as:

    • Problems causing a cascading failure of dependent systems that are all sending frequent alerts.
    • Issues right after major application deployments or infrastructure changes.
    • Bottlenecks in customer-facing applications that are experiencing unusually heavy loads.

    IT’s history of solving these complex incidents isn’t great. The incident management team responds and calls in help from Tier One support. With so many things going wrong, operational teams have no choice but to call in higher levels of support, including developers. By the time someone communicates to the business on the incident’s status, leaders are irate over the lack of communication, and the time it is taking to recover from the issue.

    CIOs and IT leaders dislike being yelled at and seeing long outages. Their easiest management response is to get all the experts in the room, often called the war room, in the hope that having more people involved is better. War rooms often include bridge calls to allow remote people to attend, and bridge calls are standard practice for major incidents occurring off-hours.

    Bridge Calls Don’t Solve Incident Management Problems Better

    It surely isn’t better for the operational engineers to be responding to off-hour incidents regularly. It’s not better if a seasoned major incident manager is needed to oversee the bridge call and ensure differing opinions don’t escalate into arguments or finger-pointing. It isn’t better if the recovery times miss business objectives and if root causes are never identified. It isn’t better if the resolution times to recurring problems don’t improve.

    It’s also really bad if the number of complex issues is increasing. This is likely to be the case as application architectures based on hybrid cloud or containers and microservices add complexity while frequent releases from DevOps teams add to risk. Also, global events such as COVID-19 create usage uncertainties and network bottlenecks during periods of increased business importance.

    Solving for the Root Cause of Inefficiencies

    Bridge calls and war rooms are inefficient practices to solve complex problems. They require too many people, take too long and require too many more follow-ups to identify the root cause.

    IT has built up an arsenal of tools to better manage myriad operational domains. We’re using one set of monitoring tools for the data center and a separate one for the public cloud. Every database, API, microservice, application component and type of IoT device has different tools to monitor performance. In fact, according to the future of monitoring and AIOps survey, about 20% of respondents reported having 25 or more monitoring tools. In addition, there’s also all the system, network and application log files that often have the most critical information when troubleshooting a complex issue.

    That’s many tools, considerable data and countless alerts to sift through under the pressure of a major incident. These tools often require different skills and subject matter expert participation, which is another reason why bridge calls and war rooms are so crowded.

    Now, AIOps can mean many different things to different people. But if it’s going to improve major incident recoveries without requiring bridge calls or war rooms, then it must:

    • Aggregate all the data from these monitoring tools.
    • Aggregate data from all of your change and topology tools, so you can find out which changes led to an incident or what’s affected by the incident.
    • Sequence the data into a time series showing what issues came first.
    • Correlate the hundreds or even thousands of alerts into discrete events that operators should review.
    • Present the information in an easy console for operators to assess during major incidents.
    • Enable engineers to automate steps in the recovery.
    • Integrate with different workflow, collaboration and communication tools to automate incident response steps.
    • Share the event sequence with engineering and development teams.

    During incidents, the differentiator is that machine learning models have gotten a head start in processing all the data. Instead of dozens of tools requiring lots of people, the start of the incident review can begin by examining information in a single platform. Instead of IT Ops being inundated with too much noisy data and alerts, machine learning has already processed and correlated events into a more easily decipherable storyline. Instead of manual recovery steps, automated steps can be centralized and orchestrated from a single console. Lastly, since the organization’s workflow, collaboration and communication tools are integrated, no one should feel out of the loop on issue status and postmortem steps to make improvements.

    IT teams often celebrate when a legacy system is shut down because operations have fully transitioned to a modernized platform. Isn’t it time to do the same with war rooms and bridge calls?

  • When IT Disaster Strikes, Part 1: Resolving Incidents

    When IT Disaster Strikes, Part 1: Resolving Incidents

    As a developer or operations team member, there is nothing quite like the dread you feel when you hear the familiar ringtone of your on-call page at 3 a.m. Being on call means that you may be contacted at any time to investigate and fix issues that arise for the system, but that doesn’t mean you can’t get ready beforehand. There are steps your team can take to prepare for incidents and streamline the process of resolving them, leading to fewer 3 a.m. wake-up calls and better-running software and services.

    Below is an overview of a proven approach your organization can take before, during and after an IT incident to reduce headaches and increase hours of sleep.

    Before: Define the Who, What, When of Incidents

    The first step in any incident prevention and orchestration process is determining how your organization defines an incident. Generally, this is done by defining the varying degrees of severity, with lower-numbered severities more urgent. Operational issues can be classified at one of these severity levels. As a general rule of thumb, if you are unsure which level an incident is, treat it as more severe to ensure it is dealt with timely.

    In addition to aligning your team on defining an incident to communicate the sense of urgency involved, there are several main roles that should be designated ahead of potential IT issues. Certain roles have only one person per incident, while other roles can have multiple people assigned to it. It’s all about coming together as a team, working the problem and getting a solution quickly. Generally speaking, developer, IT Ops and DevOps teams must designate the following roles:

    • A point person or “Incident Commander”—someone who will drive the process forward, but not be involved in the actual remediation;
    • Someone to document a timeline of an incident as it progresses for future analysis and learning and to act as a backup for the point person;
    • One or more subject matter experts who are deeply familiar with the specified component or service and who will take the remediation steps; and
    • One person to manage customer support and communication during and after an incident.

    During: Keep Calm and Communicate

    If you are alerted of a major incident, don’t panic. The first step is to join a previously agreed-upon method of communication to be used during incidents to ensure that communication can run smoothly throughout the resolution process. The incident commander (IC) and the IC deputy should announce the issue in appropriate communication channels and lead the process, and it is best to defer to the subject matter experts assigned to the incident to ensure non-essential communication is kept to a minimum.

    The steps you take to prepare for an incident have a significant impact on how quickly you are able to move when disaster strikes. If incident prep has been done correctly, each member of the incident response team will have a very specific role and set of responsibilities carved out, ranging from someone to provide regular updates in the chat client to someone to modify your company’s status page to keep customers informed. By having these roles defined beforehand, you don’t have to spend valuable time during an incident figuring them out instead of fixing the problem. In a future article, I will break down each member of the incident response team’s specific role and the steps they should follow during an incident.

    After: Conduct a Post-Mortem Without Finger-Pointing

    For every major incident, you should follow up with a post-mortem—a blame-free, detailed description of exactly what went wrong to cause the incident, along with a list of steps to take to prevent a similar incident from occurring again in the future. The incident response process itself should also be included as part of the review.

    As the IT incidents we deal with daily become increasingly tied to larger organizational success and business objectives, streamlining the resolution process is a must. According to a report from IDC, the average hourly cost of an infrastructure failure is $100,000 per hour, and the average cost of a critical application failure per hour is $500,000 to $1 million. In future articles, I will break down the varying roles on an incident response team, specific steps to follow during an incident, a tried-and-true template for conducting a successful post-mortem and more, to ensure effective incident prevention and resolution.

    About the Author / Eric Sigler

    Eric Sigler is the Head of DevOps at PagerDuty, helping protect its customers from the pains of downtime. Before his current role, Eric led infrastructure teams at Minted, Expensify, and the Missouri University of Science and Technology. Connect with him on Twitter.

  • An outage war room primer

    An outage war room primer

    One aspect of the DevOps movement I’ve seen adopted at numerous companies is the idea that everyone supports their products by being on-call for any incidents that occur in the production environment. This responsibility often leads to participation in the outage war room. For those of you who may be new to this experience, I’m offering this primer from a grey-hair ops guy who’s served countless hours in every manner of war room you could imagine, some that stretch the bounds of that imagination, and a few that I just still can’t talk about. Been there. Done that. Still doing it.

    In all that time, however, I’ve observed  four distinct war room dynamics that can cause conflict. Understanding how that conflict arises is key to mitigating it.

    The war room is a stressful place

    This statement sounds obvious, but it’s fundamental to understanding the tensions that can arise in a war room. The clock is ticking. In fact, the clock has already been ticking for some time when you join the room. These things don’t happen instantaneously. When you get the call to join a war room, the problem has already manifested and is causing an impact. Even with automation, it takes time to notify people and have them connect to the war room, in whatever form that takes. In environments with a very high SLA target, the set up time for the war room alone can exceed the monthly allowance for downtime. So once you get into the war room, it’s serious business. The clock keeps ticking.

    The war room is often a stressful place as a result. Tensions, and even tempers, can run high. But each and every person in the war room has an obligation to keep themselves under control. Tension and stress feed on each other, and if you succumb, it makes it worse for everyone. Even more important, if you get angry, emotional, or display some other decidedly not cool way of coping with the situation, you are likely to lose the confidence of your peers. That really sucks, so it’s important to force yourself to project calm confidence and be deliberate and measured in your actions. If you have the opportunity (and the ability) and can help to disperse some of that tension, do it. Those are the kinds of moments that build credibility.

    And there is another very personal connection to the outage that can be stressful. A professional, worst-case scenario. In the worst case, you are actually responsible for the whole mess. When that happens, you are going to be in your own special hell in the war room. But you should know exactly what is expected of you – the one thing you can do that will really make a difference. You need to tell everyone exactly what you did, without skipping anything, and without regard to how stupid it might sound in hindsight. As in, “I continued to run through all the commands in the procedure even though they were failing with errors and even though I had guidance to stop and seek help if they failed.” To help prop you up in this most difficult of tasks, consider this: everyone else in the room has done the same thing. We’ve all been there. If we haven’t, it’s probably just a matter of time. So you have to swallow your pride, tell the whole story and then get actively involved in fixing things, because that’s what the war room is all about.

    The goal of the war room is to restore service as quickly as possible

    A war room is bad. If you are in one, it probably means that your customers are impacted. You have an obligation, working with the rest of the team, to recover the service in the most effective and least risky way. In a perfect world, that method is also fast, but life doesn’t always work out that way. If there is no obvious and simple solution, though, things can slow down a bit and we see different approaches forming across groups of people in the war room. Some, traditionally the operations staff, want a way to restore service quickly by a reboot, reset or other “big red button” kind of operation. Others feel it’s important to fully understand the nature of the problem before taking action. These are clearly often opposing goals.

    One way this manifests in war rooms is when, in order to further analyze the problem, someone asks to run some experiments, or capture data, or otherwise use the production environment to debug the problem. If there is no method available to recover service with a reasonable level of risk, such experiments may be justified. Otherwise, the team has an obligation to execute whatever procedure they have to bring the system back up, even if that destroys information that might be helpful to the root cause investigation.

    Another thing I see happen frequently is when the someone jumps the gun in trying to fix things. It plays out like this:

    Jane: Do you think we should reboot the server?

    Dick: I’ll get set up for it.

    Jane: I think we should do it.

    Dick: Done!

    Jane: Wait…

    A good war room manager will easily spot and work through these kinds of conflicts. If there is such a person on your war room, listen to them and obey them. The war rooms I’ve participated in are not democracies. They are run by a strong leader who is empowered to make extremely critical decisions. It’s crucial to respect that authority in the heat of the moment rather than try to be the hero. Not so coincidentally, this is my next point.

    We need solutions, not heroes

    In a war room, you need to separate your ego from your idea. The war room does not need heroes. The war room needs solutions. Be advised that no matter what part of the organization you come from, your idea is probably going to go up against some alternative ideas. They will not be judged, generally speaking, on how clever or elegant they are. They will be judged on how effectively they can recover the service and how much risk there is. And if yours is not the chosen idea, it does not mean that you suck. The only people who really suck in a war room are those who interfere with the process of service recovery. You know, the people who interrupt the flow of a critical conversation to ask for a status update, or who won’t stop debating a relatively minor point when the team really needs to focus on a larger issue. That kind of thing. But I digress.

    Sometimes, the right path forward does not actually address the root cause of the problem.

    Sometimes, the right path forward precludes ever figuring out what the root cause is.

    Heroes have a hard time coping with that.

    You must accept that you may never know why

    The incident and problem management processes in most reasonably mature organizations are designed to recover the service, find the root cause and then ensure that it never happens again. But reality is often different than these goals. Incidents sometimes last a lot longer than you like, as when you discover that something has been broken for a long time and you did not know about it. And we should all know by this point that sometimes you never really know why it happened. You just can’t figure it out. It sucks, but again, it doesn’t mean that you suck. It means that the cost of determining the root cause has crossed the point at which it is feasible to continue, given the impact of the problem when it manifests. That’s just common business acumen, but it’s worth stating. Because when you realize that this is a possible outcome, it can give you a better perspective on evaluating your options. Suddenly, restarting the system and walking away from any run-time state information that might be helpful in root cause determination seems more viable. And those are the kinds of trade-offs that everyone in the war room has to come to terms with.

    Conclusion

    It sucks being called into a war room due to an outage. It is the very definition of failure in the SaaS world. But you can make a bad situation into a great and positive experience if you handle it properly. You can take an event that should mean that you suck and turn it into something that demonstrates your excellence. Being an effective member of the war room is a way for you, as an individual contributor, to demonstrate that awesomeness. You do it through calm, reasoned action and sound leadership. That builds confidence and trust. I guarantee you that your peers in the war room will notice and respect you for it and your customers will benefit from it. Let’s not let any of them down.