Tag: postmortem

  • The Single-Sentence Postmortem

    The Single-Sentence Postmortem

    Do you have four seconds?  In the time it takes you to read this sentence you could be done with your incident’s postmortem.

    “But my postmortem has 8 sections!  I have to construct a critical timeline, restate my followup tasks, show the major milestones, evaluate root cause…”

    Stop right there.

    First of all, your incident management system should be automatically capturing your timeline, follow-up tasks and major milestones as they happen — so you shouldn’t be worrying about reproducing that content at all.

    Second of all, the most critical thing a responder can bring to a postmortem, in the moments right after the incident concludes, is the one-sentence personal context of what actually happened. It’s the same sentence they’d say to a co-worker in the lunch room when asked what went wrong. But to achieve organizational resilience, the knowledge in that sentence has to be captured and distributed. I can’t emphasize this enough – a single sentence can unlock 80% of the value. Writing something, anything, is so much better than writing nothing, and yet, too often, we see companies skip this part of the incident management process.

    Modern incident management, as a practice, was born on factory floors.  Instead of firing the person who was considered accountable for a major incident, having that person write up their view of what happened was more impactful at preventing future incidents. That’s because only the responder has the complete context to shed light on the contextual, systemic failures that actually caused incidents, as compared to the outside view of management and other employees who will often see problems as human-centered or due to obvious oversights (using 20/20 hindsight).

    When asked if he would fire an employee who made a $600,000 mistake at IBM, Thomas J. Watson famously replied, “No! I just spent $600,000 training him – why would I want somebody to hire his experience?”

    The reality is that any amount of your unique perspective as a responder is invaluable when compared to nothing, and without it, the chances of your company learning from these mistakes is slim. That’s why we encourage a unique approach: we tell our customers to spend four seconds on their postmortem.

    Especially for those new to incident response and management practices, it’s important to develop a practice of doing postmortems right away, as soon as the incident closes. As quickly as possible, open up your postmortem, jump to the ‘What Happened?’ section, and write one useful sentence. That’s it.

    Once you do that, we know that one of two things will happen:

    1) You’ll have other things to do, so you’ll publish the postmortem, and that one critical piece of knowledge will be shared throughout the company. That is infinitely better than nothing. 

    2) Or, you’ll realize you have more to say and get it done, right then and there.

    Like putting your running shoes on even if you don’t want to go for a run — at the very least, you’re up and active; at most, you’ll get out there and rack up some miles.

    Working with companies across industries and seeing thousands of incidents filed; hearing the stories of hundreds of companies’ incident management implementations has taught us that the alternative to a quick postmortem often isn’t often a more in-depth one, it’s no postmortem at all.

    So if you’ve just finished an incident– and if you’re about to pick up your coffee and move on to the next task at hand – instead, spend four seconds writing one sentence. It could make all the difference.

  • Atlassian Brings Postmortem Analysis to DevOps

    Atlassian Brings Postmortem Analysis to DevOps

    Atlassian has added a postmortem capability to JiraOps, which gathers a list of all the key events from any IT incident in chronological order to make it easier to analyze what happened, identify root causes and raise issues directly within the JiraOps platform to ensure subsequent actions are taken.

    At the same time, Atlassian is extending the Automation Actions, a set of automated scripts and playbooks made available in Opsgenie to include support for Amazon Web Services (AWS) Systems Manager and Generic REST Endpoint. Those capabilities extend the ability of DevOps teams to launch automated tasks from within an Opsgenie console or mobile applications. Opsgenie is a notification application that Atlassian gained via an acquisition announced the same day the company launched JiraOps.

    Atlassian is also sharing incident management data it has collected throughout the year. Collectively, its customers have recorded more than 190,000 incidents and scheduled maintenance events. Updates to those events were made more than 564,000 times, which in turn fueled the delivery of more than 130 million email and text messages to end users.

    Unfortunately, the Atlassian report also finds only 3 percent of incident reports include any kind of postmortem analysis—an issue the company is trying to address via the latest update to JiraOps.

    Danny Olinsky, head of product marketing for incident management at Atlassian, said that as organizations make the shift from primarily deploying application on-premises to the cloud, many of them are starting to more fully embrace best DevOps practices. One of the central tenets of DevOps is ruthlessly automating as many processes as possible, including incident management. By integrating incident management and DevOps processes, the amount of downtime experienced by any application over the course of its life cycle can be reduced substantially, he said.

    The biggest challenge many IT organizations now face is melding DevOps practices and incident management process, which frequently evolve in isolation from one another. Those processes, however, are being unified as more organizations digitally transform business processes that need to always available. Any incident that might lead to application downtime now has potential revenue implications that go well beyond adversely impacting employee productivity. In fact, clear, precise communications involving any IT incident is now an expectation in the DevOps era, said Olinsky.

    The degree to which organizations will be automating incident management naturally will differ. But at a time when mean time to resolution is being more aggressively measured than ever, many IT professionals have an incentive to automate as many processes as possible. Murphy’s Law also dictates that incidents will manifest themselves at the most inconvenient time, so being able to automate a process from a mobile device can eliminate the need to make a trip to the office in the dead of night.

    It may be a while before IT organizations are able to fully automate every response to an IT incident. But as process that can be automated are identified, the lower the probability a truly catastrophic event capable of paralyzing the entire organization becomes.

    — Mike Vizard

  • Security: How to Conduct an Agile Incident Postmortem

    Security: How to Conduct an Agile Incident Postmortem

    In a perfect world, every organization could block every attack, no employee would ever make a mistake, and there would be advance warning that an organization is on some cybercriminal’s list of targets. Since organizations operate in a world that is far from perfect, however, they are forced to accept that bad things will happen. History and headlines show they cannot erect enough barriers to stop cybercriminals from trying to penetrate defenses, they cannot hire perfect employees and they must assume their name has been bandied about as a potential target.

    One of the things they can do, however, is use each incident as a learning opportunity by conducting a thorough postmortem. To maximize the benefits of a post-incident analysis, organizations should remember two things: agility and blamelessness.

    Agility

    To a developer, agility is a development methodology that provides significant benefits over the traditional waterfall method. However, agility implies a system of thinking, processing information and executing plans. As such, agility belongs in the realm of security as much as it belongs in the realm of development.

    Currently, the advantages offered by agility tend to be reaped more by cybercriminals than by security professionals. The bad guys get to go first, and since they are not bound by compliance issues or regulations, have no concern for the rights of individuals and don’t have to answer to advocacy groups or government agencies, they have the advantage. Cybercriminals embrace agility to launch attacks that can change from one day to the next.

    Unless security professionals are as agile as the crooks, they will be limited to reacting to incidents instead of proactively preventing them. One way to help level the playing field is to build in the ability to move quickly when an incident occurs—and this includes a plan to conduct an effective, thorough and efficient postmortem.

    However, a postmortem implies that an incident is over and is now undergoing a final review. If organizations are truly agile, they will have conducted at least one retrospective prior to the postmortem. Agile retrospectives are conducted to assist the team in making immediate changes and are more about action than review.

    Blamelessness

    For retrospectives and postmortems to be effective, the second key is to ensure that postmortems are blameless. The end goal is to get to the truth so that organizations can better protect their operations in the future. Assigning blame can make it much more difficult to uncover the truth.

    For example, if an employee clicks on a link that spreads malware throughout the corporate network, he or she might be afraid to admit the mistake for fear of being chastised privately or blamed publicly. The organization then wastes time searching for the source of the infection, instead of being able to deal with the incident immediately. Therefore, the postmortem must be blameless and cordial, with no finger-pointing allowed. Keep the focus on what happened rather than who is responsible.

    Postmortem Best Practices

    Understanding the two “prime directives” for conducting a postmortem, organizations have a few other considerations to help them gain the most from their post-incident evaluation. First, they need to learn precisely what happened and how it can be explained concisely, accurately and appropriately. If the chain of events is not understood, an organization will not be able to enlighten team members or non-technical employees.

    Next, the incident should be used as a learning experience to ensure that the problem will not occur again. Organizations should evaluate whether the issue is a simple employee mistake, a tool on which staff members were not trained, miscommunication or a process that is broken. Everyone should be focused on preventing another occurrence, as well as ways to achieve continuous improvement.

    After the facts are uncovered, an organization must take action, ignoring “should have” and “could have” statements and focusing energies on “will” and “must.” In addition, organization must establish priorities and assign follow-up actions before ending the postmortem.

    Also, employees and team members should be encouraged to discuss potential issues whenever they arise. Employees may need guidance on whether an action is safe, team members may need to confess that they made a mistake or supervisors may want to discuss ways to help workers learn how to identify suspicious emails.

    Trust and communication are essential if organizations want to gain the greatest benefit from their postmortem. Organizations need to stay open and approachable so they can reduce the number of incidents, improve incident response and strengthen their company’s defenses.

    — Erin Swanson

  • Understand Why You’re Doing DevOps

    Understand Why You’re Doing DevOps

    DevOps has been all the rage for a few years now. It has quickly matured from a fringe concept to a mainstream imperative, and organizations that haven’t already embraced DevOps are scrambling to understand and implement it to keep up with rivals and remain competitive. Despite its benefits, though, DevOps may not be for everyone. Before diving into the DevOps deep end, you first should consider what your objectives are.

    To frame the issue more precisely, it’s not that organizations necessarily should avoid DevOps altogether. It’s just important to go into it with your eyes wide open and to understand why some DevOps projects fail. According to Andrew Storms, VP of Security Services at New Context, it’s always for one of four reasons: cultural roadblocks, failure to identify and take action on mistakes as you go, failure to learn from past mistakes and most importantly, failure to include security—either early on or at all.

    Let’s break down those four hurdles.

    Face Cultural Roadblocks

    DevOps is different things to different organizations, or even to different teams within an organization. There are many definitions of what DevOps is or is not. There is a fairly unanimous consensus, though, that DevOps is first and foremost a function of culture.

    Cultural roadblocks, Storms said, are by far the most common reason he sees DevOps fail. “It’s rarely a technical tools problem; usually there’s a disconnect between the DevOps team and executives or other parts of the organization. A concerted effort must be made to get everyone in the organization on the same page. Everyone is fighting for the same thing, but too often, they don’t realize it.”

    Fix Mistakes

    One of the outcomes of the DevOps culture shift is a fundamental change in corporate bureaucracy and the rigid processes that segregate teams and impede progress. DevOps enables organizations to work more seamlessly and produce results more quickly. It also allows issues and mistakes to be identified and resolved on the fly. “Smaller incremental changes are always easier to handle than a large monster of a release, which can include hundreds of changes,” explained Storms. “For example, teams should look for the single change in a monthly release that resulted in an unexpected security hole, instead of thinking they’ll tackle all changes with a huge release down the road.”

    Learn from the Past

    One of the most important things for business in general to help them understand is learning from past mistakes. With DevOps, it’s crucial. Organizations often only conduct investigations and postmortem analyses in the event of an issue or catastrophic failure. In that situation, the teams and individuals involved are immediately on the defensive and likely to go into “CYA” mode.

    To effectively understand and learn from the past the discussion has to be neutral and without blame or consequence. It’s a difficult concept for organizations to grasp. The key to blameless meetings is to conduct them as a regular occurrence rather than a response to a negative situation, and to foster a culture where failure is acceptable as long as it is used as a learning experience.

    Storms also suggests changing the name to “After Action Reports” to remove the negative stigma. “Postmortem implies a dead body. After Actions Reports are imperative after any event regardless of the type of outcome.”

    Start with Security

    Security is an Achilles heel of development in general, but the rapid pace of development in a DevOps environment makes it even more critical that security be integrated from the beginning. Failing to consider security is a disaster waiting to happen.

    Those responsible for security often don’t even know about a project until after it’s already in production. Storms cautions organizations, “Untested and unreviewed code or designs in production represent both a large security risk, but also a risk to the business and your customers. Security needs to be part of the SDLC (software development life cycle) and DevOps from day one.”

    DevOps is not a silver bullet. Nothing is. There are a variety of unique advantages and benefits that come from successfully embracing DevOps, but to realize them you need to start with a clear understanding of your own objectives, and how DevOps can be leveraged to help you achieve them.