devopsy.kinsta.cloud

Tag: roles

  • When IT Disaster Strikes, Part 1: Resolving Incidents

    When IT Disaster Strikes, Part 1: Resolving Incidents

    As a developer or operations team member, there is nothing quite like the dread you feel when you hear the familiar ringtone of your on-call page at 3 a.m. Being on call means that you may be contacted at any time to investigate and fix issues that arise for the system, but that doesn’t mean you can’t get ready beforehand. There are steps your team can take to prepare for incidents and streamline the process of resolving them, leading to fewer 3 a.m. wake-up calls and better-running software and services.

    Below is an overview of a proven approach your organization can take before, during and after an IT incident to reduce headaches and increase hours of sleep.

    Before: Define the Who, What, When of Incidents

    The first step in any incident prevention and orchestration process is determining how your organization defines an incident. Generally, this is done by defining the varying degrees of severity, with lower-numbered severities more urgent. Operational issues can be classified at one of these severity levels. As a general rule of thumb, if you are unsure which level an incident is, treat it as more severe to ensure it is dealt with timely.

    In addition to aligning your team on defining an incident to communicate the sense of urgency involved, there are several main roles that should be designated ahead of potential IT issues. Certain roles have only one person per incident, while other roles can have multiple people assigned to it. It’s all about coming together as a team, working the problem and getting a solution quickly. Generally speaking, developer, IT Ops and DevOps teams must designate the following roles:

    • A point person or “Incident Commander”—someone who will drive the process forward, but not be involved in the actual remediation;
    • Someone to document a timeline of an incident as it progresses for future analysis and learning and to act as a backup for the point person;
    • One or more subject matter experts who are deeply familiar with the specified component or service and who will take the remediation steps; and
    • One person to manage customer support and communication during and after an incident.

    During: Keep Calm and Communicate

    If you are alerted of a major incident, don’t panic. The first step is to join a previously agreed-upon method of communication to be used during incidents to ensure that communication can run smoothly throughout the resolution process. The incident commander (IC) and the IC deputy should announce the issue in appropriate communication channels and lead the process, and it is best to defer to the subject matter experts assigned to the incident to ensure non-essential communication is kept to a minimum.

    The steps you take to prepare for an incident have a significant impact on how quickly you are able to move when disaster strikes. If incident prep has been done correctly, each member of the incident response team will have a very specific role and set of responsibilities carved out, ranging from someone to provide regular updates in the chat client to someone to modify your company’s status page to keep customers informed. By having these roles defined beforehand, you don’t have to spend valuable time during an incident figuring them out instead of fixing the problem. In a future article, I will break down each member of the incident response team’s specific role and the steps they should follow during an incident.

    After: Conduct a Post-Mortem Without Finger-Pointing

    For every major incident, you should follow up with a post-mortem—a blame-free, detailed description of exactly what went wrong to cause the incident, along with a list of steps to take to prevent a similar incident from occurring again in the future. The incident response process itself should also be included as part of the review.

    As the IT incidents we deal with daily become increasingly tied to larger organizational success and business objectives, streamlining the resolution process is a must. According to a report from IDC, the average hourly cost of an infrastructure failure is $100,000 per hour, and the average cost of a critical application failure per hour is $500,000 to $1 million. In future articles, I will break down the varying roles on an incident response team, specific steps to follow during an incident, a tried-and-true template for conducting a successful post-mortem and more, to ensure effective incident prevention and resolution.

    About the Author / Eric Sigler

    Eric Sigler is the Head of DevOps at PagerDuty, helping protect its customers from the pains of downtime. Before his current role, Eric led infrastructure teams at Minted, Expensify, and the Missouri University of Science and Technology. Connect with him on Twitter.

  • Services As A Model For DevOps Communications

    Services As A Model For DevOps Communications

    DevOps is an interesting movement. As with any effort of this magnitude, there will be many opinions on what is entailed in adoption and implementation. The issue is that businesses are so diverse and varied in culture that no single method for adoption is guaranteed to fit all businesses. In large IT organizations, this issue is additionally hampered by the fact that multiple cultures and dominions exist.

    Thus, I am not a believer that DevOps adoption comes from the approach that the business commits to adopting DevOps. The business has committed to many initiatives over time that have not panned out. This is not necessarily failure, but perhaps just shifting priorities, shifting political landscape, lack of training or simply a lack of desire to change. “The best laid plans of mice and men ….” and all.

    If commitment is not hampered by people, then there’s a good chance that DevOps adoption may be hampered by silos. This does not make silos our enemy. Silos help to compartmentalize and provide focus. Still, silos limit the ability to orchestrate process without involvement and participation from each of the silos.

    Regardless of the reason, the challenge stands, how to align a process without the use of force. The answer is protocol. With protocol, an individual or group does not need to buy into the entire process, but just needs to understand their role. Short of incompetence or insubordination, or even both, which will eventually result in the removal of these particular types of hurdles, following protocol ensures that every role in an activity will complete their action and, thereby, facilitate the whole with roughly an 80% success rate.

    I posit that the service is the means for defining protocol and of communicating that protocol into an IT organization, such that DevOps is the natural activity and outcome. A proper service entails defining a number attributes, for example:

    • Metered
    • Valued (priced)
    • Advertised / Catalogued
    • Governed
    • Monitored
    • Supported
    • Documented
    • Quality
    • Available
    • Intuitive

    It is up to each area of IT participating in the DevOps process—Design, Develop, Test, Stage, Deploy, Operate—to contribute to defining and providing their completed artifacts that pertain to formulation of the service. The service will guide the development in the understanding of operational characteristics and will allow operations to ensure they can deliver and support the service to the level of expectations of the consumer. Moreover, in those cases where the service is not meeting appropriate levels, it will allow them to provide development with the appropriate data and metrics to take corrective action, thus leading to fewer and shorter outages and failures.