devopsy.kinsta.cloud

Author: Orlee Berlove

  • The State of IT Ops

    The State of IT Ops

     In our most recent article, we highlighted some of the results of our recent IT Ops survey. We noted, for example, that 80 percent of IT teams are alerted to critical events via email and 41 percent of teams are experiencing excessive alerts. These stats go a long way to explaining why IT Ops is experiencing so many issues with alert fatigue.

    In this article, we will delve into the second half of our findings and examine our results on how IT teams reported they handle alerts after they come in.

    Off Scheduler

    Efficient IT operations teams realize the need for directed action when critical issues come in. This means that when significant IT issues occur, efficient teams will have alerts sent directly to the on-call engineer, who can manage the incident most effectively. The engineer will be the individual who is best-equipped to manage and quickly resolve the issue at hand.

    Our survey however showed that most teams lack automatic schedulers which could ensure the correct individual is alerted. Rather, alerts are often sent to the team as a whole, or to individuals who are unable to resolve the issue. According to our statistics, teams are frequently alerted (67.3 percent) instead of the on-call individual.

    The impact of this poor workflow is not only that issues are sent to individuals who are unable to resolve them, but time is wasted by this very process. Ideally, team managers should invest in scheduling software that sends alerts directly to the engineer on-call. By sending it to the wrong individual or team, companies are adding fuel to the fire of alert fatigue and burnout.

    Related Articles:

    Site Reliability Engineering: How to Make the Operations Side of DevOps Actually Work

    Slack and Critical Alerting

    Unfortunately, our team also highlighted that most teams lack scheduling technologies to enable this outcome. Our survey highlighted that only 28 percent of teams use an automated scheduler to ensure the correct individual or team is alerted.

    By failing to use a system that ties alerts to on-call schedules, managers receive alerts via email, SMS or phone that they are not necessarily responsible for resolving. Managers then need to track down the on-call engineer who can manage the incident. The pain of this scenario is multiplied if the alert comes after-hours or during the night. Again, this workflow leads to significant delays in resolution time.

    No Time for Downtime

    As IT Ops are often tasked with issues such as maintaining server uptime or site uptime, the prospect of downtime is something engineers work hard to avoid. IT Ops are under significant pressure to maintain five-nines uptime.

    For this outcome to occur, teams need to automate menial tasks and ensure they are quickly alerted to complex downtime issues that demand their attention. The goal needs to be getting the issue as quickly as possible in front of the individual who is best able to resolve the matter.

    However, our survey showed that 45 percent of teams indicated it takes more than 10 minutes before IT teams even begin to resolve critical IT issues. Given that the cost of downtime is more than $5,000 per minute, according to Gartner, downtime can easily balloon to cost companies or their clients more than $50,000 in 10 minutes.

    The significant expense cause by downtime has been seen in many recent scenarios. For example, in mid-May 2017, British Airways had a spectacular downtime event due to the failure of a power system in the company’s data center. British Airways was forced to cancel all its flights leaving from London airports after having a major IT system failure. The result was cancelled flights affecting more than 75,000 passengers. The costs of reimbursement were approximately $68 million and a 2.8 percent drop in the stock price of the parent company.

    Failure to Collaborate

    In the world of DevOps and IT Ops, we are often told of the importance of collaboration and how it leads to great innovation. Indeed, the goal of DevOps is to bring developers and operations into closer proximity, so they can collaborate more effectively. Better collaboration means better innovation and decreased costs.

    However, our survey shows that while IT Ops might have the mindset for collaboration, they lack the necessary tools for ensuring that collaboration happens. Our survey showed that IT Ops teams often collaborate on tickets by email. According to the results of our survey, 86 percent of teams collaborate through email.

    Yet, email is a truly poor way to collaborate when critical incidents come up. Email has few ways to bring critical emails to the top of the stack and keep them center of mind. Instead, email’s default is to let messages get buried under other emails. In attempting to collaborate on key issues, email fails to offer rapid communication methods that highlight critical messages.

    Tools are available that enable better and faster communication among team members. Slack is a tool popular among many IT Ops engineers who are key drivers of effective collaboration. Yet, our survey showed that only 16 percent of ITOps use ChatOps as a form of collaboration during incidents. Slack’s motto is, it’s “where work gets done.” On their own, however, tools such as Slack lack a method for bringing important issues to the top. However, if attached to incident management systems such as OnPage, Slack and other tools like it can enable users to receive alerts that a critical issue needs to be addressed.

    Improving Workflow

    The goal of improving how teams handle alerts is to minimize downtime and improve efficiency. These shouldn’t seem like lofty goals. However, by not developing specific workflows that bring issues to the top and enable them to be resolved quickly, IT Ops teams are adding to the time until resolution and the ensuing cost.

    At its core, the lack of specific workflows and tools to enable improved workflow points to teams that have much room for improvement. IT Ops teams need to find better ways to manage day-to-day operations so that when critical incidents do arise, the path toward resolution is quick and doesn’t require engineers to embark on a guessing game.

    Conclusion

    Unfortunately, the state of IT Ops has not progressed to the level of maturity one would hope. There is still a lot of poor alerting and workflow practices that could be improved if a more robust incident management system were employed.

    There are a number of additional insights that can be garnered from our survey. I encourage you to take a moment and download a copy of the study and see what you can learn that will help your team.

    — Orlee Berlove

  • The State of Alerting in the IT Ops world

    The State of Alerting in the IT Ops world

    OnPage Corp. just finished a survey of more than 100 ITOps professionals from across the United States. Our goal was to acquire a greater understanding of how well engineers in the industry are performing when it comes to critical alerting and alert management of their IT teams.

    We wanted to understand the antecedents of alert fatigue for ITOps that appear earlier in the food chain. We also wanted to see how many alerts teams receive per day as well as who gets alerted. We wanted to understand how alerts are managed. And, we wanted to see how well teams analyze their actions and take those lessons forward.

    In many ways, the survey was successful. We received a large number of responses from a number of industries and acquired a strong sense of how ITOps is performing across the country. Unfortunately, we also saw that for all the Chaos Monkeys and strides toward improved response to alerting, there is still a significant lack of progress.

    What’s All the Buzz About?

    Automated alerting is an essential component of monitoring. Automated alerts are what allow teams to receive automatically generated alerts from multiple points along their IT stack and software. In theory, this multitude of alerts is what enables teams to more quickly identify the causes of a problem and minimize the severity. The hope is, that with early recognition of the issue, engineers will be able to minimize service degradation and disruption.

    But alerts aren’t always as effective as they could be or need to be. Real problems often are lost in a sea of noisy alarms. As our survey showed, this is because teams are inundated with alerts coming in via multiple formats. Moreover, the barrage of alerts leaves teams inundated and practically unable to cope.

    The Law of Above Average

    Our survey showed that more than 80 percent of IT teams are alerted to critical incidents via email. General best practices would dictate that email is fine for daily communication inside a business. However, for critical incidents, email is less than ideal, as it allows critical incidents to get buried under a pile of other emails. There is no way for critical issues to rise to the top of the pile.

    Since our questions were multiple choice, respondents could provide multiple answers with regard to how they received notice of critical incidents. So, while email was the most prevalent form of communication, individuals indicated that they are also simultaneously receiving alerts by SMS or phone call.  Our survey showed that 58.9 percent and 51.4 percent of respondents received alerts via these methods.

    Already from this nugget of information, we see the opportunity for both information overload and an opportunity for missed alerts. By simultaneously receiving alerts through multiple formats, the level of irritation and overload inevitably rises. At the same time, if emails are the only form in which IT professionals receive alerts then there is a high opportunity to miss alerts all together.

    How Many Alerts Was That?

    The survey results also indicated that just over 41 percent of ITOps receive 11 alerts or more per day. Additionally, just over 20 percent of this group received 40 alerts or more per day. While 40 alerts is clearly more than a team can reasonably manage or should manage, this figure also goes a long way toward explaining why some alerts just get missed. If more than 40 alerts are sent to you and your team every day, it becomes very difficult to prioritize alerts and determine which should be handled first.

    Perhaps to better manage this large number of alerts, many teams use escalation procedures. Our survey showed that 76.6 percent of respondents have some sort of escalation procedure in place. At the same time, the most frequent ways to escalate critical responses was through email or SMS.

    The conclusions one can draw from these numbers are that, despite the large number of papers written on improving alert management, many ITOps have not been able to achieve this end. While our survey did show that just shy of 59 percent receive a manageable number of alerts, 41 percent are inundated.

    Not Just Intelligence, Business Intelligence

    Perhaps analysts of the industry could be more optimistic if they saw that teams were using analytics to track how well they are performing. If teams employed analytics, they would be better able to review their progress, see where they are failing to meet the grade and then embark on routines to improve. Unfortunately, this is not the case.

    When asked whether their team has employed any type of business intelligence to review and analyze their team’s performance, more than 70 percent reported that they did not subscribe to any BI platform. The problem with this result is more than just a missed opportunity; it is also the loss of opportunity to fundamentally improve the business at many levels.

    One of the most important reasons why you need to invest in an effective BI system is because such a system can improve efficiency within your organization and, as a result, increase productivity. Effective business intelligence can also improve the decision-making processes at all levels of management and improve your tactical strategic management.

    Yet by forgoing investments in these BI tools, teams are failing to investigate their processes and methods that would improve their team and minimize alert fatigue.

    A Call for Smart Alerting

    The lesson can be drawn from this is that companies don’t necessarily need more alerting. What they do need is to shift toward more smart alerting.

    Smart alerting means that not every bump on the monitoring screen gets tied to an alert. Instead, monitoring output is calibrated so that possibilities are aligned with probabilities and impacts. Alerts also get sent to the teams or individuals that are best able to manage the issue. Additionally, alerts are actionable and come with instructions regarding what the problem might be.

    Smart alerting also means that teams use business intelligence tools such as reports and graphs and charts to determine which of their practices have been effective or not effective. Without this insight, teams are often unaware of the subtle points that could really impact their team and provide them with a way to improve their output.

    Conclusion

    There are a number of insights that can be garnered from our survey. I encourage you to take a moment and download a copy of the study and see what you can learn that will help your team.

    — Orlee Berlove