devopsy.kinsta.cloud

Tag: incident response

PagerDuty acquires Event Enrichment HQ to speed incident resolution

Together PagerDuty and Event Enrichment HQ Integrate Event and Incident Management

I had a chance to speak yesterday with CEO Alex Solomon, PagerDuty’s Product VP Jonathon Wilkinson and Ophir Ronen, Event Enrichment HQ Co-Founder and now Product Director at PagerDuty. They all believe that this is a significant move by PagerDuty to help overcome the desensitization that many IT teams suffer as a result of so many alerts about incidents and events. Trying to turn down the signal to noise ratio and allow IT team members to concentrate on what is most important is a big part of the Event Enrichment platform.

The company already has begun integration into the PagerDuty and the whole Event Enrichment team is staying on with PagerDuty.

Below is the press release:

San Francisco – November 3, 2015 — PagerDuty, the leader in operations performance management, today announced that it has acquired Event Enrichment HQ and their Event Enrichment Platform (EEP). IT operations teams are facing a flood of monitoring data and notifications as the number and sophistication of monitoring tools increases. Integrating the Event Enrichment Platform with PagerDuty reduces noise, surfaces actionable alerts, and delivers context-rich remediation information so businesses resolve critical incidents faster.

“Event Enrichment HQ realized early that the proliferation of tools within IT operations generates a lot of additional noise and wastes critical time,” said Alex Solomon, PagerDuty CEO and Co-Founder. “The company’s work directly aligns with PagerDuty’s continued commitment to helping businesses deliver uptime and great customer experiences.”

The integrated solution helps businesses decrease resolution times by reducing noise so incident responders can spend less time on manually triaging events and by providing remediation information within the notification itself as the first source of truth. PagerDuty’s core functionality has addressed the challenge of assembling the right incident responders on the issue, but now teams have even better focus with intelligent control of monitoring system events.

“Having felt the pain ourselves for many years, we set out to provide the IT operations community with the tools to suppress noise and enrich events” said Ophir Ronen, Event Enrichment Co-Founder and PagerDuty Product Director. “Joining PagerDuty allows us to continue that vision and extend into helping customers streamline the entire incident lifecycle to decrease downtime.”

PagerDuty now offers a private beta of the Event Enrichment functionality to select customers. Additional details are available here.

ABOUT PAGERDUTY

PagerDuty’s operations performance platform reliably increases the uptime of businesses’ applications, services and infrastructure. By connecting people, systems and data into a single view, PagerDuty helps all teams — from DevOps to Enterprise IT — deliver powerful visibility and improved incident resolution times. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Adobe, Airbnb, Etsy, GitHub, Intuit, National Instruments, New Relic, Panasonic and Rackspace. The company is headquartered in San Francisco and backed by leading venture capital firms Andreessen Horowitz, Bessemer Venture Partners, Baseline Ventures, Ignition Partners and others. To try PagerDuty for free, or to get more information, visit www.pagerduty.com.

ABOUT EVENT ENRICHMENT HQ

Event Enrichment HQ’s Event Enrichment Platform (EEP) allows IT professionals to significantly reduce their downtime by eliminating noise and enriching actionable events with remediation information. IT teams using EEP have the ability to route, manage, and enrich their alerts without interacting directly with critical production monitoring systems or code. Privately held, Event Enrichment HQ is made up of IT operations veterans who are focused on providing solutions for reducing downtime. The company is headquartered in Seattle, Washington. For more information, or to try EEP for free, visit www.eventenrichment.com.

November 3, 2015
Reducing Incident Resolution Time
From raw incident count to response time, there are several key metrics that top Operations teams track to measure and improve their performance. One of the most popular metrics teams track for is mean time to resolution (MTTR). It’s the time between failure and recovery from failure, and is directly linked to your uptime. While MTTR may be the gold standard when it comes to operational readiness, it’s important for teams to look at the bigger picture to effectively decrease incident resolution time.

Putting MTTR into perspective
Your overall downtime is a function of the number of outages as well as the length of each. Dan Slimmon does a great job discussing these two factors and how you may want to think about prioritizing them. Depending on your situation, it may be more important to minimize noisy alerts that resolve quickly (meaning your MTTR may actually increase when you do this). But if you’ve identified MTTR as an area for improvement, here are some strategies that may help.

Working faster won’t solve the problem
It would be nice if we could fix outages faster simply by working faster, but we all know this isn’t true. To make sustainable, measurable improvements to your MTTR, you need to do a deep investigation into what happens during an outage. True – there will always be variability in your resolution time due to the complexity of incidents. But taking a look at your processes is a good place to start – often the key to shaving minutes lies in how your people and systems work together.

Check out your RESPONSE time
Some call it MTTA (mean time to acknowledge) or MTTR (same acronym, but meaning mean time to respond), but the clock starts ticking as soon as an incident is triggered, and with adjustments to your notification processes, you may be able to achieve some quick wins.

If your response time is on the longer side, you may want to look at how the team is being alerted. Do alerts reliably reach the right person? If the first person notified does not respond, can the alerts automatically be escalated, and how much time do you really need to wait before moving on? Setting the right expectations and goals around response time can help ensure that all team members are responding to their alerts as quickly as possible.

Establish a process for outages
An outage is a stressful time, and it’s not when you want to be figuring out how you respond to incidents. Establish a process (even if it’s not perfect at first) so everyone knows what to do. When you’re designing your process for responding to incidents, make sure you have the following elements in place:
1. Establish a communication protocol – if the incident is something more than one person needs to work on, make sure everyone understands where they need to be. Conference calls or Google Hangouts are a good idea, or a room in HipChat or Slack.
2. Establish a leader – this is the person who will be directing the work of the team in resolving the outage. They will be taking notes and giving orders. If the rest of the team disagrees, the leader can be voted out, but another leader should be established immediately.
3. Take great notes – about everything that’s happening during the outage. These notes will be a helpful reference when you look back during the post mortem. At PagerDuty, some of our call leaders like using a paper notebook beside their laptop as a visual reminder that they should be recording everything.
4. Practice makes perfect – if you’re not having frequent outages practice your incident response plan monthly to make sure the team is well-versed. Also, don’t forget to train new-hires on the process.
To learn more, check out this talk about incident management from Blake Gentry, a former lead software engineer at Heroku.

Find and fix the problem
Finding out what is actually going wrong is often the lion’s share of your resolution time. It’s critical to have instrumentation and analytics for each of your services, and make sure that information helps you identify what’s going wrong. For problems that are somewhat common and well understood, you may be able to implement automated fixes.
March 31, 2015
PagerDuty goes beyond alerting to advanced analytics of incident resolution

PagerDuty has made a major move from simply alerting team members to potential issues via pager or web to actually becoming an advanced analytics platform to measure team and system performance regarding incident response. They announced their new functionality today (release below). This is really a big step up for PagerDuty which also recently announced closing on another round of venture caiptial.

I spoke with Alex Solomon, CEO and Co-founder, David Shackelford, Product Manager and Nisha Ahluwalia, VP of Marketing recently about these new features. I had spoken with Alex some time back. He moved from Canada to Silicon Valley to found Pager Duty at the successful Y-combinator accelerator. Since then they have made steady progress on executing on their plan. However, this represents really a whole new market for PagerDuty. Proving insight into both team performance as well as system performance in regard to incident response is a big opportunity and Solomon, Shackelford and Ahluwalia are confident that PagerDuty is ready to capitalize on it.

I am personally very excited by this new functionality. A mantra of business that I learned from my friend and serial entrepreneur, Rajat Bhargava is that you need that you need to measure everything and everything can be measured. This gives organizations the ability to measure incident response across teams and systems.

Here is a sample of the system report:

While there are vendors who do system analysis reporting, the combination of this with team analytics data should prove very powerful.

PagerDuty’s Advanced Analytics Delivers Comprehensive View of Incident Resolution Across the Organization

New Resolution-Time Metrics and Drill-Down Capabilities Enable Companies to Make Proactive Improvements to Increase System Reliability

San Francisco – September 16, 2014 —PagerDuty, the leader in operations performance managementtoday announced the public release of its Advanced Analytics tools, which provide insights IT teams can use to improve team and system performance. Leveraging PagerDuty’s robust data on key operational metrics like incident frequency and time to respond and resolve, companies can now drive even faster incident resolution. The new capabilities further expand PagerDuty’s operations performance platform by giving managers the ability to analyze and improve key drivers of uptime.

“Throughout the lifecycle of an incident, responders make real-time decisions, but little of that institutional knowledge is saved to make systematic improvements. Advanced Analytics gives responders and managers the tools they need to identify, analyze and share trends about their resolution times and incident load,” said PagerDuty Product Manager David Shackelford. “We’re excited to help our customers make smarter, data-backed decisions about their operations.”

PagerDuty brings together data from all monitoring systems into a unified view, giving teams a single place to compare data about incidents with people and process metrics. With Advanced Analytics, managers gain better insight into how teams are responding to and resolving incidents, and determine whether teams are spending time on the right problems or spinning their wheels on unnecessary alerts. The reports provide a holistic view of team and system performance, enabling everyone from individual operations engineers to CTOs and CIOs to have richer discussions about these metrics.

“PagerDuty’s Advanced Reporting saves us time since we’ve been calculating this data ourselves,” said Matthew Green, Senior Engineer at Rackspace. “We’re using the new reports to better understand the incidents we see in our environment on a daily basis. Also, seeing the resolution time data for our distributed teams will help us improve our response time.”

Advanced Reporting is immediately available as part of PagerDuty’s enterprise plan.

About PagerDuty

PagerDuty’s operations performance platform helps companies increase reliability. By connecting people, systems and data in a single view, PagerDuty delivers visibility and actionable intelligence across global operations for effective incident resolution management. PagerDuty has over 80 platform partners, and is trusted by Fortune 500 companies and startups alike, including Microsoft, National Instruments, Electronic Arts, Adobe, Rackspace, Etsy, Square and Github. The company is headquartered in San Francisco and backed by leading venture capital firms Andreessen Horowitz, Bessemer Venture Partners, Baseline Ventures, Ignition Partners and others. Try PagerDuty for free, or to get more information, visit www.pagerduty.com

September 16, 2014