devopsy.kinsta.cloud

Tag: alert management

  • Filter the Firehose

    Filter the Firehose

    We are tired. Information overload is a problem in the modern world. We hear instantly about events we never would have known about otherwise, or that we would have learned about months after the fact. Today, moments after an event, we have thousands of “professionals” analyzing it for us, a millions-strong army of amateurs telling us they know everything about it and a legion of bots telling us what to think about the event. No matter what the event is. Often “the event” isn’t even newsworthy, and yet the process feeds on itself.

    We in IT have it worse. We have a flood of security and application events constantly flooding in and so much data that we call it a ‘data lake.’ More like a data ocean at some organizations. We’re tired. They call it by various names—alert fatigue, data overload, event flood, “the firehose.”

    No matter what it’s called, it’s a big problem. And if it is not effectively managed, we will make mistakes. Particularly in security—but also across IT—we cannot afford to allow the flood to distract us or burn us out.

    Look into AI. The amount of data, alerts, information, metrics, logs, traces, etc. that we are accumulating—and the rate at which we are doing so—is growing faster than humans can reasonably keep up with—even if there were no staffing constraints. Don’t hesitate any longer; don’t assume you or a vendor NOC are managing this well enough with only eyeballs. AI solutions have matured enough to reliably sift through at least the top layer(s) of noise, reducing the burden without increasing risk. In fact, given the increasing possibility of human error as volumes increase, probably with less risk.

    Ask vendors about it; talk to your security vendors about their offerings. A huge number of security vendors have NOCs now that they would love to have you subscribe to. Many of them are highly automated and use AI. Data vendors are behind a bit in this regard (for a lot of reasons) but they are catching up. Not too long ago, the vast majority of work done to normalize disparate datasets was done via trial and error. AI and scripts written from massive experience are lightening that load, at least.

    So check it out. Find ways to make things manageable again. We all know things are slipping through the cracks as the volume of items reported and the number of locations they are reported from increases. Don’t wait until you have another emergency; look into the options and get something running.

    For those of you already using AI/ML, unless you implemented it very recently, you’ll probably want to validate what you’re using, at what level and how much the AI/ML solution is helping, and tweak or replace. This space is moving too fast for “set it and forget it” today. Maybe in a few years, we can.

    With the rare exception of something I use every day and think you might benefit from, I adhere to a policy of not recommending products here, so I can’t point you in the right direction other than to say that your vendors (data, SIEM and security) can offer solutions and/or suggestions. That, and “You shouldn’t wait” are my suggestions.

    And keep rocking it. Part of that data/log/alerting growth is all the apps and infrastructure you have helped put in and that are critical to the organization. So protect it with every tool available—and use that free time to work more miracles.

  • Three Ways Machine Learning Can Change Incident Management

    Three Ways Machine Learning Can Change Incident Management

    Incident management for important applications has many connotations, many of them negative. Most are well-deserved. Stress, panic, hurry, urgent, drop-everything, all-hands-on-deck, emergency, to name a few. Typically, there is a critical problem, ranging from complete failure to loss of certain functionality or performance. The problem or outage potentially means loss of revenue, loss of customers, negative social media commentary or bad reviews and more. Time is of the essence, dictating that the primary objective is fixing the incident as quickly as possible. Often, this means that symptoms or partial causes are mainly addressed rather than the underlying root cause.

    Without fully addressing the underlying issue, companies virtually guarantee that the same problem – or a similar one – will reoccur. Not identifying root cause often prevents a durable fix. In addition, companies lose the opportunity to proactively improve application code or infrastructure based on real-world experiences and issues. Postmortems may only result in reviews of monitoring and observability solutions and the inevitable updates to alert rules. Most DevOps professionals not only understand but have lived through these frustrations on an ongoing basis. Management, then, often wonders why their systems are so unstable.

    Changing the model for incident management has been limited by a combination of the overriding urgency combined with short-staffed, overworked teams. Although AI and machine learning has been positioned as the panacea for nearly every kind of technical ill, this is a clear case where “machines” could fundamentally enhance human efforts to improve a situation. The best troubleshooters exhibit a combination of instinct, experience and patience to carefully sift through reams of data, spotting unusual events and their correlation with bad outcomes. This turns out to be a perfect application for machine learning.

    By ingesting all logs and metrics to find meaningful problems or anomalies, systematized machine learning can quickly and comprehensively sift through all data to gain a thorough understanding of the real problem. Using this approach significantly speeds up the root cause analysis. Such an applied solution introduces three game-changers for incident management.

    Three Game-Changers for Incident Management

    First, incident remediation times can be reduced dramatically and with less of an all-hands-on deck, drop-everything frenzy. Rather than SREs and engineers spending hours digging through dashboards, traces and millions of log lines, an automated approach can accomplish it more quickly, more thoroughly and with less impact on human professionals, allowing them to focus on other responsibilities. This instantly relieves stress and panic and alleviates pressure because incidents can be remedied better and faster, restoring the application to its expected service level. The value is, of course, two-fold: faster time to resolution and greater efficiency or productivity of the teams who would otherwise be involved. Ending or mitigating the frustration professionals have to deal with is an additional bonus.

    Second, besides faster MTTR, an automated, machine-learning-driven approach is more comprehensive and gets to the root problem. This means that future incidents may be avoided altogether. This also means a potential end to the alert rule “hamster wheel.” The knee-jerk tendency to continually adjust alert rules and settings to enable better “early warning systems” is far less necessary. Instead of trying to gain an early indication of a similar failure in the future, the cause of such failures can be found and addressed proactively. Eliminating such problems in the first place is far better than trying to evolve alerts to catch unresolved causes. Alerts are still important, but machine learning can shift the basis for them to more root-level issues, and they can be constructed automatically by the system.

    Third, automated, machine-learning-based incident management can help proactively catch silent bugs that may not yet directly be part of an incident. It can also inform developers earlier in the cycle. Not terribly long ago, new releases were tested extensively before deploying to production, allowing for the creation of specific test plans, conducting stress tests and potentially catching bugs that might have potentially terrible consequences in the future. Now with faster, non-stop development cycles, testing mainly occurs during production. Machine learning can surface new or unusual errors, event patterns and patterns in metrics. The approach can proactively surface subtle bugs early, before they impact users or cause widespread failure.

    Changing the DevOps Life Cycle

    Each of these three results significantly changes not only the incident management life cycle, but also DevOps as a whole. Poring over logs and metrics is a perfect application for machine learning and offers the opportunity to fundamentally change incident management, transforming it from a scourge and necessary burden to a competency. An automated system of machine learning applied to incident management advances DevOps rather than encumbers it.

  • Getting Started With Continuous Monitoring

    Getting Started With Continuous Monitoring

    DevOps continues to gain traction among organizations as demand grows for digital product and platform development. According to Gartner, 87% of business leaders believe digitalization is a priority. Every DevOps transformation requires a dedicated, continuous learning process and effective implementation to reach maturity. If a practice or pattern is passed over or ignored, it can put a damper on DevOps success. Continuous monitoring, for example, is a very important part of every DevOps life cycle that is often overlooked.

    Through effective continuous monitoring efforts, organizations can scale and deliver digital products and platforms faster and more securely. In fact, these continuous monitoring advantages tie directly to modern customer needs. If they haven’t already, business leaders must embrace continuous monitoring along every point of the DevOps life cycle. These best practices can help you get started.

    Understanding Continuous Monitoring

    The practice of continuous monitoring helps to collect and analyze outcomes, statuses, exceptions and key metrics within each step of the DevOps process – from development to deployment and production. Additionally, it offers deep insight for DevSecOps teams by enabling compliance triggers and security alerts configuration. It allows the organization to detect issues or security concerns throughout every phase of the DevOps lifecycle.

    Understanding what continuous monitoring does is a good start. However, to effectively adopt continuous monitoring, the organization must know what to monitor.

    What Should Be Monitored

    An effective continuous monitoring process collects enough data to gain sufficient understanding of both successes and failures of the organization’s systems and processes. Some of the events collected by the continuous monitoring systems will require immediate attention, while others may lead to longer-term strategy discussions.

    Collecting the right information is always tricky considering the number of endpoints that generate logs and events. One should rank the significance of each source in relation to the type of data it produces, the functional area it supports and the impact it has on the system’s performance, as well as on organization’s goals and commitments (requirements posed by SLA’s, regulations, contracts, etc.).

    A combination of technology and strategy helps ensure that the right data is collected at the right time. The next step, and perhaps one of the most significant challenges, is finding the balance between monitoring and analysis.

    How to Achieve Balance

    Balance is, by far, one of the most difficult decisions in log monitoring and analysis. Gather too little data and risk missing early trouble indicators or important alerts. Collect too much and get buried in analysis paralysis – essentially, leading to the same situation of overlooking items of significance. Striking the right balance requires understanding the system’s continuous monitoring processing capacity and using its full potential. This may also involve data preparation steps during intake to ensure optimal processing and storage of the collected material.

    Offloading mundane tasks and parts of the analysis to machine learning and artificial intelligence (ML and AI) and fuzzy logic systems will enable big data processing gains. With this approach, the continuous monitoring capacity can be significantly increased, along with the semantic quality of the alerts and notifications produced by the system. As a result, security operation center (SOC) staff can devote their attention to the analysis of events which are more meaningful to the system and operations, leading to a more efficient continuous monitoring process.

    A DevOps transformation will likely not reach full maturity unless the organization applies continuous monitoring to the DevOps life cycle. Although often overlooked, it remains central to DevOps success. With this in mind, a basic understanding of continuous monitoring, what to monitor and finding the balance between log monitoring and analysis, serve as a foundation for getting started with continuous monitoring.

  • PagerDuty Applies AIOps to IT Incident Management

    PagerDuty Applies AIOps to IT Incident Management

    PagerDuty, at its online Summit20 conference today, unveiled a bevy of updates that apply machine learning algorithms across a wide range of workflows across the PagerDuty incident response platform.

    In addition, PagerDuty today announced it will acquire Rundeck, a provider of an IT automation platform that auto-remediates incidents by enabling playbooks to automatically execute whenever there is an issue, for $100 million.

    Jonathan Rende, senior vice president of products for PagerDuty, said the overall goal is to make IT teams more efficient by applying AIOps at a time when organizations are trying to reduce to the total cost of managing increasingly complex IT environments.

    At the same time, Rende said PagerDuty is also moving to reduce the fatigue that inevitably ensues when trying to determine the relationship between the thousands of alerts that all the elements of an IT environment can now generate. The goal is to provide IT teams with more context and actionable intelligence, said Rende.

    As part of that effort, PagerDuty is now integrating change events that occur in code repositories via new integrations with GitHub, Puppet and Evolven. The bulk of IT incidents stem from changes made to the IT environment so via these integrations it will become easier to identify the most likely root cause of an issue, said Rende.

    PagerDuty has also added new capabilities to its Service Directory that employs machine learning algorithms to dynamically surface dependencies between people, changes, incidents and services in real-time. As IT environments become more complex thanks to the rise of microservices, he said it’s now more critical than ever to track relationships across workflows spanning multiple IT teams. The challenge is that existing manual processes can keep pace with the rate of change now occurring in IT environments, Rende added.

    PagerDuty is also moving to reduce the number of alerts that are generated by enabling the PagerDuty Event Intelligence module to surface recommendations generated by machine learning algorithms that suggest ways to group alerts to better streamline incident responses. In addition, IT teams can now pause alerts if they decide to allow systems to auto-remediate before they decide to intervene.

    The company is also extending the reach of an application programming interface (API) for analytics it created to enable IT teams to launch queries against the data collected by PagerDuty. In addition, PagerDuty is making available an Analytics Maturity Model that provides access to benchmark data from more than 13,300 customers to both compare their efforts with other IT teams and discover additional best practices.

    PagerDuty today also announced one-click integration with the Zoom video platform to enhance collaboration, as well as extending its existing integration with Microsoft Teams to include a similar one-click capability.

    Finally, PagerDuty also launched an instance of its platform for customer service teams that also provides enhanced integrations with Zendesk and Salesforce Service Cloud. With more organizations dependent on digital business processes to survive, Rende said the need to automate customer service processes has never been more pressing.

    At the same time, IT teams are struggling to keep pace with the rate of tickets being generated by employees who are now working from home. Put it all together and it becomes abundantly clear that continuing to rely on legacy approaches to IT incident management will simply not suffice.

  • DevOps Chat: Alert Management with Everbridge’s Prashant Darisi

    DevOps Chat: Alert Management with Everbridge’s Prashant Darisi

    IT service management (ITSM) has been around for a long time, but its relationship to DevOps has been nebulous at best. In this DevOps Chat, we explore the connections between DevOps and ITSM with Prashant Darisi, vice president of Product at Everbridge.

    Everbridge is swiftly making a name for itself in the alert management space as a pure cloud company whose roots are firmly planted in DevOps. As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Alan Shimel: Hello, everyone, this is Alan Shimel, and you’re listening to another DevOps Chat. Today’s DevOps Chat features Prashant Darisi, VP of Product at Everbridge. Prashant, welcome to DevOps Chat.

    Prashant Darisi: Thank you very much, Alan. Thanks for having me.

    Shimel: Oh, it’s our pleasure. So, Prashant, you know, just let’s get off the bat, many of our listeners may not be familiar with Everbridge. Everbridge is not quite your usual startup in that it didn’t announce funding last week or something like that. So, why don’t you give your listeners just a quick kinda corporate background on Everbridge?

    Darisi: Absolutely. Now, in the simplest single sentence, I would say Everbridge is the global leader for integrated critical event management solutions. We are a publicly traded company and we were founded following the tragic events of 09/11, where we realized that whether it is cross-team collaboration, cross-department collaboration, cross-city collaboration—there wasn’t a system out there that can connect all the dots to make sure that most important and pertinent information about safety of people, safety of enterprise, IT systems, physical assets and non-physical assets can easily ________.

    And so, that’s how this company got started. We are truly born in the cloud. So, the good news for our cloud customers is that we were not a bunch of on-prem companies that we pushed into one of the cloud platforms and said, “Hey, here we come, adopting the cloud.” From day one, a pure SaaS company. [Cross talk] Today we have more than 5,000 global customers, and we serve about 9 out of the 10 largest U.S. cities, nine out of the 10 largest U.S. investment banks, seven out of the 10 largest U.S. software companies, 6 out of the 10 largest global consulting firms, 6 out of the top 10 U.S. universities, 5 out of the top 10 pharmaceutical companies, all the 4 large global accounting firms, right, and 25 of the 25 busiest North American airports use our SaaS platform to ensure critical communications for anything that affects public safety, enterprise IT outages, cyber attacks, or even terrorist attacks and severe weather conditions. So, that’s who Everbridge is.

    Shimel: Excellent. And that’s very impressive, Prashant. Thanks for the background. You know, certainly, alert management alerting is in and of itself something that is not foreign to our DevOps audience, right? There are several sort of alert management, event management players who have really—I mean, they are sort of your traditional startups, right? They’re not public. I think one was actually acquired yesterday for $120 million.

    Darisi: That’s true.

    Shimel: But—you know, but they don’t, frankly, they don’t have the scale and the girth that Everbridge has, right? So, Everbridge was doing this before there was this DevOps movement and we were all excited about continuous delivery and continuous integration and that kind of thing.

    So, but now, you know, what—I guess my question is, Prashant, what has made the DevOps market interesting to Everbridge?

    Darisi: Yeah, for multiple reasons. One is, we have the platform that can support it, right? So, when we talk about how do we go and reach people—well, DevOps, for us, is more of a cultural change, right? I don’t want to pretend here, saying, “Hey, we looked at DevOps, this was the most exciting thing on planet Earth, and we start calling ourselves DevOps.” So, we are a critical event management company, but the beauty of this is that we drink our own champagne. When we talk about continuous integration and continuous delivery, we have, from a pure enterprise perspective, over 3,800 customers, and all of them get our new releases on the same instance, right? We offer it through ________ as well.

    Now, we could not, obviously, do any of this using a traditional waterfall method, either in terms of the velocity, or in terms of quality. So, when we talk about, do you guys use Datadog, do you use Jenkins, Drone, do you use GitHub, do you put your artifacts in JFrog—any of the popular tools that are out there. We’re not trying to promote any one product. We live, breathe, and die by these processes day in and day out. That’s a survival issue, because we’ve never had what we call an on-prem piece. This is the only way we deliver value.

    Now, if that’s what we’re doing and that’s the platform that helps people communicate and collaborate, why wouldn’t we naturally go into the world of DevOps? Traditionally, of course, we started into the ITSM piece of the world, right? This is where the ServiceNow, Remedies, and the HP SM worlds came into the picture. But, as organizations started shifting, Alan, I remember, I think you wrote—what, two, three years ago, that all the CIOs said 88 percent of them will adopt DevOps in the near future? That’s when even we started pivoting—how can we help these folks bring these solutions together?

    So, the reason why we’ve gone here is, we can obviously help folks communicate with each other. We have about, what, 100-plus communication devices, and one of the advantages we have is, why DevOps is great—companies are also starting to spread their development centers across the world. You know, I’ve done development for—what, two decades, as an engineering guy? But I never was on call, right? But look at where the world stands today.

    And so, if, just like technical support, engineers are starting to follow the sun model where you have a communication system that can cut across countries and do language translation, can you have collaboration tools like Slack, Tribe, HipChat, put it all together? Can you help customers do them with automation? Can you do language processing on who said what, what kind of diagnostic scripts were run? Last time when you patched the system, what were the results of it?

    The analytics of those things—we have the platform, and the fact that we manage 210 million contacts worldwide, and the fact that we can send 2 billion messages, that’s the number we delivered in 2017, and we have access to 100-plus communication devices—it became a very natural fit for us from a domain expertise from within the organization, what a customer that’s doing CI/CD needs, and from a platform perspective of what kind of scale and geographic reach you need to have to bring the two worlds together.

    So, I don’t want to pretend that we are born a DevOps delivery company, but we certainly are a born DevOps product company that today is public.

    Shimel: Got it. So, Prashant, you mentioned a couple things, there, Prashant. The first thing is, there’s a very common fact pattern that I’ve observed in talking to many vendors in the DevOps space, and that is that their involvement in the DevOps market started with their own use of DevOps, right? They started using tools like Jenkins and JIRA and some of the others you mentioned, and that their own use and adoption of these tools allows them to kinda see the benefits, right, of using DevOps, and also then allows them, frankly, to have some credibility when they’re talking to DevOps teams in the marketplace, because you yourself are a DevOps team yourself, right? You understand what it’s about.

    Darisi: Yeah.

    Shimel: Understand? Yeah?

    Darisi: Yeah, absolutely—completely understand. And, I think it’s, for us, just look at our own organization, right? And I don’t know how many of your listeners will agree. We have offices in Los Angeles, right here in Burlington where I’m sitting, Lansing, Beijing, London and Stockholm. Now, not all of them are development organizations, because we have grown organically. But we need to scale down that framework and true DevOps practices to keep ourselves, have some semblance of sanity. Otherwise, you know, we go nuts.

    So, I do agree that, because we live and breathe these systems, I hope we are able to deliver our solutions with a credibility both from a domain expertise perspective, but also, a very natural alignment with the tools that the customers are using, right?

    Shimel: Yep. Excellent. And then, Prashant, you brought up another kinda topic that we have been kinda running into and writing about more recently here at devopsy.kinsta.cloud, and that is the whole relationship between ITSM and DevOps, right?

    I think initially, you know, we started devopsy.kinsta.cloud back in 2014, there was this feeling of sort of ITSM goes with waterfall, if you will, or it’s from that generation, and that somehow DevOps and ITSM were mutually exclusive. But the fact is, what we’ve seen in real life is that that’s not the case, right? That there are—ITSM, there is a bridge from DevOps to ITSM and back. And that there are, there’s—you know, I always like to tell people, Prashant, there’s only one truth. I don’t really care what you call God or, you know, we’re talking about religion, right? Good is good, bad is bad, and truth is truth.

    So, the right things that we’ve learned to do in ITSM shouldn’t become wrong things in DevOps, right?

    Darisi: Yeah. [Laughter] Yeah. So, since you used the word God, I will say amen—you hit the nail on the head, right?

    Shimel: Yeah. [Laughter] Okay.

    Darisi: I’ll give you a simple, Alan, here’s the analogy, right? I need to go from here, Burlington downtown, a lot north of Boston downtown, and I need to go from here to there. I could walk, I could take a small car, or I could take a BMW. Now, if my GPS is wrong, a BMW will only take me to the wrong place faster, right? It’s, in the end, whatever we use—tools, methodologies, anything—they’re a means to efficiency. It starts with a cultural change, right? Putting the right processes in place, and then automating these processes for efficiency reasons.

    So, I would argue that—like I said, I just came back from meeting this very large priced customer, oil, and you could feel that the DevOps team thinks ITSM or ITIL or COBIT or any of these things—these are dinosaur practices. It’s not. Actually, you can use the elements of DevOps principles within ITSM. And you’re not going to solve everything using DevOps, because if you look at our shadow IT, there are some systems and infrastructures that are still being held traditionally in data centers that need to be highly secure. Then there is this concept of business driven apps where folks like to use Salesforce.com for their pipeline management. You’re not in control of that any more.

    So, somehow, you have to be able to figure out what’s the dependency of the so-called shadow IT system sitting in the cloud, completely managed by a third party vendor through an SLA agreement, and you’re on prem in your own private cloud enterprises.

    And so, I think the reason why people thought or still think, I think, ITSM is waterfall is this whole notion that, culturally, ITSM or ITIL or COBIT practices are all about who controls, right? Well, rather than talking about who controls, if the same principles can be applied within an IT organization to start enabling the business, start empowering the business to say, “Okay, you are in the better position to make the judgment call on something simple like access request, or what are the things that you need to look for when you deploy a patch or upgrade, and we will take care of the security aspects of it, and if something goes wrong, we will ensure proper communication and collaboration”—I don’t see any friction between the two worlds.

    There’s a tremendous difference in lexicon, for sure, but many, many banks and many, many organizations which are true IT organizations are saying, “Prashant, do you follow the agile model?” Right? “We are doing project management doing agile now. Oh, you’re coming here to do an implementation and integration of DNC Committee or ServiceNow or HP SM? Can your PS implementation team help us break it down across our sprints?”

    So, everybody has now started looking at the value of agile frameworks and beginning to see that, you know, the two worlds are coming together. And obviously, they’re embedding, from an HR perspective, people into each other’s team as well. So, then we talk about site reliability engineer, DREs, ________ operation folks, development folks, and perhaps even infrastructure folks all sitting in a standup or around a standup table. It’s all coming together.

    So, I think this is a little artificial, whether we’re trying to drive this line or a hard line between it, there are definitely different lexicons, but I think there is an easy way to bring these people together from a process perspective.

    Shimel: I agree 100 percent. And I think we’re seeing that kinda play out, right before our eyes in real time now. You know, we’ve seen, for instance, ServiceNow come out with a DevOps related system, right, that brings a lot of this together as well. I’m also involved and one of the co-founders of DevOps Institute, the DevOps Institute, and we have a drill down class on the bridge DevOps to ITSM.

    So, I think that’s all there. But Prashant, as much as there’s a bridge being built and we’re learning how ITSM and DevOps works together, another thing we’re learning is how critical event management works with so many of these DevOps tools—and you mentioned some earlier on, but can you share with our audience, give us an idea, you know, we talk about automation and stuff like that—how does Everbridge help with that?

    Darisi: Yeah. So, we’ll take it at the, let’s say, 10,000-foot level, and I’ll quickly and rapidly try to bring it down to a simple tool. But when we talk about critical event management, for us at Everbridge, a critical event or a disruption is a disruption is a disruption.

    And so, since I gave the example of a tornado approaching a building in Oklahoma City, let’s stick with that. Now, whenever such an event occurs, we know what is going to get impacted, but I’m not sure people do get that. For example, clearly, we know some people will be affected from purely an evacuation perspective. We know that the buildings and the physical assets within the buildings will get affected, and then we also know that we might be hosting a data center or a set of services that are going to get impacted.

    So, how can we help our customers figure out what we have coined as a consequence map. What is the consequence of a critical event or a disruption to a company’s business, both from a people perspective, physical assets perspective, and enterprise IT operations perspective?

    And this is where automation starts becoming very critical. How do you understand, from a CMDB integration, what are the components out there, or which business units are affected, right? So, Everbridge, from our perspective, we do CMDB integration on a very lightweight basis. We obviously don’t want to get into the CMDB business, but our job is response correlation, right? We are not in the business of reading logs and packets and telling you about events. We are not a log miner company, we are a response correlation company. When a critical event occurs, there are a lot of connectors and automated tools that will tell us something is not right. Now, there could be a direct correlation from a response perspective on a denial of service attack that goes and impacts the performance of a website.

    How do we make sure that we are helping our customers correlate their responses, right? How are we able to look at the product and location, the services that are impacted both upstream and downstream and trying to figure out, from a communication and collaboration perspective—Alan, you; Prashant, you—both of you are working on this critical event. One came through an item system, a new relic, or somebody telling you, and the other thing comes from a QRadar or a Resilient kind of a system saying denial of service underway. How do you know that you need to combine forces?

    So, Everbridge wants to be in the business of response correlation, far less about event correlation. So, our automation platform or the smart orchestration platform starts picking up data from these disparate sources, integrates with stuff like the CMDB and looks at the CIs and says, “Hey, I see that these critical events are related,” right? For example, a fire in a building can—it’s police, it’s ambulance, and it’s the IT systems that are sitting there—how do we correlate this? That’s the business we are in. And, for that, you need a smart orchestration platform that is looking at data or alerts coming from these different places.

    And then, when it comes to the pure IT pieces of it, right, how do we help from a DevOps perspective? Well, let’s start with the communication part of it. Can we reach out to these people, whether it is on call calendars, making sure that we follow the proper escalation procedures? I heard something else, Alan, you were talking to somebody, saying, “Hey, people still prefer e-mail,” or that’s still the number one choice. Well, in our Everbridge platform, for certain types of events, based on the criticality, we do not give the choice, even though every individual can pick 100 different types of modalities in which they can receive an alert, but for certain critical events, we will override that. We will wake you up during the morning—because guess what? An e-mail cannot wake you up, right?

    So, it depends on those business policies and rules, and that’s the kinda of work Everbridge wants to do. And then, once you have communicated with these people, how do we bring them together? I mean, just today, right, we had to start GoToMeeting, and if I had used the phone, I would put in a 10 digit number, then a 9 digit code, then an audio PIN. Well, the way we help our customers is, when you receive that SMS text, all you do is press 1, and you’re on the Smart Conference page. No numbers to type, no nothing. And if you are sitting somewhere in Europe, somewhere in the Philippines, India, China, U.S., each one of them will receive a local code. So, nobody’s on an international call. If there is a language translation that needs to get done of the same, exact trigger, we help our customers do that, too.

    So, now that you have a bunch of developers who are all responsible for taking care or of the resolution process, people typically will say audio is great, right, and oh, by the way, recording the audio is great as well. Can you enable us through a collaboration tool? So, this is where the second aspect of it comes, right? Whether it is using Slack, using HipChat, Tribe, Cisco Spark—we bring these people together.

    And all of this is recorded or, as long as the customer prefers, they’re all associated with the IT incidents, and then we want to leverage this data for knowledge transfer, for blameless root cause analysis, a post mortem of the events that happened so that next time when a similar pattern of incidents occur, we should be able to say, “Hey, last time when this happened, go and look at that Slack channel or the HipChat channel. We ran these two diagnostics ________. Have you seen this? Do you want to try this again?” And then, our connectivity on the remediation side, people could say, “Hey, I think we should move to Apache server,” right?

    And the way our infrastructure is set up, you can pick a certain set of Runbook procedures that are automated or, Alan, you could get a call and say, “The team that’s presently on call has decided that they need to reboot the server. Alan, you have been determined as the security architect that needs to approve this. Press 1 to approve an automatic reboot of the server, press 2 to join the conference bridge, or press 3 to enter the Slack channel.”

    This whole process is automated, right? Whether it is smart conferencing or the on call schedule or integration with these collaboration tools or the Runbook automation tools and the recording for archival knowledge base and analytics purposes—everything is done with our platform today.

    So, it can take various shapes and forms, but these are some of those key macro cases, Alan, of how our customers are using our product.

    Shimel: Got it, got it. Alright. Prashant, I told you when we started, Prashant, the times goes so quick here. We’re way over time already, but—so, we’re gonna have to call a wrap. Maybe we can come back and revisit and talk more about this within the larger kinda landscape of what we see today in IT as well.

    But for now, let’s call it a day. Prashant Darisi, VP Product for Everbridge, thanks for being our guest on DevOps Chat today, and we hope to have you back soon.

    Darisi: Alan, thank you very much for having me. Looking forward to another conversation. Thank you.

    Shimel: Okay. This is Alan Shimel for devopsy.kinsta.cloud, and you’ve just listened to another DevOps Chat.

    — Alan Shimel

  • Splunk Acquires VictorOps: The End of the Road for Standalone IT Alert Management Tools?

    Splunk Acquires VictorOps: The End of the Road for Standalone IT Alert Management Tools?

    This week, Splunk, the leader in machine data intelligence, announced the acquisition of IT alerting solution VictorOps for $120 million. My congratulations to the VictorOps team for building a solid product for alert management and collaboration, demonstrating a wicked sense of humor and creating some cool T-shirts.

    While the acquisition makes sense on many levels, it does raise broader questions about the need for standalone IT alerting tools. Let me explain why IT alert management as a category faces strong headwinds and how these tools will need to evolve if they are going to remain relevant in a world driven by digital transformation.

    What Do IT Alert Management Tools Actually Do?

    When an IT outage or service disruption occurs in an enterprise, the first order of business is to restore operations back to normal. IT outages can cost you a lot (an hour of downtime can cost you upwards of $100,000 per hour), disrupt your customer experiences and derail employee productivity. To quickly troubleshoot an IT incident, you’ll need to notify the right subject matter experts who can collaborate together to pinpoint root cause for an outage.

    Every day, enterprise network operations center (NOC) teams receive hundreds of alerts from different IT management tools for monitoring, service desk, event correlation and configuration management. A certain percentage of these alerts demand urgent attention from NOC teams who are on-call and readily available for incident support.

    Enter IT alert management tools, whose job it is to manage the notification of messages to personnel based on predefined rules regarding who to notify for a given categorization of event or incident. Tools from PagerDuty, VictorOps, xMatters, OpsGenie and Everbridge alert the right team members using on-call schedules and notify them using a variety of channels including email, text, voicemail and chat. IT alerting tools help you manage the entire incident life cycle, using on-call schedules, alert routing rules and collaborative war rooms.

    Embedded Alert Context: The Missing Piece of the Alert Puzzle

    The fundamental challenge with scaling enterprise alert management is about establishing the right context for a flood of alerts that you’ve just received. Instead of forwarding every single alert that you receive, can you share actionable alerts that your NOC teams can immediately act upon?

    Your IT alerting tool needs the ability to correlate a bunch of alerts into a single event for faster acknowledgment and troubleshooting. Alert consolidation and correlation call for intelligence that can extract signal from the noise and compress alert volumes with the right inferences. Artificial intelligence for IT operations (AIOps) helps you do just that, by applying big data and machine learning techniques to event management data. With AIOps tools, you can handle events and minimize alert noise by delivering timely and actionable insights for incident support. This agility and insight, coupled with intelligent correlation to stay on top of major outages, helps NOC teams by focusing their attention on the incidents that truly matter.

    AIOps + Alert Management = Smarter Incident Management

    Without the ability to apply machine learning algorithms to alert floods, alert management tools act as “dumb forwarders” of IT alerts, inducing alert fatigue and adding to the overall noise. It’s time for alert management to get smarter and contextual with machine learning insights.

    Splunk’s acquisition of VictorOps is a clear sign that alert management tools as a standalone category can’t truly scale without a solid AIOps foundation. With artificial intelligence, IT teams can stop wasting valuable time figuring out incident triage and root cause analysis and ensure faster recovery from crippling outages with the right event analytics.

    — Deepak Jannu