devopsy.kinsta.cloud

Tag: AIOps

  • How AIOps Makes DevOps Less Noisy

    How AIOps Makes DevOps Less Noisy

    For DevOps engineers, “noise” is the enemy of productivity. In this context, the noise we’re talking about is unnecessary or low-priority alerts and notifications that distract engineers from identifying serious issues—and ultimately can cause alert fatigue syndrome, in which alerting systems are ignored altogether.

    Without the application of a well-constructed noise reduction plan, alert noise will end up taking away from the focus of solving the actual problem that may be causing a more critical issue.

    So, how can you fight alert noise? You might be tempted to assume that the answer lies in reducing the number of alerting tools you use, but that’s probably unrealistic. Teams today are dealing with complex application stacks and multiple tools and data sources, all of which can produce their own alerts for issues that may be interdependent or isolated. You simply can’t take tools out of your stack.

    What you can do, however, is analyze and manage alerts more effectively. Dealing with alert noise ultimately boils down to developing an effective problem analysis strategy. Let’s explore how, with a specific focus on how AIOps factors into keeping alerts manageable.

    Reduce the Time in Analyzing Alerts

    While reducing noise is important, DevOps teams risk losing valuable time solving critical problems if they also have to create and/or add systems to properly analyze alerts and their sources and priority level. Maintaining this system also would be a source of unnecessary labor that may reduce the alert noise, but also reduces the amount of time that DevOps staff can work on the solution to the root cause— which brings up another consideration. When coming up with a plan, the amount of complexity involved in adding the monitoring tools necessary to cover an entire application and/or service can be equally laborious compared to not having a solution at all. At this point, we may be merely shifting the labor emphasis.

    This is Where AIOps Comes In

    Now, let us envision a way that a team can gain full insight into a root cause without exposure to alert noise. The team doesn’t need to analyze every single alert, which would include:

    • Duplicate alerts.
    • Alerts from multiple sources (tools, microservices).
    • Informational alerts (warnings).

    Instead, the team can let AIOps analyze and classify the stream of alerts and identify critical issues. DevOps no longer would need to spend valuable collaboration time analyzing every link in the toolchain. When a team has to analyze every tool, the team could produce false positives, redirect resources toward the wrong solution, spend too much time trying to identify the root cause and create excessive alert noise. Let us envision a way in which the team does not need to spend excessive time in analysis and can experience more effective collaboration in service of solving the actual problem.

    AI and machine learning can relieve this manual process through AIOps. Automated collection of data across the application stack, alert correlation, identifying root cause and decision-making can be leveraged by a team to get to the problem immediately without all of the messy work of manual analysis, sorting and identifying root cause. This level of automation allows for needed human interaction through collaboration across teams, which can result in faster resolution, decreased downtime and effective communication of key events to company staff and end users.

    Analyzing to Predict Future Problems

    Another benefit of AIOps and the power of automated data collection, analysis, issue identification and decision-making is the ability to predict future events. A DevOps team can leverage these insights into effective preventative measures. The ability of AIOps to produce these insights will help a team find ways to automate and/or improve operations. Reducing alert noise no longer needs to be a goal because it is happening through a smooth interaction between the human team and the AI.

    There is a growing need for DevOps teams to utilize the power of AIOps. While human teams are not replaceable, redundant and time-consuming tasks are. The ability of AIOps to effectively traverse a complex toolchain with isolated issues to analyze alert importance, correlate alerts, collect and analyze relevant data, identify root cause and produce holistic actionable insights eliminates inefficiencies in problem-solving and reduces alert noise.

    Making DevOps less noisy is a goal with its roots in effective use of information. Alerts serve to inform, but thanks to the increasing complexity of applications, alerts become noise. Leveraging the power of AIOps serves to do a lot of the dirty work for a team so that noise can be reduced—but more importantly, it can make operations run smooth and keep customers happy.

    — Phillip Lorenzo

  • How AIOps Helps You Get More Out of DevOps

    How AIOps Helps You Get More Out of DevOps

    DevOps may seem like an overused term. It involves taking an application’s source code and running it in an environment. It can cover the processes, the technology or even the people that maintain the technology that is running those very processes.

    At its heart, though, DevOps is about helping developers to be self-sufficient when it comes to the basic operations around getting their application to a real environment. Once things are running in a test or production environment, there are other IT functions that are required to keep it running and performing well. While there are teams that specialize in these functions, such as site reliability engineers (SREs) and system administration, they are usually overwhelmed by the sheer amount of data their areas of responsibility generate without considering the data that applications going through their DevOps processes are also creating.

    This is what makes AIOps the perfect complement to the DevOps space.

    What AIOps Does

    What AIOps does best is break down silos within IT. Having data from all the functional areas processed and handled by modern machine learning (ML) in a single place allows correlation, which leads to faster problem identification and resolution.

    Before machine learning was introduced, for humans to process data, more or less they operated in three ways:

    • They knew exactly what data to look for, such as when a specific business function isn’t working. There is a typical message in the monolithic application that handles all the core business.
    • They routinely sifted through data only from their area of expertise.
    • When they went outside their area, any logs and other data they had access to often was summarized data, which has time gaps and can miss a lot of detail that proper correlation needs.

    Now with ML, the AIOps data platform can perform trend analysis, look for patterns and exceptions across all the data that is being generated in the environment and highlight what it considers important through web interfaces or using an event management solution for after-hours support.

    Over time, as the AIOps platform continues to learn the environment, it can be given the ability to automatically resolve certain error conditions, as it will be able to identify them faster than human operators and often resolve them before they become service-impacting.

    How This Helps DevOps

    Machine learning aids DevOps processes in three ways when working with logs and data:

    • Knowing exactly what data to look for: In a true polyglot DevOps environment that is leveraging microservices, it is next to impossible for any one person to know the most common error messages that might occur. In this model where a new business function can be conceived, written in one of many languages, and deployed into production without Operations even being notified, ML allows the AIOps platform to quickly identify what “normal” looks like for a new microservice and start looking for anomalies.
    • Only routinely sifting through data from their area of expertise: When working in DevOps, developers need to be able to quickly find data from multiple areas in IT that is related to whatever they are trying to diagnose. Using AIOps, they will be able to quickly pull data from systems directly related to their incident. Whether it is as simple as using timecodes or a correlation GUID (globally unique identifier) that is put in the logs purposefully to aide tracking, AIOps will guide the developer through the other systems’ data.On the flip side, if there is an infrastructure problem, the same AIOps platform can help correlate what downstream services may have been impacted, from failed builds to Kubernetes pods that were restarted.
    • Summarized and incomplete data: The true value of machine learning is that it has the ability to process a higher volume and variety of data at a much higher speed and accuracy than a human, and sort and index it for later use. Not having to summarize the data to one data point every five minutes could result in a problem being discovered that has occurred often but not consistently, as it may have been dropped in the summarization.

    Conclusion

    AIOps is very young in the technology industry, but has great promise as it builds on previous generations of IT operational analytics products, combined with the latest in machine learning capabilities. DevOps is a key field in IT that has more logs than it knows what to do with. Combining the technologies and expertise in these two growth areas will have a huge positive impact on productivity across enterprises with improved service management.

    — Vince Power

  • DevOps Chat: AIOps and APM with Unravel’s Kunal Agarwal

    DevOps Chat: AIOps and APM with Unravel’s Kunal Agarwal

    Unravel is making a name in the AIOps, APM and big data markets. In this DevOps Chat, we speak with Unravel CEO Kunal Agarwal about what makes Unravel a must-have for its customers. We also take a sneak peek ahead to what the near and not so near future holds for big data, AIOps and the tech scene in general.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, this is Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Today’s DevOps Chat features Kunal Agarwal, CEO of Unravel or Unravel Data. Kunal, welcome to DevOps Chat.

    Kunal Agarwal: Thank you, Alan. Thank you so much for having me here.

    Shimel: Excellent. It’s my pleasure to have you here. So, Kunal, let’s get right to it. “Unravel” is the name of the company, correct? Not “Unravel Data.”

    Agarwal: “Unravel,” yes.

    Shimel: But yet Unravel is about data to a certain degree. I don’t know if everyone in our audience is familiar with Unravel. Why don’t we start there?

    Agarwal: Sure. We are unraveling the mysteries of big data, Alan. That’s why the name of the company.

    Shimel: Excellent.

    Agarwal: Everybody’s jumping on to big data apps, and what I mean by that is you may be doing something with machine learning, artificial intelligence, IoT, even big-scale data warehousing. And there’s just so many problems that can happen, running these applications at scale, that people just have not seen before. And this is a problem because of all these new systems and technologies that are coming about, such as Spark, Kafka, Hadoop, Cassandra, HBase, many, many more. And the complexity’s all about “Hey, how do I figure out why’s my application not performing well or if it’s failing?” And it’s usually a multi-tenant environment that these big data applications run on, which means there may be hundreds of users, thousands of applications, that you are creating every day and running them.

    And figuring it all out and making sure that they run well is a black art, so there’s a lot of trial-and-error that goes on into fixing these problems, finding the root cause of these problems. And that’s what we are automating, meaning Unravel, after it’s installed in your big data cluster, will be constantly monitoring and managing everything. And if there is an issue, Unravel will be proactive about it, tell you why that issue happened, what caused it, and then even take a step further where it would resolve those issues for you so that you can spend time creating all these amazing big data applications, rather than focusing on fixing all these systems- and application-level errors.

    Shimel: Mm-hmm. So, you know, I mean, there’s a lot there that we’re gonna dive into, but before we do, Kunal, just give our audience – ’cause I always find this fascinating. People wanna know “How do I become a CEO? How does one get to be the CEO?” Give us a little insight into your own personal journey on how you’ve come to be CEO at Unravel.

    Agarwal: CEO is “chief everything officer,” when you first start off. You gotta sweep the floors and you gotta raise the funds and you’ve gotta grow a team. There’s so many things a CEO’s role requires to do, especially in a young company, as you’re growing it. Today, now that we’ve grown to about 75, 80 people in total, across 3 different countries – we actually have offices here in the United States, one in London, and one in India – my job is becoming more about recruiting, more about getting the team together.

    And I think that’s, personally, for me, if that is taken care of, the entire company just hums and does well. Everything else is secondary for me at this time, but I do spend a lot of time doing marketing and sales and finance and product. So it really depends on the day, Alan, of what my job requires me to do.

    Shimel: Amen. You know, I’ve founded or cofounded three or four companies myself and even just running MediaOps, which is devopsy.kinsta.cloud and Security Boulevard, you are chief cook and bottle washer when you’re – you’re chief everything officer, as you said.

    Agarwal: Right.

    Shimel: You know, looking in your background, you’re a double-major graduate – electrical engineering, computer science master’s degree from Duke. I mean, you’re well positioned to be a CEO, people would say, looking at the background, but it’s – you know, you never know what’s coming out next as the CEO. If I had to ask you, Kunal, what’s kind of been the biggest surprise or biggest kind of curveball that you’ve seen in your time at Unravel that you think, “Geez, I didn’t think I’d be doing this or I didn’t think this was part of the job,” what would you say?

    Agarwal: Alan, actually, going to school and getting all those double-degrees does not help you at all becoming CEO.

    Shimel: You get it. I mean, you’re talking to someone who went to law school. I know all about it.

    Agarwal: Right?

    Shimel: Yeah.

    Agarwal: I wish there was a school for CEO, but you know what? The only school for being CEO and leading a company is actually leading the company because there’s so many things that you learn along the way. One thing, because we deal in enterprise software, there’s so many complexities to making a deal happen with a big company.

    Shimel: Mm-hmm.

    Agarwal: And I think that was one of the things that took me a while to grasp and make sure that we’re doing it right. And I mean all the boring stuff, like the contractual paperwork, all the legalities, going through info-sec requirements. All the operational stuff around making a deal happen. You know, when you’re in college and you’re starting to think about a startup, you think about the cool products you’ll make. You’ll think about the impact your product will have in the market and how so easily everybody will flock and come to you and get your product. [Chuckles] But it’s not that simple.

    Shimel: I wish it was.

    Agarwal: and it takes 9 months to 12 months to close some of these deals, so I think, more than the “surprise” factor, preparing yourself and making sure that you can drive efficiencies in all these different processes. I think that’s been one of the key challenges for us.

    Shimel: Absolutely. Absolutely. So, Kunal, if I’m a Silicon Valley VC, I’m looking at a company like Unravel, man, you’re playing in all the right places. Big data is certainly such a huge – I think, when we talk about things like machine learning and AIOps, really, they’re almost the children of big data – you couldn’t have those things without the big data, first of all – so you’re in the right place, obviously. The right time. This is a very happening kind of space. What is Unravel doing that is so – what’s the special sauce? What’s unique about it?

    Agarwal: Yeah, Alan. So, when we created this company – and by “we,” I mean myself and my cofounder, Dr. Shivnath Babu, who’s actually a professor of computer science at Duke University – we both met at Duke – and the problem we were trying to resolve is one of the most fundamental but also probably the most complex and challenging problem to solve, which is “How does one ensure that all of these big data technologies actually work and deliver on their promise?” And, you know, our approach to the problem was always “Hey, there will be a shortage of experts in the market. Even if there are experts, these problems will be requiring a machine to solve them on time and efficiently.”

    So we took the long path of actually creating not just a monitoring tool but a AIOps tool, if you may, which can decipher the problem, connect the dots, and resolve it for you. So it actually took us about three-and-a-half years to build this product or at least the first version of this product. And we moved west to start working with some companies – I lived in New York; he was in North Carolina – and we gave up our cushy jobs, moved to Silicon Valley, because you wanted to work with companies who were using big data tech at scale.

    And some of these companies, we still have advisors in, from Zynga, Rocket Fuel, Twitter, LinkedIn, and they really helped us understand the day-to-day issues that come up when you’re running these technologies at scale. And we took a lot of those learnings into our first product. So we were always about solving a core need that teams in big data tech would definitely need. And, you know, they always say, “Hindsight is 20/20,” right?

    Shimel: Yeah.

    Agarwal: But we always put a big bet that, “Hey, these technologies – Hadoop, Spark, Kafka, et cetera – will become mainstream and companies will start using this a lot more.” So what LinkedIn was doing five years ago, the biggest banks would start doing it now, and that’s where we are in the journey today.

    Shimel: Absolutely. And, you know, it’s funny, full circle, talking about that journey and what you’re doing. That’s how you become a CEO. It’s about being willing to take the risk, giving up those cushy jobs. Right? To do something that you really believe in. And I think that’s what separates CEOs from people who kinda look and say, “Boy, I’d like to do that,” but they don’t actually do it.

    Let’s talk about the market. We’re coming to the end of the year. You’re participating in our Predict 2019 panel, which will be December 18th event. You’re gonna be on the AIOps panel, looking ahead. Why don’t you give us a sneak peek? What’s kind of your take on what 2019 holds for AIOps and big data and so forth, Kunal?

    Agarwal: Well, yeah, Alan. So AIOps is an evolution of IT Ops or APM or DevOps, where you’re using machine learning to, again, understand all of the data – and this could be monitoring data, instrumentation data – that all these systems and applications are generating to draw insights from.

    Now, funnily enough, this is what our customers do with their data set, right? So with some of the biggest banks, some of the biggest insurance companies, some of the biggest health care companies, all Fortune 500. And our methodology of going about monitoring for big data stack was always the same, which is “Why aren’t we using big data technology on our data set, which is machine-generated data, to glean those insights and drive some prediction and actually make things simpler and more accurate?”

    So AIOps is also one of the use cases of artificial intelligence and machine learning. Now, just as I’ve said the industry’s evolving, what we predict to see is the evolution of AIOps, so Unravel, for example, is already on one of the end-of-the-spectrum places, where we actually make decision on behalf of the operations and the development teams, whether it is re-evaluating an application, whether it is making sure that your SLAs for your applications are being met, by moving low-priority jobs to a sub-optimized queue and high-priority jobs to a priority queue or killing bad applications in the cluster, all on your behalf, to make it more proactive.

    But what I predict is there’s gonna be an evolution in just the incumbent APM companies as well, where it’s not just about giving people a monitoring screen or even a native monitoring screen and saying, “Hey, check out all the information we know about an application,” but then I leave it up to you to decide and understand what’s amiss, what’s wrong, and how do you fix it. But I feel we’re already now in the correlation and the insights phase, which is now I can start to connect the dots, with these new software technologies that are out there. But, in 2019, we’ll start to also see some of these companies take the leap to be able to do automated root cause analysis, as well as guided remedies or automatic solutions, to get out of these problems as well.

    This industry’s evolving at a very fast pace, so I’m super excited. And it’s all about making sure that some of the biggest applications in the world, which millions of people actually rely on, all work like a well-oiled machine. And it’s all about guaranteeing reliability, which, at the end of the day, improves confidence of the customers using those products, which further fuels more innovation and more growth. So it’s an amazing cycle and AIOps is gonna be a very, very key part of it.

    Shimel: No doubt about it. No doubt about it. You know, in many ways, I look back on 30 years in IT and it really is a golden time to be alive, if you’re a developer or ops person or just in IT in general. Right? There’s the possibilities and the tools, and what’s in front of you is so exhilarating, right? I mean, I think back, if I was 20 years old now, 22 years old, coming into the –

    Agarwal: So you mean like 10 years ago, Alan?

    Shimel: Yeah, something like that. About 10.

    Agarwal: Okay.

    Shimel: About 10. And, I mean, just unlimited opportunity. I mean, that’s kinda almost the way I look at it. And, of course, it’s companies like Unravel, right, that are kinda letting these opportunities take shape and packaging them for people to run with. So great stuff. Specifically, Unravel – we only have a couple moments here left, but – so, Kunal, anything on the horizon with Unravel you wanna share with our audience?

    Agarwal: Yeah, Alan, absolutely. So, even in big data technologies, there’s so many to cover. We started our life off supporting Spark, Kafka, the entire Hadoop stack, some NoSQL systems, like HBase and Cassandra, and some newer NPP platforms. In 2019, we’re gonna further expand some of those technologies and systems that we cover, especially as we start to see more and more big data workloads move to the cloud.

    We’re also seeing a lot of our companies demanding a hybrid real estate management from the cliched “one pane of glass,” that “Can I see everything that’s in my big data environment, whether that’s on premise and on the cloud, together?” So Unravel is launching that support as well.

    And, last but not least, we are also creating a solution, from our product, which is directed towards helping companies move from their on-premise environments to cloud environment. So actually using Unravel as a migration project management tool, to help you assess, migrate, and then validate your migration. And, on the assess side, figuring out “What system should I get? Which instance type is best for my workloads? How can I keep under cost? What is the cost of running my big data workloads on the cloud?” to then validating and making sure that your SLAs are met, that you’re under cost and under time, and everything is flowing like a well-oiled machine.

    The way we look at it, Alan, again, the way to look at our road map and anything on the horizon is we are here to help our customers in the entire journey of big data tech, so, whatever they’re deploying today and they wanna use tomorrow, Unravel will be supporting that. And this is a natural evolution where companies are going from trying something out to getting something in production, first starting on premise and then going to the cloud, and those are all the shifts and transition points that we wanna hold the hands of the customers that we work with.

    Shimel: Excellent. Excellent. Well, Kunal, thanks for spending a few moments with us. As I told you, the time goes quickly.

    Agarwal: Yeah.

    Shimel: We’re way over time, but it’s okay. Hopefully, people have stayed on till now to listen. I wanna wish you lots of continued success with Unravel. We’ll be checking in on it. Again, a reminder: Kunal will be on our December 18th Predict 2019 virtual event and Kunal will be participating in the CEO panel, taking a look ahead at AIOps in 2019. So join us. You can register for that at predict2019.com.

    Agarwal: Thanks, Alan.

    Shimel: Thank you. Hello to all of our friends at Unravel. And that’s gonna make a wrap on this episode of DevOps Chat, everyone. This is Alan Shimel for DevOps Chat, devopsy.kinsta.cloud. Have a great day.

    — Alan Shimel

  • Unlocking the Power of AIOps

    Unlocking the Power of AIOps

    AIOps is becoming quite the hot topic lately. It seems as though everyone is realizing that Ops is difficult. As we have to deal with more big data at greater velocity with more automation and complexity, there is no way humans alone can tame the beast. AIOps seems to be the answer.

    I recently sat down with Dave Link, CEO and founder of ScienceLogic to discuss how to unlock the power of AIOps to empower IT ops. We video taped the conversation, along with some accompanying slides that help explain what the challenges are that ops faces and what are some of the ways to tackle this using AIOps.

    I should mention that Dave and I are old friends, having worked together 20-plus years ago at a dotcom era company called Interliant. Since 2003 or so I have watched Dave build ScienceLogic into the powerhouse it is today. I couldn’t be more happy and proud of him.

    ScienceLogic has been named by EMA as a top vendor in the AI/ML for IT Operations. Dave gives us a good background on ScienceLogic as well as a good background on the use cases and challenges that IT Ops faces that AIOps seems to be the answer for.

    Dave also talks about the differences between AI and machine learning or ML. This seems to be an area of confusion today as well.

    The video is about 30 minutes and excuse Dave and I reminiscing, but there is some really great information here.

    Video on AIOps

     

    — Alan Shimel

  • A Vendor Guy Goes To Monitorama

    A Vendor Guy Goes To Monitorama

    I attended my very first Monitorama this year, in Amsterdam, and I have thoughts!
    Monitorama Amsterdam
    Monitorama Amsterdam

    First of all, it’s deeply weird speaking at Monitorama when you work for a vendor in the monitoring and observability space, but you’re not there as a sponsor. This would be pretty odd at the best of times, but the crowd at Monitorama are next-level. They are very comfortable with highly technical content, so they get irritated very quickly with (what they perceive as) marketing fluff. “Down in the weeds” doesn’t even begin to describe it; most people had never heard of the sorts of things that are part of life in enterprise IT (AIOps, Netcool, Patrol or any of our usual suspects), but, on the other hand, everyone I met could engage in a multi-hour argument over which monitoring tool is worst.

    In fact, I would say that one of the highlights of Monitorama was the conversations in the hallway track. This is where all the Twitter loudmouths—I mean, “influencers” and “thought leaders”—who drive this industry congregate. If you manage to get into a conversation with them over coffee or beer, both of which were flowing freely, there can be a real exchange of ideas. I know I learned probably as much from those small-group chats as from the presentations on the big stage. I hope I was able to share some perspective in return.

    Heck, as ridiculous as it sounds, even the dress code was a challenge. At the events I attend as a vendor, there are basically two speeds: full suit-and-tie battle dress or a T-shirt with my employer’s logo. Neither case requires much thought when packing. Monitorama, though? I’m no longer credible in my old Thinkgeek shirts from my sysadmin days, but I’ll be laughed out of the room if I show up in a tie.

    Preparation

    I had answered the Call for Papers with a pitch about AI for distributed systems. Here’s my abstract:

    The IT industry is in the middle of one of its regular swings between centralization and decentralization, driven by increased automation of the network fabric itself, as well as new use cases such as IoT. With more and more processing and autonomy devolved to the edges, old assumptions about how to manage and operate a network have to change. It no longer makes sense to try to forward all the edge alerts to a central location for analysis, but central visibility on the health of the network is more important than ever.

    New techniques in AI-enabled observability hold the promise of helping NOC teams deliver better experiences for users of their networks, without requiring excessive manual work or heavy impact on on Operations teams’ personal lives. Machine learning enables knowledge to be captured and made available in context when it is most useful, accelerating incident resolution and improving user experiences.

    My first hurdle was that, although I have public-speaking engagements at least once or twice a month—and have done for some years now—I normally appear under the flag of my employer. This fact has both pros and cons. On the positive side, I can usually rely on a substantial library of slides that I (and others) have created in the past. I have some known high notes that I will be aiming to hit, and the call to action at the end is pretty explicit (“try our stuff”). The other side of the coin is that, because the audience is aware of these factors at least at some level, I have to work that much harder to earn their attention, as they assume they are being sold to. As a vendor spokesperson, I am also somewhat constrained in how wacky and off-the-wall I can get—although I do always try to push the boundaries. At one speech last year, I talked about bicycles for several minutes, to the consternation of the organizers, until they worked out what I was building up to.

    So I slaved away for some time over a hot Keynote, watched videos from past Monitoramas, consulted people in the industry whose opinions I respect and rocked up in Amsterdam pretty pleased with myself—to the point that I stayed out at In de Wildeman perhaps later than I should have on the night before Monitorama kicked off.

    Not Enough Preparation

    As speaker after speaker took the stage on Tuesday, I sank deeper and deeper into my seat, realizing that I had completely missed the mark—and as if that were not bad enough, several key analogies in my presentation had already been used by the first-day speakers. I did go to the party at Delirium Café, but left early to sleep the restless sleep of the nervous speaker. In the wee hours of the morning, I gave it up as a bad job, and spent several hours basically rewriting my entire talk in the light of what I had seen. Goodness knows what my neighbors thought of me rehearsing my lines, although I tried to keep the noise down.

    Here is the result in Slideshare, or you can watch the video here. That link should take you to the correct time stamp, but if not, scroll to about 5h30′ in.

    Katja Budnikov was also doing amazing sketchnotes (live!) of all the presentations. Here is the one from my session:

    Relief

     

    In the end it seemed to go down pretty well. I did my level best (honest!) to call on my own now-obsolete experience as a practitioner and on conversations with people who are more current than I am with the nuts and bolts of modern IT, rather than relying on the crutch of vendor boilerplate, and I think I was mostly successful in that.

    If you are working for a vendor and considering submitting to the Call for Papers at this sort of event, whether it’s Monitorama, DevOpsDays or Config Management Camp, I would definitely suggest test-driving your content as widely as possible. You probably have a lot of subconscious reflexes as a speaker that are not appropriate for the sort of audience these events attract. Even when you are conscious of them, it’s hard to eradicate them completely, but do your best.

    In the other direction, while I have enormous respect for the technical level of Monitorama attendees, some of them seemed to be at risk of losing track of the big picture. I would not want Monitorama to change its character as a techie’s conference, but I am grateful that we had a few presentations outside that mold, and I hope that I was able to provide some context for how all the wonderful technology gets used out there in the big bad world.

    — Dominic Wellington

  • Rethinking the APM Paradigm: A New Vision for Enterprise Application Monitoring

    Rethinking the APM Paradigm: A New Vision for Enterprise Application Monitoring

    Traditional APM systems today can’t measure up to tomorrow’s processes

    IT today is inordinately complex; continually pushing the boundaries and comprehension for even the most sophisticated minds. And while modern IT is unlocking new, innovative experiences and capabilities, the advent of cloud computing and its distributed, ephemeral nature has also made it extremely difficult for IT managers to keep apprised of their infrastructure stacks.

    This reality has taken shape so swiftly and with such a vengeance that Gartner estimates by 2021, enterprises will be able to monitor only 20 percent of their applications accurately. In a digital world, that is like a department head saying he or she knows what happens in only 20 percent of your factories or storefronts. Twenty percent, regardless of the industry or circumstance, is both unacceptable and a grave risk to business continuity and success.

    APM Helps, But Not Much

    Enterprises need to know what the other 80 percent is doing. Unfortunately, the traditional application performance management (APM) tools on which enterprises typically rely are no longer up to this task for three reasons.

    First, the modern application suite looks very different today than it did just a short time ago. The monolithic and custom ERPs, CRMs and other enterprise systems that were the hallmark of big businesses a decade ago are giving way to cloud-based, off-the-shelf, as-a-service solutions constantly updated with iterative DevOps and agile development techniques.

    Second, most traditional APM tools rely on embedded monitoring agents installed on the monitored system, or they need code-level insertions tactics to monitor. Either way, that type of burden is unsustainable at today’s enterprise scale and doesn’t work when applications are launched in containerized cloud environments. It also fails to account for the shift to more off-the-shelf or as-a-service solutions created without APM data outputs.

    Finally, the very infrastructure enterprise operates on looks nothing like it did even five years ago. Most enterprises have moved (or are in the process of moving) to cloud, multi-cloud, or at least hybrid infrastructure. The advent of cloud and the ephemeral nature of microservices and containers means that application topology looks very different at 8 a.m. when only early-bird employees are in the office, then it does at 10 a.m. when the remaining employees arrive. That unpredictability is a nightmare for APM tools, which, at best, now offer only a keyhole view of the entire IT enterprise; performance management now relies as much on the infrastructure as it does the application.

    So, What Needs to Change?

    Traditional APM tools are battleships in an asymmetric warfare age: highly specialized and designed for a single mission, but at the cost of the flexibility to adapt to modern environments as needed. We need carrier groups, not battleship armadas.

    The ideal APM solution for modern enterprises is an “agent-less” or “agent everywhere” approach. A modern solution must holistically consider how one application interacts with other applications. For instance, how does online ordering generate data and feed into warehouse processing tickets? And it must understand how those applications interact with the underlying infrastructure. For example, where are databases or applications stored and how reliable is that infrastructure?

    Current APM solutions miss the mark entirely, scratching only the most superficial layer of today’s technology stack when instead, they need to understand the relations from the deepest layers of infrastructure out to the end users’ experiences. Such an approach will allow enterprises to see into the other 80 percent of applications and deliver the insight and automation needed to drive robust and innovative digital experiences.

    However, knowing you need visibility and having it are two very different things. Fortunately, innovation within AIOps through agent and agentless technologies promises to deliver what conventional APM can no longer.

    By making visibility into application layers easier and deploying machine learning-enabled topological mapping between infrastructure and applications, AIOps is allowing for more detailed insights into the health of mission-critical business services than ever before. It allows enterprises to regain the visibility they’ve lost to an ephemeral world and be able to understand, organize and innovate with the agility they need to compete in today’s business environments.

    It is time to leave traditional APM behind and move forward into an AIOps enabled world. Neglecting to do so puts the business at ever-greater risk.

    — Erik Rudin

  • Measure What’s Important: DevFinOps

    Measure What’s Important: DevFinOps

    Lately there seem to have been quite a few DevSomethingOps terms being coined. However, I do think we need one more: DevFinOps.

    Dev-What-Ops?

    One of the biggest problems in IT is working out the cost of IT. Left unchecked, IT will expand to fill whatever budget it is given. This may mean evermore involved business continuity and redundancy, reinventing evermore elaborate wheels, irrational versus buy decisions and so on.

    In fact, the whole idea of cost estimates for IT projects has become something of a joke. Most IT projects are reputed to exceed their budgets.

    There is an entire #NoEstimates movement that rejects the idea that it is even possible to deliver a sensible estimate for a software development project. As is often the case, the basic idea is reasonable:

    it is possible to do small chunks of work incrementally, leading as rapidly as possible to a desired shippable product, and that when you do that there is no need to do much of anything in the way of estimating stories or the project.

    However, the simplistic interpretation that is all over Twitter is that because there is no estimate of the  overall project, there is no estimate at all. That is not the idea! It is simply to break down the overall estimate (which is probably wrong) and instead budget for much smaller chunks of work, which can presumably be estimated more precisely and accurately.

    So What Can We Do?

    Maybe what is required is integration of financial concerns directly into the fabric of IT development operation. We might call this DevFinOps, in line with other extensions of DevOps such as DevSecOps or DevNetOps.

    This is not such a leap; after all, we already integrate costs in areas such as SLA calculation, estimating the TCO of a tool or the ROI of a proposed purchase. However, these are generally calculations that are made after the fact, and may even be retconned to fit external assumptions.

    What I am proposing, though, is to build this information right into the IT metadata and processes to enable us to make better decisions.

    This is not IT Asset Management, although it would build on those data and extend their reach. A particular device or piece of hardware has its purchase price and current amortised value documented in the ITAM database, but the actual value it has at any given time might vary depending on which business function it is currently supporting. This calculation should change over time to reflect the real current value it is delivering to the organization at any given time.

    An idle server has no particular value beyond the intrinsic value of the hardware, back-end support (floor space, network, cooling, etc.), and perhaps any associated licensed software. However, the same server will take on a much higher value to the organization when it is provisioned as part of the infrastructure supporting a major business-critical application.

    Performing actions on these systems would therefore have different potential impacts, and should be treated as costs deducted from a limited budget. Live-patching a business-critical server is an expensive operation, not to be treated lightly. The cost of doing so should be deducted from an operational budget.

    This calculation has the benefit of making it easier to communicate the different impacts of various proposed courses of action outside the IT department. Deciding between cloud and on-premises deployment options, or between different flavors of cloud, can be a fraught process. Technical options and pricing models are rarely easily comparable, and then there are other costs such as compliance (yes, and GDPR) or vendor risk. Quantifying and normalizing these disparate factors would make it that much easier to compare options up front and calculate benefits after the fact.

    Is This Even Possible?

    Google already does something like this in its Site Reliability Engineering (SRE) program, with the notion of an error budget:

    The business or the product must establish what the availability target is for the system. Once you’ve done that, one minus the availability target is what we call the error budget; if it’s 99.99% available, that means that it’s 0.01% unavailable. Now we are allowed to have .01% unavailability and this is a budget. We can spend it on anything we want, as long as we don’t overspend it.

    There is a long and sordid history of attempts to make programming projects comparable and their estimates more reliable. Often this devolves to simply counting lines of code, according to the age-old management syllogism: “We need to measure something; this measurement is easy to gather; let’s measure this!” and of course the Mythical Man-Month.

    Fortunately, there is a way to calculate the complexity, and therefore the potential cost impact, of a piece of code: the Halstead volume. This is a measurement of the complexity of code, described in terms of the number of distinct operators, the number of distinct operands and the total number of operators and operands. While it is not guaranteed to be perfectly accurate in every case, it is at least accurate enough to be comparable, and therefore can be used to estimate the risk of a change, in terms of the relative Halstead volumes of the code to be changed and the patch to be applied.

    But What About the Squishy Meatbags?

    Connecting the human costs would also enable other metrics to be gathered. It has been historically difficult to differentiate between what is urgent and what is important, and how to prioritize each. If we connect salary data (including overtime) to changes, it becomes much easier to make the call on when enough time has been wasted on a problem and the cost-effective solution is to start from scratch.

    It sounds like a lot of work to gather all of this information, but these data already exist; they’re just not connected up in one place. When the business wants to launch a new service, there are (usually) pretty rigorous calculations behind that decision. The expectation is to acquire or retain a certain number of customers who have a certain average lifetime value and break-even points and other criteria are all mapped out ahead of time. For whatever reason, though, these islands of information are rarely if ever connected up.

    This, incidentally, is just one reason why it can be worthwhile to work with reputable and experienced vendors. I have written before about what this process looks like. The good vendor sales teams will help to unearth this sort of information—but even in the most positive sales processes, the kimono is rarely fully open. There is far more information inside the organization that could be enormously useful if only it were available in the right place and at the right time.

    Longer term, this would mean that business cases and ROI calculations would finally be concrete and factual, rather than extended performance pieces in the medium of spreadsheets. On that note, there is a wonderful little book, titled “How to Lie with Statistics,” which is even available as a free PDF. It’s one of those books that is fun to read because you start spotting the things it was talking about simply everywhere.

    If you wanted to be maximally Zeitgeist-compliant, you could probably verify all of this with blockchain, although I worry that the transaction costs (including thermal costs!) would be prohibitive. The good news, though, is that you could use this DevFinOps methodology to work that out too …

    All That Effort, for What?

    Bottom line, the goal is to be able to calculate (at least within reasonable limits) how much IT is costing today, and what the impact of any proposed changes would be—and crucially, to do so in real time.

    The goal is agile cost estimates. There is a reason I chose the DevFinOps coinage. The way we do cost estimates today is very “waterfall“: large teams have extended arguments over spreadsheets, and the process is so painful and expensive (in person-hours and disruption) that it is only undertaken for the largest changes.

    Agile’s faster release cycles do not align at all well to this process, which is where #NoEstimates is coming from. However, if we start gathering and processing all the cost data in real time, and make it available via a simple API call, it becomes possible to calculate both ends of the cost of a proposed change: the complexity of the code itself, and therefore the probability that it contains bugs, and the infrastructure impact that the deployment will have. This cost can then be compared to the team’s current SRE error budget to determine whether the change can be made automatically or whether it requires signoff as an exception.

    The result looks a lot like ITIL’s notion of a “standard change”, but without the inherent conflict between ITIL’s centralization and SRE’s decentralization. The key is a common accepted measurement of what a standard change actually is, which would otherwise require the ITIL Change Manager to make the determination ahead of time.

    What Does This Look Like in Ops?

    Let’s not forget Ops, either. Proper cost estimates like the ones described above would enable Ops to prioritize alerts correctly and estimate impacts accurately. Two outages with equal technical severity may have very different business impacts. Today, that reconciliation relies on tribal knowledge and systems that are often inaccurate or out of date. Given a central shared repository of data that describes the value of deployed systems, it becomes possible to calculate those impacts much more accurately.

    While we work to get to full DevFinOps, you might want to consider AIOps, which helps gather a contextual view of what is actually going on in IT. Gartner just released a Market Survey for this emerging space. Normally these documents cost $1,295, but you can get a free copy here, courtesy of Moogsoft.

    Dominic Wellington is Chief Evangelist at Moogsoft. He has been involved in IT operations for a number of years, working in fields as diverse as SecOps, cloud computing, and data center automation.

    — Dominic Wellington

  • AIOps: Managing the Second Law of IT Ops

    AIOps: Managing the Second Law of IT Ops

    Or, why IT systems keep breaking

    The tension between the two halves of DevOps is long-standing, going back long before the term was even invented. What underlies the split between Development and Operations is a fundamental difference, like an actual physical law. Let’s explore that analogy together.

    IT Ops is fundamentally a constant fight against entropy, meaning disorder. Sysadmins expend significant energy trying to keep everything up and running, in a process which can sometimes feel like trying to shore up a sandcastle while the tide is coming in. Systems are constantly breaking down; users need changes made that introduce risk; and, as if that wasn’t enough, the whole shebang is constantly under attack from malicious outsiders. It’s no wonder that sysadmins have a reputation for being a bit grumpy!

    Homer: In this house, we obey the laws of thermodynamics!
    It’s not just a good idea, it’s the law.

    In contrast, Dev actually increases the level of entropy. Every new release, be it as small as a one-line fix or finding that errant semicolon, is a change that has to be made, reducing the order that Ops fights so hard to maintain. Dev is entropogenic, embracing change as a positive force.

    The interaction between Dev and Ops is analogous to the second law of thermodynamics: Entropy can only increase over time, unless energy is introduced to the system from outside. In other words, unmanaged change will eventually cause something to break, unless people put in continuous effort to prevent those breaks.

    Thermodynamics: Lots of Heat, not Much Light

    These conflicting goals, taken to extremes, can get unhealthy, with Ops actively fighting change or Dev going around Ops to increase their velocity of change. Left unchecked, this tension can turn into a vicious cycle, as frustration with the “Department of No” drives developers into unmanaged “shadow IT.”

    The good news is that the analogy with hard laws of physics is not perfect. The process of increasing entropy is not irreversible, and does not require constant inputs of energy to prevent. New agile approaches to IT Ops are emerging that are light and flexible, taking their structure from that tension. In the physical world, this is called “tensegrity,” and was originally formulated by Buckminster Fuller, one of those annoying polymaths who seem to make breakthroughs in every field they touch.

    Tensegrity structure
    Tensegrity structure

    IT Ops needs its own equivalent to tensegrity, a flexible, holistic structure that is able to embrace change while maintaining its shape. Structures that try to resist change through rigidity are brittle and ultimately fragile.

    Today, this flexible strength is enabled by AIOps, a term coined by Gartner to describe a new way of operating IT, based on augmenting human skills with automated analysis enabled by modern algorithmic and machine-learning approaches.

    Complexity Breeds Complexity

    Modern IT is too complex for any single person or even team to keep track of, instead requiring multiple specializations. In turn, each of these functions requires its own specific support tools that enable that particular role. Beyond a certain point, it becomes very difficult to combine these partial views to gain a holistic understanding of what is actually happening in the environment.

    Because of the scale of modern IT environments and the sheer rate of change that businesses require, old-style rules-based “Managers of Managers” can no longer keep up, requiring more effort to build and maintain their worth. Especially as infrastructure becomes more and more self-modifying in response to changing conditions and demand, IT Operations needs to accelerate just to keep up.

    AIOps brings developing issues to the attention of sysadmins before they become visible to customers and end users, and gives them the tools to collaborate effectively between different teams and specialist roles to avoid outages or restore service quickly and efficiently. In this world, automation is used to augment the capabilities of the human operators to make them more effective, letting people concentrate on adding value, not just fighting fires.

    This is how Ops can get out of the business of fighting Dev, and instead become a supportive partner. AIOps can catch issues that are affecting services and flag them to the right people early on, including the DevOps engineers that are responsible for production support.

    For developers, one of the fears around DevOps is that in “picking up the pager,” they are opening themselves up to a constant stream of irrelevant alerts. Old Ops hands nod wryly at this, or maybe try to share their personal approach to filtering the noise. Instead, AIOps ensures that only valid, actionable alerts get as far as being shown to a human being and taking up their valuable time.

    Eyes on the Prize

    It’s not all roses. As the AIOps market begins to mature, there is a certain amount of “algorithm-washing” going on, with more and more vendors trying to claim that either their legacy tool was AIOps all along, or that their one-trick pony is actually a whole three-ring circus. What gets lost in the excitement is what all of those algorithms are actually supposed to achieve.

    Me too!
    Me too!

    In considering a shift toward AIOps, the goal should not be simply to build a better mouse trap, meaning a smarter filter for monitoring events. The success of the project can be measured not so much in the mathematical results of the algorithms, but in the smoother exchange between the human users. This easier collaboration is especially important across team boundaries, whether these be different specialist units within the Ops function or developers and software engineers who need to work with Ops to understand the performance of their code in production.

    The benefit of transforming operations this way—moving from sequential, waterfall Ops to abile, collaborative Ops—is to turn DevOps-enabled teams into a sort of perpetual-motion machine, able to continue operating without requiring constant massive inputs of energy from outside the system. Modern applications and the complex infrastructures that support them cannot be managed with the sorts of rigid, cobbled-together, industrial approaches that were appropriate in the mainframe days. AIOps is the agile, flexible approach that can not only accomodate that complexity, but thrive on it.

    — Dominic Wellington

  • From Waterfall Incident Management to Collaborative IT Operations

    From Waterfall Incident Management to Collaborative IT Operations

    What can Ops learn from Dev in the transition from sequential, reactive IT support to real-time DevOps?

    One of the signature ideas of DevOps is right there in the name: the blurring of the boundary between the historically separate worlds of Development and Operations.

    On the face of it, this seems like a great idea. Developers have always grumbled about how Operations is preventing them from deploying as rapidly or as frequently as they would like. Meanwhile, Operations people have harbored a dark suspicion that developers might code with more attention if they were the ones getting paged at 3 a.m.

    Another of the great things about DevOps is that it is pretty easy to adopt in one area or project without needing to get the entire company onboard before you can get started. This means that you can quickly produce some interesting results in terms of accelerated development times and release frequencies, faster resolution of problems and more time spent innovating instead of firefighting.

    Armed with these results, DevOps evangelists spread through the organization, telling their story and recruiting other departments and teams to join them in the bright future.

    This is Where the Problems Start

    Complex enterprise IT environments are, well, complex. Different teams have adopted different tools and methodologies to do their jobs, and are justly resistant to throwing out something that works for them. Specialists in different areas require their own specialist tools to be able to do their jobs. Because of this, when an incident occurs, all of that agility and collaboration that DevOps brings goes out of the window in favor of something that looks much more like a classic “waterfall” approach:

    • When an event is received, it is routed to one particular team
    • The event is placed in a queue
    • Eventually, someone from the relevant team takes a look at the event and makes a determination of whether it falls in their area or not
    • If it’s in their area, they begin working to resolve the issue within their team
    • If it’s not (believed to be) in their area, they reassign it, either back to the front-line team, or to whichever team they believe to be responsible (often, this is the network team)
    • Repeat until either someone takes ownership and fixes the issue, or it becomes a Major Incident and everyone gets paged at 3 a.m.

    Just to complicate matters, there may be multiple instances of this process going on at any one time, as two or more different teams grapple with what eventually turn out to be different aspects of the same problem.

    Why Does This Happen?

    In health care, there is only so much that can be done by way of prevention. There is always the need to react to patient’s issues, starting from their symptoms. In the same way, the starting point for Operations is monitoring—gathering the symptoms of the IT infrastructure to be able to return it to health as quickly as possible.

    To understand problems with the Ops part of DevOps, we needed to start at the beginning, with those symptoms. We wanted to do this the scientific way, so we polled attendees of Velocity and Monitorama to understand their top monitoring challenges.

    The interesting fact was that across both populations, the main challenge listed by attendees was “alert noise/fatigue/volume.” Simply put, Operators were drowning in alerts to the point that they had completely lost the ability to understand what was actually important, let alone an overall picture of how those alerts related to the service that IT was supposed to deliver.

    This is another parallel between IT and health care, where alert fatigue has long been a concern, and with potentially far more serious consequences than in IT.

    A big part of the cause of the alert fatigue on the IT side is the proliferation of specialist monitoring tools. In the poll, 66.67 percent of people we surveyed have five to 10 monitoring tools in place, and yet 61.90 percent of them are still struggling with that problem of alert noise/fatigue/volume.

    To be clear, those different tools are each very important. Trying to monitor the network with an APM tool is probably not going to go well for you. They only become a problem when it becomes too difficult to build up an overall picture of what is going on, and sharing that between different people to solve problems faster.

    So What Can Be Done?

    Dedicated Operations organizations have long worked to assemble tools and techniques that can give them a complete view of the environment that they are supporting, and of the services that run in it. The accelerated pace of release that comes with DevOps is causing the old models to break down, but fortunately, it also carries in itself the seeds of a new approach to the problem.

    As development moved from the sequential waterfall model to a more rapid, collaborative, and agile approach, operations teams can learn from that transition to come up with a more agile approach to IT Ops.

    What would this look like? The first thing to do is to get rid of the blame-storming and buck-passing: “It’s the network!” “No, it’s the database!” I don’t know about you, but personally, I’ve been in enough of those meetings to last me for the rest of my life, thank you very much.

    Let’s All Go Fishing in the Data Lake

    Instead, a common perspective is needed, so that everyone can quickly agree on what is going on and what needs to be done.

    This does not mean throwing out all of those specialist monitoring tools! They are there for a reason. Rather, gather all of those symptoms in one place where the whole team can collaborate to make sense of them more easily. These days, it should be easy to get information in and out of the various components of your toolchain (and if it’s not, that’s a glaring warning sign that maybe it’s time to migrate off that particular tool).

    This will, of course, produce a huge pile of data, so it’s not sufficient to stop here with the creation of a “data lake.” Things disappear into lakes and are never seen again—in which case, why did you bother gathering them in the first place?

    What you need is a way to make sense of what is going on in the lake, and show that to the right human operators. In the same way that developers resent sitting around waiting for the Change Advisory Board to meet, operations people get frustrated and bored just staring at a screen waiting for something to happen. If instead you can use automatic techniques to assemble all the related symptoms of a single issue together in one place, that will give you a good starting point to resolve that underlying issue quickly.

    Brainstorming, not Blamestorming

    This is where the collaboration comes in. Again, people are doing this already, but they are doing it in a disconnected way. The tools have evolved, from email and IRC, to Slack and Google Hangouts, or even (heaven help us all) WhatsApp. What has not evolved is the usage model; it’s still ad-hoc and reactive, and requires lots of copying and pasting.

    Operations people can learn from what their developer colleagues have built or adopted. Kanban boards are a great technique for tracking open issues and who is working on them. ChatOps is a fantastic way to integrate human-to-human communication with human-to-machine communication. Integrated collaboration tools capture knowledge that otherwise is stuck in people’s heads or inboxes, unavailable to their colleagues.

    I’m an Ops guy at heart, but I can recognize a good idea when I see it. Our Dev colleagues have come up with quite a few that we could stand to adopt in Ops. Let’s learn from what they already know to work, instead of trying to reinvent the wheel.

    — Dominic Wellington

  • Dev & Ops: These Are Reconcilable Differences

    Dev & Ops: These Are Reconcilable Differences

    The new world of IT requires a different approach from what worked in the past. As usual, the technology is the easy part, but without people and processes being aligned, it won’t deliver value. When evaluating how data science and machine learning can help manage highly dynamic IT environments, old reflexes may be actively harmful.

    By their nature and due to their experiences, Ops people tend to be a pretty (small-C) conservative bunch. Rule No. 1 of Ops is “If it ain’t broke, DON’T TOUCH IT!” and if we’re honest with ourselves, most of the rest of the rulebook is just variations and commentaries on that theme.

    One of the ways in which Ops people try to control their world is through automation. DevOps mostly looks to the moment of release and deployment, as that is where the handover to Ops typically occurs—and this is a big focus of the drive to automate. However, there is another part of Ops where automation is key, something that Dev is not always nearly as involved in; namely, day-to-day operations. Which alerts should even be sent, whom they should be routed to, what are the valid responses and so on—all these have been automated over the years, to a greater or lesser extent.

    Ops people have laboriously documented their architecture and spent a long time in meetings planning which information is relevant to share and who should respond to alarms. Much of this is implemented in various pieces of software, whether commercial, open-source, home-grown or that special grey area, where something that started out as a standard component has been so customized that it is effectively bespoke. Rules, filters and thresholds determine what action to take in which case.

    These approaches have been in place for years or even decades, but now they are starting to fail. Simply put, IT’s increased complexity and the ever-accelerating rate of change are outpacing administrators’ ability to keep up and reconfigure their management systems.

    The Coming ‘New Normal’

    New approaches are emerging to deal with the new normal, but as ever, the technology is the easy part. For new technology to be a success, people and processes need to be in sync.

    Right now, the emerging approach is to use data science and machine learning to process events, instead of the old deterministic rulesets and databases. The results speak for themselves, especially at scale and in highly dynamic environments. People building container-based or SDN architectures no longer assume that there will be a database of configurations that will always be up to date; quite the opposite.

    The new way is to throw out all the filters and rules, feed all the alerts into a single place, and then use data science and machine learning techniques to sift them and identify relationships with them. The result will be a shift in focus from individual alerts to the business problems those alerts are the symptoms of.

    Algorithms are great at these sorts of repetitive, high-volume tasks. They can look at massive volumes of events and figure out which are even relevant and worth taking a second look at, and then identify how those events relate to each other and what they really mean. This is where the humans come in, with their strengths in low-volume and unpredictable situations. Because they are not constantly drowning in irrelevant noise, they can work together effectively to understand the actual problem and get it fixed quickly.

    Where’s the Catch?

    The difficulty is that for many Ops people, there is a big cultural change required to adopt this model. The old deterministic systems may have required laborious maintenance, but at least it was possible to model them and understand why a specific result was achieved. New algorithmic techniques take a radically different approach, and while the results are impressive, they come from a black box. This does not inspire confidence for people who are very comfortable with the old approaches.

    Often, the response of Ops people coming from legacy tools is to get stuck into the minutiae of what is going on inside the black box and to look for traditional failure modes. The problem is that these approaches are no longer valid in the new world, as I discussed in my last post about getting to SRE.

    For instance, while it is indeed important to make sure that no issues are missed due to over-aggressive filtering, this does not mean that we need to see every single alert. As long as we know about the actual business problem, we don’t actually care about the single alerts, except insofar as they help us debug the underlying issue and resolve it.

    The other direction is even more insidious, where people spend a lot of time chasing down “extraneous” events that they think they should not be seeing. The risk is, of course, that the Ops team ends right back up in the same place that they were trying to escape, where they are only able to identify expected problems, but are getting blindsided on a regular basis by the “unknown unknowns,” the conditions that simply should not happen.

    Ultimately, Ops needs to let go of reflexes that were good in the old world, and focus on the goal: ensuring the uptime and stability of business-supporting systems. The how is less relevant than the what and the why; the algorithmic black box might as well be full of magic leprechauns, as long as they are telling Ops what they need to know, when they need to know it. Call it the “Chinese Room” theory of Ops: For as long as the results are useful to Ops, it does not matter what is inside the room.

    The benefit for Ops is precisely that they no longer need to focus on the minutiae of individual alerts, but can concentrate on solving actual business problems. That is how IT can make itself a value generator and avoid being viewed as another undifferentiated cost center.

    Pay no attention to the algorithm behind the curtain.

    — Dominic Wellington