TEST
— ROELBOB
Machine Intelligence-Driven Anomaly Detection
Machine learning—the science of getting computers to take thoughtful action without explicit programming—is the basis of Machine Intelligence-Driven Anomaly Detection, which is the core capability in Avik Partner’s Grok APM product, according to Casey Kindiger, veteran enterprise software solutions provider, IT automation expert, and founder of Avik Partners.
“Thoughtful action means using the type of faculties that a human being would use to make a decision and to take an action,” says Kindiger, who has more than a decade of experience in designing and developing process automation solutions for companies such as JP Morgan Chase, T-Mobile, and Allstate Insurance. Grok learns from your dev and ops environments and takes thoughtful action to detect anomalies, all in the service of Application Performance Management.
Grok works by analyzing real-time performance data from apps and infrastructure, looking for behavioral bumps in an otherwise smoothly-paved DevOps highway of deployed services and underlying infrastructure.
Grok’s Pre-Event Inferences
“Grok uses common statistical techniques to learn a pattern over a selected period of time in order to build a dynamic threshold of the continuously changing underlying patterns,” says Kindiger. In this way, Grok can establish a sort of dynamic baseline from very noisy patterns, which is something that would typically take a number of person hours of involved statistical and predictive analytics to accomplish.
Grok’s machine intelligence leverages the ability to make inferences based on this dynamic baseline information like a human being would in order to reason out a logical conclusion. Grok does this to detect the slightest of anomalies based on the difference between what reasonably should have happened and what actually did happen. Using this approach, Grok detects subtle patterns of significance in streaming data despite the fact that the data has patterns that are continuously in flux.
But rather than detect when something has gone wrong at a point in time, which is not scalable or sustainable, Grok looks for minute patterns that appear well in advance of any issue that you would normally consider looking at with traditional APM mechanisms, explains Kindiger. Rather than send alerts based on threshold-based exceptions, Grok makes a prediction on what the next data point or the next series of data points will be and then looks at deviations from those predictions, considering whether to flag those as anomalies, according to Kindiger.
Currently, Grok is particularly tuned to analyze streaming data and is, according to Kindiger, data-point agnostic. “We don’t need to teach the system what the context of a particular data point is. The system learns the context of the data over time and the subtle nuances of the pattern that produces over time, and creates the model based on that,” says Kindiger.
Watch Out for That Initial Two Week Delay
If Grok does everything that Avik Partners’ Kindiger hypes that it will, DevOps teams will soon be able to catch and act on infrastructure issues before they grow big enough for end-users to notice.
But Grok does have one wrinkle in its APM armor: it takes time for it to really learn a new customer system and get good at detection. Let’s assume that Grok’s machine learning causes it to get better at detecting anomalies in a given set of infrastructure over time. That means that there’s a point, at the beginning, given a brand new customer when Grok only starts to learn. At that point, Grok isn’t very good at knowing what’s anomalous compared to what’s normal for that infrastructure because Grok has no foundation for that particular system.
“Grok will take some time to learn the nuances of when to escalate an issue. We’ve learned from experimentation that it gets pretty good after ingesting two weeks’ worth of data,” says Kindiger. Because Avik Partners initially targeted AWS, it has taken two weeks of CloudWatch data and fed that to Grok to provide it with some backdrop so that when Grok goes live, it won’t be learning this environment entirely anew.
With so many players in the mix from operations to developers, from PMs to technical leads, to Q&A, most of their activities are automated. Systems record all their email, Wiki, and chat conversations and other notations about these activities. This gives the devops community the opportunity to learn from that data in order to improve future projects.
“We have access to a massive amount of data that can in total define what the process is that’s going on and define it in-flight in real-time,” says Aaron Cois, Researcher, CERT Division, Software Engineering Institute, Carnegie Mellon University. Through exploratory research, Cois is learning that by using machine learning tools, enterprises can analyze security activities in the development process based on data from issue trackers and commit logs.
Valleys in Topic Density Point to Software Development That is Missing the Mark
Natural Language Processing, a subset of machine learning, uses systems to process text in unstructured data and to perform topic detection, determining whether or not a portion of the text belongs to a given topic category. “In our work, we’ve taken text from issue trackers and used machine learning to classify each issue as security-related or not security-related,” says Cois.
This has practical applications for detecting the course of a project and steering it in the right direction. If the PM expects the developers to raise the quality of security for an iteration of the software package, he can monitor the chatter among developers about the issues they are adding to the issue tracker or he can monitor the comments on the code.
“By using machine learning to classify issue tracking or source code commit messages as security-related, we can see a reasonable measure of the amount of work that’s being done to ensure the security of the software system,” says Cois.
If the PM can determine based on a shortage of security chatter that in actuality not much security work is going on, he can ask why, what is the challenge? “Is the security stuff more difficult? Did we do our due diligence to allow our developers to adequately address security?” asks Cois. Then the PM and the teams can work together to remove any roadblocks before any more scope creep or veering off course occurs.
Machine Learning for Physical Modeling and Trends Analysis
If the industry can build statistical models of what a healthy software project looks like as it progresses, then it should be able to determine whether a given project is on track and what risks it may face along the way.
Issue tracking data, commit logs, wikis, chat server logs, build/deployment logs—every type of data that can lead to a full picture of project activities—would form the basis of these models, according to Cois.
Through Natural Language Processing, machine learning can help determine from this data what the rates of creation and resolution of issues are and what the rates of commits and overall progress are for the software project.
Machine learning can help PMs look at how their project’s rates in these areas compare with rates of similar projects, which developed certain problems or turned out in certain ways. If there is a strong correlation between rates of projects that had known outcomes and rates in your project, then you might want to use that information to make corrections early on, in cases where the outcomes of the other projects were undesirable.
Over time, the models become stronger and their predictive value increases, leading to some strong correlations between current projects and the models collected. With the increasing density of the data available in models as they grow, even the weakest numbers in early models can become stronger, giving the model strong correlative value in areas where strong correlations did not appear at first.
“If I have collected both activity and outcome data on thousands of projects, and a large number of them are highly similar to the current project, I’ll have much more confidence in my predictions about the potential outcomes of the current project given the behaviors I am observing,” says Cois.
Over the years I’ve helped literally thousands of customers gain efficiencies in their data centers. I’ve met and become friendly with a lot of IT pros but I’ve got to admit – I never envied them. Well, maybe I envied some of the views in NY and Jersey City. And some of the toys they got to play with over the years were amazing.
Nonetheless, I didn’t envy them because many of my friends in director and VP roles spent a huge percentage of their time fighting politics that resulted in Orwellian distortions of the truth. A couple of stories from the early 2000s might help illustrate my point:
1. Two guys pulling all-nighters delivering a $1.5B roll out of IP communications while 25+ project managers debate progress from 9-5
The headline pretty much captures it. This now too-big-to-fail bank, as well as their vendor, had brought in several project and program managers to ensure groups from facilities through supply chain through security and operations were “aligned.” I spent a couple of meetings getting to know all of these project managers before realizing that there were two guys working every night, all night, to manually, using fingers and spreadsheets, push and validate changes. Eventually those operators quit – and the project ground to a halt.
All the alignment, of course, was pretty much BS. At one point I mentioned in a meeting that our software would not run on the same server as the IP Communications System (Call Manager for those curious). I was corrected – “it isn’t a server.”
I responded with something like, “it is an HP DL something or other. Isn’t that a server?“
Answer: “No, if we call it a server then the server team will need to get involved and we don’t want the server team getting close to this project.”
You cannot make this stuff up. Servers = / servers. Got it?
2. Similar story, different bank – 1% of achievement of goal = “on plan” to outsourcer
The story starts when I was brought in by a huge vendor to a bank that is also too-big-to-fail. This bank also had a high profile project to get the bank over to IP communications. Keep in mind that this was after 9/11 and all the banks were looking up to the teams at Lehman that had been able to get Lehman back trading significantly faster in part because of their use of IP communications. So the project to get the bank onto a new approach to real time communications had CIO visibility.
One of the most surreal meetings of my career occurred when after having been vetted, tested, recommended, and designed into the solution I was granted an audience with the SVP handling the bank at EDS; I feel ok with naming EDS since they no longer exist and I’m sure this executive has retired. Like many folks at EDS he was former military and as an account executive he owned a north of $1B outsourcing contract.
I started off by saying that we can help you get back to schedule, as I know you have only done about 1% of the deployments originally planned. He looked me right in the eyes and said: “EDS is on plan, and we always will be.”
After some back and forth I got it – no matter what the facts on the ground were, he was never going to accept any culpability at all for the project delay. So missing the project time line by 99% was officially “on plan.” That’s Orwellian.
DevOps restores faith in humanity / IT
Compare any decent “DevOps” shop to the above stories and you can see why I’m so enthusiastic about doing my small part to help bring DevOps approaches to operations to a broader market.
The teamwork between not just Dev and Ops, but between everyone responsible for delivering capabilities to the organization, is so refreshing. Just last week, Google engineering blogged about using machine learning to improve operational efficiency of their data centers, with a focus on power consumption and so forth. The technology I get – we are headed down a similar path with deep learning and control theory. But what struck me more than the technology was the data they are using to draw inferences would often be siloed and tightly controlled by groups within IT at legacy companies precisely because it could be used to score points.
So DevOps entails more than infrastructure and code and collaboration. With DevOps, IT organizations move from fighting over terms like “server” and “on plan” that are often filled with well meaning and intensely frustrated teams towards an environment where cool stuff happens all the time. And IT done right – making sense of the world with software and hardware – is incredibly cool.
I’m betting on DevOps because I want the world to be filled with DevOps unicorns and because, like my friends that are operators within legacy environments, I cannot stomach more Orwellian obfuscation of the facts on the ground for political expediency. DevOps makes the world smarter and IT teams a heck of a lot happier. What’s not to like?