Continuing recapping my favorite sessions of DevOps Enterprise Summit 2015, here is Josh Corman and John Willis talking about Immutable Awesomeness. What is Immutable Awesomeness you ask? Have a listen to my interview with John and Josh and then listen into the actual session where they go over it.
Just a couple of hints: 1. Software supply chains are a great way to look at your development/deployment process 2.In a world where software controls everything, it really is a case of life and death 3. If you can’t get excited about what this all means, listen to the session again.
Of course what makes this interview and session great is the unbridled enthusiasm that both John and Josh bring to the subject. In fact there was a palpable buzz in the theater when they walked off the stage and over to my video studio. The same enthusiasm carried over to our interview too. It was a lot of fun to interview these two.
So have a listen and enjoy. Also if you enjoy John Willis presenting, he will be headlining the DevOps Connect: cdSummit SoCal on December 3rd, joining some other great speakers.
One of my favorite presentations at the recent DevOps Enterprise Summit 2015 was Jeffrey Snover of distinguished engineer at Microsoft talking about his 16 year quest to bring what is today called Windows Nano Server to market. Interestingly when I spoke with Gene Kim about what he thought were some of the best presentations of DOES he also brought up Snover’s presentation as well.
Windows Server without Windows was something once considered heretical from anyone at Microsoft. As Jeffrey relayed to me in the interview below and on stage, he paid a price for evangelizing for this. But today we have a version of Windows Server that has only 1/20th of the footprint of regular Windows Server. It runs on mainframes, command lines and has virtually (no pun intended) all of the functionality that you need for a server in the cloud without the GUI overhead that Windows usually brings.
As importantly for me, Jeffrey also gave us a glimpse of what it is like to work in Satya Nadella’s Microsoft. It sounds like a very different place than when Steve Ballmer was CEO. Is also sounds very refreshing as well.
I imagine most people are like me and have a lot of things they’d like to do if they could find the time and energy for them. Lately writing has been one of those things for me. I suppose we all have our priorities. Traveling for tech events and coaching the kids’ soccer team can make it seem like there’s no time to write, and then something legitimately disruptive happens like one of your kids landing in the hospital.
I recently read Designing Delivery with a group of folks as the inaugural title for our devOps book club. In the book Jeff Sussna discusses delivering quality experiences to your customers as early and often as possible. Most people have been exhausted from traveling at some point in their lives. Many examples from the book talk about this shared experience. This allows us to empathize with a specific type of pain many airline and hotel customers are trying to manage while going about their business.
Early engagement that sets expectations can make a huge difference. By providing realistic travel times that account for things like seasonal roadwork airlines are more likely to have their customers arrive on time and in a good mood. A hotel that knows they’re dealing with jet lagged, travel worn customers would be well served to have fresh coffee, water, and fruit for them when they arrive if not a drink at the bar.
You can probably empathize with being travel worn if not jet lagged, but I doubt you understand the level of exhaustion and confusion that comes with having one of your children hospitalized in an emergency situation unless you’ve been there. Hospitals and hotels have a lot in common. One luxury a hospital doesn’t always have is setting expectations for the patients and families. In an emergency you could have less than an hour’s notice before the hospital makes its first impression. How do you make someone comfortable and help them relax when they’re quite reasonably stressed out and concerned?
My twins were diagnosed with Twin to Twin Transfusion Syndrome (TTTS) before they were born. The illness required an emergency trip to Women and Infants Hospital in Rhode Island on short notice. Providence is a sixteen hour drive from Atlanta. Given I was still in college we stayed at the Ronald McDonald house which are around for families that have a sick child in a hospital away from home.
A trip to an emergency room for a broken arm is one thing. Not knowing if or when your child is coming home is something else entirely. There were kids getting picked up by buses for school and parents that left for work while other members of their family where in the hospital or back home in another city. I’m thankful our stay was short and relatively uneventful. Women and Infants is an amazing hospital and the Ronald McDonald house really made us feel at home.
My twins have had their lives saved at Children’s Health Care of Atlanta a total of three times since then. One was hospitalized with cardiac issues immediately after she was born and had a balloon catheter procedure done at Egleston. Both have been in intensive care at Scottish Rite for respiratory issues which includes a visit within the past sixty days.
CookieOps was something I started with them over a year ago to try to raise money for these organizations. My girls sold me on the idea by telling me, “There are sick kids that still can’t go home and play with their brothers and sisters and we need to help them”. What started out as a bake sale turned into a model for approaching problems with a devOps perspective. For us that means embracing the Three Ways and following the CALMS model (Which is CAMS with an ‘L’ for Lean/Learning). We believe that every process has an artifact and there’s room to optimize both of them. It’s this type of thinking that leads to putting a mint on a pillow and understanding the cost involved far outweighs the related expenses. It’s often the little things we find delightful that make us loyal customers for life.
Like leaving cards near the elevators to help you remember where you parked coming in from the deck to find your child. I’ve wander around in a daze looking for my car in a hospital parking lot before. They’re big buildings that often have multiple decks. It’s easy to forget where you parked when you’re in a hurry to check on much more important things.
My daughter was really excited to get a postcard in the mail from her nurses after she was feeling better. I’m not sure how much time is spent writing these cards but I’m thankful for it. Extending the hospital’s experience out to when you’re home and things are back to normal is something that should be celebrated and is well worth the cost involved.
I’ve investigated a few ways for my kids to raise money with cookieOps but so far we’ve been giving out cookies for survey responses. Publishing the data from those surveys is another thing I’d like to make a priority but haven’t had the time to complete. As raising money for the hospital was always our primary goal when we first started putting surveys out I had little choice but to do so as efficiently as possible when the opportunity presented itself.
Brawl for a Cause allows regular people to raise money for causes they believe in by stepping into a boxing ring for their first amateur boxing match. A big difference between a road weary traveler and a family with a child in the hospital is the traveler usually signed up for the ass kicking. The hospitals are dealing with families that would have been fine without ever visiting, and the kids definitely didn’t sign up to fight with their illnesses.
So I’m glad to step in the ring on November 25th, fight for the sick kids that can’t go home yet, and make good on the promise I made to my daughters. My opponent is boxing to raise money for his daughter’s college education. While I believe everyone deserves the right to an education I also believe they need to live long enough to make it through school. There are opportunities to take your education into your own hands through programs like Kahn Academy. The kids in the hospitals lives are in the hands of the doctors and nurses watching over them. Those kids didn’t sign up for the ass kicking they’re taking, but I’m willing to risk a beating if it can help Children’s successfully ease a family through such a difficult time.
If you’re in Atlanta we’re boxing at the Hyatt Regency in Buckhead on the 25th. tickets are on sale now. You can donate to support these causes no matter where you live. I’ll take the punches but I need Youto make a donation today.
Enterprise development teams find it difficult to perform QA tests on software in short DevOps cycles where they cannot squeeze in lengthy regression tests. DevOps teams must insert automated testing approaches to resolve this predicament. “DevOps teams need to be more selective in prioritizing tests that occur at each stage of the integration to delivery pipeline,” says Marc Hornbeek, Senior Solutions Architect of DevOps, Spirent.
At each point between stages of the development lifecycle, there is an opportunity to insert the most applicable tests to determine whether the software has incurred errors. It is crucial to test early. The earlier in the development process that team members discover their errors, the sooner they can fix them, ensuring fewer error-related issues in releases. The earlier in the development lifecycle that errors occur and go unchecked, the more opportunity they have to grow and become increasingly debilitating to the software.
With best practices, DevOps shops can use the right tests in the right stages of development and achieve quality assurance without depending on large test suites that run at the end of the cycle, according to Hornbeek.
The best time to create automated tests is together with or prior to developing the code for the given software feature itself. This will enable development shops to keep pace with the speed of DevOps; the old approach of automating testing after the feature is developed has truly become ineffective, according to Hornbeek.
DevOps Targets Quality / Requires Testing
DevOps has the goal of creating better software faster. Apart from effective, continuous, automated testing at keenly selected intervals, DevOps is only faster. This would achieve only a form of dev similar to what we had before DevOps, i.e., the kind where coders believe they have done well even though operations staff sees a lack of fitness of the software for production use.
Enterprises must embed a culture of QA inside the culture of DevOps rather than separate out QA testing from DevOps using organizational silos; communicating only through inefficient bug reporting methods is no longer the state-of-the-art, let alone practical with the increasing industry expectations of DevOps.
Testing: Why Must I Be Misunderstood?
The impact of testing is often overlooked. DevOps shops that take testing for granted may not invest in or improve upon testing as testing needs change and grow. “This was true before DevOps and continues to be true with DevOps. Developers focus on feature creation, while the business leaders focus on delivery,” says Hornbeek. As a result, testing is often not a focus of DevOps.
Further, there is a common misconception that testing just the part that is done after the builds are created is all that is necessary; but enterprises must integrate continuous automated testing with DevOps throughout the development-to-delivery pipeline, according to Hornbeek.
“When testing is conducted a long time after the feature is developed, there are many inefficiencies. The developers lose context of the problem and have a harder time recreating, diagnosing, and fixing problems,” says Hornbeek. Automated continuous testing eliminates this delay, improves software, and shrinks time to quality software releases.
Slothful Testing Gone Awry
There are high-profile examples of slothful software testing where projects grew out of scope and way off target. A chief example is the original rollout of Healthcare.gov, says Hornbeek. “In another example, a network equipment manufacturer tried to implement DevOps twice without success. They finally realized that they had underestimated continuous testing,” says Hornbeek of a Spirent customer.
To prevent these kinds of outcomes, says Hornbeek, follow these best practices:
• Create automated tests as you develop the features.
• Do not accept features into the integration branch without automated tests.
• Ensure that the developer test environment is a good representation of the production environment.
• Select DevOps-ready tools that have Rest APIs.
• Test priority features so you can easily integrate them into the DevOps orchestration framework.
Follow these steps, according to Hornbeek, to incorporate automated testing for QA:
• Establish a clear leader with responsibility over the entire dev-to-deliver pipeline who is visible and will champion continuous testing.
• Assign an architect for continuous testing tools selection and to oversee the technical implementations.
• Determine best practices for Continuous Testing and decide what practices best fit the organization.
• Address the culture, roles, and responsibilities for continuous testing.
• Put some automated metrics in place, such as a goal to automate 85% of all tests.
Editors Note: I had a chance to catch up with Mark Hornbeek at the Enterprise Summit 2015. Here is my interview with Mark.
Everyone wants to optimize a team’s performance, but it’s not enough to just talk about it. To truly impact your team’s performance, you need to take a page from operationally mature DevOps teams and use metrics to gain valuable insight into your team’s work, enhance their capacity and drive cultural change.
The right data allows managers to make quick decisions with confidence and minimal risk. It also lets managers see the actual outcome of those decisions, in turn allowing them to shape their team’s direction going forward and to create a happier, healthier and more productive team dynamic. However, making the shift to data-driven decision making requires more than just increased monitoring. It requires a cultural change.
Below are four key metrics DevOps teams should be monitoring, as well as the impact each metric can have on a team’s culture and performance.
Use Time to Response to Establish a Culture of High Achievement
Time to response (also called time to acknowledgement) depends on individual team members. Incident responders might not always have control over the root cause of a particular incident, but they are always in control of how quickly they acknowledge and respond. Hold team members accountable for their response time by setting internal targets and having high expectations.Most operationally mature companies use IT operations management tools to enforce a response time target. These tools allow managers to set an incident response time window, making sure the next person in line is alerted if an incident isn’t responded to. Tracking escalations also gives managers valuable data about their team.
Manage Expectations With Escalations
For an organization using an IT operations management tool, an escalation is often an exception – it’s a sign that a responder either wasn’t able to acknowledge an incident in time or that the responder didn’t have the tools or skills to address it. While escalation policies are a necessary and valuable component of incident management, teams should ideally focus on driving the number of escalations down. Managers can gauge their team’s performance by tracking the number of incidents escalated over time, and using that data to determine whether their targets need to be adjusted.That said, there are some situations where an escalation is part of standard operating practice. For example, you might have a NOC, first-tier support team or even auto-remediation tool that triages or escalates incoming incidents based on their content. For these situations, it’s important for managers to track what types of alerts should be escalated and what normal numbers should look like for those alerts.
Combat Alert Fatigue With Raw Incident Count
As an organization grows, incident counts often grow, as well. However, as a team becomes more efficient and mature, the incidents per responder should lower or, at the very least, stay constant.Implementing IT operations management tools can help teams lower their raw incident counts by helping to weed out low-quality alerts, automate common fixes and build runbooks. They can also help break down incident count by team or service, putting incidents in context with the rest of the organization. This ensures that team members are maximizing their time on attacking alerts that matter and building new features, instead of getting mired in alert fatigue.
Gauge Operational Readiness With Mean Time to Resolution Time to resolution, or how long it takes a team to resolve an incident, is the highest standard in operational metrics. The baseline for time to resolution can vary from organization to organization, depending on complexity environment, organization of responsibility, even the industry in which a company operates.Tracking time to response allows managers to determine their norm mean time to resolution and ensure their teams are able to work through the challenges of a major incident.
Use These Metrics to Foster Intelligent Change
It’s no secret that downtime is expensive, both in loss of revenue and customer trust. Metrics like the ones discussed are put in place to combat downtime and increase reliability, yet it can be easy for teams to become too focused on numbers and past performance.
It’s important for team members to not lose sight of the business goals and why they’re tracking those metrics in the first place. It’s also important for managers to avoid over-analyzing the past. Metrics measure what has already happened, and, while they can tell a lot about a team’s previous performance, they should be used as a tool to a better future and not a means of assigning blame.
Metrics are a means to an end, and having more information than you need won’t help you improve your team and refine your business. Keeping the emphasis on subsequent action is the key to using metrics to drive cultural changes.
At this years DevOps Enterprise Summit we were very happy to partner with Gene Kim’s IT Revolution and Electric Cloud to conduct on site interviews with just about all of the speakers from this years conference, as well as many other attendees. All in all devopsy.kinsta.cloud conducted something like 86 video interviews over the 3 days of the Summit.
Over the coming weeks we will be highlighting some of the best of the best here for you. We will present them in different formats.Some will have commentary around their sessions, others will feature background info on the topic and speaker and finally we will have updated info for some of the others.
In that vein I wanted to lead off the series with this interview of none other than Adrian Cockcroft of Battery Ventures and formerly Netflix. My conversation with Adrian touches on really just three questions:
What was his presentation about: Systems integration or systems thinking. It offered an inside look at the Netflix culture and talent density.
what did he think of DOES 2015 – Adrian is on the program committee though by his own admission he was a bit pre-occupied on other business this year to help too much. But like many who attended Adrian is was really impressed at how deep the penetration into different verticals were.
What does he see in the next 3-5 years for DevOps? Adrian says that companies that don’t adopt DevOps ways run the risk of disappearing.
if the short 7 minute interview above wasn’t enough, here is Adrian’s complete session at DOES 15 (warning it is 31 minutes long)
A thriving community of more than 7,000 IT professionals gathered online and in-person for Puppet Labs’ annual PuppetConf 2015 event recently to learn how leading global enterprises such as Uber, Walmart and Wells Fargo develop strong organizational cultures and leverage DevOps, automation and other modern IT tools and practices to solve problems and build high-performance data centers serve millions of customers and “change lives.”
The art of life-changing problem solving through automation isn’t lost on Puppet Labs’ customer Walmart. Senior Technical Expert Martin Jackson quoted a colleague who foresaw the benefits Puppet Labs would deliver when it first started using the company’s IT automation software, “‘We’re going to change lives with this,’” a prophecy he said has been fulfilled.
In tangible terms, the life-changing results he spoke about to the capacity audience were measurable. With Puppet Labs automation software, Walmart went from having relatively few nodes managed through automation to a whopping 48,000 in a short period.
Walmart Senior Technical Expert Martin Jackson presenting at PuppetConf 2015
With this level of automation, Jackson said his organization has been better able to support Walmart in achieving its goal to serve millions of customers at the lowest possible price.
Most consumers, or even business end users for that matter, don’t think much about what goes into processing something as a simple as a banking transaction from a mobile device or computer. But PuppetConf attendees know there is a complex spider web of tech behind making an app as common as one used for online banking work.
Wells Fargo,the world’s largest bank by market capitalization, is a Puppet Labs customer leveraging automation technology for application orchestration, automation of node configuration and application deployment.
Using “Back to the Future” as his metaphor, Wells Fargo DevOps Manager Nate Loomis talked about the need to be on the cutting edge of IT, while also keeping services to customers reliable and dependable. When Wells Fargo IT shifted from a node-centric view to an application-centric one, its provisioning started moving much faster — from months to the time it takes to get a cup of coffee.
With Puppet Labs, Wells Fargo manages 120,000 nodes and 55,000 network devices to process more than 1 billion transactions daily, hosts more than 2,300 applications in production with more than 40 percent of its operations fully automated.
Wells Fargo DevOps Manager Nate Loomis presenting at PuppetConf 2015
Culture Matters
DevOps is more than just tools. While addressing the crowd at the Portland Convention Center, Uber’s Head of Business Intelligence Komal Mangtani emphasized that its IT culture pushes engineers to experiment in the name of progress and to constantly be on the lookout for new ways to solve problems.
Uber Head of Business Intelligence Komal Mangtan presenting during PuppetConf 2015
With a picture of The Martian leading man Matt Damon on the jumbo screen, Mangtani borrowed a line from the film when she told attendees, “If you solve enough of the right problems, you get to go home,” a humorous reference to wide-spread use of Uber by patrons to get rides home that wasn’t lost on the crowd. She emphasized that while most people think of Uber as an app, ride and car, there is much more to it than just that — pointing out that that within its IT culture is a dedicated team of “fierce yet humble” engineers that are helping the company to deliver more than 1 million rides, daily.
Summing Things Up
Although he opened the conference, Puppet Labs founder and CEO Luke Kanies summed up the whole point of IT automation and DevOps during his keynote which included a live demo of Puppet Application Orchestration. While addressing the conference, he told the crowd, “We at Puppet Labs are here to help you get there. As a company, we are all about reducing the timeline from the moment you get new technology to the moment it’s in the hands of your users delivering real value. You are going to see us as a company evolve over time to get better and better at aligning the tools we provide and the work we do enable with helping you more directly, more consistently and more repeatedly delivering customer value.”
I recently had the privilege of attending the DevOps Enterprise Summit hosted by IT Revolution and Electric Cloud. There was much to learn about emerging enterprise DevOps practices, some of which I will cover in future articles. For now, I would like to share one of the biggest epiphanies about the relationship between DevOps and ITIL/ ITSM.
Since DevOps came on the enterprise radar, there have been numerous articles, webcasts and whitepapers trying to create a bridge between ITIL and DevOps. Some suggest that ITIL has passed its prime – its an out dated framework that was designed for waterfall service lifecycles. Others claim that you can’t do DevOps without underpinning control and release processes. While there is some truth in all of that, several of the presenters at DOES15 did reference ITSM processes and confirmed that service management is alive and well in DevOps. However, what became equally clear is that this is not your mother’s ITSM!
The modern IT environment expects processes that support transformational initiatives and enable rapid development and deployment. So while a process like Change Management was mentioned in several sessions, there was little focus on a Change Advisory Board or requirements for requesting a change. In fact, when one presenter proudly displayed their “change record”, it looked nothing like the ITSM database record of yore. The modern change record may resemble more of a running date stamped commentary that a collection of fields. This does not imply that you should no longer have a CAB. It implies that it is more important than ever to build change models based on risk and need that encourage more standard changes. Similarly, while there is much talk in DevOps about configuration management, it now occurs throughout the entire delivery pipeline and uses automation and mechanisms for managing configurations in real-time instead of through discovery methods. Configuration management tool integration is critical between Dev and Ops.
Processes like incident management and problem management are benefitting from the increased use of monitoring and the emerging DevOps practice of ChatOps. Dynamic chat rooms not only encourage stronger collaboration between Dev and Ops, they modernize the ability to quickly detect, investigate, diagnose and resolve incidents and problems. As one presenter pointed out, ChatOps tools allow the collaborators to bring the environment into the discussion. They can not only chat, they can see. The modern service desk may also find that ChatOps creates almost instant escalations at the point of detection instead of the point of customer contact. That’s cool.
Finally, a practitioner presentation did a fantastic job of demonstrating how they were able to marry their ITIL processes to Agile practices. There is no doubt that service management must evolve into Agile Service Management in order to be of value to continuous delivery pipelines. Putting an operational person on a Scrum team could be a good first step.
So what does all of this mean? Do you now need to throw away the ITIL books, replace your ITSM tools and start rebuilding processes? No. Many of the DevOps automation providers have built APIs between tools and into service management products such as ServiceNow or Remedy. Frameworks such as ITIL already do a good job of describing the key processes that have to underpin the service lifecyle, including the continuous delivery pipeline. However, the very nature of a static publication library makes it difficult for enterprises to look to the ITIL books to keep current..
So the time has come to heed the age old ITIL advice to “adopt and adapt”. To learn more about DevOps enterprise practices, to scale and integrate with the guidance from ITIL, Agile, Lean and other frameworks. To enable a delivery pipeline that meets changing business requirements and removes constraints across the IT spectrum. And to realize that next-gen ITSM is here!
Chef has announced some major additions to their lineup today. We will look at each of these but in total here is the news from Chef today:
First of all the previously announced Chef Delivery is now in GA for everyone. Chef Delivery defines and manages the entire DevOps workflow. The diagram below is a good illustration of this:
According to Chef, Delivery, “provides a prescriptive process to automate changes to infrastructure, applications and containers in a single prescribed pipeline.”
Beyond the general availability of the previously announced Delivery, Chef also announced that they are moving into services in a big way. They are spinning up a new DevOps transformation practice to help enterprises adopt DevOps and transform their models. Justin Arbuckle, chief enterprise architect at Chef is heading up this new practice. I would assume they will compete with professional service organizations like IBM, HP, PWC and others in helping large enterprises transform and adopt DevOps practices. It is a natural move for Chef as it continues to mature along with the DevOps market in general.
Finally, Chef is going to roll out compliance as code, part of something Justin Arbuckle has spoken to me before of compliance at velocity. In order to bring this capability to market Chef has bought a smaller German security provider called VulcanoSec. VulcanoSec has done a good job of developing a shift-left security product that allows for continuous compliance. I am sure we will see the Chef team expand this as they seek to bring more and more security into the DevOps process and they see opportunity for improving the entire development/deployment environment to make it more secure. They will also have to integrate and work with existing enterprise security solutions.
All in all this is quite a series of announcements from Chef, which were made at the European Chef Community Summit in London. Chef has been a leader of the DevOps movement from the very start and they continue to show that they will continue to lead and innovate in bringing DevOps and everything as code to enterprises everywhere.
Machine learning—the science of getting computers to take thoughtful action without explicit programming—is the basis of Machine Intelligence-Driven Anomaly Detection, which is the core capability in Avik Partner’s Grok APM product, according to Casey Kindiger, veteran enterprise software solutions provider, IT automation expert, and founder of Avik Partners.
“Thoughtful action means using the type of faculties that a human being would use to make a decision and to take an action,” says Kindiger, who has more than a decade of experience in designing and developing process automation solutions for companies such as JP Morgan Chase, T-Mobile, and Allstate Insurance. Grok learns from your dev and ops environments and takes thoughtful action to detect anomalies, all in the service of Application Performance Management.
Grok works by analyzing real-time performance data from apps and infrastructure, looking for behavioral bumps in an otherwise smoothly-paved DevOps highway of deployed services and underlying infrastructure.
Grok’s Pre-Event Inferences
“Grok uses common statistical techniques to learn a pattern over a selected period of time in order to build a dynamic threshold of the continuously changing underlying patterns,” says Kindiger. In this way, Grok can establish a sort of dynamic baseline from very noisy patterns, which is something that would typically take a number of person hours of involved statistical and predictive analytics to accomplish.
Grok’s machine intelligence leverages the ability to make inferences based on this dynamic baseline information like a human being would in order to reason out a logical conclusion. Grok does this to detect the slightest of anomalies based on the difference between what reasonably should have happened and what actually did happen. Using this approach, Grok detects subtle patterns of significance in streaming data despite the fact that the data has patterns that are continuously in flux.
But rather than detect when something has gone wrong at a point in time, which is not scalable or sustainable, Grok looks for minute patterns that appear well in advance of any issue that you would normally consider looking at with traditional APM mechanisms, explains Kindiger. Rather than send alerts based on threshold-based exceptions, Grok makes a prediction on what the next data point or the next series of data points will be and then looks at deviations from those predictions, considering whether to flag those as anomalies, according to Kindiger.
Currently, Grok is particularly tuned to analyze streaming data and is, according to Kindiger, data-point agnostic. “We don’t need to teach the system what the context of a particular data point is. The system learns the context of the data over time and the subtle nuances of the pattern that produces over time, and creates the model based on that,” says Kindiger.
Watch Out for That Initial Two Week Delay
If Grok does everything that Avik Partners’ Kindiger hypes that it will, DevOps teams will soon be able to catch and act on infrastructure issues before they grow big enough for end-users to notice.
But Grok does have one wrinkle in its APM armor: it takes time for it to really learn a new customer system and get good at detection. Let’s assume that Grok’s machine learning causes it to get better at detecting anomalies in a given set of infrastructure over time. That means that there’s a point, at the beginning, given a brand new customer when Grok only starts to learn. At that point, Grok isn’t very good at knowing what’s anomalous compared to what’s normal for that infrastructure because Grok has no foundation for that particular system.
“Grok will take some time to learn the nuances of when to escalate an issue. We’ve learned from experimentation that it gets pretty good after ingesting two weeks’ worth of data,” says Kindiger. Because Avik Partners initially targeted AWS, it has taken two weeks of CloudWatch data and fed that to Grok to provide it with some backdrop so that when Grok goes live, it won’t be learning this environment entirely anew.
