devopsy.kinsta.cloud

Tag: IT operations

From One-Size-Fits-All to Custom-Fit for IT Ops

The downfall of the all-in-one IT management suite and the rise of the customized IT Ops toolchain

Here is how enterprise IT Ops used to work: you would make your choice among the Big Four vendors and buy in (quite literally!) to their strategy. This meant a suite of products that came preintegrated (more or less) out of the box, and—at least on the sales proposal—covered all of an IT organization’s needs across a huge spectrum of activities.

Photo by Barn Images on Unsplash

Broadly speaking, each of the Big Four’s suites included some flavor of the following: monitoring, ticketing, configuration management and asset tracking and automation and orchestration. Some vendors were stronger in one area than in another, but unless you had pretty specific needs, each vendor could be your one-stop shop.

Development and improvement inside these suites was mainly incremental and occurred at a fairly leisurely pace. Meanwhile, startups with bright ideas would come along occasionally and challenge the incumbents in specific areas, only to be bought by one of the mega-vendors if they showed enough promise.

There was a symbiotic relationship between vendor and IT buyer, as IT professionals built decades-long careers as experts in this or that vendor’s technology stack, with the wall of framed certifications to prove it. CIOs were reassured by the fact that there was a readily available pool of experts to hire, and continued reinvesting in the next iteration of their chosen vendor’s stack.

The Old Model No Longer Fits Modern IT

In the last decade or so, this comfortable status quo has broken down. The cracks were already showing before then, but only the most agile and forward-thinking organizations were moving fast enough to trip over them. As more and more businesses began to expect greater and greater responsiveness and speed from their IT departments, the huge sprawling suites of the mega-vendors were no longer able to keep up with the required rate of change. Leisurely 18-month release cycles no longer cut it in an agile world, where that time span may encompass several entire strategies.

In the old world, to call something “best of breed” was a sniffy dismissal of a one-trick pony, as opposed to the more supposedly versatile offerings of the Big Four. In the new world, it became obvious that aside from a handful of tentpole products that were genuinely good and fit for purpose, most of the individual components of the big suites was distinctly unexciting. Worse, the main value proposition of the suites—that they came preintegrated out of the box—started to break down as well. Many IT professionals were still fiercely loyal to individual products, but they became more willing to pick and choose, rather than buying into the whole vision.

In parallel, the rise of simple integrations—primarily REST APIs—made it easier than ever before to connect disparate solutions together into a single IT Ops toolchain. Worse (from the point of view of the incumbent vendors), sometimes it was easier to connect modern point solutions than their own component pieces, weighted down by the baggage of decades on the market.

How to Deliver the Flexibility and Agility the Business Expects?

These days, it is becoming commonly accepted even among the most staid and risk-averse organizations that the way to go is no longer to align to a single IT vendor and trust them to deliver on their vision, but rather to design custom-fit, multisolution toolchains, including home-grown, open source and commercial tools.

Photo by Max Bender on Unsplash

But what to do if you find yourself dealing with the consequences of this sort of IT monoculture?

Look Back At How You Got Here

The first step is to re-evaluate past buying decisions and strategic alignment. It is very important to do this without assigning blame; it was a different time, and those choices may well have been the right ones for the prevailing environment at the time, when integration between tools from different vendors was legitimately difficult.

Don’t Just Throw Everything Out

In many environments, too, the correct solution may well not be a clean sweep. One or more of those tentpole products that kept the overall suite looking good past its sell-by date may well still be perfectly fit for purpose and not worth the hassle of swapping out. However, other components are probably legitimately lagging behind the state of the art, in which case it is worth replacing them with something more fit for purpose.

Count Costs – All Costs

Include cost in these calculations, but remember to include all costs. Maybe you or a predecessor signed a big enterprise license agreement (ELA), so Product X is “free” in that context. But what does it cost to keep Product X running, in terms of ancillary licenses, hardware requirements and, last but not least, skill and effort on the part of the IT administrators? I don’t want to name specific products or vendors as examples, to avoid the discussion devolving into particulars, but it’s worth bearing in mind that many legacy products have shallow talent pools—not that many people are available on the market, and many of those people are coming to the ends of their careers, which makes them both expensive and reluctant to change employers. Instead, up-and-coming technologies often have much deeper and wider talent pools, with many more experienced specialists available at all salary and skill levels. If you don’t believe me, try hiring half a dozen COBOL programmers and half a dozen Node.js coders, and then we’ll talk …

Photo by Giu Vicente on Unsplash

Your IT Ops Toolchain, For Your IT Ops Needs

Moving from one of the big all-in-one suites to an assemblage of different tools, all flying in loose formation, is not easily done. Any technology transition of this sort is a Project with a capital P, and needs to be planned and managed with due care and attention. However, there is a large payoff at the end in terms of increased agility and flexibility within IT Ops. In turn, this affords IT the ability to offer their business counterparts the sort of responsive and even proactive IT support that moves IT from being a cost center, on a par with Facilities, to a strategic differentiator that enables new business initiatives.

Business as usual doesn’t cut it any more, and assumptions that used to be valid have fallen apart. Decisions based on those assumptions need to be re-evaluated. New, more flexible approaches will set up IT Ops for success in the newly flexible world we are already living in.

— Dominic Wellington

May 11, 2018
Red Hat Aims to Push IT Management to the Cloud

Red Hat at the Red Hat Summit 2018 conference this week showed a technology preview of an instance of Red Hat Insights, a predictive analytics IT management application capable of identifying operations issues before they become a problem, running as a software-as-a-service (SaaS) application.

The instance of Red Hat Insights will be extended to apply analytics to any infrastructure platform. Red Hat plans to eventually make the rest of its IT management portfolio available as a cloud service.

Joe Fitzgerald, vice president and general manager for management at Red Hat, said the goal is to enable management of virtual machines, container, infrastructure-as-a-service and physical servers all from a single pane of glass.

Cost analytics also will be embedded in those services, adds Fitzgerald. Cost is now a much bigger issue because cloud service providers bill customers based on actual consumption. Previously, a virtual machine left running on an on-premises server might result in someone being yelled at, but no additional cost, notes Fitzgerald. Now, however, if an IT organization doesn’t turn off those virtual machines, it will end up paying more.

Fitzgerald noted that management of IT operations is becoming increasingly automated. Red Hat is now adding integrations with the agentless Ansible framework it acquired in 2015 at rate of 50 a week. There already are 1,600 integrations with Ansible. He said the fact that Ansible presents IT operations teams with a declarative approach to automating IT using YAML files makes IT automation much more accessible. Other frameworks require IT operations teams to learn how to program, which becomes a significant barrier to adoption, but YAML files allow IT operations teams to make use of familiar shell scripts.

Many organizations now are moving up the IT management curve as applications become more distributed, Fitzgerald said. Every platform they have comes with its own management tools. But the need for a control plane through which multiple platforms can be centrally managed is now becoming more evident, he said, especially as IT organizations realize they will be running many different types of platforms for many years to come.

Other benefits to shifting IT management into the cloud is that the move eliminates the need to deploy a local server just to run IT management software and all updates to the IT management platform are provided by the applications provider.

Obviously, Red Hat is not the only IT vendor building a centralized, cloud-based approach to managing distributed computing environments. But as Ansible becomes more widely adopted, the opportunity arises for Red Hat to engage in a broader IT management conversation with IT organizations.

Most enterprise IT organizations are rapidly approaching something of an IT management crisis. Each platform adopted creates yet another silo that makes adopting integrated DevOps processes even more challenging. While most organizations are not likely to abandon the management tools from the vendors that built those platforms, IT organizations are demanding thet underlying infrastructure exposes an application programming interface (API). In fact, it’s the proliferation of all that software-defined infrastructure that now makes it feasible to centralize the management of IT in the cloud.

— Mike Vizard

May 10, 2018
Microservices Momentum Accelerates

A global survey of 353 application development professionals conducted by Dimensional Research on behalf of LightStep, a provider of application performance management (APM) tools, finds a full 86 percent of respondents expect microservices to be the default architecture within five years.

A total of 60 percent of respondents said they already have microservices in production or deployed as part of a pilot, and 92 percent said they expect to grow their use in the coming year.

Top reasons cited for employing microservices are agility (82 percent) and scalability (78 percent).

The downside of increased reliance of microservices is the need to manage them all. The survey finds 99 percent report challenges in using them. More than half (56 percent) reported that each additional microservice increases operational challenges, and 87 percent of those that have deployed microservices in a production environment reported they are generating more application data.

To make matters more challenging, a full 98 percent of respondents said they have trouble identifying the root cause of performance issues associated with microservices. More than three-quarters (76 percent) said it also takes longer to resolve issues. In fact, 73 percent said it is more difficult to troubleshoot application performance problems compared to a traditional monolithic application.

Not surprisingly, just less than three-quarters (74 percent) said they plan to increase investments in microservice performance management tools in the next year.

LightStep CEO Ben Sigelman said that while microservices come in many forms, advances in how software is developed is clearly coming at a cost. Not only is software becoming more challenging to manage, APM tools are now a prerequisite. Historically, IT organizations only monitored their most mission-critical applications. But the level of dependency each microservice has on others requires a more holistic approach to APM, Sigelman said, noting that IT teams now need to be able to see well beyond the border of any one microservice.

Challenges associated with managing microservices are not likely to deter organizations from eventually adopting them. The level of agility and resilience inherently built into in a microservices-based application are too high to ignore. But the rate at which they are employed might be impacted. There’s usually a direct relationship between the availability of tooling and expertise and the rate at which any emerging technology gets deployed in a production environment. The good news is advances in machine learning algorithms and other forms of artificial intelligence (AI) should make it easier to manage microservices in the years ahead.

In the meantime, IT organizations can take some comfort in the fact that developers are being held more accountable for the microservices they build. That level of accountability tends to vary widely by organization. But the end result is generally higher levels of software quality because developers increasingly realize they can’t throw code over the proverbial IT operations wall without it coming back to them. The challenge now is to bridge the historic divide between developers and IT operations teams in a way that works best for everyone concerned.

— Mike Vizard

May 10, 2018
Dell EMC Bets on Machine Learning to Drive IT Agility

Dell Technologies announced it is infusing machine learning algorithms across its portfolio of server and storage infrastructure to optimize the performance of each individual application using artificial intelligence (AI).

At the Dell Technologies World 2018 conference, the Dell EMC arm of Dell Technologies announced it has joined the Intel AI Builders program. Dell EMC and Intel will enable customers to benchmark Deep Learning and high-performance computing (HPC) workloads running on Intel Xeon Scalable processors and Intel field programmable gate arrays (FPGAs) integrated in the company’s PowerEdge R740 servers at the Dell EMC Innovation Lab and the Dell EMC customer Solution Centers.

Dell EMC also unveiled Dell EMC PowerMax, an NVMe-based storage array that takes advantage of an embedded machine learning engine into the storage operating system to optimize I/O performance. Dell EMC said that engine is capable of making 6 billion decisions related to optimizing 40 million data sets a day.

Finally, Dell also announced Dell Precision Optimizer software that compares the behavior of each application deployed on a Dell workstation against a machine learning model that automatically optimizes system settings. According to the company, Dell Precision Optimizer can improve workstation application performance band delivery up to 394 percent.

Ashley Gorakhpurwalla, ‎president and general manager for server and infrastructure systems at Dell EMC, said the plan is make it possible to compare the state of an IT infrastructure deployment against AI models that Dell EMC is constructing in the cloud using all the data gathered from the company’s systems connected to Dell EMC support services. Armed with that data, it will become possible to optimize system settings for every type of application deployed on it, said Gorakhpurwalla.

He noted that despite advances in DevOps, provisioning resources in a local data center on demand is still too time-consuming. Many IT organizations have adopted converged and hyperconverged infrastructure (HCI) platforms to unify the management of compute, storage and networking. But machine learning algorithms combined with other emerging technologies should make it possible to make IT infrastructure resources on demand in a cloud-like manner. That increased level of automation should enable IT organizations to allocate more resources to developing applications than managing the infrastructure those applications run on.

Each IT organization today is at a different stage of maturity when it comes to IT infrastructure. Dell EMC has one customer with a single administrator who manages more than 80,000 servers, Gorakhpurwalla said. Over time, IT infrastructure will become increasingly automated across a much broader range of customers, he noted—critical because it will allow local IT teams to be as agile as any cloud service providers. Today, many workloads are deployed in the cloud because of the perceived agility of the cloud, regardless of the actual costs.

It may take some time for developers to regain confidence in the agility of local IT departments. But as AI models continue to evolve it’s now a matter of when rather than if local IT infrastructure will become much more responsive to the needs of any given application.

— Mike Vizard

May 2, 2018
NGINX Extends Microservices Ambitions

NGINX has extended the reach and scope of a namesake application platform that the company is positioning as a bridge between legacy applications and emerging microservices architectures.

The NGINX Application Platform now includes NGINX Unit 1.0, an open source application server designed to simplify microservices across multiple application types simultaneously using a single instance of a server. It supports Go, Perl, Python, PHP and Ruby, with additional support planned for Java and JavaScript support soon. Sidney Rabsatt, vice president of product management at NGINX, said NGINIX Unit 1.0 is different from other application servers in that it is configured via an application programming interface (API) and is 30 percent faster than existing app servers.

NGINX in the second quarter plans to add new monitoring dashboards, alerting and management via NGINX Controller R1 offering that provides the capabilities of a service mesh for microservices without having to set up a separate dedicated framework. Service meshes are a critical capability because they provide a means to manage interdependent microservices at scale.

Finally, NGINX Plus R15 for enterprise customers builds on new HTTP/2, gRPC and JavaScript features that have been added to the core open source NGINX platform to provide support for high-availability clustering and OpenID Connect capabilities. These capabilities extend the ability to deploy NGINX as a web server, load balancer, API gateway, Kubernetes Ingress controller and sidecar proxy, said Rabsatt.

Rabsatt noted most IT organizations do not want to set up separate infrastructure for managing microservices. Managing legacy and microservices-based applications in DevOps environments will be a fact of life for years to come, he said, adding that based on the number of downloads of NGINX software being used on Kubernetes platforms running Docker containers, that message is already resonating.

NGINX claims users have pulled a billion NGINX instances from the Docker Store, including more than a million pulls of the NGINX Kubernetes Ingress controller. According to the company, more than 3 million NGINX instances are deployed in production microservices environments across more than 250 customers. That’s critical for the company because some rivals have positioned service meshes as an existential threat to the NGINIX platform. Overall, NGINIX says there are more than 11 million instances of NGINX are downloaded each month, an average of four instances every second.

Rabsatt said most developers have made it clear they don’t have the patience to wait for IT operations teams to stand up dedicated microservices infrastructures. Most IT operations teams still don’t have the skills required. NGINX provides the ability to deploy microservices at scale using a platform most developers already know how to programmatically invoke on their own, he said.

At the same time, NGINX is also committed to making available best practices that DevOps teams can employ when trying to manage what amounts to a new type of hybrid computing environment spanning legacy applications and microservices, he added.

It remains to be seen exactly where the divide between microservices and legacy applications will be laid down inside most organizations. Some will draw a proverbial line in the sand, while others will look to deconstruct existing applications into series of microservices. Regardless of the approach taken, all those existing legacy applications will not be disappearing anytime soon.

— Mike Vizard

April 6, 2018
VMware Adds Analytics to Automate IT Operations Based on Intent

VMware is moving to make it easier for organizations to automate data centers and public cloud based on its core hypervisor technology by injecting advanced analytics into its VMware vRealize Suite, enabling them to manage IT environments based on intent rather than having to manually manage every process themselves.

At the core of those update is the Capacity Analytics Engine, which is being added to version 6.7 of vRealize Operations. Mahesh Kumar, senior director of product marketing for cloud management at VMware, said the capacity analytics engine makes use of Autoregressive Integrated Moving Average (ARIMA) techniques to forecast future system events based on historical data points that identify a long-term trend of sudden spikes in usage. Armed with that data, vRealize Operations can now provide cost analytics that include advice on ways to adjust application size and reclaim idle resources on either a public cloud or inside a local data center. IT organizations can also run “what if” scenarios, Kumar said.

On top of that capability, VMware is also now adding predictive analytics that enables IT organizations to define a set of intents such as workload balancing to reduce software license cost, prioritize one set of application to maintain a service level agreement (SLA) or make sure certain workloads only run on servers that have been shown to comply with a specific regulation or mandate. Tighter integration between vRealize Operations 6.7 and vRealize Automation 7.4 then can be continuously employed to automate DevOps processes in a closed-loop manner at higher levels of scale without having to add additional headcount, Kumar said.

In addition, the latest version of VRealize Automation includes more than 120 free, curated blueprints and OVF (Open Virtualization Format) files out of the box. VMware has teamed with Bitnami to add 20 new blueprints and 100-plus new OVFs of applications and databases including GitLab, Hadoop, Jenkins and MongoDB to speed application development and deployment. VMware also updated the Custom Form Designer and added multitenancy capabilities.

Finally, VMware vRealize Operations is sporting a new user interface and integrates with VMware Wavefront monitoring software.

Longer term, Kumar said vRealize Operations will be where VMware applies machine learning algorithms to enable artificial intelligence (AI) models to automate the management of VMware environments.

Collectively, Kumar said the VMware vRealize Suite moves IT organizations that have standardized on VMware in both their local data centers and in the public cloud much farther down the DevOps maturity curve. In effect, VMware vRealize Suite creates an “easy button” for managing complex VMware environments spanning multiple clouds, Kumar said.

VMware is also betting that AI models will make it a lot easier for organizations to make a cultural adjustment as the management of compute, storage and networking becomes more unified. It’s not clear how long it will take for those machine learning algorithms to manifest themselves in VMware management software. But it’s clear most IT organizations running VMware soon will need to rethink who is responsible for running what inside an IT operations team.

— Mike Vizard

April 4, 2018
DevOps Chat: The State of Alerting & On-Call Survival for IT Ops, with OnPage CEO Judit Sharon

In this DevOps Chat we sit down with Judit Sharon, CEO of OnPage. OnPage has been providing alerting and incident management services primarily to the healthcare industry since before DevOps was called DevOps. As DevOps has exploded on the scene, the security and functionality built into OnPage was a perfect fit for IT Ops and DevOps teams. As a result, OnPage has expanded beyond its healthcare industry roots to become a very popular solution in the DevOps space.

OnPage recently conducted a survey on the state of alerting in the IT Ops space. Judit gives us some insight into the findings, but you can download the full survey report at www.onpage.com/the-state-of-it-operations-ebook/.

As usual, the streaming audio of our conversation is below, followed by the transcript of our chat. Enjoy!

Alan Shimel: Hey, everyone, it’s Alan Shimel, and you’re listening to another episode of DevOps Chat. Today’s DevOps Chat is a really nice one that I think you’re going to enjoy. I have as my guest Judit Sharon, CEO and co-founder of OnPage. Judit, welcome to DevOps Chat.

Judit Sharon: Thank you, Alan. Happy to join you.

Shimel: Happy to have you here. So, Judit, let’s start off a little bit, as I mentioned, you were the founder and CEO of OnPage, but maybe some folks in our audience are not familiar with OnPage. Why don’t you give us a little bit of background on the company and how you came to start it?

Sharon: Sure. So, OnPage is a SaaS based incident management platform hosted in a secure facility across the U.S., and in a nutshell, we—OnPage gives you visibility and feedback on incident status. You can track alerts, delivery, ticket status, and response, and it’s rock solid reliable. As one of our customers once said, it’s a must solution for those who need to elevate critical incidents.

Shimel: Mm-hmm.

Sharon: So, if I just go a little bit deeper, OnPage was built around the incident resolution life cycle to enable an organization to get the most out of their digital investment, ensuring that sensor and monitoring system have a reliable means to escalate urgent notification to the right person and immediately.

So, you know, Alan, just a few years ago, when we wanted to come up with a product, all we needed to do is plan the product, develop, test it, and deploy it. That’s it. That was the end of it. Today, in the DevOps era, we also collect data. We use a special tool to collect data, to remove the noise. We calculate the data, we learn normal behavior, we predict anomaly, and we even attempt to correct it if possible writing a little script.

So, we invest a lot in collecting this information, and as long as the process is working well, machine to machine communication is ideal. However, in the case of failure, disaster, or outage, when a technical person is needed, e-mail and SMS or even a phone call is not a sufficient means of notification, and OnPage is the best, most reliable solution to be used. And why? Because the technical person on call will get all the information right to their device, can start working on the incident immediately. If they’re not available, there are escalation rules that are pre-defined, and they can escalate it to the next person.

Furthermore, in the event that the person did receive it and now they need help from a different person, from a different team—so, they don’t need to know who is on call in that team, all they need to do, they can forward the entire ticket, add some notes to it, and send it to the different organization. OnPage will send it to the right person on call and will escalate it within the next team.

So, in one sentence, to summarize, OnPage automates the entire notification process to avoid human errors.

Shimel: Yep. And Judit, I want to call out another not feature but characteristic in the history of OnPage. You know, we see so many companies that saw DevOps as a gold rush. You know, founders who say, “Wow, I want to start a company around DevOps,” and they build a tool for DevOps.

Not the fact with OnPage. OnPage has actually been around for some time, and its mission dovetails nicely with DevOps but it’s not one of these Johnny-come-latelies trying to capitalize on how many people are searching Google for DevOps or something like that. Correct?

Sharon: Right. So, we started about seven years ago. We started as incident notification, and actually, we started in health care. That’s why our solution is HIPAA-secure, something that is very unique about us.

The other thing that we did, because we started in health care, was, we needed to alert until read. So, we are very unique in that, that we don’t just send you an alert, like send an SMS or an e-mail or send you just a push notification. We will alert consistently for eight hours to ensure that the message was delivered to the right person. And we have the audit trail, so now, accountability is a must. Now, we have the audit trail of who got it, when, how long it took to open the message.

And another very nice feature other than the fact that we are HIPAA-compliant, so you know, we are secure, all the communication is secure at rest, and it’s encrypted in our database. We also save all the information for up to six years. So, other than the fact that we are HIPAA-compliant and other than the fact that we have the alert until read—we have other very nice features. Right now, there are a lot of companies, large companies mainly, that have a lot of systems and processes and sensors that sense just e-mail notification. And you will see from our study that we will talk about later, a lot of people are using e-mail notification for incident notification. And we just integrated those e-mails, and basically, you can leverage all the OnPage capabilities and elevate those messages if they’re important.

Yeah, that’s what we do.

Shimel: Sure, absolutely. And Judit, I wanted to kind of shift the conversation, if we will—if you will—to a larger topic, and that is sort of the state left alerting in the IT Ops world and the state of IT Ops. I know recently OnPage conducted a survey of IT Ops professionals. There’s been a few articles on devopsy.kinsta.cloud about it by Aura Lee, love of your company as well as download the full report.

But for those of our listeners who maybe haven’t had a chance to go on the site and check out the report—and we’ll put links to it in our show notes—can you talk a little bit about that, give us a little insight, here?

Sharon: Yes. So, the state of IT operations, we conducted a study where we surveyed about 96 different companies, and the nice thing about that survey, we got a few people from each organization. So, we see the response from different levels of leadership. So, 20 percent were SVP or C level, 22 percent were directors, 34 percent were managers, and 24 percent were individual contributors. So, you see a nice distribution among the different levels in the organization.

And the findings were—I mean, first, what did we want to look at? We looked at how teams are notified in critical incidents, how many alerts the team received, who received the critical alerts, and how did they get handled?

So, the findings were really interesting. As I told you before, we found that 80 percent of the respondents said that they get notified via e-mail. You know, e-mail—the same e-mail that you get as junk mail, the same e-mail that you get on a regular basis—80 percent of IT get their notifications via e-mail. Which means they need to be on top of e-mail all the time. They’re always on, they’re never off. And I don’t know if you remember Target, they had some cybersecurity?

Shimel: Mm-hmm.

Sharon: Apparently they did get alerts, right? They did get notified, but they got notified via e-mail. So, three days later, they found the e-mail that they got notified. So, what OnPage says is, “Yes, you have processes that are sending e-mails. You can always change those processes and maybe extensively change the processes.” But why don’t you elevate those e-mails so they find you in case of emergency rather than you sit there and always, always check your e-mail?

Another interesting point that we found is that 40 percent of IT teams get more than 10 alerts per day; 20 percent get more than 20 alerts per day. So, couple that with the fact that two-thirds of the respondents saying that they sent alerts to the team—so now, you don’t send alerts to one person, every alert gets sent to the entire team because they don’t want to miss the alert, right? So, what OnPage does, they say, “Okay, you get a lot of alerts. Don’t send it to the entire team. Send it to an escalation team, so one person will get it, and then always change who is the first one in line so the burden gets distributed.” So, you don’t get as many alerts, and you don’t—like, basically, the workforce, right, and the quality of their life, you don’t diminish it completely.

So, that was just a few very, very interesting facts that we found about that demographic that we surveyed.

Shimel: Yep. You know, Judit, I’d like to make a comment, but more of a question to you and get your advice, because this is an issue we deal with here at devopsy.kinsta.cloud, even. We have tried so hard to move off of e-mail as our primary communication, alerting, operational format. And we have tried. We’ve tried things like Slack and Chat and Google Chat and texting and I can’t tell you how many different tools we’ve given a try to. And yet, we keep coming back to e-mail.

So, it doesn’t surprise me that many Ops people still rely on e-mail, with all the spam and all the noise to signal ratio problems there. What do you—for people listening out here in the same boat I am in—what’s your suggestion? What’s your advice?

Sharon: Well, I think that e-mail is really good for audit trails, right? You want to know what happened, who said what. All the communication gets distributed to a distribution group—that’s fine.

What we say is, when it’s critical, right—e-mails are okay for day-to-day communication. E-mail is okay, it’s actually for events or incidents that don’t need to be taken care of right away. So, those events that don’t need to be prioritized, they should go to e-mail. That’s absolutely true.

What we say is, when there is an event that needs your attention immediately—and that’s another thing that I would like to go into are the costs, right? Those incidents that need to grab your attention immediately, those should be elevated. Those should find you rather than you always check your e-mail and always be on.

Another interesting thing that we’ve found is, 12 percent said they are still using a pager.

Shimel: Yeah?

Sharon: And then we looked into those 12 percent, what vertical they come from, and we found that they come from the healthcare industry. Come to show you that the IT follows the technology implemented in the organization; they don’t kind of bring new technology for themselves. So, if the health care organization is using a pager, the IT organization of the same organization will use pager. Only with those who responded. We have a lot of customers, health care IT, that are using OnPage to elevate those messages.

Another thing I want to get into is the cost, right? How much it costs. So, 99 percent of the organizations say that a single hour of down time costs them $100,000.00. So, every minute that we delay working on it and remedying the issue costs you about $1,670.00—every minute, right? You know, the counter is working, and we found that 62 percent of responders, it takes them 5 to 15 minutes to deliver an alert. Twelve percent said it’s more than 15 minutes to deliver an alert.

So, here we see that just the delivery costs of the down time is between 8,000 and 25,000 per incident. Now, compare that to the cost of the solution. The cost of the solution is almost like a no-brainer. When we started the company and we pitched the idea, it was a quote—it’s a no-brainer. The cost is so miniscule in comparison to the cost of the lost time in delivering the alert to the right person and having the audit trail and having the reporting so you can afterwards go and even audit your team, make sure that everyone is accountable, that everyone is reliable.

So, that was an interesting finding from this survey. Actually, we think it’s so interesting that we would like to run the survey in a different venue as well to try to get maybe a bigger audience, because this one was about 200 to 150 responders, but maybe if we can go on a larger scale and see if it’s working the same across all industries.

Shimel: Sure. That would be fascinating. And I think—you know, it’s funny, I happen to be working on an e-book we’re writing on the state of DevOps in 2017, and I will tell you, there’s frankly not enough research being done and that’s available on these kinds of subjects, right? I mean, people are going more on kinda what their gut tells them on subjective rather than objective data.

Sharon: Yeah.

Shimel: And that’s always—you know, sometimes it works, but it’s always risky. So, I would encourage you to do that if you guys can, for sure.

Sharon: Right. The other thing that we found that, you know, security became really, really important, like, this is top of mind for everyone. It used to be only health care, right, that cared about security and compliance, and now we go through one survey after another about our security policies from IT organizations, right? So, security becomes very, very important, and 40 percent of the responders said that encryption, even of alerts and messaging is very important for them. So, that’s another nice piece of information that we got from this survey.

Shimel: Absolutely. Well, Judit, as I mentioned to you off mic when we were talking before recording today, the time goes really quick here. We’re already—we’re well past 15 minutes, so we’ll try to be respectful of our audience’s time, and maybe we can have you on for another chat to continue the discussion, maybe if you continue another leg of the survey.

But thanks for being our guest on this DevOps Chat. I just wanted to end it, because I forgot to ask you in the beginning. For people who want to get more information about OnPage, where can they go?

Sharon: So, of course, there is our website, www.OnPage.com. If you want a copy of this survey, it’s located at devopsy.kinsta.cloud, and there is another location on our website, OnPage.com/DevOps-ebook. A little bit more complicated than devopsy.kinsta.cloud, but we are going to publish another part of this survey, because we put the white papers in two sections, so part two is going to come up soon.

Shimel: Great.

Sharon: And I would love you guys all coming to look into what we do.

Shimel: Okay! Judit Sharon, founder and CEO of OnPage, thank you for being our guest.

Sharon: Thank you, Alan. Pleasure to be here.

Shimel: Pleasure to have you here. Judit, we’ll see you soon.

Sharon: Sure.

Shimel: This is Alan Shimel for devopsy.kinsta.cloud. Thanks for joining us. Hope to see you soon on another DevOps Chat, and have a great day, everyone.

— Alan Shimel

March 21, 2018
When IT Disaster Strikes, Part 3: Conducting a Blameless Post-Mortem
In the first and second parts of this three-part series, we looked at how organizations can effectively resolve incidents and the role of each person involved in the resolution process. In this final piece, we’ll look at what organizations must do post-incident to better understand what went wrong and learn from their response.

At the time of occurrence, an outage is every IT practitioner’s worst nightmare. However, after the dust settles, each incident presents an opportunity to learn as a team, make an impact on your processes and improve your product. This means that even after you’ve found a resolution, your work isn’t quite done.

Enter the Incident Post-Mortem

Once you’ve successfully fixed a major issue and the chaos of the firefight is winding down, it’s time to conduct an incident post-mortem with your team to investigate the problem and figure out what went wrong. Without a post-mortem, you fail to recognize what you’re doing right, where you could improve and, most importantly, how to avoid making the same mistakes in the future.

Here are my three tried and true steps for a successful post-mortem process:

Designate a Post-Mortem Owner

To kick off the post-mortem, the incident commander should assign an owner at the very end of an incident. This owner is responsible for scheduling the post-mortem meeting, which should take place within five business days of an incident. Owners are also responsible for populating the post-mortem document, investigating the incident, pulling in other people to assist and facilitating the post-mortem meeting. In cases where a public blog post is required, the owner will also be responsible for creating the content and putting it through necessary internal review cycles.

Populate the Post-Mortem Document and Conduct Analysis

Once an owner has been designated, the team should get to work updating the post-mortem document with available information, including the timeline of status changes and key actions taken by incident responders. Every participant in the incident response should be included in this page, so be sure to go back through all communication and add relevant parties. After you’ve updated the page with the full timeline of actions, you will need to build out details for each event. For each item in the timeline, identify a metric or a third-party page where the data came from. This could be anything from a link to a Datadog graph or Splunk search to a relevant Tweet, as long as it shows the data point you’re trying to illustrate.

With the timeline in place, you can begin analyzing the incident. Capture all available data about what happened, what caused the issue, how many customers were affected, what impact they experienced, etc., to determine the underlying cause of the incident. This information should be detailed, as it will dictate follow-up action tickets your team will use to prevent recurrence of the incident and improve the incident response process. It also will be examined during the post-mortem meeting, so the more context you can include in your analysis, the better.

In the case of severe incidents where external communication is required, this is also the time when the post-mortem owner and team should develop customer communications. Pro tip: Avoid using the word “outage” if possible in this communication. Many people see the word outage and assume that your entire system was down, when in reality that was likely not the case. Be precise here, as it is important for both internal learning and sharing within the larger IT community.

Here is a general template for what to include on the post-mortem page:
- Overview
- What Happened
- Root Cause (or Contributing Factors)
- Resolution
- Impact
- Responders Involved
- Timeline
- How’d We Do?
- Action Items
- Communication (Internal and External)
- Hold a Post-Mortem Debrief Meeting
The final step in the process, the incident post-mortem meeting, should include the incident commander, service owners, key engineers/responders and the customer liaison (for severe incidents). Use your post-mortem document as an agenda to help things run smoothly. The discussion should center around what happened, what the team could have done better and any followup actions that need to be taken. Any disagreements about the facts, analysis or recommended actions should be cleared up during this time, and everyone should walk away from the meeting with a better sense of problems impacting your service’s reliability and having learned more about the service involved.

The No. 1 rule of running an incident post-mortem is to keep it blameless. The goal of the debriefing process is not to point fingers, but to learn what happened and how you can improve as a team.

Don’t make the mistake of neglecting the post-mortem process after a major incident. A well-designed, blameless post-mortem empowers your team to continuously learn and offers a way to iteratively improve your infrastructure and incident response process. Not only does this help your team, but it also helps create amazing software and services.

— Eric Sigler
March 5, 2018
The State of IT Ops

In our most recent article, we highlighted some of the results of our recent IT Ops survey. We noted, for example, that 80 percent of IT teams are alerted to critical events via email and 41 percent of teams are experiencing excessive alerts. These stats go a long way to explaining why IT Ops is experiencing so many issues with alert fatigue.

In this article, we will delve into the second half of our findings and examine our results on how IT teams reported they handle alerts after they come in.

Off Scheduler

Efficient IT operations teams realize the need for directed action when critical issues come in. This means that when significant IT issues occur, efficient teams will have alerts sent directly to the on-call engineer, who can manage the incident most effectively. The engineer will be the individual who is best-equipped to manage and quickly resolve the issue at hand.

Our survey however showed that most teams lack automatic schedulers which could ensure the correct individual is alerted. Rather, alerts are often sent to the team as a whole, or to individuals who are unable to resolve the issue. According to our statistics, teams are frequently alerted (67.3 percent) instead of the on-call individual.

The impact of this poor workflow is not only that issues are sent to individuals who are unable to resolve them, but time is wasted by this very process. Ideally, team managers should invest in scheduling software that sends alerts directly to the engineer on-call. By sending it to the wrong individual or team, companies are adding fuel to the fire of alert fatigue and burnout.

Related Articles:

Site Reliability Engineering: How to Make the Operations Side of DevOps Actually Work

Slack and Critical Alerting

Unfortunately, our team also highlighted that most teams lack scheduling technologies to enable this outcome. Our survey highlighted that only 28 percent of teams use an automated scheduler to ensure the correct individual or team is alerted.

By failing to use a system that ties alerts to on-call schedules, managers receive alerts via email, SMS or phone that they are not necessarily responsible for resolving. Managers then need to track down the on-call engineer who can manage the incident. The pain of this scenario is multiplied if the alert comes after-hours or during the night. Again, this workflow leads to significant delays in resolution time.

No Time for Downtime

As IT Ops are often tasked with issues such as maintaining server uptime or site uptime, the prospect of downtime is something engineers work hard to avoid. IT Ops are under significant pressure to maintain five-nines uptime.

For this outcome to occur, teams need to automate menial tasks and ensure they are quickly alerted to complex downtime issues that demand their attention. The goal needs to be getting the issue as quickly as possible in front of the individual who is best able to resolve the matter.

However, our survey showed that 45 percent of teams indicated it takes more than 10 minutes before IT teams even begin to resolve critical IT issues. Given that the cost of downtime is more than $5,000 per minute, according to Gartner, downtime can easily balloon to cost companies or their clients more than $50,000 in 10 minutes.

The significant expense cause by downtime has been seen in many recent scenarios. For example, in mid-May 2017, British Airways had a spectacular downtime event due to the failure of a power system in the company’s data center. British Airways was forced to cancel all its flights leaving from London airports after having a major IT system failure. The result was cancelled flights affecting more than 75,000 passengers. The costs of reimbursement were approximately $68 million and a 2.8 percent drop in the stock price of the parent company.

Failure to Collaborate

In the world of DevOps and IT Ops, we are often told of the importance of collaboration and how it leads to great innovation. Indeed, the goal of DevOps is to bring developers and operations into closer proximity, so they can collaborate more effectively. Better collaboration means better innovation and decreased costs.

However, our survey shows that while IT Ops might have the mindset for collaboration, they lack the necessary tools for ensuring that collaboration happens. Our survey showed that IT Ops teams often collaborate on tickets by email. According to the results of our survey, 86 percent of teams collaborate through email.

Yet, email is a truly poor way to collaborate when critical incidents come up. Email has few ways to bring critical emails to the top of the stack and keep them center of mind. Instead, email’s default is to let messages get buried under other emails. In attempting to collaborate on key issues, email fails to offer rapid communication methods that highlight critical messages.

Tools are available that enable better and faster communication among team members. Slack is a tool popular among many IT Ops engineers who are key drivers of effective collaboration. Yet, our survey showed that only 16 percent of ITOps use ChatOps as a form of collaboration during incidents. Slack’s motto is, it’s “where work gets done.” On their own, however, tools such as Slack lack a method for bringing important issues to the top. However, if attached to incident management systems such as OnPage, Slack and other tools like it can enable users to receive alerts that a critical issue needs to be addressed.

Improving Workflow

The goal of improving how teams handle alerts is to minimize downtime and improve efficiency. These shouldn’t seem like lofty goals. However, by not developing specific workflows that bring issues to the top and enable them to be resolved quickly, IT Ops teams are adding to the time until resolution and the ensuing cost.

At its core, the lack of specific workflows and tools to enable improved workflow points to teams that have much room for improvement. IT Ops teams need to find better ways to manage day-to-day operations so that when critical incidents do arise, the path toward resolution is quick and doesn’t require engineers to embark on a guessing game.

Conclusion

Unfortunately, the state of IT Ops has not progressed to the level of maturity one would hope. There is still a lot of poor alerting and workflow practices that could be improved if a more robust incident management system were employed.

There are a number of additional insights that can be garnered from our survey. I encourage you to take a moment and download a copy of the study and see what you can learn that will help your team.

— Orlee Berlove

February 15, 2018
HashiCorp Adds Terraform Workflow Tools for IT Ops

HashiCorp this week moved to templatize automating IT operations by making it easier to share Terraform configuration files which make it simpler to programmatically invoke infrastructure that exposes application programming interfaces (APIs) without knowing how to code.

Available in an on-premises edition or as a software-as-a-service (SaaS) application, HashiCorp Terraform Enterprise is intended for use by IT operations teams. A HashiCorp Terraform Enterprise Module Registry function has been added to enable IT operations teams to implement a publish-and-subscribe mechanism for Terraform configuration files.

Company CTO Armon Dadgar said HashiCorp Terraform Enterprise is designed to enable IT operations teams to respond faster to requests from developers for infrastructure resources without requiring them to develop programming skills. It’s unlikely developers working for traditional enterprise IT organizations will be taking over control of IT infrastructure anytime soon, he said, and Hashicorp Terraform Enterprise allows IT organizations to inject a level of agility into their workflow processes without requiring IT operations teams to completely re-engineer existing processes.

The HashiCorp Terraform Enterprise Module Registry extends that capability further by making it easier for IT operations professionals who have developed expertise employing declarative Terraform configuration files to share those files with other members of the team, Dadgar said.

Related Content:

HashiCorp Terraform Enterprise Empowers IT Administrators

Operations Management Puts the ‘Ops’ in DevOps

At the same time, IT operations teams that want to build workflow applications on HashiCorp Terraform Enterprise can use a set of application programming interfaces (APIs) that HashiCorp has exposed. In addition, the company has revamped the user interface for Terraform—workspaces now can be created by combining Terraform files into modular components that can be assigned to specific IT teams. Permissions to access are provided via support for the secure access markup language (SAML) as well as the Sentinel policy management software developed by HashiCorp.

The HashiCorp approach to automating IT operations strikes at the core of the DevOps debate in the enterprise. While web-scale companies typically can afford to hire IT professionals with programming skills to manage IT infrastructure, the average enterprise IT organization still relies on administrators to manage servers, storage and networking. Not only are IT administrators in short supply, most of them would not be in IT operations if they knew how to program—they’d be writing and testing applications instead.

Rival approaches to automating the management of IT infrastructure, despite being around for years, have yet to gain mainstream acceptance, mainly because they require programming skills. Declarative approaches allow IT infrastructure to be managed as code still, with less resistance from IT operations teams that tend to be more comfortable employing declarative tools. In fact, there’s long been a chicken-and-egg debate over the degree to which the transition to modern DevOps practices requires organizations to acquire new tools first or vice versa. The HashiCorp approach strikes a middle ground by providing tools that enable IT operations teams to first become more efficient, which ultimately should enable them to implement new processes that often is much less disruptive to the existing IT culture.

— Mike Vizard

February 9, 2018