devopsy.kinsta.cloud

Tag: SRE

Scaling Predictive Analytics With AIOps to Drive Next-Gen SRE

Enterprise systems are only as valuable as they are reliable, in the sense that they don’t suffer excessive breakdowns. Otherwise, companies experience costly downtime and added stress for engineers due to the additional burden of managing issues. This critical function of ensuring systems run reliably and optimally, at production scale and with minimum human intervention, is the purview of site reliability engineering (SRE) teams.

The SRE professional’s job is to reduce human toil by developing and implementing reliable and highly scalable systems that optimize software and applications. Their goal is to proactively identify and, where possible, anticipate potential disruptions to minimize risk and ensure optimal system performance. However, to be successful, SRE teams require a level of enterprise-wide visibility that can be difficult to provide without the right tools to capture and analyze data at the appropriate level of detail.

Let’s examine how artificial intelligence for IT operations (AIOps) can support the SRE mission by connecting data enterprise-wide for unprecedented visibility and control of systems and processes, enabling auto resolution of most issues and more efficient triaging for the few remaining cases where SRE teams must collaborate around a fix.

Challenges of Complexity and Scale in Pursuing SRE Excellence

As more enterprises digitize their operations and move to greater automation, their IT operations must leverage all data assets skillfully in order to improve reliability and reduce human toil. The SRE profession arose to satisfy this need, with a focus on monitoring systems, accessorizing automated releases, understanding change impacts and automating some of the most common system processes.

To do their job effectively, SRE teams need a wide variety of data and analytics capabilities at their disposal. This includes the ability to deep dive into descriptive and diagnostic analytics to look backward and at present conditions to discover what happened in the past and why and to baseline current operations. But this is just the beginning. The true value of SRE comes with the scaling of predictive and prescriptive analytics to draw on that historic data and apply detailed analysis to generate predictive insights into what is most likely to happen in the future; these insights are the basis for identifying proactive measures that can serve to address any potential problems and optimize those future outcomes, minimizing adverse impacts on operations

These analytic capabilities must be fed by robust data that comes from across the entire IT estate. Developers assigned to the SRE role face an especially strong mandate to have this holistic visibility as they plan, build, test, release, monitor and secure systems; their job is hampered to the extent that organizational silos or problems of scale get in the way of achieving that visibility. It is here that AIOps can help evolve IT operations to become more proactive and autonomous by scaling the power and reach of predictive and prescriptive analytics.

AIOps Empowers the SRE Mission Enterprise-Wide

AIOps is an essential tool for the SRE community in the battle to reduce operator stress, configure IT systems to be more stable and run efficiently with less human intervention. AIOps employs artificial intelligence (AI) and machine learning (ML) for observability, context, normal behavior analysis and automated health diagnostics. This, in turn, enables anomaly detection in real-time and closed-loop automatic resolution of most issues.

For example, AIOps can uncover patterns showing that, 90% of the time, a particular alert in the organization’s payment system triggers a seemingly unrelated alert within 15 minutes. Moving forward, this gives SRE teams a 15-minute head start on addressing that secondary alert; the discovery forms the basis of an auto-resolution or triaging scenario that uncovers the underlying cause of both alerts and provides a permanent fix.

That’s just one illustration of how AIOps uses advanced analytics to discern subtle patterns in data to predict when and where problems may occur, so proactive fixes can automatically be prescribed to head off those problems. In this way, AIOps is the technology backbone that scales and automates the SRE team’s insight and control across all system assets and dependencies, a critical tool for continuous optimization of performance with minimal human intervention.

Conclusion: The Game-Changing Role for AIOps in SRE

AIOps is a game-changer for the critical function of site reliability engineering. Powered by a potent blend of advanced analytics on robust data coming from systems across the enterprise, AIOps delivers unprecedented visibility and control for SRE teams in their mission to reduce toil and ensure the reliability and resiliency of enterprise systems. The result is less human intervention and enhanced value from IT systems that are made more robust, more stable and more efficient.

September 26, 2022
Kintaba Creates the First Conference Dedicated to Incident Response – Techstrong TV

John Egan, Co-founder and CEO at Kintaba, talks about IRConf, the first conference fully dedicated to Incident Response. The company behind the conference is Kintaba — a modern incident management platform for the entire organization. The video is below followed by a transcript of the conversation.

Alan Shimel: Hey, everyone. Welcome to Techstrong TV. We have got another segment for you here. I’m happy to be joined by John Egan. John is the CEO of Kintaba Incident Response.

Hey, John. Welcome to Techstrong TV.

John Egan: Thanks, Alan. Good to be here.

Alan Shimel: Nice to have you. So, John, you know, I might have messed up what your position with Kintaba is.

John Egan: That’s right. CEO and Co-Founder.

Alan Shimel: Okay. And you know what? I mean I’ve heard of Kintaba, but that doesn’t mean our audience knows Kintaba, so if you wouldn’t mind, give us maybe a quick little background on Kintaba and maybe a little bit of your own background?

John Egan: Yeah. So, Kintaba is an incident response platform. We’re designed to let any organization be able to easily implement best practices for responding to critical moments within the company, so this tends to be things like site outages, major customer issues, and other sort of technical and non-technical problems that require kind of a group real-time response, you know, separate from like project management where you might have planned things that are happening. You take, like, the outage that happened, I think yesterday, at GitLab was a good example, right? Or a slack outage like that, that has an incident response team that’s responding to that situation and making sure it gets fixed quickly, and they’re probably using some kind of tool, either internally or off the shelf, and we’re that tool that any organization can implement in minutes instead of bringing in a consultant and spending months on this kind of stuff.

Alan Shimel: Absolutely. You know what was interesting, John, is I mean look, I’ve seen, for whatever reason since I’ve started devopsy.kinsta.cloud eight years ago, incident response has always been like this, you know, it’s part of the DevOps story, and we’ve seen of course, you know, several companies have very successful exits and be, you know, very successful. They all kind of follow this pattern where they, you know, were replacing the beeper, then they became incident response. Then it becomes a whole platform, and APIs and integrations, and so forth. What makes Kintaba different?

John Egan: So, we’re really purists on the incident response side of things. I think a lot of organizations that come into this space end up either kind of pivoting over towards alerting, all right, which is sort of Pagerduty’s world –

Alan Shimel: Right.

John Egan: Which is more about automated detection, ping someone, it gets closed out, you know, you might get thousands of these things across the organization, or they go broad and start to become, you know, more of like an SRE platform to try to handle all of these different things that the site reliability engineers are really managing inside of the organization. But we really believe that incident response stands on its own as a practice that cuts across all of the organizations inside of your company. So, something unique about Kintaba is normally when you’re rolling it out within the organization you might start small. You might start with five or six seeds, a couple of people in SRE, but really rapidly we tend to get deployed across the entire organization and what we do really well is we’re a collaborative layer that helps everyone from customer success, to your SREs, to even your PR and marketing guys all be connected during these critical moments and all have one place that they can go to see what’s happening and participate in that response.

It’s a bit more of, you know, we call it an evolved sense of incident response, right? Versus kind of the old-fashioned way of looking at it, which might be two or three people in your SRE team are responsible for this. It’s really more about the entire community of your organization that responds to these company critical moments and building software that makes it easy for all of those people to participate. So, we’re a very non-technical feeling piece of software. We’re very human driven, everything from our reporting, to our collaboration experience, to the way you go and write documents, which is a new feature we’ve just released, all of this information is designed to be as easy to use as something like Google, as opposed to being, you know, more like a, you know, almost like a service now, kind of like complex enterprise implementation.

Alan Shimel: Got it. Good stuff. And, John, we’re going to talk about this conference you guys are putting on, but before we go, before we dive into that, you know, so we understand what Kintaba is. You know, in your go-to market here what are you guys finding? What are you seeing? You know, what’s kind of the state of the market, if you will?

John Egan: So, we’re really learning a lot about this community as it’s just starting to form, right? I think when communities are young and they’re new you actually find the best practices are pretty fluid and I think it’s one of the unique things about this space is Google put out a book back in 2016 where they sort of defined how they handle incident response internally. It was the first time anyone really wrote it down in a way that could be absorbed, but it turns out every organization has a slightly different attitude that they take towards this space and when you’re implementing a product, something like Kintaba, it really turns out that that product has to be pretty flexible on its way in to make sure that it’s easy for people to adopt, and then it also has to bring with it sort of almost like a light hand to say these are the very high-level macro steps you’re going to take to be successful, right?

And so, what we’ve learned early on is when Kintaba first launched, you know, over a year ago, we were very rigid about the way that you sort of had to come in and step through, you know, each part of incident response. What we’ve really learned and what the products evolve towards is to have rigidity around the high level steps report openness around visibility to when incidents are happening, like easy to use collaborative spaces, post mortems that editable and changeable, but also, be flexible in terms of who’s getting access to this thing, how are you giving people titles that are non-technical in the, like, customer success roles, you know, and in the marketing and PR roles, and how do you bring all of that together at once. We thought when we came in that, okay, maybe this would be a couple of seat experience for people, you know, you come in and maybe you need two or three seats, greater, four or five, and what we’ve really learned is these are company practices, like I was saying before, and I think that’s been kind of our big learning about the space is as the community grows this is really a cross-functional community inside of organizations rather than just a technical practice inside of DevOps and I think that’s really exciting.

Alan Shimel: Yeah. Absolutely. Absolutely. So, John, you guys are coming in behind the first dedicated incident response virtual conference coming up in April. Why don’t you – well, I already said it I guess, but why don’t you take it from there? Tell us more.

John Egan: Yeah. So, IR Conf, or Incident Response Conference, was really built out of the fact that we just couldn’t find another one to be part of that was focusing purely on the incident responder. When we talk about communities internally, you know, you think of communities as what makes people put their hands up and say, “I’m part of that,” right? I think DevOps, and SRE, and a lot of these spaces have done a good job, right? We have crowds and people will say, “Okay. I’m part of that.” Incident response has always been one of those ones where I think people sort of nervously said, “Well, I think I’m part of that.” What IR Conf really does is it focuses not on sort of that statement of I’m in the incident response department, but instead focuses on the idea that I’m a responder, right? I’ve been part of responding to a major incident and when you ask that question it turns out a lot of hands go up and it turns out that there’s a community there that’s really looking for other practitioners to come together and talk about these things I was just talking about, right, best practices, right? What are other companies experiencing? How are we defining this role and the people who are involved? And so, what’s really cool about this conference is we get to focus really directly on that incident response process, you know, not even necessarily as it relates to other parts of DevOps, but just how we deal with this very specific problem that faces all organizations.

So, I’m really excited we have one our keynote is Emily Freeman, who’s out of AWS –

Alan Shimel: Sure.

John Egan: And has done all kinds of great talks about kind of the wider, softer development lifecycle inside of DevOps, and here she can speak much more directly, you know, to this response, incident response space. I think you can see the excitement around it. You know, we have sponsors, basically everyone across the DevOps world, Honeycomb, Commodore, Rookout, BridgeCrew, Gremlin, Teleport. All of these folks, you know, really came running in and said, “Yes. This is great. We need a space to come in and talk to our sub-communities that are already using our products that care about this space,” and so really, we just felt like there was a hole in the market and so this is a free conference. Anyone can join. This is a virtual conference. It’s on April 1^st. It’s at IRCONF.IO if you’re out there looking for it. We’re really hoping to just get everyone to come along who’s ever been involved in an incident before and has experienced that fire moment inside of a company of, you know, hopefully not panic, but maybe panic, and wants to come together and talk to other folks, who have had the same experience.

Alan Shimel: Sure. I’m thrown back to my cyber – we didn’t call it cyber then – to my infosec days in the war room, you know, when stuff hit the fan. You know, best practices that were to set up a war room on how to respond to a breach, you know, some sort of attack.

John Egan: And that’s really where a lot of this grew out of, right?

Alan Shimel: Sure.

John Egan: We have incident response really lived in the security world –

Alan Shimel: Yeah.

John Egan: And it’s one of these parts of the evolution of the space, right, that now we’re adopting, you know, that process that really got its feet inside of the security incident response space across the organization to say, “Wow. These guys are doing a great job,” like, “We should be doing similar work across the org.”

Alan Shimel: Not all companies had such a great job on their war room, but that’s a whole other story.

Hey, John, you mentioned Emily Freeman speaking. Who are some of the other speakers?

John Egan: We’ve got two more who are announced right now and it’s Emily Freeman, and it’s Dave Rensin. Dave Rensin was previously at Google and ran a good chunk of the SRE Team there. He’s at a new organization now and he’s spoken at a couple of other conferences in the DevOps space, specifically about the process of incident response. I’m really excited about him being able to come in as well and talk, both, from a large company standpoint, right, a company like Google, where he really pushed a lot of customer metrics, right, for dealing with incidents, not just internal technical metrics, as well as the smaller company, like where he’s moved to now is a startup.

Those two are kind of our cornerstones. Then we’ve got a series of other speakers, who are announcing over the next couple of weeks –

Alan Shimel: Who?

John Egan: Who are coming in from all over industry, so I’m really excited. I’m can’t announce them here yet. They’re not finalized yet.

Alan Shimel: Okay. Good.

John Egan: Really exciting. Great organizations.

Alan Shimel: Can’t blame me for asking. All right. Hey, John, for people who are interested, that was IRCONF.IO?

John Egan: That’s right. IRCONF.IO. You go in there and just click Attend and register. We’re still at early bird, so everything is free right now.

Alan Shimel: And it’s April 1^st. Do you have times on that?

John Egan: I believe we’re planning to start it early, around 8:30 is the beginning of it.

Alan Shimel: East Coast time?

John Egan: It’s technically a half-day, but it runs until about 2:00, I believe.

Alan Shimel: So, that’s East Coast time though?

John Egan: East Coast time. Yes.

Alan Shimel: Yep. We’ve got a global audience here, so that’s East Coast U.S. time.

John, sounds great. You know, if there’s a need in the market, you know, the market has a vacuum, so it’s good to see it getting filled here. I want to wish you the best with it, with the conference and with Kintaba.

John Egan: Great. Thanks, Alan.

Alan Shimel: It was great having you. John Egan, CEO, Co-Founder, Kintaba, the folks behind IRCONF and you can find out more about that at IRCONF.IO. That’s coming April 1^st. Check it out.

We’re going to take a break here on Techstrong TV and we’ll be right back.

September 13, 2022
The Curious Connection Between Cloud Repatriation and SRE Ops

I have a fondness for philosophy. I’m about three classes short of a degree, and every few years I tell myself one day I’ll finish it. Thus, I am very familiar with what is known in statistics—and logic—as a post hoc fallacy, from which we get the saying “correlation is not causation.” This is the logical error of assuming that if event Y followed event X, event Y must have been caused by event X. The most famous callout of this fallacy came from Bobby Henderson, who illustrated the absurdity of assuming causation from correlation with his chart demonstrating that global warming was caused by the declining number of pirates in the world. Yeah, that doesn’t make sense, but neither do a lot of charts from which people draw causality. Just because two data points are mapped against each other does not mean one caused the other. In many cases, it doesn’t even make logical sense to correlate the two. After all, pirates and global warming? No one actually takes that seriously. But it’s an important point to make as we dive into the question of the relationship between site reliability engineering (SRE) operations and cloud repatriation.

To be clear, I’m not suggesting that adopting SRE practices causes cloud repatriation. But I am suggesting that there is a close and meaningful relationship between the two. The fact that Google—a cloud provider—created SRE as a practice is not a mistake. The model, mindset and skillsets associated with SRE are integral to successfully operating cloud infrastructure and services.

Cloud Repatriation Is Real

Public cloud repatriation itself is a somewhat taboo topic in certain circles. Consider the controversy raised by Andreessen Horowitz when it published “The Cost of Cloud, a Trillion Dollar Paradox” and suggested companies were repatriating from the cloud and realizing significant cost savings as a result. Some would have you believe it’s not happening, but there’s enough data and anecdotal evidence to indicate that yes, it is.

F5’s 2021 state of application strategy report asked the market about public cloud repatriation. A mere 13% had repatriated apps and another 14% were planning to. One year later, that combined total rose 40 percentage points to 37% and 30%, respectively. This is not an anomaly, as there are multiple credible analyst firms reporting similar results. Interestingly, the rate of repatriation is not globally universal. APCJ and LATAM are both far less likely to repatriate than EMEA and NA.

I maintain that companies are repatriating apps from the public cloud—and the question isn’t ‘Are they?’ but rather ‘How many workloads are they pulling out—and where are they going?’ That’s a question for the State of Application Strategy 2023 research.

For now, let’s consider a possible enabler of repatriation—SRE operations. Because even if the increasing cost of cloud is a driver of the desire to repatriate if you don’t have the skills to operate as efficiently elsewhere—and thus benefit from lower cost—then why would you repatriate?

And we posit that it is SRE operational practices and skills that enable companies to repatriate and maintain the efficiency and cost savings needed to justify the decision—whether they’re moving those workloads to another public cloud, on-premises or to the edge.

Digging Into the Data

On the surface, there is a strong correlation between the adoption and application of SRE practices with cloud repatriation that seems to indicate that organizations with the ability to operate in a cloud-like manner, i.e., they’ve adopted SRE practices, effectively pick up their toys (apps) and go home (on-premises or elsewhere) because they can.

Cut another way, only 4% of organizations that have not adopted SRE practices have repatriated apps from the public cloud. A whopping 73% of those who have adopted SRE practices have also repatriated apps.

Of course, adopting practices does not necessarily mean applying practices. So, we looked at how organizations are actually operating applications, systems, and infrastructure. Specifically, we looked at the percentage of their operations that use SRE practices. Perhaps unsurprisingly, that generated similar results.

Of those who operate 0% of their apps, systems, and infrastructure using SRE practices, 81% are not repatriating. Conversely, of those who use SRE practices for 76%–99% of apps, systems and infrastructure operations, 54% have repatriated. The point at which repatriation appears to begin picking up steam is when organizations surpass using SRE practices to operate more than one-quarter (25%) of their apps, systems and infrastructure.

Remember I noted that APCJ and LATAM were far less likely to repatriate? They’re also far less likely to be leveraging SRE practices to operate their apps, systems, and infrastructure. In fact, over one-quarter (26%) of respondents in LATAM and APCJ (29%) were operating ZERO percent of apps, systems and infrastructure using SRE practices. In EMEA? That’s only 5%. And in NA, even lower at 2%.

Meaningful Relationship or Curious Coincidence?

There appears to be an inarguable correlation between organizations embracing SRE as an operational practice and public cloud repatriation rates. But is it a meaningful relationship or merely a curious coincidence?

I’m going to argue that it’s a meaningful relationship.

The practices and skillsets associated with SRE are wholly suited to operating a cloudy environment—at scale. As I said before, it’s no mistake that it was Google who created SRE and has literally written the book on it. The value of cloud is in its operational model, which can dramatically lower the cost per transaction, whether measured by HTTP exchanges or customer sessions. That enables a cost-efficient scale of applications and digital services.

The use of automation and practices that tend to focus on meaningful incidents rather than non-disruptive occurrences provides the cost-efficient scale of the people (and thus their expertise) who are tasked with maintaining a high level of availability and performance.

The adoption and use of SRE practices enable organizations to efficiently scale operations whether in the public cloud or on-premises or at the edge. And what the data tells us is that organizations appear to be using the capability to do just that.

September 13, 2022
Study: Real-Time Data Has A Transformative Effect
We all know that data is the heart of the modern digital business. And real-time data is arguably the next evolution—having up-to-date data can empower energy savings and optimizations to reduce cost. Real-time data processing also aids fraud detection and improves health care responsiveness.

Especially in DevOps, access to real-time data is necessary to reduce incident response time and inform ongoing SRE objectives. Leveraging performance data can also empower future decision-making and open new revenue opportunities. In essence, real-time data can have a transformational effect on the operations of a digital business.

DataStax and ClearPath Technologies recently released the 2022 State of the Data Race Report. The study found that the majority of data leaders are using real-time data to create additional value. The report also suggested that working with cutting-edge real-time data technologies increases developer experience, which is good to maintain amid ongoing talent shortages.

Below, I’ll review the key findings from the study to infer how organizations can effectively use these technologies and how they can mitigate roadblocks that arise throughout the process.

The State of the Data Race

Simply put, customers expect real-time experiences. And it’s not only the end consumers—developers are also increasingly accustomed to building in this modality. This is evidenced by the recent rise of asynchronous, event-driven architectures. The survey depicted how fundamental real-time data is to modern software, finding that 78% of respondents agreed that it is a “must-have,” not just “nice-to-have.”

The investment appears to be paying off. A full 71% of all respondents said they can directly tie revenue growth to real-time data. Not only does it seem to have a transformative impact on revenue growth, but it enhances productivity, too—66% of real-time data-focused companies agreed that developer productivity has improved. Naturally, the report found that developers are the ones most likely to be working with real-time data.

Although modern databases and real-time data processing tools are more readily available these days (and open sourced), the maturity of data handling is not homogenous. Based on the classifications of this report, “data leaders” made up only 10% of respondents.

Roadblocks to Value

The road to data maturity has a few obstacles that could limit adoption. First is data complexity—33% said this is one of the biggest barriers to leveraging real-time data. This was closely followed by controlling data costs (32%), data accessibility (30%) and available skillsets (28%).

Most notably, a skills gap continues to be a commonly cited hindrance throughout industry reports and across technology niches. Interestingly, a lack of in-house talent rises slightly higher in organizations with a high real-time data maturity. Those surveyed in this category said that their top barrier is finding the appropriate skills in their business unit.

Cybersecurity is also a top concern—51% say security challenges are a big concern when it comes to the use of real-time data. As the report surmised, the business-critical nature of real-time data raises its worth as a commodity and area for attack. Furthermore, amid new data sharing regulations, IT departments are now under more pressure than ever to uphold data security and avoid the leakage of sensitive personal information.

Unlocking Real-Time Data

This area has a lot of potential; 86% of developers at organizations with a strategic focus on real-time data said that the “technology is more exciting than ever,” the report found. Real-time data is beginning to be applied at the edge, where AI/ML can be trained to give a more responsive advantage to modern applications. Yet, organizations must prioritize its adoption and mitigate the roadblocks mentioned above to unlock the benefits of real-time data.

Prioritizing real-time data will take more than tools—it will take vision. According to the study, those who created more value with real-time data initiatives were found to:
- Have clear product owners
- Have business unit accountability of data
- And have department representatives working together in a cross-functional team.
As you can see, these foundational goalposts have less to do with technology and more with lucid leadership and collaboration. That being said, an investment in the appropriate tooling is undoubtedly required to achieve real-time data.

For example, this tooling might encompass NoSQL databases optimized for real-time or Apache Cassandra for high throughput. The transition to real-time will likely require streaming and messaging technologies as well. Less obvious, however, is an API layer to sit between the applications and the database. This element could greatly help open data accessibility in a standardized fashion.

The State of the Data Race: Final Thoughts

Working with real-time data can be a differentiator and have a transformative impact. As a result, most developers and leaders see real-time data as an exciting area. Yet, organizations must reduce data siloes to open up data access to realize its benefits. It will also take collaboration between data scientists and business divisions to construct meaningful outcomes.

The State of the Data Race report surveyed 500 leaders across various industries. Above, we’ve summarized the key points from the survey. Readers are encouraged to pick up a copy for more nuanced data points and further commentary.
September 6, 2022
NewOps? AIOps? NoOps? Just Don’t Call Me Late for Dinner

Once again, we are seeing an uptick in the use of the term NoOps. I always felt that NoOps was a misleading and, in fact, empty phrase. I don’t care what magic you think you have; you are never going to eliminate the need to operate your software, applications and infrastructure.

I first heard the term NoOps early in the maturation curve of DevOps. Back then, the idea seemed to be that Dev was going to absorb Ops. Between that and the cloud, there really wasn’t a role for traditional Ops to deploy. Managing apps would be easy with the cloud and automation. At least, that was the pie-in-the-sky thinking—but I never bought that.

There were other terms that popped up, like NewOps and AIOps, though that became something else entirely. But really, I think the rise of the systems reliability engineer (SRE) role put the NoOps movement in its grave (where it probably belonged). SRE became the shiny new identity of the Ops crowd and SRE continues to be one of the most popular topics here on devopsy.kinsta.cloud.

This latest incarnation of NoOps seems to be a little different. From what I have seen, it is not really NoOps; instead, it is more of a “not your Ops to worry about” movement. It is shifting who is responsible for Ops from the company developing and using the apps to a service provider (or SaaS provider) who is responsible for Ops. It is more a “MyOps” not “YourOps” movement, but there is still most certainly an Ops there.

In my neighborhood growing up, we used to have a saying, “I don’t care what you call me, just don’t call me late for dinner.” Originally, to me, NoOps was very much like that. We changed the name of something, but it didn’t go away or fundamentally change the thing itself.

Now, we still have Ops, it is just a question of who is doing it. That is not NoOps to me. It is “somebody is doing MyOps for me.”

We saw a similar pattern in security. The idea of outsourcing security was never called “NoSec,” because the idea of no security is, frankly, downright scary. Instead, it just meant relying on a cloud vendor or an MSSP to handle your security for you; it was outsourcing your security. An important lesson learned from that was that you can outsource your security ops, but you can’t escape your security responsibility. That is to say, when your customers’ information is breached, saying “We had someone doing this for us!” did not absolve you of responsibility.

The same thing holds true for this new “OutsourceOps.” The outsourcer may do a better job than you at Ops. They may give you SLAs and SLOs that give you reasonable expectations of performance and remedies if those SLAs and SLOs are not met. But it is still your company’s name and reputation on the line.

So is what we seeing as NoOps really NoOps? No, it is Outsourced Ops. But as we said in the old neighborhood, I don’t care what you call it, just don’t call it late for dinner.

August 25, 2022
Everything is DevOps

It’s funny how we in IT tend to be absolutists, shoving everything into the current bucket. XML was going to eliminate programming—even though it wasn’t even particularly good at data representation. NoCode was going to eliminate programming (yeah, that’s a theme, I’ll skip the other 50 or so ‘going to end programming’ cycles we’ve been through), AI will eliminate IT, you must have a mobile app, even if no one will ever use it, my OS/browser/language is better than yours and will win—you get the idea.

We’re currently coming to the end of the “Everything is DevOps” phase. DevOps is a great methodology that helped solve several intractable IT problems but, like everything, marketing and a ton of smart people that want to be “right” have taken it too far. Or have they?

The thing with DevOps is that contrary to what those who try to make a separate market out of DevSecOps or SRE try to tell you, it was designed to literally encompass all of human IT activity. We can excuse the DevSecOps crowd, since DevOps’ roots had a definite “Security is an unimportant bottleneck” feel to it. The SRE crowd, on the other hand, appears to be another case of “my silo is special”. Or, better yet, a stompy marketer saying “We’re late to the DevOps game! Let’s invent something just like it, claim we do it more broadly and give it our own name!”

Let’s look at what each does, for those of you who are skeptical. 1) break down silos. Check. 2) Monitor everything. Check. 3) Use monitoring results to improve product performance and stability. Check. 4) Design and implement products and product features. Check. There’s more, but you get the idea.

Do you know what all of this ignores? You.

One of the issues that our absolutism brings about is that marketers get to dictate what is important. Sometimes that is what enterprise IT needs. Far too often it is what the marketer is paid to dictate.

I don’t care if you call it DevOps or SRE, I care that your organization feels like it is getting its money’s worth out of IT—something, I might add, that most organizations did not feel prior to DevOps. At least most of the ones I dealt with. I even had a job where they created a whole new department and hired me to run it to “get around those people in IT.” I spent as much time educating business leaders about not duplicating entire groups like storage services and InfoSec as I did building and deploying new products. So avoiding that kind of issue is near and dear to my heart. And we’re more ‘there’ than we have ever been.

I do care about whether you call it DevSecOps or not. It should not be different. Shifting security left is the appropriate solution to the double-edged sword of security staffing shortages and increased adversary capabilities. So if your developers are getting secure coding feedback in their IDE or each time a build is kicked off, great! But call it DevOps. Say it is part of the DevOps process. Same with configuration reviews like cloud configuration management—it’s just part of the DevOps process, not a security step to be thought of differently.

I could go on, but hopefully I’ve spurred your imagination a bit and don’t need to.

You rock. Streamline your thinking to streamline your processes, ignore absolutism and call it macaroni for all it matters. Just keep doing what needs doing—at the most convenient place to do it. And keep developing, deploying and monitoring the systems that keep the company running.

August 24, 2022
The Rogers Outage of 2022: Takeaways for SREs

When, eight years from now, folks are creating lists of the top IT incidents of the 2020s, there’s a good chance that they’ll include the Rogers outage of 2022. The failure, which made internet and cellular network service unavailable for more than 12 million users across Canada, was one of the most significant outages in memory, in terms of both the number of affected accounts and the level of service disruption.

For site reliability engineers, there’s a lot to learn from this incident, from both a technical and a crisis-management perspective. Here’s a look at our main takeaways.

The Rogers Outage: What Happened?

Before diving into lessons for SREs, let’s go over what happened during the Rogers outage and what we know about the cause of the incident.

Rogers is a major telecommunications provider that serves around 12 million customers in Canada. Early on July 8, 2022, it suffered an incident with its infrastructure that made internet and wireless phone service unavailable for most or all of its customers. The outage was so complete that users couldn’t even make 911 calls or withdraw money from ATMs–which means the failure brought down not just ordinary services, but also what you might consider resources that are essential for normal daily life.

Connectivity was restored for most users within about 15 hours. However, some customers reported continuing disruptions for several days after the beginning of the outage.

What Caused the Rogers Failure?

At first, Rogers did not reveal much technical detail about what triggered the incident. A statement attributed to the company’s CEO the day after the outage said that a “maintenance update in our core network … caused some of our routers to malfunction.” But that didn’t explain specifically which type of maintenance update the company was trying to apply, why the update didn’t go as planned or which types of routers were affected.

It wasn’t until several weeks later, when Rogers submitted a response to questions from the Canadian Radio-Television and Telecommunications Commission, that more technical detail came to light. Rogers explained to the CRTC that, during an update process, the company ran code that deleted a routing filter. It’s still unclear exactly how the bad code made its way into the update routine.

Lessons for SREs

For SREs, the Rogers outage is an example of a worst-case scenario. It was an extremely disruptive incident that took a relatively long time to mitigate, at least when you consider how severe the outage was. That’s why SREs should use Rogers’ experience to assess what SRE teams can do to avoid issues like this–and manage them more effectively when they do happen.

Test Your Updates

From a technical perspective, one of the biggest takeaways from the Rogers outage for SREs is that you should test updates thoroughly before applying them. If engineers had tested their update routine more carefully, they would presumably have detected the bad code that deleted the routing filter before the deletion actually took place.

To be fair, Rogers is hardly the first company to experience a major outage due to a bad update. Atlassian, for instance, had a similar issue earlier this year, when a buggy migration process caused a long-lasting outage.

If you’re an SRE, you don’t want your company to be the next to make headlines when something that should be a simple change, like an update, triggers a massive outage. Test, test and test again before bringing your changes into production.

Build in Redundancy

It also seems reasonable to conclude that Rogers didn’t have redundancy built into its infrastructure in the form of backup routers that could take over when the primary routers crashed due to the bad update.

This is another reminder to SREs about why it’s so critical to avoid putting all of your eggs in one basket or creating single points of failure. Redundancy may cost more, and it may not totally prevent disruptions. But it can make them less severe and easier to recover from. It’s not hard to imagine the Rogers incident being a lot less severe if even just 20% or 30% of its traffic could have been automatically redirected to backup routers. That might have been enough to keep critical services operational, at least.

Have an External Crisis Communications Plan

Rogers has received more than a little criticism for not communicating well with its customers during the outage. The company waited hours before releasing an initial statement about the failure. It was not very responsive to media queries early on, and, as noted above, it took several weeks before Rogers finally released technical details about what went wrong (and even then, the details emerged only because of a government inquiry into the incident).

The lack of rapid, transparent communication probably wasn’t deliberate. We have to believe that Rogers didn’t want to leave customers in the dark. Instead, the slow communication likely reflects the absence of a pre-planned strategy for external incident communication in a situation where Rogers’ own networks were non-operational. Without a communications plan, Rogers presumably struggled to decide which information to share, or how to share it, in the midst of a major crisis.

If Rogers had a playbook in place for communicating with the public and media about this type of issue before its network went down, it probably would have been able to share information more effectively–and it may not have faced as much of a backlash from customers and regulators.

Internal communications, at least, presumably were smoother. The company did manage to get service back up for most customers within about fifteen hours, which probably would not have happened if its engineers lacked a means of communicating with each other as they worked to resolve the incident.

Conclusion

For SREs, the Rogers outage of 2022 is a lesson in the importance of testing, designing redundant infrastructure and preparing for transparent communications before crisis strikes. While more careful planning in these respects may not have totally prevented the incident, it would likely have at least reduced its severity and mitigated the backlash that Rogers now faces from angry customers and regulators.

August 15, 2022
5 Ways to Prevent an Outage

In today’s always-on, ever-connected world, we all expect 100% availability.

What gets in the way of this? The devil is in the details. Over time, everything breaks: Disks, nodes, containers, networks, DNS servers and configuration issues can all lead to major outages.

Amazon’s 13 minutes of downtime in August 2021 translated to almost $5 million in lost revenue and Amazon Web Services’ three outages in December of last year cost millions more. And it was widely reported that the October 2021 Facebook (including related products: Messenger, Instagram and WhatsApp) outage resulted in an estimated $100 million in lost revenue due to a network configuration error.

There have been many outages of varying scope and duration—some attributed to technology breaking, others traced back to people making mistakes. In fact, a 2017 Amazon outage was attributed to “human error”–or more specifically, one employee’s typo. The employee intended to take a small number of servers offline to fix a billing system issue but the incorrect command led to several servers being affected and a major outage.

So, what can your company do to reduce the risk, duration and impact of a potential outage? For one, companies can’t pretend that if they only try harder, they can prevent humans from making mistakes. Do you have dozens of people manually keying in hundreds of commands every day? If so, a mistake is inevitable. Companies should instead investigate how and why one small blunder in a command line could do widespread damage. Guardrails and redundancies should be in place to protect against and minimize these types of incidents.

As the enterprise continues its digital transformation toward a multi-cloud, hybrid and distributed world, here are five guidelines to consider—guidelines that can help you preempt and prevent your next big outage.

1. Always Consider Your Blast Radius

Blast radius measures how one error, typo or malfunction can impact other parts of a system.

What is the chain of components or services that is impacted by one error?

What are the hidden dependencies?

How many customers would be affected and for how long?

More is connected than you think; discover how it’s all related in advance to eliminate surprises.

If too many systems in a network are mutually dependent and interconnected, the result is deadlock. In other words, when one breaks, the other also fails. We have seen this in past Facebook outages. A single erroneous command sparked a domino effect that shut down the backbone connecting all of Facebook’s data centers globally.

How can you prevent this scenario? Plan for the worst possible scenario. When it comes to mission-critical systems, take an intentional approach to situation failover and backup. What if A, B and C all happen simultaneously? Also, think about your cost versus reliability trade-offs. Maybe you can’t afford two full sets of servers but can afford a full set of archived code.

2. Smooth Out Rough Edges

Consider that core infrastructure and code get used (and tested) a lot more than edge code/infrastructure. Basically, the core is hardened while edge infrastructure is more vulnerable. You should be more proactive at the edge around identifying vulnerabilities and doing explicit testing to counter this risk.

The 2014 Google disk erase/satellite outage is a cautionary tale involving the edge. In Google’s case, the edge networks offered a key advantage: A better connection for the customer due to lower latency since data is accessed from a source closer to the user as opposed to the core network. Google had a CDN/edge network for TCP connection termination, caching for various apps, DNS regulation, etc. Teams needed to routinely upgrade a satellite rack and run decommission automation—and this automation failed part of the way through. Error message. They retried the decommission process again to debug the failure. The manual re-execution of the decommission automation meant thousands of satellite machines were decommissioned and, within minutes, the disks of all satellite machines globally were erased. The result: Customers experienced an increase in latency. Behind the scenes, it required two days of all-hands-on-deck to reinstall these machines.

3. Think Escalator, Not Elevator

A good system has both defense and depth. Considering that it takes three or four things to (really) go wrong, think about failure cascades. Bluntly put, limit dependencies. Measure the longest dependency failure path and mean-time-to-recovery (MTTR). When you do this, think escalator not elevator. What do I mean? Well, when an elevator fails, you are left with a shaft; an escalator that fails still works as a staircase.

In 2016, severe weather in Sydney, Australia resulted in an AWS EC2/EBS outage. First, the utility provider suffered a loss of power at a regional substation. This failure resulted in a total loss of utility power to multiple AWS facilities. In one of the facilities, AWS power redundancy didn’t work as designed and power was lost to a significant number of instances in that availability zone. To make matters worse, the instances that lost power lost access to both their primary and secondary power as several power delivery lineups failed to transfer load to their generators. Also, a latent bug in AWS instance management software resulted in a slower-than-expected recovery of the remaining instances.

The lesson? This correlated power delivery lineup failure illustrated that failures are not independent. You must design for the largest correlated failure in combination with the background noise of ongoing independent failure. Recovery code needs to be scale-tested against the largest expected failure.

4. Stay Ahead of Digital Certificate Rotations

Don’t underestimate the importance of digital certificates—and their expiration dates. A few years ago, LinkedIn allowed one of its SSL certificates to expire and it knocked out LinkedIn sites in the U.S., UK and Canada. In 2017, Equifax missed a breach for 76 days because of an expired certificate. The average annual cost of unplanned outages due to certificate expirations is $11.1 million.

For example, you could have a load balancer sitting in front of several servers. The load balancer could have an up-to-date certificate while the servers behind it are expiring. It is important to check for expiring servers in multiple ways. Ideally, you check the boxes in two ways: Ping the HTTP endpoints and also have scripts that check the certificate files themselves.

Expired certificates (TLS for example) can lead to encryption and authentication risks as well as lost internet connectivity and larger outages involving websites and applications. To avoid this, you continuously check, renew and deploy certificates—and this can be achieved via automation. Think about automatically rotating expiring certificates.

5. Tackle Toil to Empower Your Teams

What’s the biggest headache for site reliability engineers (SREs) and those who manage production operations? Toil.

We can define toil as manual, tedious tasks that engineers perform within production environments. Basically, toil wastes time and slows down operations. As a result, most SREs and engineering teams strive to minimize toil within their workflows.

In production operations, you should work to limit toil. Make sure your team is regularly investing time in proactive work; the reactive work shouldn’t take most of their time. Toil reduction gives developers, DevOps engineers and SREs more time to focus on projects that create net-new value for the business (new application features). Google stated that it has an “advertised goal” of keeping toil “below 50% of each SRE’s time.” Basically, no engineer should spend more than half of their time on manual, repetitive work. But the ultimate goal should be to get toil as close to zero as possible. Automating incident remediation can help SREs spend less time on toil and less time on-call.

Also, when you implement fixes—especially people and process fixes—ask yourself if this is a systemic fix or one that is relying on best intentions. Look for tools to safely empower your team (with guardrails). As you work to identify major vulnerabilities and dependencies (as discussed above), you should automate away the most common issues while planning for the “corner cases” (or the 1% of 1%) that may lead to major outages (a.k.a. the worst-case scenarios).

As you consider these guidelines, make sure you focus on putting actual mechanisms in place—not one-off, “hopeful” directives. Do not rely on people to remember best practices from a training session—these are not mechanisms. Instead, design processes that can be systematically implemented. When you do a post-mortem, look for concrete programs you can put into place that will prevent the same mistake from happening again. Automation can help you achieve this level of confidence, management and quality control—and it may just keep you one step ahead of a potential future outage.

August 15, 2022
Simplifying DevOps and Infrastructure-as-Code (IaC)

Let’s face it, tools for infrastructure-as-code (IaC) configuration management may be vital, but they’re difficult to use. But what if it’s not so much the tools themselves as it is our understanding of them? What if we looked at provisioning and configuration systems differently and did so using vocabulary that was less confusing and more common?

Simply put, we’d gain a greater understanding of these tools—and, therefore, they would be simpler to use—if the terminology was simpler. Reframing these complex tools would not only allow us to make greater sense of this realm, but we’d be more effective in addressing the DevOps and software issues we all experience.

Coming to Terms With Infrastructure-as-Code (IaC)

Software engineering encompasses all the actions developers take to create software and technology products. This typically entails writing code, testing it and making sure it satisfies the original requirements for users. Operations is the practice of ensuring those software systems are ready to go and will be able to serve users on a day-to-day basis. Operations engineers aren’t as focused on writing new code as developers are; instead, they are focused on how existing software (mis-)behaves in existing systems.

Now let’s cover some other key phrases we use and hear all the time in DevOps.

Forward engineering means creating a system based on requirements that exist for that system. It entails first building a model, implementing it in software, then configuring and deploying it into operation. While the goal is for the new system to closely match the original requirements, the modeling process can be ambiguous, causing a system to perform in unanticipated ways. Furthermore, over time, the system itself may differ from the original model due to interaction with its environment. Consider that some parts can change the configuration of other parts; procedures performed by engineers shift the system state or configuration, too.

Unfortunately, forward engineering offers no feedback with regard to the eventual success or failure of the product during the model creation process. These models and the subsequent software are created first and then deployed to test for success. Such a system can only provide feedback later based on its behavior, which can then be used to improve and build new models.

Reverse engineering calls for understanding existing systems, usually by first creating a partial mental model by inspecting such a system and then using it to plan any modifications. Often, changes are made directly to the system without always fully understanding how it works. Feedback for changes is fast because putting the system through its paces and inspecting the results is relatively simple, easy and cheap to do. A good example of reverse engineering is software security professionals who scan network services or memory to identify anomalies and exploitable gaps and then try to manipulate their content for instant feedback.

Then there’s infrastructure-as-code (IaC). This modeling technique began with declarations about how server operating system content should appear and was later adapted for the purpose of laying out and configuring cloud resources using API calls. While many vendors and products have come and gone, most tools have been declarative—with good reason.

Anything to Declare?

With declarative tools, you specify how things should appear without explicitly showing how you’ll get there. Why? It’s the easiest way to create models, templates and maps because engineers only have to declare how things should look, not get them into the correct form or place to actually do so. The emphasis is on evolving from iterative improvements that take place over time.

Another feature of declarative IaC tools is idempotence, which means repeating an action multiple times with consistent results; the system only changes when it detects deviations from the model. This can be compared with the imperative approach, which requires developers to first find all the differences between a model and system and then write each step that’s necessary for the system to match the model.

Declarative IaC tools eliminate the need for developers to write an implementation for these changes manually. These tools save thousands of labor hours by publishing templates that can be used repeatedly to build identical systems with less time and effort. The problem is that declarative tools rely on a forward engineering mindset—meaning there’s no way to get feedback from a live system—leading to many complaints from users.

The Rift With Drift

Configuration drift is what happens when a declarative forward engineering model no longer matches the state of a system. It can happen when a developer changes a model’s code without updating the systems built using that model. It could also be the result of an engineer that does exploratory ad-hoc operations and changes a system but fails to go back into the template and update its code.

Is it realistic to ban operators from ad-hoc exploration? Actually, some companies have policies forbidding any operator or developer to touch a live production environment. Ironically, when a production system breaks, it’s the first rule overridden: Ad-hoc exploration is welcomed by anyone able to get to the bottom of the issue.

Engineers who adopt IaC usually don’t like the work that comes with remodeling an existing system. Still, because of high demand fueled by user frustration, there’s a tool for every configuration management language—they just fail to live up to engineer expectations. The best-case scenario is that an engineer can use a reverse-engineered template to copy and paste segments into one but will need to manually write it elsewhere.

Bridging Gaps

What is needed is a new type of IaC tool. These must integrate reverse engineering at their core and allow feedback from living systems to update the very models that created them. In doing so, operators can adopt an ad-hoc change or simply revert back to the system as it was declared in the original model.

Yet, today’s standard tools fall short of this goal, and aren’t much good for helping engineers solve system operations issues. True, there are those capable of auditing templates and singling out misconfigurations and mistakes. Still, while helpful, the greater value would be the ability to review a living system, not just the templates that engineers initially used to create it.

We have monitoring and observability tools and IaC tools, but in between them is a wide chasm. For effective DevOps, what’s required is a new breed of IaC tool, one that integrates reverse engineering practices that supports operations engineers and enables companies to bridge any development gaps.

July 25, 2022
Why More Incidents Are Better
Ask most SREs how many incidents they’d have to respond to in a perfect world, and their answer would probably be ‘zero.’ After all, making software and infrastructure so reliable that incidents never occur is the dream that SREs are theoretically chasing.

Reducing the number of actual incidents as much as possible is a noble goal. However, it’s important to recognize that incidents aren’t an SRE’s number-one enemy. What matters more than the number of incidents you experience is how effectively you respond to each one.

Plus, there’s value in incidents. They are a learning opportunity. If your business never experienced them, it would arguably be facing more risk, not less.

We know: These ideas may sound a little counterintuitive. You might even accuse us of being “pro-incident”—which we sort of are. Allow us to explain.

The Silver Lining

In many respects, incidents are inherently bad. When one occurs, it means something broke. That’s bad. It may also mean that users were disrupted, operations halted or money was lost. Those things are even worse.

On the other hand, incidents aren’t all bad. They actually benefit SRE teams for several reasons:
- Learning opportunities: Incidents are opportunities to figure out what went wrong and prevent it from recurring. They can also help teams learn how to react more quickly or efficiently the next time something fails.
- Get ahead of bigger issues: Sometimes, working through one incident means you can avoid another that’s even worse. Perhaps one server fails, for example, and your response revealed that the failure was due to a larger issue that would have eventually caused a worse outage if left unaddressed. But thanks to the incident, you detected the larger issue before it triggered a more massive failure.
- Reinforce team culture: Nothing breeds camaraderie or a spirit of collaboration like working alongside other engineers to respond to a crisis in the middle of the night. Although being in this setting may not be anyone’s first choice, it does often have a positive impact on your team’s culture and esprit de corps.
- Demonstrating value: Assuming you handle them well, incidents are an opportunity for SREs to prove how valuable they are to the organization. If incidents never happened, it’s a safe bet that some bosses would start to wonder why they need SREs in the first place. (It would be a flawed train of thought, of course, because SREs would deserve credit for preventing incidents, but it’s a thought that may float around some C-level brains nonetheless.)
We could go on, but the point is clear: Although incidents cause problems in some respects, they actually create value in others.

Focus on Response, Not Avoidance

This is not to say that you should welcome incidents with open arms. Obviously, any decent SRE should focus first and foremost on being proactive and preventing incidents from happening whenever possible. They should use chaos engineering to identify problems that could be lurking unseen in production environments. They should leverage IaC to minimize risks. And so on.

That said, what ultimately matters more than incident frequency is the effectiveness of incident response. It’s better to experience ten incidents that you resolve in under an hour each than one incident that takes mission-critical systems offline for a week.

So, in addition to investing in tools and processes that mitigate the risk of incidents, SRE teams should place equal emphasis on ensuring that they can react quickly and effectively when an incident happens. This means having the ability to share information efficiently, define clear roles, know what to prioritize when working through complex incidents and have clear plans in place that spell out how you’ll handle a problem as soon as you detect it. Without these abilities, you’re at risk of letting incidents that should be small turn into major outages.

‘Zero Incidents’ is not Realistic

It’s important to recognize, too, that while it can be fun to imagine a world where zero incidents occur, the reality is that such a world will never exist. If it could, we wouldn’t see new records set each year for the number of security incidents that businesses collectively suffer, for example.

Nor would we see headlines about major outages at huge enterprises like Facebook or AWS on a recurring basis. If those companies, which have world-class reliability teams and virtually endless resources at their disposal, can’t reduce incidents to zero, neither can anyone else.

Conclusion

The bottom line: There is no such thing as total incident prevention, no matter how hard you try. And even if there were, that wouldn’t actually be a good thing, for the reasons explained above.

So, by all means, undertake reasonable proactive efforts to prevent as many incidents as you can from happening. But don’t let investment in prevention cause under-investment in response. Being prepared to handle incidents when they happen—which they inevitably will—is what matters most.
July 15, 2022