devopsy.kinsta.cloud

Tag: resiliency

  • Get Comfortable with Shifting Right to Improve Resiliency

    Get Comfortable with Shifting Right to Improve Resiliency

    The operations team receives a page at 2:00 a.m. Something isn’t working correctly. Do they know how to fix it? Perhaps more importantly, do they know whether it matters to customers or how it impacts the business? There are different techniques that companies implement to maximize the quality of their software. We talk about shifting left or right, chaos engineering and others. While they each have their own goals to improve quality, there’s an underlying common goal as well: All three techniques increase your team’s knowledge of the system and their ability to resolve problems quickly.

    Here, we’ll walk through the software life cycle and discuss the common challenges to software quality, running complex systems and managing dependencies between teams. The common thread is a proper observability strategy which can help software quality, hand-offs between different teams and resolve problems quickly by ensuring whole-system resiliency from design to maintenance.

    Techniques for Improving Quality

    Shift left is a concept that has attracted much attention recently. The practice revolves around the idea that detecting and correcting problems sooner minimizes their impact. As the thinking goes, they’re easier to fix, they wind up costing less and there are fewer ongoing issues with bad code.

    The rise of DevOps and DevSecOps has propelled the popularity of shift left. Yet, there’s a fact that development teams often miss: no matter how early an organization spots errors, problems will inevitably remain. Early testing can’t catch everything. As a result, some organizations have begun to rebel and move to a shift-right framework. It normalizes errors and aims to better resolve issues across the development process, including post-production.

    Shift right is the perhaps less-extreme cousin to its rebellious relative, chaos engineering. With chaos engineering, team members cause problems on purpose to keep everyone on their toes and test the limits of the system. They shut down servers, disconnect wires, delete files, etc. The goal is to confirm system resilience–from a user perspective, there should be no impact even with all of this chaos. A resilient system means that teams can fix failed components at their leisure during the day. With a resilient system, critical failures are less likely to happen and it becomes second nature to fix them when the dreaded 2:00 a.m. page does occur. Unlike chaos engineering, shift right doesn’t explicitly break things–it just makes the assumption that broken things will exist so teams need to be ready to handle them.

    Both concepts (and yes, even chaos engineering) are valid—but each company needs to carefully analyze which is right for them. One variable to consider is the complexity of the system you’re building. Working on a rocket launch or a surgery robot? Probably should shift left as much as possible. Changing the color of a button? Maybe it’s not the end of the world to verify the fix in production. For most of us, projects fall somewhere between these two extremes–so sometimes shifting left is necessary, and other times shifting right is pragmatic. A careful analysis of the risks should be performed to determine what kind of testing is needed.

    Although teams tend to be pretty good at analyzing the complexity of their software and making development process decisions, they often forget the complexity of other companies’ code which they depend on. Even a simple web application deployed today has tons of dependencies: The server it’s running on, the local ISPs, transit ISPs, load balancer, DNS provider, TLS certificate, CDN and browser, just to name a few! After analyzing the complexities of all of the dependencies across the internet, it becomes clear that finding every possible issue that might impact their application is a daunting task.

    The goal, again, is to find any issues before they reach production. Or if they reach production, fix them before they impact customers. Or if they impact customers, fix them as soon as possible after that. To do any of these things quickly, the level of observability of the overall system is paramount. With simple systems—or even complex systems that are entirely under one company’s control—tools such as application performance monitoring (APM) are common. But when the scope of the system is as big as the internet, internet performance monitoring (IPM) tools are necessary instead.

    With both APM and IPM tools, the goal is to know as much about the system as possible – enabling teams to identify problems quickly, pinpoint the root cause, and fix it. When an organization is equipped to monitor development and operational processes effectively from application to delivery, there’s a shift toward higher quality code, fewer errors, improved security and lower costs for the enterprise. There are also fewer disruptions and distractions for engineers, DevOps teams and others.

    Dealing With Disconnects

    Whether shift left or shift right is used, there will be problems that enter production. There is an invisible wall in many organizations–where engineering teams “throw code over the wall” to the team which is now responsible for running it. In some companies, this is the operations team; in others, it’s site reliability engineering or production engineering–the common trend is that in most companies, it’s not the same team who created the software.

    These operations teams often lack the nuanced information they need to resolve production issues. They might spend hours or even days attempting to track down the source of the problem. There’s often finger-pointing between different teams who created or own different portions of the system. Things can quickly devolve into chaos—and tempers can fray. After all, these kinds of problems always happen at 2:00 a.m., right?

    That’s where end-to-end observability enters the picture. Imagine a framework with proof that the development team has performed sufficient knowledge transfer to the operations team. That the operations team has asked all of the relevant questions about running and maintaining the application. Using the right tools, engineering and operations teams need to monitor every aspect and every component of the system. Creating this monitoring strategy means that everyone involved also builds an understanding of the system–everyone is on the same page. All of a sudden, there is a common language to discuss problems, and teams operating the software in production can take debugging a lot further before they need to ask for help.

    Ultimately, shifting left or shifting right is a personal choice for every organization. Even if every possible test case is executed before deployment to production though, issues will happen–so take the opportunity to implement a robust system-level monitoring strategy that can be used to solve technical questions during that 2:00 a.m. call.

  • Lightstep Extending Beyond Observability with Lightstep Incident Response – Techstrong TV

    Lightstep Extending Beyond Observability with Lightstep Incident Response – Techstrong TV

    Ben Sigelman and RJ Jainendra discuss ServiceNow’s latest announcement that Lightstep is extending beyond observability and creating a differentiated portfolio for app development with the general availability of Lightstep Incident Response, helping make organizations’ digital products and services more reliable and resilient. The video is below followed by a transcript of the conversation.

    Alan Shimel: Hey, everyone. Welcome to Techstrong TV’s segment today. We’ve got some breaking news I wanna bring to you, and I’m really happy to be joined by two gentlemen who are gonna tell you the news. I’ve had the pleasure of having them on before.

    They’re two really bright guys from ServiceNow. Let me introduce you to Ben Sigelman and RJ Jainendra from ServiceNow. RJ, I think I left out an “N” somewhere in that second or third syllable, but we’ll let you correct it. I’m gonna ask you to go first, RJ. Introduce yourself to the audience.

    RJ Jainendra: Hey, Alan, good to see you again. And no, you got it right, RJ Jainendra. I’m a VP and GM at ServiceNow, unlocking product growth at the company. It’s great to be here.

    Alan Shimel: Excellent. It’s good to see you again. And of course, RJ, you’re general manager, but you’re the portfolio you are GM with includes observability. I believe it includes value stream as well, yeah?

    RJ Jainendra: So, ServiceNow does have offerings for value stream management, which brings together a number of different product lines. Our IT business management, which we recently rebranded as Service Portfolio Management, SPM, that’s one. The second is it also brings in elements from devops and ITSM offerings to provide a value stream management.

    But I’m here today to talk about Lightstep Incident Response. That’s the product line that we’ve been building out for a little while now, and ready to launch here today.

    Alan Shimel: Absolutely, so, we’re gonna talk about Lightstep. Gotta have Ben here. Ben, welcome. Why don’t you give people a little bit of – give the folks a little bit of your background?

    Ben Sigelman: Yeah, thanks, Alan, for having us. So, I’m Ben Sigelman. I was one of the cofounders at Lightstep and was the CEO up through last year, when we were acquired. ServiceNow wasn’t willing to let me remain the CEO of ServiceNow after the acquisition, so now I’m also a VP and GM, along with RJ. 

    Lightstep’s brand and certainly our products and many aspects of our culture have actually been intentionally retained as part of the acquisition, and the brand has actually been expanded to be a multi-product brand, which is what we’re talking about today.

    Most of my day-to-day is spent doing a combination of thinking about the product vision for observability specifically within Lightstep as well as the longer-term strategy within ServiceNow, and what we can do for our customers by unifying the world of ServiceNow with the world of high-volume customer-facing applications that Lightstep tends to play in.

    Alan Shimel: Absolutely. I think the ServiceNow team already has a pretty good guy in its CEO, but hey.

    Ben Sigelman: I actually, when I talked to Bill during the acquisition, I told him he had my condolences for being a CEO of a much larger company. I knew how stressful it was for Lightstep, so I can only imagine what it’s like for him. But he’s a fantastic guy.

    Alan Shimel: Absolutely. Yes, until you’ve been a CEO, and I don’t care whether it’s a really, really big company or a really small company, yeah, it’s a lot. Anyway, guys, let’s talk Lightstep, observability. Let’s talk about the expansion. Just to level-set things – you both referenced it – Lightstep was one of the leaders in this observability revolution or wave that we saw overtake the industry, let’s say, in the last three, four years, certainly, it really caught fire.

    Lightstep, Ben and your two cofounders, I believe, all came from Google, and really in many ways helped initiate the whole observability revolution there. Then leveraging open source and working with open source, open telemetry, et cetera, really built quite a community and following.

    ServiceNow, one of the most dynamic companies in our industry, recognized this and made Lightstep and observability an important part of the ServiceNow offering. But from what I’m hearing and reading between the lines is why stop there, right? There’s more.

    There’s more that the marriage of ServiceNow’s know-how market intelligence and skill set, and the Lightstep platform can do more, and today marks a significant expansion of that. So, you’re both GMs. Who wants to go first to talk about what this new expansion is?

    RJ Jainendra: So, I can talk a little bit about what the expansion is, and Ben can, I think talk through why it makes a ton of sense, what we’re doing. The thing that we see that’s happening in the market is just the amount of investment going into what’s been called digital transformation.

    It’s just increasing significantly, and I think by some accounts, IDC forecast, it’s gonna be something like north of $10 trillion cumulative by 2025. So what Marc Andreessen said a long time ago, I think it has come to pass, which is every company is a software company, and increasingly, with the adoption of cloud, organizations are prioritizing cloud operations and software development teams they build, and they run these millions of new customer-facing, revenue-generating applications and services.

    So from that perspective, the SREs and developers who really are responsible for keeping applications reliable and resilient, they really start impacting the business outcome. That’s where if you have incidents that occur, maybe it’s a bug or some sort of an outage or poor performance, it has a major impact on revenue, on customer loyalty, on the brand perception.

    That’s why you see the option in the market to introduce a new offering around incident response, which really leverages the power of the ServiceNow platform, but then it brings together the observability capabilities that Lightstep has into sort of a holistic offering that we can bring into market to help customers become more proactive and more effective in addressing issues. Ben, do you wanna add to that in terms of your perspective from an observability side?

    Ben Sigelman: Yeah, for sure. I think the way I’d explain it is observability, Alan, I would agree with you that it’s the wave, but it’s a very confusing, diffuse – it’s a body of water right now that can be hard to wrap your head around, and you frankly have a lot of what I would call legacy observability companies trying to take what they have and call it observability, because that’s easier than reforming it.

    So, it might be helpful to spend just one very quick, 30 second snippet talking about what observability actually is, architecturally. You have some layer at the bottom of integrations and data acquisition. You mentioned Open Telemetry earlier. The Lightstep Incident Response product that we’re announcing today also has an integration ecosystem built around it.

    But you have the acquisition of data, you have some kind of data platform where you store that, which I’m not gonna talk about in detail right now but I think is an interesting topic, slightly far afield. Then you have workflow been made with your full knowledge and workflow built on top of that, and that’s where the actual value is delivered.

    A lot of the conversation around observability today is that you have a silo for your metric system, a silo for your logging system. Maybe you tack on some kind of tracing or PM system, and then you integrate them in the browser.

    Then you also have this Incident Response thing over there. That’s some other vendor, and it does not make any sense. When we talked with our customers, the three things they are prioritizing are overall performance, velocity, and actually, most importantly, incident response and timed resolution.

    It’s the most important thing for observability. Then you have not just a separate tab, but a separate product to do incident management and incident response, which is not sensible from a workflow standpoint.

    So, for the brand, this is a very obvious and natural thing, to have incident response and observability integrated into a single experience, especially incident response where seconds matter, and a context switch like that is very expensive. So, anyway, that’s why this makes a ton of sense to us.

    It goes right at the heart of the MPTR problem, to have a single platform to manage the full lifecycle of incident management on the observability side and on the incident response side.

    Alan Shimel: Absolutely, and you know, Ben, to your point – well, first of all, I think you call them “legacy observability vendors.” Really, they’re APM, application performance management vendors who – 

    Ben Sigelman: Or infrastructure monitoring vendors, more to the point.

    Alan Shimel: Right, or infrastructure – 

    Ben Sigelman: It’s not the –

    Alan Shimel: You know, everyone loves a good name.

    Ben Sigelman: Right.

    Alan Shimel: A company I had started, we had come out with a product that Gartner would later name the category NAC, network access control. We had some stupid name for it before, but Gartner, you know, did that.

    All of a sudden, out of the woodwork, all these people who were doing totally different things started calling what they do NAC. A very similar thing happened with observability, right?

    Ben Sigelman: Yeah.

    Alan Shimel: Anyone who’s doing monitoring, all of a sudden it was observability. But I think, also to your point, was that observability got siloed into just observing. Almost by definition, observing, observability, is not actionable. It’s almost like observability is a passive word, where actionable is active word.

    Ben Sigelman: You’re exactly – that’s a very good point. If you go – I tend to be a little academic, but I think this is an interesting point. Observability really originates from control theory in the 1960s, and it was this smaller, kind of lesser cousin to controllability, which is kind of what you’re talking about.

    Controllability is really the thing that matters. If you can control a system, there are scientific ways of talking about that. Mathematical ways talking about that. To a certain extent, you can think of incident response as being the controllability side of what we’re talking about here.

    It’s like we’re making insights from observability actionable with the Incidence Response product, and that’s a really important part of actually closing the loop on the control theory of modern distributed systems. That’s a great insight that you’re making, Alan.

    Alan Shimel: Yep, thank you. Every once in a while, even a blind squirrel finds something, right? So, RJ, let’s talk about this. ServiceNow is not a trivial size company. To decide that – you didn’t really have an incident response solution in market prior to this, as far as I know. I’m not an expert on everything ServiceNow, but I’m not aware of it.

    To say okay, we are going to build this in as the logical next step in the Lightstep platform, just the inertia sometimes in an organization your size, it’s difficult. Let’s hear from your point of view what it took to make that happen, turn that ship.

    RJ Jainendra: Yeah, so, Alan, we actually do have capabilities in the ServiceNow platform through our IT Service Management and IT Operation Management products, around things like event technology around event correlation, being able to take large-scale number of events from the observability tools and bring them in and understand what those events mean.

    We also do have functionality in ITSM around notifying the right person. But really, the ServiceNow technology, we have great adoption with IT organizations and a large number of customers. But what we’re seeing really is as our customers are going through the digital transformation, they’re reorganizing, and the product teams that are out there are the ones that are sent to adopt these technologies on observability. 

    That’s why we felt that we wanted to bring an offering into market that could be adopted by teams. So, the entire experience for Lightstep Incident Response is as a modern SaaS application. If you’re a developer on the team – and what we’re finding right now is a lot of our initial customers are smaller companies.

    They’re early growth-stage companies that are post-revenue. They have customers, they wanna make sure that their systems are highly available for them. So, what we’re finding is that they can come online, they can sign up, and within a few seconds get access to the system, start using it, and if they see value, then they can swipe the credit card and they’re off to the races.

    So, very much a different motion, really suited to how teams out there in smaller companies are adopting technology. And of course, the startups of today are the ones that become the unique ones tomorrow, so that’s what we’re doing that’s a little bit different. But for ServiceNow customers, they need to have those capabilities within the ServiceNow platform.

    Alan Shimel: Excellent. So, guys, let’s talk – if you don’t mind, I gotta do a little meat-and-potatoes here. You guys announced this today. Is it available? When will it be available?

    RJ Jainendra: It is available today on Lightstep.com. So, we have had the product in the market for a little bit. We did a soft launch earlier last year. So, it’s been battle-tested, and we have a number of customers that have come online. They’re using it in production. So, it is available today, and it doesn’t take more than an email to sign up and get going.

    Alan Shimel: What’s the offering? How is it priced? If you’re an existing Lightstep or ServiceNow customer, how do you – firing off an email, yeah, but how do I – what’s this gonna set – you know, what’s involved here?

    RJ Jainendra: Yep, so, actually, it isn’t a whole lot more complicated than what I just said, which is just as you sign up for a lot of consumer-type services online, developers and SREs can get started with an email. We do have a couple of different tiers. So, we do have a free tier, where if you are just getting started and you may have just a couple of services that you’re monitoring and addressing issues for with Incident Management, that’s free.

    Now, if you’re gonna start scaling up and you have more than two, that’s when the pricing kicks in. We are taking an innovative approach in terms of pricing, which is it’s based on usage. So, we look at the number of active services that a team has, and then we charge by the number of active services, which is very different than some of the other solutions in the market, which charge by the number of developers that you have in the system.

    We really think that we wanna align to the value that our customers are trying to deliver, which at the end of the day, if you’re a developer or if you’re an SRE, your goal is to deliver these resilient, reliable services. So, we feel that aligning to the services – and no matter whether you wanna bring in your whole team or maybe you also wanna bring in sort of extended folks into the solution to be able to collaborate and figure out what the problem is.

    So, we don’t wanna let the number of seats become an impediment. We really would like to have our customers drive this notion of service ownership across the entire team, as opposed to being owned by a few key people, because there’s a lot of burnout that happens.

    So, there’s different models, but after a 30-day free trial, if customers see value and they have these active services, that’s when the billing kicks in. It’s all done by credit card, so it’s a fairly streamlined experience.

    Ben Sigelman: I will add that Lightstep observability experimented with seat-pricing several years ago and abandoned it, because it’s very important for us to deliver value to our customers, and there’s nothing quite as frustrating as hearing people say, oh, we really wanna be using this, but we don’t wanna add another seat license and blah, blah, blah.

    I think especially in large organizations, for incident response, it’s just immensely frustrating to be kind of wanting to use something, like, sporadically, maybe once every couple of months, and then being blocked out of the application because of concerns over seat licensing.

    So, with Lightstep Incident Response, there’s really no restriction on that. You can have a thousand-head engineering organization and put them into the product, and that doesn’t change your bill at all. The pricing is more correlated to the value that we believe we’re delivering, and the resiliency of the services themselves.

    Alan Shimel: Sure. So, I’ve gotta ask the question. I’m gonna ask it of you, Ben. This is a step of taking observability down the control, as you mentioned, the control path, for full control. What’s next?

    Ben Sigelman: That’s a good question. We’re gonna move into crypto. No, I’m just kidding, we’re not.

    RJ Jainendra: Join the crowd, yeah.

    Ben Sigelman: I’m not here to talk about specific products, but directionally for Lightstep as a brand, I think that we see there’s an entire ecosystem out there of – I think you can kind of extrapolate from where I was going with the workflows.

    I think where we – actually, Lightstep and ServiceNow are incredibly similar in our belief in workflows being paramount, and there are a number of other workflows where people have to do a major context switch when they’re involved with the work of making reliability and velocity happen for cloud-native applications and for customer-facing applications. That’s kind of a hint as to, I think, where we’re headed with future products. But this certainly isn’t the last one yeah.

    Alan Shimel: Great. Guys, we’re way past 15 minutes, but for people who wanna go get Lightstep and Lightstep observability up and running today, where do they go?

    RJ Jainendra: www.Lightstep.com.

    Alan Shimel: That’s what I was looking for, RJ. Hey, gentlemen – Ben, RJ – thanks so much for coming on and breaking this exciting news and offering, new offering from the Lightstep/ServiceNow team. Keep us posted, best of luck with it. It sounds like a winner. We hope to hear more soon.

    RJ Jainendra: Thanks, Alan.

    Ben Sigelman: Thank you, Alan.

    Alan Shimel: Thank you, guys. All right, go check it out, Lightstep.com, their new observability and incident response now as part of it, from ServiceNow. We’re gonna take a break. We’ll be right back.

  • Empowering BizDevOps Teams With SLOs

    Empowering BizDevOps Teams With SLOs

    When BizDevOps teams are drowning in logs and false alerts and are spending more time on manual processes than on building and deploying new applications and features, the work suffers, morale suffers and the organization as a whole suffers. Releasing better and more secure software, faster and with real business impacts starts with a robust BizDevOps practice. A crucial step IT leaders can take to align the developer, operations and business teams is to create a shared language of service-level objectives (SLOs).

    SLOs are metrics for validating that software is meeting its technical and business objectives. Defining and implementing SLOs help align developer, operations and business teams, not only ensuring everyone is moving toward the same goals, but also providing a way to reliably measure how BizDevOps teams are progressing toward these goals.

    Here are three of the most impactful ways SLOs can empower your BizDevOps teams.

    1. Establish a Common Language Across the Delivery Cycle

    The best SLO gives you precise and actionable insight into your system. Take performance and resiliency engineering, for example. By defining SLOs for performance and error rates, teams can use common measurements to track progress toward meeting their quality criteria to ensure they are staying within their error budget.

    In this way, teams can use SLOs as a common language across the delivery cycle – something everyone from developers to operations to business teams can rally around, ensuring everyone is working toward the same outcomes. This aligns teams into a coordinated BizDevOps practice, where all sides have better insight into how each piece of the business feeds into the others. If an incident occurs, operations can use an AI-based observability platform to observe its trends and impacts across different product or service releases and automatically feed that information back to the developers, who can modify their builds accordingly. When everyone is using the same SLOs to speak the same language, BizDevOps teams can coordinate and communicate more easily and effectively.

    2. Creating a Culture of DevOps Resiliency by Shifting SLOs Left

    BizDevOps isn’t just a technical practice defined by tools and platforms. It’s also a culture change and a shift in mindset. SLOs can play a major part in this change. Shifting SLOs left into the delivery pipeline means giving developers feedback about the outcome of their work and real-time visibility into their business KPIs using consistent SLO measurements – ideally, on every code commit. This consistency gives them answers at their fingertips to make data-backed decisions that consistently deliver better business outcomes. Using SLOs for developer feedback is a fundamental building block of DevOps, but it’s also key to creating a culture and mindset of resiliency within BizDevOps.

    A resilient BizDevOps culture is more than just making changes to your technology stack; it’s about changing the mindsets of individual engineers and developers. Resiliency means strength and reliability; the trust that when something goes down, it’ll go right back up. It’s about building the processes and systems that guarantee flexibility for anything from managing crisis events to delivering service continuity. Shifting SLOs left helps facilitate this culture change by giving developers greater exposure to the impact their work has when it’s in production and when it’s deployed. It makes everyone feel they’re in the same boat, and recognizes developers, operations and the business teams as one cohesive unit.

    3. Integrating SLOs Into Remediation

    BizDevOps teams can use SLOs to enhance remediation and incident response efforts. With everyone working with the same metrics, BizDevOps knows with certainty when something is broken, can identify root causes more quickly and can develop agreed-upon incident responses to accelerate time to resolution. BizDevOps teams can then feed this incident response knowledge back into quality gates earlier in the development cycle to avoid future problems. This develops SLOs that have been “battle tested” and can be easily automated to make quality gate testing better on the development side, and incident response better and faster for the operations side.

    SLOs bridge the gaps between BizDevOps

    SLOs already serve a useful role in production, but they also can influence the ability to create a robust, resilient BizDevOps practice. Shifting SLOs closer to developers and using them as a shared language across the delivery cycle provides space for greater experimentation, more innovation and a culture of resiliency. SLOs can also be defined along business-relevant metrics, such as conversion rates, user experience or feature adoption. As a result, SLOs empower BizDevOps teams to respond more quickly to issues and deliver better digital experiences and business outcomes.

    SLOs aren’t just performance metrics – they’re a powerful tool for bridging the divides between the developer, operations, and business teams. SLOs can help cultivate a BizDevOps culture that delivers innovative, secure software faster and with tangible business value.

  • 7 Important Truths About Chaos Engineering

    7 Important Truths About Chaos Engineering

    As a relatively new practice, chaos engineering has plenty of myths surrounding it, from randomly shutting down production systems to requiring huge investments of time and money. There’s a lot of confusion over the purpose, the value and the practice of chaos engineering. This presents a problem for DevOps teams, especially since more than half of the teams surveyed in the 2019 Gartner DevOps Survey listed improving system reliability and release quality as one of their top five DevOps objectives.

    In this article, we’ll clear up some of this confusion by presenting seven important truths to help you make an informed decision about chaos engineering and how it can help your engineering organization.

    Truth 1: Chaos Engineering is not Chaotic

    What comes to mind when you think of chaos engineering as a practice? If the answer is causing random production outages, you’re not alone. Tools such as Chaos Monkey popularized this idea, but for most teams practicing chaos engineering, it is a well-planned, controlled process that aims to mitigate chaos rather than cause it.

    The goal of chaos engineering isn’t to add chaos, but to mitigate chaos.

    It’s true that it involves creating potentially harmful conditions on otherwise healthy systems. For example, we might test our application’s ability to handle load by increasing CPU usage on our servers. With enterprise solutions we have full control over which systems are affected by this test (known as the blast radius), how much CPU we consume (known as the magnitude) and how long it runs for. We can also immediately stop the test and roll back its impact in case of unexpected consequences.

    We also know exactly what conditions we’re introducing into our systems and can revert the changes at any time. Yes, we’re still causing harm, but we’re doing so in a way that helps us learn about our systems and reduces the risk of failures, both real-world and induced.

     Truth 2: Developers Care About Reliability

    Developers aren’t only interested in building new features. While development teams often prioritize feature development, this is likely due to business-driven initiatives resulting in rapid release schedules. Developers—especially those who have responded to incidents or worked through bug reports—understand the value of resilient software. They just don’t have time to build it.

    The problem is that this creates a reliability gap in our applications. Rapidly building features without adequately testing their resilience creates failure modes, leading to problems that developers need to go back and fix at the expense of new projects. The longer we go before finding these failure modes, the more likely we are to experience unexpected behaviors and outages in our applications, and the more expensive a solution will be.

    Increasing development velocity without testing for reliability increases the likelihood of failures, causing our expected availability to drop.

    To prevent this gap from forming, we need to prioritize and provide the appropriate tooling so developers can test their code and improve the resilience of our applications. This leads to our next truth.

    Truth 3: You Have Enough Time for Chaos Engineering

    Chaos engineering saves you the time you would spend responding to, troubleshooting and fixing production incidents. As you build resilience through chaos engineering, you’ll reduce the engineering time, effort and costs spent on incidents and can focus more on your core competencies.

    That said, adopting chaos engineering does require some time investment. Engineers will need to learn the principles and practices, adopt new tools, run experiments and implement fixes. However, this is relatively minor compared to the cost of production outages. Without chaos engineering, your engineers will need to pause their regular work and rush to restore service. This is time they could be spending on feature development and other value-creating tasks.

    A few minutes of proactive testing can uncover defects that would’ve taken hours to fix in production.

    Truth 4: Your Systems Support Chaos Engineering

    You don’t need to use specific hardware, software, or cloud platforms to benefit from chaos engineering. While some tools have specific requirements, most only require access to the host operating system. Some can even target specific resources, such as containers.

    Truth 5: You Don’t Need Comprehensive Observability (But It Helps)

    You don’t need a comprehensive monitoring or observability practice in place to benefit from chaos engineering. Basic metrics provided for free by your cloud provider can still provide important insights into how your systems are responding to experiments, and some chaos engineering tools provide basic monitoring for you.

    That said, chaos engineering provides significantly more value when used with monitoring and observability. We want to know how our systems are responding to our experiments, and the way we do that is by collecting metrics and traces. Even if you’re currently implementing a monitoring solution, chaos engineering can help streamline this process by helping you determine which metrics to track, fine-tune your alerts and avoid alarm fatigue.

    Truth 6: Chaos Engineering Pays for Itself

    Like any new tool or practice, chaos engineering requires an upfront investment of time and money. Solutions need to be evaluated and purchased, and engineers need time to warm up to the practice. However, the value it provides far exceeds these costs. Using chaos engineering can help you:

    • Avoid losses in revenue and productivity due to incidents and engineering time spent resolving incidents.
    • Improve your ability to generate sales by helping you build reliable systems and prepare for periods of increased demand.
    • Save costs by helping you optimize resource allocation and test for non-essential infrastructure.

    Truth 7: You Can Get Started Today

    Getting started with chaos engineering can be as easy as deploying an agent to your systems and choosing a resource to target for experimentation. You don’t need to have a stable, mature SRE practice or enterprise-grade infrastructure. In fact, you can use chaos engineering to help establish your SRE practice or accelerate migration to a new platform.

    The most important truth is that no system is perfect, and there are failure modes hidden all throughout our applications and infrastructure. Chaos engineering lets us uncover these failure modes and transform reliability from myth to reality.

  • How To Build a Culture of Resilience Through Good Habits

    How To Build a Culture of Resilience Through Good Habits

    Good habits are hard to form. I’ve been listening to the audiobook “Atomic Habits“ by James Clear on my morning runs, and something struck me. At Gremlin, along with our software, what we’re trying to promote are positive new habits for our customers. According to the author, one of the primary reasons new habits don’t stick is because there’s often sacrifice without immediate gratification.

    Psychologically, we’re wired to want instant gratification. But not all habits give immediate rewards; in fact, many delay our gratification for some time. So how do we help ourselves pursue good habits? Put simply: We need to make them obvious, attractive and easy.

    One of the ways to make them easy is to focus on what the author calls “gateway habits”—the smallest piece of the habit that can reasonably be achieved in two minutes. So if your goal is to eventually run a marathon, the gateway habit is putting on your running shoes every day.

    To build a culture of resilience at your company, start small and create getaway habits. If a team runs a GameDay once a month (time dedicated to experimenting on your systems) or even simply runs their first single chaos engineering experiment, then award that person or team with immediate recognition.

    Here are some other ways to build a culture of resilience at your organization:

    • Recognize the change to new habits.
    • Create DNR (do not repeat) items.
    • Adopt “You build it, you own it.”
    • Track the four golden signals.

    Recognize the Change to New Habits

    Incentives are a great way of kick-starting a new habit, but they don’t necessarily sustain the good behavior. It’s identifying the improvements that result from the new habit that really makes it stick. We’ll get to some specific metrics you can track later in the article, but notice for now that identifying the improvements is when gratification starts to drive enthusiasm. Ideally, that enthusiasm grows until the new habit becomes part of your identity.

    In our example of running a marathon, the moment of most significant change is when the person starts to self-identify as a runner. Then, the habit is no longer a chore, but rather part of who they are. Ideally, we want all computer engineers to adopt a specific set of habits until they consider themselves site reliability engineers (SREs) as part of their identity. That’s when the habit is solidified. 

    Create DNR (Do Not Repeat) Items

    Engineers and product managers want to ship new products and features. There’s nothing quite as satisfying in software development as deploying new code and seeing what you built running out in the world. But, if what you built is consistently breaking or providing a bad user experience, then you are hurting your customers and ultimately your business.

    To make sure we are always learning and getting better, at Gremlin we have what we call “DNR,” or Do Not Repeat work. This work consists of action items from outages and incidents that must not ever be repeated, lest we fail to learn our lesson from these failures. Practically, what this means is all feature work is halted until the issues highlighted as DNR work are remedied and the fixes are verified. In other words, you don’t get to write new code until your old code is fixed. We all know that many teams struggle with the trade-offs of moving fast, but ultimately, if you don’t have strict guidelines in place, then more often than not engineers will prioritize shipping something over making sure it’s reliable.

    Creating a DNR item is an easy way to incentivize the behavior you want to see internally by appealing to the engineer’s desire to produce new features. We convince them to write better code because better code means they get to spend less time fixing things.

    Adopt “You Build It, You Own It”

    This is the driving principle of DevOps. It is the reason behind shifting left. When the team that develops the software is different from the team that operates it, then there’s a misalignment of incentives. If I am a developer being tracked (and promoted) solely on the amount of code I ship, my focus will be on getting more bits out the door and not on ensuring the features I release will withstand the burdens of operation. That’s another team’s concern.

    That’s the motivation behind the proliferation of the “you build it, you own it” mindset at top-performing organizations. Hell, my first day at Amazon, they tossed me a pager and said “Good luck.” And while that may sound daunting (and it was), I can tell you that it not only motivated me to make sure I built systems to last, but it also fundamentally changed the way I thought about architecting systems. Spoiler: I’m not a big fan of my pager going off at 3 in the morning.

    In other words, the person or team building the system needs to be the same person or team that feels the pain if that system is failing. But it’s not just about punishment and pain, that team also needs to be recognized and rewarded when their system is running reliably. This creates an alignment of incentives that promotes the kind of habits seen across top-performing teams.

    Track the Four Golden Signals

    In monitoring distributed systems, Google’s SRE book outlines the four golden signals of monitoring as latency, traffic, errors and saturation. If I could wave a magic wand and immediately improve the culture of an organization, then I would have service-level objectives (SLOs) tied to these four metrics. But going back to habit formation: If you make acquiring a habit too difficult upfront, then an engineering team will ultimately reject it. So if your organization isn’t mature enough to create SLOs, simply beginning to track these metrics will up-level your game tremendously. They will give you an understanding of what is not working well and help guide your priorities.

    To Summarize

    Imagine a world where, after a major incident happens, the points of failure involved become DNR work. The same failure is not allowed to happen again and no new feature work will be completed until the fixes are implemented. And more importantly, no new feature work will be completed until those fixes are verified via a chaos engineering experiment, which is then cataloged and run continuously against your system. Then you take that knowledge and share it with other teams so they can run the same experiments and make sure they are immune to those failures as well.

    This is how you build a culture of resilience.

  • Increasing Your ‘Bionic Footprint’ with DevOps

    Increasing Your ‘Bionic Footprint’ with DevOps

    “The Six Million Dollar Man” was a 1970s TV series. In the show, NASA astronaut Colonel Steve Austin is severely injured in a crash and is “rebuilt” with bionic features that enhance his strength, speed and vision beyond normal human capabilities. The main tagline was, “We can rebuild him, we have the technology.”

    I recently was watching an old episode and it got me thinking about how the show is analogous to the genesis of DevOps—specifically, how individuals with an operations background have to “rebuild themselves” to keep up with the speed and direction of the technology industry.

    Over the years, there have been several decrees of “Unix SysAdmins are going away.” I have always firmly disagreed with that assertion. Rather, I believe that SysAdmins have to continually evolve their engineering skills to stay relevant and, more importantly, gainfully employed. It was also becoming increasingly apparent that the manual tasks of deploying infrastructure, keeping it patched, monitoring and the myriad other things a SysAdmin performs every day could not keep up with the demands of the increasing pace of the digital world and the onset of cloud computing. Infrastructure is moving rapidly to being automated and programmatically deployed and the talent needs to adjust accordingly to stay relevant.

    With many enterprises adopting the agile methodology and moving toward a continuous integration/continuous delivery (CI/CD) pipeline, the culture of DevOps became a reality. Due to the increased velocity, there became a clear need for the core tenets of DevOps culture, which include culture, automation, measurement and sharing (CAMS). In Six Million Dollar Man terms, this was simply “Better … Stronger … Faster.”

    Embracing DevOps through CAMS is similar to the superhuman enhancements that composed the rebuild of Steve Austin, which took him from man to bionic man. Here are the four ways that DevOps can increase your enterprise’s “bionic footprint”:

    Improve Culture

    Even Austin’s bionic features only got him so far; the bionic man had to rely on his team to complete his dangerous missions unscathed. The same is true in the enterprise: Building a collaborative culture is crucial for success. Culture is paramount in maintaining overall velocity without adversely affecting quality. Developers and operations teams need to function in concert to ensure a resilient application and infrastructure.

    Increase Velocity via Automation

    Automation greatly increases velocity, akin to how Austin’s bionic legs allowed him to run at high speeds and make great leaps. Automating processes saves time, money and lets your team focus on more complex tasks—a win-win for productivity gains.

    Measurement

    Steve Austin had a bionic left eye that enabled him to see in the dark as well as detect heat. This provided him precise measurements of his surroundings in any situation, enabling him to gain the upper hand against adversaries. Similarly, measurement gives insight into all aspects of the software development life cycle, application performance and other key metrics. Without insight and transparency into measurement, your team is essentially working blind.

    Sharing

    Austin had a bionic right arm that had the strength equivalent of a bulldozer. Strength and resiliency are increased by teams that work and communicate effectively made possible by DevOps.

    To summarize, I strongly believe that the mantra of “Better … Stronger … Faster” should be that of every DevOps team moving forward. DevOps’ parallel of CAMS upgrades your team’s capabilities to perform more productively. There is no end game; the objective is about continuous improvement, velocity and sustainable growth. Enterprises that want to undergo digital transformation need to adopt this cultural approach to increase their “bionic footprint” to succeed.

    About the Author / Mike D. Kail

    Mike D. Kail is chief innovation officer at Cybric. Prior to CYBRIC, Mike was Yahoo’s chief information officer and senior vice president of Infrastructure, where he led the IT and global data center functions for the company. Prior to joining Yahoo, Mike served as vice president of IT operations at Netflix, where he was responsible for employee technology and various cloud engineering components. He has been widely recognized for his insightful industry commentary on social media Twitter, and was recently named by the Huffington Post as one of the “Top 100 Most Social CIOs on Twitter.” Connect with him on LinkedIn and Twitter.