devopsy.kinsta.cloud

Author: JJ Tang

  • What Does AIOps Mean for SREs?

    What Does AIOps Mean for SREs?

    It seems SREs are of two minds when it comes to AIOps. On one hand, AIOps’ potential is pretty exciting. By automating complex workflows and troubleshooting processes, AIOps could make your life as an SRE much easier.

    But on the other hand, some SREs may choose to view AIOps with disdain and distrust. They might think of AIOps as just another trendy buzzword that doesn’t live up to the hype and that can become a distraction from the SRE tools that really matter.

    Which perspective is right? Should SREs embrace AIOps with open arms, or should they resist marketers’ efforts to position AIOps as the latest, greatest tooling innovation in the IT industry?

    Those are subjective questions that we can’t answer definitively, but let’s at least gain some perspective by examining what AIOps means for SREs.

    What is AIOps?

    If you keep up with IT buzzwords, you’ve probably at least heard the term AIOps—which is short for artificial intelligence for IT operations—and that it refers to the use of AI and machine learning (ML) to help automate IT operations workflows.

    The big idea behind AIOps is that, by using AI and ML to perform advanced analysis of large volumes of data from IT systems, IT and SRE teams can solve complex problems more efficiently than they could using a traditional, manual approach.

    AIOps can, for example, help to surface the root cause of a performance issue in a complex, multilayered environment like Kubernetes. Or it could make recommendations about how best to resolve an incident.

    AIOps entered the IT lexicon in 2016, when Gartner coined the term. Now, six years later, it’s a relatively well-established tool domain.

    How SREs View AIOps

    Despite the fact that AIOps has been around for some time, it doesn’t yet appear that many SREs have bought into the AIOps revolution. Catchpoint found in a 2021 survey that just 7.5% of SREs reported that AIOps tools delivered high value to their organizations.

    It’s unclear exactly why there are such low rates of excitement about AIOps, but we’d speculate that there are a few key factors at play:

    • AIOps is a new term for an old idea: Many monitoring and observability tools have included at least basic AI and ML analytics features for a long time, even before tool vendors slapped the AIOps label on their products. SREs probably realize this and, to some extent, view AIOps as an effort by marketers to rebrand functionality that is not actually new.
    • AIOps is hard to implement: Setting up an AIOps tool requires integrating it with diverse data sources and customizing it to fit your workflows and environment. It’s possible some SREs view this setup work as requiring more effort than it’s worth.
    • AIOps can’t replace human insight: While AIOps tools may be useful to a point, it would be unwise to place blind trust in AIOps-based analyses or recommendations. For this reason, some SREs may believe that AIOps encourages organizations to rely too heavily on automated tools at the cost of the expert analysis and perspective that only SREs can provide. (This is similar to the fact that it sometimes makes sense to prioritize human intuition and expertise over playbooks.)

    From an SRE’s perspective, then, AIOps may appear over-hyped, overly complicated and underperforming compared to traditional approaches to SRE.

    What SREs can Gain From AIOps

    SREs’ wariness toward AIOps is valid—but only to a point. It’s important not to let suspicions about the limitations of AIOps turn into excuses not to use AIOps at all. AIOps has some value to offer to SREs, even if it’s not perfect.

    For example, AIOps can play a role in reducing toil. To the extent that AIOps tools can recognize complex patterns or discover interrelated data sets more quickly than human engineers, AIOps reduces the time SREs have to spend manually troubleshooting problems or poring over complicated information.

    AIOps also helps to enable a more proactive approach to monitoring and incident management. If AIOps tools can alert SREs to emerging issues before they would otherwise recognize them, AIOps can help the SREs to get in front of the problems before they turn into true incidents. That’s better for SREs and end users alike.

    There is also an argument to be made that AIOps can help SREs do more with fewer engineering resources. If you can use AI to automate some aspects of monitoring and incident response, you can maintain the same levels of availability and performance with fewer human engineers on hand.

    Conclusion: AI Won’t Replace SREs—But it can Help

    None of the above is to say that AIOps can replace SREs, or that it magically solves every problem SREs face. Anyone who believes AIOps is a silver bullet has bought into the marketing hype to an unhealthy degree.

    Nonetheless, AIOps tools do offer value to SREs by making their jobs easier in some respects and improving reliability outcomes.

    So, while it’s wise to maintain a healthy perspective about the limitations of AIOps, SREs shouldn’t rule out AIOps tools as one way to improve reliability engineering.

  • The Evolution of Incident Management

    The Evolution of Incident Management

    Have you ever thought about the history of incident management?

    If you’re an SRE, you might be so caught up in the day-to-day work of managing reliability and responding to incidents that you never take time to step back and reflect on the evolution of your role and your responsibilities. And that’s a shame because SREs didn’t invent incident management concepts and strategies on their own.

    On the contrary, the way SREs think about incident response, structure incident management teams and rank the priority of incidents owes much to incident management strategies developed in the offline world decades ago. To understand fully what it means to be an SRE today, you have to appreciate this deep history of incident management and response.

    So, let’s take a look at that history and examine where modern incident response concepts originated.

    Historical Problems in Incident Management

    Societies have always had incidents, of course. Fires, floods, infrastructure breakdowns and similar crises have been happening for millennia.

    For most of history, however, humans lacked an efficient, purposeful way to manage these sorts of incidents. Response efforts were ad hoc, and their effectiveness owed more than a little of their success to sheer luck.

    Particular challenges included:

    • Lack of effective, consistent communication between stakeholders.
    • Varying organizational structures which made it hard to identify leaders, coordinate response efforts and delegate tasks.
    • Inconsistent response strategies.
    • Differing approaches to assessing the priority of incidents.

    Historically, organizations may have been able to handle incidents well enough if the incidents required response from only one, small group. But the more stakeholders involved, the harder it was to respond quickly and effectively.

    Putting Out Fires: The Birth of the ICS

    Things began to change for the better when stakeholders started thinking about better ways to put out fires—literally.

    By the 1960s, fire chiefs in California realized that they were struggling to respond effectively to the wildfires that broke out every summer. Each year brought worse blazes than the last, with more land burned and more buildings lost. The Laguna fire of 1970 brought matters to a head and was the catalyst for a new approach to incident response for fire agencies.

    After assessing what was going wrong, the fire chiefs determined that it wasn’t a lack of equipment or personnel. It was poor coordination among the various firefighting agencies that responded to blazes. Lacking a clear chain of command and a systematic approach to firefighting, the agencies struggled to deploy their resources rapidly and, thus, effectively.

    To fix the problem, California fire chiefs developed what became known as the Incident Command System, or ICS. The ICS defined a hierarchy for incident response with an incident commander at the top. It also defined several categories of incident response processes including operations, planning, logistics and finance. And it established a consistent set of terms that stakeholders can use to describe their actions during incident response, which makes it easier to communicate clearly.

    Although the ICS was initially conceived to fight fires, it became the de facto standard for organizing incident response strategies of all types.

    From ICS to NIMS

    The history of incident response doesn’t end with the ICS. A new chapter began in the early 2000s when the U.S. federal government developed an even more comprehensive approach to incident management called the National Incident Management System, or NIMS.

    NIMS was born in the wake of the September 11, 2001, terrorist attacks, which underlined the importance of efficient communication not just between different agencies of the same type (like fire departments), but of entirely separate organizations. To achieve this, NIMS expanded upon the principles of the ICS.

    In addition to adopting most of the incident command principles and practices included in the ICS, NIMS included standards for coordinating the distribution of resources. It also embraced the concept of the emergency operations center, which is in some ways similar to a network operations center in the digital world.

    In some respects, NIMS resembled a compliance framework (although to be clear, that’s not what it is). It includes fourteen management principles, similar to compliance controls, that organizations must implement to manage incidents using a NIMS approach.

    Incident Management Today

    Obviously, putting out forest fires and responding to terrorist attacks is pretty different from dealing with data center failures or a buggy application deployment. ICS and NIMS weren’t designed for site reliability engineering or IT teams specifically.

    Still, the influence of ICS and NIMS on the way SREs think is clear enough. Terminology like “incident commander” comes from these frameworks. So do concepts like shared accountability for incident response processes and the importance of involving all stakeholders—not just technical teams—in incident response.

    ICS and NIMS may not be acronyms familiar to most SREs. But they should be, because they are the historical sources of the incident management philosophies that form the foundation for SRE work today and offer valuable lessons for any SRE on the job today.

  • Observability Vs. Monitoring: What’s the Difference?

    Observability Vs. Monitoring: What’s the Difference?

    As more and more IT organizations pivot from strategies rooted in monitoring to ones that aim to achieve observability, one of the first questions that teams need to answer is, “What is the difference between the two?”

    That’s a somewhat loaded question because there are some differences of opinion out there regarding what, exactly, distinguishes monitoring from observability. But in general, the differences can be distilled into two main points, as we will explain.

    What is Observability and What is Monitoring?

    The first step in understanding the difference is to understand what each term means on its own.

    The classic definition of observability is that it is the use of external data outputs to understand the internal state of a system. In the IT industry specifically, observability could be defined as the use of logs, metrics and distributed traces to understand the state of a complex software system.

    Meanwhile, monitoring is the process by which data is collected from a system. In IT, monitoring is how teams collect logs, metrics and traces and move them to a place where they can be analyzed.

    Similarities

    There are some obvious similarities between the two. Both involve (at least in part) the collection of diverse sets of data. Both also serve the goal of helping teams identify problems within their software stacks and deliver a high-quality user experience.

    Two Key Differences

    But the similarities end there. Ultimately, monitoring and observability are distinct concepts and processes, with two main differences that distinguish them from each other.

    Monitoring collects data. Observability interprets it.

    The first is that observability focuses on interpreting and understanding data, whereas monitoring is merely the collection of data.

    In this sense, you can think of monitoring as one of the processes that makes observability possible. But observability requires more than monitoring. In addition to collecting data, observability also involves the correlation of data from multiple sources and the identification of relevant patterns or anomalies within that data.

    Monitoring Tells you What is Wrong. Observability Tells you Why it’s Wrong

    The second main difference is that monitoring simply alerts you when something goes wrong. Observability goes further by helping you understand why it’s wrong and how to fix it.

    For example, a monitoring tool might tell you that your application’s response rate is no longer adequate. But you need observability to figure out which specific microservices within the application are causing the problem. Using that information, you can plan and execute an effective response to reliability issues using an incident management platform.

    With monitoring alone, you only know that something is broken, not how to fix it.

    Observability and Monitoring: Better Together

    Although it’s common to speak in terms of “observability versus monitoring,” the reality is that you can’t have observability without monitoring. (You could monitor without observing, but doing so would significantly reduce your ability to understand and resolve problems quickly.)

    So, don’t think of it as an either/or proposition. Think of them as concepts and practices that go hand-in-hand, even if they perform different functions.

  • Defining Availability, Maintainability and Reliability in SRE

    Defining Availability, Maintainability and Reliability in SRE

    In the world of reliability engineering, you’ll frequently encounter the three “-ability” words: Availability, maintainability and reliability. They sound similar and have similar meanings. In fact, these words may seem so similar that it can be tempting to use them interchangeably.

    That would be a mistake. Availability, maintainability and reliability all have distinct—if related—meanings, and they each play different roles in reliability operations.

    Definitions of Availability, Maintainability and Reliability

    Let’s start by succinctly defining each of the terms.

    Availability, Defined

    Availability is the extent to which an IT resource is ready to perform a task when requested.

    Thus, an application or server that is responding to requests is available (even if it takes longer to respond than desired or otherwise operates suboptimally, in which case it may have a reliability issue but not an availability issue).

    Availability is usually measured as a percentage. A resource that has 99% availability, for example, is one that is up and responding 99% of the time.

    Maintainability, Defined

    Maintainability is a measure of how quickly and easily a resource can be fixed when something goes wrong.

    If a buggy application release can be quickly fixed by rolling back to a stable version, the application would have a high degree of maintainability. On the other hand, if you have a server that needs to be rebuilt manually after it fails, it’s not very maintainable.

    Reliability, Defined

    Reliability is the extent to which a resource functions as required upon request (as opposed to simply being available).

    For example, if your users require an application to process each transaction within one second, and it does this 99%of the time while also maintaining a 99% availability level, that would be a relatively reliable application. In contrast, an application that responds to almost all requests but that suffers from high latency or error rates would not be very reliable, although it might be highly available.

    Differences Between Availability, Maintainability and Reliability

    The simplest way to spell out the differences between availability, maintainability and reliability is to highlight what’s unique about each concept.

    Availability

    Unlike maintainability and reliability, availability is an essentially binary metric, in the sense that a system is either available or it’s not. Although availability status can change over time, there is no such thing as varying degrees of availability.

    Availability also stands out because it is often the most important metric in defining SLAs and SLOs. Contracts typically specify that resources will achieve a certain level of availability, defined in percentage terms. They may sometimes also specify metrics like response times, which are a reflection of reliability, but availability is more likely to be the most important metric within service agreements.

    Maintainability

    Maintainability is unique in that it’s a pretty subjective concept. A system that one SRE considers easy to maintain could seem difficult to maintain to another SRE. The methodologies that engineers leverage to optimize maintainability could vary, too; for instance, someone from a DevOps background may be more likely to focus on optimizing maintainability within the software delivery chain than someone trained in classic site reliability engineering, which focuses on automating IT operations through code more than on the software delivery process.

    That said, automation tools help virtually every team maximize maintainability, no matter their preferences or background.

    Reliability

    Reliability stands out because it reflects how well a system performs. Here again, this is a somewhat subjective metric, because performance requirements may vary from one user to another. Nonetheless, reliability is the most effective means of measuring whether a system meets the performance levels it needs to, even if those requirements change over time.

    Why do availability, maintainability and reliability matter?

    Because availability, maintainability and reliability each measure different aspects of a system’s status, putting them together is a useful means of gaining insight into the overall reliability of a system.

    In order to be reliable, a system requires both availability and maintainability. A system can’t be reliable if it’s not available. It’s also unlikely to be highly reliable if it takes a long time to fix issues due to low maintainability.

    At the same time, by focusing on availability, maintainability and reliability individually, you can drill down into specific issues within the IT resources you manage. For instance, you might have systems that are high-performing when they’re available, but that have low reliability rates because of availability issues. In that case, you’ll know that an investment in increased availability is likely to yield the greatest reward for increasing overall reliability.

    The Bottom Line

    The bottom line: While availability, maintainability and reliability are all reflections of the quality of an IT resource, they measure quality in different ways. Tracking each category separately is important for ensuring that you know where your weakest links exist within overall system performance and health.

    At the same time, however, it’s important to compare and correlate availability, maintainability and reliability data so that you can achieve continuous insight into a resource’s status. When you can track each item separately but also combine them together to gain a complete picture of your system, you are in the best position to optimize reliability operations.

  • Why Observability and Monitoring Matter to SREs

    Why Observability and Monitoring Matter to SREs

    Site reliability engineers, or SREs, do many things. They help developers build reliability into applications. They manage SLAs and SLOs. They play a leading role in incident management and incident response.

    For all of these tasks, SREs draw heavily on observability and monitoring. Although other parts of the IT organization also typically help to manage observability and monitoring, it’s virtually impossible for SREs to do their jobs effectively in today’s cloud-native world without observability and monitoring tools and data.

    Here’s what observability and monitoring mean, why they’re so important for SREs in particular and how SREs can systematically measure the success of their monitoring and observability practices.

    Defining Observability vs. Monitoring

    Monitoring is the collection of data from applications, infrastructure or other IT resources to track the status and performance of those resources. Monitoring has been a core IT practice since the late 1990s when the first modern monitoring tools, such as Nagios, appeared.

    When compared to monitoring, observability is more intensive. Observability is the collection of discrete types of data from multiple systems and the correlation of those data sets to achieve actionable visibility into IT resources. One thing to note: Observability has a more technical definition in the field of control theory that has to do with using external outputs to make inferences about the internal state of a system; however, when people talk about observability in the context of IT and SRE, they usually mean the collection and correlation of broad sets of data in order to understand complex systems.

    The debate about the differences between observability and monitoring is ongoing, but the consensus is that they boil down to:

    • What versus why: Monitoring helps SREs determine that a problem exists. Observability goes further by allowing them to understand why it exists.
    • Context: Observability provides context around what is wrong with a system. Monitoring, for the most part, offers surface-level information about the existence of a problem.
    • Actionability: Observability empowers SREs to respond effectively to outages, performance degradations or other problems. Monitoring merely lets them know that something is wrong and that it may require intervention.

    In most respects, monitoring is one step in the greater observability process. But observability involves additional practices, such as the correlation of various types of data, to achieve a level of context and actionability that monitoring alone can’t deliver.

    Why SREs Should Understand Observability

    Again, observability has become a core practice for engineers in a variety of roles. Developers can use observability tools to help measure and optimize application performance before deployment. IT engineers can leverage observability to gain visibility into issues that exist in a production environment. Observability tools can help QA engineers determine why an application failed a test.

    But no group has more to gain from observability than SREs. Given that maximizing the reliability and performance of systems is the core mission of SREs, the ability to not just detect problems via monitoring but also to understand them through observability is critical for modern SRE teams.

    That’s all the more true for SREs tasked with managing reliability for complex systems, like those based on microservices. In complex applications, identifying the root cause of a reliability issue—such as which specific microservice contains buggy code or which type of request triggers a performance bug—can be very difficult. By maximizing context into reliability problems, as well as allowing teams to trace requests across complex, distributed systems, observability places SREs in the strongest position to find and fix reliability problems quickly.

    What’s more, beyond detecting issues in systems that have already been deployed, SREs can also leverage observability tools to help build reliability into applications before they are put into production. By observing applications in dev/test environments, SREs may be able to identify reliability risks and then use that insight to find ways to make the application inherently more reliable. Observability might reveal reliability weaknesses that stem from an application’s architectural design, for instance, or from the orchestration tool that manages the application.

    How to Measure Observability and Monitoring

    Getting the most from observability and monitoring requires systematic measurement of observability and monitoring initiatives. It’s only by collecting metrics about observability and monitoring outcomes that SREs can ensure they are improving reliability rather than merely collecting and analyzing data to no particular end.

    While a full discussion of observability and monitoring measurement is beyond the scope of this article, consider the following types of metrics for assessing these practices:

    • Incident frequency: How often do incidents (especially severe incidents) occur that impact production systems? Ideally, SREs will see a decrease in incident frequency that correlates with increased use of observability and monitoring tools.
    • Rollback frequency: The less often an application release is rolled back to a previous version, the more effective the SRE team’s observability and monitoring practices likely are.
    • Mean-time-to-remediate (MTTR): MTTR is a measure of the overall time it takes to resolve a reliability problem. It reflects how well you monitor (which determines how quickly you detect problems) and how well you observe (which is the key to resolving problems).
    • Total data analyzed: Tracking the total volume of data that you collect and analyze through observability and monitoring tools helps you assess the overall scope and scalability of your practices.
    • Total systems analyzed: In addition to tracking total data volume, consider measuring how many discrete applications, hosting environments, CI/CD tools and other systems you are managing as part of observability and monitoring workflows.

    The ultimate goals of SREs extend beyond observability and monitoring. However, observability and monitoring serve as essential means to the broader ends that SREs are tasked with achieving: Maximizing uptime and performance. And while monitoring alone was once sufficient for supporting the goals of SREs, it’s very difficult to imagine an SRE team today that doesn’t also make extensive use of observability, which takes monitoring to the next level.

  • 5 Steps to Succeed With Blue-Green Deployment

    5 Steps to Succeed With Blue-Green Deployment

    By allowing teams to maintain two production-ready environments at the same time, the blue-green deployment technique can significantly boost reliability. But blue-green deployment can also be difficult to execute and manage. Let’s unpack how blue-green deployment works, why it’s important and which best practices to follow for blue-green deployment success.

    Blue-Green Deployment, Defined

    Blue-green deployment is a software deployment technique in which teams maintain two identical but distinct environments that are capable of hosting a production application. One environment is “live” and actively serving users, while the other is kept in reserve.

    When it comes time to deploy a new version of your application, you deploy it to the reserve environment. You then test and validate it to make sure it meets performance and reliability requirements.

    If it does, you reconfigure your DNS or load balancers so that user traffic is directed to this environment, which becomes the new production environment. Meanwhile, the previous production environment is placed in reserve.

    Benefits of Blue-Green Deployment

    The blue-green deployment technique can boost reliability in several ways:

    • Environment parity: Because the reserve environment in which you test your application becomes the live production environment, blue-green deployment ensures parity between testing and production. In turn, it reduces the risk that configuration differences could cause unforeseen problems in production.
    • Failed deployments: If a deployment fails, it doesn’t impact your users because you deploy into the reserve environment. You can fix the issue while a stable user-facing version of your app continues to run in the other environment. 
    • Redundant environments: Although blue-green deployment is not a data protection or disaster recovery technique, having two production-ready environments available is useful in the event that one of your environments is wiped out by an event like a data center failure. In that case, you can restore operations quickly by using the other environment (provided it’s hosted in a separate data center, of course).

    How to Use Blue-Green Deployment: Steps and Best Practices

    While blue-green deployment is a great way to increase reliability, it does present some challenges. The following best practices can help you to handle them.

    Use Scalable Infrastructure

    One obvious potential downside of blue-green deployment is that it can essentially double the amount of resources you need to host your application since you have to maintain two identical environments. That can double your hosting bill.

    One way to reduce the cost impact of blue-green deployment is to create environments that can scale rapidly without changing their core configuration. You could do this using cloud-based virtual machines (VMs) and autoscaling or via containers and an orchestration engine (like Kubernetes) that can scale them.

    Using this approach, you can set up two environments that are effectively identical except that the production environment includes more VM or container instances than the reserve. Although you could argue that this is not quite a blue-green deployment because the environments are not perfectly identical, that semantic downside is outweighed by the benefit of not wasting a bunch of money on infrastructure you don’t need.

    Chaos Engineering

    Typically, chaos engineering entails experimenting with a production system to test its reliability. Chaos engineering offers many benefits, but it presents the obvious risk of creating a problem in production that impacts your end users.

    Blue-green deployment allows you to work around this risk by practicing chaos engineering in the reserve environment. Of course, because you won’t have real users, performing chaos engineering under these settings may not allow you to test every aspect of your production application. But you can still test many of the configuration variables by issuing requests yourself within the reserve environment—all without worrying about breaking things for actual users.

    Managing Database State

    Keeping databases in sync when you switch between environments is one of the biggest challenges of blue-green deployment. Even small discrepancies in database state could cause big reliability issues.

    The simplest way to address this issue is to turn off the database before switching and perform tests to ensure that databases in both environments are identical before they are reenabled. Of course, this approach may impact application functionality, because the app may not work properly if the database is turned off.

    Another solution is to sync databases between your two environments. This should ensure that database state remains identical even as you switch from one production environment to the other. But it also means that problems with the database will affect both environments.

    The bottom line: There is no simple way of managing database state in blue-green deployment. You should evaluate the different options available and consider which one best meets your needs. If you can tolerate a little downtime, stop the database and restart it during the switch. If you need total continuity, use syncing, but be sure to manage the associated risks.

    Change Load Balancers, not DNS

    There are two main ways to manage where traffic is directed, and hence which is the production environment: You can change your DNS configuration when you want to switch. That’s easy to do—you would simply change the mapping for yourapp.com from one IP address to another, depending on which environment is the live one.

    You can also use load balancers to manage traffic. Load balancer configuration is more complex than simply changing DNS entries, but it also enables more sophisticated and granular control.

    In general, it’s a best practice to manage blue-green deployment switches via load balancers, due to the extra control they provide.

    Prepare for the Unexpected

    While blue-green deployment can reduce the risk of user-impacting disruptions, it’s no silver bullet. You must also be sure that you have the reliability management tools you’ll need to detect and remediate issues within both of your environments.

    Blue-green deployment is a powerful reliability engineering technique. But to get the most out of it, you must optimize the way you handle requirements like infrastructure management, traffic routing and database state. 

  • Does Shift Left Matter to SREs?

    Does Shift Left Matter to SREs?

    If you’re a software engineer, you’ve likely heard all about shift left, a practice that can streamline certain aspects of software development.

    But shift left isn’t just for developers. It can be equally valuable for site reliability engineers (SREs). Although the main mission of SREs is ensuring the reliability of software after it has been deployed rather than actually developing software, encouraging shift left practices as part of a software development strategy can nonetheless make it easier to optimize reliability.

    To prove the point, here’s an overview of what shift left means and why it matters for SREs.

    What Does Shift Left Mean?

    Shift left is the detection of application problems early in the software development process. The core idea behind shift left is that instead of waiting until just before deployment to test software for bugs, security issues or other problems, teams should begin the testing process as early as possible. In other words, they should shift some testing to the “left” of the software development life cycle instead of testing only in the middle—hence the term “shift left.”

    In so doing, they can detect problems early in the development process, at which point they are typically easier to resolve. If you don’t catch a performance, reliability or security bug until just before you plan to push your application release into production, you may have to overhaul a large part of your codebase to fix the issue. In many cases, you’ll have to change not just the specific code that triggered the bug, but also other code that depends on or integrates with the buggy code. But by finding buggy code early, developers can often fix the problem with minimal adjustments to the application as a whole.

    Shift left is a high-level concept, and it can be implemented in multiple ways. In some cases, developers may start running performance tests on new code right after they write it, even if it has not yet been integrated into the main codebase. In other cases, shift left testing could mean compiling some parts of the codebase in order to run tests against it before the application as a whole is built and tested in a dev/testing environment. Either way, the result is a testing process that begins earlier than it would under a conventional approach.

    Importantly, shift left doesn’t mean that teams skip the tests that they would normally perform just prior to deployment. Those tests, which are important for catching bugs that materialize only when testing the application as a whole rather than individual parts of it, typically still happen as part of a shift left strategy. Thus, shift left means that testing begins earlier and coincides with a broader portion of the overall development pipeline, not that testing is simply shifted to the beginning of the pipeline at the expense of mid-pipeline tests.

    Shift Left and SREs

    Again, the main job of SREs is to manage the reliability of systems. They may spend part of their days working alongside software engineers to help devise architectures and coding strategies that maximize the resiliency of applications against reliability and performance issues. But SREs also devote much of their time to managing what happens post-deployment. They use monitoring and observability tools to detect problems that arise in production environments. They then take the lead in responding to them.

    In that sense, SREs may not seem to have much to gain from shift left. Shift left is a strategy that aligns with software development first and foremost, and software development is not the core focus of SREs.

    Nonetheless, a shift-left strategy can help SREs do their jobs better, for several reasons:

    • Fewer problems in production: Most obviously, shift left reduces the rate at which reliability issues reach production environments. In turn, it minimizes the number of incidents SREs need to respond to.
    • Optimizing software pre-deployment: By exposing reliability issues early in the development process, shift left is one way for SREs to acquire the insights they need to collaborate with developers on building reliability into applications. Shift left could highlight buggy dependencies or suboptimal architectural patterns, for example, which SREs can encourage developers to address.
    • Granular reliability insights: Because shift left is a great way to trace software problems to the specific code that triggers them, it provides highly granular visibility into reliability. It can help SREs find the weakest links within an application’s codebase or architecture. These weak spots can be harder to detect when monitoring or observing an application as a whole.

    How SREs can Follow the Shift Left Methodology

    Because the implementation of shift left practices is a task that ultimately falls to developers, SREs need to work closely with development teams (and DevOps engineers, if they exist in the organization) to make shift left happen.

    The first step in that process is to get buy-in for shift left among developers. SREs can do this by highlighting the ways in which shift left can streamline the work of developers and reduce the complexity of fixing bugs. That’s important to communicate because it ensures that developers understand that shift left benefits everyone, not just SREs.

    SREs should also work with developers to identify the best approaches and practices to implementing shift left. A key factor to consider here is where reliability problems most often arise, and how shift left can help to detect them earlier. If most performance problems stem from code written by individual developers, for instance, testing code as soon as it is generated may be the best way to detect problems early on. In contrast, in situations where performance issues are triggered most often by environment configurations, the ability to test applications early under different environment variables could be the best way to surface issues early in the development process.

    Finally, SREs should partner with developers to respond to the issues that shift left reveals. After all, while developers typically don’t pay close attention to what happens post-deployment, SREs have a special ability to understand how problems in development translate to problems in production. SREs may therefore be able to help developers identify the most effective way to fix bugs based on production environment requirements, even if it’s not the quickest or simplest way.

    Why SREs Need to Shift Left, Too

    Although the shift left methodology has not traditionally been closely associated with SRE, shift left is a crucial tool in the SRE toolbox. By helping teams to detect reliability issues when they are easier to resolve, shift left puts SREs in a stronger position to maximize the overall reliability of applications. At the same time, it reduces the number of production environment issues that SREs have to manage.