devopsy.kinsta.cloud

Tag: SRE

  • Say Goodbye to Late-Night SRE Wake-Up Calls

    Say Goodbye to Late-Night SRE Wake-Up Calls

    Thousands of SREs, on-call engineers and DevOps pros all over the world dread nothing more than the late-night incident alert. The pager buzzing at 2:00 a.m. can cause panic for SREs and leave IT and DevOps teams with quite a mess to contain. But incident response doesn’t have to cause panic if you have the right automation tools in place. If you have ever wondered why companies like Amazon, Google and Zoom rarely suffer service outages and downtime while other companies struggle to achieve similar efficiencies, you’re not alone. In fact, you’re halfway to understanding what makes incident response automation such a vital component of your workflow. 

    Without a doubt, the best way to take back control over manual incident response and—once again—sleep through the night is to implement a powerful automation solution for incident response. We’ve listened to stories from SREs around the globe about the benefits of automated incident response. Plus, we’ve experienced them ourselves (from companies like Cisco, for example).  In the end, we’ve learned a thing or two about creating efficiency by leveraging automation and integration. Below, we’ll briefly cover how leveraging automation to remediate issues with or without a human in the loop is the best way to say goodbye to 2:00 a.m. wake-up calls.

    An Overview of Incident Response

    First, a staggering fact: In 2019, 17% of global enterprises lost more than $5 million every hour their servers were down, according to Statista. Even for smaller companies, the cost of servers going down is enormous. Following the news of Facebook’s recent outage (losing $13.3 million an hour; not counting the loss from the stock price drop), the need for minimizing downtime and reducing these costs is clear. 

    In order to fix issues faster, organizations need an easy-to-use tool that SREs and DevOps teams can implement to troubleshoot and automate incident response. For one, users should opt for a drag-and-drop system. This method is preferred over headless automation tools that too often result in data loss and extended downtimes.

    Next, IT organizations need to take the next step with automation and implement best practices. We can go from asking “How do we bring in automation?” and start to think in terms of “What are the use cases when I do?” A winning incident response toolset will help customers navigate this challenge. An automation platform should also allow users to fine-tune workflows with a library of connectors and actions that are comprehensive enough to get the job done. Ideally, your cloud stack should include several functions such as alerting (such as PagerDuty), monitoring (like Datadog) and analytics (Splunk, for example), while integrating with collaboration tools like Slack and Jira.

    What Are Some Typical Use Cases?

    Let’s dive deeper into three use cases that incident responders and engineers commonly encounter. These examples demonstrate why a powerful automated platform lets small businesses automate effectively and ensure long-term uptime just like larger enterprises.

    Incident Response Automation

    It’s all hands on deck when that 2:00 a.m. incident response alert goes off. Why rely on manual responses which can often lead to broken uptime SLAs and even drag down a company’s reputation? Automation is the key to operating any large-scale service in the cloud.

    Cost Management

    The costs of extended periods of downtime don’t stop at the checkbook. When incidents occur without drag-and-drop automation, long-term side effects can dramatically up the company’s losses.

    Orchestration

    Orchestrated automation tools provide credential management, templates, playbooks and data processing for any company size. They empower users to curate services to their liking as well.

    Want to Learn More about Automated Incident Response? 

    Sign up for this upcoming devopsy.kinsta.cloud webinar on Monday, October 25th at 3:oo p.m. Eastern. We hope you’ll join us to learn more about how to integrate automation into your workflow and stop those pesky 2:00 a.m. wake-up calls.

  • Why Observability and Monitoring Matter to SREs

    Why Observability and Monitoring Matter to SREs

    Site reliability engineers, or SREs, do many things. They help developers build reliability into applications. They manage SLAs and SLOs. They play a leading role in incident management and incident response.

    For all of these tasks, SREs draw heavily on observability and monitoring. Although other parts of the IT organization also typically help to manage observability and monitoring, it’s virtually impossible for SREs to do their jobs effectively in today’s cloud-native world without observability and monitoring tools and data.

    Here’s what observability and monitoring mean, why they’re so important for SREs in particular and how SREs can systematically measure the success of their monitoring and observability practices.

    Defining Observability vs. Monitoring

    Monitoring is the collection of data from applications, infrastructure or other IT resources to track the status and performance of those resources. Monitoring has been a core IT practice since the late 1990s when the first modern monitoring tools, such as Nagios, appeared.

    When compared to monitoring, observability is more intensive. Observability is the collection of discrete types of data from multiple systems and the correlation of those data sets to achieve actionable visibility into IT resources. One thing to note: Observability has a more technical definition in the field of control theory that has to do with using external outputs to make inferences about the internal state of a system; however, when people talk about observability in the context of IT and SRE, they usually mean the collection and correlation of broad sets of data in order to understand complex systems.

    The debate about the differences between observability and monitoring is ongoing, but the consensus is that they boil down to:

    • What versus why: Monitoring helps SREs determine that a problem exists. Observability goes further by allowing them to understand why it exists.
    • Context: Observability provides context around what is wrong with a system. Monitoring, for the most part, offers surface-level information about the existence of a problem.
    • Actionability: Observability empowers SREs to respond effectively to outages, performance degradations or other problems. Monitoring merely lets them know that something is wrong and that it may require intervention.

    In most respects, monitoring is one step in the greater observability process. But observability involves additional practices, such as the correlation of various types of data, to achieve a level of context and actionability that monitoring alone can’t deliver.

    Why SREs Should Understand Observability

    Again, observability has become a core practice for engineers in a variety of roles. Developers can use observability tools to help measure and optimize application performance before deployment. IT engineers can leverage observability to gain visibility into issues that exist in a production environment. Observability tools can help QA engineers determine why an application failed a test.

    But no group has more to gain from observability than SREs. Given that maximizing the reliability and performance of systems is the core mission of SREs, the ability to not just detect problems via monitoring but also to understand them through observability is critical for modern SRE teams.

    That’s all the more true for SREs tasked with managing reliability for complex systems, like those based on microservices. In complex applications, identifying the root cause of a reliability issue—such as which specific microservice contains buggy code or which type of request triggers a performance bug—can be very difficult. By maximizing context into reliability problems, as well as allowing teams to trace requests across complex, distributed systems, observability places SREs in the strongest position to find and fix reliability problems quickly.

    What’s more, beyond detecting issues in systems that have already been deployed, SREs can also leverage observability tools to help build reliability into applications before they are put into production. By observing applications in dev/test environments, SREs may be able to identify reliability risks and then use that insight to find ways to make the application inherently more reliable. Observability might reveal reliability weaknesses that stem from an application’s architectural design, for instance, or from the orchestration tool that manages the application.

    How to Measure Observability and Monitoring

    Getting the most from observability and monitoring requires systematic measurement of observability and monitoring initiatives. It’s only by collecting metrics about observability and monitoring outcomes that SREs can ensure they are improving reliability rather than merely collecting and analyzing data to no particular end.

    While a full discussion of observability and monitoring measurement is beyond the scope of this article, consider the following types of metrics for assessing these practices:

    • Incident frequency: How often do incidents (especially severe incidents) occur that impact production systems? Ideally, SREs will see a decrease in incident frequency that correlates with increased use of observability and monitoring tools.
    • Rollback frequency: The less often an application release is rolled back to a previous version, the more effective the SRE team’s observability and monitoring practices likely are.
    • Mean-time-to-remediate (MTTR): MTTR is a measure of the overall time it takes to resolve a reliability problem. It reflects how well you monitor (which determines how quickly you detect problems) and how well you observe (which is the key to resolving problems).
    • Total data analyzed: Tracking the total volume of data that you collect and analyze through observability and monitoring tools helps you assess the overall scope and scalability of your practices.
    • Total systems analyzed: In addition to tracking total data volume, consider measuring how many discrete applications, hosting environments, CI/CD tools and other systems you are managing as part of observability and monitoring workflows.

    The ultimate goals of SREs extend beyond observability and monitoring. However, observability and monitoring serve as essential means to the broader ends that SREs are tasked with achieving: Maximizing uptime and performance. And while monitoring alone was once sufficient for supporting the goals of SREs, it’s very difficult to imagine an SRE team today that doesn’t also make extensive use of observability, which takes monitoring to the next level.

  • Does Shift Left Matter to SREs?

    Does Shift Left Matter to SREs?

    If you’re a software engineer, you’ve likely heard all about shift left, a practice that can streamline certain aspects of software development.

    But shift left isn’t just for developers. It can be equally valuable for site reliability engineers (SREs). Although the main mission of SREs is ensuring the reliability of software after it has been deployed rather than actually developing software, encouraging shift left practices as part of a software development strategy can nonetheless make it easier to optimize reliability.

    To prove the point, here’s an overview of what shift left means and why it matters for SREs.

    What Does Shift Left Mean?

    Shift left is the detection of application problems early in the software development process. The core idea behind shift left is that instead of waiting until just before deployment to test software for bugs, security issues or other problems, teams should begin the testing process as early as possible. In other words, they should shift some testing to the “left” of the software development life cycle instead of testing only in the middle—hence the term “shift left.”

    In so doing, they can detect problems early in the development process, at which point they are typically easier to resolve. If you don’t catch a performance, reliability or security bug until just before you plan to push your application release into production, you may have to overhaul a large part of your codebase to fix the issue. In many cases, you’ll have to change not just the specific code that triggered the bug, but also other code that depends on or integrates with the buggy code. But by finding buggy code early, developers can often fix the problem with minimal adjustments to the application as a whole.

    Shift left is a high-level concept, and it can be implemented in multiple ways. In some cases, developers may start running performance tests on new code right after they write it, even if it has not yet been integrated into the main codebase. In other cases, shift left testing could mean compiling some parts of the codebase in order to run tests against it before the application as a whole is built and tested in a dev/testing environment. Either way, the result is a testing process that begins earlier than it would under a conventional approach.

    Importantly, shift left doesn’t mean that teams skip the tests that they would normally perform just prior to deployment. Those tests, which are important for catching bugs that materialize only when testing the application as a whole rather than individual parts of it, typically still happen as part of a shift left strategy. Thus, shift left means that testing begins earlier and coincides with a broader portion of the overall development pipeline, not that testing is simply shifted to the beginning of the pipeline at the expense of mid-pipeline tests.

    Shift Left and SREs

    Again, the main job of SREs is to manage the reliability of systems. They may spend part of their days working alongside software engineers to help devise architectures and coding strategies that maximize the resiliency of applications against reliability and performance issues. But SREs also devote much of their time to managing what happens post-deployment. They use monitoring and observability tools to detect problems that arise in production environments. They then take the lead in responding to them.

    In that sense, SREs may not seem to have much to gain from shift left. Shift left is a strategy that aligns with software development first and foremost, and software development is not the core focus of SREs.

    Nonetheless, a shift-left strategy can help SREs do their jobs better, for several reasons:

    • Fewer problems in production: Most obviously, shift left reduces the rate at which reliability issues reach production environments. In turn, it minimizes the number of incidents SREs need to respond to.
    • Optimizing software pre-deployment: By exposing reliability issues early in the development process, shift left is one way for SREs to acquire the insights they need to collaborate with developers on building reliability into applications. Shift left could highlight buggy dependencies or suboptimal architectural patterns, for example, which SREs can encourage developers to address.
    • Granular reliability insights: Because shift left is a great way to trace software problems to the specific code that triggers them, it provides highly granular visibility into reliability. It can help SREs find the weakest links within an application’s codebase or architecture. These weak spots can be harder to detect when monitoring or observing an application as a whole.

    How SREs can Follow the Shift Left Methodology

    Because the implementation of shift left practices is a task that ultimately falls to developers, SREs need to work closely with development teams (and DevOps engineers, if they exist in the organization) to make shift left happen.

    The first step in that process is to get buy-in for shift left among developers. SREs can do this by highlighting the ways in which shift left can streamline the work of developers and reduce the complexity of fixing bugs. That’s important to communicate because it ensures that developers understand that shift left benefits everyone, not just SREs.

    SREs should also work with developers to identify the best approaches and practices to implementing shift left. A key factor to consider here is where reliability problems most often arise, and how shift left can help to detect them earlier. If most performance problems stem from code written by individual developers, for instance, testing code as soon as it is generated may be the best way to detect problems early on. In contrast, in situations where performance issues are triggered most often by environment configurations, the ability to test applications early under different environment variables could be the best way to surface issues early in the development process.

    Finally, SREs should partner with developers to respond to the issues that shift left reveals. After all, while developers typically don’t pay close attention to what happens post-deployment, SREs have a special ability to understand how problems in development translate to problems in production. SREs may therefore be able to help developers identify the most effective way to fix bugs based on production environment requirements, even if it’s not the quickest or simplest way.

    Why SREs Need to Shift Left, Too

    Although the shift left methodology has not traditionally been closely associated with SRE, shift left is a crucial tool in the SRE toolbox. By helping teams to detect reliability issues when they are easier to resolve, shift left puts SREs in a stronger position to maximize the overall reliability of applications. At the same time, it reduces the number of production environment issues that SREs have to manage.

  • SRE’s Guide to Pragmatic Incident Response

    SRE’s Guide to Pragmatic Incident Response

    In my past experience as an SRE, I learned some valuable lessons about how to respond to and learn from incidents. If you want the TL;DR, I’ll summarize them here:

    • Declare and run retros for the small incidents. It’s less stressful, and action items become much more actionable.
    • Decrease the time it takes to analyze an incident. You’ll remember more, and will learn more from the incident.
    • Alert on pain felt by people – not machines. The only reason we declare incidents at all is because of the people on the other side of those machines.

    Now, let’s dive into each of these lessons a little deeper, and explore how they can help you build a better system for pragmatic incident response.

    1. Focus on the Small Incidents First

    The bad habit of ignoring small issues often leads to bigger issues. You should run retrospectives for small incidents (slowdowns, minor bugs, etc.) because these often have the most actionable takeaways—instead of shooting the moon and creating “rearchitect async pipeline” as a Jira ticket that never happens. Focusing on low-stakes incidents and retrospectives are a great introduction to behavior change across your organization.

    Let’s look at an obvious example of how a focus on small incidents can have an outsized impact. Case in point—my apartment. I live in an old candy factory retrofitted into apartment units. We have an elevator (thank God), but some of the buttons don’t light up when you press them. The LED display doesn’t match up with the numbers on the floor buttons. Yes, the elevator goes up and down, but overall, you can tell that things are wrong with it.

    One day, I came home and noticed an “Out of Order” notice on the elevator door. Was I surprised? Not at all. Of course an elevator with mislabeled buttons and broken LEDs would stop working!

    This isn’t dissimilar from an important software lesson—ignoring small issues often leads to much bigger issues. What starts as an inconvenience can lead to a complete breakdown. This is why you should focus on the small incidents first. I hear a lot of companies say, “We need to fix incident response! Every time we have an incident, it’s just chaos!”

    They are talking about unexpected downtime during a high severity level incident and they want to improve it.

    I say that’s the wrong type of incident to use to fix your incident response system and process. Focus on the small incidents first. Running retros for small incidents can help you build strong incident response models, because they have the most actionable takeaways and they are the best way to change behavior through repetition and practice.

    If you have a high-stakes, 12-hour incident and you run a one-hour retrospective you’re not going to get the results you want. You need to start small. Run retros for bugs that were introduced; maybe a bad data migration that didn’t really impact anything but took up a couple of hours of your day.

    Heidi Waterhouse captured this idea really well in this piece on reliability. Every airplane you’ve ever flown on has many tiny problems, Waterhouse said, “… like a sticky luggage latch or a broken seat or a frayed seatbelt. None of these problems alone are cause to ground the plane. However, if enough small problems compound, the plane may no longer meet the requirements for passenger airworthiness and the airline will ground it. A plane with many malfunctioning call buttons may also be poorly maintained in other ways, like faulty checking for turbine blade microfractures or landing gear behavior.”

    I couldn’t agree more. Extrapolating to software: The small things are typically indicators of bigger issues and could cause catastrophes down the road.

    2. Track Mean Time to Retro (MTTR)

    It’s important to think about what you measure in your organization. You should be measuring how you’re improving, and the most important metric here is mean time to retro (MTTR). Everyone should be tracking MTTR. It’s a great statistic to improve incident response in your team because it helps you understand the delay between incidents and retrospectives.

    The easiest way to have a bad incident retro is to wait two weeks. It’s better to get into a room quickly and hash out what happened than wait a long time until you’ve got everything perfectly prepared.

    Tracking MTTR can help you hold prompt and consistent retrospectives after incidents. Set a timer and make an SLO or SLA for yourself that says, “This is how long we take for retros.”

    Retro time will vary depending on the severity of the incident itself. If it’s a SEV1, clear everyone’s schedules, because you need to have retro within 24 hours of the incident. For SEV3 incidents, you have much more leniency.

    I also like tracking the ratio of retros to declared incidents. This is a metric that should go up; you should see your ratio of retrospectives to incidents increasing. You can break down that number by severity, as well. If you have a low retro ratio for a SEV1 versus a SEV3 incident, that might be okay at first (remember, start small) but you want those metrics to eventually become equal.

    3. Alert on Degraded Experience with the Service, and Not Much Else

    The severity of incidents is directly linked to customer pain. We would not declare SEV1s if there weren’t a lot of people feeling a lot of pain.

    Alerting on computer vitals is an easy way to create alert fatigue and burnout. As your company starts to scale, you are going to use more CPU; you’re going to use more memory. Tying alerts to computer vitals is not a good sign.

    If I run, my heart will beat faster—it’s just doing its job. Paging people at 2:00 a.m. because disc capacity is at 80% even though you won’t run out of space until next month is a good way to lose great teammates. I have worked with people who left their previous companies strictly because they got paged too many times for stuff that didn’t matter.

    This is why you need to alert on a degraded experience with the service and not much else. A CPU burning hot at 90% is not necessarily a bad thing, but you need context to decide. Create SLOs that are tied to customer experience and alert on those. People experiencing problems with the service is the only thing you should alert on—for the most part.

    One of the best ways I’ve seen to think about this came from Soundcloud developers, who explained that you should alert on symptoms, not causes. My fast heartbeat is not necessarily a problem. But if my elevated heart rate leads to lightheadedness and I fall—that’s a problem. So, I need to be able to alert on something like that. Paging people at 2:00 a.m. because disc capacity is at 80% and you won’t run out until next month is not good. But paging people because you know that disc capacity problems cascade into other, systemic problems is. You can apply the same thought to other potential causes of an outage. Paging alerts that wake you up in the night should only be based on symptoms.

  • The Key Benefits of Observability for SREs

    The Key Benefits of Observability for SREs

    In today’s technology landscape, organizations strive to champion innovative ideas, techniques and technologies to achieve success and outshine their competitors. For this reason, site reliability engineering (SRE) has become one of the fastest-growing enterprise roles and a set of organizational practices for fast and reliable software delivery. 

    SREs use various tools and practices at their disposal to manage services at scale, such as observability. Observability is the ability to infer a system’s internal state(s). It provides actionable insights into when errors occur within a system and, more importantly, why they occur. For SREs, this actionable data can be important to providing secure and reliable applications. 

    To gain more insights into the benefits of observability for SREs, I asked SKILup Day participants and DevOps Institute Ambassadors to weigh in. Here’s what they shared: 

    Sponsor, Priya Satheesh, CEO, Instana 

    “As their name implies, site reliability engineers are tasked with keeping the applications, architectures and websites up and available. Observability solutions put alerts in front of them at the first indications of trouble, with proper context and information so they can take action before issues become incidents and start affecting their customers. In addition, AIOps and machine learning help them to predict events and standardize responses for common occurrences to reduce response and repair times.”

    Helen Beal, chief ambassador, DevOps Institute 

    “Embracing observability supports SREs’ goals by:

    • Reducing the toil associated with incident management—particularly around cause analysis—improving uptime and MTTR.
    • Providing a platform for inspecting and adapting according to SLOs and ultimately improving teams’ ability to meet them.
    • Offering a potential solution to improve when SLOs are not met and error budgets are overspent.
    • Relieving team cognitive load when dealing with vast amounts of data–reducing burnout.
    • Releasing humans and teams from toil while improving productivity, innovation and the flow and delivery of value.
    • Supporting multifunctional, autonomous teams and the ‘We build it, we own it,’ DevOps mantra.
    • Completing the value stream cycle by providing insights around value outcomes that can be fed back into the innovation phase.”

    Mark Peters, technical lead, Novetta

    “True system observability means any process or event within any operational or development phase can be monitored by SREs. Systems with built-in observability provide SREs, as subject matter experts, the opportunity to examine events from different perspectives. The more observable the system, the more different places can be examined to help improve the overall flow. SREs can then create the feedback and design experiments to observe from different perspectives. If the event is showing functional difficulties and the only monitoring point is the input/output of the function and there is no process observability, SREs have to go back to the beginning to find events. If they can observe the entire process chain from commit to deployment, one can find the different areas. Think about the A/B cycle: Limited deployments are planned, and the load shifted gradually to ensure the new system can handle the increased functions. If the SRE cannot observe the function, then they cannot help improve reliability.”

    Ryan Sheldrake, field CTO, Lacework 

    “If you can ‘see’ or observe the complex and ever-changing environments that SREs are tasked with managing, creating and perhaps destroying, you can manage the associated risk posture associated with those environments and corresponding workload.“

    Vishnu Vasudevan, head of product engineering and development, Opsera

    “The top benefit for site reliability engineering (SRE) teams is the opportunity to work collaboratively with the business stakeholders and technology stakeholders to set goals and continuously improve. Based on these goals, SRE’s can create an approval process, which will help them understand where they are with respect to their own planning, security policies, quality metrics and operations practices. The ability to measure every release and verify the project helps improve speed to market and other factors once a robust observability practice is achieved. With DevOps orchestration across all the DevOps tools, applications being released and the different teams involved, SRE teams can then turn observability practices into a competitive advantage.”

    Sushant Mehta, senior manager, application development, Diyar United Company

    “Using observability, SREs can achieve a number of objectives in their day-to-day work, like identifying the root cause of production issues, faster resolution of issues and striving for self-healing infrastructure setup with no-code.”

    Tiffany Jachja, engineering manager, Vox Media

    “Every team that delivers software should be taking responsibility for their contributions to a software service. For SREs, having observability into an application means having the systems in place to help developers gain insights into how software applications function. Ultimately, it’s about building better software.”

    Parveen Arora, co-founder and director, VVnT SeQuor

    “SREs teams can leverage observability to get the following benefits:

    • To keep SLOs intact by detecting customer-affecting issues faster and rolling back before an issue affects the SLOs.
    • To have real-time health updates and transparency of information with regard to a service’s status. 
    • To create better workflows for debugging, optimizing workflows and resolving issues rapidly.
    • To simplify root cause analysis and investigation of hypotheses.”

    Jose Adan Ortiz, solutions engineer, Akamai Technologies

    “SREs are mainly concerned with the reliability of systems and usually monitor practices of toil, SLO, error, budget and stability to maintain systems working as expected.

    You could ask, ‘How can an SRE achieve such an integrated view of all systems components?’ The answer is observability. Without observability tools, it was impossible for an SRE to provide an effective response, fast RCA and analysis of data to improve SLOs.”

    Supratip Banerjee, solutions architect, Principal Global Services

    “It can provide several advantages, including:

    • Detecting customer-impacting errors faster and reverting before SLOs are broken.
    • Fostering transparency and delivering real-time updates on the status of a service. This allows SREs to be more productive and saves a lot of time.
    • Developing better methods for debugging, optimizing and quickly addressing issues.
    • Allows for feedback loops, which are crucial in the SRE role.
    • To avoid downtime, the relevance of observability rises in real-world production systems. There should be proper notification processes in place.”

    Maciek Jarosz, DevOps and process expert

    “The benefits are quite great, I’d say. Unless one is living under a rock with one local system, then it’s rather inevitable that there will be dependencies here and there. And if you’re an SRE, then I’d say that you’d rather know a bit more about dependencies from other branches of your system—or other systems altogether—than less.

    Learn more about observability and similar topics by registering for an upcoming SKILup Day. 

  • Site Reliability Engineers to the DevOps Fore

    Site Reliability Engineers to the DevOps Fore

    [pdf-embedder url=”https://devopsy.kinsta.cloud/wp-content/uploads/2022/02/site-reliability-engineers-to-the-devops-fore.pdf”]

  • Improving Resiliency by Creating Chaos

    Improving Resiliency by Creating Chaos

    In the digital economy, preventing downtime is paramount. When digital systems fail, the consequences for business can be huge. The cost of downtime can run to thousands of dollars per minute for large businesses. That’s without taking into account the impact of customer dissatisfaction and reputational damage for the company and the IT careers involved.

    No matter how you measure it, IT failure is costly. It’s also largely unavoidable due to the increasing complexity and interdependence of today’s distributed IT systems. The combination of cloud computing, microservices architectures and bare-metal infrastructure create many moving parts and potential points of failure, making those systems anything but predictable.

    That IT faults will occur is easy to acknowledge. Understanding how to fix them is harder, especially when interdependencies are not always obvious. Until recently, build testing has been the go-to method for assuring quality and resilience, but this kind of testing does not take into account environmental factors that cause unpredictable events or how cascading failures that lie dormant and unnoticed can trigger larger failures.

    How Chaos Engineering can Help

    Chaos engineering is a relatively new approach to enterprise software development and testing designed to eliminate that unpredictability.

    Introducing chaos into a system may sound counterintuitive if your end goal is to get clarity and improve resilience. Indeed, if you have heard anything about chaos engineering, you may have been alarmed at some of the terminologies: “blast radius,” “random terminations,” “fault injection” and “storms,” to name a few.

    In practice, chaos engineering is about performing controlled experiments in a distributed environment so that digital engineering teams can build confidence in the system’s ability to tolerate inevitable future failures.

    How it Works

    The process of chaos engineering involves stressing applications in testing or production environments by creating disruptive events in a controlled manner, such as server outages or API throttling. By observing how the system responds, improvements can be made before those weaknesses affect real customers.

    Experiments are meticulously planned from initial scoping to execution and the insights they deliver are far-reaching.

    Chaos planning starts with identifying the target deployment for the experiment. This process requires a comprehensive review of the application architecture and infrastructure components to first define what we call steady-state behavior. In other words, you need to understand what “normal” looks like before you start experimenting.

    You can then form a hypothesis about how the system will behave during the disruptive event. You’ll need buy-in from the business areas you are potentially disrupting and you’ll need to plan the parameters of the test carefully, reducing the scope if necessary.
    It’s a good idea to start small with chaos experiments. You’ll need to replicate them many times over in any given system to properly test its resiliency.

    Tooling for Experiments

    Luckily, many different tools already exist to help organizations implement and manage planned disruptions. Chaos engineering as we know it today originated back in 2010 from experiments conducted at Netflix using the tool Chaos Monkey, which still exists and is used today.

    Nowadays, there are many more chaos offerings available, including services from Microsoft Azure and AWS as well as Gremlin, ToxiProxy, Litmus and many more. Organizations can choose tools tailored to the size of their environment and decide just how automated they want the process to be. Tool selection will also depend on whether experiments are designed to test the system at an infrastructure, network or application level.

    Chaos Culture

    Chaos engineering is much more than a set of tools and rules. It involves adopting a culture in which teams trust each other and collaborate to build resiliency and advance innovation.

    When it comes to thinking about this culture shift, it can be helpful to think back to when DevOps was new. Sure, people would say they were using DevOps tools, but that did not necessarily mean they were actually practicing DevOps. DevOps involves breaking down siloes between different groups in an organization, creating an atmosphere of trust and enabling collaboration–some of the same attributes a chaos engineering culture needs to have.

    Why you Need Chaos Engineering

    Although chaos engineering sounds like a disruptive or uncontrolled exercise, it is actually the opposite.

    Chaos experiments require meticulous planning with an emphasis firmly on rooting out failures before they become outages. Far from lacking in control, chaos testing is a closely coordinated process and the organization retains a firm grip on everything from the speed at which testing happens to what components are tested. Chaos engineering doesn’t create problems, it reveals them.

  • Where Do SREs Go From Here?

    Where Do SREs Go From Here?

    Charlene O’Hanlon talks with Leo Vasiliou, director of product marketing at Catchpoint, about the results of a study the company fielded with VMWare Tanzu and DevOps Institute of nearly 300 site reliability engineers (SREs). This year’s report underscores the challenges of multi-cloud, calls out the underutilization of AIOps and shows a systemic shift in core baselining data. The video is below, followed by a transcript of the conversation.

    Announcer: This is Digital Anarchist.

    Charlene O’Hanlon: Hey, everybody. Welcome back to TechStrong TV. I’m Charlene O’Hanlon, and I’m here now with Leo Vasiliou who is the Director of Product Marketing over at Catchpoint. Leo, thanks so much for joining me today. Really appreciate it.

    Leo Vasiliou: Oh, of course, Charlene. Thank you very much for allowing us to have the conversation today.

    O’Hanlon: Absolutely, absolutely. I want to talk to you about a recent survey study that you guys put out in conjunction with VMware Tanzu and DevOps Institute. What were you guys looking at?

    Vasiliou: Well, from an existential perspective, we hope that the survey that we do leads into the annual report and we’d like for it to be the most data-backed set of insights in its community. There are sort of three underlying constructional essences of the report. One is to provide a core baselining data for the key SRE tenants. So DevOps, who had an amount of time, would call in reported levels of toil. The other essence is to look at what’s trending. And then the third one will generally be around the IT to business relationship. And then with each one of those constructional essences, the context and line items within each section will change from year to year.

    O’Hanlon: Gotcha. Alright. So this one focuses on site reliability engineering and what they’re seeing and what they’re dealing with on kind of a daily basis. Is that right?

    Vasiliou: Yes, yes.

    O’Hanlon: Okay. Alright, great. So tell me a little bit about some of the findings in the study. I know you guys do these, as you just said, reports yearly. So I’m interested in knowing what’s changed from year to year and what really surprised you in this year’s report.

    Vasiliou: Well, I like to say SREs are double struck with challenges because the SRE is essentially two jobs in one, right? They have to increase the efficiency of their operational activities while mitigating the risk of their transformational activities. So one of the key findings this year, which by the way this may be the most provocative dataset we’ve published just because some of the findings conflict with some popular theories in the market, but take the first one just as an example of that, is as part of the baselining set of data there was a self-reported drop in toil, which is the word they use to define repeated, mundane, busywork that doesn’t have any inherent value, of 15 percent. And one of the reasons this is such an insightful nugget to lead us into our various conspiracy theories is it was across the entire distribution of data, across all of their percentiles, as well as across all of the geographies that we looked at.

    O’Hanlon: Wow.

    Vasiliou: And so for me, when I had the blessed opportunity to do the research and sit down and co-write the report, having been stuck in the house for, let’s just call it a year and a half at this point, there was just something different about working on a piece like this and it just felt a bit more meaningful. And the conspiracy theory there or the hypothesis is that if people return to work, or return to the office I should say or some type of hybrid environment, will those reported levels of toil increase again next year as people adjust to the hustle and bustle of commutes just adding to frustration in how they think about things?

    O’Hanlon: That is a really, really fascinating thought process there. We could do an entire conversation about just that and whether what happens this year is going to be indicative of what happens in the future or if this year is going to be like a reset year. So from an SRE perspective though, the fact that the level of toil decreased 15 percent, that seems pretty significant. If you look at the entire spectrum of what they have to deal with on a daily basis, 15 percent, good lord, that’s huge in the SRE space.

    Vasiliou: Yeah. It is huge. I agree. It’s also based on the response of hundreds of survey participants year over year. And what’s interesting, Charlene, is one of the unique tactics we took in this year’s report is before we publish the reports and post survey pre report, we actually showed the results to various industry, let’s just call them, leaders. One of them we shared this particular section to had an alternative theory on why levels of toil dropped, and it had to do essentially with the fact that if people are not in the office, maybe does the very act of them not being observed compel them to do less busywork.

    So there was an alternative theory. But bringing us back to and leading into one of the other essences of the report, which is kind of the trends piece, it had to do with the amount of monitoring data sources, like the volume, variety, velocity of data, and the number of multi same-service platforms, e.g. multi CDN, multi DNS, et cetera, in use. And so will the return to work and the promise of AI ops help them keep toil levels low? Which leads us into the next sort of surprising finding, which is that SREs and their implementation or shift to AI ops is quite slow to maybe what we might be reading about in the industry, which was probably the second most provocative finding in the report.

    O’Hanlon: Yeah. So why do you think it is so slow then? Were there any trends that kind of came out of that? Any reasons why organizations are kind of slow to adopt AI ops?

    Vasiliou: Well, there is obviously a deluge of information and rhetoric in the market on the internet today. We need to seriously consider this idea of an information diet, which is a concept I first heard about a couple years ago. When we think about the idea of the hype cycle versus the challenges SREs face on a day-to-day basis, sometimes the market and the promise of what these types of solutions will offer might outpace actual implementation capabilities from, hey, we’re an SRE, let’s go implement this. I think part of the challenge here is probably the concept of AI ops is too broad and too large for consumption. It’s the idea of just saying, “Hey, let’s go solve world hunger or, hey, let’s go solve or implement AI ops,” right?

    O’Hanlon: Right, right.

    Vasiliou: You’ve got to break it down into smaller components and incrementally develop capabilities from there. Otherwise, how the heck are you going to baseline your progress for when you’re done? I think it’s too big, too broad, and some of the vendor promises may be of self-healing, auto remediation, maybe they were a little more on the hype side, where some of the other AI ops of components like root cause analysis or even correlation, maybe those are truthfully more on the hope and promise side.

    O’Hanlon: Okay. Do you think that AI ops – to your point, it is broad – do you think it just means different things to different people and so somebody actually may be implementing some type of AI ops and actually not realizing it?

    Vasiliou: The short answer is yes. So when we asked the question what monitoring tools or practices are used today, they were presented with seven options, ranging from application, infrastructure, experience monitoring, AI ops, and it was the fifth lowest, meaning that almost 40 percent of respondents said they’d never used it in the first place. The second question we asked is very simply what is the received value from AI ops, where one was the low value, nine was the high, and the range evenly spanned the value scale that was presented in the survey. What was interesting is there were a subset of people who said they never used AI ops but that they still received value, which was an additional dimension we didn’t think about when writing the survey.

    But it kind of brings me back to the essence of your question, which is are you user versus a consumer? And then go back to the previous comments about how big and broad this quote/unquote AI ops idea is, and there’s absolutely a propensity that people hear and think completely different things, unfortunately.

    O’Hanlon: Interesting. Interesting. So one of the things that we’ve been talking about in the SRE space over the last couple years is really how the role is kind of morphing and changing along with the market in general and to address a lot of organizational needs as well. Are there any findings from this year’s study that kind of bear out how an SRE role is changing and even if it is changing?

    Vasiliou: That will go back to what we were saying just a moment ago about mitigating the risk of their transformational activities. When you were asking the question, I’ll just cite the stat from our very first SRE report. We said that 85 percent of respondents said that in order to be successful in this role you have to commute into an office. So that got topsy-turvy turned upside down on its head.

    O’Hanlon: Right. Yeah. [Laughs]

    Vasiliou: But to get back to the essence of the question here, when we looked at the quantity of data sources and the prolific rise of the use of multiplatform providers, we sort of looked at this correlation between how SRE orgs are structured. Are you decentralized, are you centralized, or is there some type of hybrid in between? And then the decentralized piece was broken down even further by stack component, by platform component, or by business product or service. The vast majority of SREs said they are currently decentralized by business product or service, but I think the transformational piece here, which is something we’ll look at next year when we write the report, is because of the prolific rise of multiple third party and multiple same-service third party providers, there is a huge, gigantic spotlight to shift the primary dimension of org structure from product and service to a platform basis.

    And we talk about it like this. If you’ve got 10 different teams assigned to 10 different products or services and one member from each of those SRE teams developing capabilities that happen to be the exact same capabilities in development by those other nine teams, that’s the scalability and efficiency _____ that we start to lead in and talk about in this year’s report. So I think the transformational piece will be the assignment of SRE development engineering resources from product and service as a primary dimension to a platform – for example, cloud, CDN, DNS, API – as the primary dimension where those capabilities will be normalized and developed. Even though underlying platform structures will be different, those capabilities will then be offered as part of, let’s call it, an internal marketplace for use by multiple teams to kind of get away from that concept of silos and being segmented by product or service. So a bit of a long answer but all contextual to the key part there, which is the trend and the transformational piece that we wrote about this year.

     O’Hanlon: Right. Yeah, well it really is food for thought there if you consider how the SRE role was established and how it’s kind of – if you go down that path, how it seems to be changing and evolving as the more technology is adopted and more things are moved out to the edge and the need for SREs at basically all points on the network if you think about it. So great stuff there. Well, Leo, I know we could talk about all of the findings and have a much larger conversation, but unfortunately that’s all the time that we have for this one. But I do want to thank you very much for going through some of the survey findings. If folks are interested in taking a look at the study itself, is it readily available?

    Vasiliou: It is, it is. So it’s available at Catchpoint.com. Just head over to our learning and research section and enjoy the read. Charlene, one of the sentiments we wanted to leave readers with this year was this concept of baselining against Google’s definition of SRE, but we hereby declare by the power vested in us by absolutely nobody at all that you are hereby free to be an SRE.

    O’Hanlon: Oh, wow. Just one more thing to add to my plate. [Laughs] Well Leo, thanks again. I really do appreciate your time, and it’s a pleasure talking to you. I’m sure that we’ll talk again in the future about future studies that you guys have going on. So, thanks again. Really appreciate it.

    Vasiliou: You are quite welcome. Thank you for your time, Charlene.

    O’Hanlon: Alright, everybody. Please stick around. We’ve got lost more TechStrong TV coming up, so stay tuned.

    [End of Audio]

  • SREs Say AIOps Doesn’t Live Up to the Hype

    SREs Say AIOps Doesn’t Live Up to the Hype

    What can you expect when investing in artificial intelligence for IT operations (AIOps)? Real-time visibility across huge volumes of information. Lightning-fast event correlation and anomaly detection. Automated remediation and self-healing, without Ops personnel having to lift a finger.

    It all sounds amazing—at least, if you believe the marketing hype. But do those kinds of AIOps benefits actually translate to the real world? According to a recent survey of nearly 300 site reliability engineers (SREs), not so much.

    Among SREs surveyed—including engineers for some of the world’s largest, most innovative digital companies—just 12% use AIOps as a day-to-day part of their monitoring toolkit. Nearly 40% never use it at all.

    Why aren’t current AIOps solutions living up to the hype? And what can this gap tell us about the state of IT monitoring in general? Let’s take a closer look.

    Sprawling Complexity

    In theory, AIOps should be a perfect fit for SREs. These are, after all, the people with one foot in the world of operations, one in development, whose job is to make distributed software systems ultra-scalable and reliable. A solution that can turn mountains of data about those systems into actionable insights should be exactly what SREs need. And, in fact, 53% of SREs cite lack of unified visibility across the stack as their biggest cloud application monitoring challenge.

    And yet, if we gauge by utilization, today’s AIOps tools don’t seem to be delivering on that promise. Asked about the value of AIOps, just 7.5% of SREs, on average, view AIOps as “high-value”—even less than those SREs (9%) who perceive its value as low.

    Why is AIOps so slow to catch on? Ultimately, the barriers facing these tools are the same as those facing human engineers: massive and growing complexity in IT environments. As digital products become more dependent on third-party cloud services, as the number of things businesses want to track grows (from infrastructure to application to experience), the sheer volume, velocity and variety of monitoring data has exploded.

    Diverse Data Sources

    Compounding the problem, enterprises increasingly rely on multiple “same-service” providers for IT services. That is, they use multiple cloud providers, multiple DNS providers, multiple API providers, etc. There are sound business reasons for doing so, such as adding resiliency and drawing on different vendors’ strengths in different areas. But even when two providers are doing basically the same thing, they use different interfaces and instrumentation, and their data sources often employ different metrics, data structures, and taxonomies.

    Whether you’re asking a human being or an AI-driven tool to solve this problem, this heterogeneity makes it extremely difficult to visualize the complete picture across the infrastructure. It also creates gray areas around how best to take advantage of each vendor’s different rules and toolsets. All of which makes automating anything beyond the most superficial operations tasks hugely challenging, if it’s possible at all.

    Optimizing AIOps

    AIOps really is the solution to this problem—or at least, an important part of the solution. Machine learning is, after all, tailor-made to analyze large volumes of observational and engagement data and react in real-time, as well as to continually identify opportunities for improvement. But before we can get there, we need to change the way we think about AIOps.

    Don’t try to do everything at once.

    It would be great if we could deploy AIOps, let it learn about our environment and watch it magically start remediating issues on its own. But we’re not there yet. (And if a vendor tries to convince you we are, you should definitely be skeptical.) If we break down our expectations into smaller, more manageable chunks, however, AIOps can accomplish quite a lot.

    Event correlation, topology analytics and smart alerting, for example, are all areas where AIOps can provide significant value right now. For these more limited objectives, AIOps can absolutely reduce tedious manual effort for human operators, reduce mean time to repair, improve visibility and more, and digital businesses should be leaning on AIOps to do that.

    Invest in AI and ML training.

    Part of the reason adoption of AIOps tools remains sluggish stems from the fact that many teams still just don’t fully understand how they can use them. The more SREs are exposed to AI/ML training, the more opportunities they will identify for concrete, meaningful applications of AIOps. For example, only 27% of SREs said, “Experimenting or receiving training to expand knowledge or skills” was a major activity as part of their job.

    Embrace “Platform Ops.”

    The biggest step we can take is to rethink how we use AIOps within an organization. Today, AIOps tools—like SRE resources themselves—are often dedicated to a specific digital product or service. As a result, enterprises end up trying to apply their AIOps tools, and the SREs using them, to support everything associated with that product (DNS, API, CDN, cloud monitoring, etc). As IT environments continue to get even more complex, though, businesses will likely find that shifting to shared platform tools and teams yields better results.

    Instead of trying to apply AIOps across every part of a digital service, imagine applying it to one part of that service (for instance, focusing on just DNS, or just APIs). In this way, the AIOps tools, as well as the teams using them, gain in-depth expertise across all the different interfaces and instruments used for that function and get continually better at normalizing multi-provider data sources across them. At that point, this platform-specific AIOps capability can then be scaled and reused across the company—providing more advanced DNS AIOps (or API, or CDN, etc.) for multiple digital products and teams.

    Now, you’ve got an advanced, highly specialized toolset for optimizing an important platform function, regardless of which digital product happens to be using it. You’ve narrowed the scope of how you’re using a given AIOps implementation while going much deeper into that function than a more generic, broad-based implementation could achieve. All of a sudden, even more advanced automation and self-healing becomes possible. And the loftiest marketing claims of AIOps start to look less like hype and more like viable real-world outcomes for your business.

  • Code Ownership Is Key to Accelerate Debugging

    Code Ownership Is Key to Accelerate Debugging

    App stability is a fundamental part of every app experience. Broadly speaking, app stability is a measurement of the number of total app sessions that are crash-free or the percentage of daily active users who do not experience an error. End users have little patience for apps that crash or fail. Every application, whether it’s B2B- or B2C-oriented, suffers some level of instability and fails occasionally. Ideally, organizations should aim for app stability scores close to 99.999%. To achieve that quality of performance, software engineers must be fast and efficient in fixing coding errors that are frequently the cause of app crashes.

    But that’s a major challenge using the traditional approach to debugging. Most major applications have a wide variety of engineers, including completely separate engineering teams, all working from a single code base. Each team typically uses their own unique issue tracker to monitor errors, such as Jira or Asana. In addition, teams typically use their own channel in a collaboration platform to communicate internally, like Slack or Microsoft Teams. This causes silos between the various engineering teams, limiting communication and visibility between each team.

    When an error occurs, all the engineering teams are alerted about the bug through their own issue trackers. From there, they figure out where the error occurred in the code and who is specifically responsible for fixing it. This requires each team to manually review code as they try to locate the bug, which wastes precious time as engineers duplicate one another’s efforts. Because teams are using different communication channels, there’s little to no collaboration or alignment between the various engineering teams during this process. Overall, this approach to debugging is a cumbersome and inefficient undertaking that slows the process of repairing apps, ultimately hurting the end user experience.

    Consider a real-world example. Suppose your team is one of a dozen engineering teams, all working together on an application that creates a mobile game. Perhaps your team works on maintaining the in-app purchases feature set, while another works on the account setup flow and yet another team works on character customization features. If a bug suddenly crashes the app for a significant portion of users (resulting in a clear drop in the app stability score), every team will receive an error alert through their issue tracker notifying them that a coding issue has caused the application to fail. No one will know if their team’s code was responsible. As a result, each team is forced to go through the laborious process outlined above. If it turns out that the bug was in the character customization code, then the teams responsible for in-app purchases and account setup have wasted valuable time and resources searching for the error.

    The only team that should be notified of a bug is the one responsible for the part of the code a bug originates from. The emerging concept of code ownership makes this possible, providing a faster and easier approach to debugging. With this method, organizations create code owner files to determine which engineers or teams are responsible for each section of code, feature flags or experiments. If section X within the code fails, then team X will be responsible for addressing it. In the example above, only the team responsible for the code associated with character customization would need to find and fix the bug.

    This makes it much easier to respond to application issues and allocate the correct resources to remedy those issues. For end users, code ownership means that malfunctioning apps will be fixed faster, improving the customer experience and boosting business metrics. From an internal POV, engineering teams become far more cost- and time-efficient, freeing developers to take on new projects they actually enjoy working on.

    A culture of code ownership also helps engineering teams adopt other best practices for debugging. One major example is error prioritization. Not all bugs cause significant app instability and need to be fixed right away. When a team can ignore errors unrelated to their code, they have more time to fully hone in on and evaluate the errors they are responsible for. With this focused approach, they can establish metrics determining if they need to fix a bug immediately or if it can wait.

    Simply put, code ownership eliminates the confusion and guesswork associated with the process of debugging, allowing engineering teams to easily identify, own, prioritize and remedy bugs. Embracing code ownership is critical for companies to improve app stability, which is the most fundamental measure of an app’s health. This ties directly into business outcomes, because app stability is closely associated with customer retention, customer growth, sales and revenue. By speeding up debugging efforts, code ownership has a pronounced, positive impact on stability, boosting each of those metrics.