devopsy.kinsta.cloud

Tag: SLOs

  • The Hidden Configuration Tax Affecting Uptime SLO

    The Hidden Configuration Tax Affecting Uptime SLO

    This article is a preview of a talk by Greg Arnette for SLOconf 2023 on May 15 – 18. To watch this talk and many more like it, register for free at sloconf.com.

    Tax season was just a few weeks ago, and that got me thinking about how frustrating it is to get hit with a surprise tax bill and how that can relate to uptime: Configuration sprawl. DevOps practitioners constantly strive to maintain uptime service level objectives (SLO) while working with complex and ever-evolving infrastructure and applications. Configuration sprawl is a significant challenge many teams face, negatively impacting system performance and reliability.

    My company’s recent survey of 900 DevOps and platform engineering leaders revealed a startling (but not too surprising) conclusion: Most teams struggle managing secrets and configs at scale for infrastructure and applications.

    Where Does Configuration Sprawl Come From?

    As organizations scale their operations, the number of configuration files and settings grows, making it increasingly difficult to manage and maintain consistency across the board. They may have multiple environments, such as development, staging and production environments, often requiring different configurations, leading to duplicated settings and increased complexity.

    Configuration sprawl may also lack standardization as teams adopt different naming conventions, file formats and storage locations without enforced configuration standards, resulting in fragmentation and confusion. Relying on manual processes for configuration management increases the likelihood of errors and inconsistencies.

    Grinding to a Halt

    So you’ve got a lot of configuration–why should you care? Sure, it means more files to manage, but does it really impact your environment? There are a few places where configuration sprawl will negatively impact you. You might see an increase in error rates because as complexity grows, the likelihood of errors and misconfigurations increases, leading to system instability and downtime.

    Increased configuration complexity also means slower troubleshooting. Identifying the root cause of issues becomes more challenging and time-consuming when configurations are disorganized and fragmented. Ultimately this means less agility as you are forced to slow the deployment of new features and updates, impeding the organization’s ability to respond to market demands.

    How to Overcome Configuration Sprawl

    Don’t worry–you can tackle configuration sprawl, keep your SLOs on track and reduce the efforts required to manage and maintain configurations. To mitigate the harmful effects of configuration sprawl, consider adopting these best practices:

    Don’t repeat yourself (DRY)
    DRY config means variable names are declared once. Then flexible variable values are injected across build, deploy and runtime phases as defaults, inheritances and overrides. DRY avoids errors by allowing single changes to propagate where they are needed.

    DRY solves the common problem of forgetting to update critical secrets or parameters and failing to synchronize related changes (e.g., client and server port numbers or rotating a database password.)

    Decouple and abstract
    Decoupling and abstracting config means externalizing config from source code, separating how code and configuration interact, and isolating the specific values (log level) from the configuration interface (how the log level value is fetched.)

    This approach allows the same code to run in different environments using parameterized config. Abstracting config supports a best practice of implementing independent “code and config change” life cycles. Teams that frequently need to change config independently from code should decouple and abstract.

    Centralized config
    Consolidate configuration complexity into a single location because it’s easier to manage complexity within a single location versus multiple locations. Syncing config to the edge (where it is consumed) mitigates downtime. For example, it’s easier to fetch the configuration for service X versus coding all the logic needed to generate the configuration for service X.

    Standardize your configuration management by enforcing consistent naming conventions, file formats and storage locations across teams to reduce confusion and streamline configuration management. Centralizing the configuration and generating that configuration with transformations allows all the places that consume the configuration to be simple. Simple equals easy to understand and troubleshoot.

    Version control & monitoring
    Use a version control system like Git to track config changes to configuration files, making identifying and resolving issues easier. And continue to monitor changes and identify potential problems before they impact system performance.

    Train your team
    Ensure all team members know your best practices and patterns for configuration management. The proliferation of microservices and teams operating in parallel, establishing their unique naming schemes and patterns, exacerbates this problem for DevOps teams.

    Cutting the Uptime Tax

    By understanding the causes of configuration sprawl and implementing best practices to manage and reduce its impact, teams can improve their uptime SLOs and maintain a more reliable, high-performing infrastructure. Remember, investing in the proper tools and processes for configuration management is not only a best practice but also a critical factor in ensuring the long-term success of your DevOps initiatives. Those are tax cuts I can get behind.

  • Implementing SLOs-as-Code: A Case Study

    Implementing SLOs-as-Code: A Case Study

    This article was co-authored by Stephan Lips, Devin Cunningham and Justin Hoang of Procore and is a preview of their talk for SLOconf 2023 on May 15-18, 2023. To watch this talk and many more like it, register for free at sloconf.com.

    By managing service level objectives (SLOs) as code, we can co-locate SLO definitions and ownership with the product code and team. This supports horizontal scaling of SLO ownership while establishing a single source of truth and adding transparency, integrating with the code management process, and creating an audit trail for SLOs.

    When you measure the reliability of hundreds or even thousands of products and services across an enterprise, ownership of SLOs should not reside with a single team. There are a few aspects that make scaling SLO ownership horizontally across teams much more efficient: A standardized SLO management platform, process architecture and automation across the enterprise. Managing SLOs-as-code alongside product code is a step towards 360° product ownership and enables us to automate SLO updates via continuous integration.

    Objectives and Architecture

    The larger objective behind the SLOs-as-code approach is to move towards 360° product ownership. At Procore, teams own their product’s code and processes, including aspects like testing, performance, reliability, automated deployment pipelines and deployment configurations. Adding SLO ownership to a team’s portfolio will also help improve product reliability.

    Implementing SLOs-as-code involves three architectural components, which will be discussed in detail in their own respective sections. In summary, these components are:

    • SLO objects and definitions: Standardized SLOs and related resources facilitate centralized management and automation.
    • Automation: SLOs as code are our single and reproducible source of truth, which we automate via Continuous Integration.
    • A common platform: Nobl9 is the SLO management platform upon which SLOs, related resources, and automation are built. It aggregates the various data streams that power SLIs.

    SLO Objects and Definitions

    Procore’s Observability team has designed our SLO-as-code approach to scale with Procore’s growing number of teams and services. Choosing YAML as the source of truth allows Procore a scalable approach for the company through centralized automation. Following the examples put forth by openslo.com and embracing a ubiquitous language like YAML helps avoid adding the complexities of Terraform for development teams and is easier to embed in every team’s directories. We used a GitOps approach to infrastructure-as-code (IaC) to create and maintain our Nobl9 resources.

    The Nobl9 resources can be defined as YAML configuration (config) files. In particular, one can declaratively define a resource’s properties (in the config file) and have a tool read and process that into a live and running resource. It’s important to draw a distinction between the resource and its configuration, as we’ll be discussing both throughout this article. All resources, from projects (the primary grouping of resources in Nobl9) to SLOs, can be defined through YAML.

    Figure 2. A simple visualization of Nobl9 object relationships. A Project is a top-level object. Projects contain Services, SLOs are attached to Services, and SLOs can trigger alerts via project-scoped Alert Policies. Finally, Role Bindings define who has access to the Project contents.

    Procore adopted a hybrid approach to organizing our Nobl9 configuration so that the observability team can review systemic changes while teams still own changes to their service SLOs. A separate central repository is the source of truth for all other Nobl9 configurations, such as Project and Alert Policy definitions. As projects have a one-to-many relationship with their services, it would quickly become a game of “guess where” if the project configuration were defined within one of its service repositories. The central repository is owned by the observability team and allows us to manage permissions through pull request reviews submitted by product teams for their non-SLO Nobl9 resources. Once requests are merged, our automation applies the changes. The SLO definitions are co-located with the service’s code. Teams self-regulate these resources to maintain their SLOs internally.

    Automation

    Automating the creation and modification of Nobl9 resources is essential to Procore — it makes iterating on the deployment process quick and painless for our engineering teams. Automation removes human error and possible complications that usually come along with manually applying config files via CLI tools.

    With that in mind, the observability team created a CI job/workflow that our engineering teams copy into their project repo during their SLOs-as-code onboarding. Procore’s CI job uses the Nobl9 sloctl docker image, so we do not have to install the sloctl CLI tool in our CI containers. We configured this job to only apply the configs that have been added or updated, which helps to future-proof our pipeline as we scale the number of our SLO configs.

    Figure 3: SLOs-as-code Automation Workflow

    This requires the following steps in the workflow:

    1. An engineer creates or modifies a config and commits the changes to GitHub.
    2. The engineer opens a PR, gets reviews and merges their changes into the main branch.
    3. CI picks up the changes in GitHub and kicks off the following steps:
      1. Using git diff and regex pattern matching, we figure out what configs have been added or changed from the most recent merge to main:

    #!/bin/bash
    ADDED_OR_CHANGED_CONFIGS=$(git diff –name-only –diff-filter=d HEAD^ HEAD | { grep -E “^${NOBL9_RESOURCE_DIR}\/.*\.y[a]?ml$” || :; })

    2. We loop over the added/changed configs and run sloctl apply using the nobl9/sloctl docker image, which updates our resources in Nobl9:

    #!/bin/bash
    for file in “$ADDED_OR_CHANGED_CONFIGS”
    do
      docker run nobl9/sloctl:v0.0.80 apply -f “$file”
    done

    3. We use the same workflow in our centralized IaC repo that manages our projects, services, role-binding and alert policy configs.

    Common Platform

    Our platform requires another key feature for SLOs-as-code to work: We need to be able to consolidate different observability data sources and telemetry streams that power SLIs. This platform needs to support standardized data formats to define SLOs and related entities, such as alert policies for different conditions and trends, while allowing for the management of these formats via automation. We evaluated three vendors that offered SLO solutions, and only Nobl9 met all our requirements, which extended beyond the SLOs-as-code topic of this article.

    Observations

    Adopting SLOs-as-code can be impeded by three particular barriers of entry: New tooling (Nobl9 platform), data and format (SLO configs and resources) and process. To drive adoption, we have designed a self-service model supported by an open-door consulting practice. While this is an ongoing learning process for all involved, we are seeing early signs of success of an approach that uses:

    • Detailed, step-by-step instructions, including code snippets and templates.
    • Screencast tutorials supplementing the instructions.
    • Initial face-to-face, white-glove meetings to explain concepts and provide demos. The personal experience and conversation in our experience created considerable enthusiasm and goodwill among our teams. It proved to be a much better approach than just pointing them to documentation.
    • Fast feedback cycles with onboarding teams to improve instructions.

    Quite frequently, teams stated their primary challenges were not the technical aspects of SLOs-as-code discussed in this article, but how to design and improve on meaningful SLOs. We addressed this by authoring articles like Black Box SLIs and other internal resources for guidance. While posts can be a useful tool, we found that working with teams and discussing how their product impacts the user experience was more beneficial to modeling their SLOs using observability data.

    Last but not least, we are integrating with our embedded SRE (ESRE) team to help onboard more services. When a product team engages with ESRE on their deployment strategy and configuration, ESRE also discusses onboarding to SLOs-as-code and basic SLOs, such as error rate or duration.

  • Nobl9 Adds Service Level Analyzer to SLO Management Platform

    Nobl9 Adds Service Level Analyzer to SLO Management Platform

    Nobl9 today added a free tool for analyzing service level objectives (SLOs) to its platform to enable DevOps team to more accurately set reliability goals.

    Brian Singer, chief product officer for Nobl9, said Service Level Analyzer collects metrics and observability data from more than 24 DevOps tools and platforms that can then be used to create an SLO via a single click.

    Included as part of the free tier of the Nobl9 service, Service Level Analyzer also enables DevOps teams to more easily update SLOs as error budgets and error budget burn rates change over time based on actual incidents that have occurred, added Singer.

    DevOps teams can also perform “what-if” analysis to test out the impact of different targets and thresholds on error budgets, he added. That capability ultimately reduces costs because DevOps teams become less likely to overprovision resources to ensure application performance and resiliency, noted Singer. In effect, Service Level Analyzer is providing DevOps teams with an observability tool that spans the entire software development life cycle, said Singer.

    Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing SLOs across a DevOps workflow. SLOs, of course, are not a new idea. They have been employed as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications because they have many more dependencies than legacy monolithic applications.

    Ultimately, each DevOps team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services. SLO-as-code is intended to make it simpler to gather the metrics that confirm whether service levels are being achieved. As organizations become more dependent on software to drive digital processes, those SLOs are then tracked across everything from financial services to supply chains.

    In general, DevOps teams are more comfortable with the SLO concept, said Singer. The biggest challenge is making them simpler to implement, he added. The Service Level Analyzer lowers the bar for implementing an SLO by making historical data stored in DevOps tools and platforms more accessible, noted Singer. Rather than having to manually collect that data and then normalize it in an external repository, Nobl9’s Service Level Analyzer automates that process on behalf of a DevOps team using actual data versus relying on less precise statistical analysis, said Singer.

    As IT environments become more distributed, the number of dependencies that exist across application environments only continues to expand. IT teams can spend weeks trying to discover the root cause of issues that are negatively impacting application performance, only to discover that, for example, calls to an application performance interface (API) are being rerouted in a way that significantly increases latency. Tools that enable DevOps teams to more easily understand the relationships between all the services that make up an application environment are more crucial than ever.

  • How to Rethink Your Website Uptime

    How to Rethink Your Website Uptime

    COVID-19 rapidly accelerated e-commerce, almost shutting down in-person shopping during the lockdown. Ordering from Amazon became second nature. The online giant accounted for approximately 33% growth—50 million new Amazon Prime members.

    While there are concerns that consumer spending will fall short this year, according to Adobe’s 2022 holiday shopping predictions, online discounts are already playing a big part in creating record sales. Customers are concerned about their site’s availability. Are your stakeholders saying your service is running too slowly? Look at the latency metric and you may not see a change noted from the week before. With no real decrease in latency, your team is unaware of what “running too slow” means.

    Revisiting Technology Vendor Contracts

    Page latency is not new. Many retailers have increased their website conversion by improving page load time. The problem with some sites is that no one really knows what was defined as being fast enough and not too slow. And, of course, every stakeholder has different expectations of what the service should deliver. Internet traffic surged during COVID-19 and networks responded at different levels. While not unusual, the rapidity of growth was staggering. Globally, traffic in April 2020 grew 38% with upstream traffic skyrocketing up to 123.18% before leveling off.

    Service level objectives (SLOs) can create a common understanding of service capabilities to help define site requirements from the start. Unlike service level agreements (SLAs) which are engineering-driven, SLOs are a business decision that requires input from multiple stakeholders. SLOs allow retailers to match IT delivery models to business outcomes.

    Uptime is only one of an organization’s business objectives. First, service level indicators (SLIs) must be measured to identify users’ expectations of reliability. When you combine these individual data points into your SLO, then you’re set to meet your team’s business goals.

    SLOs for Existing and New Services

    Once your SLOs have been established, you should review all your existing and new services to make sure that your systems are running according to the objective. Moving from SLAs to SLOs isn’t simply a rebranding exercise.

    SLOs become the target for the availability of the system. Instead of looking at every individual data point, you can now look at the SLIs over a certain period — to establish an event and success criterion — and, for example, report that for 99% of them, the operation has less than five seconds to succeed, if that’s your goal. The SLO will indicate if a large portion of your SLI events were good. At 99%, there’s still a gap, but aiming for 100% reliability is unrealistic.

    No retailer can guarantee that their system will always be up and running 100% of the time. There are too many factors you can’t influence. And aiming for a higher number may not speak to your business case as it will contribute to unnecessary costs. Additionally, if your users have no expectations of the reliability of your service, it’s not prudent to make the investment. Your 1% gap, known as an error budget, is the number of negative events your system can withstand before violating your SLO—when negative events happen but are completely acceptable and fine.

    By using SLOs and error budgets, you will deliver less noise for engineers. Often engineers are paged too much which results in alert fatigue. You’ll have happier engineers who will not receive pages in the middle of the night, get fewer alerts and not have to deal with being alerted about small website spikes.

    Implementing SLOs

    While reliability is the single most important decision when making SLOs, teams are also faced with internal development challenges. Engineers would like to prioritize technical debt to rework existing services versus focusing on the business side demanding new features. When looking at SLOs, they should be based on your user expectations, not an arbitrary value.

    Undercutting the reliability of objectives also means that you’re not meeting user expectations. And not meeting user expectations will generate a direct business impact because unhappy users might move away from your service. Create this direct link between reserving tech debt and having a positive business impact by bringing reliability back up to the value which was defined. This approach will help prioritize between tech debt and new feature development.

    Getting started with SLOs for existing services is a good jumping-off point. Most likely, there will be information expectations as to what’s a service. Document the services and think about what users expect. You won’t have to deliver anything near 100% but be reassured that by making the informal formal, seeing the user perspective, and not trying to aim for perfection are areas to hone in on when delivering value for your SLO. Once defined, how often, what you want to achieve, and measuring the actual reliability of your service are the next logical steps to knowing how good you’re delivering in comparison to the objective defined.

  • Decidability: SLOs to Measure Quality Attributes

    Decidability: SLOs to Measure Quality Attributes

    You may be familiar with the “-ilities”–the quality attributes that define a system’s so-called “non-functional” parts. One that isn’t on the list is decidability, which is the ultimate quality attribute because it focuses on what truly matters to your team and your customers.

    You can’t do it all. Fast, cheap and good? Here in the real world, we have tradeoffs. Software is the mouth that keeps on eating, but it isn’t a digital free lunch. When we decide what not to do, we bring greater clarity and productivity to the team.

    Of course, the world is full of “shoulds.” We should make our product accessible, we know. It should be secure; it should be reliable. Of course. But how do you decide the limit of your shoulds? To enable decidability, you need context. To gain a contextual understanding of your most critical quality attributes, you must talk to real people and reason about your situation. There’s no other way.

    New Design, Wrong Decision

    I had an eye-opening experience that demonstrated firsthand how quality attributes don’t necessarily deliver business results. Six years ago, I was working on e-learning software for public schools. One of the main problems was user management. We were selling to massive institutions that often had only one administrator responsible for onboarding all the users across hundreds of schools. While school was out, they had to enter data for the schools, create classes and assign teachers.

    Once we had a design for improving the workflow (luckily, before implementing it), we took the mockups to a customer to show the fancy ideas we had built. They surprised us by showing us “what it looks like for them”–a spreadsheet! They were copying/pasting everything by hand.

    So we asked them, “What if we gave you a way to upload the spreadsheet? Would that work?” And, of course, the answer was yes! So we scrapped our plans to build a fancy form and made a drag-and-drop input field to validate and upload the sheet. The decision was clear.

    Measuring Decidability

    How do we measure the critical business drivers and turn them into decidability? The closest answer I’ve found is service level objectives (SLOs). These take the fuzzy -ilities and turn them into metrics and goals that can quickly inform decisions.

    SLOs are a universal tool for measuring quality attributes from an indicator expressed as a proportion of “good” events for a given quality like reliability or scalability. Decidability comes from aggregating the SLO data across your services, geographies, user groups and quality attributes over time.

    Comparing an indicator to a particular goal (tuned over time as the business and engineering context changes), we can derive an allotment of allowable “bad” time or events–an error budget. Setting an error budget for reliability, performance, scalability, etc., allows you to decide how poorly your service can operate and still keep customers happy. Further, you can find the breaking point of user behavior and defend your service proactively from violating the quality attributes you expect.

    Technical debt inevitably stems from all the -ilities you should focus on but couldn’t quite get to, given your situation. Some technical debt is deliberate and even prudent. To tackle technical debt, you must decide which new features to put on hold to make space in the roadmap. SLOs provide a data-driven baseline to make this decision quickly and accurately by telling you when your -ilities are within error budget or out of whack.

    Scaling Decidedly

    You might start on a small, single-purpose project that’s rarely used. But if that changes, you must decide how to invest, redesign and maintain it. Suddenly, you might care about performance, scalability, security and the rest. First, you need decidability so you can prioritize your work. So you gather the people together who care about the service, and you start to think and talk and stack rank. You use your instrumentation to build metrics, set clear goals and do what it takes to keep the system running within expectations.

    Automating action based on error budgets is the magic of setting SLOs. They aren’t just for sophisticated, highly available services. Error budgets are extremely valuable in systems when you aren’t sure what to do. SLOs create decidability in all the trade-offs of all the quality attributes and the functional needs of the app or service you’ve carefully built.

    If you put decidability first, the rest will fall into place.

  • Nobl9 Adds Free Tier to SaaS Platform for Managing SLOs

    Nobl9 Adds Free Tier to SaaS Platform for Managing SLOs

    Nobl9 this week announced it is making available a perpetual free tier of its software-as-a-service (SaaS) platform for managing service level objectives (SLOs).

    The free tier was announced at the AWS re:Invent 2022 conference and the company also revamped its pricing. Nobl9 Teams Edition (formerly Nobl9 Hydrogen) now includes 50 SLOs, up from 25, for $850/month. Nobl9 Enterprise Edition also has doubled the included SLOs from 75 to 150.

    Finally, Nobl9 also launched SLOcademy to help educate IT teams on how best to create and manage SLOs.

    The Nobl9 platform simplifies the creation and tracking of SLOs using metrics and observability data collected from more than 24 widely used DevOps tools including Prometheus as well as services provided by Datadog, New Relic, Dynatrace, Google, Amazon Web Services (AWS) and Splunk.

    Kit Merker, chief growth officer for Nobl9, said the company’s platform collects signals from logs, metrics and traces to define what normal should look like for any given service. The free tier of the platform has some limitations, but it does provide access to the full-featured platform to enable IT teams to get started managing SLOs.

    SLOs, of course, are not a new idea. They have been used as a metric to track the performance of IT services for decades. It’s not clear whether reliance on SLOs has waned over the years or if application environments simply became too complex to track meaningful metrics. However, as applications are increasingly viewed as services, it’s now only a matter of time before SLOs become more widely adopted across a modern application environment. The challenge, of course, is that defining an SLO is a lot easier than maintaining it—especially within today’s highly dynamic cloud-native application environments where services with lots of dependencies tend to come, change and go unexpectedly.

    Nobl9 is making a case for an SLO-as-code approach that makes it simpler to track service levels. A recent survey of more than 300 IT managers and executives conducted by Dimensional Research on behalf of Nobl9 found only 29% of respondents had no plans to implement SLOs. A full 94% of respondents that have or plan to implement SLOs intend to map them directly to business operations, with 91% reporting that they expect that effort to improve decision-making. More than 80% also said their organizations are planning to increase the use of SLOs, with 87% indicating SLOs should improve overall microservices performance.

    The challenge is that most organizations have limited visibility into their IT environments. The survey, for example, found less than half of respondents (46%) had visibility into all their IT environments. Only 45% and 35% claim to have visibility into containers and microservices, respectively. More than three-quarters (78%) said hybrid clouds make observability more difficult. Ironically, 45% of companies reported they already employed 11 or more observability and monitoring tools. On the plus side, 31% have hired site reliability engineers (SREs) while nearly half (46%) planned to create that role.

    It may be a while before most IT teams are organized around the concept of applications as services but as the management of IT continues to evolve it’s now just a matter of time before that approach becomes more the rule than the exception.

  • Nobl9 Simplifies SLO Identification

    Nobl9 Simplifies SLO Identification

    Nobl9 today added a Replay capability to its service level objectives (SLO) management platform that makes it simpler to collect the historical data a DevOps team needs to get started.

    Nobl9 COO Kit Merker said Nobl9 Replay makes service level indicator (SLI) data instantly available in minutes using 30 days of historical data collected from observability and application performance monitoring (APM) platforms. Previously, DevOps teams would have had to wait as long as a month before the Nobl9 platform could collect enough spreadsheet data to create a graph that identified the relationship between various services.

    Nobl9 now enables historical data to be merged with data collected in real-time to allow DevOps teams to view SLO charts and generate reports shortly after creating an SLO within a Git-based workflow.

    Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing them that has garnered support from Dynatrace, Sumo Logic and others. SLOs, of course, are not a new idea; they have been used as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications.

    The Nobl9 platform is designed to enable DevOps teams to employ an SLO-as-code approach that makes it simpler to gather the metrics that confirm whether SLOs are being achieved.

    It’s not clear whether reliance on SLOs has waned over the years or if application environments simply became too complex to track meaningful metrics. A recent survey conducted by Dimensional Research on behalf of Nobl9, however, found a full 94% of respondents that have or plan to implement SLOs intend to map them directly to business operations, with 91% reporting they expect that effort to improve decision-making. More than 80% also said their organizations are planning to increase the use of SLOs, with 87% indicating SLOs should improve overall microservices performance.

    The issue is that most organizations have limited visibility into their IT environments. The survey found less than half of respondents (46%) had visibility into all their IT environments. Only 45% and 35% claim to have visibility into containers and microservices, respectively. More than three-quarters (78%) said hybrid clouds make observability more difficult, even though 45% said they already employed 11 or more observability and monitoring tools. Trying to track SLOs using tools such as Grafana dashboards that DevOps teams need to set up and maintain is simply too difficult, noted Merker.

    As applications are increasingly constructed using services that drive digital business transformation initiatives, it’s now only a matter of time before SLOs are more widely tracked. That will enable DevOps teams to not only ascertain their current level of performance but also more easily work toward achieving their actual performance goals, he added.

    The challenge is that defining an SLO in a highly dynamic application environment where services are added and replaced with much greater frequency is often a lot easier than maintaining it.

  • How to Adopt an SRE Practice (When You’re not Google)

    How to Adopt an SRE Practice (When You’re not Google)

    Site reliability engineering (SRE) isn’t a new term or practice. The practice of applying software engineering skills and principles to operations problems and tasks happened even before site reliability engineer was a defined job title. But organizing a proactive approach to building and maintaining software drives long-term success in improving operational efficiency, data-driven roadmap planning and general uptime and reliability. All these advantages account for the widespread adoption of SRE. 

    In this article, I’ll dive into what it takes to get into site reliability engineering, how to adopt it within your own organization and some of the core principles and best practices you’ll need to keep in mind as you move forward in your SRE maturity journey.

    What is SRE?

    SRE, or site reliability engineering, is the practice of applying software engineering expertise to DevOps and operations problems. Often, this means proactively writing code and developing internal applications or services to combat reliability and performance concerns. SRE has been a practice for many years but has been more recently popularized by Google when they first published Site Reliability Engineering: How Google Runs Production Systems in March 2016.

    SRE is a complicated acronym because it both relates to a job title, site reliability engineer,  as well as a general practice adopted by development and IT teams, site reliability engineering. SRE teams are often organized differently depending on the organization and the services they support. Sometimes site reliability engineers (SREs) are sprinkled among development teams. This means they can vocalize concerns during roadmap planning and work closely with QA teams, collaborating on everything from shipping features to staging to managing production environments. 

    Another approach organizes an SRE team as a standalone group. The SRE team as a whole is focused on responsibilities including monitoring applications and infrastructure, establishing reliability metrics, tracking issues with new releases and handling on-call duties. No matter how they are organized, what remains the same for all these SREs is their initiative—reliability and performance.  

    DevOps Vs. SRE

    The rapid adoption of DevOps practices may have you wondering, “How is SRE different from DevOps?” Commonly, the two terms go hand-in-hand. SRE is a practice that doesn’t necessarily have to be part of DevOps culture and adoption but is often implemented by organizations that are further along in their DevOps journey. The further along on your DevOps maturity model, the more likely you are to have bought into SRE practices.

    The conversation isn’t ‘SRE versus DevOps; the conversation is about establishing a proactive SRE team and operating model to help strengthen your DevOps practice. Whether you choose an SRE framework integrated into your development and IT teams, or you want to make SRE a standalone business unit, you need to understand their general responsibilities.

    SRE Team Responsibilities

    Site reliability engineers are responsible for defining what performant and reliable really mean when you talk about your applications, services and infrastructure. The day-to-day tasks can range anywhere from instrumenting new monitoring solutions to building custom apps for technical support teams. They might ship new code to production to fix bugs or deliver new features, or they might respond to incidents in real-time and work closely with support teams to deliver positive customer experiences.

    At the end of the day, site reliability engineers are the key to understanding exactly how customers experience your product and how to track system performance and reliability through your customers’ eyes. SRE teams need to figure out exactly where service boundaries exist between what development teams ship and what customers experience. From there, they need to find ways to monitor reliability and performance concerns in a way that helps your internal teams proactively identify risks and deliver better software. 

    SRE teams spread knowledge across the product and development teams to consistently define reliability across the entire organization. With everyone on the same page, the engineering teams can make data-driven decisions when it comes to releasing new features or improving current experiences in production. 

    Site Reliability Engineering Principles and Practices

    At a very high level, Google defines the core of SRE principles and practices as an ability to ’embrace risk.’ Site reliability engineers balance the organizational need for constant innovation and delivery of new software with the reliability and performance of production environments. 

    The practice of SRE grows as the adoption of DevOps grows because they both help balance the sometimes opposing needs of the development and operations teams. Site reliability engineers inject processes into the CI/CD and software delivery workflows to improve performance and reliability but they will know when to sacrifice stability for speed. By working closely with DevOps teams to understand critical components of their applications and infrastructure, SREs can also learn the non-critical components.

    Creating transparency across all teams about the health of their applications and systems can help site reliability engineers determine a level of risk they can feel comfortable with. The level of desired service availability and acceptable performance issues that you can reasonably allow will depend on the type of service you support as well. SRE principles and practices embrace experimentation and require a dedication to proactively understanding the health of the services they support.

    An SRE Operating and Maturity Model

    You can perform a lot of the responsibilities asked of a site reliability engineer and still have a software engineer job title. So, how do you know how mature your site reliability engineering practice is? Luckily for you, we’ll lay out a quick way to build an effective SRE operating model and track your maturity against it. An SRE operating model typically includes three elements, which you might achieve in phases:

    1. A team (or at least one person) dedicated to the practice of SRE.
    2. Deep integration and influence across product, development and operations teams.
    3. Autonomy to automate workflows and write code for nearly any part of your application or system.

    Your SRE maturity depends on where your organization falls along these three elements of an SRE operating model. If you’ve taken the steps to make an SRE team or hire your first site reliability engineer, you’re at the beginning of your journey. If you have a team and they are a valued part of roadmap discussions, QA, deployment workflows, incident management processes, then you have a somewhat mature SRE practice. 

    An organization only reaches complete SRE maturity when the SRE business unit has the autonomy to automate workflows, build applications, own monitoring and alerting solutions, or interject themselves into nearly any conversation. Vocalizing performance and reliability concerns upfront and having a proactive discussion about those concerns is always better than simply ignoring them until it’s too late.

    Monitoring, CI/CD and Organizational Automation

    Site reliability engineers can and will automate just about anything and everything. If it can proactively detect, remediate or resolve an issue, it needs to be automated. From continuous integration and delivery practices to production environment monitoring, SREs should have some visibility into all of it. If they can identify ways to proactively uncover performance and reliability issues, then they need to have the authority to implement those changes.

    Today’s DevOps and IT capabilities around automation, monitoring, artificial intelligence and machine learning give SRE teams a huge advantage when identifying issues, responding to them and fixing them. Organizations with mature DevOps and SRE practices can catch problems in staging and they can also build automated incident management workflows and self-correcting systems. By determining critical components in your applications and infrastructure, SREs can narrow down the scope of things that can cause major problems.

    The Practice of Service Levels (SLIs, SLOs and SLAs)

    Service levels come into play to help SRE teams communicate the true health of digital products and services to all stakeholders. This is done by identifying and measuring critical components that are key to delivering positive customer experiences. In particular, they need to know when one or more components expose functionality to external customers. We call these intersection points system boundaries. System boundaries are the place where site reliability engineers need to apply service-level indicators and objectives to their metrics in order to tell the real story of system performance and reliability. 

    • Service-level indicators (SLIs) are the key measurements to determine the availability of a system.
    • Service-level objectives (SLOs) are goals you set for how much availability you expect out of a system.
    • Service-level agreements (SLAs) are the legal contracts that explain what happens if the system doesn’t meet its SLO.

    While SREs aren’t always responsible for managing service levels, it often falls within their purview. By tracking SLIs and tying them to SLOs, you can set goals around the performance of a system. Google’s SRE book defines the four golden signals of service levels as latency, traffic, errors and saturation. So, for example, you could look at an API call and track its number of successful/failed requests (the SLI) against a general percentage of requests that need to be successful for customers to have a good experience (the SLO).

    SRE teams will often set strict SLOs on critical components within their applications and services to better understand how strict of an SLA they can agree to with customers. From here, the team can apply error budgets as a way to understand how quickly they must resolve issues in order to stay compliant with their SLOs. Service levels allow teams to aggregate metrics and create a transparent view of uptime, performance, and reliability across the entire organization. At a glance, business leaders can use service levels to monitor compliance across multiple teams, applications, services, etc. to gain a comprehensive understanding of their system’s health.

    Adopting SRE Best Practices

    Adopting SRE best practices and principles won’t happen overnight. It takes time and effort to proactively monitor your teams and systems for performance and reliability concerns. But, in the end, your DevOps teams and especially your customers will thank you for deciding to take advantage of site reliability engineering.

  • Increasing Use of SLOs to Enable Observability

    Increasing Use of SLOs to Enable Observability

    Observability is a growing discipline among most IT and operations departments. To release stable software faster, operators need continuous visibility into metrics like performance, uptime and availability. As a result, engineers are increasing their use of service-level objectives (SLOs) across the board—a recent study found that 82% of companies are increasing their use of SLOs.

    SLOs enable deep visibility into specific applications’ performance and are often used by site reliability engineers (SREs) to ensure quality and avoid service interruptions. Not only that, but various teams are correlating SLOs to aspects of the business to reduce cost and help direct decision-making. But while many environments have visibility, there are still gaps present.

    Nobl9 recently released a global survey of IT professionals and executives that tracked the state of service-level objectives in 2022. Below, we’ll look at the key findings from the report and consider what they mean for the state of SLOs and observability in general.

    State of Observability and SRE

    In general, site reliability engineering is still maturing across organizations. Though only 31% of companies have adopted SRE, it’s slated for much future growth, as 46% said they plan to embrace SRE in the future.

    These operators are now faced with many cloud-native observability tools, which are producing a sea of data in the form of metrics, logs and traces. A full 39% of companies use anywhere from six to 10 observability and monitoring tools and 35% use more than 11.

    With so many tools now deployed, who is using this observability and monitoring data? Namely, at 74% of companies, observability data support operational requirements. Operations teams are the ones most likely to use SLOs to monitor uptime, performance and overall efficiency. After operations teams, security teams also use this observability data (71%), which makes sense as SLOs can inform incident response. Other areas that follow are customer support, compliance and capacity planning.

    One interesting finding is that only 42% of companies use SLOs for service-level agreement (SLA) adherence. This indicates that SLOs are most often applied to internal optimizations and decision-making.

    Hybrid Environments Complicate Visibility

    Companies are mostly tracking SLOs to increase their visibility into networks (83%), databases (76%) and applications (75%). Other top areas include private cloud environments and legacy computing arrangements. But although observability is trending, organizations still lack complete visibility across the entire stack.

    Of note is that nearly half (46%) of respondents said their monitoring and observability tools don’t provide full visibility into all of their company’s IT assets. For example, only 45% of companies have visibility into their containers and just 35% have visibility into their microservices architecture.

    A lack of full-stack visibility may be complicated due to rising hybrid and multi-cloud conditions, as 78% say a hybrid cloud environment makes monitoring infrastructure more difficult.

    Benefits of Tracking SLOs

    The report defined SLO as “a performance and availability target set for a given system, application [or] service over time.” And the benefits are readily apparent for organizations following such targets—they can help increase performances, direct decision-making and help avoid outages.

    As a result, 70% of companies are currently using SLOs in some fashion. Here are some benefits of tracking SLOs:

    • Increase microservice performance: 87% said using SLOs for microservices architecture would improve service performance.
    • Enable full-stack observability: 58% said some of their company SLOs are mapped to business operations.
    • Improve business decision-making: 91% agreed using SLOs can help drive better business decisions.
    • Prevent service interruption: 67% said their company prevented business interruptions thanks to SLO thresholds alerts.
    • Reduce expenses: 90% indicated SLOs saved their company money.

    Final Thoughts

    More teams are tracking service-level objectives than ever before. And most companies (71%) that aren’t using SLOs now plan on adopting them soon. The data demonstrated that the observability market is an evolving area with room to grow. And, the same can be said about the SRE role.

    Yet, it’s good to point out that not all companies will adopt precisely the same roles or monitoring procedures. Therefore, there will likely continue to be variance in who is using SLOs and how they are applied. Regardless, the study suggested that investigating SLOs has the potential to benefit the software life cycle in many ways.

    The study, “Adoption of SLOs Grow to Increase Visibility and Drive Business Improvements” surveyed 309 participants around the globe in various sectors. The study was sponsored by Nobl9 and conducted by Dimensional Research. For more information, you can download the report here.

  • Nobl9 Shares SLO-as-Code Methodology

    Nobl9 Shares SLO-as-Code Methodology

    Nobl9 has released version 1.0 of an open specification for defining service level objectives, dubbed OpenSLO, and, in addition, has defined a repeatable SLO methodology.

    Kit Merker, Nobl9 COO, said the Service Level Objective Development Lifecycle (SLODC) methodology represents an effort to create a set of best practices for achieving and maintaining SLOs that are implemented as code within an application environment.

    A survey of more than 300 IT managers and executives conducted by Dimensional Research on behalf of Nobl9 found only 29% of respondents had no plans to implement SLOs. A full 94% of respondents that have or plan to implement SLOs intend to map them directly to business operations, with 91% reporting they expect that effort to improve decision-making. More than 80% also said their organizations are planning to increase the use of SLOs, with 87% indicating SLOs should improve overall microservices performance.

    The challenge is that most organizations have limited visibility into their IT environments. The survey, for example, found less than half of respondents (46%) had visibility into all their IT environments. Only 45% and 35% claim to have visibility into containers and microservices, respectively. More than three-quarters (78%) said hybrid clouds make observability more difficult.

    Ironically, 45% of companies reported they already employed 11 or more observability and monitoring tools. On the plus side, 31% have hired site reliability engineers (SREs), while nearly half (46%) planned to create that role.

    Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing SLOs across a Git-based workflow. Since OpenSLO was initially launched, more capabilities have been added including a DataSource object that makes it easier to reuse connection details and which makes creating SLOs less verbose and three alerts have been defined: AlertCondition, AlertPolicy and AlertNotifcationTarget.

    Nobl9 is also releasing an OpenSLO-to-Nobl9 converter to transition OpenSLO YAML files into Nobl9 YAML when required.

    SLOs, of course, are not a new idea. They have been employed as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications. Ultimately, each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services. SLO-as-code is intended to make it simpler to gather the metrics that confirm whether service levels are being achieved. Those SLOs are then tracked and integrated across everything from financial operations to supply chains, noted Mercer.

    It’s not clear to whether reliance on SLOs has waned over the years or if application environments simply became too complex to track meaningful metrics. However, as applications are increasingly viewed as services, it’s now only a matter of time before SLOs become more widely adopted across a modern application environment. The challenge, of course, is that defining an SLO is a lot easier than maintaining it—especially within today’s highly dynamic application environments where services tend to come, change and eventually go unexpectedly.