devopsy.kinsta.cloud

Tag: Nobl9

  • Google Taps Nobl9 to Improve Services Reliability

    Google Taps Nobl9 to Improve Services Reliability

    Google announced this week it has added a capability for tracking reliability based on how service level objectives (SLOs) are being met that is based on a platform developed by Nobl9.

    Nobl9 launched a Nobl9 Reliability Center earlier this year that provides a single source of truth about the state of reliability in an application environment. Included in that platform is SLOgpt.ai, a generative artificial intelligence (AI) tool for programmatically creating SLOs and launching natural language queries to determine how well they are being adhered to across a distributed computing environment. The generative AI tool developed by Nobl9 is based on Vertex, a foundational large language model (LLM) provided by Google.

    Kit Merker, chief growth officer for Nobl9, said the generative AI tool also makes managing SLOs more accessible to a wider range of stakeholders.

    The expansion of the alliance with Google is the latest in a series that will make it simpler to manage SLOs across multiple ecosystems, including, most recently, partnerships that have been established with Microsoft and Cisco, noted Merker.

    The Nobl9 Reliability Center is already integrated with more than 50 DevOps, observability and incident management tools. It collects metrics, events, logs, traces and alerts to track incidents, releases, rollbacks, runbooks and other documentation. Those integrations will make it simpler for IT teams to create SLOs using historical data that would otherwise be challenging to define from scratch.

    In time, there is no doubt tools for managing SLOs will become more widely available, but given the dependencies that exist between applications and systems, DevOps teams will need an approach that can be employed across hybrid IT environments, said Merker.

    In general, as more microservices-based distributed applications are built and deployed, it’s becoming more challenging to maintain SLOs across them since they have many more dependencies than legacy monolithic applications. Each feature and application programming interface (API) added over time can adversely impact performance and reliability. The challenge is that it’s not so much that any given service will outright fail as much as it is determining where degradations are occurring that result in SLOs not being met.

    Each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services by making it simpler to track whether SLOs are achieved and maintained. Not every service needs to necessarily meet a stringent requirement, but IT teams do need to have confidence in the fact that, on average, certain levels of service are being consistently maintained.

    When it comes to SLOs, the challenge is first determining which dependencies might impact the level of reliability that can be maintained in the event of an outage and then, second, making sure whenever necessary workarounds—such as rerouting application programming interface (API) calls to other services—can be easily invoked.

    SLOs, of course, as a concept, have been around for decades, but with the rise of digital services that span multiple organizations, the level of reliability DevOps teams are expected to provide is now being written into business contracts. As a result, the overall level of stress related to managing SLOs is only going to increase in an era where businesses have never been more dependent on software.

  • Nobl9 Allies With Microsoft to Simplify SLO Creation

    Nobl9 Allies With Microsoft to Simplify SLO Creation

    Nobl9 this week revealed it has integrated its reliability center platform for managing service level objectives (SLOs) as code with the Microsoft Azure Monitor service to make it simpler to both create and track service levels.

    The Nobl9 Reliability Center provides access to dashboards for a single source of truth about the state of reliability in an application environment based on the metrics collected from the SLOs that have been defined.

    Nobl9 Chief Growth Officer Kit Merker said the integration with Microsoft Azure Monitor is the first in a series that will ultimately make it easier to define SLOs using historical data collected by third-party tools.

    The Nobl9 Reliability Center is already integrated with more than 50 DevOps, observability and incident management tools through which it collects metrics, events, logs, traces, alerts to track incidents, releases, rollbacks, runbooks and other documentation.

    Those integrations will ultimately make it simpler for IT teams to create SLOs using historical data that today are still challenging to define from scratch, noted Merker. In fact, the data tracked by Nobl9 can be used to automate remediations via those integrations, he noted.

    As more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications. Each feature and application programming interface (API) added over time conspires to adversely impact performance. It‘s not so much that any given service will outright fail as much as it is determining where degradations are occurring that result in SLOs not being met.

    Ultimately, each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services by making it simpler to track whether SLOs are achieved and maintained. Not every service needs to necessarily meet a stringent requirement. Instead, IT teams need to have confidence in the fact that, on average, certain levels of performance are being consistently maintained.

    That’s crucial because as more organizations depend on software to deliver digital services, any disruption is likely to have a direct impact on profit and revenue. As such, the reliability of application environments has become a major concern in an era where IT teams are now trying to manage distinct services rather than individual applications.

    The challenge and the opportunity now is to find a way to proactively reduce that stress by programmatically managing SLOs as code rather than having to overprovision infrastructure resources to ensure maximum availability at the highest cost possible. At a time when more organizations are encountering economic headwinds, that’s no longer a luxury they can afford.

    Each organization will need to determine for themselves how aggressively they need to manage SLOs. Of course, SLOs as a concept have been around for decades, but with the rise of digital services that span multiple organizations, the level of reliability DevOps teams are expected to provide is now being written into business contracts. Not surprisingly, the overall level of associated stress DevOps teams are experiencing has correspondingly increased.

  • Nobl9 Unfurls Reliability Center for Managing SLOs

    Nobl9 Unfurls Reliability Center for Managing SLOs

    Nobl9 today launched a next-generation platform for managing service level objectives (SLOs) in a way that promised to improve overall reliability.

    Brian Singer, chief product officer for Nobl9, said in addition to enabling DevOps teams to meet SLOs, the Nobl9 Reliability Center also now provides a single source of truth about the state of reliability in an application environment.

    That capability is critical because as DevOps teams add new features to applications, they need insights into how they are impacting the reliability of the software environment, said Singer. No matter how great any new capabilities might be, it won’t matter much if the application environment slows to a crawl or outright crashes, he added.

    Nobl9 Reliability Center addresses that need by providing dashboards and reports to assign reliability scores based on the metrics collected from the SLOs defined. Those scores can then be shared with senior business and IT leaders so they can better understand the actual state of an application environment, noted Singer.

    In addition, the platform will recommend SLOs and policies and can be used to automate runbooks to improve reliability. The goal is to enable DevOps teams to continuously monitor and manage using SLOs via alerts that are received in a way that is timely enough to act on, added Singer.

    That approach enables DevOps teams to make sense of conflicting alerts generated today by multiple monitoring and management tools. A recent survey published by Nobl9 found 72% of respondents are using six or more monitoring and observability tools.

    The Nobl9 platform collects SLOs by tracking workflows through more than 50 integrations with various DevOps tools and platforms.

    SLOs, of course, are not a new idea. They have been used as a metric to track the performance of IT services for decades.

    However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications. Ultimately, each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services. The Nobl9 platform makes it simpler to track whether SLOs are achieved and maintained.

    That’s crucial because as more organizations depend on software to deliver digital services, any disruption is likely to have a direct impact on profit and revenue. As such, the reliability of application environments has become a major concern. The issue, of course, is that many of those application environments have dependencies on legacy platforms that, in some cases, have been running for years. Each feature and application programming interface (API) added over time conspires to adversely impact performance. In an era where IT teams are now trying to manage distinct services rather than individual applications, latency can significantly slow down response times.

    Each organization will need to determine for themselves how aggressively they need to manage SLOs. But thanks to the rise of digital business transformation, more senior IT and business leaders are focused on them than ever.

  • The Role of AI in ITSM: SLAs, SLOs, SLIs and Error Budgets

    The Role of AI in ITSM: SLAs, SLOs, SLIs and Error Budgets

    In my recent blog Revolutionizing the Nine Pillars of SRE with AI-Engineered Tools, I classified SRE into nine pillars of SRE practices, a comprehensive framework that covers the full scope of SRE practices and suggested how AI can help with each of the pillars.

    For SLAs, SLOs, SLIs and error budgets—arguably the most important pillar of SRE—I indicated that AI can assist by providing accurate predictions of system performance and identifying potential bottlenecks. In this blog, I’ll explain in more detail how AI-engineered tools can be used to improve SLAs, SLOs, SLIs, error budgets and error budget policies within IT service management (ITSM).

    AI for SLAs, SLOs, SLIs, Error Budgets and Error Budget Policies

    Service Level Agreements (SLAs): An SLA is a contract between a service provider and the end user. It outlines the metrics by which the service is measured as well as remedies and penalties if agreed-upon service levels are not achieved. Setting the right SLAs is crucial, as they directly impact on customer satisfaction and the service provider’s reputation.

    The challenges in defining SLAs often involve understanding the trade-off between the cost of providing the service and the level of service that customers expect. AI can assist in this process by providing data-driven insights to inform these decisions. By analyzing historical performance data, customer usage patterns and feedback, AI models can predict the impact of different service levels on both customer satisfaction and operational costs. This can help service providers to set SLAs that meet customer expectations without unnecessarily inflating costs.

    AI can also be used to proactively monitor the agreed-upon SLAs. By continuously analyzing performance data, AI tools can detect potential SLA breaches before they occur, allowing the service provider to take preventative action. This kind of proactive SLA management can help to improve customer satisfaction and reduce the risk of penalties for SLA breaches.

    Service Level Objectives (SLOs): Establishing SLOs involves making informed predictions about system performance, defining realistic yet challenging targets that align with user expectations and business goals. A common challenge in defining SLOs is dealing with the complex nature of distributed systems and their interdependencies, making it difficult to predict performance and set appropriate targets. AI can assist in this process by analyzing historical data to model system behavior and predict future performance trends. This allows SRE teams to set more accurate, data-informed SLOs that reflect the system’s true capabilities. For instance, machine learning models could identify patterns in traffic and use this information to predict periods of high demand, allowing SRE teams to define SLOs that account for these peaks.

    Service Level Indicators (SLIs): SLIs are quantitative measures of some aspect of the level of service such as latency, throughput, availability, error rates, etc. The challenge lies in identifying the right metrics that accurately reflect the user’s experience and the system’s health. Traditionally, these decisions are based on domain knowledge, experience, and sometimes trial and error. AI can assist by analyzing a broad range of metrics across the system, correlating them with issues, incidents and user satisfaction levels to identify which indicators provide the most meaningful measure of service quality. By doing so, AI can help teams select the most appropriate SLIs, thus providing a more accurate picture of their system’s health and performance.

    Error Budgets: An error budget represents the acceptable level of risk or unreliability in a service, providing a balance between reliability and the pace of innovation. Determining an appropriate error budget involves an understanding of system behavior, user tolerance for errors, and business needs. The challenge lies in predicting future performance and understanding the likely impact of potential issues. AI can help overcome these challenges by simulating different scenarios and predicting their impact on the error budget. Machine learning algorithms can learn from past incidents and understand the correlations between different factors, helping teams predict future performance, establish an appropriate error budget, and make better-informed risk decisions.

    Error Budget Policies: These are policies that dictate actions based on the remaining error budget. These decisions often involve human judgment, like deciding when to freeze releases due to an exceeded error budget or when to allow more risk for a new feature rollout. AI can support these decisions by using predictive models to estimate the likely impact of different actions on the error budget. For example, it could simulate the impact of a new release on system reliability, providing insights to help decide whether to proceed with the release or hold it back. By doing so, AI can help teams manage their error budgets more effectively and make more data-driven decisions.

    Implementation Roadmap

    Implementing AI tools for SLAs, SLOs, SLIs, error budgets, and error budget policies must be well-planned and executed systematically. Here’s a practical implementation roadmap:

    Assess the Current Situation: Start by examining the current situation, identifying the areas of need and understanding the existing process of how SLAs, SLOs, SLIs and error budgets are being managed. Also, evaluate the data you currently have available and its quality because AI tools rely heavily on data.

    Define the Goals: Clearly outline what you hope to achieve with the implementation of AI tools. This could include better prediction of system performance, quicker identification of potential bottlenecks, more effective allocation of error budgets, etc.

    Select the Right Tools: Research and select suitable AI tools. Examples of AI-engineered tools that could be useful include:

    • Nobl9 for SLO management: It can assist with setting, monitoring and managing SLOs with AI-driven predictions.
    • Dynatrace: This tool uses AI to detect and diagnose system anomalies, improving observability and helping to inform SLOs and error budgets.
    • Datadog: Offers machine learning-based features that can help forecast future infrastructure metrics, contributing to better SLA and SLO management.
    • New Relic: Its applied intelligence feature can help with incident detection and resolution, helping to better manage error budgets.

    Data Preparation: Aggregate data from various sources, clean it and possibly format it in a way that’s suitable for the AI tools.

    Integration and Deployment: Integrate the AI tools with your existing systems to ensure that the AI tools can effectively communicate with your existing systems.

    Training and Calibration: Many AI tools require a period of training and calibration where the tool learns from your data to make accurate predictions and recommendations.

    Monitoring and Evaluation: Once the AI tools are in place, continuously monitor their performance.

    Iterate and Improve: Based on the evaluations, make necessary adjustments to the tools, your data or your goals.

    Training the Team: They need to understand how to interpret the results and how to take action based on the recommendations of these tools.

    Summary

    In the complex landscape of ITSM, AI is emerging as a transformative force, enabling organizations to manage their systems with unprecedented precision and proactivity. Through smart analysis of historical data and predictive modeling, AI tools are elevating how we define and handle SLAs, SLOs and SLIs. Gone are the days of generic, one-size-fits-all service-level targets. Today, AI empowers us to set realistic, data-informed goals that accurately reflect a system’s capabilities and align with user expectations, ensuring high customer satisfaction and operational efficiency.

    Furthermore, AI brings remarkable improvement to the management of Error Budgets and Error Budget Policies. By simulating different scenarios and predicting their impact, AI helps teams proactively manage risk, making well-informed decisions about error allowance and guiding actions when error budgets are threatened. AI enables us to strike the perfect balance between reliability and innovation, allowing IT teams to continue pushing boundaries without compromising service quality. As the era of AI in IT service management dawns, we stand on the cusp of a new phase of efficiency, control, and excellence in system management. So, join us as we delve deeper into this fascinating intersection of AI and SRE practices.

  • Nobl9 Adds Tools to Make SLO Queries More Efficient

    Nobl9 Adds Tools to Make SLO Queries More Efficient

    At its SLOconf event today, Nobl9 added a Query Tester tool that validates whether queries made against the data its service level objectives (SLO) management platform collects will deliver meaningful results. Nobl9’s SLO management platform collects data in real-time from data sources such as New Relic, Datadog and Dynatrace to help customers meet SLOs.

    In addition, a Query Checker tool will now automatically put any new SLO into a testing state to automatically validate that the query against that SLO will run correctly.

    The company has also improved the precision of its calculations to provide DevOps teams with more accurate, reliable and actionable insights into the performance of services and added a Metrics Health Notifier toolset that provides insights into anomalous data being collected from various sources as well as what data might be missing. Nobl9 has also added a Custom Query Delay tool that gives DevOps teams more control over when data is pulled along with access to data source logs.

    Kit Merker, chief growth officer for Nobl9, said the overall goal is to improve the overall resiliency of its platform for managing SLOs.

    Finally, Nobl9 also announced that its platform is now available for the Google Cloud Platform (GCP) in addition to existing support for Amazon Web Services (AWS). As part of that GCP support, Nobl9 made available an educational site that provides example of how DevOps team can take advantage of the large language models (LLMs) that Google is making to improve SLOs or generate an SLO that is automatically converted into a set of YAML files.

    A survey of 314 IT professionals conducted by Dimensional Insight on behalf of Nobl9 found 69% have implemented SLOs, with more than three-quarters of those respondents crediting SLOs for preventing business disruptions. A total of 80% also noted their organization has an increased focus on system reliability, with just over a quarter (27%) saying SLOs saved their organizations a half million dollars or more. A full 95% said SLOs are also driving better business decisions.

    However, 97% also reported it is difficult to manage SLOs because of a lack of reporting, lack of application and environment support and interoperability issues.

    Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing SLOs across a Git-based workflow. Nobl9’s platform based on that specification has now been integrated with 24 data sources, said Merker.

    SLOs, of course, are not a new idea. They have been employed as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications.

    Ultimately, each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services. SLOs-as-code are intended to make it simpler to gather the metrics that confirm whether service levels are being achieved. Those SLOs are then tracked and integrated across any set of application services a DevOps team cares to track.

  • The Hidden Configuration Tax Affecting Uptime SLO

    The Hidden Configuration Tax Affecting Uptime SLO

    This article is a preview of a talk by Greg Arnette for SLOconf 2023 on May 15 – 18. To watch this talk and many more like it, register for free at sloconf.com.

    Tax season was just a few weeks ago, and that got me thinking about how frustrating it is to get hit with a surprise tax bill and how that can relate to uptime: Configuration sprawl. DevOps practitioners constantly strive to maintain uptime service level objectives (SLO) while working with complex and ever-evolving infrastructure and applications. Configuration sprawl is a significant challenge many teams face, negatively impacting system performance and reliability.

    My company’s recent survey of 900 DevOps and platform engineering leaders revealed a startling (but not too surprising) conclusion: Most teams struggle managing secrets and configs at scale for infrastructure and applications.

    Where Does Configuration Sprawl Come From?

    As organizations scale their operations, the number of configuration files and settings grows, making it increasingly difficult to manage and maintain consistency across the board. They may have multiple environments, such as development, staging and production environments, often requiring different configurations, leading to duplicated settings and increased complexity.

    Configuration sprawl may also lack standardization as teams adopt different naming conventions, file formats and storage locations without enforced configuration standards, resulting in fragmentation and confusion. Relying on manual processes for configuration management increases the likelihood of errors and inconsistencies.

    Grinding to a Halt

    So you’ve got a lot of configuration–why should you care? Sure, it means more files to manage, but does it really impact your environment? There are a few places where configuration sprawl will negatively impact you. You might see an increase in error rates because as complexity grows, the likelihood of errors and misconfigurations increases, leading to system instability and downtime.

    Increased configuration complexity also means slower troubleshooting. Identifying the root cause of issues becomes more challenging and time-consuming when configurations are disorganized and fragmented. Ultimately this means less agility as you are forced to slow the deployment of new features and updates, impeding the organization’s ability to respond to market demands.

    How to Overcome Configuration Sprawl

    Don’t worry–you can tackle configuration sprawl, keep your SLOs on track and reduce the efforts required to manage and maintain configurations. To mitigate the harmful effects of configuration sprawl, consider adopting these best practices:

    Don’t repeat yourself (DRY)
    DRY config means variable names are declared once. Then flexible variable values are injected across build, deploy and runtime phases as defaults, inheritances and overrides. DRY avoids errors by allowing single changes to propagate where they are needed.

    DRY solves the common problem of forgetting to update critical secrets or parameters and failing to synchronize related changes (e.g., client and server port numbers or rotating a database password.)

    Decouple and abstract
    Decoupling and abstracting config means externalizing config from source code, separating how code and configuration interact, and isolating the specific values (log level) from the configuration interface (how the log level value is fetched.)

    This approach allows the same code to run in different environments using parameterized config. Abstracting config supports a best practice of implementing independent “code and config change” life cycles. Teams that frequently need to change config independently from code should decouple and abstract.

    Centralized config
    Consolidate configuration complexity into a single location because it’s easier to manage complexity within a single location versus multiple locations. Syncing config to the edge (where it is consumed) mitigates downtime. For example, it’s easier to fetch the configuration for service X versus coding all the logic needed to generate the configuration for service X.

    Standardize your configuration management by enforcing consistent naming conventions, file formats and storage locations across teams to reduce confusion and streamline configuration management. Centralizing the configuration and generating that configuration with transformations allows all the places that consume the configuration to be simple. Simple equals easy to understand and troubleshoot.

    Version control & monitoring
    Use a version control system like Git to track config changes to configuration files, making identifying and resolving issues easier. And continue to monitor changes and identify potential problems before they impact system performance.

    Train your team
    Ensure all team members know your best practices and patterns for configuration management. The proliferation of microservices and teams operating in parallel, establishing their unique naming schemes and patterns, exacerbates this problem for DevOps teams.

    Cutting the Uptime Tax

    By understanding the causes of configuration sprawl and implementing best practices to manage and reduce its impact, teams can improve their uptime SLOs and maintain a more reliable, high-performing infrastructure. Remember, investing in the proper tools and processes for configuration management is not only a best practice but also a critical factor in ensuring the long-term success of your DevOps initiatives. Those are tax cuts I can get behind.

  • Implementing SLOs-as-Code: A Case Study

    Implementing SLOs-as-Code: A Case Study

    This article was co-authored by Stephan Lips, Devin Cunningham and Justin Hoang of Procore and is a preview of their talk for SLOconf 2023 on May 15-18, 2023. To watch this talk and many more like it, register for free at sloconf.com.

    By managing service level objectives (SLOs) as code, we can co-locate SLO definitions and ownership with the product code and team. This supports horizontal scaling of SLO ownership while establishing a single source of truth and adding transparency, integrating with the code management process, and creating an audit trail for SLOs.

    When you measure the reliability of hundreds or even thousands of products and services across an enterprise, ownership of SLOs should not reside with a single team. There are a few aspects that make scaling SLO ownership horizontally across teams much more efficient: A standardized SLO management platform, process architecture and automation across the enterprise. Managing SLOs-as-code alongside product code is a step towards 360° product ownership and enables us to automate SLO updates via continuous integration.

    Objectives and Architecture

    The larger objective behind the SLOs-as-code approach is to move towards 360° product ownership. At Procore, teams own their product’s code and processes, including aspects like testing, performance, reliability, automated deployment pipelines and deployment configurations. Adding SLO ownership to a team’s portfolio will also help improve product reliability.

    Implementing SLOs-as-code involves three architectural components, which will be discussed in detail in their own respective sections. In summary, these components are:

    • SLO objects and definitions: Standardized SLOs and related resources facilitate centralized management and automation.
    • Automation: SLOs as code are our single and reproducible source of truth, which we automate via Continuous Integration.
    • A common platform: Nobl9 is the SLO management platform upon which SLOs, related resources, and automation are built. It aggregates the various data streams that power SLIs.

    SLO Objects and Definitions

    Procore’s Observability team has designed our SLO-as-code approach to scale with Procore’s growing number of teams and services. Choosing YAML as the source of truth allows Procore a scalable approach for the company through centralized automation. Following the examples put forth by openslo.com and embracing a ubiquitous language like YAML helps avoid adding the complexities of Terraform for development teams and is easier to embed in every team’s directories. We used a GitOps approach to infrastructure-as-code (IaC) to create and maintain our Nobl9 resources.

    The Nobl9 resources can be defined as YAML configuration (config) files. In particular, one can declaratively define a resource’s properties (in the config file) and have a tool read and process that into a live and running resource. It’s important to draw a distinction between the resource and its configuration, as we’ll be discussing both throughout this article. All resources, from projects (the primary grouping of resources in Nobl9) to SLOs, can be defined through YAML.

    Figure 2. A simple visualization of Nobl9 object relationships. A Project is a top-level object. Projects contain Services, SLOs are attached to Services, and SLOs can trigger alerts via project-scoped Alert Policies. Finally, Role Bindings define who has access to the Project contents.

    Procore adopted a hybrid approach to organizing our Nobl9 configuration so that the observability team can review systemic changes while teams still own changes to their service SLOs. A separate central repository is the source of truth for all other Nobl9 configurations, such as Project and Alert Policy definitions. As projects have a one-to-many relationship with their services, it would quickly become a game of “guess where” if the project configuration were defined within one of its service repositories. The central repository is owned by the observability team and allows us to manage permissions through pull request reviews submitted by product teams for their non-SLO Nobl9 resources. Once requests are merged, our automation applies the changes. The SLO definitions are co-located with the service’s code. Teams self-regulate these resources to maintain their SLOs internally.

    Automation

    Automating the creation and modification of Nobl9 resources is essential to Procore — it makes iterating on the deployment process quick and painless for our engineering teams. Automation removes human error and possible complications that usually come along with manually applying config files via CLI tools.

    With that in mind, the observability team created a CI job/workflow that our engineering teams copy into their project repo during their SLOs-as-code onboarding. Procore’s CI job uses the Nobl9 sloctl docker image, so we do not have to install the sloctl CLI tool in our CI containers. We configured this job to only apply the configs that have been added or updated, which helps to future-proof our pipeline as we scale the number of our SLO configs.

    Figure 3: SLOs-as-code Automation Workflow

    This requires the following steps in the workflow:

    1. An engineer creates or modifies a config and commits the changes to GitHub.
    2. The engineer opens a PR, gets reviews and merges their changes into the main branch.
    3. CI picks up the changes in GitHub and kicks off the following steps:
      1. Using git diff and regex pattern matching, we figure out what configs have been added or changed from the most recent merge to main:

    #!/bin/bash
    ADDED_OR_CHANGED_CONFIGS=$(git diff –name-only –diff-filter=d HEAD^ HEAD | { grep -E “^${NOBL9_RESOURCE_DIR}\/.*\.y[a]?ml$” || :; })

    2. We loop over the added/changed configs and run sloctl apply using the nobl9/sloctl docker image, which updates our resources in Nobl9:

    #!/bin/bash
    for file in “$ADDED_OR_CHANGED_CONFIGS”
    do
      docker run nobl9/sloctl:v0.0.80 apply -f “$file”
    done

    3. We use the same workflow in our centralized IaC repo that manages our projects, services, role-binding and alert policy configs.

    Common Platform

    Our platform requires another key feature for SLOs-as-code to work: We need to be able to consolidate different observability data sources and telemetry streams that power SLIs. This platform needs to support standardized data formats to define SLOs and related entities, such as alert policies for different conditions and trends, while allowing for the management of these formats via automation. We evaluated three vendors that offered SLO solutions, and only Nobl9 met all our requirements, which extended beyond the SLOs-as-code topic of this article.

    Observations

    Adopting SLOs-as-code can be impeded by three particular barriers of entry: New tooling (Nobl9 platform), data and format (SLO configs and resources) and process. To drive adoption, we have designed a self-service model supported by an open-door consulting practice. While this is an ongoing learning process for all involved, we are seeing early signs of success of an approach that uses:

    • Detailed, step-by-step instructions, including code snippets and templates.
    • Screencast tutorials supplementing the instructions.
    • Initial face-to-face, white-glove meetings to explain concepts and provide demos. The personal experience and conversation in our experience created considerable enthusiasm and goodwill among our teams. It proved to be a much better approach than just pointing them to documentation.
    • Fast feedback cycles with onboarding teams to improve instructions.

    Quite frequently, teams stated their primary challenges were not the technical aspects of SLOs-as-code discussed in this article, but how to design and improve on meaningful SLOs. We addressed this by authoring articles like Black Box SLIs and other internal resources for guidance. While posts can be a useful tool, we found that working with teams and discussing how their product impacts the user experience was more beneficial to modeling their SLOs using observability data.

    Last but not least, we are integrating with our embedded SRE (ESRE) team to help onboard more services. When a product team engages with ESRE on their deployment strategy and configuration, ESRE also discusses onboarding to SLOs-as-code and basic SLOs, such as error rate or duration.

  • Nobl9 Adds Service Level Analyzer to SLO Management Platform

    Nobl9 Adds Service Level Analyzer to SLO Management Platform

    Nobl9 today added a free tool for analyzing service level objectives (SLOs) to its platform to enable DevOps team to more accurately set reliability goals.

    Brian Singer, chief product officer for Nobl9, said Service Level Analyzer collects metrics and observability data from more than 24 DevOps tools and platforms that can then be used to create an SLO via a single click.

    Included as part of the free tier of the Nobl9 service, Service Level Analyzer also enables DevOps teams to more easily update SLOs as error budgets and error budget burn rates change over time based on actual incidents that have occurred, added Singer.

    DevOps teams can also perform “what-if” analysis to test out the impact of different targets and thresholds on error budgets, he added. That capability ultimately reduces costs because DevOps teams become less likely to overprovision resources to ensure application performance and resiliency, noted Singer. In effect, Service Level Analyzer is providing DevOps teams with an observability tool that spans the entire software development life cycle, said Singer.

    Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing SLOs across a DevOps workflow. SLOs, of course, are not a new idea. They have been employed as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications because they have many more dependencies than legacy monolithic applications.

    Ultimately, each DevOps team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services. SLO-as-code is intended to make it simpler to gather the metrics that confirm whether service levels are being achieved. As organizations become more dependent on software to drive digital processes, those SLOs are then tracked across everything from financial services to supply chains.

    In general, DevOps teams are more comfortable with the SLO concept, said Singer. The biggest challenge is making them simpler to implement, he added. The Service Level Analyzer lowers the bar for implementing an SLO by making historical data stored in DevOps tools and platforms more accessible, noted Singer. Rather than having to manually collect that data and then normalize it in an external repository, Nobl9’s Service Level Analyzer automates that process on behalf of a DevOps team using actual data versus relying on less precise statistical analysis, said Singer.

    As IT environments become more distributed, the number of dependencies that exist across application environments only continues to expand. IT teams can spend weeks trying to discover the root cause of issues that are negatively impacting application performance, only to discover that, for example, calls to an application performance interface (API) are being rerouted in a way that significantly increases latency. Tools that enable DevOps teams to more easily understand the relationships between all the services that make up an application environment are more crucial than ever.

  • Nobl9 Adds Free Tier to SaaS Platform for Managing SLOs

    Nobl9 Adds Free Tier to SaaS Platform for Managing SLOs

    Nobl9 this week announced it is making available a perpetual free tier of its software-as-a-service (SaaS) platform for managing service level objectives (SLOs).

    The free tier was announced at the AWS re:Invent 2022 conference and the company also revamped its pricing. Nobl9 Teams Edition (formerly Nobl9 Hydrogen) now includes 50 SLOs, up from 25, for $850/month. Nobl9 Enterprise Edition also has doubled the included SLOs from 75 to 150.

    Finally, Nobl9 also launched SLOcademy to help educate IT teams on how best to create and manage SLOs.

    The Nobl9 platform simplifies the creation and tracking of SLOs using metrics and observability data collected from more than 24 widely used DevOps tools including Prometheus as well as services provided by Datadog, New Relic, Dynatrace, Google, Amazon Web Services (AWS) and Splunk.

    Kit Merker, chief growth officer for Nobl9, said the company’s platform collects signals from logs, metrics and traces to define what normal should look like for any given service. The free tier of the platform has some limitations, but it does provide access to the full-featured platform to enable IT teams to get started managing SLOs.

    SLOs, of course, are not a new idea. They have been used as a metric to track the performance of IT services for decades. It’s not clear whether reliance on SLOs has waned over the years or if application environments simply became too complex to track meaningful metrics. However, as applications are increasingly viewed as services, it’s now only a matter of time before SLOs become more widely adopted across a modern application environment. The challenge, of course, is that defining an SLO is a lot easier than maintaining it—especially within today’s highly dynamic cloud-native application environments where services with lots of dependencies tend to come, change and go unexpectedly.

    Nobl9 is making a case for an SLO-as-code approach that makes it simpler to track service levels. A recent survey of more than 300 IT managers and executives conducted by Dimensional Research on behalf of Nobl9 found only 29% of respondents had no plans to implement SLOs. A full 94% of respondents that have or plan to implement SLOs intend to map them directly to business operations, with 91% reporting that they expect that effort to improve decision-making. More than 80% also said their organizations are planning to increase the use of SLOs, with 87% indicating SLOs should improve overall microservices performance.

    The challenge is that most organizations have limited visibility into their IT environments. The survey, for example, found less than half of respondents (46%) had visibility into all their IT environments. Only 45% and 35% claim to have visibility into containers and microservices, respectively. More than three-quarters (78%) said hybrid clouds make observability more difficult. Ironically, 45% of companies reported they already employed 11 or more observability and monitoring tools. On the plus side, 31% have hired site reliability engineers (SREs) while nearly half (46%) planned to create that role.

    It may be a while before most IT teams are organized around the concept of applications as services but as the management of IT continues to evolve it’s now just a matter of time before that approach becomes more the rule than the exception.

  • Nobl9 Simplifies SLO Identification

    Nobl9 Simplifies SLO Identification

    Nobl9 today added a Replay capability to its service level objectives (SLO) management platform that makes it simpler to collect the historical data a DevOps team needs to get started.

    Nobl9 COO Kit Merker said Nobl9 Replay makes service level indicator (SLI) data instantly available in minutes using 30 days of historical data collected from observability and application performance monitoring (APM) platforms. Previously, DevOps teams would have had to wait as long as a month before the Nobl9 platform could collect enough spreadsheet data to create a graph that identified the relationship between various services.

    Nobl9 now enables historical data to be merged with data collected in real-time to allow DevOps teams to view SLO charts and generate reports shortly after creating an SLO within a Git-based workflow.

    Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing them that has garnered support from Dynatrace, Sumo Logic and others. SLOs, of course, are not a new idea; they have been used as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications.

    The Nobl9 platform is designed to enable DevOps teams to employ an SLO-as-code approach that makes it simpler to gather the metrics that confirm whether SLOs are being achieved.

    It’s not clear whether reliance on SLOs has waned over the years or if application environments simply became too complex to track meaningful metrics. A recent survey conducted by Dimensional Research on behalf of Nobl9, however, found a full 94% of respondents that have or plan to implement SLOs intend to map them directly to business operations, with 91% reporting they expect that effort to improve decision-making. More than 80% also said their organizations are planning to increase the use of SLOs, with 87% indicating SLOs should improve overall microservices performance.

    The issue is that most organizations have limited visibility into their IT environments. The survey found less than half of respondents (46%) had visibility into all their IT environments. Only 45% and 35% claim to have visibility into containers and microservices, respectively. More than three-quarters (78%) said hybrid clouds make observability more difficult, even though 45% said they already employed 11 or more observability and monitoring tools. Trying to track SLOs using tools such as Grafana dashboards that DevOps teams need to set up and maintain is simply too difficult, noted Merker.

    As applications are increasingly constructed using services that drive digital business transformation initiatives, it’s now only a matter of time before SLOs are more widely tracked. That will enable DevOps teams to not only ascertain their current level of performance but also more easily work toward achieving their actual performance goals, he added.

    The challenge is that defining an SLO in a highly dynamic application environment where services are added and replaced with much greater frequency is often a lot easier than maintaining it.