devopsy.kinsta.cloud

Tag: reliability

Why Reliability Guardrails Are Needed in Every AI Coding Pipeline
We’re in the middle of a reliability reckoning. Thanks to AI, companies are shipping code much faster than before. But if there’s anything to learn from the surge in high-profile outages over the last couple of years, it’s that with more code comes more reliability risks. And when those risks do lead to an outage, the impact can be a whole different order of magnitude.

It’s like driving around a race track. At low speeds, it’s quick and easy to recover from a spinout. But when you’re going significantly faster, a single slip-up can spell catastrophe.

The nature of these risks is also changing. With AI code, we’re less likely to find typos but more likely to find unplanned dependencies, configuration drift, or infrastructure changes due to AI agents not having the proper context.

Any company that cares about its reliability needs to implement AI reliability guardrails: automated feedback loops that safely create real failure conditions to validate resilience, propose solutions for any issues found, and then verify the fixes once they’re implemented.

While operating independently of your AI agents, they provide an automatic governance mechanism that keeps your code within policy and prevents outage-causing risks from being introduced.

When done right, these guardrails will add reliability without slowing us down. And that’s just the first step. The reliability metrics from those tests provide valuable context to improve code quality in the future and help AI SREs track down root causes faster.

Reliability Guardrails Need to Be Based on Actual Performance

One of the earliest concepts of chaos engineering was that you can’t test every possible failure combination with unit and integration tests. Running realistic failure modes against real services tests many things at once, making it much more efficient.

This concept still holds true with reliability guardrails for AI code. Code analysis will only get us so far. If we want to see how the code will actually perform in production, we need to see how it responds to failures. And because we don’t have the time for manual chaos engineering experiments, we need to set up automated tests based on known failure modes.

Our goal with these guardrails is to validate resilience and ensure the code complies with resilience policies. Ideally, the code will respond correctly, for example, by falling back to the primary database when the cache is unavailable. If that happens, then we’ve verified resilience and the code can ship.

But what about when the test fails? To keep things moving quickly, the reliability guardrails should include proposals to address any issues that come up, such as suggesting an adjustment to the database timeout.

Once the fix is implemented, then the test can run again, verify the fix worked, and promote the code.

The goal is to embrace automation and AI integration to maintain reliability without slowing down. When set up correctly, these guardrails can keep everything on the road while still going fast.

Cover the Most Common Failure Modes First

Many outages are caused by issues with the same five things: CPU, memory, disk, I/O, and network. After all, software runs on computers, and computers have the same set of resources. Every architect knows this, and we all try to design systems that are resilient to failures like scaling, increased latency, or unavailable resources. But we also know that things don’t always go to plan.

So instead of waiting to see what happens in production, we need to create those failure conditions in a safe, automated way with tests that cover these key areas:
- Zone or host redundancy: What happens if entire zones or regions go down? What happens when a host or container is unavailable?
- CPU scalability: Is scaling set up correctly for CPU surges? Can it scale back down after the surge is done?
- Memory scalability: Same thing with memory. Can it scale? Will it gracefully degrade the way it’s supposed to?
- Dependency failure: Modern applications are a massive web of third-party and internal dependencies. What happens when your application can’t reach them?
- Dependency latency: Even if they can be reached, what happens when there’s too much latency? Will your application gracefully degrade, resort to a backup database, or just fall over?
The thing is, we should already know the answer to these questions, and we usually have policies in place that govern this kind of behavior. All we’re doing with these guardrails is making sure the candidate being proposed meets those policies and standards.

Use the Reliability Guardrails for AI Context

So we put these guardrails at the end of the CI/CD pipeline, right before promotion. If any test fails, the code gets flagged along with the test information, any recommended fixes, and is sent back to the agent. That alone will help improve reliability.

But those test results, fixes, and resolutions also provide valuable context to strengthen our agentic workflows. Every time a test is failed, a fix is applied and the test is passed, that data should be returned to the AI agents. We’ll still want to test every single time, but this context should mean that new code will pass more tests than it fails.

That same data can also be used to give context to AI SREs. Because while reliability guardrails will make outages less frequent, they will still happen. But if we feed the results into the AI SREs, there will be a contextual record of which failure modes have been tested, the test results, and the fixes implemented, allowing AI SREs to narrow the scope so the severity, duration, and customer impact can be reduced.

Which is, after all, the whole point: to make sure that your applications can be there when your customers need them. And with the right reliability guardrails, they can.
August 6, 2026
The Velocity Trap: Why Shipping Faster Is Making Systems Worse

There is a particular flavour of engineering dysfunction that looks, from the outside, like peak performance. Deployments are frequent. Sprint velocity is high. The feature backlog is shrinking. Leadership is pleased. And underneath all of it, the system is quietly rotting. Technical debt compounds with every rushed deployment. Observability gaps widen because nobody has time to instrument the new services properly. The on-call rotation gets noisier every month. But the velocity metrics keep climbing, so nobody sounds the alarm until something breaks badly enough that velocity stops being the conversation.

I call this the velocity trap, and it is the most common failure mode in engineering organizations that have adopted DevOps practices without internalizing DevOps principles. The practices say: automate, deploy frequently, iterate fast. The principles say: build quality in, create feedback loops, continuously learn and improve. When you execute the practices without the principles, you build a system that can push code to production at extraordinary speed with zero assurance that the code should be in production.

The trap is insidious because it feels productive. Engineers are busy. Features are launching. The dashboard shows green. But the leading indicators of system degradation are all trending the wrong way. Mean time to recovery is creeping up because each incident involves more entangled services. Change failure rate is stable only because the definition of “failure” has been quietly narrowed to exclude degradations that do not trigger a full outage. Customer-facing reliability is declining in ways that SLO dashboards do not capture because the SLOs were defined when the system was simpler.

Breaking out of the velocity trap requires leadership courage. It means telling stakeholders that the team needs to slow feature delivery to invest in the system that delivers features. It means redefining success from “features shipped” to “features shipped that stay shipped without constant intervention.” It means creating space for engineers to instrument, refactor, write tests, and pay down the debt that is silently accumulating behind every rapid deployment.

The organisations that build genuinely reliable, high-performing systems are not the ones that ship fastest. They are the ones that know when to slow down, invest in their foundations, and then ship with confidence. Speed without structural integrity is not velocity. It is just falling, with style.

May 1, 2026
What to do About AI’s Forced Rethink of Reliability in Modern DevOps
For years, reliability discussions have focused on uptime and whether a service met its internal SLO. However, as systems become more distributed, reliant on complex internet stacks, and integrated with AI, this binary perspective is no longer sufficient. Reliability now encompasses digital experience, speed, and business impact.

For the second year in a row, The SRE Report highlights this shift. DevOps and SRE leaders across industries are redefining reliability, with AI accelerating change by revealing the limitations of legacy monitoring and requiring teams to link technical signals to real-world outcomes. Traditional availability metrics continue to fall short because they fail to capture how users actually experience reliability.

From Uptime to Experience-Driven Resilience

This year’s SRE Report reinforces that slow performance is as critical as downtime, with nearly two-thirds of respondents considering performance degradations as serious as outages. This is good news as users do not distinguish between down and slow, instead perceiving both as a disruption.

According to the report, “Speed is now one of reliability’s clearest trust signals.” Fast performance builds reputation, while slow performance quickly leads to lost conversions, customer churn, and diminished trust. Resilience is now defined not only by incident survival, but by maintaining acceptable user experience under real-world conditions such as high load, congestion, and third-party failures.

AI accelerates this transition. AI-driven features, agentic workflows, and LLM-based applications introduce new latency paths and probabilistic behaviors. A component may be “up,” but the system may still provide a degraded or unexpected experience to your users. In this context, uptime metrics may produce data, but they do not produce insight.

Where Legacy Monitoring Breaks Down

Traditional monitoring tools were designed for systems fully controlled by internal teams. Today, modern digital experiences rely on:
- Third-party providers and managed services
- SaaS platforms and APIs
- Internet routing, CDNs, and millions of connected devices
- AI/ML components whose behavior changes over time
The SRE Report highlights that most teams still rely heavily on dashboards and alerts to detect performance issues. While familiar and trusted, these tools are good with the “what” but not so much the “why”, especially when the root cause lives outside the application boundary.

This gap becomes even more pronounced with AI systems. Only a small minority of respondents say they feel highly confident in monitoring the reliability of AI/ML components. For most teams, AI remains a black box, introducing new failure modes, subtle regressions, and cascading performance issues that legacy alerting was never designed to catch.

AI not only introduces new reliability challenges but also reveals the fragmentation in current observability methods. Siloed metrics, disconnected tools, and static thresholds cannot keep up with systems that learn, adapt, and rely on changing external conditions. In other words, AI is best poised to amplify whenthe underlying data is connected and integrated.

AI as the Integrator and the New Operating Model for Reliability

Increasingly, AI is being used to:
- Correlate performance, network, and user experience signals
- Surface patterns humans would miss across tools and domains
- Assist during incident response with context-rich suggestions
But AI only helps if someone owns it. Teams need people who can lead AI systems in production—setting guardrails, validating what the system suggests, tuning it over time, and being accountable when it’s wrong. That’s the new job: not “watch dashboards all day,” but guide the automation and keep it effective as the environment changes.

The report also warns against expecting automatic benefits. AI shifts the work: maintaining model drift, validating recommendations, governing agent behavior, and measuring impact. The bigger question is whether reliability work is changing business outcomes, not just engineering metrics.

Connecting Reliability to Business Outcomes

The most significant gap in the data is organizational, not technical. Only about a quarter of teams consistently assess whether performance improvements impact business metrics such as revenue or NPS. Even fewer quantify the financial cost of downtime or severe slowness.

This is a missed opportunity. The report is clear: when reliability is expressed in business terms, it gains influence. Teams that quantify the cost of delay can prioritize more effectively, justify investment, and elevate reliability from an operational concern to a strategic one.

AI can assist in this area as well. By correlating user experience data with business KPIs, AI-driven analytics help answer key executive questions:
- How much revenue does a 500 ms slowdown cost?
- Which customer journeys are most sensitive to performance?
- Where should we invest to protect trust and growth?
When reliability becomes a shared language between engineering and leadership, it shifts from being a cost center to a competitive advantage.

What This Means for SRE and DevOps Leaders

AI is prompting a reevaluation of reliability, though the goal of building trust remains unchanged. What is changing is how trust is established and measured.

For modern SRE and DevOps leaders, that means:
- Expanding reliability beyond service level objectives to speed and experience level objectives
- Acknowledging that internet stack and AI dependencies are first-class reliability risks
- Using AI thoughtfully to amplify integrated signals, or integrate edge cases
- Tying reliability work directly to customer and business outcomes
The 2026 SRE Report makes it clear that reliability has moved from the server room to the boardroom. In an AI-driven environment, reliability is how trust is earned, and the teams that understand that will be the ones that scale with confidence.
February 20, 2026
If Planes Can Fly Themselves Then Why Can’t IT Management be Autonomous?

When it comes to discussing how AI will impact the future of software development and IT management, most vendors hold on to the objective that it’s important to keep the human in the loop. They are afraid to publicly acknowledge what’s always been true — that machines are better than humans at many things. Meanwhile, that list of things continues to grow.

When it comes to observability and IT operations, our goal should be to get humans out of the loop as much as possible. In a world run by software, where SLAs guarantee 99.999% uptime, even one incident in a month — which requires human intervention — is enough downtime to violate your commitments to your customers.

With the rise of distributed, complex applications exacerbated by more and more AI-generated code, it is difficult for traditional IT operations to keep up across troubleshooting, provisioning infrastructure, performance monitoring, security, etc. Developers now deploy code 70% faster using AI, according to the latest DORA report — you can’t have enough SREs and security personnel to keep up with this kind of activity. The companies that succeed will leverage technology to get humans out of the way as quickly as possible.

Failed Promises of AIOps

It is worth stating the obvious that there’s a difference between where things are today and where they are heading in the future. As someone who has been in IT management for decades, I have seen all the buzzwords as well as both marketing nonsense and genuine progress.

The term ‘AIOps’ was coined by Gartner in 2016 to address the increasing complexity and data volume in IT environments, aiming to automate processes such as event correlation, anomaly detection and causality determination. But as it turned out, many of the vendors who claimed to offer AIOps were nothing more than empty shells when you looked under the hood.

It was essentially the same AI/ML that had been used for a decade beforehand, branded in a new way and making outsized claims that didn’t map to reality. We saw tech giants making acquisitions of point solutions, and then bundling them under the ‘AIOps’ category because it was trendy to do so and they had nowhere else better to put them.

But I would argue that many of the companies that continued to trumpet these claims without actually delivering on the promises eventually suffered a hit to their reputation. We still aren’t at that stage where machines are acting autonomously to solve complex problems across performance, reliability and security. But they are doing more than ever before, and there’s every reason to believe we will reach that goal in the future.

The Future of Autonomous Reliability

Autonomous reliability platforms of the future will not only surface actionable insights, but they will also be competent enough to make autonomous decisions without human intervention. And why should this be impossible? If planes can mostly fly themselves, why can’t IT management become autonomous?

The decades-long trend of collecting more and more data in the name of observability isn’t rendering autonomous service reliability. Nor is it feeding that data into a machine and magically hoping for answers. Machines trained on yesterday’s patterns might explain what went wrong in the past, but they can’t make the real-time decisions required to keep systems running, especially in dynamic, cloud-native environments.

What we need is a paradigm shift from data collection to causal understanding. By capturing causal knowledge as part of an ontology, we can reason about cause-and-effect in complex, ever-changing systems — the key is to move beyond reactive alerting into autonomous reliability. Unlike the growing industry trend of offloading alerts to LLMs, causal reasoning gives us the context and clarity needed to take real control.

May 16, 2025
Let’s Finally Build Continuous Database Reliability! We Deserve It
Over the past few decades, we have undergone significant transformations, enhancing our ability to improve software delivery and creating new methodologies and frameworks to improve collaboration among teams. Despite these advancements, the existing software development lifecycle (SDLC) is still far from flawless. Teams invest a considerable amount of time in the handover of artifacts, and early pipeline checks are inefficient or lacking.

At the same time, we want to achieve a reliable and robust SDLC. We want our deployments to not get blocked, our applications to not fail and our databases to not slow down. We want continuous reliability around deployments, applications and databases. While we worked hard to make sure our CI/CD pipelines are fast and learned how to deploy and test applications reliably, we didn’t advance our databases world. It’s time to get continuous reliability around databases as well.

To do that, developers need to own their databases. Once developers take over the ownership, they will be ready to optimize the pipelines, thereby achieving continuous reliability for databases. This shift of ownership needs to be consciously driven by technical leaders. The potential for platform engineers to revolutionize the industry by implementing proactive measures to safeguard databases is evident. However, having the right tools and processes is essential. Let’s see how to do it and what we need.

The World Moved on But Databases Stayed Behind

Two decades ago, cloud environments were non-existent, and the majority of software operated on on-premise servers. Applications were confined to a few blocks, typically comprising one database, a couple of web servers and file storage. In those times, troubleshooting was relatively straightforward as logs and traces were easily accessible when bugs arose.

However, our software delivery capabilities were hindered by inefficient processes. Developers worked on changes that were later encapsulated in changesets and handed over to system engineers for deployment. This segregation meant that developers were not actively involved in the deployment and maintenance phases. When bugs occurred, system engineers had to step in, leading to prolonged remediation processes due to communication barriers.

Recognizing the inefficiency of excluding developers from deployment and maintenance, the concept of DevOps was introduced, emphasizing the collaboration between developers and system engineers. However, it became evident that collaboration alone did not suffice. DevOps engineers emerged, aiming to merge competencies for smoother development and deployment. These engineers can now develop business code, deploy it and manage cloud infrastructure using infrastructure as code (IaC) solutions equipped with tools and processes for efficient operations.

The landscape has evolved significantly since then, with the adoption of microservices, independent databases for each small application and increased inter-service communication complexity. Bug identification has become challenging, given the distributed nature of systems and scattered signals throughout the ecosystem. Although component deployment has accelerated, managing this complexity remains a struggle. Effective solutions to prevent production issues, streamlined debugging processes and scalable teams are still elusive.

In the era of single-database applications, a team of database administrators sufficed for managing tasks such as database configuration, replication, maintenance, partitioning and optimizations. However, with the current deployment frequency of numerous applications daily, identifying performance issues and scaling teams of database administrators present challenges. The absence of adequate tools and processes exacerbates the situation.

Platform engineers now face the task of addressing these challenges. Just as we learned to bring developers and system engineers together in the realm of DevOps, the next step is to empower developers with ownership of all components, including databases. While developers already manage deployment, CI/CD pipelines and IaC, they lack control over databases due to monitoring challenges, insufficient troubleshooting tools and a lack of alignment with CI/CD pipelines to prevent faulty changes in production. Platform engineers are poised to address these gaps. The subsequent section explores how they can affect this change.

How to Build Database Reliability

Following are the three parts that platform engineers need to cover to build reliability in the database domain:
- Tools and processes that work across the pipeline
- Observability and semantic monitoring of the databases
- Automated troubleshooting.
Let’s go through each of these areas to understand what we need.

Tools and Processes

Various issues concerning databases can occur without developers noticing. These include the N+1 queries problem, an inadequate or excessive number of indexes, challenges related to eager loading versus lazy loading in ORMs, schema migrations and impedance mismatch — just to highlight a few.

It is crucial to recognize that developers are unable to proactively prevent these problems. They lack effective tools and processes to identify performance issues during the development of their applications. Testing databases is often insufficient, as discussed in our article on how to test databases. This inadequacy is due to the limitations of current CI/CD solutions and the testing pyramid, which struggle to detect these issues. Unit and integration tests primarily focus on data correctness and do not address concerns such as the N+1 queries problem, the use of indexes or the impact on performance when utilizing common table expression (CTE). While load tests may offer some insights, they are conducted late in the pipeline, near the end of the deployment process, providing little help to developers in terms of time efficiency.

We need robust database guardrails to enable developers to identify these issues early in the development process and shift checks to the left as much as possible. These guardrails can identify issues such as unused indexes, incorrect configurations, performance concerns and improper settings in object-relational mapping (ORM) systems precisely when developers are writing their code. By implementing these measures before committing any code changes, the turnaround time is significantly reduced. This approach empowers developers to take ownership of their databases’ performance, providing them with the necessary tools without hindering their productivity. Since the ownership is kept within one team, the turnaround increases significantly, which leads to higher reliability.

Observability for Databases

Another dimension that we must address to build reliability is monitoring. Present monitoring solutions fall short of perfection, inundating users with raw data, aggregating signals, obscuring problems within specific user cohorts, or hindering easy debugging to pinpoint issues.

Enabling developers to assume control of their databases requires the development of tools attuned to database-related activities and developers’ workflows. Database monitoring tools should comprehend schema migrations, maintenance tasks, diverse hosting methods, multi-tenancy applications, database extensions, configurations and numerous other facets. Demanding developers to take ownership becomes impractical if monitoring tools overwhelm them with raw data devoid of explanations about the system’s actual workings.

Nonetheless, platform engineers can transition from mere telemetry and monitoring to achieving comprehension and observability, as elucidated in our article on observability. By integrating database-aware tools, platform engineers can empower developers to utilize them. Once implemented, developers can effectively monitor their databases and gain insights into their evolution over time.

Automated Troubleshooting

Developers cannot assume ownership of their databases if burdened with labor-intensive and manual tasks. Activities such as setting thresholds, configuring alarms, reviewing dashboards or correlating queries with REST commands can all be automated. Instead of relying on monitoring systems to report generic issues like ‘high CPU usage,’ we need comprehensive narratives like ‘we deployed these changes to production, altering data distribution, leading to the application’s failure to use an index due to an outdated execution plan when executing the query in this particular part of the code’. This detailed account is what we require.

Platform engineers must furnish developers with tools that narrate the entire story rather than merely elucidating symptoms. This approach enables developers to address issues more expeditiously, avoiding the laborious troubleshooting process, including collecting logs from various sources and using grep to search for correlation IDs. Automation based on our knowledge of databases is essential. Various strategies to enhance database performance, as outlined in our earlier discussions, should be automated within the system. Once these three areas are fortified with database guardrails, developers can once again take charge of their databases. Let’s explore the benefits this approach can yield.

Benefits of the Shift in Ownership

The primary advantage of implementing database guardrails and empowering developers to take ownership of their databases is scalability. This approach eliminates team constraints, unlocking their complete potential and enabling them to operate at their optimal speed. By removing the need to collaborate with other teams that lack comprehensive context, developers can work more swiftly, reducing communication overhead. Just as we recognized that streamlining communication between developers and system engineers was the initial step, leading to the evolution into DevOps engineers, the objective here is to eliminate dependence on other teams. Developers are no longer reliant on system engineers or database administrators; they can independently manage and maintain their databases.

This results in a significantly accelerated evolution process. With each database now under the ownership of the respective microservice owner, any database issues are promptly addressed and resolved by the owner. There is no need for centralized performance management or maintaining teams of database administrators capable of optimization but unable to keep pace with the speed of development.

Another noteworthy aspect is the reduction of the bus factor. As the knowledge of the database becomes concentrated within a single database administrators’ team, concerns about staff turnover or extended vacations are alleviated. Database task handovers can be managed like regular development workstreams, aligning with the principles of agile methodology. Database-related tasks seamlessly integrate into the scrum methodology.

Ultimately, developers taking ownership of their databases minimizes the time required to identify and address database issues. Developers are freed from the burden of slow and mundane tasks. Thanks to semantic monitoring, they promptly identify issues, automated troubleshooting provides a comprehensive understanding of the problem, and they can independently rectify the issues. This eliminates the need for war rooms or call bridges to decipher the situation.

What is Ahead of Us

Continuous reliability is a must regardless of the company size. The shift of ownership provides a way to achieve it. Database guardrails mark the inception of a new era for developers and databases, but this is just the starting point. With the integration of machine learning (ML), automated troubleshooting can evolve into automated code changes. Similar to static code analysis that identifies common issues in programming languages, tools can generate automated pull requests to address typical problems, leveraging production database data captured automatically. Rather than initiating a ticket for ORM configuration changes, database guardrails can autonomously modify the code, seeking approval as a formality.

As developers take charge of their databases, they can employ CI/CD best practices to enhance the database’s state. The testing pyramid will expand beyond checking business requirements to encompass ‘how to do that,’ ensuring not only correct actions but also correct implementation.

Ultimately, this approach will reduce communication bottlenecks between teams and roles, transitioning from DevOps to DevDbOps. This is the path we must tread to unlock the full potential of developers.

Summary

In recent years, the global landscape has grown significantly more intricate. The proliferation of databases, services, communication channels and dynamic components has added complexity. Similar to the shift toward DevOps and the implementation of CI/CD using infrastructure as code (IaC) for expedited change deployment, integrating database guardrails is essential to empower developers to take ownership of their databases. It falls upon platform engineers to advocate for and implement this innovative approach within their organizations.
July 11, 2024
Google Taps Nobl9 to Improve Services Reliability

Google announced this week it has added a capability for tracking reliability based on how service level objectives (SLOs) are being met that is based on a platform developed by Nobl9.

Nobl9 launched a Nobl9 Reliability Center earlier this year that provides a single source of truth about the state of reliability in an application environment. Included in that platform is SLOgpt.ai, a generative artificial intelligence (AI) tool for programmatically creating SLOs and launching natural language queries to determine how well they are being adhered to across a distributed computing environment. The generative AI tool developed by Nobl9 is based on Vertex, a foundational large language model (LLM) provided by Google.

Kit Merker, chief growth officer for Nobl9, said the generative AI tool also makes managing SLOs more accessible to a wider range of stakeholders.

The expansion of the alliance with Google is the latest in a series that will make it simpler to manage SLOs across multiple ecosystems, including, most recently, partnerships that have been established with Microsoft and Cisco, noted Merker.

The Nobl9 Reliability Center is already integrated with more than 50 DevOps, observability and incident management tools. It collects metrics, events, logs, traces and alerts to track incidents, releases, rollbacks, runbooks and other documentation. Those integrations will make it simpler for IT teams to create SLOs using historical data that would otherwise be challenging to define from scratch.

In time, there is no doubt tools for managing SLOs will become more widely available, but given the dependencies that exist between applications and systems, DevOps teams will need an approach that can be employed across hybrid IT environments, said Merker.

In general, as more microservices-based distributed applications are built and deployed, it’s becoming more challenging to maintain SLOs across them since they have many more dependencies than legacy monolithic applications. Each feature and application programming interface (API) added over time can adversely impact performance and reliability. The challenge is that it’s not so much that any given service will outright fail as much as it is determining where degradations are occurring that result in SLOs not being met.

Each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services by making it simpler to track whether SLOs are achieved and maintained. Not every service needs to necessarily meet a stringent requirement, but IT teams do need to have confidence in the fact that, on average, certain levels of service are being consistently maintained.

When it comes to SLOs, the challenge is first determining which dependencies might impact the level of reliability that can be maintained in the event of an outage and then, second, making sure whenever necessary workarounds—such as rerouting application programming interface (API) calls to other services—can be easily invoked.

SLOs, of course, as a concept, have been around for decades, but with the rise of digital services that span multiple organizations, the level of reliability DevOps teams are expected to provide is now being written into business contracts. As a result, the overall level of stress related to managing SLOs is only going to increase in an era where businesses have never been more dependent on software.

December 1, 2023
Nobl9 Allies With Microsoft to Simplify SLO Creation

Nobl9 this week revealed it has integrated its reliability center platform for managing service level objectives (SLOs) as code with the Microsoft Azure Monitor service to make it simpler to both create and track service levels.

The Nobl9 Reliability Center provides access to dashboards for a single source of truth about the state of reliability in an application environment based on the metrics collected from the SLOs that have been defined.

Nobl9 Chief Growth Officer Kit Merker said the integration with Microsoft Azure Monitor is the first in a series that will ultimately make it easier to define SLOs using historical data collected by third-party tools.

The Nobl9 Reliability Center is already integrated with more than 50 DevOps, observability and incident management tools through which it collects metrics, events, logs, traces, alerts to track incidents, releases, rollbacks, runbooks and other documentation.

Those integrations will ultimately make it simpler for IT teams to create SLOs using historical data that today are still challenging to define from scratch, noted Merker. In fact, the data tracked by Nobl9 can be used to automate remediations via those integrations, he noted.

As more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications. Each feature and application programming interface (API) added over time conspires to adversely impact performance. It‘s not so much that any given service will outright fail as much as it is determining where degradations are occurring that result in SLOs not being met.

Ultimately, each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services by making it simpler to track whether SLOs are achieved and maintained. Not every service needs to necessarily meet a stringent requirement. Instead, IT teams need to have confidence in the fact that, on average, certain levels of performance are being consistently maintained.

That’s crucial because as more organizations depend on software to deliver digital services, any disruption is likely to have a direct impact on profit and revenue. As such, the reliability of application environments has become a major concern in an era where IT teams are now trying to manage distinct services rather than individual applications.

The challenge and the opportunity now is to find a way to proactively reduce that stress by programmatically managing SLOs as code rather than having to overprovision infrastructure resources to ensure maximum availability at the highest cost possible. At a time when more organizations are encountering economic headwinds, that’s no longer a luxury they can afford.

Each organization will need to determine for themselves how aggressively they need to manage SLOs. Of course, SLOs as a concept have been around for decades, but with the rise of digital services that span multiple organizations, the level of reliability DevOps teams are expected to provide is now being written into business contracts. Not surprisingly, the overall level of associated stress DevOps teams are experiencing has correspondingly increased.

November 17, 2023
Declarative Compliance With Policy-as-Code and GitOps
In recent times, the quest for greater agility, faster releases, enhanced scalability, security and performance brought forth the advent of several automation tools, technologies and frameworks. Software development has evolved considerably over the years to mitigate these challenges.

Monoliths have been split into microservices for improved scalability, maintenance and faster releases. Since DevOps is adept at bridging the gap between development and operations teams, now these two teams can work together to improve productivity and speed up release cycles.

The Need for Compliance

One of the critical strategies for business success is to stay compliant since processes are always improving and evolving. Compliance ensures that your DevOps processes and practices are consistent and meet regulations or regulatory standards.

An organization with a compliance-driven DevOps culture can cut down on operational costs, improve efficiency and reduce risks considerably.

What is Declarative Compliance?

Declarative compliance is defined as the process of managing and enforcing compliance in which policies are defined using code, and the system’s desired state is declared rather than explicitly programmed. To enforce compliance policies, you don’t write scripts or programs; instead, you use a high-level language or configuration file.

Declarative Compliance provides several benefits:
- Increased efficiency
- Better managed compliance
- Minimized risk of human errors
- Enhanced security
- Faster deployment
- Better collaboration
With declarative compliance tools, the system’s state is defined as a set of policies, and the tool makes sure it matches the desired state. The tool will automatically fix anything out of compliance, bringing the system back into compliance if there’s a deviation.

Tools for Declarative Compliance

Some of the popular declarative compliance tools are:
- Open Policy Agent (OPA)
- Terraform
- Kubernetes Policy Controller
- Chef Compliance
- AWS Config
Declarative Compliance for Infrastructure and Application Deployment

Often, declarative compliance is used in infrastructure management and application deployment, where policies are written as code and enforced across multiple environments. In addition to managing compliance at scale, declarative compliance tools ensure that the system’s desired state stays consistent across multiple systems and applications over time.

With declarative compliance, organizations can manage compliance more efficiently, effectively and securely since they can define policies as code and enforce them automatically. In addition to reducing errors, downtime and security breaches, this approach enables organizations to stay compliant and maintain their system consistently.

Introduction to Policy-as-Code

A policy-as-code approach allows you to write programs that govern security, compliance and rules throughout your application’s life cycle. It entails defining and applying code to manage and automate policies. It shares the same ideas as infrastructure-as-code (IaC) and other DevOps practices like continuous integration and continuous delivery (CI/CD).

A policy enforces constraints and restrictions to prevent unauthorized access to resources such as databases, storage, services, etc. By codifying policies, you can enforce rules and constraints throughout the development life cycle to prevent non-conforming resources from being deployed by integrating these policies into the software development life cycle.

Figure 1: Policy-as-Code at work!

Why Policy-as-Code?

Typically, businesses have the required processes, tools and procedures to guarantee the deliverable’s quality. Performance, scalability, ease of use and accuracy are typical quantifiable quality metrics used to ascertain the quality.

You can examine the quality of your application using these metrics, but a manual procedure will take time and is prone to mistakes. A better alternative is to leverage policy-as-code.

The traditional policy enforcement processes have certain drawbacks. It is challenging to perceive that your policies would be entirely adhered to, i.e., they will not be broken. It is cumbersome to check against a list of policies manually.

Moreover, the traditional policy enforcement method cannot scale in today’s agile world, where organizations have been growing continuously with more and more employees, services and teams.

Here are the key benefits of policy-as-code at a glance:
- Faster and more efficient deployments
- Automated governance, approvals and policy enforcement
- More reliable and secure applications
- Centralized policy management
- Leverages version control
- Better visibility of policies
- Enhanced collaboration
How Does it Work?

Policy-as-code involves three key elements:
1. The policy itself, which contains the required code to model the decision-making process
2. The data and information about the environment, service or application
3. The query that is responsible for triggering the decision-making process
Here’s how policy-as-code works:
1. Policies are defined as code in a human-readable format
2. These policies are then translated into machine-readable code using Rego or HCL
3. The policies are integrated with the IaC pipeline
4. A policy engine evaluates the infrastructure and the application resources against these policies
5. The policy engine accepts the policies as input, processes them and then produces a query result
6. The policy engine reports violations (if any) to the respective stakeholders, such as the development or security teams
7. If violations are reported, the necessary remediation is performed automatically or manually
Types of Policies
- Security policies: Security is enforcing adequate technological controls to safeguard company assets.
- Compliance policies: Compliance policies are designed to help organizations adhere to governance standards. Typical examples include PCI DSS, GDPR, HIPPA, etc.
- Operational excellence: This encompasses policies pertaining to service degradation or interruption.
What is GitOps? How can GitOps Help With Compliance?

GitOps is a subset or extension of DevOps encompassing a collection of practices that can simplify deployments using Git. It is an operating model for cloud-native applications that leverages Git as a single source of truth for declarative applications and infrastructure. Using GitOps, you can implement a standardized policy-as-code approach across your CI/CD and GitOps pipelines.

The key principles of GitOps are:
- The state of the system is captured and stored in a Git repository
- The entire system is configured declaratively
- Automated deployment
You can create policies in Git and store them in a particular repository, which is essential since Git offers sophisticated change management tools like version control. Only the compliance team would have admin access to the repository.

It’s easy to enforce policies with Git, and you can apply them to any execution engine, like CI or CD. With GitOps, you can audit and document every action that affects data consumption.

You can expose system changes, optimize deployment and use a version-controlled infrastructure. By managing compliance with GitOps, you can make sure that the changes are visible, verifiable and auditable.

Figure 2: GitOps at work!

Using GitOps as the Execution Engine in a GitOps Process

There are several benefits to using GitOps as a process execution engine, including:
- Increased automation: GitOps automates software delivery, reducing manual intervention and increasing deployment accuracy and speed.
- Better transparency: GitOps records all changes to the system in a transparent and auditable way so that you can track compliance easily.
- Maintaining consistency: GitOps keeps the system in the desired state defined in the Git repository, reducing non-compliance risk.
- Fosters collaboration: All changes to the system are made through the Git repository, so GitOps encourages collaboration across teams.
Using Policy-as-Code and GitOps: Policy-Based Compliance

By using GitOps and policy-as-code, teams can manage compliance-as-code in an auditable, declarative way and enforce it via GitOps, ensuring that the desired state is always maintained. This can help your teams to codify compliance requirements as code, store them in version control and automatically enforce them through GitOps.

Organizations can improve the consistency and reliability of their systems by defining and enforcing compliance requirements using policy-as-code and managing their deployment processes with GitOps. While this can minimize compliance violations, it can also maximize efficiency.

By following policy-based compliance declaratively, businesses can define policies as code and enforce them automatically and repeatedly. By adopting such an approach to compliance management, you can better manage compliance in complex systems in a consistent, scalable, agile, transparent and collaborative manner.

For example, you can leverage policy-as-code to define policies for infrastructure provisioning and configuration management to ensure that the infrastructure complies with organizational policies and industry standards. Typical examples are defining policies for network configuration, access control and resource allocation.

Use Cases: Using Policy-as-Code and GitOps in a Production Environment

Infrastructure Provisioning

Open Policy Agent (OPA) can define policies for infrastructure resources like AWS Elastic Compute Cloud (EC2) servers. You can use GitOps to manage the deployment of infrastructure resources by storing IaC files in a Git directory. When the IaC files are changed, GitOps is adept at automatically deploying the changes while OPA checks for policy compliance.

Deployment

You can use policy-as-code tools such as OPA to create application deployment policies requiring specific security settings, while GitOps manages the application deployment by storing manifests.

Manage Kubernetes Clusters

You can leverage policy-as-code to better manage your Kubernetes clusters. So, you can write your code to define policies to manage Kubernetes resources, such as nodes, pods, clusters, etc., declaratively.

Compliance Management

You can use policy-as-code to establish and implement rules and regulations, such as those pertaining to security, to help businesses ensure that their systems comply with industry regulations and standards.

OPA can be used to set compliance policies, like restricting access to sensitive data to authorized users only. You can use GitOps to manage the deployment of compliance policies seamlessly.

Trusted Delivery: Policy-as-Code Integrated With GitOps Workflows

Trusted application delivery is a software delivery approach emphasizing security and trust in the software supply chain. This ensures that any software delivered to the clients is devoid of security flaws, malicious code and other attacks.

This approach prioritizes security and trust at the development, testing, packaging and deployment stages and integrates security and compliance checks into the GitOps pipelines. This quality check reassures clients about the product’s quality and ensures that any modifications adhere to predetermined rules and principles.

Trusted application delivery allows for quick deployment while securing apps with automated guardrails. These guardrails are implemented using policy-as-code. By enforcing these guardrails, you encourage frequent deployments while ensuring that the reliability and security of the application are not compromised.

Conclusion

Declarative compliance with policy-as-code and GitOps is a best practice for achieving greater compliance, reliability and agility in software development and deployment processes. You can simplify defining and enforcing compliance policies using declarative configuration files and automation tools. Using Git as the single source of truth for all configuration changes, an organization can achieve greater efficiency, visibility and auditability in the compliance process.
April 6, 2023
Debunking Myths About Reliability

“Our service should always be up.” Some myths just won’t die.

Engineering for reliability is well understood by engineering leaders, less so by bosses demanding unreasonable uptime with minimal resources and immense feature pressure. Business leaders tend to waffle between ignoring reliability with a hand wave and freaking out after an outage. How can they understand the reality of reliability and forget the myths?

Myth One: Our Service Should Always be Up

Reality: Our Service is Engineered to Consistently Exceed Expectations

Some might imagine reliability like a light switch—it is either on or off. But reliability is about consistent repetitions and managing the risk of minor, occasional, recoverable failures. Can we consistently exceed customer expectations?

Understanding customer expectations is challenging because you can’t ask people what they expect from you (unless you have a peculiar customer). So instead, we want to define and quantify the impact of our service not working and how that could negatively impact customer experience based on observed behavior. For example, “People abandon their shopping cart when our checkout experience takes more than 10 seconds to load, which means we lose money.”

We reflect business impact in our reliability goals and then engineer our system and processes to meet this goal. We might change how we do releases, ensure tests pass or avoid changes during times of peak usage. These engineering decisions use reliability as a business metric based on customer expectations.

Myth Two: Innovation is More Valuable Than Reliability

Reality: You Need to Balance Innovation and Reliability Engineering Work

The constant drive for new features over reliability is the most frustrating myth. “We need to release new feature X, or we won’t have customers; we can worry about reliability later.” In reality, most customers care about reliability; they don’t mention it until it’s a problem.

You can have the most fantastic whizbang feature in the world, but if it doesn’t work–and Murphy’s Law will make sure that it doesn’t work at the worst possible time–no one can use it, no one will be impressed and it will turn your technology into a laughingstock.

Innovation excites customers, but trust comes from reliability, which you must earn through hard work and clever engineering. Depending on your business context, you may require reliability more than ever. If you are facing headwinds, you may need to scale back your ambitions regarding innovative features. Still, you can’t scrimp on reliability or customers will justifiably leave.

Myth Three: Five Nines is Normal and Incremental From Four Nines

Reality: Five Nines is Expensive—10X the Cost of Four Nines

No one–not even massive cloud providers or telcos–can consistently deliver at 99.999% across all their services by accident. Reliability at that level–less than six minutes downtime per year!–is an engineering marvel. A bridge or a dam might look simple after completion, but the engineering required to create a reliable physical infrastructure is immense, as everyone knows. Why is it so hard to understand the complexity, design, engineering and redundancy required to deliver a highly available and performant digital system? Further, it’s easy to think that 99.999% is just a bit more than 99.99%. After all, it’s “just one more nine.” Remind your less-technical counterparts that each nine requires ten times more effort!

Why is it so expensive to deliver? Because the failure tolerance (also known as an error budget) is 1/10th the size but the risk of missing the goal increases exponentially. You’ll need more redundancy, careful testing and certification of releases, increased on-call rotations, extra hardware or cloud capacity and extensively tested backup plans to achieve this goal.

Worst of all, higher reliability will slow you down. You can’t innovate or deliver updates as fast when you need to ensure absolute uptime.

But what if there was a limit to how much reliability we need?

Myth Four: More Reliability is Always Good

Reality: Reliability Engineering Has Diminishing Returns

There is a point at which being “too reliable” is terrible for business. It’s expensive to build all that redundancy, testing, responding to tiny glitches and all the rest. And most of your users won’t notice. We must avoid the large blowups that put us in the headlines and manage expectations everywhere else. The significant outages that can impact thousands, if not millions of customers come from this reductionist view of reliability as a by-product of conscientious work rather than an engineering problem with well-defined tolerances and thresholds. You earn the trust of your customers by properly engineering reliability into your delivery process.

Consider tardiness at meetings. If you wanted to be 99% on time, you’d need to join a one-hour call within 36 seconds and at 99.9%, you would need to enter a Zoom within 3.6 seconds of its start time, a timescale so small you don’t even notice it. You would have to do this for every meeting you attended, no excuses–bio breaks, last meeting ran long, someone at the door, etc. None of these things matter when defining and measuring reliability. This metaphor also provides a common sense way to think about risk and error budgets. Your other meeting attendees can’t possibly notice (or care) if you’re 3.6 seconds late, no matter how prestigious or impatient the other party is.

You could apply this same reasoning to catching a flight, picking up your kids from school, completing an exam, building a woodworking project or any human endeavor. The concept is so intuitive to daily life that even pointing it out seems absurd. But this is the fundamental concept from which reliability engineering stems. To build a reliable system, we must define acceptable failure boundaries. Otherwise, we will spend precious time and resources to eliminate the 3.6 seconds of delay that no one cares about and miss the more significant issues–like being present and engaged in the discussion.

Busting Reliability Myths

Understanding reliability is vital for engineers and business people alike. It all comes down to intentionally designing a customer experience, keeping up with expectations and, in some cases, even promises. Right-sizing reliability lets you find the perfect balance between delivering excellent service and efficiently running your organization.

Image Source: Indira Tjokorda via Unsplash

March 31, 2023
Nobl9 Adds Service Level Analyzer to SLO Management Platform

Nobl9 today added a free tool for analyzing service level objectives (SLOs) to its platform to enable DevOps team to more accurately set reliability goals.

Brian Singer, chief product officer for Nobl9, said Service Level Analyzer collects metrics and observability data from more than 24 DevOps tools and platforms that can then be used to create an SLO via a single click.

Included as part of the free tier of the Nobl9 service, Service Level Analyzer also enables DevOps teams to more easily update SLOs as error budgets and error budget burn rates change over time based on actual incidents that have occurred, added Singer.

DevOps teams can also perform “what-if” analysis to test out the impact of different targets and thresholds on error budgets, he added. That capability ultimately reduces costs because DevOps teams become less likely to overprovision resources to ensure application performance and resiliency, noted Singer. In effect, Service Level Analyzer is providing DevOps teams with an observability tool that spans the entire software development life cycle, said Singer.

Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing SLOs across a DevOps workflow. SLOs, of course, are not a new idea. They have been employed as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications because they have many more dependencies than legacy monolithic applications.

Ultimately, each DevOps team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services. SLO-as-code is intended to make it simpler to gather the metrics that confirm whether service levels are being achieved. As organizations become more dependent on software to drive digital processes, those SLOs are then tracked across everything from financial services to supply chains.

In general, DevOps teams are more comfortable with the SLO concept, said Singer. The biggest challenge is making them simpler to implement, he added. The Service Level Analyzer lowers the bar for implementing an SLO by making historical data stored in DevOps tools and platforms more accessible, noted Singer. Rather than having to manually collect that data and then normalize it in an external repository, Nobl9’s Service Level Analyzer automates that process on behalf of a DevOps team using actual data versus relying on less precise statistical analysis, said Singer.

As IT environments become more distributed, the number of dependencies that exist across application environments only continues to expand. IT teams can spend weeks trying to discover the root cause of issues that are negatively impacting application performance, only to discover that, for example, calls to an application performance interface (API) are being rerouted in a way that significantly increases latency. Tools that enable DevOps teams to more easily understand the relationships between all the services that make up an application environment are more crucial than ever.

February 22, 2023