devopsy.kinsta.cloud

Author: Adam Frank

Using DORA Metrics to Measure DevOps Success

The DevOps Research and Assessment (DORA) metrics have fundamentally changed how leaders interact with developers—and, in turn, how developers interact with their code. Although these metrics have changed significantly since their introduction in 2014, their intent remains to provide a numbers-based approach to improving the developer experience.

DORA metrics play a critical role in many development workflows. These metrics posit that four focus areas—deployment frequency, lead time for changes, change failure rate, time to restore service and reliability—can differentiate high-performing software engineering teams (elites) from lower-performing teams (non-elites).

Why is this a meaningful difference? High-performing teams reliably achieve faster time-to-market by removing menial tasks from their workload and expediting the development and deployment processes. Elite teams provide a better customer experience, deploy robust code faster and maintain better availability. And engineering teams that codify their procedures and track using DORA metrics are more likely to achieve these results in a shorter time frame.

Why are DORA Metrics the Baseline for Many Developers?

Most departments have tangible objectives and key results (OKRs) to present to executives. For example, the finance department typically looks to achieve a certain revenue threshold, while the marketing department may report on social media traffic or engagement.

Software engineers have spent years grappling with how to demonstrate success to non-engineering departments. Although it’s relatively easy to identify when a deployment goes wrong, many developers have difficulty translating how these breakdowns occur. Engineering teams cannot generate buy-in for necessary process improvements or tool adoptions without the vernacular to discuss success benchmarks.

DORA metrics provide a language for developers to use when speaking with executives and other teams. Moreover, DORA metrics enable developers to understand their process’ weak points and identify areas for improvement internally.

Making the Most of DORA in 2023

It’s incredibly frustrating to fall short of development goals or hit roadblocks like post-deployment errors and bugs. DORA metrics are the secret to circumventing these challenges. Here’s how DORA can go beyond codification to help your team achieve quicker, more reliable development and deployment:

Accelerate Time to Market

Tracking DORA metrics can significantly curb a team’s time to market—a key competitive advantage in the fast-paced world of software development. Enterprises that deploy more quickly and frequently accrue greater revenue over time than less efficient enterprises.

A streamlined deployment process looks different from organization to organization. That is why DORA metrics are essential; they pinpoint each organization’s specific areas of improvement, enabling DevOps leaders to make informed decisions about remediation. Common indicators for lagging time to market include non-orchestrated deployment processes and manual testing procedures, which distract developers from what they do best: coding.

Leaders looking to isolate issues with time to market should scrutinize their organization’s deployment frequency, as this metric can indicate process inefficiencies. Deploying irregularly? Your team may be batching too many updates, leading to overloaded, riskier deployments. Alternatively, infrequent updates may mean your organization has outgrown manual deployments. If that’s the case, consider adopting continuous integration and deployment (CI/CD) processes—these offerings improve time-to-market through immediate feedback and automated testing. Successful CI/CD implementations are proven to increase daily deployments by over 1000%.

Improve Development Processes

Although highly interconnected, deployment and development delays stem from different issues. Think of it this way: Low-performing engineering teams might deploy frequently—several times a day, even—but without quality assurance, those deployments are likely erroneous and may lead to extended system downtime.

DORA metrics are a powerful way to isolate deployment issues from gaps in the development process. For example, an engineering team that tracks DORA may find their deployment frequency optimal, but their time to restore service and change failure rate lagging. This discrepancy indicates the team needs to improve its testing practices by rigorously reviewing code and managing application dependencies better. Additionally, the team should consider implementing more advantageous deployment and rollback processes, including but not limited to progressive deployments and automated rollbacks.

Remove Menial Tasks From the Workload

Some organizations approach DORA metrics and other OKRs as productivity benchmarks that can be wielded against individual engineers. This perspective has even tinged how many developers view DORA metrics. Yet Google research demonstrates DORA metrics are most successful when adopted by high-trust, low-blame cultures. In these enterprises, DORA metrics reveal organizational opportunities for improvement—never should they reveal lagging individual performers or provide reasons for unnecessary busy work.

In fact, DORA metrics often eradicate rote tasks from developers’ workloads. Teams that use these metrics to identify and implement better incident management processes spend less time fighting fires (e.g., reworking code after a costly manual rollback) and more time doing higher-value work, like building new experiences. Automated processes like automated rollbacks and CI/CD empower developers to improve the quality of individual deployments while also reducing stress

Make DORA Work for Your Engineering Team

DORA metrics aren’t going anywhere soon. While they continue to evolve, like the recent removal of the elite category, they remain the industry standard for engineering OKRs, they provide a robust framework for success and a common language for developers and non-engineering stakeholders to communicate about performance, challenges and improvements in the software development process. DORA eases the collaboration and decision-making processes, creating an environment where leaders can consistently and effectively make data-driven improvements.

But these improvements are only possible when engineers are empowered by beneficial processes and tools. I encourage leaders who implement DORA metrics into their benchmarking to take results seriously. Is your organization’s time to market lagging? Interrogate your current developer workflow—and actively take steps to address issues through developer-first solutions. Doing so will not only improve your organization’s software delivery capabilities but also foster a culture of continuous learning and improvement.

October 10, 2023
The Evolution of SRE and Developer Roles

As an industry, we’ve seen cultures and roles shift throughout IT teams — specifically, the role of ITOps site reliability engineers (SREs), various developer roles and the culture of DevOps. Most of these roles operated – or still operate – in silos, and didn’t understand (or care to understand) how each other’s role worked. More importantly, they didn’t know how to work together. But these roles are starting to overlap, driven by the increase in demand for customer experience and, subsequently, in intelligent observability, or observability with AIOps. Before we dive into the ‘how’ behind the overlap, let’s look at how each of these roles performs today.

Developer

While the specifics of what a developer does varies by project, the overarching responsibilities haven’t changed much since the introduction of the role. Simply put, developers write code to create and improve product functionality.

IT Operations and SRE

Traditionally, an IT operations professional focused on operating the code a developer put into production; aka keeping the lights on. However, we’re now seeing evolution of the role of highly skilled operations pro to that of an SRE who oversees automation and features surrounding the product, like CI/CD pipelines, infrastructure-as-code and the ability to monitor and observe the services and components within.

QA

Traditionally, QA would test the code the developer wrote and move it down the deployment pipeline. But we’re moving away from traditional QA. Now, the role sets up an automated framework for the developer so they can automate tests — similar to how the SRE sets up the CI/CD pipeline and observability framework to build, test, deploy, verify and monitor their code.

DevOps

DevOps has, in some cases, evolved from a practice and culture into a role. The DevOps engineer prepares the automation and the CI/CD pipeline for the developer to push out their code to test and, ultimately, production. But let me be clear – I believe the idea behind the practice of DevOps is that the practitioner does everything — they own testing frameworks, CI/CD pipeline and infrastructure. They own end-to-end product development and operation, so when anything does go wrong, they know exactly what to do, where to go and how to fix it.

In the end, if a SRE can develop everything around the product code and knows how to fix the product — isn’t an SRE actually a developer? What if we have a really skilled developer who sets up the automation and observability framework to get their code from their keyboard to the user as quickly as possible — without breaking anything, of course — while also building and improving all the infrastructure and components surrounding the product? Isn’t the developer an SRE, or at least performing site reliability engineering?

The short answer, in my opinion, is yes. And there’s one guiding force behind this amalgamation, this crossover; ultimately, this evolution: intelligent observability.

The role of observability with AIOps

Traditionally, and with the use of many tools, DevOps practitioners and SREs have had to — manually — battle floods of data coming in from all their endpoints, services and products. This can feel like a never-ending struggle. It can lead to a lot of toil and time spent on maintenance, when what developers really want is focused time to do what they love – write code, create and innovate. Not to mention, the result has a negative impact on the customer experience.

But now, new tools are surfacing to support the everyday needs of SREs and DevOps practitioners. For example, intelligent observability, or AIOps, applied to observability data like events, metrics, traces, changes and logs, gives teams clearer insight into not only where complex data is coming from, but why an issue is happening and what action they need to take to resolve it. They can spend less time operating and more time innovating.

As these roles continue to evolve, it’s important SREs and DevOps practitioners remain agile and willing to adjust to keep up with the latest technologies supporting their day-to-day workflow. AI can be the key to keeping the incident count down and the customer experience up.

Looking Ahead

If you’re currently in one of these roles, consider how you are practicing. Whether you’re a developer, a QA engineer, a DevOps practitioner, or an SRE, have you evolved your role to fit your company and customers’ needs right now? As these roles overlap with the application of intelligent observability throughout your development life cycle, it’s possible to deliver better products and fix reliability issues faster. It’s up to you to decide if you want to embrace the evolution of your role and intelligent automation or stay put and risk falling behind.

May 11, 2021
How to Make a Self-Healing IT Infrastructure a Reality

In 2019, enterprises worldwide reported that every hour of infrastructure downtime cost them an average of $301,000- $400,000. If a system is down for five hours, that’s $1.5 million lost—and that’s on the low end. Five hours of downtime can impact companies for months down the road. It’s in the hands of DevOps practitioners to ensure systems are up and running and budgets are kept in line. But if they don’t have the proper tools on hand that allow them to juggle their responsibilities while adding value to the business, it makes their jobs that much more difficult—or, dare I say, impossible.

Teams simply can’t afford to go offline to fix incidents, but neither can they afford to manually sift through data to find the root cause. By automating the incident management process, developers have more time to focus on building new products and capabilities that drive revenue. With a self-healing IT infrastructure, teams can tackle these issues before they become larger issues costing millions of dollars.

Imagine you’re running to catch a bus and your heart rate increases, but when you sit down, it doesn’t come back down to a normal rate. This could quickly become a much larger, even fatal issue. In this instance, the body should “self-heal” to get back to a normal rate. On a similar note, a self-healing IT infrastructure allows teams to quickly get back on track by fixing issues before they become a $1.5 million problem.

Let’s look at how DevOps practitioners can turn a self-healing IT infrastructure into a reality.

Self-Healing Infrastructure Needs Observability and AI

Observability is the practice of collecting deep data from applications and services to provide insights through three core components: logs, traces and metrics. While these three components are essential to a self-healing infrastructure, they are most powerful when used together. Similar to our senses of sight, hearing and touch, they each tell us something different but are equally important. When combining the power of all three, development teams can determine where, when and how an incident occurred and take action as needed.

We all know the pain of traditional monitoring tools. They don’t surface immediate issues or identify root causes. They rely too heavily on developers to interpret and analyze immense amounts of data then sift through it in a tedious and time-consuming process requiring extra budget for more human power, and extra time to identify and remediate incidents. Using today’s cloud-native self-service approaches to combine observability and artificial intelligence, teams can set up tools themselves that automate the process of gathering and correlating metrics, logs and traces at machine speed to provide a complete, intelligent and actionable picture of what’s happening and why. And that’s where a self-healing IT system starts.

The Future Is Within Reach

By ingesting observability data, applying AI to analyze that data and create insights around root causes, then leveraging automation in a closed-loop, dev teams can not only see what’s happening but also take action on it for remediation. When this closed-loop process is done intelligently at machine speed, that’s when self-healing becomes a reality.

The idea of a self-healing IT infrastructure doesn’t have to be a distant vision. In fact, the democratization of cloud computing and advanced data science have put the required observability technology within reach of teams of any size with any budget. When artificial intelligence and observability come together, DevOps practitioners can operate less and innovate more.

November 18, 2020
AIOps and the Myth of ‘Move Fast and Break Things’

Observability through AIOps enables developers to find problems faster by cutting through the noise

Years and years ago, a little-known entrepreneur named Mark Zuckerberg uttered the phrase, “Move fast and break things,” to his development team. Since then, dev teams have stuck to this “move fast and break things” mantra, where fast development far outweighs dealing with a few bugs down the road. Under this mindset, finding and fixing problems is essentially the last step in the development process.

The Problem With Breaking Things

This way of working has served dev teams well over the years as consumers continue to demand speedier innovation. But as end user expectations around deliverability speed rise, so do expectations around service reliability. Customers now want better products faster. Not only does the “break things” way of development lead to extra work for the dev teams down the road, but it also degrades customer trust over time as issues, from a buggy user experience to major security concerns, pop up once the product is already shipped.

Even Zuckerberg realized this years later, admitting, “What we realized over time is that it wasn’t helping us to move faster because we had to slow down to fix these bugs and it wasn’t improving our speed.” He realized this so much so that the Facebook team officially changed their mantra to “move fast with stable infra” in 2014. Sure, Zuckerberg shouldn’t be our role model after everything that has happened, but if he can realize the need for stable infrastructure, so should we.

So, how do dev teams meet growing end user expectations while cleaning up the now hurried development process for more stable systems?

Observability With AIOps and the Development Process

This is where observability strengthened by AIOps (artificial intelligence for IT operations) comes into play. Observability—or simply making data readily available from a system that you want to monitor—enhanced through AIOps allows dev teams to automate their monitoring. This automation can include applying AI/ML algorithms to all data; eliminating noise; detecting anomalies; correlating relevant metric anomalies, traces, changes and log events triggered by incidents; using contextual data to surface incidents; and identifying probable root causes of those incidents. This gives the teams a more complete picture of what’s happening in their systems, where the bugs are and how to resolve issues quicker, ultimately speeding the development processes overall.

Through observability with AIOps, dev teams can actually build systems faster by finding problems earlier, cutting out the time-consuming process of retroactively fixing mountains of problems after the product has already shipped. On top of that, finding these problems earlier ultimately leads to more reliable services—and reliable services mean happier customers. When the product actually works the first time around and there’s a system in place to find and fix evolving issues, there’s less legwork to make customers happy. It’s a win-win.

Observability with AIOps also means freeing developer time and effort to focus more on innovation and less on operations. And who doesn’t want to work on higher impact projects more often instead of spinning their wheels making sure the work they’ve already done is actually functioning correctly?

We Need to Stop Stalling

It’s easy to get stuck in your ways, but that is the only reason dev teams haven’t improved their processes to streamline workflows and build better products. Simply put, we have to stop stalling on making this change because, finally, moving fast doesn’t have to mean breaking things.

We can continue to move fast while building products that are better quality and keep customers happier if we change our mindset around the development process. As developers, it’s our responsibility to evolve with the best strategies for our team, customers, and business—and we now have a way to do so by embracing observability with AIOps.

September 30, 2020
DevOps and AIOps: Better Together

DevOps is an attempt to scale technology with humans but AIOps is the ultimate answer.

In today’s world, where so many essential business tasks have become digitized, IT teams must deal with constant change while ensuring zero downtime.

The irony is that, although IT has become business-critical, the productivity and agility of the people building and supporting the customer experiences has plummeted. This has serious consequences for businesses. When your apps and services malfunction, you lose customers and revenue drops. When IT teams focus mostly on fixing emergencies, agility suffers and innovation shrinks. You need seamless operations in order to grow, compete and thrive.

Historically, IT teams grew along with technology, beginning with the mainframe through distributed computing. But, the advent of virtual computing led to a world of microservices and ephemeral logic. Now companies simply generate too much data for humans to monitor and understand manually and with legacy tools.

This opened the floodgates for an AI-led operations evolution, known as Artificial Intelligence for IT Operations (AIOps). By streamlining and automating IT monitoring, AIOps has helped IT Ops teams regain control over IT environments, quickly detect and repair problems, and prevent outages.

DevOps: Back to the Future?

However, there’s recently been a shift in technology operations to once again attempt to scale technology with humans. The DevOps movement has placed the ownership, support and success of their services on the developers that write the code. This has decentralized technology operations teams that primarily operated with little context and unclear escalation paths, creating several smaller teams of DevOps engineers.

DevOps teams work together, each on their own microservice, toward the company’s desired customer experience and business goals. Depending on the operations model, there can also be a dedicated site reliability engineering (SRE) team or individuals on the DevOps team with the primary SRE role of monitoring observability data to discover potential issues. In this scenario, SREs feed the insights they uncover back into the development cycle to correct and strengthen the reliability and scalability of the DevOps teams’ services.

But, as incidents occur in real-time, the challenge remains with SREs and DevOps teams to achieve insights and awareness across their applications, infrastructure and, ultimately, business services. To understand where incidents are occurring and what impact they’re having on the services and customers, it’s necessary to surface important events from the noise, understand the relationships between alerts and obtain the context needed to engage the right teams and people.

This leads to another challenge: Getting the right people to respond and resolve incidents before there’s any business impact. Each DevOps team has its own responsibilities and tools. Often, DevOps teams don’t communicate except through their APIs. When incidents occur that need immediate attention from multiple people from multiple geographically-dispersed teams, how do you contact them and get them to collaborate, considering they have complex and different on-call schedules and escalation processes?

As companies continue to transform with a customer experience focus and a digital-first mentality, they realize the seriousness of these challenges, the magnitude of their monitoring data and its negative effect on their business. Part of transforming their business is transforming their technology operations: Enter AIOps with the light to shine on DevOps processes, leading to a manageable, efficient and profitable future.

What AIOps Brings to DevOps Teams

AIOps provides a unique solution to address operational challenges and cover every aspect of your service assurance strategy and business.

Bottom line: You need to free people to focus on mission-critical tasks and empower them to build better services for better customer experiences—not scale them along with technology in an attempt to operate it.

AIOps allows you to scale by integrating with the tools and infrastructure you’ve invested in over the years and by adding critical layers of AIOps (intelligence) throughout. By applying patented AI and machine learning algorithms to your observability and monitoring data, AIOps learns your environment’s normal behaviors and generates alerts accordingly. This must be done at the edge, close to where digital services produce vast amounts of data.

Based on the anomalies generated both locally and environmentally, AIOps will then surface and correlate important alerts from all sources into actionable, contextual insights. A true, comprehensive AIOps solution also identifies root causes and impacts, and prescribes potential solutions based on previous resolution steps and feedback. This all occurs in a virtual workspace, where team members can collaborate, visualize and provide feedback.

AIOps notifies and empowers the right people to take the right action. It simplifies and understands complex team structures, engagement methods, on-call schedules and escalation paths, ensuring the right people are engaged and collaborate, even when they’re located around the globe.

When a situation or incident is surfaced, it is sent to users in real-time. Based on the insights derived from the underlying data and machine learning applied, AIOps knows the exact teams and people that need to take action, and the ones that need to be informed. These users have the context they need to respond, share a consistent view and stay in sync throughout the lifecycle of the incident. Once the incident is resolved, a streamlined post-mortem speeds up future response, by using similar incidents and predictive analysis to spot repetitive and future problems.

AIOps: A Key Ingredient for Effective DevOps

The only way to scale with the technology being created in today’s and tomorrow’s world by DevOps teams is with AI. It frees humans to focus on creating and improving the customer experience that drives maximum profitability for your business.

May 7, 2020