devopsy.kinsta.cloud

Tag: KPIs

What to do About AI’s Forced Rethink of Reliability in Modern DevOps
For years, reliability discussions have focused on uptime and whether a service met its internal SLO. However, as systems become more distributed, reliant on complex internet stacks, and integrated with AI, this binary perspective is no longer sufficient. Reliability now encompasses digital experience, speed, and business impact.

For the second year in a row, The SRE Report highlights this shift. DevOps and SRE leaders across industries are redefining reliability, with AI accelerating change by revealing the limitations of legacy monitoring and requiring teams to link technical signals to real-world outcomes. Traditional availability metrics continue to fall short because they fail to capture how users actually experience reliability.

From Uptime to Experience-Driven Resilience

This year’s SRE Report reinforces that slow performance is as critical as downtime, with nearly two-thirds of respondents considering performance degradations as serious as outages. This is good news as users do not distinguish between down and slow, instead perceiving both as a disruption.

According to the report, “Speed is now one of reliability’s clearest trust signals.” Fast performance builds reputation, while slow performance quickly leads to lost conversions, customer churn, and diminished trust. Resilience is now defined not only by incident survival, but by maintaining acceptable user experience under real-world conditions such as high load, congestion, and third-party failures.

AI accelerates this transition. AI-driven features, agentic workflows, and LLM-based applications introduce new latency paths and probabilistic behaviors. A component may be “up,” but the system may still provide a degraded or unexpected experience to your users. In this context, uptime metrics may produce data, but they do not produce insight.

Where Legacy Monitoring Breaks Down

Traditional monitoring tools were designed for systems fully controlled by internal teams. Today, modern digital experiences rely on:
- Third-party providers and managed services
- SaaS platforms and APIs
- Internet routing, CDNs, and millions of connected devices
- AI/ML components whose behavior changes over time
The SRE Report highlights that most teams still rely heavily on dashboards and alerts to detect performance issues. While familiar and trusted, these tools are good with the “what” but not so much the “why”, especially when the root cause lives outside the application boundary.

This gap becomes even more pronounced with AI systems. Only a small minority of respondents say they feel highly confident in monitoring the reliability of AI/ML components. For most teams, AI remains a black box, introducing new failure modes, subtle regressions, and cascading performance issues that legacy alerting was never designed to catch.

AI not only introduces new reliability challenges but also reveals the fragmentation in current observability methods. Siloed metrics, disconnected tools, and static thresholds cannot keep up with systems that learn, adapt, and rely on changing external conditions. In other words, AI is best poised to amplify whenthe underlying data is connected and integrated.

AI as the Integrator and the New Operating Model for Reliability

Increasingly, AI is being used to:
- Correlate performance, network, and user experience signals
- Surface patterns humans would miss across tools and domains
- Assist during incident response with context-rich suggestions
But AI only helps if someone owns it. Teams need people who can lead AI systems in production—setting guardrails, validating what the system suggests, tuning it over time, and being accountable when it’s wrong. That’s the new job: not “watch dashboards all day,” but guide the automation and keep it effective as the environment changes.

The report also warns against expecting automatic benefits. AI shifts the work: maintaining model drift, validating recommendations, governing agent behavior, and measuring impact. The bigger question is whether reliability work is changing business outcomes, not just engineering metrics.

Connecting Reliability to Business Outcomes

The most significant gap in the data is organizational, not technical. Only about a quarter of teams consistently assess whether performance improvements impact business metrics such as revenue or NPS. Even fewer quantify the financial cost of downtime or severe slowness.

This is a missed opportunity. The report is clear: when reliability is expressed in business terms, it gains influence. Teams that quantify the cost of delay can prioritize more effectively, justify investment, and elevate reliability from an operational concern to a strategic one.

AI can assist in this area as well. By correlating user experience data with business KPIs, AI-driven analytics help answer key executive questions:
- How much revenue does a 500 ms slowdown cost?
- Which customer journeys are most sensitive to performance?
- Where should we invest to protect trust and growth?
When reliability becomes a shared language between engineering and leadership, it shifts from being a cost center to a competitive advantage.

What This Means for SRE and DevOps Leaders

AI is prompting a reevaluation of reliability, though the goal of building trust remains unchanged. What is changing is how trust is established and measured.

For modern SRE and DevOps leaders, that means:
- Expanding reliability beyond service level objectives to speed and experience level objectives
- Acknowledging that internet stack and AI dependencies are first-class reliability risks
- Using AI thoughtfully to amplify integrated signals, or integrate edge cases
- Tying reliability work directly to customer and business outcomes
The 2026 SRE Report makes it clear that reliability has moved from the server room to the boardroom. In an AI-driven environment, reliability is how trust is earned, and the teams that understand that will be the ones that scale with confidence.
February 20, 2026
Survey Surfaces Disconnect Between DevOps Metrics and Business KPIs

A survey of 418 DevOps professionals finds that while DevOps teams closely monitor and observe the performance of applications, not nearly as many are able to correlate the value of those efforts to the business.

Conducted by Catchpoint, a unit of LogicMonitor, the survey finds more than two-thirds (67%) believe application degradations are either always or often as important to the organizations as outright downtime. However, only 26% directly evaluate business metrics after any improvement in application performance is achieved, compared to 32% that sometimes make that evaluation.

In fact, only 22% have any type of formal financial model for tracking actual downtime costs and even fewer (21%) have formal key performance indicators for tracking business metrics, the survey finds.

Leo Vasiliou, director of product marketing for Catchpoint, said the survey makes it clear that there remains a significant disconnect between the metrics that DevOps teams typically track and their relevance to the business. Most organizations are all too well aware this issue exists, but not enough political will exists to bridge that gap, he added.

As a result, many organizations are not correlating efforts to improve application performance to an actual return on the investment that was required, noted Vasiliou.

Overall, the survey finds 67% of respondents make use of dashboards and alerts to monitor applications, with 54% actively making use of synthetic probes and testing. Less than half (45%) are using passive monitoring tools such as real-user monitoring (RUM), while 50% use load testing/capacity planning tools (40%.

Less than a quarter (24%), however, are tracking metrics such as service level objectives (SLOs) or latency/error budgets and even fewer (22%) have any ability to detect anomalies using artificial intelligence (AI).

On average, 34% of the work respondents do could be classified as toil, the survey finds.

Additionally, 42% also noted their organization has a tolerance for any planned disruption, such as conducting chaos engineering tests.

Among the small percentage of respondents that work for organizations that have adopted AI (18%), just under half (49%) report seeing a reduction in toil, compared to 16% seeing an increase.

Just over a third (34%) said purchased third-party AI capabilities (34%) versus 33% that built their own. A third (33%) also report using free or open source AI tools, the survey finds.

A full 60% of those respondents are optimistic about AI, but only 42% said they are moderately confident in machine learning/AI versus 13% that are extremely confident.

At the same time, 65% of respondents spend less than two hours a month on learning new skills, which suggests that many DevOps teams are spending most of their time trying to optimize application environments using tactics and techniques that with each passing day are becoming more antiquated.

Each organization will need to determine the degree to which improving application performance provides meaningful value to the business, but software engineers are likely to continue paying close attention to any set of metrics they can directly influence. The challenge, and opportunity, is to get the rest of the organization to understand the importance of those metrics as it pertains to an actual impact on the bottom line that business leaders appreciate.

February 2, 2026
Why Up to 70% of SRE Initiatives Stall Before They Scale — and How to Break the Plateau
Over the last decade, site reliability engineering (SRE) has shifted from a niche Google practice to a mainstream aspiration for enterprises modernizing their technology operations. Yet despite this momentum, a substantial number of organizations — in some industry analyses, as many as 70% — struggle to scale SRE beyond the initial launch phase. Most companies start strong: They assign a few engineers the SRE title, implement some monitoring tools and hold one or two incident postmortems. But then progress slows. The cultural shift stalls. Engineering teams become skeptical. Leadership moves on to the next initiative. Eventually, SRE becomes a title without the discipline to support it.

This plateau is not due to a lack of interest — it is due to a mismatch between the intention and actual implementation of SRE.

When SRE is Adopted in Name Only

A surprising contributor to stalled initiatives is the widespread habit of rebranding existing production support or teams. These teams often continue doing what they have always done — reactive incident response, manual change approvals or tool administration — without practicing core SRE principles such as SLO-driven operations, reducing toil, error budget-based decision-making or engineering automation.

This ‘title-first, principle-later’ approach creates an illusion of maturity that delays real transformation. Organizations believe they have SRE, but reliability outcomes don’t improve because the underlying mindset hasn’t changed. Even worse, engineers quickly recognize the mismatch, creating cynicism and resistance.

When Legacy Authority Becomes a Modernization Bottleneck

Another underexamined blocker emerges from within: Long-tenured individuals in positions of influence whose skill sets have not evolved alongside today’s reliability and cloud-native practices.

As these individuals often hold senior authority or control critical processes, they unintentionally slow progress by resisting changes to ownership models, blocking automation in favor of manual controls, discouraging experimentation, avoiding new tooling or cloud-native concepts and promoting old operational habits that conflict with the SRE mindset.

No organization intentionally designs this bottleneck — but it is common, and it is one of the reasons initiatives quietly stall.

Why So Many SRE Initiatives Plateau

Below are the most common failure patterns observed across organizations that attempted but struggled to scale SRE.

Lack of Clear, Top-Down Vision

SRE is fundamentally a leadership-driven function. Without a clear, transparent and consistent executive vision — translated into simple messaging, measurable business outcomes and a multiyear roadmap — teams often default to their old ways of working. When vision is absent, fragmented interpretations emerge: Some think SRE is a tool replacement project, some treat it as a production support upgrade, some assume it is DevOps 2.0 and some see it as an operational gatekeeping role. Lack of unified direction leads to misalignment, which stalls progress before it truly begins.

Misaligned Incentives Between Leadership and Engineering

Leadership may seek higher uptime, lower incident volume and fewer customer escalations. Engineers may want less toil, rationalized on-call responsibilities and automation. Yet neither side succeeds unless both share a unified, measurable reliability mission. When incentives and KPIs differ, momentum breaks.

Cultural Resistance and Fear of Change

Cultural resistance is often underestimated. Teams fear losing autonomy, being judged for outages and exposing gaps through blameless practices, face job insecurity due to automation and are reluctant to move from reactive to proactive work. For an SRE transformation to succeed, teams must trust the process and believe the goal is empowerment — not redundancy.

Over-Focus on Tools Instead of Principles

Tooling is valuable but not foundational. Many organizations start SRE by buying a new observability platform, building dashboards and implementing incident tooling. These are important, but without SLOs, error budgets and ownership clarity, the tools remain underutilized. The initiative becomes a tooling project, not an engineering practice.

Absence of a Measurable Maturity Model

Without a maturity baseline and ongoing measurement, teams cannot prove progress or detect stalls. SRE maturity must evolve in stages — not be treated as a binary — we have SRE/we don’t. Organizations without maturity tracking often struggle to show value, which weakens long-term sponsorship.

How to Break the Plateau and Build a Sustainable SRE Practice

Establish a Clear, Top-Level Reliability Vision

Every successful SRE program begins with explicit leadership sponsorship that answers: Why are we doing this? What business value will SRE unlock? What are the expectations for the next 12–36 months? How will we measure success? This vision must be communicated consistently — not once.

Build Cross-Level Motivation With Shared KPIs

SRE only thrives when leadership and engineering teams share the same metrics and incentives. Effective KPIs include:
- SLO adoption rate per service
- Error budget burn trends
- MTTR and incident response effectiveness
- Toil reduction targets
- On-call health metrics
- Automation adoption
These KPIs must be reviewed by both leaders and engineers, creating shared accountability instead of pressuring one side.

Start With Small, Value-Driven Wins

Large-scale transformations rarely succeed without early proof. Teams should begin with small, visible improvements that create immediate value — both for leadership and engineers — examples include eliminating a top recurring incident, reducing alert noise by 40% in two weeks, implementing a first SLO for a critical service and automating 2–3 high-toil tasks. These wins build credibility, reduce skepticism and show that SRE produces tangible outcomes — not overhead.

Invest in Training and Enablement

SRE requires foundational knowledge — SLOs, incident management, automation, observability and cloud-native reliability patterns. To build this knowledge, organizations must provide hands-on workshops, brown-bag sessions, paired engineering, internal SRE office hours and platform runbooks with shared libraries. Such training reduces fear, accelerates adoption and builds engineering confidence.

Address Legacy Bottlenecks Transparently

Organizations must proactively identify and address outdated processes or authority structures that hinder modernization. This should be handled respectfully but decisively — either through upskilling, role redesign or clarified ownership models. Ignoring bottlenecks is one of the fastest ways for initiatives to stagnate.

Embrace SRE as a Change-Management Program, Not a Technical Project

SRE changes how teams collaborate, deploy, operate, measure and respond — making it a cultural transformation. Successful teams treat it as long-term change management, behavior redesign, leadership alignment and engineering empowerment. Tools help, but culture determines success.

Continuously Measure and Publicize Maturity

Use a simple maturity model (e.g., Level 0 to Level 3) that tracks SLO coverage, operational toil, engineering automation, incident response discipline and reliability outcomes. Share progress regularly. Consistent visibility reinforces sponsorship and ensures that the initiative never quietly stalls.

Conclusion

SRE succeeds when organizations understand that it is not a role, not a toolset and not a renamed operations team. It is a structured, engineering-driven approach to reliability that requires leadership alignment, cultural transformation and measurable progress.

Organizations that overcome the plateau do so by starting small, demonstrating value early, empowering teams with training, creating shared KPIs, addressing internal blockers and tracking maturity continuously. SRE is a long-term investment — and for organizations that commit to it with clarity, transparency and discipline, the payoff is stability, customer trust and operational excellence.
November 26, 2025
Unlocking Better DevEx: Four Areas to Improve and KPIs

In the constantly changing world of software development, the concept of developer experience (DevEx) is now nearly mainstream. DevEx, as most developers know these days, refers to the overall experience developers have while working on a project, including the tools, processes and environment.

Therefore, a well-designed DevEx can drive a happy, productive and efficient development team. However, assessing the current DevEx to improve it isn’t as easy as you might think. Forcing a lead developer and their team to document how they spend every moment of the day to inform these assessments on DevEx is only going to backfire.

After all, filing and counting Jira tickets is not going to improve any developers’ experience. If anything, things such as Jira tickets can be used as a legacy productivity assessment but not an experience assessment. But in truth, neither Jira tickets nor even the total number of products developed by an engineer, are going to give you a real assessment of productivity if you want to examine quality alongside quantity.

Therefore, DevEx can also be a better way to examine productivity, by further identifying how developers feel, think and value their work. Uncovering these can lead to faster project completion, improved code quality and reduced employee turnover. Companies that prioritize DevEx, such as Google and Netflix, have seen remarkable improvements in productivity, which serves as a testament to its importance.

To emulate Google and Netflix with your DevEx strategy, you need to identify where to focus on improvements. I suggest focusing on accessibility, discoverability, usability and credibility. By examining these four areas, we can determine if developers are being hindered by cumbersome tools or complex processes and how we can work together more efficiently and enjoyably.

However, optimizing DevEx comes with many challenges. From quickly changing platforms and user expectations, developers face numerous hurdles that can hinder day-to-day experiences. Another challenge is the lack of standardized guidelines within many enterprises and across the industry at large. With new technologies like AI emerging by the day, and even legacy solutions constantly evolving, it can be hard to stay up-to-date on best practices for developers. Let’s hone in on our focus areas above, to unlock and optimize a better DevEx.

To Elevate Your DevEx, Focus on These Main Areas

Improving Accessibility: It isn’t only about developer tools that work, it’s also important to ensure everyone has access and it’s easy to get in and out of each tool. Sometimes gaining access to a developer tool can be a huge time suck. This could be because it was misconfigured or the person who configured it did it with the wrong user in mind.

Streamlining Discoverability: Finding all the tools and information you need for a development project is vital. Developers need to be able to easily discover all the things they want to view at different parts of their development journey. At different stages of the planning to post-production journey, developers will want to be able to find different information easily, with the context they need in front of them.

Using an internal developer portal (IDP) is one great way to centralize where developers find and discover tools, resources and documentation. An IDP simplifies the entire developer ecosystem for engineers so they can use it as self-service alongside their development journey. It has all the tools in one place, so developers can get a full picture of everything they have to use. IDPs can also streamline development workflows and enhance productivity. One key benefit of an IDP is improved collaboration and knowledge sharing among developers.

Easing Usability: It’s not just access to the tools and features, the usability of those tools matters. It’s pretty common for conversations and collaboration among developers to happen in tools or dialogue channels that they weren’t designed to – simply because it seems to work the best for everyone. One way to ensure all of the features are accessible is to use feedback tools at the end of crucial points in the journey. This way you can find out how easily your developers can find the relevant information or feature.

Platform engineering approaches can support developers by providing them with internally standardized tools, infrastructure and support they need to build and deploy applications efficiently. The key here is reusability and platform engineering is based on creating the components and services needed to deliver applications that can be used repeatedly.

Validating Credibility: Every tool a developer uses within your enterprise must be credible. Whether it’s bugs, troubleshooting errors, uptime, or even the ability to ensure that the data within the tool is always accurate, developers can easily sway from thinking that your tool is credible or not, and subsequently, this can also affect how they think about the overall team/organization

Enhancing these four areas will also require KPIs, and these are some ideas for creating baselines to measure and improve against.

API Response Time and Efficiency: Speedy responses mean less frustration and more productivity for developers.

Bug Fix Rate: The bug fix rate indicates how quickly and effectively a team addresses and resolves software bugs. Monitoring this metric can help teams prioritize bug fixes, improve product stability, and enhance customer satisfaction.

Contribution analysis. Assessing individual contributions to a team’s backlog can help unearth the trends that block the optimization of that team’s capacity. Using this kind of insight, team leaders can manage clear expectations for output and improve performance, identify opportunities for individual upskilling or training, evaluate role distribution within a team and make adjustments.

Developer Velocity Index: A benchmark that measures an enterprise’s technology, working practices and organizational enablement and benchmarks them against peers. This comparison helps unearth specific areas of opportunity, whether in backlog management, testing, or security and compliance.

Developer Satisfaction Score: By regularly surveying your developers, you can check in with them to see how satisfied they are with a platform, tool, service, or feature. Regular surveys about the overall developer experience and correlating this with employee turnover and retention rates are essential.

Onboarding time: Ensure that the onboarding process is swift and that limited time is wasted during this process.

Platform stability: Uptime and reliability should be monitored, ensuring developers can focus on coding and solving problems.

Talent capability score: Based on industry-standard capability maps, this score summarizes a specific engineer’s knowledge, skills and abilities. It can present coaching and upskilling opportunities and inspire a rethink of talent strategy.

Using these KPIs doesn’t guarantee the transformation of your DevEx. However, they do help establish baselines. Therefore, in addition to starting to use them, it’s important to bring management up to speed with details about the software development process and how it’s evolving.

In addition, as you assess your DevEx, many enterprises will need to refine their tech stacks. Start by identifying friction points and bottlenecks within that tech stack.

No matter how comprehensive a plan to improve DevEx is, there is still no silver bullet. But these focus areas and KPIs will provide structure to your approach.

July 24, 2024
Survey Surfaces Lots of Software Supply Chain Insecurity

A global survey of 900 application security professionals finds nearly two-thirds work for organizations that have had their software supply chains compromised in the past two years, with 18% being victimized in the last year.

Conducted by Checkmarx, a provider of application security testing tools, the survey also finds that 100% of respondents are aware of a breach of their software supply chain that occurred at some point in the past.

Not surprisingly, the survey finds three-quarters of respondents (75%) are now either very concerned (39%) or concerned (36%) about the security of their software supply chain.

Additionally, well over half (57%) said software supply chain security is a top or significant area of focus, with 54% planning to use or investigate some type of solution. However, only 7% have acquired and implemented a tool or platform to specifically secure their software supply chains, the survey finds.

On the plus side, half (50%) are now requesting software bill of materials (SBOMs) from entities that provide their organization with software, but fewer than half (47%) said their organization can effectively operationalize SBOMs, the survey finds.

Renny Shen, vice president of portfolio marketing for Checkmarx, said the challenge is the scope of the effort required to secure software supply chains is significant. In addition to SBOMs and deploying tooling to discover vulnerabilities and malware, DevSecOps teams also need to, for example, embrace zero-trust initiatives to secure access to their software supply chain.

There is no single tool or platform that enables DevSecOps teams to achieve that goal, he added.

In general, securing software remains challenging because even if a vulnerability is discovered a DevSecOps team may not be able to fix it, noted Shen. For example, well over half (56%) of the applications deployed are based on open-source code packages that organizations are dependent on external maintainers to update.

On the plus side, however, software supply chain security has become a C-level discussion, said Shen. The challenge is that many organizations have yet to define a set of key performance indicators (KPIs) that enable them to ensure best DevSecOps practices are being adopted. Right now, there is not enough emphasis on application security so developers will continue to focus most of their time and effort on writing new code rather than fixing existing code, he noted.

Unfortunately, most organizations don’t prioritize application security so until there is a fundamental shift in incentives it’s not likely developers will spend more time remediating vulnerabilities. There has been an effort to shift more responsibility for security further left toward developers, which has been met with mixed success. Developers generally have access to more tools but unless those tools are surfacing issues as developers write code it’s not likely they will have a meaningful impact on improving the overall state of application security.

Hopefully, the rise of generative artificial intelligence (AI) will soon make it simpler to find and remediate vulnerabilities by, for example, automatically creating the required patch. In the meantime, however, securing software supply chains requires a level of vigilance that despite best intentions remains difficult to achieve and maintain.

July 3, 2024
GitLab Strengthens Remote DevOps Management

GitLab this week moved to provide remote developers with access to a more secure continuous integration/continuous delivery (CI/CD) platform while at the same time adding a dashboard that makes it simpler to track the overall status of application development projects.

David DeSanto, chief product officer for GitLab, said GitLab Remote Development takes advantage of an instance of the GitLab CI/CD platform running on a Kubernetes cluster. That cluster is accessed via a Web-based integrated development environment (IDE) to eliminate the need for developers to create development environments on their local desktops.

That capability reduces developers’ time and effort on tasks other than code and gives organizations more centralized control over the intellectual property being created, noted DeSanto. Organizations will no longer have to worry, for example, about theft of a developer’s laptop that has sensitive data on it, he added. In addition, organizations will be able to more easily create “golden” images that can be consistently accessed by developers residing in multiple locations.

In addition to making it easier to centrally manage a distributed team of developers, GitLab is also adding a GitLab Value Streams Dashboard that provides senior executives with access to metrics that assess the overall health of an application development project. While GitLab has always provided metrics to DevOps teams, DeSanto said this dashboard, available as a private beta, makes it possible to identify, for example, bottlenecks that might result in delayed delivery of an application.

Value stream management (VSM) is, of course, not a new idea. The issue that DevOps teams are trying to come to terms with is whether they need to adopt a dedicated VSM platform. The other option is to simply provide a set of dashboards that enable senior IT and business executives to track projects’ progress without deciphering a set of much lower key performance indicators (KPIs). Previously, the GitLab platform surfaced a lot of nitty-gritty details that don’t provide the context senior executives required, noted DeSanto. The Value Streams Dashboard metric surfaces KPIs based on DevOps Readiness Assessments (DORA) metrics from Google alongside other flow metrics.

These latest offerings are part of a larger case being made for an integrated DevOps platform that reduces the total cost of application development by eliminating the need for organizations to integrate various tools themselves. Each DevOps team can employ as many of the modules GitLab makes available as they need. Once invoked, they take advantage of the same data store as every other element of the GitLab platform. That capability is becoming more critical as organizations embrace platform engineering to reduce the cost of application development using DevOps best practices, said DeSanto.

It’s not clear how much interest there is in replacing legacy CI/CD platforms that, in some instances, have been in place for nearly a decade. However, as the pace at which applications are being built and deployed continues to accelerate, there is no doubt that organizations of all sizes will need to modernize application development processes.

January 25, 2023
Best of 2021 – Nine Pillars of DevOps Best Practices

As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the fourteenth in our series of the Best of 2021.

In a prior blog I explained my 27-factor DevOps assessment model which includes nine pillars of DevOps and three dimensions of people, process and technology.

The nine pillars represent categories of practices. In this blog, I include examples of practices for each of the nine pillars. In a future blog I will explain how to use these practices to conduct a gap assessment for your DevOps practices.

Leadership Practices for DevOps

• Leaders demonstrate a long-term vision for organizational direction and team direction.
• Leaders intellectually stimulate the team by encouraging them to ask new questions and question basic assumptions about the work.
• Leaders provide inspirational communication that inspires pride in being part of the team, says positive things about the team, inspires passion and motivation and encourages people to see that change brings opportunities.
• Leaders demonstrate support by considering others’ personal feelings before acting, being thoughtful of others’ personal needs and caring about individuals’ interests.
• Leaders promote personal recognition by commending teams for better-than-average work, acknowledging improvements in the quality of work and personally complimenting individuals’ outstanding work.

Collaborative Culture Practices for DevOps

• The culture encourages cross-functional collaboration and shared responsibilities and avoids silos between Dev, Ops and QA.
• The culture encourages learning from failures and cooperation between departments.
• Communication flows fluidly across the end-to-end cross-functional team using collaboration tools where appropriate (for example Slack, HipChat, Yammer).
• The DevOps system is created by an expert team, and reviewed by a coalition of stakeholders including Dev, Ops and QA.
• Changes to end-to-end DevOps workflows are led by an expert team, and reviewed by a coalition of stakeholders including Dev, Ops and QA.
• DevOps system changes follow a phased process to ensure the changes do not disturb the current DevOps operation. Examples of implementation phases include: proof of concept (POC) phase in a test environment, limited production and deployment to all live environments.
• Key performance indicators (KPIs) are set and monitored by the entire team to validate the performance of the end-to-end DevOps pipeline, always. KPIs include the time for a new change to be deployed, the frequency of deliveries and the number of times changes fail to pass the tests for any stage in the DevOps pipeline.

Design-for-DevOps Practices for DevOps

• Products are architected to support modular independent packaging, testing and releases. In other words, the product itself is partitioned into modules with minimal dependencies between modules. In this way, the modules can be built, tested and released without requiring the entire product to be built, tested and released all at once.
• Applications are architected as modular, immutable microservices ready for deployment in cloud infrastructures in accordance with the tenets of 12-factor apps, rather than monolithic, mutable architectures.
• Software source code changes are pre-checked with static analysis tools, prior to commit to the integration branch. Static analysis tools are used to ensure the modified source code does not introduce critical software faults such as memory leaks, uninitialized variables, and array-boundary problems.
• Software code changes are pre-checked using peer code reviews prior to commit to the integration/trunk branch.
• Software code changes are pre-checked with dynamic analysis tests prior to committing to the integration/trunk branch to ensure the software performance has not degraded.
• Software changes are integrated in a private environment, together with the most recent integration branch version, and tested using functional testing prior to committing the software changes to the integration/trunk branch.
• Software features are tagged with software switches (i.e., feature tags or toggles) during check-in to enable selective feature-level testing, promotion and reverts.
• Automated test cases are checked in to the integration branch at the same time code changes are checked in, together with evidence that the tests passed in a pre-flight test environment.
• Developers commit their code changes regularly, at least once per day.

Continuous Integration Practices for DevOps

• A software version management (SVM) system is used to manage all source code changes. (Git, Perforce, Mercurial, etc.)
• A software version management (SVM) system is used to manage all versions of code images changes used by the build process. (Git, Perforce, Mercurial, etc.)
• A software version management (SVM) system is used to manage all versions of tools and infrastructure configurations and tests that are used in the build process. (Git, Perforce, Mercurial, etc.)
• All production software changes are maintained in a single trunk or integration branch of the code.
• The software version(s) for supporting each customer release are maintained in a separate release branch to support software updated for each release.
• Every software commit automatically triggers a build process for all components of the module that has code changed by the commit. The system is engineered such that resources are always sufficient to execute a build.
• Once triggered, the software build process is fully automated and produces build artifacts, provided the build time checks are successful.
• The automated build process checks include unit tests.
• Resources for builds are available on-demand and never block a build.
• CI builds are fast enough to complete incremental builds in less than an hour.
• The build process and resources for builds scale up and down automatically according to the complexity of the change. If a full build is required, the CI system automatically scales horizontally to ensure the builds are completed as quickly as possible.

Continuous Testing Practices for DevOps

• Development changes are pre-flight tested in a clone of the production environment prior to being integrated to the trunk branch. (Note: “production environment” means “variations of customer configurations of a product.”)
• New unit and functional regression tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
• A test script standard is used to guide test script creation, to ensure the scripts are performing the intended test purpose and are maintainable.
• Tests are selected automatically according to the specific software changes. CT is orchestrated dynamically, whereby the execution of portions of the CT test suites may be accelerated, or skipped entirely, depending on how complex or risky the software changes are.
• Test resources are scaled automatically according to the resource requirements of specific tests selected and the available time for testing.
• Release regression tests are automated. At least 85% of the tests are fully automated and the remaining are auto-assisted if portions must be performed manually.
• Release performance tests are automated to verify that no unacceptable degradations are released.
• Blue/green testing methods are used to verify deployments in a staging environment before activating the environment to live. A/B testing methods are used together with feature toggles to try different versions of code with customers in separate live environments. Canary testing methods are used to try new code versions on selected live environments.
• The entire testing life cycle, which may include pre-flight, integration, regression, performance and release acceptance tests are automatically orchestrated across the DevOps pipeline. The test suites for each phase include a predefined set of tests that may be selected automatically according to predefined criteria.

Elastic Infrastructure Practices for DevOps

• The data and executable files needed for building and testing builds are automatically archived frequently and can be reinstated on demand. Archives include all release and integration repositories. If an older version of a build needs to be updated, then the environment for building and testing that version can be retrieved and reinstated on demand and can be accomplished in a short time (for example, minutes to hours.)
• Build and test processes are flexible enough to automatically handle a wide variety of exceptions gracefully. If the build or test process for a component is unable to complete, then the process for that failed component is reported and automatically scheduled for analysis, but build and test processes for other components continue. The reasons for the component failure are automatically analyzed and rescheduled if the reason for the failure can be corrected by the system; if not, then it is reported and suspended.
• System configuration management and system inventory is stored and maintained in a configuration management database (CMDB).
• Infrastructure changes are managed and automated using configuration management tools that assure idempotency.
• Automated tools are used to support immutable infrastructure deployments.
• Equal performance for all. The user performance experience of the build and test processes by different teams are consistent for all users, independent of location or other factors. There are SLAs and monitoring tools that ensure the user performance experience is consistent for all users.
• Fault recovery mechanisms are provided. Build and test system fault monitoring, fault detection, system and data monitoring and recovery mechanisms exist. They are automated and are consistently verified through simulated failure conditions.
• Infrastructure failure modes are frequently tested.
• Disaster recovery procedures are automated.

Continuous Monitoring Practices for DevOps

• Logging and proactive alert systems make it easy to detect and correct DevOps system failures. Logs and proactive system alerts are in place for most DevOps component failures, and are organized in a manner to quickly identify the highest-priority problems.
• Snapshot and trend results of each metric from each DevOps pipeline stage (for example, builds, artifacts, tests) are automatically calculated in process and visible to everyone in the Dev, QA and Ops Teams.
• Key performance indicators (KPIs) for the DevOps infrastructure components are automatically gathered, calculated and made visible to anyone on the team that subscribes to them. Example metrics are availability (uptime) of computing resources for CI, CT and CD processes, time to complete builds, time to complete tests, number of commits that fail and number of changes that need to be reverted due to serious failures.
• Metrics and thresholds for DevOps infrastructure components are automatically gathered, calculated and made visible to anyone on the team that subscribes to them. Example metrics are availability (uptime) of computing resources for CI, CT and CD processes, time to complete builds, time to complete tests, number of commits that fail and number of changes that need to be reverted due to serious failures.
• Process analytics are used to monitor and improve the integration, test and release process. Descriptive build and test analytics drive process improvements.
• Predictive analytics are used to dynamically adjust DevOps pipeline configurations. For analysis of test results, data may indicate a need to concentrate more testing in areas that have a higher failure trend.

Continuous Security Practices for DevOps

• Developers are empowered and trained to take personal responsibility for security.
• Security assurance automation and security monitoring practices are embraced by the organization.
• All information security platforms that are in use expose full functionality via APIs for automation capability.
• Proven version control practices and tools are used for all application software, scripts, templates and blueprints that are used in DevOps environments.
• Immutable infrastructure mindsets are adopted to ensure production systems are locked down.
• Security controls are automated so as not to impede DevOps agility.
• Security tools are integrated into the CI/CD pipeline.
• Source code for key intellectual property on build or test machines are only accessible by trusted users with verified credentials. Build and test scripts do not contain credentials for access to any system that has intellectual property. Intellectual Property is divided such that not all of it exists on the same archive and each archive has different credentials.

Continuous Delivery Practices for DevOps

• Delivery and deployment stages are separate. The delivery stage precedes the deployment pipeline.
• All deliverables that pass the delivery metrics are packaged and prepared for deployment using containers.
• Deliverable packages include sufficient configuration and test data to validate each deployment. Configuration management tools are used to manage configuration information.
• Deliverables from the delivery pipeline are automatically pushed to the deployment pipeline, once acceptable delivery measures are achieved.
• Deployment decisions are determined according to predetermined metrics. The entire deployment process may take hours, but usually less than a day.
• Deployments to production environments are staged such that failed deployments can be detected early and impact to customers isolated quickly.
• Deployments are arranged with automated recovery and self-healing capabilities in case a deployment fails.

What This Means

DevOps is a powerful tool that enables many benefits for organizations that use it. Achieving performance efficiently with DevOps depends on following best practices. By following the nine pillars of practices enumerated in this blog, organizations can achieve the performance potential that DevOps has to offer.

December 26, 2021
Allstacks Adds Free Dashboard to Track Software Dev Benchmarks

Value stream intelligence platform provider Allstacks today added a free live dashboard populated with anonymized data that DevOps teams can use to benchmark their own progress.

Hersh Tapadia, Allstacks co-founder and CEO, said the company is making this data available to make it easier for DevOps teams to uncover bottlenecks in their software development processes.

As part of that effort, the dashboard includes pre-configured benchmarks for the following:

How many days per week do benchmark teams write code?
What percentage of issues are planned versus unplanned for benchmark teams?
What are the cycle times per issue type for benchmark teams?
What types of code do benchmark companies produce?
What is the average commit size for benchmark customers?

Allstacks has committed to making 12 months of data publicly available on the dashboard and said it will update that data on an ongoing basis. Allstacks customers, however, can also access five years of data to compare historical trends.

Tapadia said Allstacks plans to add additional benchmarks to a dashboard that can collect data via the various application programming interfaces (APIs) that most DevOps tools now routinely provide. The overall goal is to increase transparency into software development processes, he said.

Allstacks’ value stream intelligence platform uses machine learning algorithms and artificial intelligence (AI) models to identify projects that are at risk of not meeting delivery goals. That data is then used to analyze the impact those missed deadlines are likely to have on various business objectives.

Value stream management platforms automatically take technical metrics, gathered via a DevOps process, and match them to a set of key performance indicators (KPIs) defined by business leaders. The challenge many organizations still face is making sure they have the tools in place to collect the appropriate technical metrics and then define the KPIs that truly matter to the business. Once those metrics are gathered and KPIs established, organizations are then able to apply advanced analytics to the massive amounts of DevOps data they collect.

The concept of value stream management traces its lineage back to lean manufacturing methods, which called for each step of a manufacturing process to be continuously measured. As software development has evolved from being a craft to a process that is highly automated using DevOps best practices, an appreciation for the value of monitoring the impact of missed software development deadlines on the business has grown as organizations realize how dependent they are on software.

As a result, more organizations realize there is a clear disconnect between their business goals and development processes, noted Tapadia. Business leaders are especially anxious to understand how a software development project delay might impact future revenue projections. Rather than being a set of tools employed mainly by IT leaders to track software development projects, value stream management platforms make it possible for other stakeholders to track KPIs spanning multiple software development projects. It then becomes easier to make informed decisions about how to reassign resources across various projects as different bottlenecks are encountered.

It may be a while before organizations achieve that level of transparency into their software development processes, but as software continues to ‘eat the world,’ it’s only a matter of time before value stream management becomes a required set of DevOps tools for every organization.

November 10, 2021
How to Optimize Data in Software Development

Data can drive significant business value for every company. Globally, we are collecting more of it than ever before, and it keeps growing—from 45 zettabytes in 2019 to a projected 175 zettabytes by 2025, according to IDC. However, companies are struggling to monetize all the personal, behavioral, engagement and transactional data from customers and operations and turn it into actionable intelligence to drive their digital strategies. Leveraging insights from your data is critical for success and the development of new digital solutions to better serve customers, grow sales, improve operations and more.

Data helps organizations develop the right software for the right customer at the right time. Data-driven organizations are 23 times more likely to acquire and retain customers than their peers, and are growing at an average of more than 30% annually, according to Forrester.

Organizations with a deep and dedicated focus on data and analytics tend to follow three common paths to successfully drive their software development initiatives.

Discovery Matters

It is nearly impossible to draw value, understanding, and insights from the massive amount of data businesses collect without a proper discovery system in place. Having a comprehensive process of collecting data from various databases and silos and consolidating it into a single source that can be easily and instantly evaluated is critical for any software development project.

During client engagements, data scientists perform deep-dive research into clients’ data pools and workflows to support client roadmaps. We recently partnered with a global e-commerce company to create a recommendation system for their customers. The discovery process allowed us to verify the feasibility of recommended products using their existing data before moving ahead with the project. Using machine learning, the team was able to clean and analyze information from multiple angles. With that knowledge and confidence, the rest of the development project proceeded quickly knowing the data was in place and the project was viable.

Align KPIs

Software development metrics help organizations and their partners track and measure how an application is performing and how effective the development team is in its work. IT organizations rely on a range of KPIs to fully understand software engineers’ progress, as well as gauge software quality including performance, engagement and user satisfaction. While each engagement is different and requires a different set of KPIs, ensuring that the organization and development team are on the same page is crucial.

During the e-commerce project, success was measured through conversion and user retention rates. Other KPIs that are regularly used include developer productivity and performance metrics like cycle time and team velocity, and software developer metrics like response time, usability and UX data, among others.

Deploy Agile

By using agile methodologies at all stages of development, developers are able to focus on business results, not just architecture and new technologies. A similar agile approach can also be applied to data management. When it comes to data science, it’s all about extracting useful information from raw, structured and unstructured data and implementing machine learning models to analyze vast amounts of information. This process requires more creativity and is a key driver for understanding potential risks and failures over the course of any given project.

With agile, data models and analyses are broken down into manageable bite-sized chunks that can be built and tested quickly—the same way software is developed. If the analytics do not yield the expected results, data scientists see the failures immediately and can correct their course. They could ask the team to analyze different data sets, adjust the models, tweak algorithms or make an educated decision about the project’s viability.

A multinational credit company took this approach to the next level when it used agile to develop a new, big data hybrid cloud platform. After anonymizing, aggregating and storing information in a data lake, the team quickly realized the database it was planning to use was limited to a certain number of partitions and entities. By running agile sprints and developing and testing the minimum viable product (MVP) continuously, the team was able to seamlessly switch to a new database that had more scale and functionality.

The value of data is only going to increase with time. Data-driven organizations embrace the use of data and analytics to gain unique business insights to support their software development projects and overall business. By focusing on the data in each and every development project, companies have a tremendous opportunity to gain a competitive advantage and grow their businesses.

August 26, 2021
CircleCI CEO Sees Growing Need for Developer Experience Engineers

There is no shortage of titles to go with all the tasks that make up a DevOps workflow, but given the critical role developers now play within any organization, it might be time for organizations to start creating a developer experience engineer (DXE) function within DevOps teams that is specifically tasked with increasing developer productivity.

CircleCI CEO Jim Rose said it’s become apparent there is more friction than ever in the application development and deployment process, as more complex applications are constructed, thanks, in part, to the rise of microservices.

A DXE, as outlined in a report published by CircleCI, would be responsible for implementing a common set of DevOps principles, maintaining the right tools and putting in place a cohesive set of standard processes that ensure developer success. Today, there are simply too many languages, frameworks, engineering styles and processes being employed within the same organization, and that conspires to reduce overall developer productivity, said Rose.

While software engineers are, today, addressing subsets of these issues, Rose said the time has come to assign these tasks to an engineering team that views itself as an advocate for developers by making available more responsive infrastructure.

As organizations of all sizes increasingly realize how dependent they are on software to drive digital business transformation initiatives, Rose noted that developer productivity is becoming a major corporate issue. Revenue forecasts and profit margins are now being directly impacted by the rate at which software is being delivered and updated.

Software engineers today routinely build DevOps pipelines for developers, but not much time and effort is put into optimizing the developer experience, noted Rose. That’s become not only a major productivity issue, Rose said, it has a material impact on the ability of organizations to attract and retain developer talent. One of the reasons so many developers prefer to work for smaller startups than large enterprises is because their daily work experience is simply better. Large organizations need to reduce the amount of friction developers experience if they want to be able to compete for that same talent, said Rose.

A DXE would also be more inclined to solicit input from developers on tool selection to best optimize productivity, noted Rose. Many software engineers today construct pipelines with little to no regard for how simple or hard it might be for a developer to employ tooling.

Organization today are, of course, starting to become more obsessed with measuring application development velocity. However, in many cases, simply making someone responsible for achieving a goal is going to result in more gains at a lower cost. Analytics tools have a role to play, but obsessing over key performance indicators (KPIs) may not generate the same result as quickly as simply giving an engineer a specific mandate.

As responsibility for application management continues to shift left toward developers, it should not come as a surprise that there is a lot more interest in automating the underlying infrastructure. The issue, at this point, is determining to what degree software engineers are going to achieve that goal versus forcing developers to seek out yet another platform that promises to do it on their behalf.

June 28, 2021