Metrics are important for any manager seeking to continuously improve critical work processes and the resulting work-product.
That’s why DevOps leaders need DevOps metrics. With the right ones, those leaders can guide their organization’s adoption of DevOps best practices—progressively optimizing staff productivity, business agility and customer experience.
But, what are the right metrics for DevOps success? And, what are the wrong ones?
These next few blogs will offer field-proven guidance on DevOps metrics. Much of the material in these upcoming blogs can be found in the book, “DevOps for Digital Leaders,” which I co-wrote with Kieran Taylor and Peter Waterhouse. And I’d love for you to comment below so we can all share lessons learned about how to effectively measure DevOps maturity—and, just as important, how not to measure it.
Change Your Metrics, Change Your Culture
DevOps leaders tend to struggle with metrics because DevOps entails changes that are both operational and cultural—and span different teams or functional areas, including development, test and operations. So inexperience may cause leaders to adopt new ones that are counterproductive—or, as is often the case, continue using traditional ones that don’t align well with the cross-functional nature of DevOps objectives.
Three pitfalls have proven to be especially common in the transition to DevOps:
- Vanity metrics. These appeal to the “hero” mentality of individual coders, rather than the collaborative process goals of cross-disciplinary DevOps and continuous delivery teams. They include total lines of code produced and total function points created. If incentives are linked to these metrics, developers can focus too much on volume at the expense of other essential activities such as refactoring and design simplification.
- Those that promote interteam conflicts. Metrics that pit teams against each other undermine critical DevOps behaviors—such as code-sharing, mentoring and constructive peer reviews. So, while gamification may be useful in other areas of the business, Agile leaderboards that rank teams based on the prevention of extraneous deployments and changes can actually work against DevOps culture.
- Traditional metrics. Many DevOps leaders persist in utilizing traditional/incumbent ones such as mean time between failures (MTBF) and the ratio of full-time equivalents (FTEs) to servers. But while these may have been useful in the past, they can work at cross-purposes with DevOps transformation. After all, with faster delivery of services, some failure is to be expected. And as IT improves its responsiveness to the business, it must look past costs alone to total economic impact.
What Makes a DevOps Metric Useful?
While future blogs will propose some specific metrics that DevOps leaders should find useful in their efforts to bring about cultural transformation, here are a few general guidelines to consider:
They must be obtainable.
A metric won’t do you any good if you can’t actually capture it. So you need DevOps tools that help you capture the metrics you need. You may also have to measure some attributes indirectly. For example, rather than measuring employee morale directly, it may make sense to use staff retention rates—which you can readily capture—as your performance indicator.
They must be reviewable.
Any metric you choose to employ as part of your DevOps leadership effort must be able to stand up to rigorous scrutiny in a business context. Your credibility as a leader depends on your ability to communicate its relevance to the stated values and mission of the team.
They must be incorruptible.
It’s never a good idea to promote metrics that can be influenced by individual team members or any group as a whole. You should also be on guard against metrics remaining entrenched—despite the fact that they may inhibit change—because bonuses are linked to SLAs associated with them.
Metrics must be actionable.
Useful metrics must enable DevOps leaders to make better decisions about workflows, incentives, policies, training, tools or some other “lever” of transformation. Not every such decision will be a game-changing one—but incremental gains in knowledge lead to incremental improvements in process, which collectively do change the game.
In my next blog, I’ll discuss the metrics that DevOps leaders are using today to successfully drive the digital transformation of their businesses. In the meantime, if this is a topic that interests you, be sure to request a free copy of my book, “DevOps for Digital Leaders.”
Surveys such as Puppet Labs’ 