devopsy.kinsta.cloud

Tag: adoption

DevOps storytime: MTTR vs. Goodheart’s Law

I am not a fan of metrics. Wait. That’s not right. Let me try again. I am not a fan of singular metrics that stand by themselves. Nope. That’s still not right. I loathe singular metrics that stand by themselves as if they themselves are solutions to problems. Ok. That works. Let me explain.

Last week, I told a tale of a secret passageway into DevOps called Mean Time To Resolution (MTTR) and explained how the introduction of this one tiny metric can magically open the door to all things DevOps. Given that MTTR and cycle time are inextricably linked, any shop that is serious about MTTR will sooner or later find itself using DevOps practices.

Metrics + Crafty Humans = Doom

This is very different than two weeks ago when I told a different tale. A tale of how the very origins and roots of DevOps gives the DevOps movement an opportunity to do what ITIL could not; Change the relationship between business and IT. In the telling of this tale, rife with philosophical terms like reductionism and formalism, I quoted a brilliant English economist, Charles Goodheart, who noted that “when a measure becomes a target, it ceases to be a good measure”. Goodheart discovered that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” The layman’s explanation is that humans are crafty little buggers and will find ways to artificially hit any target because the quantitative systems can always be rigged.

So what is a DevOps voyager to do? Introduce the metric, get the change, and then proceed to watch devops die a slow death? No, my fellow DevOps devotees, that shall not be our fate. We can avoid this fate by knowing that halfway down the path of the secret passageway known as MTTR, there is a fork!

More Metrics to the Rescue!

Instead of introducing and presenting a flat and un-nuanced view of MTTR by itself, we can get a little crafty ourselves and find an appropriate hedge against Goodheart’s lay and instead present a weighted view of MTTR with a few other key metrics to allow for a more sophisticated view of the health of the IT factory floor.

Apply Weighted Averages – Weight the MTTR of defects and their associated resolutions by priority to intentionally skew the measurement towards the highest priority defects. Clinging to an absolute view of defect free software doesn’t work for a large code base with limited funding and time (aside from being scientifically impossible to achieve via godels proof)

Track Inbound Defect Counts – Track and present the inbound defect count, keeping the inbound defect count low is important, but pushing it to zero is more likely to be indicative of a shop that is more concerned with perfection than throughput

Track Outbound Defect Counts – Track and present the defect close count. keeping a negative inbound-outbound differential is a sign of a shop that is arcing in the right direction. Several weeks or months of positive differential is a warning sign that a quality mindset is slipping into dangerous territory.

Track Backlog Defect Counts – Track and present the count of defects in the backlog. A backlog of zero can be incredibly hard to maintain in a large enterprise with a large code base. In my experience 20 and 50 have been ominous numbers. Any more than 20 defects in an application and the backlog starts to become a fearsome beast that can only be tamed by stopping the factory. Any more than 50 and the incremental additions of any new defect seem trivial.

Track The Age Defects – Tracking the average age of defects in the backlog is critical for any presentation of MTTR. A spike in MTTR can actually be a good thing if it represents a wave of old defects being cleared from the back of the backlog.

Resist The Urge to Simplify

A five part metric begs the question of an balanced score card approach with a grading formula and despite my belief in the value of this tactic for availability and performance dashboards, I advise you to strongly resist the temptation to apply it here (otherwise, we are right back where we started with Goodheart’s Law. Damn you crafty humans!).

If you decide to bring the magical metric of MTTR into your shop, this type of reporting may be one area where starting small is quite possibly worse than doing nothing. Without the nuanced view, true visibility into the health of the shop isn’t possible. Worse than that, a flat view of just MTTR has the potential to mask real problems and raise false ones.

When you pull all of the metrics together in a single view, like in the above graphic, the real message becomes clear. The health of the factory floor isn’t tied up in a neat little bow by a single number. It isn’t even made perfectly clear by three, four or five numbers. The real health becomes clear in the trend. Is your shop doing better week over week? Month over month? Getting better every day is the name of the game, and MTTR (with a few other metrics) can be a lever that moves your IT world.

April 28, 2014
DevOps adoption can succeed where ITIL failed, if we let it
When I look at the future of DevOps adoption and acceleration I contrast it with what I feel was the failure of ITIL to really move beyond large enterprise IT shops. I am of the firm belief that DevOps has the legs to succeed and soar where ITIL sort of crashed and burned. OK if not crashed and burned, at least sputtered out. To understand why I feel this way, let me share a little personal history with you.

ITIL demands you to come to IT

I remember my first ITIL project. I was working at a large retailer and I was asked to help our IT department understand what this ITIL thing was and to help drive adoption of some of ITIL’s basic concepts. First on the list was to get a service catalog up and running. In the space of just a few weeks, I realized the main barriers to adoption and helped our IT executive team understand that getting IT consumers to understand, much less embrace an IT-centric taxonomy such as ITIL was not going to be an easy task.

This scenario points directly at one of the big reasons ITIL failed to achieve critical mass (i.e., adoption of the model outside of large enterprise IT) and also reveals why DevOps has a much better chance of success of achieving critical mass (as long as we ourselves don’t mess it up).

Mind you I’m not saying that ITIL is dead or that it has nothing to teach IT shops. I am saying that ITIL alone has not really changed much of anything inside of IT shops, let alone improved the reputation of IT with it’s consumers and customers.

When you contrast ITIL and DevOps several things really set the two approaches apart:
- ITIL makes it easy to blame the business for not talking IT’s language while DevOps continuously seeks to meet people on whatever terms they are capable of communicating in. DevOps inspires, while ITIL deflates
- DevOps is centered around improving the lives of IT workers everywhere while also improving bottom lines at the companies they work for. ITIL is centered around improving stability in the most efficient manner possible. I don’t know about you, but I just don’t see a room full of people, much less an entire organization getting excited or inspired by the concept of efficient stability. The only people likely to get sustainably excited by “efficient stability” are the accountants who can now start to push cost (i.e., people) out the door.
- ITIL at its core is reductive (or at least allows for reductive methods and approaches to grow unchecked). For those uninitiated in philosophy or how language influences our ideas and behavior, reductionism is when people essentially believe that a complex system is nothing more than the sum of its parts. This reductive approach at the core of the ITIL movement is what reinforced the division between Dev and Ops – because the concept of team chemistry has no place in ITIL given that the organization is nothing more than a collection of faceless workers who play out a prescribed role.
Reductive goals and processes, essentially reduce people and their efforts to numbers. The act of using quantitative targets as goals in large initiatives is a dangerous practice in general and grows more dangerous in larger groups and organizations – case in point the recent GM ignition failures that resulted in 13 deaths in an effort to save .57 cents per car affected by a defect.

Before you get excited, I’m not proposing that ITIL will kill people. I am stating that allowing an organization to be led by reductive philosophy allows a whole bunch of unintended consequences to come into play and often leads to holding dollars as singularly more important than anything else; sometimes even human life.

Whether or not you agree with my point that efficiency driven quantitative metrics are not inspiring, it’s kind of hard to argue with 40 years of scientifically proven fact that when “a measure becomes a target, it ceases to be a good measure”.

DevOps shops “live to serve”, while ITIL shops “serve to live”

DevOps contains a concept of craft that extends beyond efficiency. When a problem arrives in a DevOps shop (or at least one that really embraces the cultural ethos), the staff looks at the real resolution of that defect as a matter of personal pride. This pride is not a statement of “pride in self” however, but a statement in “pride in craft”. To hammer this contrast home, does ITIL by itself even have a deep cultural bias at all, much less an ethos?

This perspective is what allows DevOps shops to act in a manner that transcends the checkbox approaches that pervade formalist driven shops that believe in “adherence to process” above all rather than the “results the process is intended to achieve”. If you think I’m wrong, go ask your business clients how they feel about the DMV-like experiences they get when trying to get a problem solved.

This perspective is also why DevOps forms strategic pillars in automation and performance. Performance and automation efforts are not only about the manufacturing of time. Automation isn’t only a method of driving towards error free processes, or even a liberator of individuals from mundane tedium. Driving highly performant code and infrastructure isn’t only about conservation of compute resources.

Well implemented automation routines and highly performant applications are also a deeply creative expression of both development and operations practitioners. The same sense of artistry and elegance that was once reserved for just the software developer has been opened up to all IT practitioners. With the arrival of DevOps, being a great IT professional is not out of reach of any IT practitioner no matter their discipline.

We just can’t afford to mess this one up. In order not to, here is some practical advice:

As you go forth in creating or playing your part in your organization’s DevOps journey, look out for these pitfalls and pivot back to the center.
1. Reductionism – Don’t let numbers be the mission. This doesn’t mean don’t have numbers in your goals. It only means to make sure that your number oriented goals have a basis in making things better for people. When you hear someone say “We need to get this deployment routine under 4 hours!”, make sure that someone reminds everyone that the reason to get it under 4 hours is to allow employees to have a reasonably well balanced life outside of work. Off hours deployments that last four hours, when things go right, should not be a target that anyone is proud of. Getting people home to their families and allowing people to be able to keep a healthy sleep schedule is goal worth shooting for.
2. Formalism – Worshipping the shovel rather than the hole it digs is an easy trap to fall into for people whose jobs revolve around buiding and using tools. Remember that processes and tools are mechanisms intended to guide novices and intermediates through decisions. When you see someone turning a process or a tool into a sword and shield for working with others, make sure you remind them that results are better than processes.
People Over Process

Neither processes nor tools solve problems; People do. Processes and tools don’t create business opportunities by themselves; People do. Isn’t that why we chose this profession in the first place? To be useful in solving problems and to create opportunities to make things better while allowing people (ourselves included) to have a good standard of living.

Now go DevOps practitioners! Go kick some ass and make the internet awesome!
April 18, 2014
How We Can Help DevOps Cross The Chasm

The problem that DevOps solves is universal, urgent and important. Every IT organization is responsible for two things: deliver fast flow of projects and features, while preserving reliable, stable and secure services. Until very recently these were thought to be mutually exclusive, we thought it was only possible to achieve one of those objectives, at the expense of the other. This caused years (or even decades) of inter-tribal conflict between Development and IT Operations; led to chronic underperformance, with features taking slower than ever to reach market, deployments taking longer, an ever-increasing number of Sev 1 outages, and IT Operations becoming buried in technical debt and unplanned work; and a variety of other symptoms.

We now know that there is a better way. The proof is that high-performing organizations such as Amazon, Google, Twitter, Etsy and Netflix, are adopting a set of techniques that we now call DevOps, and are routinely deploying hundreds or even thousands of code deployments per day, while preserving world-class reliability, stability and security.

During the 2012 Puppet Labs DevOps Survey of Practice ), we benchmarked over 4000 organizations. We found that the high performers were massively outperforming the low performers. Those employing DevOps practices were doing 30 times more frequent code deploys, and had deployment lead times measured in minutes or hours, versus weeks, months or quarters.

DevOps organizations also had far better deployment outcomes: Their changes and deployments had twice the change success rates, and when the changes failed, they could restore service 12 times faster.

This is what every IT organization needs to be able to replicate.

DevOps Isn’t Just For The Unicorns

“If there’s anything that all horses [enterprise IT organizations] hate, it’s hearing stories about unicorns [DevOps shops]. Which is strange, because horses and unicorns are probably the same species. Unicorns are just horses with horns.” — Christopher Little

We know that without something like DevOps, IT organizations will fall into the downward spiral. However, most enterprise IT organizations will come up with countless reasons why they cannot adopt DevOps, or why it is not relevant for them.

One of the primary objections from horses is that all the unicorns (e.g., Google, Amazon, Twitter, Etsy, etc.) were born that way. In other words, from their beginning all unicorns used DevOps style work patterns.

In reality, every unicorn at some point in their history had the same problems as horses: an inability to grow capacity fast enough to keep up with market demand, dealing with years of accumulated technical debt, lack of automated testing to sustain high rates of change, monolithic and overly tight architectures that prevented developers from being productive, and so forth.

The unicorns have made the journey that every horse will need to make.

The DevOps Survey of Practice showed that DevOps isn’t just for unicorns. Not all of the 4000 or so respondents were startups, and they weren’t all web operations companies either. Instead, 42% of the organizations that were implementing DevOps practices had over 500 employees. The link to those slides is here.

DevOps is equally relevant for startups as it is for legacy enterprises. The need to decrease time to market and shrink lead times is universal. In fact, it may even be more urgent for the multi-billion-dollar enterprises – creative disruption is making the tenure of Fortune 500 companies ever more tenuous. Richard Foster, a professor at Yale University, has said that the average life of a Fortune 500 company has decreased from around 75 years to 15 years in the last half century. Of the Fortune 500 companies in 1955, 87 percent are no longer on the list.

Increasingly, IT is the value creation and customer acquisition engine for organizations. As my friend Chris Little says, “Every company is an IT company regardless of what business they think they’re in.”

DevOps Isn’t A Fad

I compared the plight of DevOps to apartheid. With ridiculously worse conditions and a much more self evident problem, Nelson Mandela sat in jail for 27 years to show a minority of people in only one country that racism is wrong. Industry level change doesn’t take seven years. It takes upwards of 27. — Stephen Fishman

Some may worry that as more people write about or seek knowledge about DevOps, that DevOps will become a dismissible fad.

I’d encourage anyone who wants DevOps to become a pervasive reality to look at it this way: there is a spectrum of awareness, spanning from “no one has heard of it, and dismiss it as a movement that is irrelevant to them” on one side, to “everyone has heard of it, and rolls their eyes that it’s the latest fad.”

If I had to choose one of those scenarios, I’d choose “fad” every time. Why? Because there is nothing worse than trying to solve a problem that no one else thinks is important. The fact that so many communities are working to solve the DevOps problems is amazing. Development, testers, agile coaches, IT operations: We all believe it’s important.

As DevOps advocates, we must always be ready to respond when someone asks us, “I’ve heard about this DevOps thing. Is it something we should be doing?” The response should always be, “Yes, of course, we should be doing it! Here’s what I can do to help!”

As the DevOps community grows past what Geoffrey Moore called the “innovators” and “early adopters” (i.e., the unicorns), we must continue to tell the story so that the “early majority” and “late majority” will realize that DevOps is for them, too. Only by doing this can we have mainstream adoption, a phenomenon that Moore called “crossing the chasm.”

To create DevOps outcomes, we need a coalition that spans the entire value stream, from Marketing, to Product Owners and Product Managers, to Development and Testers, IT Operations and Infosec. Only then can we help our organizations go from low-performing IT to high-performing, which we’ve shown we can do through DevOps practices.

Conclusion

When IT does poorly, the business will do poorly. And when IT helps the organization win, those organizations will outperform their competitors in the marketplace. DevOps allows IT to win.

March 11, 2014