Recently, some folks have been suggesting that we need a new form of DevOps, something called “Enterprise DevOps” or “Big DevOps.”
DevOps is great, the argument goes, but large enterprises don’t operate the same way as the agile web startups where DevOps was birthed, and thus they can’t implement the same DevOps methodologies as those startups. Large enterprises have bigger teams, more inherent operational complexity, and greater governance controls (both self-imposed and externally-imposed); therefore, they need a different type of DevOps that caters to their sensibilities and not those of agile web startups.
While it’s true that large enterprises do operate in a different environment than agile web startups, none of that makes a difference from the point of view of adopting DevOps.
In short, if you’re advocating for Enterprise DevOps as something different than Plain Ol’ DevOps, then you’re missing the fundamental essence of DevOps.
Here’s why.
Early Thinking: DevOps = Tools + Culture
To best understand DevOps, we need to understand where it came from.
DevOps started as a bottom-up collection of best practices with few formal underpinnings. System administrators and developers were looking at the successes of agile development practices and trying to apply some of the same ideas to the whole development-to-operations process. Practitioners in the trenches stepped forward and said, “This is what we’re doing and it seems to work for us.”
One of the earliest and most visible examples of that was John Allspaw’s and Paul Hammond’s presentation at Velocity in 2009, titled “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr.”
[youtube http://www.youtube.com/watch?v=LdOe18KhtT4]The video is well worth your time, as is the presentation on Slideshare:
[slideshare id=1628368&doc=allspawhammondvelocity2009-090623161942-phpapp01]
Note that this presentation predates the actual creation of the name “DevOps.” Allspaw and Hammond talk about how Dev and Ops teams can work productively together, but they never say “DevOps.” Patrick DeBois would coin the term later that year when he created the first DevOpsDays.
In the presentation, Allspaw and Hammond discussed the success they were able to achieve at Flickr and provided a set of recommendations that they think are important, separated into two buckets:
- Tools
- Culture
In the Tools category, they provided six recommendations:
- Automated infrastructure
- Shared version control
- One step build and deploy
- Feature flags
- Shared metrics
- IRC and IM robots
Along the way, they mentioned that Flickr used:
- Chef, Puppet, CFEngine, and other automation tools
- SVN for version control
- Hudson for builds
- Ganglia for monitoring
Finally, in the Culture category, they listed four additional recommendations:
- Respect
- Trust
- Healthy attitude about failure
- Avoiding blame
Unfortunately, most people missed Hammond’s remark ahead of the Tools section of the presentation (timecode 7:40):
These are some of the tools that work for us. They are not necessarily going to work for everyone, and throughout we’re going to give examples of the specific tools that we use, but the important thing that we’re trying to get across is the concepts that are important.
In the years following, many self-proclaimed “DevOps engineers” (a bit of an nonsensical title if you understand that DevOps is about bringing Dev and Ops engineers together, not creating new types of engineers focused on nothing but DevOps) insisted that if you were automating your releases using Jenkins and Chef and treating the other members of your team with respect, you were “doing DevOps.”
While Allspaw’s and Hammond’s presentation was excellent (seriously, spend the time watching it) and the advice was practical and valuable, many people walked away with only a vague notion of what would later be called DevOps.
For instance, in the “One step build and deploy” recommendation, Allspaw and Hammond describe their automated system that moves code into production. Developers can click a single button to make a build live. The button is jokingly labeled “I’m feeling lucky” (in homage to Google, I’m sure).
If you’re an agile web startup, that’s par for the course.
But if you’re a large financial services company, you’re likely to think, “No! Our governance folks will never let developers push code straight to production with a single click, never mind the ‘I’m feeling lucky’ labeling. No way. Not gonna happen.”
When people suggest that we need “Enterprise DevOps,” they’re typically reacting negatively to the choices that others have made. They recognize that those solutions won’t work for them for any number of reasons, and they say, “I can’t use that. I need something different.”
But here’s the thing — DevOps isn’t about the specific implementation choices. Those are all optional. You can make a completely different set of choices than Allspaw and Hammond and still be “doing DevOps.”
Unfortunately, subsequent conferences were filled with DevOps presentations that just reiterated the “DevOps = Tools + Culture” mantra and provided similar sets of recommendations that were often drawn from agile startups rather than from large enterprises.
So, does that mean Tools and Culture aren’t important for DevOps?
No, not at all. Both Tools and Culture are critical for DevOps in the long run.
The problem that DevOps seeks to solve, however, is not a Tools problem or a Culture problem but rather something more fundamental.
In order to understand DevOps, you need to focus on the ultimate goal.
The Firm Foundation: The Toyota Production System and Lean Manufacturing
In 2013, Gene Kim, Kevin Behr, and George Spafford wrote The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.
The trio’s contribution to the DevOps movement was in recognizing that enterprises had confronted a similar set of issues previously, but in a different domain — manufacturing — and that similar solutions could be applied to software engineering and application operations. (The group took another cue from the manufacturing world, copying the novelization technique from Goldratt’s The Goal to create a fictional case study where the characters were allowed to wrestle with the fundamental DevOps truths being discussed.)
The key takeaway from The Phoenix Project is that DevOps seeks to create a smooth and increasingly speedy flow of change from development into production, with as little waste along the way as possible.
The models here were the Toyota Production System (TPS), Lean Manufacturing, and Goldratt’s Theory of Constraints.
In the world of manufacturing, we have a physical product that moves through a factory, either on a formal assembly line, or from processing step to processing step, as part of a value stream. Each step along the path is supposed to “add value” to the product until it completes its journey and exits the factory to be shipped to customers.
If you’re a manufacturing engineer, you need to do everything possible to create a smooth flow through the value stream by identifying constraints in the system and eliminating waste of time or resources.
In the software world, application artifacts and changes move through the software development and operations value stream, with each step designed to “add value” and get the changes into production.
If you’re a development or ops engineer, you need to do everything possible to smooth the flow through to production. The techniques you use are similar to a manufacturing engineer — you identify constraints in the system and go about eliminating waste of time and resources.
The Kata is the Key
Following Toyota’s success, the TPS and Lean Manufacturing were all the rage in the manufacturing world.
Consequently, many western manufacturing engineers studied Toyota’s systems in great detail. They noticed a variety of tools and solutions that Toyota used to optimize manufacturing flow, and they recommended their use to their employers and clients. Toyota’s Kanban, for instance, was heavily promoted in the manufacturing world, and later made the jump to software engineering.
Interestingly, the manufacturing world has suffered much of the same confusion that’s now happening in the DevOps world. When they tried to employ the tools and techniques they had learned from studying Toyota, things didn’t work as expected.
Mike Rother, author of Toyota Kata: Managing People for Improvement, Adaptiveness, and Superior Results, described the situation in the manufacturing world this way:
Looking back, we naturally put Toyota’s visible tools in focus first. That is where we started— the “door” through which we entered the Toyota topic. It was a step in the learning process (which will also, of course, continue after this book). Since then I went back to the research lab–several factories–to experiment further, and present what I learned in this book. The visible elements, tools, techniques, and even the principles of Toyota’s production system have been benchmarked and described many times in great detail. But just copying these visible elements does not seem to work. Why? What is missing?
Later, Rother continues:
Toyota opens its factory doors to us again and again, but I imagine Toyota’s leaders may also be shaking their heads and thinking, “Sure, come have a look. But why are you so interested in the solutions we develop for our specific problems? Why do you never study how we go about developing those solutions?” Since the future lies beyond what we can see, the solutions we employ today may not continue to be effective. The competitive advantage of an organization lies not so much in the solutions themselves—whether lean techniques, today’s profitable product, or any other— but in the ability of the organization to understand conditions and create fitting, smart solutions.
Focusing on solutions does not make an organization adaptive. For example, several years ago a friend of mine visited a Toyota factory in Japan and observed that parts were presented to production-line operators in “flow racks.” Wherever possible the different part configurations for different vehicle types were all in the flow racks. This way an operator could simply pick the appropriate part to fit the particular vehicle passing down the assembly line in front of him or her, which allows mixed-model assembly without the necessity of changing parts in the racks. Many of us have been copying this idea for several years now.
When my friend recently returned to the same factory, he found that many of the flow racks along that Toyota assembly line were gone and had been replaced with a different approach. Many of the parts for a vehicle are now put into a “kit” that travels along with the vehicle as it moves down the assembly line. When the vehicle is in an operator’s workstation, the operator only sees those parts , and she always reaches to the same position to get the part.
My friend was a little upset and asked his Toyota hosts, “So tell me, what is the right approach? Which is better , flow racks or kitting?” The Toyota hosts did not understand his question, and their response was, “When you were in our factory a few years ago we produced four different models on this assembly line. Today we produce eight different models on the same line, and keeping all those different part variations in the flow racks was no longer workable. Besides, we try to keep moving closer to a one-by-one flow. Whenever you visit us, you are simply looking at a solution we developed for a particular situation at a particular point in time.”
As we conducted benchmarking studies in the 1980’s and 90’s and tried to explain the reasons for the manufacturing performance gap between Toyota and other automobile companies, we saw at Toyota the now familiar “lean” techniques such as kanban, cellular manufacturing, short changeovers, andon lights, and so on. Many concluded— and I initially did too— that these new production techniques and the fact that Western industry was still relying on old techniques were the primary reasons for Toyota’s superior performance.
Rother concludes that the secret to the Toyota Production System and Toyota’s success was not any one particular tool or technique, but rather the Toyota Improvement Kata, which focused on continuous improvement of everything at Toyota, seeking to establish flow and eliminate waste.
Toyota Kata is a great book and if you want to understand DevOps, I encourage you to get a copy and read it for yourself. Kevin Behr recommended it to me and I was pleased to see Jez Humble reference Rother’s story, above, in his recent QCon talk on DevOps.
What is DevOps?
So, after all that, what are we left with?
What is DevOps?
Cue the trumpet fanfare…
Fundamentally, DevOps is the activity of optimizing the development-to-operations value stream by creating an increasingly smooth, fast flow of application changes from development into operations, with little waste. Optimization of the value stream takes place continuously using various continuous improvement techniques like the Toyota Kata.
“Is that it?!” I hear you cry.
Yes, that’s it.
“No way!”
Way.
“But what about Tools and Culture?! What about continuous delivery and Jenkins?! What about automation? What about mutual respect?!”
Yes, those are all in there, too. But those are the equivalents of Rother’s “kanban, cellular manufacturing, short changeovers, andon lights, and so on.” They are solutions to particular problems that you might be faced with.
If, during the course of trying to optimize your value stream, you’re faced with the problems that those tools and techniques solve, then you might try them out as solutions. Or you might not, if your situation dictates differently.
The key is to understand that simply deploying a tool or even a broad solution without thinking through your unique situation and the current state of your value stream and its current constraints is foolish. After all, the specific solutions that are applicable to an agile web startup have a high likelihood of being inapplicable to a large enterprise. And vice-versa.
Why “Enterprise DevOps” Doesn’t Make Sense
So, that brings us right back to where we started.
Enterprise DevOps doesn’t make sense because it confuses the forest for the trees. Those advocating for Enterprise DevOps make the mistake of focusing on specific tools and solutions (Jenkins, Kanban, etc.), then find them wanting for various (extremely valid) reasons in an enterprise context, and then reject DevOps in favor of Enterprise DevOps, which they suggest is somehow different.
But DevOps is about flow and continuous improvement, not about specific solutions. Flow and continuous improvement are equally applicable to a large enterprise as they are to an agile web startup. And if you miss that, you’re lost, regardless of the tools and solutions you choose.
To implement DevOps, you need to learn how to map your value stream, identify the constraints in it, and then use a continuous improvement kata to develop specific solutions for your particular problems at this point in time in your particular organization.
If you’re a large enterprise, the problems and constraints that you face will undoubtedly be different than those of an agile web startup. So, you’ll necessarily need to select different solutions.
That doesn’t mean you “aren’t doing DevOps!”
Rather, it means that you’re doing DevOps correctly.
And remember that whatever solutions you come up with to your particular problems and bottlenecks, they are likely to be temporary. As Toyota said to Rother’s colleague, solutions are only applicable to a particular problem at a particular point in time.
Epilogue: A Bitter Irony
Some of the earliest work on what would become DevOps grew out of some ideas that Andrew Shafer was kicking around in late 2008.
At the Agile 2008 Conference in Toronto, Shafer proposed a birds-of-a-feather session about “Agile Infrastructure.” Only a single person showed up — Patrick DeBois. He found an empty room. It turned out that Shafer skipped his own session because he received such poor feedback about it and thought nobody was interested. Shafer and DeBois later discussed Agile Infrastructure and many ideas that would later become DevOps in the hallways at the Agile 2008 Conference.
Shafer also presented on Agile Infrastructure at Velocity 2009:
[slideshare id=1637386&doc=velo09-agile-inf-090625020850-phpapp01]
Note slides 92 through 97. The key points from this sequence are:
- Dogma Sucks
- “Do not seek to follow in the footsteps of the wise. Seek what they sought.” This is a quote from Japanese poet, Matsuo Bashō.
- The Tao that can be spoken is not the eternal Tao.
- You are a special snowflake, just like everybody else.
Even from the start, Shafer was on to the point that a binding set of rules is self-defeating. DevOps cannot be codified in a set of rules. Because it’s bigger than that.
Note the Matsuo Bashō quote:
Do not seek to follow in the footsteps of the wise. Seek what they sought.
The wise DevOps practitioners do not seek to implement any given tool or technique out of adherence to DevOps dogma. Rather, they seek to create a smooth, quick flow of changes through the value stream. Do not try to follow in their footsteps using the exact same set of tools or techniques, but rather seek the same thing that they seek and find your own way to the same destination.
So, the bitter irony here is that we’ve been warned all along, by Shafer, Allspaw, Hammond, and others, to focus on the foundational problems, not the specific solutions they have employed. Yes, we can learn something from their experiences, but every situation is unique.
(Unfortunately, I haven’t been able to find video of Shafer’s Velocity 2009 presentation. If you find it, let me know. But you can see a later version of Shafer’s ideas, mixed with those of Paul Nasrat, in a presentation from Agile 2009, made available on InfoQ.)
Other Resources
I’ve hopefully salted this post with enough links to key presentations and resources, but there’s one more that should be included. If you’re at all interested in the history of DevOps, and you should be if you really want to understand it better, then you need to view Damon Edwards’ “The (Short) History of DevOps”on YouTube:
[youtube http://www.youtube.com/watch?v=o7-IuYS0iSE]
Connect With Me
Think I’ve left something out or put too much in, got it totally wrong or completely nailed it? Let me know! Continue the conversation by commenting on this post and following me on Twitter.
About the Author
Dave Roberts brings 25 years of technology experience to his role of Executive Advisor at BMC Software. A thought leader in agile IT and digital transformation, Dave routinely speaks at conferences around the world. Dave is a member of the Twitter #clouderati and blogs at BMC.com and Leverhawk.com. He holds nine patents and has written three books on computer programming and data communications technology.
