devopsy.kinsta.cloud

Author: Stephen Fishman

Seeking the Balance Between Federalism and States’ Rights
When you overhear someone talking about federalism versus states’ rights, you are bound to think they’re talking about national politics. In the meeting rooms of corporate America, however, it’s often a very different discussion going on.

For product engineering teams in a medium to large corporate context, many topics and discussions boil down to federalism versus states’ rights wherein federalism relates to the interests of the corporate parent and the states in question are the business units or individual product teams. The representatives from each of these bodies often have distinct agendas and priorities which sometimes compete with each other and all too often leave employees and executives alike flummoxed and resigned to pain.

The Federalist Perspective

The people who are charged with driving unification and standardization may indeed use their process and policies as sword and shield, but this doesn’t mean they don’t have a well-reasoned position that drives them. Almost every employee is familiar with the drive to create revenue. Not nearly as many are familiar with the charge of “driving the spread.”

A business could be viewed as healthy when revenue grows at a rate that exceeds the cost of capital (i.e., “investment in the business beats investment in the market”). Better yet, a business can be viewed as sustainably healthy when revenue grows at a rate that exceeds the cost of capital AND operational expense grows at a smaller rate. This is what is known as increasing “the spread.”

When the spread grows year over year, it means the business is succeeding at increasing both effectiveness and efficiency at the same time (i.e., “doing the right things right”). In most cases that don’t have explosive market growth, driving the separation between revenue growth trend lines and operational expense growth trend lines is one of, if not the best, metrics for the sustainable health of an enterprise, because the increasing spread acts as a lever to fend off disruption and competition by raising barriers to entry or funding acquisition of potential disruptors.

If your job within the company is to increase the spread for the entire enterprise, the best tools at your disposal are to drive standardization and consolidation in all things by default. In this context, “all things” really does mean “all things.” It includes—but is not limited to—tech stacks, support levels and processes, operating procedures, job descriptions, programming languages and frameworks, employee policies, employee tools, business process systems and core capabilities.

This drive for rationalization often will come into conflict at the business unit because individuals at all levels within the business unit will many times be asked to drive change that could sacrifice the performance metrics and commitments that are measured at the business unit level.

In the eyes of the federalist, sacrificing pennies for the sake of dollars is almost—if not—always the right thing to do for the enterprise.

The States’ Rights Perspective

The people who are charged with driving business unit performance may indeed voice their displeasure at corporate direction with an attitude that displays a lack of professionalism and respect, but this doesn’t mean they aren’t fighting for a just cause. Supporting the whole at the expense of piece is something that most people can understand. Not everybody, however, can see the cost that comes when unification erodes peoples’ sense of self-determination.

Individuals and individual business units alike are extrinsically driven and rewarded by achievement of individualistic goals and commitments. Many of these same individuals are intrinsically driven by a sense of autonomy, mastery and purpose. Autonomy at its basic level is well-understood.

When new directives for unification or standardization are rolled out, this introduces both direct and indirect cost at the business unit level. These costs aren’t always funded to the extent necessary to drive the required rationalization. Even in the case of reasonable funding, relief from existing commitments isn’t usually factored into corporate directives.

Funding and wholistic concerns aside, what can be the hardest dynamic to work through is directed idleness while fully aligned decisions are being generated. From the practitioner perspective, this dynamic can be perceived as contemptuous because it simultaneously says, “Your concerns are not sufficient to warrant our attention,” and “Don’t take any action to address your concerns because that will warrant our attention.” In the most extreme cases, this is coupled with an unyielding expectation to meet or exceed performance goals.

This approach breeds a combination of labor attrition, forced helplessness and a jaded attitude within teams that can take years to unwind and change.

The Balanced Perspective

Splitting the middle can be difficult in emotionally laden high-stakes areas. Add distrust and a high degree of power separation, and the degree of difficulty gets intense for this effort. Let’s start from the top because the onus is on leaders to go first when an impasse arises:

Where federalists can give in efforts to support self determination:
- Even the most ardent federalists agree that autonomy is the only path to achieve scale in a modern corporation. Central bodies of authority can go out of their way to demonstrate their commitment to this by endorsing explicit role-specific areas where experimentation and innovation is needed (i.e., a message of, “Don’t advance here just yet,” goes down easier if it’s coupled with a list of topic areas that have few or no guard rails).
- Assuming current commitments are not in jeopardy, acknowledge that work done in excess of current commitments is a contribution and not a distraction, even if it has the localized possibility of deviating from to-be-set standards (i.e., a message of, “Don’t spend time on improving area x,” goes down a lot easier if it’s coupled with an acknowledgment that, “If a person is willing to dedicate their own personal time, that does not come at the expense of the company. We value that level of commitment”).
- Find ways to recognize and reward individuals and teams for avoiding local optimization at the expense of the whole to compensate for locally defined goals and compensation systems.
Where states’ rights activists can give in an effort to support a broader set of goals:
- Even the most ardent localists acknowledge the pricicipal within a hierarchy of goals (enterprise goals > unit goals > team goals > personal goals). Local leaders and practitioners can go out of their way to demonstrate their faith in leadership by going the extra mile to find and generate goal alignment that drives benefits to more than one organizational unit (i.e., a stance of, “We’ve prioritized local remediation in this one area because it helps our business unit meet some secondary objectives,” invites less opposition if it’s demonstrably linked to the broader set of enterprise goals.
- Assuming room for innovation and continuous improvement is met with tolerance, local leaders and practitioners can make themselves beyond reproach by ensuring goal priorities have been kept in mind and also focusing improvements that have limited to no impact outside of their local responsibility (i.e., a local development push to a microservice architecture can have deep impacts to a centralized operations unit without a diligent effort to manufacture a mature operational capability).
- Find ways to drive local improvement communications up to broader system owners in an effort to facilitate collaboration and longer-term planning (i.e., a message of, “Business unit x has developed a new solution for y,” raises a lot less concern when it’s paired with a statement of, “… and the teams acted responsibly by collaborating with the enterprise team responsible for keeping our long-term efforts here aligned with a consensus road map.”
When deference and forbearance meet, graceful and elegant solutions emerge and ultimately harmony is preserved. If only more people understood that the adversarial noise in between us is more harmful than either minor deviance, or momentary ambiguity.

— Stephen Fishman
May 3, 2017
DevOps Storytime: The Line, The Loop and The Cube

In a time not so long ago, but long enough to be before the mainstreaming of magical ideas such as “configuration as code” or “platform as a service (PaaS),” in a period where the enterprise IT shops were ruled by the one-two punch of Six Sigma and ITIL, there were three magical artifacts sought by the people who spent their working days lubricating the gears of the consulting, development and operations factories throughout the lands of corporatedom.

The three mystical objects consisted of The Line, The Loop and The Cube. Each were first envisioned as possibilities by “automation zealots” as they roamed the corporate cubicle farms in vain attempts to provide enlightenment to their enterprise brethren. As the quest to drive evermore efficacy out of technology shops of all shapes and sizes has progressed, strategists, designers, architects and developers have been slowly creating these possibilities into being. Much like the legend of the Deathly Hallows, if these once-mythical objects come fully into being, the enterprise that can behold all three of the artifacts can create the nigh invincible product/technology factory as it becomes the master of time.

Continuous Integration – The Line

In conceptual land, The Line was just what it sounds like: a singular line, never bending and never deviating from its course. It connected the space from development to production. The idea of The Line has been around a long time—long enough for many shops to now have working implementations running as standard operating procedure. But it wasn’t always this way.

In reality land, The Line was a patchwork of open-source frameworks, glue code, xml files and a few engineers unlucky enough to be tagged with role of “Build Master.” Woven together by the Build Master, these transformative robots and would take coders’ output, compile it into a workable build that merges all of the developed functionality, run and validate test harnesses against it. Once validated, the newly minted build then would be promoted to the higher-level environment, ultimately moving from dev to test, test to stage and, lastly, from stage to prod.

The tools to build and manage The Line have been through many iterations since it first came into use and many shops have progressed far enough into the advanced territories of The Line to call themselves masters of the once-mythical Line. The shops exploiting The Line are now so prolific and the tools have advanced enough that it’s no longer an onerous task to get change-resistant enterprises to try it out.

The Line has transformed from mythical artifact to concrete reality so much so that the believers now outnumber the holdouts. The end result is that shops not embracing the basics of continuous integration risk losing credibility in the eyes of their staff and any potential recruits.

OODA is a Way of Being – The Loop

When someone starts talking about “getting the right information to the right person at the right time,” that person is talking in fairly vague terms about the the long sought-after grail of The Loop. The Line is a good start, but the robot watching the line is only as smart as the humans who created and configured it. This is one of the many reasons to chase The Loop.

What is the use in “shifting left” if you can’t get the alerts and metrics to the people who care? How can the metrics from “shifting left” be acted upon if they can’t be pushed into the view of those who can act upon them? The Loop is the answer, because it is the mechanism to determine many things beyond “the right person.” The Loop knows how to get messages to all interested and responsible parties. The Loop has mechanisms to account for an ever-fluid organizational structure and backup delivery methods for backup delivery methods.

All enterprises have feedback loops. The key differentiator for moving toward mastery of The Loop is the degree to which you have automated the and abstracted the mechanisms inside the feedback loops.

Many people and enterprises are resigned to life without The Loop. The signs are easy to see if you know where and how to look. Are mail distribution lists and org charts always a pain in your organization? Are executive-led fire drills the norm when systems are down? Are system alerts dropping on the floor rather than getting to the first responders and application owners?

If you answered yes to any of the above, your enterprise has not committed to developing and leveraging The Loop. Basic “loop-like” alerting functionality is commonly available from alerting vendors, but true mastery of The Loop cannot be achieved without robust identity management and synchronization technology. Without this critical integration point, alerts and information flow can only be as effective as your manual processes that update the myriad data islands in each and every alerting system within your enterprise.

When enterprises grow, being able to have the enterprise robot army seamlessly drive information flow without requiring sieve-like manual processes is also the key to tightening the enterprise OODA (observe, orient, decide and act) loop. Done with full mastery, one could imagine the possibility of acquiring the last of the three magical grails and maybe the most sought after: The Cube.

Perceiving the Matrix – The Cube

The story of The Cube begins with a question: “What defines the so-called ‘right’ information?” When enterprise technology and design evangelists talk of The Cube they often wax poetic and describe it as all things to all enterprise people at every level within the organization. It provides not only the integrated matrix of the fully faceted enterprise from an organization perspective, but also extends beyond operational information rights into decision rights, affinity groupings and cross-functional teams.

We have begun to see promising beginnings of The Cube pop up within medium and large enterprises through popularizing algorithms and outfitting teams with ubiquitous mission control-style dashboards and screens in team areas. But it is only when these dashboards can integrate and harmonize wide varieties of development, operational and business KPI data that we will be able to claim that we have delivered full and timely context (i.e., “the Right Information”) to the staff sailing the enterprise ship.

The hard part of creating The Cube is not so much the technology required to achieve its development and integration. The really difficult part is coaxing the organizational will to define and maintain the KPI taxonomies for each organizational unit and the integration points between them. Given that a fully mastered Loop doesn’t necessarily understand how KPIs, infrastructure and applications interrelate and affect each other, the Loop will forever be but a required stepping stone to The Cube.

Will a Unicorn be Required for Unification?

The current landscape of open-source and vendor tools to wield and tie together these ungainly enterprise models may not, as of yet, be sophisticated enough to both integrate and simplify the provisioning and cultivation of The Lines, The Loops and The Cubes. The sultry, siren-like call of the Unicorns, however, will likely beckon the cloud builders of today and tomorrow to create not just development to deployment pipelines, but also fully integrated feedback loops with intelligent understanding for how the data connects to form perspective.

It is odd to think about, because with the idealized future state so close to actual possibility, it begs a new set of questions. Hordes of IT professionals around the world have been working to find faster and more fool-proof ways to deliver meaningful perspective into the hands of decision makers and first responders. We are so close now to that mission being accomplished that there are a few instances of the perspective itself not only being generated by “the robot” but also being handed to “the robot” to take the necessary remediation steps.

The ending we are approaching, much like the beginning, circles around a question: “If we truly automate all the things, will there be any more work to do?”

September 1, 2016
An open letter to Jeff Knupp
Dear Jeff,

First off, nice recovery. I commend your gesture to return to positivity and make what to my eyes, looks like a kind and altruistic offer to an underserved class of engineers. I personally sent your offer to all the female engineers on my team and got very good feedback from several of them.

Some people will try to tear you down and criticize your offer to make a difference in the lives of female engineers as patronizing or sly self promotion, but I take you at your word and applaud your offer as both poised and graceful.

Given your offer to one underserved population in IT, I thought you would be OK with some constructive feedback from one dev guy concerned with the welfare of IT workers to another and hope you take this feedback as it is intended, with the same eye towards a desire to be a force for positive change to the plight of all IT workers, everywhere. Please remember that this is just my perspective. I could be wrong, and often am. So, here goes:

DevOps does not mean everyone has to do everything

DevOps is not just for startups.

I work in a big shop and we have all the sorts of disciplines you call out in your posting on how DevOps is killing the developer. We are also embracing the DevOps movement not to force anyone to take on roles they aren’t optimized for, but rather to inspire them to expand the boundaries they have set for themselves and others in an effort to make things better for everyone.

We want QA professionals to know how to script and automate tests. This is good.

We want operations staff to know how to script and automate changes. This is good.

We want release managers who know how to triage problems, automate configuration management and instrument the throughput of the IT machine. This is good.

We want developers and architects who know how to go deep when required to help solve performance problems. This is good.

You may disagree and I respect your right to do so. I just am not that interested in hiring an IT professional of any type who says “That ain’t my job”. That’s not a guy or gal who gets things done or makes things better.

I’m not saying that everyone has to be able to do every role or even that developers must branch out. I am saying that making others wrong for increasing their value prop is not likely to do anyone any good.

Injustice anywhere is a threat to justice everywhere

Using narratives of which type of IT worker is more important than another type of IT worker does us all harm. All of us. All men. All women. All IT workers. All Developers. All operations staff. All employees from any department of any company no matter the size.

Now, I realize that this is a bold statement (even from the guy who wrote “Sharepoint is Crack and Microsoft is the Pusher”), and I have no problem admitting that you probably have no desire to harm any employee anywhere (Except for the guy or girl who invented IVRs. I can understand wanting to do bodily harm to whoever did that). I would ask that you consider a few ideas before writing me off as a troll.

1) The schism upstream is the enemy to be fought, not the people downstream from you

You may not be aware of the schism to which I refer, but I would argue that you are probably very familiar with it’s effects. The deep and distrusting divide that exists between enterprise IT and the customers of enterprise IT across the industry is the single most destructive and demoralizing artifact in the lives of IT employees everywhere.

Have you ever worked in a shop where IT employees are casually referred to as “technical” while marketing, product and design employees are referred to as “creative”. Have you ever been exposed to a shop where the developers were treated and talked about like they were worker insects who’s job it was to keep their mouths shut and code what they’ve been told to?

If you have somehow managed to have an IT career and avoided exposure to this phenomena that seems to pervade so many (but not all) IT shops around the world, I would ask you to pull back and reflect for minute. Spend some time and have some gratitude for the fact that you have not been either witness to, or been the subject of the message: “we are the so and so team and you are not on our team so you have no say IT boy, go back to your nerd closet and build what I tell you to build”.

If you are familiar with this idea that robs developers everywhere of the right to claim their creativity and basic need for self realization through recognized craft, then I ask you this: Aren’t you all sick and tired of the completely loathsome ideas that we are not “creative” because we are IT? Doesn’t that make you want to throw up? Don’t you want to go postal when you hear people push developers down the collectively agreed to illusion, referred to by you as the “totem pole”, that creates a veritable corporate caste system?

I would run out of my word limit if I were to try and cite the publications that point out that silos are the prime roadblock to unlocking business potential, but I will point you to a few modern classics which make the case much better than I ever could:
- The Advantage: Why Organizational Health Trumps Everything Else In Business by Patrick Lencioni, best known for The Five Dysfunctions of a Team: A Leadership Fable
- Silos, Politics and Turf Wars: A Leadership Fable About Destroying the Barriers That Turn Colleagues Into Competitors also by Lencioni
- Orbiting the Giant Hairball: A Corporate Fool’s Guide to Surviving with Grace by Gordon MacKenzie
This isn’t just about IT workers. This is about every employee everywhere who has their role demeaned and belittled by another. There are three basic roles inside of most companies in the products and services business. Sell the work, do the work, support the work. One thing in common to them all is that if any of the three roles don’t get done well, then everyone is at risk of not being paid.

We are all necessary and none of us is sufficient at any real corporate scale. The moment we allow division (i.e. the totem pole) between ourselves, we have ceded the ground that other people upstream from us or up the hierarchy from us have the right to demean whomever they want to.

You can’t possibly eliminate injustice via injustice

2) The schism will always exist if we reinforce it

Being of service to another is an honor and all who claim to want to improve the lives of workers anywhere, should hold themselves accountable to not belittle the contributions of others.

Every single employee has the potential to be innovative and creative in their work. We constantly create and reinforce the concept of commodity items and jobs and then are paradoxically shocked and surprised by:
- how the unmotivated and unengaged masses of people in jobs frustrate us with their lack of passion for doing a job well
- how new people constantly create a “new and improved” version of something perceived to be commoditized forever and then dominate the market because they believed it was possible to reinvent a new level of quality
Innovation is not a department. Creative is not a department. Aren’t you all sick and tired of narratives of ownership and control from other teams who claim to own these abstract qualities when they are delivered at you?

When we the development community adopt the same basic ideas, we allow it to continue.

If you want to world to be different, than you must have the courage to be willing to be one of the first ones who fully live in that new different world.

What world are you working towards?

3) How will you know your ideas and contributions are better if you set up barriers that make it harder for others to contribute?

I am privileged to lead one of the best dev teams in the world. I am privileged to work in a shop that is committed to breaking down the barriers to collaboration and cross-pollinating innovation. To keep us all all at our best, we have to be brave enough to say that our dev team and our creations can more than likely withstand any criticism or contribution from any team up or down stream and if they don’t because some other idea was better, so be it.

I won’t take the cheap and easy shot and ask “Are you afraid that the outside idea is better and that you will seem weak for having a lesser idea?”, because I doubt that this is true. I honestly think this kind of resistance has roots in the human condition itself. It probably has more to do with fear of the clock running out or the possibility of lesser ideas winning the day than of fear of being shown either as “weak” or “not as smart”. With these dreadful possibilities in mind, I would ask you consider this: The way to maximize the likelihood of these undesirable outcomes is by allowing others to say we are closed off to outside ideas. This is what leads to goat-rodeos, witch hunts and ultimately those with more authority than we have removing the little authority and control we actually have.

Let the best ideas win, not the ones made by those with the authority backed up by a title.

If you truly are looking for the best and most game changing ideas to help your enterprise, be aware that the richest source of these ideas will inevitably come from people who’s blinders have not been firmly affixed by the discipline they are embedded in. This is a widely accepted historical truth laid out by Thomas Kuhn in the timeless masterpiece The Structure of Scientific Revolutions and if you doubt me, remember that Albert Einstein was a patent clerk when he gave humanity the theory of relativity.

I’m not trying to make you out to be a bad guy or anything close to one. In fact I praise you for having the guts to articulate a fairly controversial point of view that you believe in.

When I write this letter, I’m not fighting for or against DevOps in particular (even though I am DevOps fan). I’m fighting for the end to the schism between Business and IT.

I’m fighting for the end of narratives of control and authority.

I’m fighting for the end of the un-empowering messages that authority gives to individuals every day.

If you want to know why enterprise IT employees everywhere are asked questions from their business partners that display lack of trust over and over and over; This is why…because the schism exists.

If you want to know why enterprise IT can’t complete things it starts; This is why…because the schism exists.

If you want a world where people trust IT employees expertise and allow IT shops to complete work without questions of minutia that spawn from lack of trust, we must not embody what we oppose – narratives of authority and ownership combined with an “us vs them” mentality.

I don’t say any of this to scold you Jeff. I say this as an attempt to call upon you to join us . Help us tear down the walls. We need more help. We need you.

Respectfully,

Stephen Fishman – Creative IT Professional
May 2, 2014
DevOps storytime: MTTR vs. Goodheart’s Law

I am not a fan of metrics. Wait. That’s not right. Let me try again. I am not a fan of singular metrics that stand by themselves. Nope. That’s still not right. I loathe singular metrics that stand by themselves as if they themselves are solutions to problems. Ok. That works. Let me explain.

Last week, I told a tale of a secret passageway into DevOps called Mean Time To Resolution (MTTR) and explained how the introduction of this one tiny metric can magically open the door to all things DevOps. Given that MTTR and cycle time are inextricably linked, any shop that is serious about MTTR will sooner or later find itself using DevOps practices.

Metrics + Crafty Humans = Doom

This is very different than two weeks ago when I told a different tale. A tale of how the very origins and roots of DevOps gives the DevOps movement an opportunity to do what ITIL could not; Change the relationship between business and IT. In the telling of this tale, rife with philosophical terms like reductionism and formalism, I quoted a brilliant English economist, Charles Goodheart, who noted that “when a measure becomes a target, it ceases to be a good measure”. Goodheart discovered that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” The layman’s explanation is that humans are crafty little buggers and will find ways to artificially hit any target because the quantitative systems can always be rigged.

So what is a DevOps voyager to do? Introduce the metric, get the change, and then proceed to watch devops die a slow death? No, my fellow DevOps devotees, that shall not be our fate. We can avoid this fate by knowing that halfway down the path of the secret passageway known as MTTR, there is a fork!

More Metrics to the Rescue!

Instead of introducing and presenting a flat and un-nuanced view of MTTR by itself, we can get a little crafty ourselves and find an appropriate hedge against Goodheart’s lay and instead present a weighted view of MTTR with a few other key metrics to allow for a more sophisticated view of the health of the IT factory floor.

Apply Weighted Averages – Weight the MTTR of defects and their associated resolutions by priority to intentionally skew the measurement towards the highest priority defects. Clinging to an absolute view of defect free software doesn’t work for a large code base with limited funding and time (aside from being scientifically impossible to achieve via godels proof)

Track Inbound Defect Counts – Track and present the inbound defect count, keeping the inbound defect count low is important, but pushing it to zero is more likely to be indicative of a shop that is more concerned with perfection than throughput

Track Outbound Defect Counts – Track and present the defect close count. keeping a negative inbound-outbound differential is a sign of a shop that is arcing in the right direction. Several weeks or months of positive differential is a warning sign that a quality mindset is slipping into dangerous territory.

Track Backlog Defect Counts – Track and present the count of defects in the backlog. A backlog of zero can be incredibly hard to maintain in a large enterprise with a large code base. In my experience 20 and 50 have been ominous numbers. Any more than 20 defects in an application and the backlog starts to become a fearsome beast that can only be tamed by stopping the factory. Any more than 50 and the incremental additions of any new defect seem trivial.

Track The Age Defects – Tracking the average age of defects in the backlog is critical for any presentation of MTTR. A spike in MTTR can actually be a good thing if it represents a wave of old defects being cleared from the back of the backlog.

Resist The Urge to Simplify

A five part metric begs the question of an balanced score card approach with a grading formula and despite my belief in the value of this tactic for availability and performance dashboards, I advise you to strongly resist the temptation to apply it here (otherwise, we are right back where we started with Goodheart’s Law. Damn you crafty humans!).

If you decide to bring the magical metric of MTTR into your shop, this type of reporting may be one area where starting small is quite possibly worse than doing nothing. Without the nuanced view, true visibility into the health of the shop isn’t possible. Worse than that, a flat view of just MTTR has the potential to mask real problems and raise false ones.

When you pull all of the metrics together in a single view, like in the above graphic, the real message becomes clear. The health of the factory floor isn’t tied up in a neat little bow by a single number. It isn’t even made perfectly clear by three, four or five numbers. The real health becomes clear in the trend. Is your shop doing better week over week? Month over month? Getting better every day is the name of the game, and MTTR (with a few other metrics) can be a lever that moves your IT world.

April 28, 2014
DevOps storytime: The tale of the secret passageway named MTTR

When DevOps practitioners and evangelists talk about adoption, they mostly fall into one of two camps. People who talk about driving devops adoption by helping people understand the core ethical reasons of improving the world for those who work in it (I’m usually in this camp with Gene Kim).

Others talk about driving DevOps adoption by helping people understand the monetarily valuable things that come out of DevOps like improved user engagement and conversion via better performance, more resilient operations via adaptive architectures, or an increase in IT throughput delivered by a use of automation technologies. There is, however, a third way to drive adoption; rather than introduce devops directly, introduce mean time to resolve (MTTR).

MTTR is an Industrial Age metric that measures the time from a problem arises to the time it solved. It just so happens that shops that make a nuanced version of MTTR a very visible metric and seek to drive it down, have no other choice but to adopt DevOps methods because DevOps methods and beliefs are more and more required to push MTTR closer and closer to zero.

A Story We Are All Too Familiar With

Let me tell you a story of Alice, Bill and Carl. Alice and Bill work for Carl inside an IT shop where nothing of DevOps is known or practiced. All is manual in the land of Alice and Bill and the tyranny of work in progress keeps productivity and throughput capped.

One day, when Carl asks Alice and Bill to get the cost of system defects and outages under control. Alice and Bill both start to research the methods for managing defects. Bill, the more linearly aligned of the two people chooses to focus on prevention of defects and starts to introduce checkpoints and controls for how changes get made with the aim of preventing defects from being introduced.

The Plot Thickens

Alice on the other hand thinks a little differently than most. Alice believes the answer for reducing costs related to defects is by reducing the impact of each defect introduced. In her mind, she sees the only way to reduce defect counts to zero is to either add more and more gates, barriers and checkpoints thus providing additional layers of certainty, or to stop all change completely. Neither of these options smells right to Alice and she decides the first step for her is to start measuring the costs of the outages.

In her efforts to measure the costs, Alice discovers and then introduces a metric that neither Bill and Carl have seen before called MTTR. Alice explains to Bill and Carl that she doesn’t believe that speed and quality are naturally opposed to each other, but rather that the game to be played is how to be both fast and good. After Carl hears from both Bill, who wants to limit the number of defects introduced, and Alice who wants to limit the impact of each defect, he understands the logic of Alice’s approach, but that it’s just plain crazy to not try to reduce the number of defects. Carl instructs the team to go after both objectives, and limit the number of defects and reduce the impact of each one by measuring and driving down MTTR. When leaving the room Carl like every other executive says, “Remember that you can’t add any new headcount, we have to get costs down, not move them up!”

As Alice and Bill work together to establish the metric and reduce the number of defects, a funny thing happens. Both Alice and Bill see that automation of deployment tasks will positively impact both of their goals without requiring any additional headcount. Alice gets to reduce lead time on applying fixes while Bill gets to decrease the likelihood of errors as the automation scripts improve. While working through the deployment automation, it becomes clear that the exact same effect happens when the QA process is automated. Even funnier, when the previously manual processes are automated, peoples’ attention begins to free up to accelerate the pace of automating manual processes.

Again and again and again it happens, first in automation moving backwards through the development lifecycle and then in instrumentation to both lower the time to notice that a defect has occurred and also to catch a defect earlier in the lifecycle before it gets deployed. Defects and outages go down, speed and throughput go up and what before Alice and Bill can say “Bob’s your uncle”, their teams are doing everything they can to speed up the automation processes themselves to drive cycle time even lower. What a wonderful coincidence! Performance engineering has now become something that the teams now actively engage in.

The Moral of the Story

Did Alice, Bill and Carl live happily ever after again? Time will tell and they are more certain to get better results out of their teams given that they have visibility into, and are grounded by, a metric that puts how you respond to an error as more important than error prevention. Next week, I’ll go under the hood yet again and give a little more insight into how to create a robust, mature and almost qualitative view of my favorite metric: MTTR.

April 22, 2014
DevOps adoption can succeed where ITIL failed, if we let it
When I look at the future of DevOps adoption and acceleration I contrast it with what I feel was the failure of ITIL to really move beyond large enterprise IT shops. I am of the firm belief that DevOps has the legs to succeed and soar where ITIL sort of crashed and burned. OK if not crashed and burned, at least sputtered out. To understand why I feel this way, let me share a little personal history with you.

ITIL demands you to come to IT

I remember my first ITIL project. I was working at a large retailer and I was asked to help our IT department understand what this ITIL thing was and to help drive adoption of some of ITIL’s basic concepts. First on the list was to get a service catalog up and running. In the space of just a few weeks, I realized the main barriers to adoption and helped our IT executive team understand that getting IT consumers to understand, much less embrace an IT-centric taxonomy such as ITIL was not going to be an easy task.

This scenario points directly at one of the big reasons ITIL failed to achieve critical mass (i.e., adoption of the model outside of large enterprise IT) and also reveals why DevOps has a much better chance of success of achieving critical mass (as long as we ourselves don’t mess it up).

Mind you I’m not saying that ITIL is dead or that it has nothing to teach IT shops. I am saying that ITIL alone has not really changed much of anything inside of IT shops, let alone improved the reputation of IT with it’s consumers and customers.

When you contrast ITIL and DevOps several things really set the two approaches apart:
- ITIL makes it easy to blame the business for not talking IT’s language while DevOps continuously seeks to meet people on whatever terms they are capable of communicating in. DevOps inspires, while ITIL deflates
- DevOps is centered around improving the lives of IT workers everywhere while also improving bottom lines at the companies they work for. ITIL is centered around improving stability in the most efficient manner possible. I don’t know about you, but I just don’t see a room full of people, much less an entire organization getting excited or inspired by the concept of efficient stability. The only people likely to get sustainably excited by “efficient stability” are the accountants who can now start to push cost (i.e., people) out the door.
- ITIL at its core is reductive (or at least allows for reductive methods and approaches to grow unchecked). For those uninitiated in philosophy or how language influences our ideas and behavior, reductionism is when people essentially believe that a complex system is nothing more than the sum of its parts. This reductive approach at the core of the ITIL movement is what reinforced the division between Dev and Ops – because the concept of team chemistry has no place in ITIL given that the organization is nothing more than a collection of faceless workers who play out a prescribed role.
Reductive goals and processes, essentially reduce people and their efforts to numbers. The act of using quantitative targets as goals in large initiatives is a dangerous practice in general and grows more dangerous in larger groups and organizations – case in point the recent GM ignition failures that resulted in 13 deaths in an effort to save .57 cents per car affected by a defect.

Before you get excited, I’m not proposing that ITIL will kill people. I am stating that allowing an organization to be led by reductive philosophy allows a whole bunch of unintended consequences to come into play and often leads to holding dollars as singularly more important than anything else; sometimes even human life.

Whether or not you agree with my point that efficiency driven quantitative metrics are not inspiring, it’s kind of hard to argue with 40 years of scientifically proven fact that when “a measure becomes a target, it ceases to be a good measure”.

DevOps shops “live to serve”, while ITIL shops “serve to live”

DevOps contains a concept of craft that extends beyond efficiency. When a problem arrives in a DevOps shop (or at least one that really embraces the cultural ethos), the staff looks at the real resolution of that defect as a matter of personal pride. This pride is not a statement of “pride in self” however, but a statement in “pride in craft”. To hammer this contrast home, does ITIL by itself even have a deep cultural bias at all, much less an ethos?

This perspective is what allows DevOps shops to act in a manner that transcends the checkbox approaches that pervade formalist driven shops that believe in “adherence to process” above all rather than the “results the process is intended to achieve”. If you think I’m wrong, go ask your business clients how they feel about the DMV-like experiences they get when trying to get a problem solved.

This perspective is also why DevOps forms strategic pillars in automation and performance. Performance and automation efforts are not only about the manufacturing of time. Automation isn’t only a method of driving towards error free processes, or even a liberator of individuals from mundane tedium. Driving highly performant code and infrastructure isn’t only about conservation of compute resources.

Well implemented automation routines and highly performant applications are also a deeply creative expression of both development and operations practitioners. The same sense of artistry and elegance that was once reserved for just the software developer has been opened up to all IT practitioners. With the arrival of DevOps, being a great IT professional is not out of reach of any IT practitioner no matter their discipline.

We just can’t afford to mess this one up. In order not to, here is some practical advice:

As you go forth in creating or playing your part in your organization’s DevOps journey, look out for these pitfalls and pivot back to the center.
1. Reductionism – Don’t let numbers be the mission. This doesn’t mean don’t have numbers in your goals. It only means to make sure that your number oriented goals have a basis in making things better for people. When you hear someone say “We need to get this deployment routine under 4 hours!”, make sure that someone reminds everyone that the reason to get it under 4 hours is to allow employees to have a reasonably well balanced life outside of work. Off hours deployments that last four hours, when things go right, should not be a target that anyone is proud of. Getting people home to their families and allowing people to be able to keep a healthy sleep schedule is goal worth shooting for.
2. Formalism – Worshipping the shovel rather than the hole it digs is an easy trap to fall into for people whose jobs revolve around buiding and using tools. Remember that processes and tools are mechanisms intended to guide novices and intermediates through decisions. When you see someone turning a process or a tool into a sword and shield for working with others, make sure you remind them that results are better than processes.
People Over Process

Neither processes nor tools solve problems; People do. Processes and tools don’t create business opportunities by themselves; People do. Isn’t that why we chose this profession in the first place? To be useful in solving problems and to create opportunities to make things better while allowing people (ourselves included) to have a good standard of living.

Now go DevOps practitioners! Go kick some ass and make the internet awesome!
April 18, 2014
No DevOps Roadmap? No Problem.

I’m a huge fan of The Phoenix Project and I’m honored to count Gene Kim both as one of my professional inspirations and also as a personal friend. There is, however, a dangerous precipice inside of the Phoenix Project that, if misunderstood, could cause many DevOps practitioners and enterprise IT shops a daunting problem in adopting and progressing in the ways of DevOps.

Roadmap or Roadblock?

One of the main concepts discussed in both Kim’s The Phoenix Project and Goldratt’s The Goal is “the bottleneck” and what what to do about it. The basic premise explained is that all actions to increase throughput before a bottleneck, pile work up at the bottleneck and all actions to increase throughput after a bottleneck create a void of work to address after the bottleneck. The logical conclusion from this idea is that: in order to have a net positive effect on throughput, action must be taken at the, holistically viewed, tightest constraint on throughput within the operational system.

This concept model is both accurate and valuable as the IT shop and the manufacturing floor have a lot in common. The power of this concept model, also happens to be where the aforementioned dangerous precipice is revealed – getting alignment on which bottlenecks to focus on can turn into a multi-month effort in many enterprises which can require consultants, workshops, deep-dives, process studies and other activities that are at once highly valuable and not always likely to be approved by a shop that has not fully bought in to the three ways of DevOps.

One of the lesser understood secrets of corporate america is that even when you have formal authority, permission is required to do almost anything of substance. One “no” from the wrong party can kill even the most promising initiatives. This “veto effect” doesn’t require an absolute “no” either. Much like a parent saying “let me think about it”, the veto effect only requires the absence of a “yes”. This absence can take the form of “maybe”, “let me think about it”, “let’s wait and see”, or even a “run this by the other department heads and see what they think”.

Given that any reasonably sophisticated roadmap takes more than a few weeks to develop, vett and roadshow around a medium to large enterprise, a mature DevOps roadmap suffers from a classic chicken/egg paradox for many organizations. I extend my deepest sympathies to those who have found that a clear and easily understood roadmap can somehow both provide and require the catalyst to consensus and permission to proceed in the ways of DevOps. You can take heart, however, in the fact that neither you, nor DevOps, are alone in this conundrum as this is what can be referred to as “the corporate condition” that we must all bear.

While company may not always provide solace to the miserable, there is however, another way.

Just Follow the Sun

What is this one side road still available to you that has the power to relieve almost any non-alignment traffic jam? Start heading west! Just like the American pioneers, you can start following the sun to the wide open spaces of DevOps land where automation is plentiful and provable performance gains via instrumentation are gold. All you need is one or two comrades with some skills worthy of the road ahead and you can be off!

Just because you can’t get the unix team to automate environment creation, doesn’t mean you can’t head west and automate your build processes. Just because you can’t automate your entire production deployment doesn’t mean you can’t head west and automate your performance testing and benchmarking processes. Just because you can’t get everyone to agree on the tightest constraint, doesn’t mean you can’t head west and alleviate the ones you already know about. To head west on any of these things, you don’t need full concensus, you only need a traveling partner or two.

The Trail Has Already Been Marked

For mid to large size enterprises, a true DevOps transformation happens over the course of three or more years. Allowing a lack of consensus to become a total standstill is not the right answer. The tool at your disposal to overcome the veto affect is public consciousness.

Take UX design principles for example. It was less than a decade ago that design activities were considered a luxury item that produced pretty pictures; and then Apple happened and tipped public consciousness over. How many people are willing to casually veto design now?

When public consciousness to the value of DevOps tips over, it will no longer be you going to the powers that be to get the “OK to proceed”. It will be them coming to you and asking “what is taking so long?”. In that one moment, you’ll be able to fire back at them: “Here’s my roadmapping proposal. I’d love to have you be my business owner!”

April 9, 2014
DevOps & UX: A Star-Crossed Love Story
“Two households, both alike in dignity
In fair Verona, where we lay our scene
From ancient grudge break to new mutiny
Where civil blood makes civil hands unclean.
From forth the fatal loins of these two foes
A pair of star-cross’d lovers take their life
Whose misadventured piteous overthrows
Do with their death bury their parents’ strife.” ~ Romeo and Juliet: Prologue, William Shakespeare

For more than 15 years I have watched as they battle. The chasm between the two houses was so wide that I often wondered if it would ever be bridged. I never wavered from my view that we were all one house striving to find common ground for making working solutions for the world around us. Sometimes there were allies beside me. Sometimes I stood alone, For the first time in my professional history, it seems like the blood-feud could be put aside. Is it time for peace? Even more ambitious, is the day of an alliance at hand?

The Feud

For many years, corporate IT and finance were the only voices in the room. Ruled by an unquestioned and unmeasured desire for quantification and linear solutions, design considerations were usually an afterthought. All parties were subservient to the almighty ROI which ruled over all with an iron fist. Claims of possible productivity gains through better tooling were always met with doubt and even when taken seriously, you could not claim productivity gains unless you were willing to commit to lowering headcount. Tooling for technical employees to manage the environment and its operating data was so scarce, that there were more applications with no maintenance tooling than applications with operational plans. Operational changes were made by going into databases by hand and making changes in a “hit or miss until you hit” fashion.

In the late 90’s UX showed up to the party and with a basis in humanism, the equation changed. This was the time that the “great misunderstanding” happened and 15 years were wasted building up a corporate landscape strewn with enmity and false narratives. The “great misunderstanding” goes something like this:
- “Let’s make something that people can use” – UX practitioner espouses user-centric design and introduces new methods and artifacts into the SDLC processes which take interaction design specifications to a new level in corporate America
- “That’s going to take more time and I don’t agree that it’s any better than what I could do myself” – IT staff view this incursion as a mechanism to take what little time they are given to complete projects while also limiting their creative outlet
- “I don’t care what you think and neither does anyone else” – UX practitioners view the response as a denial of their value proposition and a fundamental lack of care about the people who interact with technology
Several of the basic assumptions in these archetypal interactions between disciplines were short-sighted. For example:
- The “more time” fallacy – adding a design process may take more time if you only think about a single iteration in development and that’s only if you are lucky enough to have a complete interaction model worked out in your head that doesn’t have any design holes in it. Designing interactions on paper saves time in the same way that technical design saves time over “jumping straight to coding”
- The “fighting on the wrong front” mistake – It was never the goal of UX to remove IT budget. The goal was to make things better for the humans that interacted with technology while creating business value. The two different disciplines never saw their goals as aligned. We will never know what the alternate history would have been like had UX and IT teamed up to get larger budgets in an effort to create better and more sustainable experiences for users, developers AND operators.
- The “reductive labeling” error – Both IT and UX created a narrative where the interaction and visual design tasks were the “creative tasks” for “creative people”, while the development was the “technical tasks” for “technical people”. These labels have stuck around for the last 15 years and we are still paying the price for removing the idea that IT staff is fundamentally non-creative and devoid of craft.
- The “self-reinforcing paradox” tragedy – It was never the goal of IT to demean the UX discipline. When one neglected child is joined by another neglected child, the two will almost unavoidably fight over the little attention afforded to either of them. The irony here is that UX’s basis in cognitive science could have possibly seen the hostile response from IT for what it was; a cry for help. Instead, UX took the bait offered by IT and thus we are here 15 years later with a wide chasm between two disciplines that should be united by common causes:
  - a desire to make awesome stuff
  - a desire to create sustainable solutions
  - a desire to express one’s self at work
  - a desire to improve the lives of their customers and employees at the same time
Oh The Humanities

When IT resources and UX practitioners first came together the familiar plot “boy meets girl, boy loses girl” emerged and we may now, thanks to DevOps finally have arrived at the time when “boy gets girl again”. Gene Kim and the folks at IT Revolution Press have hit the target dead center in their manifesto:

“Working in most IT organizations is often thankless and frustrating. People feel as if they’re trapped in an ever-repeating horror movie, helpless to change the outcome The organization abdicates their responsibility to ensure that IT is managed well, plunging the department into relentless intertribal warfare between development, IT operations and information security. And of course, the auditors.

What inevitably results is chronic under-achievement. The life of an IT professional is often demoralizing and frustrating, typically leading to feelings of powerlessness and rife with stress which seeps into every aspect of life. From stress-related health problems, to social issues, to tension at home, it has become clear that the current state of the IT professional is not only unhealthy, but unsustainable.

As people, we’re wired to contribute and to feel like we’re actively making a difference. Yet, all too often when IT professionals ask their organization for support, they’re met with “you don’t understand,” or worse, a barely masked, “you don’t matter.”

As a human being, this is the worst response we could receive. Because the opposite of love isn’t hate — it’s apathy.”

Despite what some IT step-children will tell you, UX is not about making pretty pictures. UX as a discipline was formed to combat apathy and contempt by corporations through a mechanism of introducing empathy. Similarly, DevOps wasn’t formed to “make web pages faster”. DevOps clearly has a tactic of emphasizing performance (as well as other technical tactics), and performance improvements are clearly one of the benefits that come out of a DevOps organization. DevOps was formed to make IT a more sane and human space inside companies for IT employees, it just so happens that DevOps produces other impacts that are financially beneficial for enterprises.

Avoiding The Fates of Romeo And Juliet

Knowing that waters of DevOps and UX flow from the same wellspring allows the practitioners of DevOps several opportunities to repair the tragedies that have already befallen our enterprises before they escalate to the Shakespearian proportions. Three broad categories of opportunities would be:
1. Enroll UX into the DevOps movement – Knowing that we are aligned creates multiple possibilities to bring many industry and enterprise change agents into our cause. How much would the DevOps adoption curve accelerate if people like Don Norman, Jesse James Garret, Alan Cooper or Steve Krug were visible advocates for DevOps? Support from UX at the leadership level in the industry and within your enterprise is an advantage you cannot afford to pass up. The number of consensus driven organizations are just too many to count and the more groups and individuals that believe in advancing our cause, the better.
2. Learn from UX’s travails – Creating artificial barriers to discipline inclusion is ultimately an adoption limiting factor. When we make things pass-fail or turn collaborations into confrontations or competitions for who is DevOps and who is not, we all lose out on the what a wider community could offer. Hold yourself accountable to pivot confrontations into constructive opportunities where learning can happen for people of different disciplines.
3. Take some pages from UX’s playbook – UX’s biggest lever for creating believers is by involving non-UXers in the research methods and activities (think about the last time you went to a usability test behind the 2 way mirror and saw a person struggle to use a system you were part of implementing). One of the best things we the practitioner community can do is to demonstrate what life is like without DevOps in real-life scenarios. When someone really attaches to how ugly a manual process is, or how soul-crushing it is to not be able to self-actualize in their job, they can’t help but want to be a part of making it better. Problem solving is hard-wired into the human condition, let’s leverage it.
April 1, 2014