devopsy.kinsta.cloud

Category: Enterprise DevOps

Five Top Tips for Cloud and DevOps Automation

A while back, I overheard someone asking why many of the ‘cloudy people’ have moved on to DevOps.

One reason is certainly that cloud is at the heart of DevOps. Many of the grounding principles of DevOps – rapid iteration, agile development, automated testing, continuous delivery and continuous integration – are barely even possible without cloud. However, I think it runs even deeper than that. DevOps and Cloud are both fundamentally enabled by the same technology I have been working with my entire career – automation.

When IT leaders get excited about cloud, it is mainly because of automation. Sure, cloud providers get excited about essential characteristics like broad network access and resource pooling, because that is how they maximize reach and minimize costs. Similarly, CFOs get excited about the measured service, because that means they only pay for what they use.

However, according to a recent Luth Research study, most IT leaders prioritize the higher-order values enabled by on-demand self-service and rapid elasticity – faster innovation, superior performance, better scalability and higher resiliency. These capabilities are deeply rooted in automation – of provisioning, deployment, configuration, scale-up/scale-out and other processes.

Similarly, DevOps – beyond a certain point – also builds on a foundation of automation. As Rackspace CTO John Engates noted in his recent column in this journal, “DevOps is more than a set of automation tools.” He is right, of course. You can achieve a great deal through cultural and structural changes – better collaboration, shared goals, team building, process improvements, etc.

However, tools for release automation, configuration automation, process automation, test automation, service virtualization and provisioning – just to name a few – are adding tremendous value with a rapidly maturing DevOps automation toolchain. Indeed, in recent research from Vanson-Bourne, over half of senior IT decision-makers worldwide rate automation as the most important DevOps component – above such putative stalwarts as agile development, collaborative teaming and process alignment.

Looking at the history of enterprise automation, and considering my own experience, I believe there are many common principles that you can apply to both cloud and DevOps. Here are just five of my top tips:

Don’t Automate Bad Processes

Automating bad process just makes bad things happen faster. This seems obvious, but I often see automation that only encodes (and accelerates) a bad existing practice. To avoid a high-speed train wreck, first identify, catalog, document, review and refine key manual processes to make sure you are automating the right activities, at the right time, in the right way. Solve for simplicity, speed and throughput, even before you apply automation. Remove friction across silo boundaries – departmental, technological and disciplinary. Start simple, be inquisitive, and iterate methodically. Then make sure you automate deeply, thoroughly and efficiently.

Cross-skill Existing Resources

Great people are hard to find and salaries for cloud- and devops-relevant skills are amongst the highest in IT. This explains why, according to the same Vanson Bourne research, over 70 percent of DevOps leaders are retraining existing personnel, while only 50 percent are hiring new resources. Look for staff with a history of automation in their resume, through multiple waves of technical change, rather than specialists in one automation technology or product. Find the IT equivalent of a “good athlete” rather than a “position player”.

Security Cannot Be an Afterthought

Automation systems and the staff responsible for them have always needed highly privileged (and frequently shared) user IDs, whether automating a mainframe IPL or scripting a production code release. This is an audit red flag, and can result in high-impact incidents, whether accidental or malicious. Identity and access management provides essential audit and control, while giving users (including automation tools and APIs) easy access to the resources they need (locally and in the cloud), when they need them.

Document, Measure, and Backup

If you do it right, automation quickly becomes pervasive and mission-critical. Without documentation, you create inscrutable systems, tribal knowledge, the hero and other ‘anti-patterns‘ in your Cloud or DevOps machine. Without measurement, you cannot know if (or when) complex autonomous systems run amok or stop working. Without backups, critical systems are prone to irrecoverable failure. Cloud and DevOps teams must value and prove traditional disciplines like documentation, measurement and backup – painful as they may be.

Build Awareness into Automation

As automation becomes mission-critical, it must be resilient to errors, failures, exceptions and other unexpected conditions, and still deliver a correct result. Remember when ‘autonomic computing’ was a thing, when we dreamt of ‘self-aware’ systems? Similarly, concepts like ‘desired state’ and ‘idempotence’ go back at least as far as mainframe and server automation. Similar concepts apply to cloud computing and DevOps too. In both, it is critical that automation routines are aware – of their own state and of the surrounding environment – so that they can test for and accommodate unexpected conditions, and still deliver the expected results.

Given the depth of history in automation, and the many parallels with modern computing, some may claim that cloud is nothing new, that we have been doing that since the mainframe; while others will say something similar about DevOps. I think they are wrong, but there are definitely parallels, and George Santayana has a point – we should be at least informed by history, if we are going to try to write the future, while avoiding the mistakes of the past.

June 16, 2014
Enterprise DevOps: organizational change a must

There has been a lot of discussion in the blogosphere, including here at devopsy.kinsta.cloud, lately about the challenges of adopting DevOps in the Enterprise space. These challenges at the end boil down to the need for organization change. There are organizational structures that appear to hinder DevOps adoption in large enterprises, that are inherent to enterprises and smaller companies do not have. In enterprises different functional teams – Dev, Ops, Security, QA, etc. may be under totally different management chains. It is not uncommon to have to go up thru several layers of management to reach a common senior executive. In other cases, there may be an entire software supply chain in place that is made up of not just internal but also outsourced or external suppliers. These suppliers may or may not ‘play ball’ when it comes to the changes in processes required to adopt DevOps. Especially if they are bound by outsourcing contracts that do not make change possible or profitable for them.

DevOps as a philosophy has had as its centerpiece the principle that all the application delivery teams (Dev, QA, Ops, Business Analysts, Architects, Security…) need to align better and collaborate towards the common goal of delivering applications, in a efficient and constantly improving manner. This is a people and organizational principle, not just a process centric principle. To me adopting this is more important when adopting DevOps than any process or tool.

This post discusses key guiding principles on which this DevOps driven alignment of different teams in an enterprise should be based. They are designed to improve collaboration between these teams and to break down the proverbial ‘wall’; to end the water-SCRUM-fall, when it comes to the relationships between the different application delivery stakeholder teams in the enterprise. This is not an easy effort. It depends on the existing culture at the organization, the presence of outsourced suppliers, politics, contracts and above all, the desire to change.

These guiding principles are:

1. Shift Left:

Organizationally the goals of DevOps are to bring all the teams involved in application delivery together closer. Not just at deployment time, as they may do already, but all thru the delivery cycle. It requires Ops to allow Developers to take more responsibility of the operational characteristics of the applications they are building, for example. In turn, it requires Developers to keep Operational considerations in mind while building their applications. It requires analysts and architects to think holistically and work with Dev and Ops teams as they design and enhance the application. This is referred to in the DevOps world as ‘Shift Left’. As in shifting towards the left, Ops responsibilities. Shifting it to earlier in the software delivery lifecycle, towards Dev. It is shifting left the testing of your software to as early in the lifecycle as possible. It is shifting left the point at which collaboration between teams begins. It is shifting left the responsibility to deliver code to production – way left. Can someone in finance be held responsible for code not being delivered on time because they delayed approving a budget line item?

2. Collaborate across all teams:

Different functional teams typically use different tools to manage their projects and activities, for change management and (outside of email) different collaboration tools. In order to better collaborate across the organizational boundaries, all teams should start using the same Change Management and Work Item Management tools or in the worst case, use tools that are integrated. This allows seamless visibility across tools and traceability between respective Change Requests. Real-time collaboration using a common tool is obviously the best scenario.

3. Build ‘Application Aware’ Environments:

As we move towards Software Defined Environments, we have the ability to build, version and manage complex environments, all as code. All of the benefits of this are moot if the environments are not a perfect fit for the applications and more importantly the changes to the applications being delivered. The goal is hence to build Environments that are ‘Application Aware’ or are fine-tuned for the applications they are designed to run. No more cookie cutter Virtual images for all kinds of applications. More importantly, one needs to ensure that the environments are architected in a manner to allow for the evolution of the applications, both as they are developed; as they are projected to change or as they may evolve in the future. This obviously, would require close collaboration between Dev and Ops. Development Architects and Product Managers need to work with the IT Architects and Operations Managers to architect environments and their projected evolution to align with that of the application. Not only that, but also allow enough resilience in the environments to allow for unexpected change. For example, massive change caused by a super successful App. Think Instagram and how the founders had to keep changing their server environment almost daily as millions joined their service.

4. Environment Sprints:

Agile Development Principles prescribe that at the end of every Sprint, developers have a build. An executable build that runs, even if does not do much in terms of functionality. The next evolution of this concept is to extend it to environments. This would mean that at the end of each Sprint, the Dev team would have an executable AND the Ops team would have an environment, a production-like environment ready that the executable can be deployed on. This would allow the build to be tested. That too in a production-like environment. This would also provide immediate feedback to the Ops teams on the behavior of the application in their environment and use that feedback to improve the environment. This requires the Ops and Dev teams to align better. They will both have to use a single Sprint plan for releases – of their Builds and Environments respectively and will provide feedback to both at the end of every Sprint, using which Dev can enhance their Apps to improve functionality and performance and Ops can enhance the environment for the same. Leveraging full stack deployment – deploying at each instance the full stack that includes the environment, middleware and the application being deployed – could very well be called the ‘holy grail’ of DevOps. (Brief commercial break – IBM just announced the new offering UrbanCode Deploy with Patterns that provides full stack deployment leveraging OpenStack HOT patterns).

The Human factor

These principles are designed to foster Application Delivery team alignment from a teaming perspective, from a people perspective. These however do not replace the need for good old team building. Whether it is thru face to face get togethers or in todays distributed worlds, thru regular virtual meetings and online collaboration in real time. The best teams are the ones where the people know each other, trust each other, look out for each other and when there are challenges, know who to pick up the phone (or launch Skype) and talk to.

June 11, 2014
Lean versus the bandwagon, lean wins

I’ve just finished reading “The Phoenix Project,” by Kim, Behr and Spafford. To be honest, it was on my reading list but had I not picked up a copy at Camp DevOps I don’t know that I would have ever read the book. My main issue with books of this ilk, such as “Who Moved My Cheese” and “The Energy Bus,” is that the story is often contrived for purposes of delivering the learning points. So, I was surprised by how much I was able to relate to the issues in Phoenix giving it an air of reality I was not expecting.

Phoenix follows the life of a new VP of IT Operations through his early weeks on the job. Plucked from obscurity as the Director of Mid-range Computing after his management chain was seemingly asked to depart the company, Bill’s eyes are opened to a world of heroics and chaos that surrounds him in IT and has had detrimental impact on the company’s financial and competitive performance. For anyone who has logically questioned, “how can that happen?” in their own companies, the authors do a great job of illustrating the paths that lead to the unimaginable outcomes we see every day in our own IT world.

That said (spoiler coming), I ask the authors, did we really need to make Bill’s dad an alcoholic? Sorry, completely tangential point, but interestingly, one of the few times where I thought the authors had reached into their “drama box” for reasons that are still unclear. Nonetheless, after reading this book, I reflected upon the content that is being created around DevOps movements and realize that there is a polarization effect occurring.

Camp 1 I categorize as the “lean camp”. This camp represents those that believe DevOps should be the application of lean manufacturing principles on IT. In this perspective IT is represented as a critical hub through which business is enabled and transacted. In accordance with this perspective, work enters the hub through multiple endpoints and must be prioritized and coordinated in order to ensure that a) critical IT resources are sourced appropriately and with maximum value to the business and b) constraints and bottlenecks are minimized, thus allowing work-in-progress to complete successfully. Anyone who has studied any aspect of logistics, manufacturing or supply-chain management understands the number of variables involved in this type of scheduling, but applying this to IT is significantly more challenging since it requires the cooperation at both the 30,000 foot view and the 10 foot view, with the latter required to understand the impact of scheduling conflict and help foster collision avoidance that could lead to outages.

Camp 2 I categorize as the “bandwagon camp”. This camp represents individuals that are using DevOps as a lever for their careers. Certainly, DevOps is the hot topic of the day and having it on the resume is certainly good for a few extra brownie points and higher salaries. Ask these individuals what DevOps is about and you hear answers, such as, “it’s about changing the culture in IT,” “it’s about getting development and operations to cooperate more effectively,” or my personal favorite, “it’s about automation.” The truth is that DevOps needs aspects of all of these factors but driven under an umbrella of work described in the Camp 1 description above. Acted on independently under the guise of DevOps and we will see DevOps nose dive into Gartner’s trough of disillusionment very quickly.

Will things get fixed in Camp 2? Probably. Certainly implementing controls around who can make changes and when they can occur may help many businesses limit the unplanned work that is crippling their ability to address other work efforts. However, since this is not the focus of the DevOps effort, unplanned work will continue to come from other areas, it’s just the nature of the beast and without correcting the actions that foster unplanned work, it will continually have a negative impact on the business.

The authors of Phoenix have done a fantastic job of providing IT leaders with a goal for their DevOps programs. If they are wise, they will rapidly adopt the principles defined therein and make sure that all other so-called DevOps efforts in the company are aligned for purposes of reigning in the chaos and free-wheeling efforts of rogue IT employees. I recently met with a VP of IT for one of the major food distribution companies. I was awed by how effectively his IT organization operates, but now I realize, that his company is primarily a logistics and supply-chain company that happens to move food. It only makes sense that the principles they use to run their business would be adopted for IT as well; especially with their IT leaders coming from other parts of the business. In combination with Phoenix, the meeting locked it for me that the lean principles are key to positioning IT as the business enabler needed to be competitive in the current era.

June 2, 2014
Avoiding Parrot Processes

One of the things devops practitioners are tasked with is the provisioning and configuration of all sorts of infrastructure. Application servers, web servers, load balancers, proxies and database servers are among the lengthy (and no doubt growing) list of “boxes” devops needs to get up and running to support just about any given application today. One of the key value propositions of a devops approach to operations is that it can reduce the time it takes to get applications to market by getting them up and running in production faster. That’s increasingly important as we move into the Era of Things, driven by extreme connectivity of everything, because all those “things” need are going to need to be talking to applications on the back side.

So speed is of the essence, but not at the cost of accuracy. We’re encouraged, then, to automate tasks and orchestrate processes to achieve that speed. The side effect of codification of tasks is higher accuracy (aka fewer mistakes).

So far so good.

But you may notice that I have (and will continue to) clearly delineate between “automate” and “orchestrate”, between “task” and “process”.

The reason for that is that, simply, they are not the same thing. We should strive not to confuse these two because while the two share some benefits, there are other benefits that are achievable only by one or the other – or the combination thereof.

Process optimization, for example, can’t be achieved by simple automation. Automation is concerned with a single task – launching a web server, configuring a web server, stopping a service. Orchestration, however, is concerned with automating, if you will, the execution of a workflow – of a process. A provisioning process may be comprised of multiple tasks and involve multiple systems. An “application” is not just a single server, after all, it’s likely several servers – web, app and database in a traditional three-tier architecture.

The goal of orchestration is not just to automatically execute a service, which brings speed to the deployment process and gets applications into production faster. It also affords an opportunity to streamline – to optimize – those processes for even greater gains in deployment velocity.

One of the simplest ways to optimize a process is to eliminate repetitive steps. This entry on Vagrant explains it fairly well :

“We could just SSH in and install a webserver and be on our way, but then every person who used Vagrant would have to do the same thing. Instead, Vagrant has built-in support for automated provisioning. Using this feature, Vagrant will automatically install software when you vagrant up so that the guest machine can be repeatably created and ready-to-use.” [emphasis mine]

— Vagrant Docs “Getting Started | Provisioning”

Now certainly the step of installing a web server on a new box with vagrant is hardly all that time consuming, it’s just a single line of text. But every line of text must be manually typed, and that introduces the possibility of fat fingering the command, which either consumes time in fixing it or introduces an error into the system. Neither are desirable results.

Ensuring that the process is repeatable and orchestrated by automating all the tasks that comprise that process can optimize it by eliminating redundancy. That results in greater deployment velocity.

So while you’re “doing devops”, remember to evaluate the processes and consider whether or not they’re repetitive, and how you might be able to eliminate that repetition through more modern architectures or toolsets.

May 20, 2014
Doing Devops: URL Management
A fairly common task for operations folks is managing URLs. URL management either uses redirection or rewrites to manage things like vanity URLs or marketing campaign URLs or even just getting users to current URLs from old, deprecated ones. The business purpose behind the need to manage URLs aside, the process for doing so can become quite unwieldy and time consuming.

That’s because traditionally URL redirection and rewriting is a layer 7 proxy or web server function that requires modification of a configuration file. For example, here’s a simple URL rewriting configuration excerpt for Apache:
```
01.RewriteEngine On
```
```
02.RewriteBase /books
```
```
03.# If there's a cookie called thgtp-pre-version set,
```
```
04.# use its value and serve the page
```
```
05.RewriteCond %{HTTP_COOKIE} thgtp-pre-version=([^;])
```
```
06.RewriteRule ^guide-to-node$ %{REQUEST_FILENAME}-%1.html [L]
```
The key here is that it’s in the configuration file. This is just one rewrite rule. Imagine needing hundreds of these rules and then imagine them being changed on a weekly or even daily basis. Not only is this likely not the job you signed up for, it’s inevitable that at some point you’re going to fat finger a rule change and you know what that means. Yup. Apache no start. And you get to figure out where it is.

In some architectures, rewrite/redirect rules are simply made to be part of the web server configuration. Each and every web server has to have its configuration file updated when a single rule changes. More modern approaches take advantage of proxies, which greatly minimizes the number of instances of Apache (or whatever proxy you’re using) that need to be updated.

Regardless, it’s a pretty mundane and boring thing to do, and honestly, the whole process of having marketing (or whoever) kick off a request for a new rewrite/redirect rule and having the ticket make its way to your desk probably takes more time than it does to enter the rule and type restart httpd.

So how about we leverage some programmability in the network to automate this bad boy and, if you want, hand off the tedious management of the URLs to the people who have ownership of them (business and marketing folks)?

Programmable Proxies

I’ve explained before about programmable components, particularly those in the data path, and one of those components can be described as simply a “programmable proxy.” Proxies are not unfamiliar to dev or ops, so I’m going to assume we’re good on that front. The programmable piece comes in when you basically move all the logic that’s encoded in the configuration file (like rewrites and redirects) into a programmatic environment instead. So rather than configure the behavior of the proxy, you’re going to code the behavior of the proxy instead.

So how does that help, you might ask? Well, for starters it makes the phrase “infrastructure as code” ring a heck of a lot more true because, well, it really is code. That means it becomes an artifact like any other piece of code and gets managed like code. That means simpler rollbacks and version control.

Second, it means if you’ve got a robust enough environment you can extend the logic to include data sources like a database (SQL or NoSQL, your choice) that manage a mapping of old URL to new URL. And if it’s in a database, it’s a pretty simple thing to have someone put a pretty HTML form in front of it that enables business folks to manage that list.

The actual execution is straightforward: the proxy receives the request for one URL, say www.example.com/TVad and queries the database, gets back the real URL, say www.example.com/marketing/tv/2014 . It’s not that much different than a configuration-based approach, with the exception that you’re able to dynamically determine the mapping rather than hard-code it in a file.

Now, what the proxy does with it is up to you. You can send a redirect to the client with the “real” URL, or (a better choice if you care at all about performance) rewrite it and send it on to the right server.

A programmable proxy should be a key tool in every devops practitioner’s back pocket. With both network and application fluency, a programmable proxy provides the foundation for a variety of architectural patterns that rely on dynamic evaluation of application-layer requests in real time. The programmable proxy is well-suited for these tasks because it can be inserted into the data path nearly transparently, and doesn’t require special networking (or perhaps more importantly, the cooperation of the networking teams).
May 12, 2014
The DevOps master process: design

In speaking, counseling and talking to people with regard to DevOps, I often frame the discussion in terms of the DevOps master process. While all of the parts of the master process are important, one aspect very often is overlooked. I call this often overlooked aspect the design stage. The design stage includes the gathering and visualization of requirements. If we assume that our goal is to deliver higher quality and more consistent and resilient deployments, then we must include expectations for how the software should operate and perform.

When we are able to streamline and optimize the DevOps master process we ensure maximum velocity and agility for delivery of IT services to the business. The importance of the master process is that it applies for both businesses that develop their own software as well as those that operate using only Commercial-Off-The-Shelf (COTS) and Software-as-a-Service (SaaS) products. Before discussing what is involved with design in more detail, let me take a moment to better define what I mean by the DevOps master process:

1. Design

2. Build

3. Test

4. Stage

5. Deploy

6. Operate

In future articles I will explore each of these elements in greater depth. For this article I wanted to focus in on the design element specifically.

To ignore the design aspect of the master process means that we have no linkage for what we are building or testing during the Dev phase and no concern for performance and service levels in the Ops phase. Simply because there are no tools that help to automate this aspect of master process, does not mean that it is not an important aspect to fulfilling our DevOps mission; it simply means that we need human intervention to ensure that our requirements are met at each stage of the process.

However, this is not all a one-way street with requirements only dictating to operations. In the past, traditional requirements gathering often focused on function with very little thought given to performance or consumption. Through our new DevOps collaboration, we introduce operations into the requirements gathering phase, which means that we can capture critical elements, such as service levels and usage models, ensuring all work across the master process leads to meeting these operational requirements. This will, hopefully, limit the spike in support calls operations needs to respond to immediately after deploying new software into production.

Perhaps the reason we don’t yet hear about the importance of this is because it’s difficult and the tools for requirements gathering and automation around testing against the requirements traceability matrices just don’t exist in the current popular DevOps tool chains. Dare I say, these have been completely ignored by the open source community? It would be understandable since open source communities grow out of interest more so than need and developers are more inclined to work tools and technologies that are interesting to their own efforts. Perhaps we will see better integration of automation of testing not only features but also service levels, from the enterprise DevOps software providers, such as IBM who are applying their Rational suite of tools toward DevOps initiatives.

Regardless, to achieve optimal benefits from DevOps, we need to automate around the entire master process. This means that we must translate requirements into automatable tasks by linking them to specific unit tests and load stress testing and ensuring that the requirement is only satisfied once these tests have passed.

May 5, 2014
The Push to Make Ops More Social

Given that social coding is a thing, what about the other side? Operations has the same image of lone wolves solving problems in the dark. The culture of on-call is built around the concept that when you’re holding the pager, you’re the one on the front lines, so that everyone else can sleep.

Social operations, the ops in DevOps, is going to be about the same things that made social coding important: knocking down walls between individuals; making the knowledge of a system or infrastructure explicit and available, in context. But social operations is more difficult in two major ways: operations problems are often multidisciplinary, and often temporal – meaning that you have to be social when the situation arises,not on your time!

When I say that operations problems are multidisciplinary, I mean that solving a problem in an operational context might require a number of people with different domain expertise. The solution is going to be much more complex and may not be portable to other people. Fixing something that’s wrong with a puppet script inside of our own infrastructure is fundamentally different than contributing a bug fix back to an open source project, or otherwise contributing to an open development environment.

And when I say that operations problems are temporal, I mean that many of the forms of social engagement we’re getting used to in a social coding environment – forums, pull requests, etc. – are eminently useless when you’re in a firefight. When something’s wrong enough that you’re being paged at 2am, waiting for someone to respond to a comment thread on a pull request isn’t an option. You have to be able to get the right eyes on the problem, in a timely manner, if you want to solve infrastructure problems faster.

So, how can we make the operations side of DevOps social? What parts of “social” are useful in the context of an operations issue?

Opening up the knowledge of your team is a good place to start. Having the ability to get people involved is another good thing. Basically, what can we do to make it so that you know what you need to know, whether that’s how a problem was solved in the past, who’s available to get help, or what else is going on. That’s what we’re trying to figure out at VictorOps.

May 1, 2014
DevOps and Change Management
In “The Phoenix Project”, the character Erik introduces the underpinning principles of DevOps as “The Three Ways”. Let’s talk about the First and Third Ways and how change management works with DevOps, but first let’s spend a few minutes talking about baseball caps.

Last weekend I was looking for my Atlanta Brave’s baseball cap and found three as I expected for a family of three with two adults and one child. Since they all looked alike, I compared sizes to figure out which one was mine and was surprised to find they were all the same size! How can one size fit three very different sized people? The answer was simple: They have an adjustable strap. Let’s use this idea to explore the relationship between DevOps and change management.

One size fits all with an adjustable strap

IT is being asked to iterate faster and keep our systems highly available while doing so. Those two ideas seem at odds: How do we fail fast and keep our infrastructure highly available? Change management can help!

Many people I speak with view change management as an anathema, as being incompatible with the principles DevOps. The word “onerous” comes up a lot. Inflexible is also quite popular. I understand the sentiment and would like to help the community see change management differently.

I think of change management as a continuum spanning rigorous to mild, heavyweight to lightweight. Just like the baseball cap with an adjustable strap, change management can be adjusted for as needed.

The First Way

When describing the First Way, Erik invokes Deming’s quote, “appreciation for the System”. In “The Top 11 Things to Know about DevOps” Gene Kim refines the idea as, “seeking to achieve profound understanding of the system”. In my article on devopsy.kinsta.cloud titled “Systems of Things”, I assert that understanding the system is required before improving the system. It seems pretty clear the First Way is about understanding the system we want to improve. It is also about delivering business value to the customer.

The Third Way

The Third Way is about encouraging experimentation and building a culture that rewards risk as a way to speed maturation. Rajat Bhargava expands upon that idea in his article, “DevOps Demands we Fix Twice” where he asserts that we should release quickly to speed product maturation. It seems pretty clear to me that the Third Way is about speeding maturation of the features and functionality that deliver value. The Third Way suggests we iterate faster, accepting some amount of risk as a trade-off. I assert this risk is solely focused on features and functionality and not infrastructure.

Tighter for infrastructure and looser for features

After reading “The Goal” I began to think of code, the features and functionality that is ultimately delivered to the customer, as the material that flows through the factory and IT infrastructure as the machinery in the factory. The machinery factories only exist to process material. IT infrastructure is analogous to machinery; it only exists to allow features flow through to customers.

In a well-run manufacturing facility the material flowing through the factory and maintenance on the machinery are both governed by well-defined and well-understood processes. These processes are the analog to IT change management.

The machinery should only be down for maintenance at very specific times. At all other times the machinery should be ready to produce widgets. The same is true for the Ops part of DevOps; Ops exists to allow feature and functional to flow through to the customers. Ops should endeavor to keep the systems up and available for Dev to push code.

Practical Application

In practical terms, this means that infrastructure changes and code releases should be treated very differently. The frequency and timing of code releases should be decoupled from the frequency and timing of infrastructure changes. Infrastructure changes should not impact code releases and vice versa. Infrastructure changes should come with a high amount of rigor while feature functionality should come with much less.

A single change management process accomplishes this by:
- Decoupling code releases and infrastructure changes when possible
- Allow the frequency and timing of code release to be independent of infrastructure changes
- De-conflicting infrastructure changes and code releases
I think my colleague Stephen Fishman summed it up nicely, “The mission of Change Management… is to simultaneously increase the frequency of intentional change executions while reducing the duration and impact of interruptions for services provided….”

Rent to own

It can be tough to see change management and DevOps as compatible. Instead of agreeing or disagreeing, just try the concept on for a while. Rent the idea (it’s free) that change management rigor is a continuum, an adjustable strap and see how if it works for you.
May 1, 2014
Does DevOps spell death for Line-of-Business (LOB) Applications?

Perhaps wrongly, “DevOps” has been associated with “new” and “that thing for the cool kids” and the “hipster developers”. This is used both as an excuse and as a source of animosity against the movement. But more subtly, I sometimes observe IT departments that acknowledge DevOps as a thing, but claim it wont work for their applications. I always pause and ask: What are the applications they are referring to, and are they correct?

(more…)

April 30, 2014
DevOps storytime: MTTR vs. Goodheart’s Law

I am not a fan of metrics. Wait. That’s not right. Let me try again. I am not a fan of singular metrics that stand by themselves. Nope. That’s still not right. I loathe singular metrics that stand by themselves as if they themselves are solutions to problems. Ok. That works. Let me explain.

Last week, I told a tale of a secret passageway into DevOps called Mean Time To Resolution (MTTR) and explained how the introduction of this one tiny metric can magically open the door to all things DevOps. Given that MTTR and cycle time are inextricably linked, any shop that is serious about MTTR will sooner or later find itself using DevOps practices.

Metrics + Crafty Humans = Doom

This is very different than two weeks ago when I told a different tale. A tale of how the very origins and roots of DevOps gives the DevOps movement an opportunity to do what ITIL could not; Change the relationship between business and IT. In the telling of this tale, rife with philosophical terms like reductionism and formalism, I quoted a brilliant English economist, Charles Goodheart, who noted that “when a measure becomes a target, it ceases to be a good measure”. Goodheart discovered that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” The layman’s explanation is that humans are crafty little buggers and will find ways to artificially hit any target because the quantitative systems can always be rigged.

So what is a DevOps voyager to do? Introduce the metric, get the change, and then proceed to watch devops die a slow death? No, my fellow DevOps devotees, that shall not be our fate. We can avoid this fate by knowing that halfway down the path of the secret passageway known as MTTR, there is a fork!

More Metrics to the Rescue!

Instead of introducing and presenting a flat and un-nuanced view of MTTR by itself, we can get a little crafty ourselves and find an appropriate hedge against Goodheart’s lay and instead present a weighted view of MTTR with a few other key metrics to allow for a more sophisticated view of the health of the IT factory floor.

Apply Weighted Averages – Weight the MTTR of defects and their associated resolutions by priority to intentionally skew the measurement towards the highest priority defects. Clinging to an absolute view of defect free software doesn’t work for a large code base with limited funding and time (aside from being scientifically impossible to achieve via godels proof)

Track Inbound Defect Counts – Track and present the inbound defect count, keeping the inbound defect count low is important, but pushing it to zero is more likely to be indicative of a shop that is more concerned with perfection than throughput

Track Outbound Defect Counts – Track and present the defect close count. keeping a negative inbound-outbound differential is a sign of a shop that is arcing in the right direction. Several weeks or months of positive differential is a warning sign that a quality mindset is slipping into dangerous territory.

Track Backlog Defect Counts – Track and present the count of defects in the backlog. A backlog of zero can be incredibly hard to maintain in a large enterprise with a large code base. In my experience 20 and 50 have been ominous numbers. Any more than 20 defects in an application and the backlog starts to become a fearsome beast that can only be tamed by stopping the factory. Any more than 50 and the incremental additions of any new defect seem trivial.

Track The Age Defects – Tracking the average age of defects in the backlog is critical for any presentation of MTTR. A spike in MTTR can actually be a good thing if it represents a wave of old defects being cleared from the back of the backlog.

Resist The Urge to Simplify

A five part metric begs the question of an balanced score card approach with a grading formula and despite my belief in the value of this tactic for availability and performance dashboards, I advise you to strongly resist the temptation to apply it here (otherwise, we are right back where we started with Goodheart’s Law. Damn you crafty humans!).

If you decide to bring the magical metric of MTTR into your shop, this type of reporting may be one area where starting small is quite possibly worse than doing nothing. Without the nuanced view, true visibility into the health of the shop isn’t possible. Worse than that, a flat view of just MTTR has the potential to mask real problems and raise false ones.

When you pull all of the metrics together in a single view, like in the above graphic, the real message becomes clear. The health of the factory floor isn’t tied up in a neat little bow by a single number. It isn’t even made perfectly clear by three, four or five numbers. The real health becomes clear in the trend. Is your shop doing better week over week? Month over month? Getting better every day is the name of the game, and MTTR (with a few other metrics) can be a lever that moves your IT world.

April 28, 2014