devopsy.kinsta.cloud

Category: Blogs

Why Vertical Collaboration Matters for DevOps

Vertical collaboration is defined by Rick Defour as a team of people working interdependently toward a common goal for which they are all mutually accountable.

Now that everyone is clear on the importance of collaboration and the different types, it’s time to delve into the importance of one type of collaboration in particular: vertical. Vertical collaboration is defined by Rick Defour as a team of people working interdependently toward a common goal for which they are all mutually accountable. You can find examples of vertical collaboration in educational settings where teachers of different levels work together to insure student success and in manufacturing settings where the supplier & retailer can better leverage themselves to enhance the bottom line for both.

When you think of vertical collaboration in the tech space, the example of a dev team pushing out code is a good one. Each dev has a different area of specialty but they must work together in order to implement the new feature. Taking this into consideration then, it’s easy to see why a vertical collaboration tool has to be easy to use and not create any new boundaries. The more vertical a collaborative tool is built, the more useful it is to real people.

The problem, of course, is that many companies view all collaboration tools as vertical. They only want to spend money on one tool so they pick the horizontal one (thinking that it’s the one more people will use) and it turns out to not actually be helpful to anyone. So that’s when companies start adding to their toolbox and then end up with a hodgepodge of tools. Instead of making their collaboration process more effective, they have made it more convoluted.

I created VictorOps because there was no existing vertical collaboration platform for DevOps. Going into it, I knew I was solving a problem I’d had at a number of previous companies and therefore already knew the exact requirements needed. The collaborative platform would have to be mobile first, would have to allow for synchronous engagement while enabling asynchronous problem solving and would have to give all participants symmetric access to information from their enterprise monitoring platform.

What I realized after creating VictorOps and using it to solve our own internal issues is that vertical collaboration gives team members the opportunity for scalable “leaning in” while also providing the freedom to lean out to reflect. What’s not to love about that?

April 17, 2014
Delivering Delight At ChefConf 2014

Chef CEO Barry Crist stirs the pot to open ChefConf 2014

The theme from ChefConf 2014 is stirring delight and if you listened to Barry Crist during todays opening keynote, then you would know its more appropriately “F****ing Delight”. Amidst high energy, loud music and a smattering of exciting expletives, Barry Crist called for attendees to stir delight in their enterprises using DevOps methodologies and, of course, Chef.

Crist defined delight when the customer’s experience exceeds expectations. Using Uber as an example when traveling in New York, he joked that the influx of Uber cars was so high that it took only 30 seconds for him to wait for a ride. And what more he didn’t have to deal with payment when exiting the car at the airport. In short, the service not only provided the expected functionality of something as simple as a car ride, but provided a delightful experience. The new economy, we are part of he said, is the “delight economy”. And the call to action for ChefConf attendees is to bring others along with them into this new and leading economy.

Using Volvo as a case study, Crist also touched on the concept that many companies, not just web development organizations, can utilize DevOps practices. When his wife’s Volvo navigation system needed a map update, he received a set of DVDs. The DVDs needed to be manually loaded into the on-board computer while leaving the car running for 5 hours. Crist countered that Volvo is more like a software company than a car manufacturer, but they just didn’t know it yet. For example, this would have been an excellent case for continuous delivery in that the Volvo probably didn’t need to ship him a handful of DVDs, but just the small changes and probably could have been delivered in a much more economical means.

Crist concluded that using DevOps methods and delivering delight isn’t limited to any industry and certainly isn’t just for web dev shops. And so ChefConf 2014 begins.

April 17, 2014
An outage war room primer

One aspect of the DevOps movement I’ve seen adopted at numerous companies is the idea that everyone supports their products by being on-call for any incidents that occur in the production environment. This responsibility often leads to participation in the outage war room. For those of you who may be new to this experience, I’m offering this primer from a grey-hair ops guy who’s served countless hours in every manner of war room you could imagine, some that stretch the bounds of that imagination, and a few that I just still can’t talk about. Been there. Done that. Still doing it.

In all that time, however, I’ve observed four distinct war room dynamics that can cause conflict. Understanding how that conflict arises is key to mitigating it.

The war room is a stressful place

This statement sounds obvious, but it’s fundamental to understanding the tensions that can arise in a war room. The clock is ticking. In fact, the clock has already been ticking for some time when you join the room. These things don’t happen instantaneously. When you get the call to join a war room, the problem has already manifested and is causing an impact. Even with automation, it takes time to notify people and have them connect to the war room, in whatever form that takes. In environments with a very high SLA target, the set up time for the war room alone can exceed the monthly allowance for downtime. So once you get into the war room, it’s serious business. The clock keeps ticking.

The war room is often a stressful place as a result. Tensions, and even tempers, can run high. But each and every person in the war room has an obligation to keep themselves under control. Tension and stress feed on each other, and if you succumb, it makes it worse for everyone. Even more important, if you get angry, emotional, or display some other decidedly not cool way of coping with the situation, you are likely to lose the confidence of your peers. That really sucks, so it’s important to force yourself to project calm confidence and be deliberate and measured in your actions. If you have the opportunity (and the ability) and can help to disperse some of that tension, do it. Those are the kinds of moments that build credibility.

And there is another very personal connection to the outage that can be stressful. A professional, worst-case scenario. In the worst case, you are actually responsible for the whole mess. When that happens, you are going to be in your own special hell in the war room. But you should know exactly what is expected of you – the one thing you can do that will really make a difference. You need to tell everyone exactly what you did, without skipping anything, and without regard to how stupid it might sound in hindsight. As in, “I continued to run through all the commands in the procedure even though they were failing with errors and even though I had guidance to stop and seek help if they failed.” To help prop you up in this most difficult of tasks, consider this: everyone else in the room has done the same thing. We’ve all been there. If we haven’t, it’s probably just a matter of time. So you have to swallow your pride, tell the whole story and then get actively involved in fixing things, because that’s what the war room is all about.

The goal of the war room is to restore service as quickly as possible

A war room is bad. If you are in one, it probably means that your customers are impacted. You have an obligation, working with the rest of the team, to recover the service in the most effective and least risky way. In a perfect world, that method is also fast, but life doesn’t always work out that way. If there is no obvious and simple solution, though, things can slow down a bit and we see different approaches forming across groups of people in the war room. Some, traditionally the operations staff, want a way to restore service quickly by a reboot, reset or other “big red button” kind of operation. Others feel it’s important to fully understand the nature of the problem before taking action. These are clearly often opposing goals.

One way this manifests in war rooms is when, in order to further analyze the problem, someone asks to run some experiments, or capture data, or otherwise use the production environment to debug the problem. If there is no method available to recover service with a reasonable level of risk, such experiments may be justified. Otherwise, the team has an obligation to execute whatever procedure they have to bring the system back up, even if that destroys information that might be helpful to the root cause investigation.

Another thing I see happen frequently is when the someone jumps the gun in trying to fix things. It plays out like this:

Jane: Do you think we should reboot the server?

Dick: I’ll get set up for it.

Jane: I think we should do it.

Dick: Done!

Jane: Wait…

A good war room manager will easily spot and work through these kinds of conflicts. If there is such a person on your war room, listen to them and obey them. The war rooms I’ve participated in are not democracies. They are run by a strong leader who is empowered to make extremely critical decisions. It’s crucial to respect that authority in the heat of the moment rather than try to be the hero. Not so coincidentally, this is my next point.

We need solutions, not heroes

In a war room, you need to separate your ego from your idea. The war room does not need heroes. The war room needs solutions. Be advised that no matter what part of the organization you come from, your idea is probably going to go up against some alternative ideas. They will not be judged, generally speaking, on how clever or elegant they are. They will be judged on how effectively they can recover the service and how much risk there is. And if yours is not the chosen idea, it does not mean that you suck. The only people who really suck in a war room are those who interfere with the process of service recovery. You know, the people who interrupt the flow of a critical conversation to ask for a status update, or who won’t stop debating a relatively minor point when the team really needs to focus on a larger issue. That kind of thing. But I digress.

Sometimes, the right path forward does not actually address the root cause of the problem.

Sometimes, the right path forward precludes ever figuring out what the root cause is.

Heroes have a hard time coping with that.

You must accept that you may never know why

The incident and problem management processes in most reasonably mature organizations are designed to recover the service, find the root cause and then ensure that it never happens again. But reality is often different than these goals. Incidents sometimes last a lot longer than you like, as when you discover that something has been broken for a long time and you did not know about it. And we should all know by this point that sometimes you never really know why it happened. You just can’t figure it out. It sucks, but again, it doesn’t mean that you suck. It means that the cost of determining the root cause has crossed the point at which it is feasible to continue, given the impact of the problem when it manifests. That’s just common business acumen, but it’s worth stating. Because when you realize that this is a possible outcome, it can give you a better perspective on evaluating your options. Suddenly, restarting the system and walking away from any run-time state information that might be helpful in root cause determination seems more viable. And those are the kinds of trade-offs that everyone in the war room has to come to terms with.

Conclusion

It sucks being called into a war room due to an outage. It is the very definition of failure in the SaaS world. But you can make a bad situation into a great and positive experience if you handle it properly. You can take an event that should mean that you suck and turn it into something that demonstrates your excellence. Being an effective member of the war room is a way for you, as an individual contributor, to demonstrate that awesomeness. You do it through calm, reasoned action and sound leadership. That builds confidence and trust. I guarantee you that your peers in the war room will notice and respect you for it and your customers will benefit from it. Let’s not let any of them down.

April 15, 2014
Nagios is not a monitoring strategy

When I visit clients to talk about DevOps, I usually ask them what their monitoring strategy is. Too often, the answer I hear is “We use Nagios”. I think Nagios is a great tool, but it sure is not a strategy. Nagios does a good job of monitoring infrastructure. It will alert you when you are running out of disk, CPU, or memory. I call this reactive monitoring. In other words, Nagios is telling you that your resources are getting maxed out and you are about to have issues. Proactive monitoring focuses more on the behavior of the applications and attempts to detect when metrics are starting to stray away from their normal baseline numbers. Proactive monitoring alerts you that the system is starting to experience symptoms that can lead to a degradation of performance or capacity issues which is more preferable than Nagios telling you are about to be screwed. With reactive monitoring, it is not uncommon that customers start complaining about the same time that the Nagios alerts start going off. The goal of proactive monitoring is to head off issues so that customers don’t even notice.

The next question I ask is “What things are you monitoring?” A typical answer usually revolves around various infrastructure assets and databases. That’s a good start but there is much more to consider. But first, let’s talk about why proactive monitoring is so critical. In the pre-cloud days we used to ship software to our customers where they would install the software, perform capacity planning tasks, manage the infrastructure, and operate the day-to-day activities. Once we shipped the code we were done. In today’s world, we are no longer shipping product. Instead we are delivering services that are always on. The customer no longer owns and operates the infrastructure and the software. Instead they pay for a service and expect that service to run reliably all the time. To meet those expectations, we need a more robust monitoring strategy. We need to monitor more than just the infrastructure.

A good monitoring strategy starts by identifying all of the actors who needs access to data and all of the categories of data that needs to be tracked. Some metrics are monitored in real-time while others are mined from log data. Every good monitoring strategy is accompanied with a sound logging solution. In order to perform analytics to predict trends within the data, one must collect various data points ranging from customer usage activity, security controls, deployment activities, and much more. The following presentation goes into much more detail about the different areas that should be monitored and why different actors need these data points to perform their jobs.

[slideshare id=33320431&doc=monitoringstrategies-0408014v1-140409081614-phpapp01]

The bottom line is, before building in the cloud, it pays to invest some time into a sound monitoring strategy. I have seen too often where teams don’t think through how to support these highly distributed, always on SaaS solutions and end up delivering software that does not meet the reliability and quality expectations of customers. Monitoring provides feedback to developers, product owners, operators, and even customers so that systems can continuously be improved. Nagios is great, but there is no single monitoring solution that can implemented to effectively operate today’s always on services.

April 15, 2014
From here to there: practical first DevOps steps
I’m a bad programmer.

There, I got that out of the way.

The truth is, I’m a sysadmin that writes code infrequently, on an island, and out of disciplined requirement, release, or QA practices. I need to run an ETL that kicks a file transfer. I’ll write some code. I’ve got a workflow that needs a crappy but functional web UI. Now I’m a developer. But in between these kinds of projects are days, and sometimes weeks of meetings and planning and dozens of things that are not code, which means by the time I come back to actual programming, I’m starting fresh every time.

I don’t think I’m alone. I think there are a lot of sysadmins in the world with functional perl chops, but because they don’t code daily, they don’t approach problems with an inherently code-focused approach. Systems are still running, the internet is still on, so this way of solving problems must be OK. It is, but there are drawbacks.

Rolling changes out to 4 dozen of your 8 dozen systems and verifying each received a 4 line change to that logship script with scp? Good luck with that. The old SA just left, and now you need to find all his stuff? GLHF. Some report hasn’t been sending for 4 days, what changed?! Yeah… yeah.

Where software development got it right

Great systems need three things: consistent configuration, maintainability, auditability. Good dev & deploy practices get you all three. Version control systems like git provide a great mechanism to manage code, version, and collaborate in reliable and predictable ways. Issue trackers, like Jira or Rally that integrate with version control let you easily tie specific changes back to the original reason for the change. Deployment systems like Jenkins and Capistrano give you one stop interfaces to track what got deployed where and when and by whom.

Take it as given that few SA teams are using all of these approaches for their work today. Probably you’ve got a ticket system, and you definitely get yelled at if you don’t comment your scripts. Maybe those comments link back to the work order number? Migrating what you do today to a more DevOps focused approach isn’t going to happen overnight. So how do you start?

Version control or GTFO – Yeah. I know. It’s on the list! No. Stop what you are doing, and gather all of your nginx.conf’s, your limits.conf, pg_hba.conf, and commit them to a git repo. I can’t emphasize this enough – if you are not enforcing version control in your systems administration team you’ll reap the whirlwind.

Change, Commit, Test. Welcome to the new DevOps discipline. You’re going to hate it at first, but 7 months from now when the webservers are flopping because somebody changed something, you’ll be grateful for a quick and reliable rollback.

Puppet or Chef or Ansible – get a basic setup. One master, three nodes. My starting point here was user management, I setup a basic user-add class, and let it manage ssh keys. Let that soak for a couple weeks and slowly roll it out to the rest of your infrastructure. Ten bucks says you find a dozen hosts with inconsistent UID’s that you’ve been meaning to fix for years.

Once that feels comfortable, move some basic scripts into your setup, roll those out slowly. Take the db configurations, commit them to the repo, and move ’em over to configuration mangaement too. The point is, moving to configuration management systems is not an atomic event across your entire infrastructure, iterate against it. Make small weekly changes that roll up to quarterly progress.

Here’s where the rubber meets the road:

“Git is going to suck because deploying is so much easier when you just vim a file on a server” – Every Sysadmin Ever

If you’ve got a repo, and you’ve got say, puppet… it’s a pretty easy step to have commits to that repo pulled and auto-deployed to your puppet managed nodes. Which means rollbacks can be too.

Pay off your tech debt – I don’t have to tell you about the 4 dozen scripts you’ve got out there, unmanaged, brittle, and needing constant tweaks to keep them lined up with system changes. These are the bread and butter of system administrators, and can provide a great opportunity to implement change. Next time you have to touch that script for a new SCP server, stop. Stop! Do this instead:
- check it in to your git repo, with some meaningful comments about what it does
- add it to your configuration management system (puppet/chef/ansible)
- if it’s bash or perl, rewrite it as python. It’s going to take a bit more time to rewrite all your scripts, but spending this time gets you more comfortable writing code regularly, and gets you more familiar with a modern scripting language.
Walk the path

Whatever your system or your goals, the hardest part of adopting DevOps approaches is going to come down to time. Reading through this myself, I’m struck by what an insensitive jerk I am. “Hey guys I know you totally have tons of free time, just starting doing everything you regularly do but add a ton of extra work too. Kthxbai”. The reality is, I probably am a jerk, but you don’t have to think about moving to DevOps as a shining city on a hill that you can reach with enough late nights. It’s a journey, a path, a way, not a place to arrive.

Find a way to start making small changes to your existing processes that line up with the best of the development world. Decide to add 10% more time to a proposed project to allow for it to be handled with these methods. If you begin to turn your steps onto the DevOps path, you’ll quickly realize you’ve already arrived at your destination.
April 14, 2014
DevOps Frequent Friday Follow on Twitter

Continuing our Friday practice of highlighting people to follow in the DevOps space, here are this weeks entries:

@jpmorgenthal @lmacvittie @AndiMann @mongodbinc @kevinbehr @devopscloud @amyl_bishop @kristiancarter

Already on the list:

@patrickdebois , @realgenekim  @botchagalupe  @jezhumble  @garethr

@kartar  @krisbuytaert  @jordansissel  @agentdero  @allspaw  @monkchips

@devops_borat @lnxchk  @damonedwards   @lusis  @obfuscurity @shanley

@adrianco  @scriptrock @jumpcloud  @sd_architect  @govictorops @softlayer

@docker @chef  @hullsean @champagnie @planetdevops @krishnan

@tarunbhatti @wickett @nathenharvey @andimann @byron_miller

You can find all of these people on a Twitter list I have started at: https://twitter.com/devopsdotcom/lists/devops-twitters-to-folow

If you think you should be on the list, drop me a line here. Have a good weekend and change those passwords if you haven’t already thanks to Heartbleed

April 11, 2014
Trust and Computers Make the News

The Heartbleed bug in OpenSSL was major news this week. While you are waiting for sudo apt-get dist-upgrade to run on all your servers, let’s take a minute to reflect on how the SSL trust system works and about the kinds of system relationships that depend on it.

Just to recap, SSL-based trust derives from a “blessed” certificate authority, and this certificate authority can create other certificates that can be “trusted” as long as their lineage can be traced back to the original authority. In this system the answer to “Should I trust you?” is binary; the answer is either “yes” or “no” based on the certificate chain.

In the event the certificate authority is compromised, then all of the derivative trusted systems become suspect as well. Real certificates become indistinguishable from fake certificates and it is no longer possible to know “who or what to trust”. Absolute all-or-nothing trust is an inherent weakness in certificate based trust and the X.509 based trust system.

This situation got me thinking about the differences between human trust systems and computing trust systems. It’s interesting that people take very different approaches to trusting computers as we do with each other. We rarely can answer the question “Do I trust you?” about another person with a simple “yes” or a “no” answer to cover all scenarios. Most of the time the answer is “it depends” on what we are trusting another person to do, which is better captured by the concept of trustworthiness. In human networks you establish trustworthiness based on knowledge that you have about the person, their honesty and their reputation, and information from other people who have had experiences and interactions which are similar to one you are trying to assess.

SSL doesn’t let us assess the trustworthiness of computers. It presents a “definitive” trust answer, in which we place too much confidence. Based on the SSL attestations of websites, we readily hand over our credit cards to strange computers in a way that we wouldn’t do with people.

There are alternatives to SSL for establishing trust in computing. For example, Pretty Good Privacy (PGP) is a trust system that operates a bit closer to how humans think about trust in one another, although it requires additional overhead to manage trust systems that are established on a case by case basis. Moreover, PGP doesn’t necessarily incorporate historical experience akin to the “reliability” parameter of trustworthiness.

This week’s news highlights the need for accessible and manageable trust systems that provide estimates of trustworthiness somewhere in between the type of trust provided by SSL and case by case managed trust systems like PGP.

April 11, 2014
Not all collaboration types are equal

Talking about collaboration in general terms is easy. I’ve done it and so has everyone else. It’s time now to dig deeper into the specifics of collaboration, meaning the different types – asynchronous vs. synchronous, vertical vs. horizontal – and how different companies fit into the magic collaboration quadrant. And yes, the quadrant is indeed magical.

Let’s start with the two types of collaboration that are time-relevant, asynchronous and synchronous. Asynchronous collaboration takes place outside of real-time. The best example of a ubiquitous asynchronous collaboration tool is email. You send an email and the receiver has time to digest the information. The drawback to asynchronous communication is that it’s inadequate as a means of communicating a sense of urgency. You have to wait for someone to respond to an email and that lack of immediacy can cause damage in troubleshooting or problem solving situations. Email replies tend to include the original message for context because you can’t rely on the timing of the message to keep the context.

On the other side of that collaboration coin, you have synchronous collaboration. Synchronous communication takes place like a conversation and more closely resembles a face-to-face interaction. A chat session is the ideal illustration of this concept. The obvious benefit to synchronous collaboration is the ability to send and receive information right away. This adds to the feeling of immediacy and tends to provide a more effective means of working together to solve issues in real-time.

Now that we have that covered, it’s time to move to the other defining aspect of collaboration: whether it’s vertical or horizontal. Vertical collaboration consists of problem solving tools that are domain-specific. An example of a vertical collaboration tool is one built for certain parts of the business like Salesforce for the Sales & Marketing teams or Rally for the development team. Horizontal collaboration consists of non-domain specific tools, meaning that they have potential use across all teams of an organization. Project management tools like Campfire or remote meeting solutions like Join.me or WebEx are examples of horizontal collaboration tools which tend to be widely adopted but lack the ability to really go very deep in helping in a domain-specific way.

Check out the quadrant I created to better show where different companies currently fit into the ever-growing collaboration landscape, based on the definitions provided above.

When looking at the state of collaboration tools, it’s interesting to note where most companies land within the quadrant. Why so few vertical synchronous ones? I’ll answer that next time.

April 11, 2014
Programmability in the Network: Stop a Bleeding Heart…

It is not often the case that a security vulnerability can get the entire Internet talking. And not just the security community on the Internet, but everyone. End-users and IT alike are looking for answers and trying to mitigate Heartbleed. It has its own web site and logo. It’s that big of a deal.

Many service providers have already patched their systems, but there are a whole lot of sites on the Internet and it’s estimated that a significant number of them are vulnerable.

Netcraft notes that Heartbleed, based on OpenSSL, “affects around 17% of SSL web servers which use certificates issued by trusted certificate authorities.” One of the most trusted sources of data regarding web server software in use today, Netcraft’s “most recent SSL Survey found that the heartbeat extension was enabled on 17.5% of SSL sites, accounting for around half a million certificates issued by trusted certificate authorities.”

But it’s not just server software that may be vulnerable. Any device in the data path that terminates SSL and is based on OpenSSL could be vulnerable. Any “thing” that might use OpenSSL to enable secure communication and accepts connections could be vulnerable. In other words, hearts are bleeding all over the Internet right now, and it could take quite a while for the Internet in general to plug the holes.

The hole itself is fairly simple – it takes advantage of the mishandling of what seems like an innocuous meta-protocol that runs atop TLS and DTLS that consists of two message types: HeartbeatRequest and HeartbeatResponse. It’s used to “ping” the other party in a secure connection to make sure the peer is alive. It can be used to offset idle times that might otherwise cause the connection to time out.

It can also, thanks to a missing bounds check on the message structure, arbitrarily and completely silently grab 64 kilobytes of the server’s memory. That memory might include sensitive things like the private key used to encrypt communications or passwords or pieces of critical documents.

Whatever might be in memory when the vulnerable software goes to grab it is leaving the building faster than Elvis – and without a trace. There’s no logs, no evidence of the theft, nothing. It’s clean and remotely executable by anyone on the Internet. That may explain the attention it’s currently getting.

The answer to any vulnerability is ultimately to patch it. Depending on how many servers or systems are impacted, that could take some organizations days – assuming patches are available immediately. In the meantime, the heart is still bleeding and the only way to prevent more information from leaking is to shut off the hose.

Except it’s not the only way.

Programmability in the data path can mitigate this, now. Right now. Like in the next five minutes.

Let’s review programmability in the network for a brief moment: you have, in the data path, a device. That device (or software) is capable of programmatically intercepting, inspecting, and transforming the data that’s traversing it. That means such a system can intercept a request to initiate SSL/TLS traffic, inspect it and, if it finds a request to include Heartbeat support it can reject the connection.

Interestingly, one of the primary motivators of early SDN architectures was the ability to support new or experimental protocols by inserting them into the data path in the network. While that’s a driver for academics, a more realistic driver for businesses is the ability to interact with protocols in flight and “fix” them – especially when the “fix” involves preventing the exploitation of a vulnerability.

Programmability in the network enables this kind of rapid response to threats. Being able to deploy a solution very quickly to prevent further data leakage provides protection for both the organization and those who interact with its properties (employees, partners and customers). It also offers time for IT to patch and update all impacted systems without remaining vulnerable or taking the drastic step of going completely dark until the situation is resolved.

By enabling a view of “infrastructure as code”, programmability not only enables a more software-oriented approach to deploying services, but also a software-oriented approach to quickly creating and deploying an appropriate response to emerging threats.

A devops approach can facilitate this capability, and push protection into the network as code.

April 10, 2014
Services As A Model For DevOps Communications
DevOps is an interesting movement. As with any effort of this magnitude, there will be many opinions on what is entailed in adoption and implementation. The issue is that businesses are so diverse and varied in culture that no single method for adoption is guaranteed to fit all businesses. In large IT organizations, this issue is additionally hampered by the fact that multiple cultures and dominions exist.

Thus, I am not a believer that DevOps adoption comes from the approach that the business commits to adopting DevOps. The business has committed to many initiatives over time that have not panned out. This is not necessarily failure, but perhaps just shifting priorities, shifting political landscape, lack of training or simply a lack of desire to change. “The best laid plans of mice and men ….” and all.

If commitment is not hampered by people, then there’s a good chance that DevOps adoption may be hampered by silos. This does not make silos our enemy. Silos help to compartmentalize and provide focus. Still, silos limit the ability to orchestrate process without involvement and participation from each of the silos.

Regardless of the reason, the challenge stands, how to align a process without the use of force. The answer is protocol. With protocol, an individual or group does not need to buy into the entire process, but just needs to understand their role. Short of incompetence or insubordination, or even both, which will eventually result in the removal of these particular types of hurdles, following protocol ensures that every role in an activity will complete their action and, thereby, facilitate the whole with roughly an 80% success rate.

I posit that the service is the means for defining protocol and of communicating that protocol into an IT organization, such that DevOps is the natural activity and outcome. A proper service entails defining a number attributes, for example:
- Metered
- Valued (priced)
- Advertised / Catalogued
- Governed
- Monitored
- Supported
- Documented
- Quality
- Available
- Intuitive
It is up to each area of IT participating in the DevOps process—Design, Develop, Test, Stage, Deploy, Operate—to contribute to defining and providing their completed artifacts that pertain to formulation of the service. The service will guide the development in the understanding of operational characteristics and will allow operations to ensure they can deliver and support the service to the level of expectations of the consumer. Moreover, in those cases where the service is not meeting appropriate levels, it will allow them to provide development with the appropriate data and metrics to take corrective action, thus leading to fewer and shorter outages and failures.
April 10, 2014