devopsy.kinsta.cloud

Tag: devops adoption

DevOps Leadership Series: Accelerating Adoption

We are at an inflection point in the adoption of DevOps. Now is not the time to blink.

DevOps was once thought to be a practice attainable by a select few. The early pioneers were the ones with arrows in their backs. For those who succeeded a new level of awesomeness was achieved and admiration was deserved. But we have reached a new level of DevOps maturity today – one with broader participation across functional disciplines, where more rigorous processes being applied, and significantly more organizations are seeing a return on their early investments.

This week, I caught up with three amazing guys with a wealth of DevOps experience to add to the DevOps Leadership Series. We now have 18 free videos in the series.

Dave Farley – DevOps for Everyone

First, I sat down with Dave Farley (@DaveFarley77), co-author of Continuous Delivery. Dave first talks about experimenting with and optimizing the development process from beginning to end, and then shares that DevOps is truly an approach available to everyone in the organization.

https://youtu.be/1VLb3XQcRbA

Brian Dawson – Cross-Functional DevOps

I then sat down for a chat with Brian Dawson (@BrianVDawson), DevOps Evangelist at CloudBees, who is observing that cross-functional participation in DevOps is on the rise. It is no longer a practice relegated to a select few participants in a development or operations team.

https://youtu.be/B2JhCMlYBDg

Curtis Yanko – The ROI of Software Factories

Finally, I caught up with Curtis Yanko (@onCommit), the former Architecture Manager for Cigna’s DevOps Center of Excellence (and now colleague of mine at Sonatype). Curtis has been seeing the successful adoption of “software factories” reduce day-to-day burdens on DevOps teams which allows them to further accelerate learning and process improvements.

https://youtu.be/Qv2wleI3TRAhttps://youtu.be/Qv2wleI3TRA

On October 1st, we are going to extend these three conversations even further. I have invited Dave, Brian, and Curtis to join me for a discussion on how to get started with DevOps or how you can reach the next level of maturity in your existing DevOps practice. Be sure to sign up for this event – you won’t want to miss it.

Dave, Brian, and Curtis will share 45 minutes with us discussing DevOps processes, adoption practices, technologies, and cross-functional alignment that are critical to DevOps success. We’ll also share and discuss DevOps reference architectures that can jump start your own practice or further validate the path you are already on. You can RSVP here.

September 17, 2015
Is DevOps on the cusp of Disillusionment?

If you are familiar with, or buy into the idea of the Hype Cycle, you are familiar with the Trough of Disillusionment. When it comes to adoption of pretty much anything, there seems to be a point, just after the peak of enthusiasm, where those in the know are disappointed with the practicality and adoption rate of their thing or cause. Now that DevOps is several years old, and, in the last year enthusiasm has been at an all time high, is DevOps on the cusp of Disillusionment?

Hype Cycle – Wikipedia

(more…)

May 11, 2015
Mixcloud Makeover: Faster Development, More Confidence

“Mixcloud provides audio hosting and distribution for millions of podcasters and DJs,” says Matthew Clayton, CTO & Co-Founder, Mixcloud. Mixcloud’s developers program the site in python (Django) and javascript, and develop their native mobile apps in Java and Objective-C. The Mixcloud team grew its mammoth Django monolith application over a five year period. Mixcloud’s primary server infrastructure supports this application, which makes up most of its code base.

The Legacy Development Workflow & Goals For The Future

The Mixcloud team used to code its application on a branch, deploy the branch, check for stability, merge it into the master branch and redeploy it. “We used to branch the code on git, then push this to github so that all the team could see the branches developing in parallel,” says Clayton. To check code stability, the Mixcloud team used to simply watch to see whether the site crashed on a single server. “We originally didn’t have any tests, so this was really the only way we could see whether there was a problem,” says Clayton.

The old overall development methodology also produced challenges every time Mixcloud hired new developers. “Adding developers who didn’t understand the entire stack massively increased the likelihood of bugs and the risk of downtime,” says Clayton.

When Mixcloud went looking for a DevOps / continuous delivery approach, it had in mind to increase the quality of the code its developers produced and to increase the team’s confidence in the code they shipped to production while overall having faster development.

New Workflow, Bumps In The Road

Today, Mixcloud hosts and reviews its code using github.com. The company uses Slack for internal communications. The team uses getsentry.com for exception handling and Asana for product feature and bug tracking. “We use Codeship for running Linting (python/JS/Templates), JS unit tests, Python tests, Selenium integration tests, and database integration tests,” says Clayton.

Python/JS/Templates enable code linting by leveraging tools such as pylint, jslint, and Django-extensions to perform static analysis and to lint the appropriate code.

But the path to DevOps enlightenment and improvements is never ending. It took some time for Clayton and his team to understand how all the working components affect each other as they integrate together. “We still have not included database migrations in our release process. This is still a manual process,” says Clayton. Mixcloud will eventually add this too to its list of automated endeavors.

Navigating The DevOps / Continuous Delivery Maze

At the very start of a round trip through the new development process, Mixcloud developers will open up a pull request on github.com and assign it to another team member. This action automatically sets a complete build in motion and deploys a staging infrastructure. “The assigned developer will now review the pull request using github, and also the running code. Once this review process is complete, the branch merges with the master,” says Clayton.

This will next set off a deployment in production. When the team is ready, they will run a single command to activate the new code for each and every user. “The whole test and deploy process now takes about three minutes. This enables a rapid iteration,” says Clayton.

Benefits & Surprises

Two of the biggest benefits for Mixcloud were adding code linting and pull request reviews. “These helped the team become more aware of changes across the entire code base, and eradicated entire classes of bugs for what is essentially a fixed, one-off engineering cost,” says Clayton.

When compared with the legacy development methodology that could introduce new bugs every time a new developer came on board, the Mixcloud team now only introduces bugs of a magnitude that could create website outages and downtime in the rarest of cases.

But the major unpredicted upside was how the automation and the reduction in manual testing added to the productivity and development speed. “Surprisingly, there have been massive improvements to interaction speed among team members. We are seeing productivity and developer happiness that are significantly better than before the change,” says Clayton. The ability for the development process to refactor huge amounts of code with confidence has also made great strides forward.

April 14, 2015
Why do DevOps? Winning hearts and minds with a Lean boxscore

Change is difficult. It can be painful, disruptive, and costly. Making a case for change requires us to win over hearts and minds. In my experience, we need to win the hearts of the ‘do-ers’, the people whose daily work is going to be disrupted. And the minds of the executives, whose financial support and personal sponsorship we need to make change happen.

A Lean boxscore allows you to address both the hearts, and the minds. Boxscore is a simple tool which we use to compare the current state with the future state, and turn that comparison into a compelling case for change.

Step One: Current State Baseline
Let’s start with where we are today, by capturing performance metrics that tell an accurate story of today’s development lifecycle. This may mean looking beyond the project plan to the reality, by the way. By interviewing or workshopping with development and operations teams, we get a true picture of the way the development process currently performs.

We’re always interested in end-to-end leadtime, of course – from the moment the business provides a prioritized set of requirements, to the moment that functionality is in production and working as intended. We’re also interested in how much wait time there is within the process – how much time between steps, waiting for an approval, a developer, or a tester. We do this by creating a value stream map, capturing all those metrics and validated by the teams who work within this process every day.

We want to look beyond leadtime and wait time too. We capture the number of resources involved in each release, the number of handoffs, and the number of bugs caught at each stage of development and deployment. If we can get to it, we might capture the percentage of time a developer spends writing code for their current release, as opposed to firefighting issues from the last release, arguing over schedules, trying to secure an appropriate environment, or other ‘non value adding’ activities. We might also capture some usage statistics, or other measure of customer satisfaction, and any relevant performance indicators, like server response time.

Before moving on, it’s crucial to ask stakeholders: ‘is this a fair reflection of the way things currently work?’

Step Two: Future State Target
This next step requires some imagination, and some bold thinking. Imagine a world where you’ve adopted DevOps: you’ve removed the waste within the development lifecycle and instilled the ‘shift left’ principle in all your working practices, your ‘4 in a box’ teams are rapidly developing, testing and deploying code, Cloud-based environments give you immediate provisioning, and automation and smart tooling have taken the bellyache out of testing. What does that future look like? How quickly can you develop – if provisioning is immediate? How many bugs do you expect to put live – if testing is automated and your production environment matches your test environment? How many changes to scope – if you’re working with smaller batches and faster delivery?

We create a new value stream map, which details the way this brave new process will work in practice. This allows us to set sensible but challenging targets for our future state.

Before moving on, be sure to ask stakeholders: ‘are these targets achievable, if we make the changes we have outlined?

Step Three: So What?
Finally, we get to the money question. What’s it all worth? For each improvement there will be an impact. Many of these will be financial. Ask your stakeholder ‘if we can get this new functionality out to market 2 months faster, what does that mean for revenue?’ Or ‘if we reduce the number of defects in production by 90%, can we reduce our operating costs?

Putting a dollar value on the financial impact will win over those minds with a logical, fact-based argument. Now, investing in a DevOps pilot seems like something they can’t afford NOT to do.

And the case for change for hardworking developers, testers and production resources? Metrics like number of handoffs, bugs and rework, and batch sizes can easily be translated into ‘how much pain you have to live with each day at work’ to win those hearts over.

April 9, 2015
Adopt DevOps Practices Organically, But Don’t “DO DevOps” (or is DevOps bullshit?)

In a post on his blog , programmer Josh Johnson points out that “radical claims of great cultural shifts, especially in technology” are usually “bullshit”. “Over time technology changes, but true radicalism is rare. Most often, a reinvention or revisiting of past ideas forms the basis for such claims. This DevOps thing was no different,” says the post from Josh Johnson, Software Engineer at Adzerk.

In 15 years as a technology journalist, I’ve heard vendors use labels and marketing speak to sell solutions. You can’t judge the efficacy of a solution by its label any more than you can “tell a book by its cover.” But is there something to these devops practices that we should take seriously? Should we make the effort to separate the wheat from the chaff, the faster, better software builds from the buzzwords and BS? I think so. And when I read Josh’s post, I came to believe that he thinks so, too. Josh shared his thoughts with me. Here are the fruits of that exchange.

First, no one should be doing devops. It’s not an action, it’s not a title, it’s a blanket term for approaches that bridge the gap between traditional operations and development groups, says Johnson. Second, having devops teams is wrong. “Sectioning off a group of people and giving them special projects and expecting devops skills doesn’t fix the divide, it creates new divides,” says Johnson. This leaves other developers and ops people, not to mention management, out of that inner circle. “The value of devops is in the cross-functional aspects of bringing people together from different disciplines,” says Johnson. True devops should include everyone.

A healthy, organic transition to devops practices should encourage people who specialize in operations or development to apply their specialties to the fullest while learning what the other side does. “If I’m the security expert,” says Johnson, “I’m going to monitor CVE feeds and recommend policy and practice. On a cross-functional team, I will also perform code reviews, participate directly as a peer in planning, and help the QA specialist by turning them on to fuzz testing. They’ll teach me how to use Selenium. I’m still the security expert.” In this scenario, the security expert grows and benefits and so does everyone whose work he touches.

Hurdles

The company will make compromises. The enterprise is asking a lot of its people and will have to pay to get it. The turnover in human resources can slow change to a crawl. The organization will probably have to outsource at least some tools and talents. “You may have to use GitHub instead of hosting your own git server,” says Johnson.

The more the devops metamorphosis progresses, the more the organization can lose central control of critical components. All things being equal, the bigger the enterprise, the harder it will be to let go. Initial costs and security concerns when getting started, together with outsourcing, can drive project scope through the roof. And that’s not the worst of it.

“You might lose some great people. There are some people who really need to focus exclusively on their specialty. There are managers who don’t thrive outside of silos. There are people who will not be able to rise to the occasion and work well with their new teammates,” says Johnson.

Don’t Force It

To achieve a thriving instantiation of devops while maintaining your sanity, you have to guide everyone to simultaneously want these practices. Start by poking holes in silos and letting members of different teams peek through. “Take a team of bright, collaborative people, give them a goal (say, reduce downtime in deployments by half) and empower them to make it happen, and it will happen. DevOps will magically spring forth,” says Johnson.

This organic approach is far afield of simply adding a devops team in hopes that it will save money by reducing staff. “Four DevOps engineers are not directly equivalent to two systems engineers and two software engineers,” says Johnson. If that’s the case, it’s worse yet to believe that you can replace four specialists with three experimental devops positions and maintain quality output at a savings of one FTE. Instead of saving money, this move may increase costs due to compressed time to burn out, expensive errors, and an enveloping drop in moral. “That will blow up in your face,” says Johnson.

March 23, 2015
Why are we still asking, “Can DevOps work here?”

It’s easy to get tired of buzzwords, especially in the software development and testing world where they’re so prevalent. But while it’s easy to blame the thought leaders and evangelists for creating them, and publishers and powerful marketing teams for inserting them into every content channel imaginable, the buzzwords that truly stick around do so on their own merit.

Agile, big data, and cloud have been written about and philosophized for years—and I hate to break it to you, but they’re not going anywhere. While they may not be the panacea that some make them out to be, their impact on the way we develop, test, deploy, and release software has been documented in every market and industry around the globe. This gives the words their staying power. Not relentless press coverage that’s been covertly sponsored by marketing and PR dollars.

Members of the academy and esteemed buzzwords in the software hall of fame: join me in welcoming our newest member: DevOps.

You may say it’s too early, and that DevOps hasn’t been around long enough to earn such universal acceptance. Many quickly cite John Allspaw and Paul Hammond’s 2009 presentation, “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” as the real inspiration behind Patrick Debois’ coining of the term, and if you’ve never viewed the session, I can’t recommend it enough.

Skip ahead to today and we’ve got multiple conferences that focus on DevOps exclusively. Devops Days, DevOps Summit, DevOps Connect and DevOps Enterprise feature some the most respected speakers and writers alive today.

But they’re not the only ones speaking there. CTOs, CIOs, chief architects, IT heads, and QA directors from every bank, retailer, hospitality group, government agency, and software development shop are there, too, and with a captivatingly similar story behind their employer’s decision to “do” DevOps.

They were stuck. Their software was late. It didn’t address their customers’ real needs, and it didn’t address their own company’s needs. Their application stacks were growing increasingly complex, the bottlenecks in their SDLC were increasing, and their ability to innovate, accelerate, and stay ahead of their competition was slipping further away from them.

What I find especially interesting in re-watching many of these conference sessions is that a lot of them were also leveraging the other buzzwords mentioned above. They’d been practicing agile for years, and they’d started to look to the cloud to lighten their hardware dependence and provisioning headaches.

What they all needed, and thankfully found, was a disruptor. And no matter what definition you attribute to DevOps (and good grief, there are so many) there’s no denying its disruptive nature.

It disrupts the culture of your dev, QA, and ops teams. It requires constant communication, more deploys, earlier testing, shifting left, an embracing of “failing fast and failing often” and what’s often described as “relentless automation.” Some would even argue that the hardest part of DevOps is getting executive buy-in and sponsorship so that you can even begin your initiative. Though to be fair, your ground level troops should be just as on-board as your commanding officers in order to have the best chance for success.

Joachim Bauernberger at InfoQ recently looked into whether DevOps can work in Telecoms, and it turns out, like in any industry looking to resolve the constraints and bottlenecks listed above, it most certainly can—however, it requires support and commitment from “all links in the delivery chain.”

And this is the only reason you should be asking, “Can DevOps work here?” Do you have the right people, and are you the right person to increase the likelihood of this disruptive shift within your company? Stop wondering if (or worse, doubting) your industry will support DevOps, because each year and each conference that passes proves that it can.

And soon enough, if the people at your organization are exposed as the real constraints and bottlenecks, they’ll be removed. Don’t be one of them. Be disruptive.

About the Author/Noel Wurst

Noel Wurst, Managing Editor at Skytap, Noel is responsible for setting and executing Skytap’s content marketing strategy. That includes overseeing the creation and distribution of high-quality, targeted content, and managing the team and external resources associated with those efforts. By leveraging deep online marketing, project management, and editorial experience, Noel plays a crucial role in helping generate and nurture leads with compelling content. You can follow Noel and Skytap at @NoelWurst and LinkedIn and @Skytap, Skytap’s LinkedIn

February 12, 2015
The real cost of downtime
Stephen Elliot and the IDC team recently released a report (registration required) sponsored by AppDynamics that examines the true cost of downtime and infrastructure failure, as well as DevOps adoption. The headline numbers are pretty eye-opening:
- For the Fortune 1000, the average total cost of unplanned application downtime per year is $1.25 billion to $2.5 billion.
- The average hourly cost of an infrastructure failure is $100,000 per hour.
- The average cost of a critical application failure per hour is $500,000 to $1 million.
Let those numbers just sink in a moment. Between $1 to $2.5 billion lost to application downtime a year! That is a big number, but if you look at the follow on numbers you can see how you can get there. Infrastructure failure has an average hourly cost of $100k per hour, critical application failure has an average hourly cost of $500k to $1m per hour. It doesn’t take a lot of hours for those numbers to add up. You can read in the report the actual math IDC applied to reach these conclusions based on the responses to a survey they undertook.

There was also some good insight into DevOps and DevOps tools adoption. Remember this was aimed at the Fortune 1000 market segment, so large enterprises. About 43% of respondents were using DevOps currently, with another 40% planning on adopting DevOps soon. That 83% number is in line with another recent survey by CA that showed an 88% number.

But all is not as rosy as that picture paints. When looking at the biggest impediments to DevOps, the usual suspects rear their ugly heads:
- Cultural inhibitors – 56.7%
- Fragmented processes – 43.3%
- Lack of executive support – 26.7%
All three of these topics have been the fodder for stories here on devopsy.kinsta.cloud. There are specific strategies to overcome each of these, but the fact is they are still major obstacles to successful DevOps adoption despite overwhelming desire to do so.

There was another interesting metric in the study that bears noting as well. It is about companies who try to use their existing tool sets in a DevOps setting. There is something like an 80% failure rate for companies who try to adapt their existing tools for DevOps practices. That screams out something that many a man already knows, “you need the right tool to do the job right.” The moral of that story is if you are going DevOps you damn better be buying some DevOps specific tools to get the job done right.

Again in line with other surveys I have seen the biggest initiatives driving DevOps adoption are:
- Automation 60%
- Continuous delivery 50%
- Continuous integration 43.3%
- Automated testing 43.3%
- Application monitoring/management 43.3%
DevOps was being initiated by slightly more development teams that IT Ops teams by about a 40% to 33% margin respectively.

You can download the full report from the link at the top of the story to get the rest of the data. Ultimately like similar surveys there are some assumptions that you have to swallow in order to give credence to the findings. For me the biggest grains of salt to consider are:
1. This survey was based on a relatively small amount of respondents. On average like 50 per question and some of those not from organizations in the scope of the survey. With that small a group trends could be skewed by a handful of outlier responses.
2. While IDC is a very reputable company, the fact is this was a sponsored survey by a DevOps tool vendor. AppDynamics likewise is a reputable company (and in full disclosure a sponsor here at devopsy.kinsta.cloud), nevertheless it is something you should take into consideration.
3. Perhaps the biggest grain is: Do you buy their math in calculating downtime? According the report’s authors here is how they arrived at their numbers:
To examine application downtime impact per year across the Fortune 1000, we conducted the

following analysis:
- 1,000 companies multiplied by an average of 250 critical applications per company equals 250,000 total applications. 250,000 multiplied by an average of $500 to $1 million per hour of downtime cost equals $1.25 billion to 2.5 billion of unplanned application downtime costs per year.
That sounds like back of napkin stuff to me. But really at the end of the day whether the number is $1b, $2b or more, suffice to say it is a big number.

Surveys like these continue to contribute to a broader understanding of the DevOps market. It also provides cover to IT executives seeking support to kick off DevOps in their own organizations. Hopefully we are approaching a critical mass of data that will make the job of “selling” DevOps up the org chart an easier sell.

UPDATE:

Here’s a great upcoming webinar that will include thought leadership from Stephen Elliot, VP of analyst firm IDC on “DevOps and the Cost of Downtime” based on research of Fortune 1000 companies. Register for it now, here.
February 11, 2015
The Trouble With Enterprise DevOps

The literary world and beyond has been up in arms as of late regarding the use of the word ‘literally’. It seems that many are using the word literally when they mean figuratively. For example, if you say, “I literally split my pants laughing,” it would be plausible for one to look at your pants looking for a tear. Words are funny that way. When used to emphasize the action or outcome, some will take literary privilege. Enterprise is also one of those words that are used to convey emphasis often with disregard to the implied meaning.

When it comes to DevOps there has been a lot of debate about the modifier ‘enterprise’. Modifiers are supposed to add specificity to the noun it is modifying. Agreeably, enterprise is a horrible modifier for DevOps when used without additional modifiers representing scale. I have been, and continue to be, a big advocate of the concept of Enterprise DevOps. That said, I too have been remiss in my use of the word enterprise to further modify with the term ‘large-scale.’

I don’t remember exactly what triggered me to seek out if the word enterprise implied large, but when I did I was surprised to find that there is no implied meaning that enterprise means large and, in fact, can be modified with small-, medium- and large-scale to represent the appropriate size. As further evidence to the ambiguity of the use of the word enterprise, I worked for a company that divided their sales organization into enterprise and commercial accounts. Clearly, this organization believes that enterprise connotes large-scale since this group was responsible for selling to a specific class of customer that spent a significant amount of money with them. In this case, scale was identified by how much money the business spent with that vendor versus annual revenues versus number of employees. So, clearly, not only does the word enterprise not imply scale, but it also has different metrics that will determine the scale of an enterprise.

Merriam-Webster defines Enterprise as follows:

en·ter·prise

noun
\ˈen-tə(r)-ˌprīz\

: a project or activity that involves many people and that is often difficult

: a business organization

: the ability or desire to do dangerous or difficult things or to solve problems in new ways

Obviously, none of these meanings relate to size or scale, but two of the meanings do indicate difficulty. Let’s put that in the parking lot for later.

Why is this important? With regard to IT, I believe there is a tendency to throw around the word enterprise with no modifiers when we mean large-scale. For example, “Enterprise Software” is a term that is often used to describe very expensive and complex software applications that are designed to operate at the scale of the entire business versus the individual. Typically, the software described by enterprise software is found in mid- to large-scale enterprises where they can afford to acquire, configure and operate this type of software. Likewise, “Enterprise Storage” is a way to describe compute data storage environments for large organizations.

Likewise, “Enterprise DevOps” is a term that that is often used to represent adoption of DevOps practices in large, complex organizations with many different stakeholders and sub-organizations making it much more difficult to coordinate efforts around efforts like continuous delivery and continuous improvement.

Interestingly, and I believe accidentally, implementing enterprise software and adopting enterprise DevOps are difficult projects. So, based on our Webster’s definition, enterprise works as a modifier for these nouns. But there’s still the issue that for small-, mid- and large-scale enterprises the challenges and hurdles associated with DevOps adoption vary enough that the constraints and bottlenecks require different approaches to remove. This is especially true of large-scale global enterprises where development exists in four continents, operations data centers exist in multiple countries on four continents and the business operates in six continents with a comprised IT staff of 120,000 people.

I for one will be very careful in the future to define what I mean when I’m using the word enterprise in the future. If I’m talking about large-scale enterprises, which I usually am, I will make sure that early on in my writings and presentations I specify that when you see enterprise going forward I’m talking about large-scale enterprises.

January 23, 2015
What are the most critical DevOps Tools?

What are the most critical DevOps tools categories? According to the results of a new survey by CA Technologies the answers may be surprising. That is just one of the insights reported on. The survey results and report are chock full of other great insights about DevOps usage as well.

The survey shows that a staggering 88% of respondents either are already utilizing or expect to be utilizing DevOps within the next 5 years. The over 1400 respondents to the survey represented IT or line of business executives and is a significant increase over last years 66% response rate. This would certainly indicate that DevOps is crossing over in the mainstream and is no longer solely in the realm of early adopters and visionaries.

So according to the study what are the most critical DevOps tools?

APM as the most critical? It is a popular segment with players like New Relic, App Dynamics, Dynatrace (Compuware) and CA to name a few. But I thought that change/configuration management which would include Chef and Puppet would be number 1. Also performance testing and functional testing were 2nd and 3rd. Could this be a case of categories CA plays in being over emphasized, the “everything looks like a hammer to a nail syndrome”? I don’t know, but the numbers speak for themselves. I was happy to see enterprise security coming in 4th on the list (another area CA plays in). Bringing up the rear at only 11% was network performance management. I think that may have a lot to do with the fact that with the rise of the cloud, so many organizations are less invested about network as the cloud and Internet has replaced much of the internal LAN.

I spoke to Andi Mann, VP in the CTO’s office at CA and a frequent contributor to devopsy.kinsta.cloud about the survey. Andi said that it is re-enforcement that despite all of the talk about DevOps being about culture, we should not forget that it is about tools as well.

So what are the reasons given for this massive adoption statistic. What are the perceived benefits and reasons driving DevOps adoption? According to the survey results:

As can be seen over 40% cited improved quality and performance of applications as the driver for DevOps. For those who wonder about whether you can have increased speed only at the expense of quality, this would seem to show that you can have both with DevOps. Perhaps equally as rewarding was the 2nd most popular response, improving the end customer experience.

In any case with 88% of respondents on board with DevOps, one thing is for sure, DevOps is the worst kept secret in IT. However, let’s not paint an overly rosy picture. Not every DevOps effort succeeds. What are the biggest obstacles to DevOps adoption? Again the survey says:

So while enterprise security is a big benefit for adopting DevOps, security and compliance concerns are the biggest obstacles to DevOps adoption as well. I guess that is the Jekyll and Hyde nature of security? I don’t buy the ROI being difficult to justify. Most DevOps examples I have seen show a clear positive ROI. Identifying the right DevOps consultant firm? I don’t know about that one either. But then again I am bullish on DevOps, so I may be as guilty of the “hammer looks like a nail” as anyone. I am surprised though that under obstacles we didn’t see that our culture is not trusting or some other culture related reason.

You should go have a look at the survey yourself as there are other good insights there. In the meantime here is another indicator of just how deep DevOps is penetrating the IT market.

January 16, 2015
DevOps storytime: The tale of the secret passageway named MTTR

When DevOps practitioners and evangelists talk about adoption, they mostly fall into one of two camps. People who talk about driving devops adoption by helping people understand the core ethical reasons of improving the world for those who work in it (I’m usually in this camp with Gene Kim).

Others talk about driving DevOps adoption by helping people understand the monetarily valuable things that come out of DevOps like improved user engagement and conversion via better performance, more resilient operations via adaptive architectures, or an increase in IT throughput delivered by a use of automation technologies. There is, however, a third way to drive adoption; rather than introduce devops directly, introduce mean time to resolve (MTTR).

MTTR is an Industrial Age metric that measures the time from a problem arises to the time it solved. It just so happens that shops that make a nuanced version of MTTR a very visible metric and seek to drive it down, have no other choice but to adopt DevOps methods because DevOps methods and beliefs are more and more required to push MTTR closer and closer to zero.

A Story We Are All Too Familiar With

Let me tell you a story of Alice, Bill and Carl. Alice and Bill work for Carl inside an IT shop where nothing of DevOps is known or practiced. All is manual in the land of Alice and Bill and the tyranny of work in progress keeps productivity and throughput capped.

One day, when Carl asks Alice and Bill to get the cost of system defects and outages under control. Alice and Bill both start to research the methods for managing defects. Bill, the more linearly aligned of the two people chooses to focus on prevention of defects and starts to introduce checkpoints and controls for how changes get made with the aim of preventing defects from being introduced.

The Plot Thickens

Alice on the other hand thinks a little differently than most. Alice believes the answer for reducing costs related to defects is by reducing the impact of each defect introduced. In her mind, she sees the only way to reduce defect counts to zero is to either add more and more gates, barriers and checkpoints thus providing additional layers of certainty, or to stop all change completely. Neither of these options smells right to Alice and she decides the first step for her is to start measuring the costs of the outages.

In her efforts to measure the costs, Alice discovers and then introduces a metric that neither Bill and Carl have seen before called MTTR. Alice explains to Bill and Carl that she doesn’t believe that speed and quality are naturally opposed to each other, but rather that the game to be played is how to be both fast and good. After Carl hears from both Bill, who wants to limit the number of defects introduced, and Alice who wants to limit the impact of each defect, he understands the logic of Alice’s approach, but that it’s just plain crazy to not try to reduce the number of defects. Carl instructs the team to go after both objectives, and limit the number of defects and reduce the impact of each one by measuring and driving down MTTR. When leaving the room Carl like every other executive says, “Remember that you can’t add any new headcount, we have to get costs down, not move them up!”

As Alice and Bill work together to establish the metric and reduce the number of defects, a funny thing happens. Both Alice and Bill see that automation of deployment tasks will positively impact both of their goals without requiring any additional headcount. Alice gets to reduce lead time on applying fixes while Bill gets to decrease the likelihood of errors as the automation scripts improve. While working through the deployment automation, it becomes clear that the exact same effect happens when the QA process is automated. Even funnier, when the previously manual processes are automated, peoples’ attention begins to free up to accelerate the pace of automating manual processes.

Again and again and again it happens, first in automation moving backwards through the development lifecycle and then in instrumentation to both lower the time to notice that a defect has occurred and also to catch a defect earlier in the lifecycle before it gets deployed. Defects and outages go down, speed and throughput go up and what before Alice and Bill can say “Bob’s your uncle”, their teams are doing everything they can to speed up the automation processes themselves to drive cycle time even lower. What a wonderful coincidence! Performance engineering has now become something that the teams now actively engage in.

The Moral of the Story

Did Alice, Bill and Carl live happily ever after again? Time will tell and they are more certain to get better results out of their teams given that they have visibility into, and are grounded by, a metric that puts how you respond to an error as more important than error prevention. Next week, I’ll go under the hood yet again and give a little more insight into how to create a robust, mature and almost qualitative view of my favorite metric: MTTR.

April 22, 2014