Category: Features

The results and analysis of the State of DevOps Report 2017 report are due out in early June. just in time for the DevOps Enterprise Summit London June 5-6. In this DevOps Chat we speak with Puppet exec and one of our favorite interviews, Nigel Kersten.

Of course, Nigel and the folks over at Puppet have been co-producing the State of DevOps report and survey for six years now. For this year, Nigel along with Dr. Nicole Forsgren and Jez Humble of DORA, will be co-presenting at the DOES London, where they will announce the results of this year’s survey. It should be quite interesting.

Nigel gave us some sneak peeks and early trends for this years report, as well as discussed the DevOps Enterprise Summit and, as usual, some good info on what is happening at Puppet and the DevOps world in general.

You know the drill by now: The streaming audio of our conversation is immediately below and the transcript of our conversation is below that. Enjoy!

Streaming Audio

Chat Transcript

Alan Shimel: Hey, this Alan Shimel, DevOps.com, and we’re here for another DevOps Chat. And we’re happy to be joined again by a repeat guest on DevOps Chat, none other than Nigel Kersten of Puppet. Nigel, welcome.

Nigel Kersten: Hey, Alan. Nice to talk to you again.

Shimel: Good to speak with you. So, Nigel, we’re not talking about the latest and greatest from Puppet today. Well, maybe if you wanna just mention it quickly, feel free, but I wanted to really talk a little bit about the upcoming DOES London event, Gene Kim’s DevOps Enterprise Summit, which is June 5th and 6th in London at the QEII Center, and you will be presenting there. Correct?

Kersten: I will be. Yep, with Dr. Nicole Forsgren, who’s one of the other authors of the DevOps report, and DevOps Enterprise Summit London last year, I think, was honestly one of the best events I’ve been to in this space for a long time and I adore London for lots of reasons, and so it’s always great to be back there.

Shimel: Yeah. And, Nigel, correct me if I’m wrong – is Jez Humble also presenting with you and Dr. Nicole?

Kersten: I think Jez probably will be, but I’m not sure if it’s quite confirmed yet. We’re sort of trying to work out how we’re covering all of the different events, for the various roadshows and webinars we’re doing around the DevOps Enterprise Summits and the DevOps enterprise report.

Shimel: Yep. Okay, so I guess we’ll stand by for final confirmation on that, but, certainly, you and Dr. Nicole will be on, and, of course, Nicole has appeared on our show before and we’ve talked a little bit about not only this year’s DevOps survey but past years’ as well. But, Nigel, for some of our listeners who maybe weren’t lucky enough to hear previous talks about it, this is what? The sixth or seventh year of the State of DevOps survey?

Kersten: This is the sixth year, which is kind of amazing ’cause, you know, I think, when we first started with the fantastic Alanna Brown, who also works here at Puppet, everyone was a bit skeptical about it. I think this was even before you’d started DevOps.com and people were like, “Is this really a thing? Do we really need a survey about all of this?” but I think it’s just gone from strength to strength over the years. So, yes, I guess, for the people who are sort of new and may not have come across the report before, what we essentially do is we survey a whole bunch of practitioners—we’ve done tens of thousands of ’em around the world over the last six years—and what we do is we ask about the practices and the tools that they use, but also some metrics about “What’s your mean time before failures? How quickly can you deploy your software? What sort of practices do you use?” as well as what sort of outcomes you have. And then we do a statistical analysis on all of that so that we can—and we actually end up showing that there’s a huge gap between what we call “high-performing” IT organizations and everyone else.

And, in fact, interestingly enough, over the last few years, we’ve shown that the gap is increasing over time, so the people who’ve adopted DevOps practices—because, essentially, what we’re showing is we’ve been able to show statistically that what we tend to call “DevOps practices,” around using CI, using CD, using infrastructure-as-code software, doing continuous deployment, using testing, storing things in version control, all of these practices, when you cluster them, are the people—the people who employ these practices are the people who have high-performing organizations. And they’re really huge statistics—they tend to be thousands of times better, being able to deploy just ridiculously faster—and, essentially, what this ends up showing is you really don’t have to sacrifice quality for speed or vice versa, so we’re seeing that these people are both producing high-quality infrastructure and applications and they’re doing it far more quickly than everyone else.

I think one of the things I’m really proud of with the DevOps report is that it’s useful to sell DevOps practices within your organization, so I talk to an awful lot of customers and practitioners out there who are really thankful for the report because it tends to be focused on business outcomes. So, even though we’re surveying practitioners, it’s something that’s really applicable to C-levels and VPs and directors. They can look at this with them and go, “Ah, this is where the industry’s moving and these are the sort of benefits we’ll actually get.”

Shimel: Excellent. And, Nigel, in typical humble Nigel self, you’re—I mean, let’s give a little bit more flavor, if we can. Over the five previous years, I think there’s over 25,000 responses to these surveys.

Kersten: Yeah, no, that’s right. 25,000 over five, six years. It’s pretty crazy.

Shimel: So that does represent a—in a field like DevOps, which is fairly new, right, it represents a sizable amount of people and organizations who have taken the time and contributed to the data that you guys have analyzed over the years. And another important thing, I think, to note for our audience is not only the analysis of the data but even the questions themselves are, from a statistics point of view, a statistician’s point of view, rigorous—rigorously designed and rigorously analyzed. And, look, having a Ph.D. statistician helping you with it doesn’t hurt, but so these are not just some –

Kersten: No, it’s not your average –

Shimel: – go up and doing surveys.

Kersten: – sort of marketing survey at all, and I think this is why it’s always really important for us that—you know, we’re a software vendor and we’re in this space and there’s a lot of people in our space who do pretty fluffy kind of surveys that only take two or three minutes and then some sort of ridiculous claim gets made, so this is why we’ve always been really keen on working with the external parties of Gene Kim, Jez Humble, Dr. Nicole Forsgren. Particularly Nicole. She’s a trained statistician who’s literally got a Ph.D. and has spent her career working on studying organizational performance, which overlaps with DevOps in really interesting ways, but I think having that external rigor, and I think it helps keep us honest as a software vendor. So this is something we really loved, having a partnership with them, ’cause this is their specialty.

They actually—to give them a little bit of a plug, they have their own organization, DORA, DevOps Research and Assessment, where they’ll come in and do benchmarking of your organization and actually measure how you’re actually doing, so I think it’s just really great. We’re clearly really plugged in as Puppet into this whole community, but they plugged in at a different level, and working together, I think, means that we both get to expand reach and bring a bunch of statistical rigor to something that could easily devolve into a self-serving survey, which ultimately doesn’t –

Shimel: Hey, Nigel?

Kersten: Yep?

Shimel: I thought I lost you there for a second. I apologize. Agreed, I mean, on all of the above. And that’s really—I want people to understand that, that this isn’t a marketing survey they up and have done now for six years running. I wanted to make another point, Nigel, and that is, certainly, over the last two years at least, and I would imagine for this year as well, one of the real nuggets of wisdom that have come out of this analysis is really just how high-performing or what high-performing IT organizations mean to the bottom line of these companies that are doing it. I think, at some level, we all at a gut level say, “Well, if you’re performing better, you’re releasing faster, it should translate into a better company,” but, certainly, the State of DevOps report—survey and report is one of the few that are out there that really connects the dotted lines here.

Kersten: Absolutely. So you look into the sort of things that people are actually sort of seeing. What we’re seeing, so to sort of hit some of the boon stats from last year, is that we’re seeing the high-performing IT orgs, three times lower change-failure rate, 200 times more frequent deployments, and 24 times faster recovery from failures. And those are really amazing stats because I think they really resonate, both with practitioners who’re the people who are getting woke up in the middle of the night or having to deal with the pain of a deployment that didn’t work properly, but they’re also the sort of stats that your average, been-away-from-the-technology-keyboard for a few years, high-level executive, can look at it and they go, “No, that’s actually what we want. We want to be able to deploy more frequently,” because, I think, one of the lessons that SAS has taught us is the reason a lot of SAS companies outstrip on-premise competitors is because, every time they deploy, it’s a chance for learning, it’s a change for feedback.

And Agile has taught us, from traditional software development, the more frequently you can deploy, the smaller the deltas are in those deployments and you can do more frequent cost correction, so you can actually produce a much better outcome over time by deploying more frequently and being able to correct course more frequently.

And, to, I think, bring it back to enterprise folks, not everyone always thinks about deployments in this way, but, every time you get a patch from your vendor, that’s another deployment. Every time you respond to a security incident internally, every time you have to follow through a recommendation of your audit and compliance committee—all of those things are deployments and the more pain-free and friction-free you can make the ability to deploy within an organization, the more responsive you’re gonna be to all the different stakeholders.

Shimel: Yeah. Let me give you my favorite stat that you omitted or you just didn’t get a chance to, and that is these high-performing IT organizations have 50 percent less vulnerabilities post-deployment.

Kersten: Yep.

Shimel: That’s huge.

Kersten: It’s a major. And you’re spending less time on that. And I think the other related one that’s really amazing is that the high-performers spend 22 percent less time on unplanned work and rework. And, for anyone’s who’s been neck-deep in infrastructure, it’s the unplanned work that kills you. It’s the context switching. You’re not making progress on your defined road map every time you’re having to spend time on unplanned work.

Shimel: Yeah. So, Nigel, these are all fantastic insights and metrics, so now, of course, though, we’ll turn the table on you and say, “Hey, what have you got for us this year?” I know it’s early; I know you haven’t released the report yet. You’ll probably be announcing a lot of the results, I think, come out just at the time of DOES London, which, again, is June 5th and 6th, but is there anything you can share with our audience as maybe a preview?

Kersten: So I think one of the things that I could share is we’re still going through doing the analysis, but we’re seeing that, again, the high-performing orgs are achieving just as well compared to everyone else, as they have in previous years. It doesn’t look like there’s only sort of peak, as far as implementing these practices. I think the whole industry, even people who have been doing this for a while, have still a fair way to improve.

I think one of the really interesting things—well, there’s two big ones I wanna call out. One is that, this year, we spend a bit of time working on non-profit performance ’cause, in previous years, we’ve shown that commercial companies who are high-performing IT orgs tend to outperform their financial metrics, outperform in the stock market, their internal company goals, but, this year, we decided to focus on non-profit performance, and so we’re looking at things like, “What’s their operating efficiency like? Customer satisfaction? Quality of the products that have been shipped?” So we’re actually, I think, showing that DevOps practices in high-performing IT orgs impact in a positive way all sorts of industries, so I think that’s gonna be one of the really interesting sections.

The other section, I think, that’s gonna be really fascinating is that we’re actually looking at what’s the—there’s a myth out there, to sort of back up a little bit, that DevOps practices tend to work if you’re building all of the software yourself, if you’ve got a web app, if you’re building a whole bunch of stuff in Node.js or something like that, and deploying it to the cloud, sure, you can do DevOps—you can do continuous integration; you can do continuous delivery—but, for those of us who are sitting there running commercial, off-the-shelf software, we can’t actually do that and we can’t actually obtain the benefits that are available by employing DevOps practices. So I think what we’re gonna be showing is that this isn’t true, that there’s a couple of things you can do as an organization, in the way you consume commercial, off-the-shelf software, to actually let you employ continuous integration, continuous delivery.

Now the reality is you’re not gonna be able to test all the same kinds of things because, unfortunately, you know, lots of vendors that release software don’t necessarily have an API-first policy, so you’re not always gonna be able to test everything inside the application, like you can’t run unit tests, necessarily, ’cause it’s not your code, but there are a whole bunch of things you can do. And one of the big bits of guidance that we’re gonna be pushing is that, when it comes to commercial, off-the-shelf software, really try to make the installation as vanilla as possible. Don’t spend forever customizing it. And, if you’re having to customize it because that’s what your business processes say, in many ways, the lower-cost and better-outcome approach is to not customize the software and adjust your business processes because there’s something within your control, generally. And if you adjust those, then it becomes simpler to deploy that software into a CI system, to have Jenkins or Bamboo or whatever do deployments of it, do external black-box testing, make sure that it actually works, and then roll it out to production.

So we’ve got a pretty strong belief that, when it comes to utility computing, to sort of use Martin Fowler’s distinction between utility and strategic computing.

Shimel: Yep.

Kersten: There are huge efficiency gains to be made there, but there’s a few things you can do to avoid those problems. The analogy I kind of come back to is that there was a period, I think, in the late ’90s and early 2000s, when lots of people would customize their Apache installs. They would sit there and come up with this fine-grained list of modules, configured in specific ways, and, every time they deployed a new version of Apache, they’d eke out another one or two percent performance by having this really customized installation. It turns out the actual cost to the business is really great because those tend to be manual processes—you’re turning yourself into a snowflake when you didn’t actually need to be—and I think the more you can commoditize the software you’re buying and that you’re building and that you’re deploying, the faster and more friction-free a lot of these processes are gonna be.

Shimel: Got it. You know, Nigel, that—so I’m old. That’s firing up some old synapses in my brain back from the dot-com days, when I helped start a company called “Interliant.” We were an early ASP. And, mind you, this is before we have a modern cloud and virtualization and ubiquitous connectivity, but we were offering hosted Lotus Notes, PeopleSoft, Oracle applications, Lawson ERP—I don’t know if you remember Onyx was an early CRM tool back then.

Kersten: Oh, yeah. Yep.

Shimel: That’s a blast from the past, huh?

Kersten: Totally.

Shimel: Anyway, so our thought was to offer hosted versions of these applications. This is early, right? We called them “application service providers.” And what we quickly learned in—you know, when you go to market, and we did the whole IPO thing, all of that, but what we learned from our enterprise customers was that, as much as we wanted them to run a vanilla PeopleSoft or a vanilla Lawson or Onyx or even Lotus Notes, for that matter, the best we could hope for was about 80 to 85 percent of out-of-the-box, just plug-and-go. They always needed—and so what happened was we had to change our whole business model and we wound up acquiring a bunch of service bodyshops, for lack of a better word, who would come in and do that last-mile customization, if you will, that last 15 percent. And, as a result of that, I will tell you, the business model failed, ultimately bankrupt. And, if you look at all those early ASPs, none of them really succeeded—USinternetworking and Corio, some others—because we weren’t able to get people to use the vanilla configs that would have been easy for us to host, spin up, and put down.

I think you see the same thing a lot today, with kind of bare-metal hosting. Right?

Kersten: Yep.

Shimel: Versus a traditional cloud virtual, you know, kind of hypervisor thing. How many people are using just vanilla bare-metal kind of config versus how many have—and you guys are Puppet, right? You’re dealing with this more than most, right? How many people are highly, highly customizing their bare-metals?

Kersten: I think we’re seeing people shift away from customization. I think there’s probably a few things. One is I think people are realizing that a lot of this industry is sort of commoditizing, and so you may as well use off-the-shelf stuff. And so, if you’re just consuming stuff the way it was sort of shipped from the vendor, whether it be hardware or software, there’s sort of a general expectation that, I think, both software quality has improved—as much as we like to bitch and complain about software, I generally think software has gotten better than it used to be. There was no sort of Golden Age of incredibly rigorous enterprise software, as far as I’m concerned. I think stuff’s getting better. So I think there’s a couple of different pressures there, is that software’s gotten better, people have gotten more used to the idea that various things are being commoditized, and so they’re more willing to entertain the notion that the next layer up is getting commoditized.

Shimel: Yep.

Kersten: I think we’re also seeing things like—so here’s a good example, I think. If you consume Atlassian’s JIRA, the hosted version, versus the on-premise version, you can’t customize it as much. And, if you’re consuming GitHub the public service, versus GitHub enterprise, you can’t customize it as much. And so there’s a general understanding from people that, you know, an uncustomized version, or as much as is sort of possible, is actually pretty good and we don’t need to make everything completely different for everyone else because it means that documentation just works out of the box and people who’ve used the software somewhere else can join the company and have much faster on-ramp.

And I think we’re seeing that—and this is a somewhat optimistic approach—that there’s a general desire for people to adopt software engineering principles and sort of DevOps cultural changes to get to that higher-performing IT organization state. And I genuinely think, if we keep pushing this sort of attitude out to people that, “If you don’t customize your software, adjust business processes to fit instead, that’s cheaper and you’ll get a better outcome,” I think all of those things combined sort of give me hope that we’re actually heading towards a world where people are giving up unnecessary customization.

Shimel: I hope so. But, Nigel, we’re so far off our course here. You know, we were supposed to be talking about DOES London, and let me try to bring you back. So, I mean, obviously, all of this is background and the—this is why this survey is so important, so that we see these trends and we see it manifest itself in how companies are doing things and so forth. Let me ask a little bit—so, Nigel, you’ve presented at DOES before and I did an earlier interview today—I was talking to one of the program committee’s members for London, Jonathan from Hiscox—and –

Kersten: Ah, yes.

Shimel: Yep. And, forgive me, his last name—Jonathan Fletcher from Hiscox.

Kersten: Yeah.

Shimel: You know, and I’ve known Jonathan for some time—great guy. What we—you know, we forget—all rolled into one place in the space of two days, we’re gonna be jam-packed with all of this great information around DevOps and what we’re seeing and not only what we’re seeing but people are gonna talk about what they’re doing right now and what they did last year versus this year versus next year, what the findings are in our surveys, and I think that’s what really makes for a great event. Right? I think –

Kersten: Yeah, I think there’s a couple of things that are really unique about it. I think one is that it’s not a vendor event, and I’m in the business of working the software company and we make and sell software, but it’s also really refreshing to go to an event where everyone who’s on stage is really there just telling case studies, their real-world stories. They’re not product-pitching to you. We’re not getting up and pitching Puppet Enterprise to you at all. We’re getting up and talking about the DevOps report, and everyone who’s there is—I think this is Gene’s big focus, is that everyone’s telling a real-world case study about what they’re actually doing, where the challenges were. It’s war. It’s honest. So I think that’s one section of it, that it’s real stories from out in the field.

But, perhaps more importantly, it’s not just practitioners. And I think those of us who sort of have had a technical background can sometimes get too in the weeds about going and watching someone’s talk about this incredibly interesting deployment they’ve done and what they achieved, but we’re seeing business leaders talking about the actual impact on the business, how they ran large departments, large transformation projects. You know, the term “digital transformation”’s been kicking around for quite a long time, but we tend to see very sanitized case studies, driven either by a software vendor or a large consulting firm or the PR department of some multinational corporation talking about what they’re doing, and these tend to be really warts-and-all stories, going, “This is what we did and this is how it worked or this is how this didn’t work.”

And I think some of those are some of the more fascinating stories, where people are getting up and talking about “Here were our lofty ambitions and we only got 60 percent of the way there and these were the problems we ran into.” And that, I think, is just really, really powerful, so I think the fact that you’ve got senior business leaders and executives there, as well as the fact that they’re all real-world case studies, is just a really, really powerful combination.

Shimel: Yep. I agree with you. As a matter of fact, one of the things I’d love to see is more people get up and talk about failures, right, where maybe DevOps, they didn’t do it right, or what they could have done differently or what have you, where it didn’t work out right and this is what we did as a result, ’cause I think too many people come out of these events, out of Gene’s things, sometimes, and it’s like it’s all rainbows and unicorns and horses and such, and that’s great, but we—you know, as a dad, I try to teach this to my kids—you don’t always win. And that’s why you gotta learn to deal with failure and how to recover from that as well. And I think those are some of the best lessons for people to learn, is we learn more from our failures than from our successes sometimes, so.

Kersten: Absolutely. And I think it’s hard because we haven’t always had a culture in the IT industry of actually performing postmortems and retrospectives on large projects that we’ve done, so it’s people don’t always have great answers for how they would have done things differently, but I think that’s something that’s actually also starting to shift quite considerably. We’re seeing DevOps has been a big part of that, of “Let’s do blameless postmortems,” and, hopefully, that continues so that we see more people being open about their failures.

‘Cause I think one of the things I find really frustrating—this has happened to me more than once and I don’t think this is gonna identify me, one customer or user, because they do so many of them—seeing your executives at Amazon summits get up and talk about how fantastic their whole cloud migration journey has been and that they’re all in the cloud and everything’s done and it’s absolutely wonderful. And then you go talk to the VPs or directors who are actually running those deployments and they’d be like, “Oh, God, no. We’ve only moved five percent of things and it’s been a miserable failure and we’ve now added three years to the project because we’ve fallen over organizational boundaries.” So I think that gap between the sort of PR-style business leader talk and the reality—that’s not a gap that actually exists at the DevOps Enterprise Summits, which I think’s been really fantastic.

Shimel: Yep. Agreed. Agreed. Well, Nigel, our little 15-minute chat has turned into a solid half-hour, and I think we’re gonna need to call a break here. We are looking forward to the release of this year’s analysis of the survey, looking forward to your presentation along with Dr. Nicole at Gene Kim’s DevOps Enterprise Summit in London, June 5th and 6th, and just always looking forward to what you’re up to and what the folks in Puppet are up to. Many people, you know, when they think DevOps, they think of companies like Puppet, right? A leader in—I don’t know if they’re ever gonna do a DevOps Magic Quadrant, but—or some sort of stuff like that, right? But, I mean, seriously, Puppet’s one of the cornerstones here and you’re one of the cornerstones in Puppet, so it’s always great to hear what’s happening from you and getting caught up. It’s a pleasure.

Kersten: Aw, thanks, Alan. So I will get one little pitch in, which is, because of DevOps Enterprise Summit being in London that week, we’re going to be announcing pretty soon that we’ve got a Puppet Camp happening pretty soon afterwards, so there’ll be a Puppet Camp in London, and this will be a slightly different tweak on the Puppet Camp format. We’ve run hundreds and hundreds of these events over the last few years, but we’re doing a slightly different thing, where we’re having a sort of more executive-manager-focused morning and a more practitioner-focused afternoon. So, if you’re someone who’s an IT architect, you might find mornings and afternoons both applicable, but we’re really looking to try and join together the conversation for sort of senior executive leaders as well as practitioners on the ground. So we’re already collecting a whole bunch of really great talks, both for the morning and for the afternoon, but, for those of you who are into Puppet, look out for a Puppet Camp London coming soon after DevOps Enterprise Summit.

Shimel: All right. We’ll try to get info on that up on DevOps.com for anyone interested. Nigel, fantastic, man. Thank you so much for the time today. We appreciate it. Nigel Kersten of Puppet here on DevOps Chat. This is Alan Shimel for DevOps Chat. Hopefully, we’ll see you soon at the next—on the next DevOps Chat. Until then, have a great day.

— Alan Shimel