DORA’s annual “State of DevOps Report” is something I look forward to every year. There is no other report out there that gives us a view and insight into the industry the way this one does. And I know I’m not alone in my beliefs. That’s why it’s always a treat to speak with one of the folks from DORA—DevOps Research and Assessment—about the “State of DevOps Report” and get some real takeaways.
In this DevOps Chat, Jez Humble, CTO and co-founder of DORA, and I talk about some of the findings in this year’s “Accelerate the State of DevOps Report”—a new name, but the same valuable information.
As usual, the streaming audio is immediately below, followed by the transcript of our conversation.
Alan Shimel: Hey, everyone. It’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Today’s DevOps Chat is one I look forward to every year and that’s the one where we speak about the latest “State of DevOps Report.” Helping me, or actually our guest and star for this year’s “State of DevOps Report” review, is none other than Jez Humble, co-founder, CTO for DORA, DevOps Research and Assessment and just all-around DevOps dude and great guy to talk to. Jez, welcome to DevOps Chat.
Jez Humble: Thanks so much for having me, Alan. Always a pleasure.
Shimel: Thank you. And Jez, you know, you guys, Nicole and you and Gene and the rest of the team have done another bang up job this year in the “State of DevOps Report.” It has a little bit of a different nomenclature this year. It’s actually – if I mess it up, correct me –”Accelerate the State of DevOps Report”?
Humble: Yeah, that’s right. It’s the “Accelerate State of DevOps Report” this year but continuing the same rigorous research as previous years and extending it with some exciting new findings this time.
Shimel: Yeah. And another thing I will mention, right, let’s get it out of the way early on, is that this year’s report diamond sponsor is Google Cloud, correct?
Humble: Yep, that’s right. We have 15 gold sponsors as well, but our main collaboration this year is with Google Cloud. That’s correct, thank you.
Shimel: Fantastic. All right, now that we’ve got the commercial out of the way, Jez, let’s dive in. So this is what, the fifth or sixth year you guys have worked on State of DevOps Report?
Humble: Yeah, so this is the fifth year that DORA has been involved in or been leading the research for this State of DevOps Report. That’s correct.
Shimel: And so, you know, when we look at it from that point of view, you know, with hindsight, it really is a massive amount of data there to dig into, but let’s concentrate on this year’s report, Jez. Let’s start with what do you think for our listeners, what do you think the biggest takeaway for them is?
Humble: So yes, as you say, this is the biggest research program of its kind looking into DevOps, what DevOps means, why it’s important, the factors that predict your ability to deliver with speed and stability. You’d think, you know, this is the fifth year that we’re doing this that we’d run out of things to talk about. That’s absolutely not the case. This is the biggest and most comprehensive one yet. I think a few big takeaways.
No. 1, again, we found that our measure of software delivery performance is still valid. That’s looking at speed and stability. We find again that the high performers are not trading off one against the other. They’re actually increasing both the speed and stability of the delivery process. We’ve extended that measure out also to look at availability as well. So availability is availability in production. It’s a measure of operational performance. Because we’re also looking at that, we now talk about software delivery and operational performance. So we’re covering the whole delivery life cycle from development right through to operations in production. Again, we find the high performers are not making trade-offs there. They’re achieving high levels of throughput and stability and availability as well.
We’ve also looked at the key technical practices that drive high performance, and we’ve expanded that this year to also look at monitoring and observability to look at the impacts of continuous testing, to look at database change management, and to look at shifting network security. We’ve also looked in more depth at culture. We’ve looked at the management practices that shape culture to improve performance outcomes. This year we’ve also looked at the role of leaders on influencing culture through giving their teams autonomy and building trust. We find that promoting a culture learning fostered through retrospectives learning reviews has an impact on culture. The highest performers are 1.5 times more likely to consistently hold retrospectives and use them to improve their work which is critical.
We also looked at cloud infrastructure. A lot of people are moving to the cloud. That’s very hot, and it’s seen huge adoption across many different industries, but what we found is that how you do it matters. Now the National Institute of Science and Technology has a definition of cloud infrastructure and they talk about five essential capabilities that you have to implement – things like on demand self-service whereby customers can unilaterally provision cloud resources. We find that high performers are 23 times more likely to be using those five essential characteristics that NIST talks about. So there’s lots of people who are using cloud but don’t actually have those capabilities implemented. That’s a problem. How you do this stuff matters. Just putting your stuff in the cloud and pretending it’s a data center doesn’t actually work. You need to actually use the capabilities the cloud provides as well.
We found that using open source software improves performance. And finally, again, we’ve shown that industry doesn’t matter. We find high performers among all different market groups, all different industries whether that’s financial services, health care, other highly regulated groups, as well as, you know, startups and small companies in the low performing groups. So anyone can do this stuff.
Shimel: Absolutely. [clears throat] You know, Jez, I find it not comforting but we hear you talk and you talk about, well, companies that are moving in the cloud and stuff. You know, I guess was it around maybe 2005 where all of a sudden the cloud really started kind of bursting on the scene and here we are 13 going on maybe 15 years and, you know, it’s almost like, well, when will we really see – are we in the midst of like the – it’s mainstream certainly, but have we seen – where are we in cloud migration and how this is impacting us? Like has the bulk of it already taken place or is the bulk of it in front of us?
Humble: Yeah. I mean that’s a great question, and I think we see the same trend with DevOps and with, you know, continuous delivery as well. People say they’re doing and the question is are they really doing it because everyone wants to do it and then say they’re done and what’s the next big thing? I mean if we look at our data, what we see is 17 percent of respondents are not using cloud at all and the rest which is 83 percent are using cloud including 40 percent using multiple cloud providers. But if we look at who’s actually doing it right, only a very small proportion of people who actually say they’re doing it in fact are. Twenty-two percent of our respondents said they met all those essential characteristics. So I would say that there’s some way to go. I mean it’s literally the Pareto principle, right? Eighty percent of people are doing it; only twenty-two percent are doing it right.
So I think there’s a long tail with all this stuff. Whether it’s DevOps adoption, whether it’s cloud adoption, we’re all learning as we go along. Change takes time. It’s hard. To do this stuff right you need – and this again is common across tech and DevOps – you need to change the way you do things. You need to change your practices. You need to change your processes. That’s hard and that’s the bit that takes the time. It’s complicated. People are very keen to adopt new technologies, you know, Kubernetes, Docker, Cloudstaff, and they adopt it; but changing their processes and practices, that’s the hard work; but it’s also the important work. We see that very clearly in our research.
Shimel: Yeah, yeah. And you’re right. I mean I’ve often thought that, you know, there’s a chicken and egg relationship between DevOps and many of the megatrends that we’ve seen pop up over the last decade including the cloud and ubiquitous connectivity and the move to mobile, you know, and application driven sort of IT world. I don’t know, you know, if any one of those things happens without any of the other things including DevOps, frankly, which makes this so important. So Jez, when we look at this year’s findings in historical perspective to previous years, any kind of anomalies, anything that jumps out that says, boy, this seems different than years before, this seems kind of not to be moving along that same track that we’ve seen in years before?
Humble: Yeah, great question. So a few things. Firstly, we’ve got a new high performance group, the elite performers. So in previous years, our cluster analysis has shown us that there’s three groups – high performers, medium performers, low performers. So when we did the cluster analysis this year, we found a couple of interesting trends. So firstly, we’ve got an elite performance group which shows that the industry continues to improve its software development and delivery practices.
And our high-performance group has grown in terms of the percentage of respondents which shows that high performance is attainable for many teams in the industry, not just reserved for an exclusive group of teams with unique characteristics. So high performers continue to deliver and operate software at the highest levels, but the bar for excellence has continued to evolve across the industry and this fourth group of performers, elite performers is still optimizing and getting better. That’s what we see in general. The best organizations are the ones that keep trying to get better and are never satisfied.
The other interesting trend we saw with the cluster analysis was we found that a small group, about 7 percent that we’ve called the misguided performers. So the misguided performers is interesting as well. So this is a group that’s basically optimizing for going slowly. They’re making safe choices but they pay a really expensive price. So they’re taking this very cautious approach, and they want to release infrequently, and they see that as an effective strategy and that extra time between deployments for testing and quality checks, but they report really long times to restore service when there’s outage like months and months of time to restore service which is a huge problem.
So our research shows that making large batch, infrequent changes introduces risk to the deployment process; failure is inevitable in complex systems; and when large failures do occur, these big batch changes can contribute additional complexity when attempting to both understand failures or when dealing with catastrophic or cascading failures through the system. Sorry, 5 percent of teams that are doing exactly this and seeing the consequences. So those are kind of some interesting things that we saw in this year’s data.
Shimel: Absolutely. So you know, Jez, certainly since the founding, the creation, the launch of DORA, the survey has taken on not a greater mission but it’s so intertwined with what DORA itself does, right, to a large degree. And I don’t know if our audience – I think they’ve all been reading this survey for years. They may have had heard of DORA. They know Nicole. They know you. They certainly know Gene. But lay it out. Connect the dots for us. How does the survey go sort of arm in arm or hand in hand, whatever the case may be, with DORA and its mission.
Humble: Yes. So our mission, as DORA, DevOps Research and Assessment, our mission is to help the industry create excellence in software delivery. We really want to help organizations get better. We have a product. Our product is an assessment tool that can help you see how you’re doing and see what the constraints are in terms of your ability to get better. There’s a bunch of assessment products out there. We really wanted to take a scientific approach to this. So four years of doing this previously with Puppet and now doing it with Google led by Nicole – Nicole was an academic. She was a tenure track professor. She has a long background in research from an academic point of view. She really has found a way to work out what matters in an academically rigorous way that’s scientific.
All this research we’ve been doing actually allows us to say, here’s how you measure performance, here’s how we can show it has an impact on the organizational outcomes you care about, here are the things that predict your ability to lever with speed and stability and that’s a combination of technical and cultural and process based capabilities. So we can say these things actually matter. So here are the things that matter. Here’s how you measure them. Here’s how they impact your software delivery performance, and here’s how that predicts the organizational performance. So it’s really about taking a scientific approach to measuring these things and improving them.
Actually this year we came out with a book, “Accelerate,” which talks about that research program and the results we found over the last four years of doing that program. So it’s all about taking a scientific research based approach to improving technical capabilities around delivering software.
Shimel: Absolutely. Absolutely. And Jez, believe it or not, we’re almost out of time already. As I mentioned, it does go quickly. But I want to do a little bit more on DORA for people who don’t get it. I know you’ve worked publically – I think it’s announced DORA’s worked with Capital One among other enterprises. Give our audience, you know, again, a little bit more about DORA’s mission and how you’re helping companies there.
Humble: Yeah. Well, I mean thanks very much for the opportunity to talk about this. We have a solid number of companies who are our customers from all kinds of industries including very large, highly regulated industries including finance and government who are interested in finding out how to get better. We basically apply our research that we’ve gathered over the last several years and help them understand what’s holding them beck, what their constraints are, and then how to actually improve that in a research databased way.
So it’s something that applies everywhere. It’s something that applies – and this is one of the great things about this research is that it shows that anyone can do this stuff. So this applies anywhere. You know, I’ve often heard people say, “You know, well, we can’t do that. This won’t work here. DevOps, whatever cloud, continuous delivery, it sounds great, won’t work here.” And we can really come in and say, “Well, actually, you know, that’s really not the case. Anybody can get better, and here’s how the data shows that people in all kinds of different industries of all different sizes can improve.” And again, we’ve been able to help companies actually find out where they are and find out their constraints and understand how to improve.
It’s been very helpful from that perspective, not just – and this is a great think about working at DORA that I love is that not only is it scientific and research based, it’s also pragmatic and practical and it has real impact on the ground with companies looking to improve their software and development capabilities. So it’s really a total pleasure and great fun being able to not just do the research but also apply it in practice. That’s a real privilege for me being in that position to be able to apply this in this way.
Shimel: Absolutely. Well, Jez, you know, continued success with DORA and I’m looking forward to – oh my god, I forgot to mention, Jez, if people want to download the report, where can they go get it?
Humble: Yes, thank you so much. That’s an excellent question to ask. So you can go to our website. DevOps-research.com. If you go to the research, it’s on the front page. It’s on our research page. The link to download the report from Google Cloud. You can also go to bit.ly/2018-devops-report. That will take you straight to the registration page so you can download the report. Also a reminder that we’ve discussed the last four years of our research and our research program in Accelerate, the book that we came out with earlier this year. But yeah, please go and look at the report. Obviously, we’ve only been able to touch the tip of the iceberg in this discussion. There’s loads of, if I say so myself, very interesting stuff there. So encourage you to go and download it and check it out.
Shimel: Absolutely. All right, Jez Humble, always a pleasure, my friend, to have you on our show. Thank you very much for doing the report. Continued success. Say hello to Dr. Nicole and Gene and the rest of the gang and we’ll have you back again soon on DevOps Chat.
Humble: Thanks so much, Alan. Always a pleasure. Thank you.
Shimel: My pleasure. This is Alan Shimel for devopsy.kinsta.cloud, DevOps Chat. You’ve just listened to another chat. Have a great day everyone.
André brings more than 20 years of experience in high technology marketing and communications to his role as vice president of marketing at CloudBees. He has experience in several enterprise software markets including application development tools, middleware, manufacturing and supply chain, enterprise search and software quality and testing tools. André has particular expertise in working with the executive teams of enterprise software companies to create unique competitive positioning, build pipeline and establish sustainable market leadership. Prior to joining CloudBees, he served as principal analyst serving technology marketers at Forrester Research. He has also held a number of senior management positions in marketing and communications with companies such as Segue Software, Hewlett-Packard, Unisys Corp. and Bluestone Software. André holds a BS from Eastern University.
Alanna Brown heads up DevOps research and go-to-marketing strategies at Puppet Labs. She also works closely with Puppet Labs product owners and engineers to develop a roadmap and support for Puppet on Windows.
