Tag: continuous integration

Alan Shimel: Hey, everyone, it’s Alan Shimel for devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Lucky you, folks out there, because you’re catching I think what will be one of the best chats of the year. I’m joined by my good friend and DevOps survey expert, guru—just all around great person, Dr. Nicole Forsgren. Nicole, welcome.

Nicole Forsgren: Hey, thanks for having me.

Shimel: It’s my pleasure to have you on, Nicole, and you know I always get excited to have you on our show. But Nicole, double whammy reason to be excited in that the survey is out. I know you put your heart, soul and guts into this thing for 10 months a year or so and, you know, between drawing up the questions, getting the responses, analyzing them, doing the report, then talking about the report, it is—it’s a lot of work. I don’t know if people at home realize exactly how much.

Forsgren: I know, I know. Every year, it’s so much, and every once in a while, I’m like, “No more!” So, I have one week where it’s just crazy and I’m like, “I’m not doing it again!” And two days after release, I’m like, “This is my favorite time of year! It’s the best!” [Laughter]

Shimel: [Laughter] Absolutely. And, you know, just for folks out there who, I don’t know, maybe they’re not familiar or they are familiar but they don’t know the whole story—you’ve been involved in the State of DevOps Report now, is it six years?

Forsgren: Yeah! This is six years. I’ve been lead investigator on the report. So, it’s the State of DevOps Reports. It’s the longest running, largest, academically rigorous research into DevOps. And so, when we say DevOps, we’re talking about developing and delivering software with speed and stability, in ways that drive business outcomes and the capabilities and practices that are impactful. And when I say impactful, I mean things that are statistically significant.

So, when people say, you know, “Does this make a difference? Does CI actually matter? What type of automation is important? Do we care about trunk based development? Everyone says culture is important—does culture matter, and what type of culture?” These are the types of things we’ve been researching for six years now. And it’s—when I say it’s a huge study, we’ve hit over 31,000 data points now.

And—I keep talking, I’m so excited [Laughter]—

Shimel: No, no—go.

Forsgren: —academically rigorous. When I started doing this, I was a professor. So, this has been written up in academic peer review, but because we want to make an impact in industry and because we want everyone to be able to read it and access it and understand it, we translate it, right? We make it accessible.

So, we put it out in the State of DevOps Report. So, I sometimes joke it’s like an adult picture book. It’s larger font, it’s big print, it’s graphs, it’s charts. We do layout really carefully so that if you want, you know, some people print it out and they put it on the wall. You can screenshot it so you can put it in slide decks.

And I also keep saying “we.” So, historically, the DORA authors have been myself, Jez Humble, Gene Kim. The authors this year are myself, Dustin Smith—he’s a new researcher who joined me, he’s with the team at Google Cloud—Jez Humble and then this year, we were also joined by Jesse Frizzell.

Shimel: Yes, absolutely. We ran—we covered that on devopsy.kinsta.cloud, actually, I remember.

Forsgren: Yes!

Shimel: A couple things—and, you know, you guys did the research here, but let’s be clear. You could go to any DevOps event the world over—and I do mean the whole world over—

Forsgren: Yep.

Shimel: And listen to a presentation and, you know, nine out of 10 dentists agree, nine out of 10 presentations will have some metric from this survey that they use to cite as authority on where we are in DevOps, what is the state of CI/CD, et cetera, et cetera. So, it really is—

Forsgren: So, it comes out of the survey, or it comes—

Shimel: From [Cross talk].

Forsgren: Okay, Accelerate, and it comes out of—so, like you said, it could reference a DevOps event or an agile event or a testing event. Because we do tend to cover so many things and we do cover them in a very, very rigorous way, right? So, it’s not—you know, the joke is “lies, damn lies, and statistics,” right?

Shimel: Yep.

Forsgren: We do cover it in a very rigorous way, and we put it forward in the finding, the things that we find, and we say, “This is what we find and these are some of the limitations, and these are the evaluative criteria,” right? Like, it does not include—like, this is not a typical vendor report, right? Like, we don’t—

Shimel: Right, and that’s important.

Forsgren: We don’t talk about tools. And why don’t we talk about tools? Some people are like, “Well, what tool should I use?” I’m like, “I don’t know, what tool should you use?” Like, for example, CI. CI is predictive of better continuous delivery and better speed and stability.

Now, when you do CI, it should—every time you check in code, it should result in an automatic build of the software. It should trigger automated tests. You should get results of your automated build and tests at least every day, and your developers should see that at least every day.

So, here’s your four evaluative criteria. Now, whatever tools you wanna use to do your CI, look at those four pieces. Now, I don’t run through a set of tools because there’s a million tools, and what happens when your tool comes out with a new release? It could change your, I mean—

Shimel: Well, not only that. I mean—

Forsgren: So many things.

Shimel: You wanna know, Nicole, I’m on the other side, right? I’m running this devopsy.kinsta.cloud, we’re media. We do cover tools. It’s a little—it dirties the water. It muddies the water. Because then right away, people wanna know what’s your bias, who’s sponsoring, you know what I mean?

Forsgren: Right.

Shimel: I just, I kinda like it better this—and I think most people do. We’re not looking for you to pick one tool or, this isn’t a magic quadrant nonsense kind of thing.

Forsgren: Well, and this way, I don’t have to keep up with toolset changes or feature set changes or versioning changes.

Shimel: Or who bought who and—yeah, all of that stuff.

Forsgren: Yep. Anyone can take those evaluative criteria, they can apply it to whatever tool they wanna use. Heck, vendors can use it. They can say, “If I’m building a tool, this is what I should be looking at.”

Shimel: Absolutely, absolutely. Let’s talk, you mentioned lies, damn lies, statistics, math.

Forsgren: [Laughter] So much math.

Shimel: You know what, let’s dig in on that a second, because—so, first of all, obviously, you’re a woman, right? And it’s great, we have a woman here talking math and—

Forsgren: Yeah.

Shimel: —for my audience out there, you know what, you can grow up and be Nicole, too.

Forsgren: Uh huh.

Shimel: So, it’s encouraging. But too many people, Nicole, they see big numbers—and I don’t know if it’s the law of big numbers or what, but their eyes glaze over.

Forsgren: Yep.

Shimel: And they won’t recognize something that’s kind of as plain as the nose on their face, right? So, let’s get some of these highlights. Thirty one thousand plus respondents over the last six years.

Forsgren: Yep.

Shimel: That’s huge! I mean, that’s a big pool, right?

Forsgren: We’ve got a lot of data points around the world, all industries. So, it’s helps our data to be more generalizable. So, that’s kind of a research word. What that means is, it means that there’s a high likelihood that any of our findings will work for you—which is important to me, because I used to be an engineer. I also was a consultant for a few years. And so often, I would come forward with a proposal for a solution, and if it sounded hard, many times, my manager or the lead on the project would say, “Oh, well, that won’t work here. That only worked on that other team.” Well, if my results are generalizable, that means it has a high likelihood of working in any team.

Shimel: Absolutely.

Forsgren: Yep.

Shimel: Let me throw some more numbers out at you.

Forsgren: Yep.

Shimel: Normalized annual deployments range from 1,460 deploys a year for high performance down to seven for low performers, right? Seven deploys per year for low performers. Elite performers deploy code 208 times more frequently than low performers, according to the report.

And here’s another one, and from my mind, and I’ll ask you—probably one of the most thought provoking, controversial kind of stats in the report. And that is that 20% of respondents consider their organizations to be elite DevOps performers, up from 7% just last year, right?

Now, guys—don’t get hung up on the math, here. Look at those numbers. They clearly say something, right—1,460 deploys per year is roughly four a day, right That’s what high performers are doing, four a day. Seven a year, maybe a little under two a quarter, right? This isn’t calculus, okay? This is plain arithmetic.

Nicole, what about that 20% number, though?

Forsgren: So, that’s interesting. Let me take it back just a step.

Shimel: Okay.

Forsgren: When we asked people—so, how did we come up with this classification of low, medium, high, elite performers? I asked people four questions about how they’re developing and delivering software, what the outcome is, right? And so, I say, “How often do you push code?” Right?

And the nice thing about that question is, almost everyone can answer it at a very high level. “Can you push code on demand?” Right? “Can you do it when you want to?” So, that’s kind of deploying on demand, multiple per day-ish. “Can you do it on demand? Do you do it about, between once a week—between once a week and once a month?” Sometimes people are like, “That’s a really rough number.” And I’m like, “Yeah, it’s a rough number, but pretty much anyone can answer that. Are you between once a week and once a month?”

Shimel: Right.

Forsgren: You know, notice this is—for people who are kinda into math, this is a log scale. What I mean by log scale is, it’s increasing in big pieces. On demand between once a day and once a week, between once a week and once per month, between once a month and once per six months, right?

Shimel: Yep.

Forsgren: These are getting bigger in big chunks.

Shimel: Yeah.

Forsgren: Which is nice, because it gives me pretty good reliability. I’m not asking for, “Are you deploying between one minute and two minutes, between two minutes and three minutes?” People aren’t good at that. Systems are good at that, people are not. So, I’m asking in big pieces.

Shimel: Yeah, these are big buckets.

Forsgren: Yeah, these are—and they’re large, increasing buckets.

Shimel: Yes, agreed.

Forsgren: Okay, yeah. Okay, so then lead time for changes—that’s the next question I ask. “How long does it take to get from I commit code to code runs in production?” Same type of buckets. It takes less than a day, it takes between a day and a week, it takes between a week and a month, it takes one to six months. People can about answer that question.

Okay, then I have two stability related questions. When something’s broken or when something needs attention, how long does it take to fix? Less than an hour, less than a day, less than a week, between one and six months? Same types of buckets, right?

Now, change/fail rate. When we introduce something to production, what’s the likelihood that something goes wrong? Now, these buckets are basically like, we put into about six buckets, right? So, like, 0% to 15%, 15 to 30, right, 30 to 45, 56 to 60. And people can generally bucket that in their heads.

Then what we do is, I take those four numbers and I throw it into the stats hopper, the stats bucket. And then I—it’s actually called a cluster analysis, because we wanna see how they cluster. So, for people who answer those four questions, I say, “How does those four questions, at the team level”—this is not at the organization level. Because why? Let’s think about organizations. Organizations have teams that operate at very different levels, right?

Shimel: Mm-hmm.

Forsgren: If we think about, especially big organizations, you have some teams that are super, super fast, and you have some teams that are kind of on the struggle bus. Which is fine, because we do things differently.

So, I analyze this at the team level, and the we say, “Okay, how do these four things fall out?” And it’s important to investigate this, because for a long time, for decades, a couple decades, we were always told that in order to be stable, we had to slow down, right? So, remember I said, we had the speed metrics—how often do you deploy, ____ changes? Those are speed, those are throughput. And then we had stability, which is time to restore and what’s your change/fail rate.

Now, here’s the interesting thing—for six years in a row now, those four things have always clustered together. They’ve always grouped together. Now, occasionally, I have people say, “You’re mistaking correlation for causation.” This is not correlation, this is not causation. We are observing in the data, no one—there are no trade-offs. Those things always group together. They are always clustering together.

Shimel: Yep.

Forsgren: For six years in a row. Now, you also commented on—now, how do we get there? Everyone put a pin in that in your head—how do you get to better speed and stability together? Or, conversely, how do you suck at all the things at the same time? Okay, we’ll get there.

But, now, you noted—and this is super interesting and super important—we went from 7 percent of people being amazing at optimizing and speed and stability to 20 percent. That’s an almost tripling. Now, some people are saying, “How are they elite performers? I don’t see 20 percent of people being elite.” Now, this is a definitional thing. This is very much a research definitional thing. This doesn’t mean that—this is me also being a very nice mother. Like, I love all of my children. Everyone is—I love everyone equally. All of my children are special, right?

Shimel: Equally. Right. [Laughter]

Forsgren: Not everyone is going to be a Netflix type of, you know—

Shimel: Organization.

Forsgren: —deploying a thousand times a day. Remember, that definition of elite was, you can deploy on demand, multiple times a day. It could be two or three or four times a day, which was how we did the math, because we were like, “We’re gonna be conservative. We’ll call it four times a day.” And I say this in the report, I randomly picked a number, I said four. Four seems reasonable. Some of these companies are deploying thousands of times a day. I picked four instead of five.

Shimel: Look, I personally think four times a day is amazing, if you’re deploying four times a day—

Forsgren: Yeah, oh, that seems amazing. But some companies are doing thousands, okay? So, when I say elite, I’m not saying elite is thousands, I’m saying on demand and four seems fine. Making time for changes, it’s less than a day. I didn’t say it had to be 10 seconds, right? But that is also my upper limit, because of how I’m collecting my data. I’m not collecting my data from systems, I’m not saying that elite is 5 seconds versus 10 seconds, because I cannot ask a person, right—

Shimel: And who is—right, you’ll never be able to measure that.

Forsgren:—or I can’t ask a person five seconds versus 10 seconds. I can only ask—less than a day.

Shimel: But let’s put that in context.

Forsgren: Yep.

Shimel: So, elite performers do it in less than a day. Low performers are something like 2,500 hours.

Forsgren: Yes, low performers are between one month and six months. This is still a huge—

Shimel: You’re talking a day—so, we don’t need computer precision to the millisecond.

Forsgren: Nope, nope, nope.

Shimel: We’re talking a day versus months.

Forsgren: Months.

Shimel: And people, you don’t have to be a genius to see the difference there.

Forsgren: Exactly. And the great thing is that this is absolutely achievable. And think about what this means for your companies. If you go from deploying code two, three times a year to being able to push code in less than a day and every single day? Some people are like, “Oh, I don’t need to push features that often to my customers.” That’s great, but you know what you probably do need to do is keep up with compliance and regulatory changes.

Shimel: Yep.

Forsgren: You need to keep up with security threats. You need to keep up with infrastructure. This allows you to pivot any time you need to pivot. This allows you to keep up with so many things in your environment.

I gave a great talk at Black Hat with Kelly Shortridge, and she pointed out that one of the best things you can do to be a pain in the butt for security is to re-roll your infrastructure. Because every time you re-roll a container or re-roll your IP is—an attacker has to completely start over from scratch, now.

Shimel: Yeah.

Forsgren: This is so smart, and so good.

Shimel: Well, and that was one of the great things about immutable infrastructure, right?

Forsgren: Exactly!

Shimel: Is, you can absolutely just start from, let them start back—you know, pull the rug out from under the hacker. I mean, but it’s such a—these numbers, like you said, you see these clusters always come together.

Forsgren: Yep, yep.

Shimel: They always follow each other. So, time to remediation, right, becomes such a huge thing. Failure rates drop through the floor when you start doing this.

Forsgren: Yep.

Shimel: You know—

Forsgren: And I love that you point out, when we compare the high and the low performers, our speed metrics, our elite performers are 208 times—they’re seeing 208 times more frequent code deployments. They’re 106 times faster, code commit to deploy. The stability metrics are hug. The elite performers are over 2,600 times faster recovering from incidents. And they see seven times lower change/fail rate. This—this is huge.

Shimel: The numbers are real. Can I throw one other thing at you?

Forsgren: Yeah, yeah, of course.

Shimel: These people who tell you, “Our customers don’t care if we update frequently, it’s not important”—I used to believe them, Nicole. But I’ve come to the conclusion that they’re full of crap. That’s arrogant elitism at its worst, and if you don’t think your customers want you to stay on top of things and be current and constantly improving—show me who you are, because I can’t wait to disrupt you and take your customers from you.

Forsgren: Well, and it’s interesting—

Shimel: But I don’t buy that any more.

Forsgren: I have spoken with one or two government and military contractors who have said, “We live in a different space. I do not have a customer”—

Shimel: Yes, they do.

Forsgren: Well, and they have said, “I have a customer that doesn’t require a feature change.” And I’ve said [Cross talk]—

Shimel: No, they do. I’ve sold to the military for years.

Forsgren: Sure, sure.

Shimel: They want—you know what? And as counterintuitive as it may be, the military—and I’ve sold to all of our armed service divisions as well as the Secretary of Defense at the Pentagon—they, in a lot of ways, lead. Because their requirements are constant. This is why they have things like DISA and they have things like In-Q-Tel that are investing in the next gen.

Forsgren: Oh, absolutely.

Shimel: Because their requirements are constantly pushing the envelope.

Forsgren: Oh, sure. But to that person in particular, I did have someone come up to me after a talk and say, “I actually do not have any competitors. I do not have feature changes.”

Shimel: No, they do. It’s a single world. [Cross talk]

Forsgren: Well, but, to him, I said, “Okay, sure, but you do have compliance and regulatory changes, and you do have security changes.”

Shimel: And you have changing missions.

Forsgren: Yep.

Shimel: Their mission changes. They need—so, again, I just, I don’t buy those people. I think that arrogance is misplaced.

Forsgren: Well, but, it was interesting, because as soon as I pointed out compliance and regulatory and security, he said, “Oh, well, that’s right.”

Shimel: Oh, yeah, yeah.

Forsgren: And that’s what we struggle with.

Shimel: Yep, I agree with you. So, a couple other things, and we’re running low on time, I apologize.

Forsgren: Yep, yep. [Laughter]

Shimel: We could go all day. So, you know, this is the first year that it was Google Cloud involved. Well, Google Cloud was a major sponsor last year.

Forsgren: Yep, we collaborated with them this year—or last year, and then this is the first year that we’ve joined Google Cloud.

Shimel: So, let’s talk about what they brought to the party.

Forsgren: So, they were just a wonderful partner this year. What I really appreciated about working with them this year is, they still allowed me full research independence, and it was fantastic to work with them.

Shimel: Yeah. Well, and it also seems like they’ve brought, they’ve given you the ability to add more resources, perhaps, to the whole process.

Forsgren: Yeah. I really loved being able to work and collaborate with Dustin Smith this year. He brought a fantastic eye for research and analysis this year. He has a phenomenal, phenomenal statistics background. And it was almost like being back in academia again where I could have another academic—he has a Ph.D.

Shimel: Collaborator.

Forsgren: Yeah! So, I had, like, a full academic peer where I used to be able, when I was a professor, to walk next door and bounce ideas off of someone and have someone replicate every single one of my analyses.

Now, in the past, I used to pull former Ph.D. students to replicate a handful of my findings. But, as a tiny startup, you’re real scrappy.

Shimel: Yeah.

Forsgren: This year, I had Dustin, and he replicated every single one of my findings, and then he came up with new ideas, and we could really spitball off of each other and we had full peer review on every single one of our analyses. It was, like, it was just this academic research dream. It was wonderful.

Shimel: No, it’s great. I mean, but this is, for entrepreneurs out there, for people in startups listening to this, this is one of the great things about when you do do a merger or acquisition or whatever. The idea that you now have peers—

Forsgren: Yes!

Shimel: —to bounce around stuff with, to collaborate with. Because, you know, startups, we run lean and mean.

Forsgren: [Laughter]

Shimel: But sometimes, it’s—that’s the code word for by yourself. [Laughter]

Forsgren: Yep, yep. [Laughter]

Shimel: And so, it’s not as cool. I just wanted to mention, you know, quick commercial—we’re doing The DevOps Experience, that’s coming on, I think it’s October 10th. Nicole, you’re headlining. You’re gonna go into greater detail on this, so if anyone’s listening—

Forsgren: Yes, yes! And this year, for the first time—so, in years previous, we’ve always investigated performance. How to make software development and delivery both fast and stable. This year, in 2019, for the first time, we also dug into productivity. So, we have two full research models. So, I will be sure to dig into that.

Shimel: Absolutely. So, you can check that out on DevOpsExperience.io. This is a virtual event, guys, you can watch it from your desktop.

Secondly, Nicole, I guess the obvious question is—alright, what’s the time frame for next year?

Forsgren: Oh, well, I need to take a nap first. [Laughter]

Shimel: Okay. [Laughter]

Forsgren: But we tend to release the report late August, so I’m guessing that neighborhood, that time frame.

Shimel: Well, that’ll be the report, but when do we start, when do the next year’s survey questions come out and all of that good stuff?

Forsgren: So, we usually release the survey and collect data in the March/April time frame.

Shimel: Yep.

Forsgren: So, I’m guessing that as well.

Shimel: So, are you already working on next year’s questions?

Forsgren: I mean, I’ve got a couple ideas in the back of my head, but research is a very creative process, so I try to take at least a month to just unwind my brain.

Shimel: Totally [Cross talk].

Forsgren: Yep, and then—

Shimel: Well, it’s well deserved.

Forsgren: Dustin and I and the team will start kind of opening our brains again. Luckily, we will be at DevOps Enterprise Summit, where we’ll be giving a talk, Dustin and I are speaking, and then that’s one of the best places—

Shimel: And that is October 28th and 29th at the Cosmopolitan in Las Vegas.

Forsgren: Yep.

Shimel: And you can get information, I guess, at IT Revolution.

Forsgren: Yep. And that’s a fantastic opportunity for us to attend talks, speak to everyone who’s really involved in transformation, the wins, the questions, the challenges—that’s where I get so much inspiration. I get all of it, year ‘round. I’ve got, like, my favorite Google Doc where I keep track of ideas. [Laughter] But that’s where I get a lot of the stuff that people don’t necessarily share out in the wild. That’s where I get a lot of my really great research ideas.

Shimel: We’ll be broadcasting live from there, so you’ll have to come down and do a quick little live interview—

Forsgren: Yeah, for sure!

Shimel: —with Digital Anarchist. Very cool. Can’t wait to see you there. So, Nicole, I apologize. I told you this was gonna be 15 minutes and I think we ran 30-something. [Laughter]

Forsgren: Sorry, everyone!

Shimel: But it was well worth it, it was well worth it. Hey, we’ll see you on DevOps Experience, I will see you in person at The DevOps Enterprise Summit in Vegas at the end of October, but in the meantime, go take some well-deserved rest time, my friend. You did good.

Forsgren: Thanks so much.

Shimel: Thank you. Dr. Nicole Forsgren, Founder, CEO of DORA, part of Google Cloud, our guest on this DevOps Chat. This is Alan Shimel for DevOps Chat, devopsy.kinsta.cloud. Have a great day, everyone.

— Alan Shimel