devopsy.kinsta.cloud

Tag: Organizational Change

  • Change Management in a Fast-Paced DevOps World

    Change Management in a Fast-Paced DevOps World

    While DevOps speeds up software release cycles, change management often slows them down. So, how do you do change management in a fast-paced DevOps environment? Change control enables organizations to meet compliance and governance while managing changes at the speed of business.

    In this interview for TechStrong TV, Greg Sanker, former CIO, IT Operations specialist, author, speaker and contributor to the “ITL4 Change Enablement Guide,” joins us to share his views on the tremendous impact of Agile, DevOps and CI/CD on traditional change control teams. Greg talks about the new role of change control and his contributions to the “ITL4 Change Enablement Guide.”

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript: Change Management in DevOps

    Mitch Ashley: Well, I have the pleasure of being joined today by Greg Sanker who is former CIO, operations specialist, he’s a speaker/author, a bit of a futurist in leadership and digital transformation. He’s also a contributor to the “ITIL4 Change Enablement Guide” and a reviewer of a lot of other content—so, long introduction, there. Welcome, Greg, to [Cross talk].

    Greg Sanker: Yeah, wow—thank you. That’s kind of awesome. Thanks for having me. I appreciate being on the show. I think we’re gonna have—

    Ashley: It’s always—always good to have a fellow musician. Anybody with a Telecaster in their background can be on this any time you want.

    Sanker: Yeah.

    Ashley: So, tell us, [Laughter] in addition to your wonderful guitar sitting behind you, tell us a little bit about yourself.

    Sanker: Well, you mentioned, I am—I specialize in IT Operations. That’s what I’ve done for more decades than I’d maybe like to admit. But service management has been a core part of that, and it’s become really clear to me that my view of service management is a lot more streamlined and agile than a lot of people perceive it as. And so, I’ve begun speaking at conferences and whatnot. I have a book of my own that’s published—I’ll let you guys go find it if you’re interested—about change management.

    But most recently, I was asked to work on the ITIL project on a number of ways, but most notably, the change, what we now call “Enablement Guide.” And one of my prerequisites to get engaged with the project was, if you’re gonna ask me to write it, I’m gonna write it the way I see it, which is very different than what we’ve been doing in the past, and we’re gonna be pushing it forward, so. So, that’s a bit about me.

    Ashley: That’s wonderful. That’s definitely a consistent theme with ITIL4 and other folks, Mark Smalley and others that we’ve talked with is really recognizing—I think ITIL’s had its sort of process moniker, right?

    Sanker: Yeah, pretty much.

    Ashley: But there’s a huge emphasis on, you know, how do you fit ITIL and all of the aspects of operations into something that is more of a dynamic DevOps-y agile kind of world, if that’s a way to describe it. And not to say that it didn’t fit in before, but I think it’s making it more prescriptive in that kind of setting is what a lot of the good work that you and others have done.

    Sanker: You know, I was an early embracer of DevOps, and part of the reason for it is the mindset that explores possibility that ignores the boundaries in an organization to achieve outcomes. That’s, like, music to my ears—I’m all in. So, it’s a match made in heaven, but I can mention, it’s like, there’s a lot of organizations that have adopted service management, they have used ITIL in ways that are very prescriptive, very heavy weighted, very bureaucratic and slowed down. And it was never intended to be that way.

    Ashley: Mm-hmm.

    Sanker: It just is, it got adopted that way, right?

    Ashley: Well, and I think a lot of the work that’s gone into ITIL4 will help make some of that shift.

    Sanker: Yeah.

    Ashley: Certainly, it’s positioned that way, and I know from other authors, that was the mindset of how the contributions were made. Now, you specialize around change management and we were chatting when we first got introduced about, you know, how do you do change management in a rapid development, agile, DevOps kinda world? If you were gonna be ask that at a conference by somebody, how would you—what guidance would you give them?

    Sanker: Wow. So, I’ve got an hour and a half platform, right, for that?

    Ashley: [Laughter] At least an hour and a half compressed into 20 minutes, yes.

    Sanker: Twenty minutes, right? Yeah, absolutely. Well, see—well, let me put it this way. I think it really comes down to one fundamental paradigm that we need to change is, traditional change management, which is what I call your CAB-based, you know, Razor request for change, blah blah blah, is based on the idea that we inspect every single change. And as you probably know, Edwards Deming’s said many, many decades ago, “We’ve got to cease reliance on mass inspection to get quality products.” Well, that’s exactly what we’re doing when we say, “Hey, I need to make a change. Well, let’s just take a look at it,” you know, so on Thursday, we’re gonna get together and look at that. But you know full well that, in a DevOps environment or a continuous integration, I mean, we are moving all the time. We don’t slow down and stop.

    And so, that mindset, just the mindset of stop and let’s inspect is completely incompatible with a flow-based, whether it’s agile or DevOps or CI/CD or any of the others that are out there that are more flow based—you know, rapid feedback loops, short cycles, et cetera—it’s just incompatible.

    And so, what we have to do is elevate out of the inspect mindset into the expectation mindset. And that is—what is it that this organization needs from change management? And some of it’s governance and compliance, some of it’s operational stability and recoverability and some of those kinds of things. But let’s state what that is, we’ll state what those things are for the organization so that those that are working in the pipelines, the value streams can actually optimize how well we achieve those expectations and don’t dictate how they do it, let’s just dictate what the outcomes that are required by the organization. That’s the fundamental shift, and it’s big. I mean, it’s more than a 180-degree shift. It’s very significant.

    Ashley: Definitely a different mindset for sure, you know, the inspection versus not. It’s not unfamiliar, though—that’s what security is doing about shifting left and moving ______.

    Sanker: Yeah.

    Ashley: Iterating the process, DataOps is also—you know, everyone wants to move early in the process, but it’s more than that, because you’re talking about a pretty different mind shift change. So, talk about compliance. Talk about governance from a change management standpoint in this new paradigm.

    Sanker: Yeah. So, you jumped right to the jugular. That is the million dollar question.

    Ashley: [Laughter] No, but it needs to be done and, you know, we all sort of pooh-pooh it.

    Sanker: Right.

    Ashley: But you know what? If you don’t do it, it’s really not fun doing it at the end, right?

    Sanker: So, maybe you remember, because it certainly rattled my world—Dr. Forsgren said on stage at DevOps Enterprise Summit 2018, “CAB is useless.” And everybody in the audience laughed and cheered and duh duh duh duh duh.

    Ashley: You’re not talking about Cabernet, you mean change advisory board.

    Sanker: No. [Laughter] I bet you probably would enjoy a good Cabernet, and I’d—

    Ashley: CABs, yeah. Those others are still good, but anyway.

    Sanker: Merlot, et cetera, they’re all good. But the thing is, she’s not wrong. But she was looking at it from a very specific lens, and that lens was about failed changes—in other words, the success of changes and the recoverability of changes, two things that are really important to the stability of the business. And what she found in the data, she’s a data expert, she has a Ph.D. in this, so there’s no mistake, here.

    She’s absolutely right—however, CABs frequently are part of the compliance equation. In other words, it’s useless for those things, but it has a use elsewhere in the organization, and for many organizations, that’s a very scary thought. Because that’s where we’re able to document and demonstrate for internal and external auditors and validators—are we doing the things that we’re required to do? PCI DSS, HIPAA, GDPR—you know, all of those other compliance framework, NIST 800-53, which hopefully we’ll get to in a minute, but they all have requirements around managing changes within your organization. So, if we don’t do it in the CAB, how do we do it? And that’s precisely the point.

    So, what I’ve sought to do is help organizations build those compliance points into the pipeline itself. In other words, we can point directly to where that’s being done in the pipeline and don’t require a “let’s slow down and actually check the box, here.”

    You know, the governance is—well, can I just say this? There’s a difference between governance and compliance. They’re two very different topics, right?

    Ashley: Yeah.

    Sanker: So, governance is about how organizations make decisions, and making sure that those decisions that are made throughout the organization align with or help fulfill the purpose of that organization, whereas compliance really has to do with the conformance. Are we conforming to applicable standards or expectations, either that the organization has or we have externally?

    So, but when we talk about governance and compliance, we think about performance and conformance. I’m sure you’ve heard that—COBIT and lots of other frameworks have talked about that for years. But performance is about, you know, your strategy and are you utilizing your resources in pursuit, or are you producing value or value creation? And I think that’s fascinating.

    I’ll bet you’ve read or heard of Mark Schwartz’ new book, “The Art of Business Value.” It’s a fascinating read.

    Ashley: I actually haven’t read that one. I’ll have to check it out.

    Sanker: I would put it on your reading list, and I know that sometimes he rubs people the wrong way. I found out that he rubs me exactly the right way, because he asks some really challenging questions, like, “What is business value?” You’d think that’s an easy question to answer, right? It’s like, you know, for a for-profit corporation, it’s like, you know, making your market windows, making your profitability, adding features that attract and retain customers, those kinds of things.

    But what if you’re in—as I was as a CIO—what if you’re in a government agency? That’s not a for-profit organization, so what is business value? Delivery of services to citizens, upholding environmental standards? There’s a lot of things that governments do, but they’re not profit motive. In fact, in some cases, it’s not even financial, so what is business value?

    And so, when we’re running our backlog, you know, we’re scoring our user stories, like, how do we score that in that kinda context? And then, just when he, when I thought that he had confused me enough, he asked, “Well, what if your business strategy”—and I don’t wanna give away his book. Go buy his book. But—

    Ashley: No, just tell me the last chapter, that’s all I need.

    Sanker: Exactly. [Laughter] I actually sent him an e-mail. He’s a great guy and he thinks very, very big. But one of the things that he said was, “Well, what if your strategy is, we want to move customers off of this platform and onto one that’s more profitable to us, or one that’s in alignment with our roadmap and our strategy?”

    So, if we’re down in the value streams or in the pipelines, scoring our user stories against customer satisfaction, customer value, things that customers really, really demand and want, we’re actually creating negative value—these are my words, not his. We’re creating negative value because we’re making our customers like the platform that we want to get them off of.

    Ashley: Mm-hmm.

    Sanker: So, business value would actually be—no, do things that make them less likely to like this product. Make them want to migrate. Make it easy for them to say, “You know, this is really getting old. I’m gonna try your news service.” And so—

    Ashley: Right, that behavior change.

    Sanker: Yeah, exactly. And so, when we really start thinking about, we’re getting, you know, your governance, especially your governance and your compliance, we have to have adaptive, rapidly adaptive systems that allow us to, at any point in time, understand what the current strategy is and how it affects the decision making that we’re doing at any point in time—all without, of course, getting your board of directors involved with every single decision throughout the organization.

    Ashley: Well, let me ask you this. I’m really interested, I’m really curious about how you take this kinda shift in mindset and how you’ve woven that into the “Change Enablement Guide“ that you were the author of. So, you know, it isn’t a process discussion.

    Sanker: No.

    Ashley: You’re talking about viewing things a different way, valuing some things maybe we didn’t value before, maybe some other things less, and maybe delivering a different set of things. Let me just share, briefly. I remember a day, probably not too long ago, where CABs saw their role as saving us from developers and all the bad decisions and lousy code they were gonna create, so that’s what—

    Sanker: [Laughter] Oh, yeah! Absolutely.

    Ashley: – that’s what they say in those CAB meetings.

    Sanker: Yes. Oh, absolutely.

    Ashley: But, you know, that’s not what we’re talking about. Tell me how—so, how do you build that into, in a way that’s gonna help people make that shift through ITIL4?

    Sanker: So, thank you for saying that, because there is some unspoken or lightly spoken or spoken only in factions kind of frustration, there. It’s like—let’s be honest. I used to be a network engineer, I did network security, I did all things network. So, I used to be a change manager at one point in time, and who the heck are you, Greg, to cast judgment on this piece of code and decide whether that’s gonna work appropriately? Everybody knows that I can’t add value to that, and everybody knows that this external body who’s not familiar with it, certainly not at the pace—you can’t add value, and yet you act like you do. And that just—that doesn’t sit well with anybody.

    So, we really need to, if we shift our focus to, “What is it that we’re trying to achieve?”—so, I talk about putting the right measures on the pipeline. How well is this pipeline producing outcomes, those outcome expectations? Is it producing consistently high quality changes? Is it consistently not delivering failed changes—or, for the failed changes that do go through, how quickly and easily can we recover from that with minimizing impact to the customers?

    Those are the things that your CAB or whatever your body is should care about, not the individual changes. You’ve got to get away from the individual changes. You can’t add value to that.

    And so, I get asked a lot of the times, it’s like, “Well, what do we do with our CAB?” since, you know, they’re very unpopular. And in some cases, the best answer is—now I’m gonna say it publicly, so I’m gonna be quoted on this, but—“Blow it up. Just get rid of it. And don’t ever use the C-A-B word again, because it has such a bad name.”

    But it isn’t without value. So, if you have a CAB and the CAB starts saying, “Look, developers, we can’t add value to what you’re doing, but what we really care about is some of these things. So, let’s talk about those things.” And if you’re talking to, particularly DevOps, agile, automated pipeline folks—they were gonna welcome that conversation because they’ve already got it covered. They just do. You know, like you know, I’m the change manager—“Well, have you tested that code before you released it?” Well, dude, that’s the only way it gets in production, if it passed tests. There’s no way for it to make it into production. So, not only is it insulting, but it’s a dumb question, and undermines your credibility.

    Ashley: Well, let me ask you this, then. I think where you’re going—and tell me if this is right—is, “We have so much automation in this pipeline process, right? That’s the whole idea of being able to produce software more reliably, more rapidly.”

    Sanker: Oh, yeah.

    Ashley: There is so much data that’s generated and captured, but not leveraged.

    Sanker: Yep.

    Ashley: We treat it as data exhaust, right? That’s sort of this fancy term for it.

    Sanker: Yep.

    Ashley: In that is a lot of compliance, is a lot of governance information—

    Sanker: Yeah.

    Ashley: – that you can then elevate and connect, and basically save developers time and not having to produce reports at the end. “I’ve already got three-fourths of what I need or what you need, and let me help you put the finishing touches on it so you’ll sail through the next whatever NIST whatever it is.”

    Sanker: Yeah.  So, there, you get to write a chapter in my next book, because that’s exactly, exactly what we want to focus on is, we want to be—you know, alright, this is potentially controversial, but change management is a feedback loop. It helps us understand the quality of changes that are coming through our pipeline. Sitting there and passing judgment and saying, “Nothing passes until I say so”—that’s not adding value. That’s absolutely antithetical—you know this—to the DevOps continuous flow kinda mindset.

    Ashley: So, they’ll go every way around you you can think possible, you know?

    Sanker: Yeah, and the other thing, as we well know, is like, develop is most frequently used in the flagship products. “This is the core business. Get out of our way. We have to deliver these changes to maintain our profitability, maintain our market windows—all of those things that are core to the business, and you think I’m gonna stop for three days when I went from, you know, a user story to a value in under 24 hours in some cases? There is no way I’m gonna stop. I’m just not going to. I will take the risk of it failing and you saying, ‘You shouldn’t have done that.’” And that’s—

    Ashley: You can talk to the chief product officer about—

    Sanker: Exactly.

    Ashley: – the stock’s direction. [Laughter]

    Sanker: And I helped an organization who was, you know, they had agile coaches and they were doing all that. It’s a large company, but I’d rather not name them. And I was asked to come in and look at their change management.

    Well, what they had done was, rather than having one CAB a week, we were having daily CABs. And we were having daily pre-CABs and then the CABs. And one of the things that I tried to get them to understand is, these two are entirely incompatible. If you’re going towards agile DevOps as a development methodology, then you must move away from CAB. CAB has got to go.

    And what was interesting is, their flagship products, the ones that really made money for the organization, they could get away with murder. Why? Because it was so critical to massive revenue in the case of this organization.

    Ashley: Mm-hmm.

    Sanker: And so, we have a big outage, we recover, we move on, that’s okay, it’s an acceptable risk. And change management will still know, we can’t have any defects or whatever in production.

    And so, we’ve got to get ourselves wrapped around the idea of managing business risk. You know, you mentioned something about the compliance, and just this last week—like I said, I’m working on a new book, but I was looking at the NIST 800-53. Everybody’s familiar with it, but they finally released version 5. And so, I’m reading through and the author’s made a very fascinating statement, and I encourage everybody to go look at this. Version 5 is phenomenal, because what they said was, “We have eliminated or reduced the specification of who actually does the activity, and we’ve instead focused on the outcomes that need to be achieved.” In other words, some of the compliance requirements or the controls said, “IT will review” or, “The change manager will review.” And so, if you’re gonna comply with NIST 800-53 previous versions, there’s not a lot of wiggle room around that.

    But I’ve been saying for over 10 years and others have been saying, it’s like, if we focus on the outcomes, then who’s doing it, how they’re doing it is open for us to do it in the most effective way. And that gets us into the pipeline space. And it’s a major step forward, because we struggle with that. Like you said, we have to meet our compliance, you know? If you’re dealing with sensitive information, you know, pharmacy has HIPAA and credit cards and all that sort of stuff, you have to. It’s just not optional. But you don’t have to do it the way we’ve always done it, and that’s the piece that I like to bring to organizations. It’s like—how are you doing it, and can we demonstrate that we’re doing it? Those are basic audit business controls minimums.

    Ashley: I’m curious about, we’re sort of talking also about the whole issue of agility and speed, and on the other end of it of the belief of the best way for something to be stable is don’t change it.

    Sanker: Oh, yeah, absolutely.

    Ashley: Which is antithetical to the environment we’re in, for sure. And so, it’s kinda like you have to embrace, you just have to say, “We love change, and how do we be super effective at it?” instead of, “How do we not change so we don’t mess it up?”

    Sanker: So, there’s two places, two things that I’m gonna mention briefly, but we won’t have time to explore them. There’s the field of study around complex, you know, and Cynefin is one of the most notable frameworks. And the idea that there’s simplex, complex—or complicated, complex, and chaotic, and we treat changes frequently like they’re either simple or complex. Meaning, we need to get some eyeballs on this thing.

    But the fact of the matter is, most systems, especially systems that we’re DevOps-ing, are more like a complex adaptive system. There’s a level of complexity that we’ve not seen in days of old, like back in the ‘80s and ‘90s and early 2000s where these systems were, we could actually try to understand them. Modern system—

    Ashley: They’re not linear. They’re more, you know, dynamic and self-creating, if you will, in run time.

    Sanker: Yeah, exactly. So, that hit me a number of years back. It’s like, this whole mindset, it’s around the idea of, it either is simple or it should be simple, and we can understand it, and we can predict the outcomes, and you can’t. You cannot do that.

    And so, we need to build adaptive systems that say, “Hey, look, this is complex.” And you’re not gonna send it to a board of examiners, you know, a CAB who are gonna say, “Let’s take a look at all the risk,” because it would take you years to do that in a complex system. That’s just—that’s not gonna happen.

    So, we need to adopt practices that are more adaptive. And one of the things in that complex area within the Cynefin framework says that we need to run some experiments to better understand the mechanism. And how that applies to change management—see, this hit me between the eyes because it was like, “No failures in production. No failures will be allowed”—that’s antithetical to learning how the system works. If you don’t make some changes to see how the thing works in operation, you’re denying yourself learning and you’re probably deluding yourself that you’re actually actively controlling it, right?

    The other piece of study that I’ve brought to the table is—and you’ve probably heard of it—this study of anti-fragile. There’s a number of authors whose understanding is astronomically higher than my IQ, you know, Ph.D.s who actively debate such things. But I take complex things and make ‘em simple. Here’s the thing—some things, like a tree, when the wind is blowing, the tree is waving, but there’s more than that going on. It’s the actually fibers, the plant itself becomes more resilient to wind if it’s been exposed to wind.

    Here in Oregon, often there’s logging operations that have a lot of controls for the environment—trust me on that. It’s changed a lot since the good, ol’ days. But one of the things that we do hear is, we leave a segment of trees at the edge of a piece of property that’s being logged, and we do that for the animals, for the environment, we protect the riparian areas. But one thing that you’ll notice is, within one year of leaving that stand of trees that used to be next to a large strand of trees—they break. A storm comes, and they get hit. Why? Because they’ve relied on the strength of their neighbors up to this point, and they’re not suited to stand the wind alone, and so those trees break.

    Here’s the thing—if we don’t inject some failure into the system, for a system that is truly anti-fragile, we’re actually making our systems more fragile. And so, a mindset of change management that says, “No errors, no problems, no failures ever, ever, ever”—you are making your system more fragile, and you’re treating a complex system like it’s simple. And so, we’ve got to change our mindset.

    Now, you know full well—and let me just blast into this—you know full well that, as we take these mammoth changes that we used to make in the old days, you know, it took 6, 8, 12 months to produce, and we’ve turned it into these microscopic things that can go through your pipeline and ours and produce values is your blast radius. What bad thing could happen in production has materially changed. The risk has become very, very small. If that—

    Ashley: And your ability to fix it—

    Sanker: And your ability to fix it skyrockets.

    Ashley: – your responsiveness, it isn’t six months to fix it or three, it’s three minutes.

    Sanker: Yeah. So, one of the change outcome expectations that I encourage people to establish is, what are your expectations for recovery from any given change, and can we realistically do that?

    So, for instance, if I put something in my flagship product that doesn’t work as desired, if I can recover within two minutes, no harm, no foul, is that okay? In a lot of cases, it is. If it’s a defibrillator keeping somebody alive—probably not, that’s probably not okay. But in a financial transaction or a video streaming service or whatever, you know, modern digital services, it’s probably okay, especially if you’ve engineered the release in such a way that you do it like that. You know, blue/red switches, on/off—all of that kind of stuff that we’re engineering in become, you start seeing why we’re able to do that and what that actually does in production. It’s wonderful.

    Sorry, you got me excited about that.

    Ashley: Yeah, boy, I could take you a lot of places with that. [Laughter]

    Sanker: Yeah.

    Ashley: So, let’s take it back to someone who’s gonna pick up a digital version or a printed version of the “Change Enablement Guide” author.

    Sanker: Yeah.

    Ashley: How is that approachable for folks? Let me be honest. I’m probably not gonna go download that to get my heart pumping to go work out, get a better workout—or maybe I will.

    Sanker: [Laughter]

    Ashley: Maybe after listening to this, I will—but how do you make that engaging for people?

    Sanker: Yeah. Well, so, one of the things—well, I’ll put it this way. It’s like, we’re not selling a product, we’re selling an opportunity to solve some problems. That’s why we call it best practice, right? And there’s a lot of best practices, and I don’t feel like they’re nearly at odds as maybe some of the industry might feel like.

    But change enablement is—and I’ve been saying for years that this day has come and it’s now here is, change management is in the crosshairs of modern digital organizations. It’s a pesky remnant of days past. And what the “Change Enablement Guide” is—and there are was more. I mean, for production reasons, I had to kind of limit it. But what I wanted to do was point in this direction of what I’m talking about. It’s like—let’s get people moving towards that. Because like you said, we’re not gonna change overnight. I’m not gonna read the “Change Enablement Guide” and go back to work the next day and say, “Alright, I can save all the world’s problems.”

    And so, what I’ve subsequently started doing is writing guidance around, “But what do I do today? You know, I went to a DevOps conference, I’m excited, but what do I do when I go back to work? How do I start moving in that direction?” And blowing up the CAB, while we might all celebrate that, that’s not a good first step. Instead, what we need to do is start moving ourselves in a better direction. It’s not more of same, it’s—what do we actually need to be doing here, and what’s the best way to accomplish that?

    And it’s difficult, culturally, because it becomes more of a culture. Like you said, there are CABs galore who despise developers probably as much as developers despise CAB. It’s a match made in—

    Ashley: Cats and dogs, living together.

    Sanker: – yeah, exactly. [Laughter] Good reference. And so, that group of people who are widely known for having that kind of mindset are not gonna be welcome when they show up to your spring and say, “Hey, we’d really like to be involved earlier in your change management so that we can help you.” It’s like—yeah, sorry, bud. Let’s start somewhere else.

    So, if change management starts with, “We’ve gotta get out of your way. We’re gonna start delegating more and more of your changes into your value stream, and instead, we’re gonna be focusing on, we’re gonna partner with you on, let’s look at the results of what you’re doing. Because we believe that you are optimizing your delivery and we believe that you have the right tools and controls in place.” So, we are moving more towards, “Are we achieving all the outcomes that the organization has? Is the organization even clear on what those outcomes really, really are?”

    You know, the one that always—I love this one—“All changes must be tested before going into production.” It’s like—well, how many shops have you been in where that’s not true, at least in modern times?

    Ashley: Mm-hmm, yeah. I call that the “meh” policy. “We’re gonna test it? Meh.” No, of course we’re gonna test it.

    Sanker: Yeah, we call it blow testing, right? You know, it’s like, “Eh, okay, I think we’re alright.” That is true. In modern terms, though? For flagship products, for things that are really, really core to the business, most organizations have some form of automated testing, or that’s just part of the process. So, rather than make everybody say, “Did ya test it?” at a CAB, just, like, “Hey”—

    Ashley: “Of course you did.”

    Sanker: Yeah, exactly. Is that happening in the value stream? If it is, I’m not—don’t worry about it. We’ll just move on. Is somebody with the appropriate authority approving that? And that’s—I am gonna take a minute to talk about this one, Mitch, is that everybody talks about change approval in terms of who’s gonna approve this idea going into production, this change going into production? Because if something goes bad, we wanna know who to blame, right? In fact, we used to say, “Who has enough—who is sufficiently authority that they’ll get fired if something goes bad?” Which is, of course, the wrong question.

    Ashley: Yeah, [Cross talk].

    Sanker: Because the right question is, “Who has enough authority that, if it goes bad, they don’t get fired, because they have the authority to put on that level of business risk?” Right?

    But if we move—and we could talk for hours, but the idea of product ownership or taking a product view of services or modern systems is, “Isn’t it true that the product owner is accountable to the organization to decide what things we’re going to move into production?” So, to me, that’s the change approval. We’re approving that we want this change to happen.

    So, nobody’s adding value by making approval at the end of the chain. We’ve already decided that. We already know that we’re gonna make this change.

    So, those are two different approvals. One of them is about a product decision or a governance decision, the other is about a business risk decision of, “Is it actually going to achieve the results that we set out to do?” I believe that most of the latter can be automated. If it’s passed its test, if the testing systems are appropriate, if we’ve done—in some cases, I love peer reviews. It’s like, you know, the code needs to be reviewed by a peer, stamp it, and we ship it out. It happens in a minute or two, you know, not hours or days or whatnot like that.

    So, I think as we kind of retool or thinking around it, I think that’s where we can start really solving actual problems that organizations have.

    I am reminded, though, I saw this—you’ve seen these before, these overhead views of an Indy car or a Formula One car coming in for a pit stop and all this stuff happens and off they go. I saw one this week that was 1.8 seconds. It was awesome, right? And people like thus always share those on social media of, like, “That’s an agile organization, doing the things” and blah blah blah.

    And so, you can’t really argue with 1.8 seconds—four tires, top off the gas, and out you go.

    Ashley: Impressive.

    Sanker: But the question, to me, is—like, I’m a bit irreverent. Why isn’t it zero? Why can’t we make it zero? If faster is better, let’s make it zero. And the answer is, we won’t make it to the end of the race unless we have fuel. We won’t make it, because [Laughter] when you’re racing at 200 miles an hour around corners, your tires literally don’t last that long. It’s not a matter of preference, it’s a matter of life and death, right? So, we have to change tires.

    So, the question isn’t, “Are we balancing speed and agility against risk mitigation?” It’s what it is that we’re trying to accomplish here. And in the case of Formula One, we’re trying to win the race. And to do that, we have to take on certain activities that cause delay—that’s a requirement.

    So, the question isn’t, “Do we have to take a pit stop?” We do. We need tires and we need fuel and sometimes minor adjustments. So, the question then becomes, “How quickly can we achieve those outcomes?” And putting four tires on in under two seconds—that is phenomenal. Hats off to those folks that are doing that. And how do you get however many gallons of fuel we put in a Formula One car in two seconds, I don’t know, but that’s amazing, right?

    And I feel like that’s the mindset that we need to adopt. We need to quit asking, “Why do we have to do change management?” We have to do change management because part of it is to be compliant. That keeps us in business. That keeps us out of trouble.

    Ashley: But it’s a really good example, because it’s not an absolute, it’s not, “Do we need to fuel?” but, “What’s just enough fuel for us to win?” right?

    Sanker: Exactly. Oh, and trust me, you’ve watched it. I know, I can see it in your eye, you watch Formula One or Indy and there’s like a thousand computers over there and stopwatches and gauges and dials and they are calculating exactly how much fuel based on the game strategy or the race strategy, the conditions, who they’re up against, what they think their strategy is—that’s how much fuel we need to take on, right? So, it’s—

    Ashley: Great model for us to think about of, you know, it’s not absolute quality or absolute whatever.

    Sanker: Right.

    Ashley: It’s what’s appropriate for the organization. We’re gonna have to leave it at there.

    Sanker: Okay.

    Ashley: And I want you to come back some time, because the next topic is Stevie Ray Vaughan or Eric Clapton. But I’m gonna—

    Sanker: Oh!

    Ashley: You can’t answer it. You have to come back and answer that, so we’ll have you here another time. It’s been a real pleasure talking to you, Greg Sanker, who is the author of the “Change Enablement Guide” of ITIL4. So, I think you really gave us some compelling reasons to go check it out. So, thanks a lot—it was great talking with you.

    Sanker: Thanks for having me, Mitch.

    Ashley: You bet.

  • With DevOps Culture, It’s Mind Over Matter

    With DevOps Culture, It’s Mind Over Matter

    There is quite a bit of conversation about tools versus culture, and sometimes, those constantly involved in the back-and-forth forget that not everyone is so steeped in the finer points of the conversation.

    Tools is the easy part. There are a ton of tools available to optimize IT operations, from small bespoke products such as the Antsle Private Cloud Server to CA’s broad suite of Dev/Test/DevOps/ARA tools. There are plenty of places to find lists or charts of tools, and I’ll leave you to them, just mentioning the two I’ve written about in the not-too-distant past: Xebia Labs’ Periodic Table of DevOps, and CA/Automic’s Continuous Delivery Landscape.

    My youngest just built a Cube-solving robot, so this picture inspired me.

    But what of culture? When pundits and old hands talk of culture, what exactly are they talking about?

    Well, it still varies a bit from organization to organization, but there are some definite underlying foundations.

    Increased Cross-Team Communications

    Even if an org is full-on DevOps, there is still the requirement of cross-team communications to make sure everyone understands what is going on.

    One of the things that this does not make abundantly clear, I will belabor a bit (sorry, old hands). Communications is not just talking. It is not just email exchanges, it is not just sitting in another team’s meetings while researching your current problem on your phone.

    It can be these things, but it is a breadth of options. People are using Slack channels to keep everyone updated, common notifications or even pub-sub systems to make certain information about the state of IT, operations and builds are getting wide distribution. Increase communication, not chatter.

    Organizational Change

    You hear that a lot, and it can mean complete overhaul of an IT organization, but in an enterprise setting, it rarely does. The reason is simple. There may only be one or two Cisco specialists, or one or two EMC specialists, and these people don’t properly belong on one or two product/project teams in the long term. The same goes for security and a whole further swath of operations team members.

    So in the enterprise, what “organizational change” often means is moving those responsible for development and core deployment closer together on small, product- or portfolio-based teams. Networking staff, security staff and infrastructure staff will join the teams as needed, but are not part of the core team.

    Management Change

    This is another often spoken-of mantra that easily can be misunderstood. In short, managing a small, agile, entire product lifecycle team is not the same as managing a pool of resources that pop from project to project and are either (a) not involved in the operations portion at all, or (b) not involved in the development portion at all.

    For a great set of perspectives on this topic, I’ll recommend our partner The DevOps Institute. It has an eBook and a certification course (DevOps Leader) on this topic.

    Decentralization, Yet Increased Oversight

    This is often a confusing pair of messages we send when talking about what DevOps culture change does. It really is not that complex—pundits just assume everyone already knows, and, honestly, those knee-deep in DevOps do know.

    Decentralization, as taught in the DevOps Leader course mentioned above, is about driving as much decision-making as possible down to the team. When teams are built around a product or group of products, then those with the most pertinent information about what is best for the product quite often can be made within the team, speeding decision-making and implementation.

    Increased oversight is an acknowledgement of two things. First, there is more information available in an agile environment, and that information is available more frequently. Standup meetings offer some, while milestones offer other data. Second, tools are starting to allow rollup of information in a uniform fashion for CIOs, CTOs and business leaders to look at. This means they can quickly see the status of projects they have a vested interest in and provide feedback into the process. To go from, “Oh, this roadmap item is going to take a lot more work than we thought,” to, “Great, we have a plan, let’s keep moving,” is shorter when there is more information available to the decision-makers on both the business side and the IT side.

    There is no doubt plenty I have missed, but these seem to be the core things that those of us discussing this topic a lot assume others know, and that those just looking into DevOps really don’t. Hopefully it will help someone push through the clutter and come to a decision on DevOps in general, or culture changes in particular.

    — Don Macvittie

  • Overcoming the People, Culture Barriers to DevOps Adoption

    Overcoming the People, Culture Barriers to DevOps Adoption

    The fact you’re reading this article on devopsy.kinsta.cloud suggests you’re already a DevOps fan. Or maybe you’re visiting this site because you’re considering moving to DevOps in your organization, or you’ve been told you must move to DevOps and are figuring out how on earth to do it.

    Whatever your view on DevOps, let’s get my position clear from the outset: I am a big believer in the power of DevOps to deliver continual innovation to an organization. I advocate DevOps in every project I do because I believe those organizations that innovate and change the fastest become the leaders in their sector, and since almost every business innovation relies on a technology innovation first (“every company is becoming a technology company,” after all), DevOps is the best way for an organization to adapt to, and benefit from, new technologies and gain a lead in the market.

    However, as anyone who has managed the transition from waterfall to DevOps will know—or, in fact, anyone who has overseen any change management project will know—the transition is rarely a smooth one. So, what is behind this reluctance to innovate and work in new ways?

    First, the Technical Arguments

    At some point, most organizations have been burned due to a lack of quality control in technology, whether a security breach, bugs or implementations/changes that went horribly wrong. We all know stories like these. They can have an immediate and catastrophic effect on the organization’s ability to do business. Therefore, there has been a tendency to inhibit change in favor of “stability” and known entities, and to adopt a “Better the devil you know,” strategy.

    DevOps has rocked this view of the world, and teams are reticent to move away from tried and tested approaches and processes because they are held responsible for operational stability. In addition, those people in a quality-control role believe their very purpose and job is under threat. There is also a view that DevOps is system- or application-centric, and can introduce risk to an interconnected IT estate.

    While these are all valid concerns, it is worth bearing in mind they are all technical arguments against DevOps, and they have all been countered before. Implemented correctly, DevOps does not jeopardize stability or undermine quality (in fact, many would argue it does the exact opposite), and it does not undermine a person’s role in development or operations, but rather liberates them from their silos.

    Culture is the Real Reason

    From my experience, innovation of any kind cannot be held back by technicalities alone. While the technical arguments are certainly valid, in most instances, the reluctance to adopt DevOps comes from something much more basic: the organizational culture borne from natural human behavior. Most people are simply risk-averse and resistant to change, so the resistance to DevOps is just a manifestation of human nature.

    The only way for DevOps to succeed is to change the organizational culture to be more open to change and to adopt an “innovation mindset.” The change must also encompass the whole organization. If you only bring half of the business with you, the innovators will become frustrated and find ways to circumvent processes and policy, resulting in additional operational risk and shadow IT.

    The influence that senior managers can have in facilitating—or undermining—the move to an innovation mindset must also be factored in. It is common for senior management to inadvertently foster a hero culture, where those individuals who get things back on track after a catastrophic event are rewarded, yet the well-structured and “stable” DevOps deployments are not given the same weight or focus. Again, it is human nature to be drawn to the dramatic event than the non-event. But you must ask yourself: As exciting as firefighting is, is it a sustainable way to run a business?

    Instilling a new mindset into a business is the same as any other transformation project. It requires you to tackle the four elements of process, organization, technology and information to bring about meaningful and sustainable change. To bring DevOps into your organization you must:

    • Align operational teams with development teams, embedding where possible for early involvement of Ops.
    • Ensure that operational/non-functional requirements have the same weight and priority as functional requirements—no user story is complete without a stable and secure operational environment.
    • Automate where possible to reduce human error and increase the speed of delivery.
    • Improve knowledge management to reduce “hero culture” or bottlenecks where knowledge is concentrated across a few people or one team. Never make exceptions for “heroes”; you need their support more than anyone else because they hold the biggest sway over the rest of the team.
    • Have accurate and comprehensive configuration information. Having a good understanding of the IT estate (including interconnectivity) can help move an organization from a one-size-fits-all governance model to a tailored risk assessment approach through better understanding of the impact of changes.

    DevOps often fails when people can revert to their default behavior. Only by actively identifying and supporting a project in a DevOps manner (considering process, organization, technology and information as previously explained) will you be able to change to a new “business-as-usual” innovation mindset and behavior. But this needs strong sponsorship, guidance and patience.

    DevOps must become the culture of the organization, driven from the top as well as the engineering teams, and people need to understand the implications for their role and responsibilities. CEOs and leaders need to actively support and drive the change, while providing the necessary working environments (tools and training) for those people who will be held accountable for operational stability. It is no good investing in technology and training for engineering to innovate if your ITSM functions are not brought along with the change, for example. But if you bring everyone with you, and address all concerns along the way, you will soon be reaping the rewards of the DevOps approach.

    About the Author / Romy Hughes

    Romy Hughes is director at Brightman. Romy is a business change specialist with expertise in service integration and management (SIAM) for managed services, business transformation and IT service management (ITSM and ITIL). Connect with her on LinkedIn.

  • The Conundrum of Adopting DevOps within Organizational Constraints

    The Conundrum of Adopting DevOps within Organizational Constraints

    I wish I had a dollar for each time I’ve now heard, “we’re adopting DevOps, but we have to operate within the constraints of what the business is ready to accept.” Basically, this is becoming the battle cry of many enterprise IT professionals that are DevOps believers, but are stuck with the limitations of operating within a complex, political and immovable business structure.

    One on hand, it is very pragmatic to approach adoption of new methods, techniques, technologies, etc. on a small scale and then expand based upon success. However, what is the result of a DevOps initiative in which existing organizational constraints and bottlenecks continue to remain in place?

    If you’ve been in the business world for more than a few years you’ve probably already had the experience of having a really good idea only to come to the realization that implementation of that idea requires cooperation of another organization. For example, Marketing might have a great concept for how to acquire new customers, but would require sales management to provide incentives to the sales team for it to succeed. For purposes of this example, let’s assume that marketing has made its case to sales and sales rejects the proposal (many times there is a perception that the idea will be rejected limiting those from even trying; this is a worst-case scenario for any business). In this case, Marketing might attempt to operate around the need for the sales incentive and, ultimately, come to the realization to abort the plan or lower expectations for results.

    Is anything more debilitating to morale then end up with these two outcomes as the only choices? What would you do if you were in the position of driving this for Marketing?

    Often times in businesses there is no grievance committee. To steal a line from “A League Of Their Own”, “there’s no crying in business!” What DevOps is surfacing is that business is about relationships and managing cross-departmental and extra-departmental relationships is not something that many IT professionals have had to or want to foster. Ultimately, this will have a detrimental impact on DevOps adoption.

    In most cases, DevOps requires that some leader within IT steps up with the idea to foster a frictionless ideation-to-operation lifecycle. By frictionless I mean the removal of as many bottlenecks and constraints for transitioning from the idea phase to a production-ready support application. Like Marketing in my earlier example, this may require this leader to “sell” their idea to the heads of IT Operations, Application Development, Quality Assurance, Information Security, and potentially C-level executives. A “No” response by any of these teams will result in the leader working around this limitation resulting in no change for the processes and work output of that particular group or attempting to do what they can within the organizational boundaries within their control.

    Hence, business agility is tied to velocity of accepting change, which is further dependent upon individuals fostering a strong enough personal relationship–at a minimum one of mutual respect–to be willing to accept risk in support of a larger goal. I believe many would see this as a high-risk constraint on the business agility deliverable.

    One way to increase likelihood of avoiding this trap? DevOps needs to be sponsored from C-level management from the start inclusive of the necessary incentives required to help drive the changes that will lead to more rapid adoption of DevOps practices and speed the way to business agility. There’s enough evidence now available to convince most C-level executives that a DevOps program is a worthy investment in moving their IT organization toward greater alignment with their business goals, producing faster, reducing operating overhead and improving quality.

    As with most changes within IT, and DevOps adoption is no different, there is a requirement for bottom up support as those closest to the work need to believe, and there is a top-down requirement that provides the coverage, financing. motivation and accountability to the business to allow the change to occur.

  • Time-Sharing to Unicorns: The DevOps Problem Facing Enterprise IT

    Time-Sharing to Unicorns: The DevOps Problem Facing Enterprise IT

    How did we in Enterprise IT get to this point? There’s a lot of talking and writing about the value of adopting DevOps but very little analyzing the problem domain we’re trying to correct. There are some schools that will say we can move forward without looking back, but I believe a key element of the dysfunction that faces many enterprise IT organizations can directly be attributed to their history and how they were formed.

    One of the clear lessons we can learn from businesses that we call “unicorns”–termed this because of their mythical capabilities as an IT organization that can deliver high quality software very quickly–is that they have very little friction in the process from ideation to operation. Startups and greenfield initiatives are practically operating in a vacuum relative to the amount of friction they encounter compared to operations around entrenched decade-old legacy mission-critical applications. In retrospect, this should not be surprising given frictionless delivery is a mandate for many of these efforts and has been from the start.

    You create what you focus on!

    Some will say we don’t require the capability to deliver to production ten times a day (or even a year) for our SAP Enterprise Resource Planning (ERP) system. So they will conclude that perhaps DevOps isn’t right for our needs. This shows the skewed thinking that current literature on DevOps is fostering. It is not the need to deploy changes quickly into production that is at issue, but the ability to deliver business capabilities when they are required as quickly and with little impact as possible. Current processes around changes to the ERP are usually so fraught with friction that when the time comes to fulfill a need for the business so that the business can continue to compete effectively, it takes weeks or months instead of days or hours.

    I researched how the division between development and operations was formed and interestingly, there’s very little history about this division of work. We can trace this division back to the early days of time-sharing systems in which the division of work actually represented the exact advice DevOps prescribes today. Deliver and operate a platform on which developers can build and deliver their applications. In the days of time-sharing systems, operations was not responsible for what developers put into the box, just ensuring that the box was available, secure, and backed up. If developers applications had bugs or crashed, it was the developers that were responsible for repair.

    Then came the era of client/server. Application pieces ended up distributed across application infrastructure like a kid that dumped their Legos all over the floor. Managing these applications became as painful as stepping barefoot on a Lego as well. Developers didn’t work on the complete application, but instead worked on modules and these modules were split across various services each requiring their own levels of expertise to keep running in an optimal manner. The DBA was responsible for ensuring the database could meet transactional requirements and support data design and management, the network professionals dealt with protocols and traffic issues caused by these modules communicating with one another. Each layer of the application had it’s own capacity management and operational concerns. The only chance we had to keep the ship from sinking was to put it all in the hands of the one team that had access to all the components — IT Operations.

    Things have not progressed much since this divergence away from time-sharing to client/server either. Today’s applications are more complex and more distributed than ever before. We have clusters of the same server working in unison to handle scalability exponentially increasing the complexity and difficulty for operations. A single application may now require anywhere from 5 to 25 servers to deliver the required capacity.

    On top of all this we’ve never modified the division of work from the days of time-sharing and continue to push the burden for monitoring and uptime to the operations staff. We have turned IT operations into the Jigsawyer’s–someone who does jigsaw puzzles–and developers into the puzzle makers. Moreover, the puzzle is never the same; the puzzle makers change the picture all the time as they add new piece forcing the Jigsawyers to figure out how to retrofit the new piece into the whole.

    While researching the history of IT operations and trying to figure out the path from computing’s earliest organization until now, I came across a great piece that really helps identify the problem we have created. So What Is A Deployment Really? by Robert van Loghem illustrates what it takes to move from ideation to operation with today’s application platforms. What I really like about this piece is the illustration that even if we want to go back to earliest division of work, it may not be possible. IT operations cannot simply provide a black box for developers to deliver their applications on and expect them to work as planned. Firewalls need updating, web servers need configuration changes to support the new domain requests and the security architecture must now allow these new requests to be routed appropriately based on rights and source.

    Deployment is a team activity!

    So, what have most of us in enterprise IT done? We broke down the tasks of deployment further into divisions of work so that we can apply the necessary level of specialization to ensure the application can be delivered to the end user at the cost of adding a ton of friction into the process slowing things down by a logarithmic factor when compared to our unicorn friends.

    What is surprising is the number of pundits that put forth that the answer to this problem domain is empathy and culture change. Clearly, we have organizationally written ourselves into a corner because this is how large enterprises solve problems. Here’s a simplification of how this decision was most likely made:

    Question: There’s a lot of issues attempting to monitor and operate these distributed applications and the developers don’t have the means or transparency into all the components.

    Retort: Then who does have access and insight into all these components?

    Answer: IT operations can see what’s running and has the tools to monitor the health, let’s give them the task.

    At a time when IT was viewed as an expense versus a strategic advantage, there was no offsite planning to assess what the impact to this answer means to the business ten or twenty years hence. It’s solve the problem and keep moving because that’s what enterprises need to do in non-strategic areas. At the time this answer was formulated most businesses equated this decision with “which office supply vendor should we use?” Moreover, the leaders in client/server barely had any experience with IT operations, they were primarily comprised of application developers. Hence, there was no experience to guide businesses away from this decision. IBM, Unisys, Honeywell, etc. the vendors that were a prime source of guidance at the time were still leading with a time-sharing mentality.

    This entire trip down memory lane was with one purpose in mind, figure out what we created and how it was created so we can consider the means to unwind it and provide a path for enterprise IT to move forward in an agile manner. The history provides the answer. We need to move away from the traditional time-sharing division of work that was formulated before the age of the Internet and realign along frictionless surfaces. What does that mean? Well, that, my friend, is an entire book that happens to be in the works! 😉

  • Enterprise DevOps: organizational change a must

    Enterprise DevOps: organizational change a must

    There has been a lot of discussion in the blogosphere, including here at devopsy.kinsta.cloud, lately about the challenges of adopting DevOps in the Enterprise space. These challenges at the end boil down to the need for organization change. There are organizational structures that appear to hinder DevOps adoption in large enterprises, that are inherent to enterprises and smaller companies do not have. In enterprises different functional teams – Dev, Ops, Security, QA, etc. may be under totally different management chains. It is not uncommon to have to go up thru several layers of management to reach a common senior executive. In other cases, there may be an entire software supply chain in place that is made up of not just internal but also outsourced or external suppliers. These suppliers may or may not ‘play ball’ when it comes to the changes in processes required to adopt DevOps. Especially if they are bound by outsourcing contracts that do not make change possible or profitable for them.

    DevOps as a philosophy has had as its centerpiece the principle that all the application delivery teams (Dev, QA, Ops, Business Analysts, Architects, Security…) need to align better and collaborate towards the common goal of delivering applications, in a efficient and constantly improving manner. This is a people and organizational principle, not just a process centric principle. To me adopting this is more important when adopting DevOps than any process or tool.

    This post discusses key guiding principles on which this DevOps driven alignment of different teams in an enterprise should be based. They are designed to improve collaboration between these teams and to break down the proverbial ‘wall’; to end the water-SCRUM-fall, when it comes to the relationships between the different application delivery stakeholder teams in the enterprise. This is not an easy effort. It depends on the existing culture at the organization, the presence of outsourced suppliers, politics, contracts and above all, the desire to change.

    These guiding principles are:

    1. Shift Left:

    Organizationally the goals of DevOps are to bring all the teams involved in application delivery together closer. Not just at deployment time, as they may do already, but all thru the delivery cycle. It requires Ops to allow Developers to take more responsibility of the operational characteristics of the applications they are building, for example. In turn, it requires Developers to keep Operational considerations in mind while building their applications. It requires analysts and architects to think holistically and work with Dev and Ops teams as they design and enhance the application. This is referred to in the DevOps world as ‘Shift Left’. As in shifting towards the left, Ops responsibilities. Shifting it to earlier in the software delivery lifecycle, towards Dev. It is shifting left the testing of your software to as early in the lifecycle as possible. It is shifting left the point at which collaboration between teams begins. It is shifting left the responsibility to deliver code to production – way left. Can someone in finance be held responsible for code not being delivered on time because they delayed approving a budget line item?

    2. Collaborate across all teams:

    Different functional teams typically use different tools to manage their projects and activities, for change management and (outside of email) different collaboration tools. In order to better collaborate across the organizational boundaries, all teams should start using the same Change Management and Work Item Management tools or in the worst case, use tools that are integrated. This allows seamless visibility across tools and traceability between respective Change Requests. Real-time collaboration using a common tool is obviously the best scenario.

    3. Build ‘Application Aware’ Environments:

    As we move towards Software Defined Environments, we have the ability to build, version and manage complex environments, all as code. All of the benefits of this are moot if the environments are not a perfect fit for the applications and more importantly the changes to the applications being delivered. The goal is hence to build Environments that are ‘Application Aware’ or are fine-tuned for the applications they are designed to run. No more cookie cutter Virtual images for all kinds of applications. More importantly, one needs to ensure that the environments are architected in a manner to allow for the evolution of the applications, both as they are developed; as they are projected to change or as they may evolve in the future. This obviously, would require close collaboration between Dev and Ops. Development Architects and Product Managers need to work with the IT Architects and Operations Managers to architect environments and their projected evolution to align with that of the application. Not only that, but also allow enough resilience in the environments to allow for unexpected change. For example, massive change caused by a super successful App. Think Instagram and how the founders had to keep changing their server environment almost daily as millions joined their service.

    4. Environment Sprints:

    Agile Development Principles prescribe that at the end of every Sprint, developers have a build. An executable build that runs, even if does not do much in terms of functionality. The next evolution of this concept is to extend it to environments. This would mean that at the end of each Sprint, the Dev team would have an executable AND the Ops team would have an environment, a production-like environment ready that the executable can be deployed on. This would allow the build to be tested. That too in a production-like environment. This would also provide immediate feedback to the Ops teams on the behavior of the application in their environment and use that feedback to improve the environment. This requires the Ops and Dev teams to align better. They will both have to use a single Sprint plan for releases – of their Builds and Environments respectively and will provide feedback to both at the end of every Sprint, using which Dev can enhance their Apps to improve functionality and performance and Ops can enhance the environment for the same. Leveraging full stack deployment – deploying at each instance the full stack that includes the environment, middleware and the application being deployed – could very well be called the ‘holy grail’ of DevOps. (Brief commercial break – IBM just announced the new offering UrbanCode Deploy with Patterns that provides full stack deployment leveraging OpenStack HOT patterns).

    The Human factor

    These principles are designed to foster Application Delivery team alignment from a teaming perspective, from a people perspective. These however do not replace the need for good old team building. Whether it is thru face to face get togethers or in todays distributed worlds, thru regular virtual meetings and online collaboration in real time. The best teams are the ones where the people know each other, trust each other, look out for each other and when there are challenges, know who to pick up the phone (or launch Skype) and talk to.

  • Step One Of DevOps Ignores Technology

    Step One Of DevOps Ignores Technology

    I’ve spent a bunch of time talking with Adrian Cockcroft, former cloud tzar at Neflix, and listening to his various talks and presentations. I’ve also spent a bunch of time looking around the various rooms within which he presents and seen the dazed look of attendees who have rightly assumed that just seen a picture of the way organizations should be. Netflix is unquestionably an exemplar for IT reinvented – where innovation, agility and developer enablement are natural parts of what they do.

    But then I’ve seen these new disciples of the “Netflix way” write a list of the technologies that Cockcroft detailed and attempt to shoehorn those same approaches into their own organization. All to often the result is disappointment, failed projects and a swing back to the traditional rigid ways of doing things.

    These thoughts struck me then I hear about this new masthead and its aim of creating a hub for DevLops thought leaders – surely giving IT practitioners a good place to find the latest tools and techniques will help spread the DevOps gospel?

    Well yes, and no. You see I’m a firm believer in the viewpoint that DevOps has very little to do with products, processes and policies, and instead is a direct result of a particular type of culture. On the one hand you have Netflix, clearly a cultural outlier with its openness, willingness to experiment, embracing of failure-driven iteration and aim of planning for failure. On the other hand you have traditional IT shops that are top-down, driven by a rigid policy framework and consider the “operations department” and the “developer team” as two distinct entities.

    This was a topic that Christian Reilly (he of the sartorial elegance and funny accent tendencies) and I discussed a couple of years ago, before the DevOps hype had really started to hit. In the video, he reflected directly on his organization, Bechtel, as an outlier at one end of the spectrum, while Netflix is an outlier at the other. It’s a somewhat dated video, and talks about public/private cloud, but the themes in terms of organizational readiness hold true for DevOps as well.

    [youtube http://www.youtube.com/watch?v=gVG1Pj4cW-g&hd=1]

    So my aim in my regularly irregular DevOps column in here is to talk about everything but the technology. And where better to start that the first step towards DevOps which is, I believe, all about building cultural bridges rather than making technology choices. It should be a natural concluding that, since DevOps is all about bridging the developer/operations span, that perhaps getting people in a room to talk about their particular drivers and metrics would be a good idea. But alas most people go from talking about DevOps to deploying their choice of automation tool.

    There’s so many things to think about in embarking on this process – over on the ScriptRock blog Alan points out that visibility is a key first step before automation is embarked upon. While I agree with him that visibility is key, for me the money quote comes in his post where he writes:

    Don’t automate what you don’t understand

    But to then suggest that the first step should instead be a visibility tool does little to improve the outcomes – unless both sides fo the organizational divide are in the same room and accepting a common lingua franca and viewpoint on the data – visibility achieves little. So takeaway #1 focus on the culture. There’s lots of ways to build the bridges needed to change the way an organization looks – I’m going to cover some of that stuff over time.