devopsy.kinsta.cloud

Category: Enterprise DevOps

  • Change Management in a Fast-Paced DevOps World

    Change Management in a Fast-Paced DevOps World

    While DevOps speeds up software release cycles, change management often slows them down. So, how do you do change management in a fast-paced DevOps environment? Change control enables organizations to meet compliance and governance while managing changes at the speed of business.

    In this interview for TechStrong TV, Greg Sanker, former CIO, IT Operations specialist, author, speaker and contributor to the “ITL4 Change Enablement Guide,” joins us to share his views on the tremendous impact of Agile, DevOps and CI/CD on traditional change control teams. Greg talks about the new role of change control and his contributions to the “ITL4 Change Enablement Guide.”

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript: Change Management in DevOps

    Mitch Ashley: Well, I have the pleasure of being joined today by Greg Sanker who is former CIO, operations specialist, he’s a speaker/author, a bit of a futurist in leadership and digital transformation. He’s also a contributor to the “ITIL4 Change Enablement Guide” and a reviewer of a lot of other content—so, long introduction, there. Welcome, Greg, to [Cross talk].

    Greg Sanker: Yeah, wow—thank you. That’s kind of awesome. Thanks for having me. I appreciate being on the show. I think we’re gonna have—

    Ashley: It’s always—always good to have a fellow musician. Anybody with a Telecaster in their background can be on this any time you want.

    Sanker: Yeah.

    Ashley: So, tell us, [Laughter] in addition to your wonderful guitar sitting behind you, tell us a little bit about yourself.

    Sanker: Well, you mentioned, I am—I specialize in IT Operations. That’s what I’ve done for more decades than I’d maybe like to admit. But service management has been a core part of that, and it’s become really clear to me that my view of service management is a lot more streamlined and agile than a lot of people perceive it as. And so, I’ve begun speaking at conferences and whatnot. I have a book of my own that’s published—I’ll let you guys go find it if you’re interested—about change management.

    But most recently, I was asked to work on the ITIL project on a number of ways, but most notably, the change, what we now call “Enablement Guide.” And one of my prerequisites to get engaged with the project was, if you’re gonna ask me to write it, I’m gonna write it the way I see it, which is very different than what we’ve been doing in the past, and we’re gonna be pushing it forward, so. So, that’s a bit about me.

    Ashley: That’s wonderful. That’s definitely a consistent theme with ITIL4 and other folks, Mark Smalley and others that we’ve talked with is really recognizing—I think ITIL’s had its sort of process moniker, right?

    Sanker: Yeah, pretty much.

    Ashley: But there’s a huge emphasis on, you know, how do you fit ITIL and all of the aspects of operations into something that is more of a dynamic DevOps-y agile kind of world, if that’s a way to describe it. And not to say that it didn’t fit in before, but I think it’s making it more prescriptive in that kind of setting is what a lot of the good work that you and others have done.

    Sanker: You know, I was an early embracer of DevOps, and part of the reason for it is the mindset that explores possibility that ignores the boundaries in an organization to achieve outcomes. That’s, like, music to my ears—I’m all in. So, it’s a match made in heaven, but I can mention, it’s like, there’s a lot of organizations that have adopted service management, they have used ITIL in ways that are very prescriptive, very heavy weighted, very bureaucratic and slowed down. And it was never intended to be that way.

    Ashley: Mm-hmm.

    Sanker: It just is, it got adopted that way, right?

    Ashley: Well, and I think a lot of the work that’s gone into ITIL4 will help make some of that shift.

    Sanker: Yeah.

    Ashley: Certainly, it’s positioned that way, and I know from other authors, that was the mindset of how the contributions were made. Now, you specialize around change management and we were chatting when we first got introduced about, you know, how do you do change management in a rapid development, agile, DevOps kinda world? If you were gonna be ask that at a conference by somebody, how would you—what guidance would you give them?

    Sanker: Wow. So, I’ve got an hour and a half platform, right, for that?

    Ashley: [Laughter] At least an hour and a half compressed into 20 minutes, yes.

    Sanker: Twenty minutes, right? Yeah, absolutely. Well, see—well, let me put it this way. I think it really comes down to one fundamental paradigm that we need to change is, traditional change management, which is what I call your CAB-based, you know, Razor request for change, blah blah blah, is based on the idea that we inspect every single change. And as you probably know, Edwards Deming’s said many, many decades ago, “We’ve got to cease reliance on mass inspection to get quality products.” Well, that’s exactly what we’re doing when we say, “Hey, I need to make a change. Well, let’s just take a look at it,” you know, so on Thursday, we’re gonna get together and look at that. But you know full well that, in a DevOps environment or a continuous integration, I mean, we are moving all the time. We don’t slow down and stop.

    And so, that mindset, just the mindset of stop and let’s inspect is completely incompatible with a flow-based, whether it’s agile or DevOps or CI/CD or any of the others that are out there that are more flow based—you know, rapid feedback loops, short cycles, et cetera—it’s just incompatible.

    And so, what we have to do is elevate out of the inspect mindset into the expectation mindset. And that is—what is it that this organization needs from change management? And some of it’s governance and compliance, some of it’s operational stability and recoverability and some of those kinds of things. But let’s state what that is, we’ll state what those things are for the organization so that those that are working in the pipelines, the value streams can actually optimize how well we achieve those expectations and don’t dictate how they do it, let’s just dictate what the outcomes that are required by the organization. That’s the fundamental shift, and it’s big. I mean, it’s more than a 180-degree shift. It’s very significant.

    Ashley: Definitely a different mindset for sure, you know, the inspection versus not. It’s not unfamiliar, though—that’s what security is doing about shifting left and moving ______.

    Sanker: Yeah.

    Ashley: Iterating the process, DataOps is also—you know, everyone wants to move early in the process, but it’s more than that, because you’re talking about a pretty different mind shift change. So, talk about compliance. Talk about governance from a change management standpoint in this new paradigm.

    Sanker: Yeah. So, you jumped right to the jugular. That is the million dollar question.

    Ashley: [Laughter] No, but it needs to be done and, you know, we all sort of pooh-pooh it.

    Sanker: Right.

    Ashley: But you know what? If you don’t do it, it’s really not fun doing it at the end, right?

    Sanker: So, maybe you remember, because it certainly rattled my world—Dr. Forsgren said on stage at DevOps Enterprise Summit 2018, “CAB is useless.” And everybody in the audience laughed and cheered and duh duh duh duh duh.

    Ashley: You’re not talking about Cabernet, you mean change advisory board.

    Sanker: No. [Laughter] I bet you probably would enjoy a good Cabernet, and I’d—

    Ashley: CABs, yeah. Those others are still good, but anyway.

    Sanker: Merlot, et cetera, they’re all good. But the thing is, she’s not wrong. But she was looking at it from a very specific lens, and that lens was about failed changes—in other words, the success of changes and the recoverability of changes, two things that are really important to the stability of the business. And what she found in the data, she’s a data expert, she has a Ph.D. in this, so there’s no mistake, here.

    She’s absolutely right—however, CABs frequently are part of the compliance equation. In other words, it’s useless for those things, but it has a use elsewhere in the organization, and for many organizations, that’s a very scary thought. Because that’s where we’re able to document and demonstrate for internal and external auditors and validators—are we doing the things that we’re required to do? PCI DSS, HIPAA, GDPR—you know, all of those other compliance framework, NIST 800-53, which hopefully we’ll get to in a minute, but they all have requirements around managing changes within your organization. So, if we don’t do it in the CAB, how do we do it? And that’s precisely the point.

    So, what I’ve sought to do is help organizations build those compliance points into the pipeline itself. In other words, we can point directly to where that’s being done in the pipeline and don’t require a “let’s slow down and actually check the box, here.”

    You know, the governance is—well, can I just say this? There’s a difference between governance and compliance. They’re two very different topics, right?

    Ashley: Yeah.

    Sanker: So, governance is about how organizations make decisions, and making sure that those decisions that are made throughout the organization align with or help fulfill the purpose of that organization, whereas compliance really has to do with the conformance. Are we conforming to applicable standards or expectations, either that the organization has or we have externally?

    So, but when we talk about governance and compliance, we think about performance and conformance. I’m sure you’ve heard that—COBIT and lots of other frameworks have talked about that for years. But performance is about, you know, your strategy and are you utilizing your resources in pursuit, or are you producing value or value creation? And I think that’s fascinating.

    I’ll bet you’ve read or heard of Mark Schwartz’ new book, “The Art of Business Value.” It’s a fascinating read.

    Ashley: I actually haven’t read that one. I’ll have to check it out.

    Sanker: I would put it on your reading list, and I know that sometimes he rubs people the wrong way. I found out that he rubs me exactly the right way, because he asks some really challenging questions, like, “What is business value?” You’d think that’s an easy question to answer, right? It’s like, you know, for a for-profit corporation, it’s like, you know, making your market windows, making your profitability, adding features that attract and retain customers, those kinds of things.

    But what if you’re in—as I was as a CIO—what if you’re in a government agency? That’s not a for-profit organization, so what is business value? Delivery of services to citizens, upholding environmental standards? There’s a lot of things that governments do, but they’re not profit motive. In fact, in some cases, it’s not even financial, so what is business value?

    And so, when we’re running our backlog, you know, we’re scoring our user stories, like, how do we score that in that kinda context? And then, just when he, when I thought that he had confused me enough, he asked, “Well, what if your business strategy”—and I don’t wanna give away his book. Go buy his book. But—

    Ashley: No, just tell me the last chapter, that’s all I need.

    Sanker: Exactly. [Laughter] I actually sent him an e-mail. He’s a great guy and he thinks very, very big. But one of the things that he said was, “Well, what if your strategy is, we want to move customers off of this platform and onto one that’s more profitable to us, or one that’s in alignment with our roadmap and our strategy?”

    So, if we’re down in the value streams or in the pipelines, scoring our user stories against customer satisfaction, customer value, things that customers really, really demand and want, we’re actually creating negative value—these are my words, not his. We’re creating negative value because we’re making our customers like the platform that we want to get them off of.

    Ashley: Mm-hmm.

    Sanker: So, business value would actually be—no, do things that make them less likely to like this product. Make them want to migrate. Make it easy for them to say, “You know, this is really getting old. I’m gonna try your news service.” And so—

    Ashley: Right, that behavior change.

    Sanker: Yeah, exactly. And so, when we really start thinking about, we’re getting, you know, your governance, especially your governance and your compliance, we have to have adaptive, rapidly adaptive systems that allow us to, at any point in time, understand what the current strategy is and how it affects the decision making that we’re doing at any point in time—all without, of course, getting your board of directors involved with every single decision throughout the organization.

    Ashley: Well, let me ask you this. I’m really interested, I’m really curious about how you take this kinda shift in mindset and how you’ve woven that into the “Change Enablement Guide“ that you were the author of. So, you know, it isn’t a process discussion.

    Sanker: No.

    Ashley: You’re talking about viewing things a different way, valuing some things maybe we didn’t value before, maybe some other things less, and maybe delivering a different set of things. Let me just share, briefly. I remember a day, probably not too long ago, where CABs saw their role as saving us from developers and all the bad decisions and lousy code they were gonna create, so that’s what—

    Sanker: [Laughter] Oh, yeah! Absolutely.

    Ashley: – that’s what they say in those CAB meetings.

    Sanker: Yes. Oh, absolutely.

    Ashley: But, you know, that’s not what we’re talking about. Tell me how—so, how do you build that into, in a way that’s gonna help people make that shift through ITIL4?

    Sanker: So, thank you for saying that, because there is some unspoken or lightly spoken or spoken only in factions kind of frustration, there. It’s like—let’s be honest. I used to be a network engineer, I did network security, I did all things network. So, I used to be a change manager at one point in time, and who the heck are you, Greg, to cast judgment on this piece of code and decide whether that’s gonna work appropriately? Everybody knows that I can’t add value to that, and everybody knows that this external body who’s not familiar with it, certainly not at the pace—you can’t add value, and yet you act like you do. And that just—that doesn’t sit well with anybody.

    So, we really need to, if we shift our focus to, “What is it that we’re trying to achieve?”—so, I talk about putting the right measures on the pipeline. How well is this pipeline producing outcomes, those outcome expectations? Is it producing consistently high quality changes? Is it consistently not delivering failed changes—or, for the failed changes that do go through, how quickly and easily can we recover from that with minimizing impact to the customers?

    Those are the things that your CAB or whatever your body is should care about, not the individual changes. You’ve got to get away from the individual changes. You can’t add value to that.

    And so, I get asked a lot of the times, it’s like, “Well, what do we do with our CAB?” since, you know, they’re very unpopular. And in some cases, the best answer is—now I’m gonna say it publicly, so I’m gonna be quoted on this, but—“Blow it up. Just get rid of it. And don’t ever use the C-A-B word again, because it has such a bad name.”

    But it isn’t without value. So, if you have a CAB and the CAB starts saying, “Look, developers, we can’t add value to what you’re doing, but what we really care about is some of these things. So, let’s talk about those things.” And if you’re talking to, particularly DevOps, agile, automated pipeline folks—they were gonna welcome that conversation because they’ve already got it covered. They just do. You know, like you know, I’m the change manager—“Well, have you tested that code before you released it?” Well, dude, that’s the only way it gets in production, if it passed tests. There’s no way for it to make it into production. So, not only is it insulting, but it’s a dumb question, and undermines your credibility.

    Ashley: Well, let me ask you this, then. I think where you’re going—and tell me if this is right—is, “We have so much automation in this pipeline process, right? That’s the whole idea of being able to produce software more reliably, more rapidly.”

    Sanker: Oh, yeah.

    Ashley: There is so much data that’s generated and captured, but not leveraged.

    Sanker: Yep.

    Ashley: We treat it as data exhaust, right? That’s sort of this fancy term for it.

    Sanker: Yep.

    Ashley: In that is a lot of compliance, is a lot of governance information—

    Sanker: Yeah.

    Ashley: – that you can then elevate and connect, and basically save developers time and not having to produce reports at the end. “I’ve already got three-fourths of what I need or what you need, and let me help you put the finishing touches on it so you’ll sail through the next whatever NIST whatever it is.”

    Sanker: Yeah.  So, there, you get to write a chapter in my next book, because that’s exactly, exactly what we want to focus on is, we want to be—you know, alright, this is potentially controversial, but change management is a feedback loop. It helps us understand the quality of changes that are coming through our pipeline. Sitting there and passing judgment and saying, “Nothing passes until I say so”—that’s not adding value. That’s absolutely antithetical—you know this—to the DevOps continuous flow kinda mindset.

    Ashley: So, they’ll go every way around you you can think possible, you know?

    Sanker: Yeah, and the other thing, as we well know, is like, develop is most frequently used in the flagship products. “This is the core business. Get out of our way. We have to deliver these changes to maintain our profitability, maintain our market windows—all of those things that are core to the business, and you think I’m gonna stop for three days when I went from, you know, a user story to a value in under 24 hours in some cases? There is no way I’m gonna stop. I’m just not going to. I will take the risk of it failing and you saying, ‘You shouldn’t have done that.’” And that’s—

    Ashley: You can talk to the chief product officer about—

    Sanker: Exactly.

    Ashley: – the stock’s direction. [Laughter]

    Sanker: And I helped an organization who was, you know, they had agile coaches and they were doing all that. It’s a large company, but I’d rather not name them. And I was asked to come in and look at their change management.

    Well, what they had done was, rather than having one CAB a week, we were having daily CABs. And we were having daily pre-CABs and then the CABs. And one of the things that I tried to get them to understand is, these two are entirely incompatible. If you’re going towards agile DevOps as a development methodology, then you must move away from CAB. CAB has got to go.

    And what was interesting is, their flagship products, the ones that really made money for the organization, they could get away with murder. Why? Because it was so critical to massive revenue in the case of this organization.

    Ashley: Mm-hmm.

    Sanker: And so, we have a big outage, we recover, we move on, that’s okay, it’s an acceptable risk. And change management will still know, we can’t have any defects or whatever in production.

    And so, we’ve got to get ourselves wrapped around the idea of managing business risk. You know, you mentioned something about the compliance, and just this last week—like I said, I’m working on a new book, but I was looking at the NIST 800-53. Everybody’s familiar with it, but they finally released version 5. And so, I’m reading through and the author’s made a very fascinating statement, and I encourage everybody to go look at this. Version 5 is phenomenal, because what they said was, “We have eliminated or reduced the specification of who actually does the activity, and we’ve instead focused on the outcomes that need to be achieved.” In other words, some of the compliance requirements or the controls said, “IT will review” or, “The change manager will review.” And so, if you’re gonna comply with NIST 800-53 previous versions, there’s not a lot of wiggle room around that.

    But I’ve been saying for over 10 years and others have been saying, it’s like, if we focus on the outcomes, then who’s doing it, how they’re doing it is open for us to do it in the most effective way. And that gets us into the pipeline space. And it’s a major step forward, because we struggle with that. Like you said, we have to meet our compliance, you know? If you’re dealing with sensitive information, you know, pharmacy has HIPAA and credit cards and all that sort of stuff, you have to. It’s just not optional. But you don’t have to do it the way we’ve always done it, and that’s the piece that I like to bring to organizations. It’s like—how are you doing it, and can we demonstrate that we’re doing it? Those are basic audit business controls minimums.

    Ashley: I’m curious about, we’re sort of talking also about the whole issue of agility and speed, and on the other end of it of the belief of the best way for something to be stable is don’t change it.

    Sanker: Oh, yeah, absolutely.

    Ashley: Which is antithetical to the environment we’re in, for sure. And so, it’s kinda like you have to embrace, you just have to say, “We love change, and how do we be super effective at it?” instead of, “How do we not change so we don’t mess it up?”

    Sanker: So, there’s two places, two things that I’m gonna mention briefly, but we won’t have time to explore them. There’s the field of study around complex, you know, and Cynefin is one of the most notable frameworks. And the idea that there’s simplex, complex—or complicated, complex, and chaotic, and we treat changes frequently like they’re either simple or complex. Meaning, we need to get some eyeballs on this thing.

    But the fact of the matter is, most systems, especially systems that we’re DevOps-ing, are more like a complex adaptive system. There’s a level of complexity that we’ve not seen in days of old, like back in the ‘80s and ‘90s and early 2000s where these systems were, we could actually try to understand them. Modern system—

    Ashley: They’re not linear. They’re more, you know, dynamic and self-creating, if you will, in run time.

    Sanker: Yeah, exactly. So, that hit me a number of years back. It’s like, this whole mindset, it’s around the idea of, it either is simple or it should be simple, and we can understand it, and we can predict the outcomes, and you can’t. You cannot do that.

    And so, we need to build adaptive systems that say, “Hey, look, this is complex.” And you’re not gonna send it to a board of examiners, you know, a CAB who are gonna say, “Let’s take a look at all the risk,” because it would take you years to do that in a complex system. That’s just—that’s not gonna happen.

    So, we need to adopt practices that are more adaptive. And one of the things in that complex area within the Cynefin framework says that we need to run some experiments to better understand the mechanism. And how that applies to change management—see, this hit me between the eyes because it was like, “No failures in production. No failures will be allowed”—that’s antithetical to learning how the system works. If you don’t make some changes to see how the thing works in operation, you’re denying yourself learning and you’re probably deluding yourself that you’re actually actively controlling it, right?

    The other piece of study that I’ve brought to the table is—and you’ve probably heard of it—this study of anti-fragile. There’s a number of authors whose understanding is astronomically higher than my IQ, you know, Ph.D.s who actively debate such things. But I take complex things and make ‘em simple. Here’s the thing—some things, like a tree, when the wind is blowing, the tree is waving, but there’s more than that going on. It’s the actually fibers, the plant itself becomes more resilient to wind if it’s been exposed to wind.

    Here in Oregon, often there’s logging operations that have a lot of controls for the environment—trust me on that. It’s changed a lot since the good, ol’ days. But one of the things that we do hear is, we leave a segment of trees at the edge of a piece of property that’s being logged, and we do that for the animals, for the environment, we protect the riparian areas. But one thing that you’ll notice is, within one year of leaving that stand of trees that used to be next to a large strand of trees—they break. A storm comes, and they get hit. Why? Because they’ve relied on the strength of their neighbors up to this point, and they’re not suited to stand the wind alone, and so those trees break.

    Here’s the thing—if we don’t inject some failure into the system, for a system that is truly anti-fragile, we’re actually making our systems more fragile. And so, a mindset of change management that says, “No errors, no problems, no failures ever, ever, ever”—you are making your system more fragile, and you’re treating a complex system like it’s simple. And so, we’ve got to change our mindset.

    Now, you know full well—and let me just blast into this—you know full well that, as we take these mammoth changes that we used to make in the old days, you know, it took 6, 8, 12 months to produce, and we’ve turned it into these microscopic things that can go through your pipeline and ours and produce values is your blast radius. What bad thing could happen in production has materially changed. The risk has become very, very small. If that—

    Ashley: And your ability to fix it—

    Sanker: And your ability to fix it skyrockets.

    Ashley: – your responsiveness, it isn’t six months to fix it or three, it’s three minutes.

    Sanker: Yeah. So, one of the change outcome expectations that I encourage people to establish is, what are your expectations for recovery from any given change, and can we realistically do that?

    So, for instance, if I put something in my flagship product that doesn’t work as desired, if I can recover within two minutes, no harm, no foul, is that okay? In a lot of cases, it is. If it’s a defibrillator keeping somebody alive—probably not, that’s probably not okay. But in a financial transaction or a video streaming service or whatever, you know, modern digital services, it’s probably okay, especially if you’ve engineered the release in such a way that you do it like that. You know, blue/red switches, on/off—all of that kind of stuff that we’re engineering in become, you start seeing why we’re able to do that and what that actually does in production. It’s wonderful.

    Sorry, you got me excited about that.

    Ashley: Yeah, boy, I could take you a lot of places with that. [Laughter]

    Sanker: Yeah.

    Ashley: So, let’s take it back to someone who’s gonna pick up a digital version or a printed version of the “Change Enablement Guide” author.

    Sanker: Yeah.

    Ashley: How is that approachable for folks? Let me be honest. I’m probably not gonna go download that to get my heart pumping to go work out, get a better workout—or maybe I will.

    Sanker: [Laughter]

    Ashley: Maybe after listening to this, I will—but how do you make that engaging for people?

    Sanker: Yeah. Well, so, one of the things—well, I’ll put it this way. It’s like, we’re not selling a product, we’re selling an opportunity to solve some problems. That’s why we call it best practice, right? And there’s a lot of best practices, and I don’t feel like they’re nearly at odds as maybe some of the industry might feel like.

    But change enablement is—and I’ve been saying for years that this day has come and it’s now here is, change management is in the crosshairs of modern digital organizations. It’s a pesky remnant of days past. And what the “Change Enablement Guide” is—and there are was more. I mean, for production reasons, I had to kind of limit it. But what I wanted to do was point in this direction of what I’m talking about. It’s like—let’s get people moving towards that. Because like you said, we’re not gonna change overnight. I’m not gonna read the “Change Enablement Guide” and go back to work the next day and say, “Alright, I can save all the world’s problems.”

    And so, what I’ve subsequently started doing is writing guidance around, “But what do I do today? You know, I went to a DevOps conference, I’m excited, but what do I do when I go back to work? How do I start moving in that direction?” And blowing up the CAB, while we might all celebrate that, that’s not a good first step. Instead, what we need to do is start moving ourselves in a better direction. It’s not more of same, it’s—what do we actually need to be doing here, and what’s the best way to accomplish that?

    And it’s difficult, culturally, because it becomes more of a culture. Like you said, there are CABs galore who despise developers probably as much as developers despise CAB. It’s a match made in—

    Ashley: Cats and dogs, living together.

    Sanker: – yeah, exactly. [Laughter] Good reference. And so, that group of people who are widely known for having that kind of mindset are not gonna be welcome when they show up to your spring and say, “Hey, we’d really like to be involved earlier in your change management so that we can help you.” It’s like—yeah, sorry, bud. Let’s start somewhere else.

    So, if change management starts with, “We’ve gotta get out of your way. We’re gonna start delegating more and more of your changes into your value stream, and instead, we’re gonna be focusing on, we’re gonna partner with you on, let’s look at the results of what you’re doing. Because we believe that you are optimizing your delivery and we believe that you have the right tools and controls in place.” So, we are moving more towards, “Are we achieving all the outcomes that the organization has? Is the organization even clear on what those outcomes really, really are?”

    You know, the one that always—I love this one—“All changes must be tested before going into production.” It’s like—well, how many shops have you been in where that’s not true, at least in modern times?

    Ashley: Mm-hmm, yeah. I call that the “meh” policy. “We’re gonna test it? Meh.” No, of course we’re gonna test it.

    Sanker: Yeah, we call it blow testing, right? You know, it’s like, “Eh, okay, I think we’re alright.” That is true. In modern terms, though? For flagship products, for things that are really, really core to the business, most organizations have some form of automated testing, or that’s just part of the process. So, rather than make everybody say, “Did ya test it?” at a CAB, just, like, “Hey”—

    Ashley: “Of course you did.”

    Sanker: Yeah, exactly. Is that happening in the value stream? If it is, I’m not—don’t worry about it. We’ll just move on. Is somebody with the appropriate authority approving that? And that’s—I am gonna take a minute to talk about this one, Mitch, is that everybody talks about change approval in terms of who’s gonna approve this idea going into production, this change going into production? Because if something goes bad, we wanna know who to blame, right? In fact, we used to say, “Who has enough—who is sufficiently authority that they’ll get fired if something goes bad?” Which is, of course, the wrong question.

    Ashley: Yeah, [Cross talk].

    Sanker: Because the right question is, “Who has enough authority that, if it goes bad, they don’t get fired, because they have the authority to put on that level of business risk?” Right?

    But if we move—and we could talk for hours, but the idea of product ownership or taking a product view of services or modern systems is, “Isn’t it true that the product owner is accountable to the organization to decide what things we’re going to move into production?” So, to me, that’s the change approval. We’re approving that we want this change to happen.

    So, nobody’s adding value by making approval at the end of the chain. We’ve already decided that. We already know that we’re gonna make this change.

    So, those are two different approvals. One of them is about a product decision or a governance decision, the other is about a business risk decision of, “Is it actually going to achieve the results that we set out to do?” I believe that most of the latter can be automated. If it’s passed its test, if the testing systems are appropriate, if we’ve done—in some cases, I love peer reviews. It’s like, you know, the code needs to be reviewed by a peer, stamp it, and we ship it out. It happens in a minute or two, you know, not hours or days or whatnot like that.

    So, I think as we kind of retool or thinking around it, I think that’s where we can start really solving actual problems that organizations have.

    I am reminded, though, I saw this—you’ve seen these before, these overhead views of an Indy car or a Formula One car coming in for a pit stop and all this stuff happens and off they go. I saw one this week that was 1.8 seconds. It was awesome, right? And people like thus always share those on social media of, like, “That’s an agile organization, doing the things” and blah blah blah.

    And so, you can’t really argue with 1.8 seconds—four tires, top off the gas, and out you go.

    Ashley: Impressive.

    Sanker: But the question, to me, is—like, I’m a bit irreverent. Why isn’t it zero? Why can’t we make it zero? If faster is better, let’s make it zero. And the answer is, we won’t make it to the end of the race unless we have fuel. We won’t make it, because [Laughter] when you’re racing at 200 miles an hour around corners, your tires literally don’t last that long. It’s not a matter of preference, it’s a matter of life and death, right? So, we have to change tires.

    So, the question isn’t, “Are we balancing speed and agility against risk mitigation?” It’s what it is that we’re trying to accomplish here. And in the case of Formula One, we’re trying to win the race. And to do that, we have to take on certain activities that cause delay—that’s a requirement.

    So, the question isn’t, “Do we have to take a pit stop?” We do. We need tires and we need fuel and sometimes minor adjustments. So, the question then becomes, “How quickly can we achieve those outcomes?” And putting four tires on in under two seconds—that is phenomenal. Hats off to those folks that are doing that. And how do you get however many gallons of fuel we put in a Formula One car in two seconds, I don’t know, but that’s amazing, right?

    And I feel like that’s the mindset that we need to adopt. We need to quit asking, “Why do we have to do change management?” We have to do change management because part of it is to be compliant. That keeps us in business. That keeps us out of trouble.

    Ashley: But it’s a really good example, because it’s not an absolute, it’s not, “Do we need to fuel?” but, “What’s just enough fuel for us to win?” right?

    Sanker: Exactly. Oh, and trust me, you’ve watched it. I know, I can see it in your eye, you watch Formula One or Indy and there’s like a thousand computers over there and stopwatches and gauges and dials and they are calculating exactly how much fuel based on the game strategy or the race strategy, the conditions, who they’re up against, what they think their strategy is—that’s how much fuel we need to take on, right? So, it’s—

    Ashley: Great model for us to think about of, you know, it’s not absolute quality or absolute whatever.

    Sanker: Right.

    Ashley: It’s what’s appropriate for the organization. We’re gonna have to leave it at there.

    Sanker: Okay.

    Ashley: And I want you to come back some time, because the next topic is Stevie Ray Vaughan or Eric Clapton. But I’m gonna—

    Sanker: Oh!

    Ashley: You can’t answer it. You have to come back and answer that, so we’ll have you here another time. It’s been a real pleasure talking to you, Greg Sanker, who is the author of the “Change Enablement Guide” of ITIL4. So, I think you really gave us some compelling reasons to go check it out. So, thanks a lot—it was great talking with you.

    Sanker: Thanks for having me, Mitch.

    Ashley: You bet.

  • DevSecOps Implementation: Interactive Testing

    DevSecOps Implementation: Interactive Testing

    This is the fourth installment in this series on DevSecOps. Read the first installment, on static analysis, here the second installment, on source composition analysis, here, and the third installment, on dynamic scans, here.

    Dynamic testing looks at the running application, poking and prodding to see how it reacts to known vulnerabilities. A complete dynamic scan watches a test runner as it runs against the application and tries all points of entry it can find, normally a highly automated process that takes a while.

    Where dynamic scans fall short is in pinpointing the problem within the application. Knowing that things went wrong when a given CVE was exercised against the app is one thing, but knowing what went wrong and where it went wrong is much more useful to make development teams productive. This is where interactive application security testing (IAST) tries to lend a hand. Using tooling in the application, interactive scans can offer the exact module/source file/shared library (etc) that caused the error.

    Generally speaking, interactive tests are best run at a targeted subset of an application, finding details needed to resolve issues the organization is already aware (or suspect) exist. In our test environment, we use interactive scans to zero in on issues that were detected by static and/or dynamic scans.

    It is likely that the interactive testing market will be the one to continue to move down the line to full-on security monitoring and/or it will encompass a growing list of environmental issues—it is, after all, designed for a running application to get exercised and find the issues. It is not a stretch to see these tools move in the above directions, and indeed you can see the beginnings of these moves in different vendors.

    But for today, the things-you-want-to-look-for list is here. My usual disclaimer applies … Through my (and others’) work for ASG, we’ve turned up a lot of information about DevSecOps tools and what to look for. We don’t have room in these blogs to delve into the depths of that research, but I’m offering an abbreviated hit-list of things to look for when moving toward this type of product. There is a lot to look for, and please be aware that this list is just to get your journey jump-started.

    • Language/framework support: It doesn’t need to be as tightly integrated as language-dependent static scans, but the tool still needs to support your specific development efforts. Support for Ruby/RoR, for example, is important if that is your dev platform of choice.
    • Source of vulnerabilities: Like other security tools, understand what interactive testing tools cover and how it is covered. Currently, most of these tools count on an external test running tool to actually run the test, while the dynamic scan watches what is happening inside the application.
    • Toolchain integration and support: Can the tool be called from your build/test toolset? Does it have integrations to feed results back into your build/bug-tracking systems? While these tools offer good interfaces to see what is going on, the explosion in tools that DevOps brought is making us dashboard-weary. Make certain your centralized code quality information store can be used as the single pane (or pain, depending upon implementation) of glass for initiating scans and reporting results.
    • Test tool support: As mentioned above, most of these tools are watchers that rely on another source to run tests. Make certain that there is some manner of coordination between these two so the test runner can be kicked off from the dynamic scan or vice-versa. More manual steps aren’t needed here and just slow the process.
    • False-positive rate: Interactive testing still struggles with false positives, though these tools are getting better. Have an understanding about average false-positive rates of the product in question, then ask about customization options to reduce this number in your given environment without reducing security posture.
    • Cost to configure/maintain: As is probably obvious, tooling an application to look inside of it is a step beyond normal production needs. Make certain the overhead of adding interactive testing to a project and the cost of integrating are worth the results to be had.

    Vendors call it “interactive” because it watches the interaction of the pieces of code inside the app as it’s running. Many customers call it “interactive” because the “tool” used to run apps for testing ends up being people “interacting” with the application. Both are somewhat accurate, though automation is coming apace, as it is for everything software testing.

    Choose a product that has enough automation today to suit your organization’s needs, and is progressing to wow you. Of all the security tools out there right now, interactive holds the greatest promise for wowing us. Since it is happening at runtime and can see inside the application, everything from environment to installed applications on the host could be covered, and that’s where some vendors are headed. An integrated environment security test tool with deep knowledge/reporting that includes knowledge gained from static scans and source code analysis is on the horizon—and, given the direction of all of DevOps, making that solution fully automated and reporting back to build systems is coming also.

    Interactive won’t replace the other steps in DevSecOps, but it will replace some of their functionality—or more often enhance some of their functionality. If a static scan notes a potential vulnerability, dynamic could verify if it is a real vulnerability before a human ever sees a report on the issue, for example.

    I’ll end this as I’ve ended the others: You all are kicking rear and taking names in a tough IT environment that exploded into near-100% remote for most of you. Keep kicking rear and don’t forget security. It is more important as your organization becomes geographically dispersed because now applications that were “secure enough” for the internal network are shared publicly. Even with infrastructure protections such as VPN in place, more caution should be applied to those applications simply because they are more exposed than “must be on our LAN” allowed for.

    Most users forget that in the end, someone is running those servers and keeping things working. So from me, for all those who’ve never seen the inside of IT, thank you. And keep rocking it.

  • DevOps Deeper Dive: ServiceNow and DevOps, One Year In

    DevOps Deeper Dive: ServiceNow and DevOps, One Year In

    It’s now been a year since ServiceNow launched its DevOps integration initiative. Since then, thanks to the arrival of the COVID-19 pandemic, the way organizations manage IT has utterly changed. With more employees now regularly working from home, more organizations have shifted toward relying on IT service management (ITSM) platforms that, like ServiceNow, run in the cloud.

    At the same time, the overall IT environment has become more complex as a greater number of applications based on microservices that span everything from containers to serverless computing frameworks are deployed in production environments. Most of those distributed applications are being built and deployed by DevOps teams who are finding it necessary to collaborate more with IT operations teams who still rely on ticket-based workflows to manage jobs and tasks.

    As organizations invest more in digital business transformation initiatives fueled by these applications, the pressure has intensified to bridge the cultural divide that has long existed between traditional IT operations teams and application development teams that have embraced best DevOps practices.

    At the core of that divide is a desire on the part of DevOps teams to automate IT process management. Tasks that once required IT personnel to implement manually increasingly now are being automated by developers employing tools and platforms that expose both APIs and command-line interfaces (CLIs). Not every IT task can be fully automated just yet, but with the rise of machine learning algorithms and other forms of artificial intelligence (AI), a great leap forward is clearly being made.

    In fact, the rise of AI coupled with a need to more aggressively reduce costs are motivating organizations to investigate how they can better unify the management of IT, said Judith Hurwitz, principal analyst for Hurwitz and Associates, an IT consulting firm.

    “It’s a major change,” said Hurwitz. “There’s a lot more focus on the issue of managing errors.”

    ServiceNow is investing in AI to address that issue, including partnering with providers of DevOps platforms to gain access to the data required to train AI models. That effort will simultaneously improve the developer’s experience at a time when organizations need to accelerate the rate at which applications are being developed, deployed and updated, said RJ Jainendra, vice president and general manager for IT business management and DevOps at ServiceNow.

    “The change management process is becoming automated,” he said.

    ServiceNow, which has more than 1,000 customers and contract values in excess of $1 million, is not the only provider of an ITSM with similar ambitions. Not only are providers of rival ITSM platforms moving down the same path, but new entrants have also emerged in the form of vendors that are extending DevOps platforms into the realm of ITSM. Atlassian, for example, has launched an ITSM initiative aimed specifically at DevOps teams looking to eschew a ticket-based approach to IT in favor of workflows that are managed directly by DevOps teams.

    As the responsibility for managing IT end-to-end continues to shift left toward DevOps teams, many won’t embrace ticket-based workflows. Eventually, DevOps platforms and legacy approaches to managing IT using tickets will converge. ITSM platforms are still the systems of record for recording IT events, so integration between DevOps and ITSM platforms is required.

    In the meantime, replacing IT processes that employ ticket-based workflows which adhere to frameworks such as ITIL will not be easy, said Mitchell Ashley, CEO and managing analyst at Accelerated Strategies Group.

    “Ticket-based workflows are incredibly sticky,” he said. “It’s not a question of undoing them as much as it is unraveling them.”

    In fact, Ashely said a ticket-based approach to managing IT may be too deeply ingrained in the psyche of IT teams to be replaced.

    “Ticket-based systems are a lot like email,” said Ashley. “They will be with us forever.”

    As long as ticket-based approaches to managing IT workflows remain in place, however, the potential for culture clash across teams that have divergent approaches to managing IT will remain high.

  • Out of the Silo: Big Data and DevOps Meet in the Middle

    Out of the Silo: Big Data and DevOps Meet in the Middle

    Given the myriad benefits they provide, it seems crazy that big data and DevOps have been implemented independent of each other

    If you have been working from home during the recent lockdowns, you probably don’t feel like you are making many new connections, but when we come to look back on this time in a few years, it might prove to have been one of the most effective networking sessions of the past decade.

    Nowhere is this more true than in the twin worlds of big data and DevOps. It seems that being forced to work from home, far from locking engineers in both disciplines deeper within their silos, has brought them out of their shells. In recent months, we’ve seen an outpouring of speculation on how DevOps and big data can work together, and even a number of impassioned manifestos on why it is time to bring DevOps ideas to big data.

    In this article, we’ll look at why this crossover is suddenly so popular, how these two disciplines can engage in a symbiotic relationship with each other, and how they are already doing so.

    The Pandemic and the Silo

    For consumers, the experience of lockdown has accelerated internet usage trends. While some of the changes that have recently swept the online world were easy to guess, the speed at which they have taken hold has been astounding. It’s therefore no exaggeration to say, as the New York Times put it recently, that the virus has changed the way we use the internet.

    These consumer trends are driving changes throughout the software development industry, forcing firms to adapt to rapid release, testing and update cycles.

    One of the biggest changes in the way that developers collect data over the past decade has been the source of the data. This year, and largely due to the COVID-19 pandemic, we passed a notable watershed: Mobile devices and smartphones now account for more than half of all internet traffic, resulting in an increased data flow that boggles the mind and creates issues of its own. This increased data load from consumer mobile devices, rather than from corporate data acquisition systems, is forcing teams to work more closely together.

    In the words of Bill Detwiler, editor in chief of TechRepublic, in a recent CBS interview: “The COVID-19 pandemic has accelerated the blending of data analytics and DevOps, meaning developers, data scientists, and product managers will need to work more closely together than ever before.” For most firms, the way in which these consumer trends are fed into the development life cycle is via analytics teams. This has meant that, during the pandemic, analytics teams were generating actionable insights at an increased rate, and this has been the driving force behind the newly intimate relationship between big data and DevOps staff.

    Waterfall vs. DevOps

    To understand the way in which big data and DevOps teams can best work together, it’s instructive to visualize the way in which developers, operational staff and data analytics agents work.

    A decade ago, most software developers were using some form of waterfall development methodology. As the name suggests, this approach was essentially linear—the specifications for software were set in advance, and after this point multiple teams worked in tandem to deliver a continuous “stream” of software updates.

    The DevOps framework changed that by encouraging communication and collaboration between operations and development staff. The idea, in short, was to ensure that software underwent continual testing—not just in terms of usability and security, but also against the requirements of the operations staff who were charged with maintaining it.

    Now, we might be facing a shift of similar scale. Many firms have now adopted big data frameworks, but these systems generally have been seen as a way of harvesting and collating market insights. This was always a waste of potential for big data because as long as it was regarded as part of marketing, it’s utility in other parts of an organization were difficult to discern.

    Not any longer, it seems. The changes caused by the pandemic—both in consumer behavior and in the way that organizations communicate internally—have given developers an appreciation of the value of the data that analytics teams hold.

    Pioneers Show the Way Forward

    It’s also worth noting that big data and DevOps teams working in tandem is not a utopian vision; rather, this way of working was already being explored within leading companies even before the pandemic.

    In retrospect, in fact, it seems crazy that these two teams ever worked independently. If you’re a transportation company, the ability to track your loads on the road and the health and safety of the cargo that they’re carrying has become mission-critical. If you’re in the armed forces and using drones on the battlefield to conduct and report reconnaissance in real-time flyovers, the ability to adapt software parameters on-the-fly can be the difference between success and failure.

    The military is, in fact, the place to look for examples of how big data and DevOps will shortly become one process. Big data in the military has long been more advanced than that in the civilian sector and is beginning to inform everything from IT predictive analytics to real-time usage analytics.

    The Future of Big Data and DevOps

    There remain, of course, significant challenges to be overcome before the promise of DevOps working with big data teams can be realized. Perhaps the most thorny of these is the fact that computing performance is crucial for big data in a way it hasn’t been for development: The latter teams may need to build capabilities to make use of the insights generated by big data teams.

    Still, communication can only be a good thing, and if the pandemic has forced some of us to emerge from our bunkers and start to work together, then at least one good thing will have come from it.

  • DevSecOps Implementation: Dynamic Scans

    DevSecOps Implementation: Dynamic Scans

    This is the third installment in this series on DevSecOps. Read the first installment, on static analysis, here and the second installment, on source composition analysis, here.

    One weakness of static analysis is its failure to account for environment and use. Running static analysis on a code base as the only check before production deployment is akin to looking at the engine of a VW Golf before assembly and pronouncing the vehicle fit to use. The way a vehicle runs after assembly, and the way it is—or, more precisely for dynamic scans, could be—used, matter to any product. That is where dynamic scans come in.

    Exercising the product as it runs is a big deal, and that is what dynamic scanning attempts to do. Automating the testing of applications by exercising inputs and watching the results, dynamic scans can detect a variety of issues that static analysis simply cannot. These tools are the source of a lot of the noise in DevSecOps because they’re testing a variety of scenarios with each run, and things that a dynamic scan sees as vulnerabilities might very well be known to not be a risk at all by the dev team. As with static analysis, this has been improving over time, and they’re well worth a look at this point.

    The best use for these tools is as gateways—before going to test, in test and before going to production, set a policy for promotion requirements and then run dynamic scans to determine if the application meets that policy, needs remediation or needs approval to go live with possible issues known.

    With all of that said, here is a starter list of things to look for when considering using dynamic scans:

    • Production Safe: These tools run tests that try a documented vulnerability to see how the application responds. The thing is, if you’re not very careful, this can cause serious problems in production. Since long-term the trend is to eliminate a pure test environment for all but the most stability-conscious organizations, knowing that the product you choose can run in prod without causing outages is an important step.
    • Vulnerabilities Tested: Most vendors offer a list of what they look for. Compare these lists to your organization’s needs and other vendor’s lists. Since choosing a product is unique to each org, I don’t have much in the way of buying advice, but here I will say if the product does not support at least the OWASP top 10, look elsewhere, simply because OWASP is both a recipe for you to prevent attacks and a handy list of easy vulnerabilities for attackers to look for. All of the vendors we looked at for our ASG work do support OWASP, so this might be non-advice.
    • Integration: Dynamic scans need to be run after the product is fully assembled. This is not to say complete, but the various pieces need to be together—the scans generally look at an application, not pieces thereof. This means they can be spawned at any point after build—or, more specifically, after post-build deployment to dev/test/prod. Make certain there is a place in the toolchain to call the scanner from and proper gateways to suit your organization’s needs. Part of that evaluation is how the application is kicked off: Can your build tool call it directly? Is it deeply integrated into your chosen environment, or will you have to do work?
    • Reporting and Gating: What level of reporting is available: Can dynamic reports and static reports be integrated to show that a potential problem reported by static scan is found to actually be a problem by the dynamic scan? Can an overall status report be found (à la C-level dashboards) to show the security posture of a given application? Can alerts be configured for apps that fail certain tests (so you can make informed decisions before going to production, for example)?
    • RBAC: This is applicable to all scanning, but I’ve chosen to include it here because for dynamic (and interactive), it’s a bit more important. Can you limit what an individual can see/do? In the end, dynamic scans are reporting on actual vulnerabilities in your codebase—only those who need to know to fix the issue should be able to see it; others should just see pass/fail overall for the app, not a list of known vulnerabilities in the app. The ability to run these scans should likewise be limited. Who/when is a corporate decision that is as much policy and process as it is the chosen tool, but if the scanner doesn’t support that policy, enforcement becomes more difficult.

    When you get to interactive scanning, it is time to consider how security issues traditionally have been handled versus how they should be moving forward. Most organizations hold security issues separate from other application development issues. It is time for that to end—and indeed, some organizations ended this process long ago, but too many are hanging on to the past. Security vulnerabilities that can result in a risk to the organization need to be run through the system as high-priority bugs. Because they are. We used to think that something making the UI look bad for 5% of customers was more important than a security issue impacting no one. A truly global internet and the rise of automation even among attackers make this a silly stance to take. No one is impacted by security issues at this second, but sometime in the next minute, everyone might be impacted. It’s like saying you don’t need scalability for Black Friday because the servers are handling traffic well right now (written in early November). The issues that dynamic scanners alert on are generally real (removal of false positives has come a long way in the dynamic space) and generally dangerous. If your scan tool can find it, assume attackers’ scan tools can, too.

    As with the other installments, I’ll thank you all for keeping the lights on in a really topsy-turvy IT environment—you are doing us all a service—and mention that all of your hard work is at risk if security at your organization isn’t taken seriously. We quickly identified the threat from the sudden opening of millions of RDP ports on the public internet, and most of you took steps to secure your own. Keep it going. Make certain the apps you are developing and deploying are as safe as you can make them. And keep kicking rear—you are the rock that’s keeping the suddenly dispersed company clipping along.

  • FinConDX: The State of DevOps in FinTech

    FinConDX: The State of DevOps in FinTech

    The FinTech space has grown dramatically this year as more consumers shift their financial transactions online due to the COVID-19 pandemic.

    Financial services companies are learning how to ship better code, faster by adopting DevOps methodologies. DevOps helps companies develop fast, fail fast and learn fast so they can fulfill customer expectations and deliver quality features faster to market than their competitors. However, the shift to DevOps is not that easy.

    The digital transformation journey for banking and financial services organizations can be quite challenging. That is why, on Dec. 3, the top financial services and tech leaders in the industry will come together at the FinConDX Virtual Summit, a free virtual conference focused on the DevOps, DevSecOps and digital transformation practices and tools you need to thrive in the FinTech industry today.

    FinConDX features a stellar lineup of industry experts, including:

    • Yaniv Yehuda, co-founder and CTO at DBmaestro
    • Viktor Farcic, principal DevOps architect at Codefresh
    • Sladjana Jovanovic, vice president of payments technology, TD Bank Group
    • Shannon Lietz, director of DevSecOps at Intuit
    • Sanjeev Sharma, senior vice president, head of automation and platform engineering at Truist
    • Rosalind Radcliffe, distinguished engineer at IBM
    • Don Duet, CEO and co-founder at Concourse Labs
    • Ravi Lachhman, chief evangelist at Harness
    • Matt Wilson, senior product manager at GitLab
    • Mitchell Ashley, CEO and founder at Accelerated Strategies Group
    • Keith Murphy, senior solutions architect at OutSystems
    • Jonathan Smart, partner, business agility lead at Deloitte
    • John Rzeszotarski, senior vice president of technology infrastructure at PNC
    • Dan Garfield, chief technology evangelist at Codefresh
    • Gary Gruver, CEO at Gruver Consulting and analyst at Accelerated Strategies Group

    The virtual environment features auditoriums with different tracks where attendees can watch the sessions by industry leaders and an exhibit hall lined with booths showcasing the latest solutions from leading DevOps companies. In the exhibit hall, attendees can download resources and interact with partner sponsors including Codefresh, GitLab, WhiteSource, DBmaestro, Harness, Traceable, OutSystems and more.

    Breakout sessions and panel discussions will fall under following topics:

    • Leading digital transformation
    • Emerging technology
    • Security and audit
    • Modernization tools

    Attendees can participate in a scavenger hunt throughout the virtual environment for a chance to win amazing prizes. Participants earn points by visiting booths, watching sessions, downloading resources, networking with speakers in the chat and finding hidden objects in the environment.

    Registered attendees will also receive a link to access the virtual environment on-demand for a limited time once the live event is over.

    To see the full agenda and to register, please visit the FinConDX website.

  • DevOps Chat: Adopting Agile + DevOps in Large Orgs

    DevOps Chat: Adopting Agile + DevOps in Large Orgs

    In this DevOps Chat, Akshay Anand, ITSM product ambassador at AXELOS, and David Crouch, senior advisor at Beyond20, join Mitch Ashley to share best practices and learnings from introducing and adopting Agile and DevOps in large organizations. Akshay and David discuss overcoming change management challenges and obstacles to increase adoption.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everyone. This is Mitch Ashley with DevOps.com and you’re listening to another DevOps Chat podcast. I’m joined by Akshay Anand, who’s product ambassador with AXELOS, and David Crouch, who’s a senior advisor at Beyond20. Welcome. Good to have you on, good to be talking with you.

    Akshay Anand: Thanks for having us.

    David Crouch: Thank you.

    Ashley: Great. Would you first both start out – let’s do a little bit of introduction just to introduce yourself, a little bit of what you do and a little bit about the company that you’re with, if you want to start Akshay.

    Anand: Sure. Yeah, my pleasure. Thanks for having us, as I said. My name is Akshay Anand. I’m the product ambassador for ITIL, working for Axelos Global Best Practice. Axelos was a company that was formed about five years ago as a partnership between Her Majesty’s Government and a large private sector company called Capita. We like to think of ourselves as the custodians of best practice bodies of knowledge, not just ITIL. We also manage PRINCE2, which is our project management, MSP which is Managing Successful Programmes, which is our program management, and a few others besides.

    Before joining Axelos I was a consultant for most of my career in the IT service management space across both the US and the UK. I spent a couple of years as the head of service management for Macmillan Publishing as well. And that’s me.

    Ashley: Great. David?

    Crouch: Hi. My name is David Crouch, affectionately called Lower Baltimore at the office because that’s where I live and where I’m coming to you from right now.

    Ashley: [Laughs]

    Crouch: It doesn’t come with a _____. I spent about 20 years at Johns Hopkins University and Hospital before coming to Beyond20. Finished my time there working in the IT solution center which was our service management center for excellence. At Beyond20 we are a company that focuses on digital transformation. At the core of it we really try to improve the way people and organizations work by looking at their digital transformations and looking at how to improve their IT service management processes. We do consulting work, we do training, and focusing on ITIL but also project management and Agile and DevOps and these other areas as well, and we also have a team of folks who implement IT service management tools.

    Ashley: Excellent. Perfect timing for our topic today. So let’s kind of jump into that. We’re talking about Agile and DevOps and ways that that’s introduced into companies’ best practices, learnings from adopting or not [laughs] those different practices. Maybe we can start with how we introduce or you’ve seen companies introduce either Agile or DevOps or both into organizations and kind of thinking probably larger organizations where there’s more change management involved. I imagine that’s a lot of work you get into, David. Is that correct?

    Crouch: It is. You know, we see a lot of organizations who know a little bit about DevOps and a little bit about Agile but haven’t really adopted an Agile mindset and don’t really have the full understanding of what organizational structure can really support DevOps, and yet they need these things to really support a digital transformation. So typically we’ll start with doing some sort of assessment to see where they are now to see what pain points they have right now, and then try to figure out, okay, does it make sense to introduce Agile, where does it make sense to introduce DevOps, and how quickly can that be done or how gradually do we need to introduce these concepts?

    Ashley: How about you, Akshay? I’m sure you worked with plenty of organizations that have introduced this and gone through that process.

    Anand: Yeah. I’ve actually seen broadly two major patterns. The first being where the organization transformation is driven top-down, so it’s usually a board of directors or whatever governance structure says, “Look, you know, this is the vision of what we want the company to be,” and the senior-most leadership team says, “Well, in order to be able to do that we need to be able to iterate faster, we need to be able to have much more of a focus on pushing out digital products and services,” and that change then percolates through the rest of the organization, so very much a top-down thing. The other major pattern that I’ve seen and experienced is where you might have a small team who with the help of let’s just say a more enlightened manager is given the freedom to start to experiment and innovate and change ways of working while sort of at the same time remaining compliant with the rest of the organization’s policies and what have you, and they’re able to demonstrate improvements in either efficiency, productivity, experience, whatever it might be, to the point where the rest of the organization says, “Okay, so what are these guys doing and what can we learn from that and how can we start to pick up some of those changes ourselves?”

    So in a way at that point, it’s a branch, that sort of new ways of working or new principles of working start to diffuse horizontally, so there are other sorts of teams, engineering teams, operational teams, et cetera, who start to think and work in those ways. Or it could be that the stories of that change percolate upwards, where the middle management or the senior management say, “This is interesting. Are other organizations doing something like this? Would it benefit us to roll this out a bit wider and give it the sort of support and evangelism and investment that we need in order to create a larger change?” So that’s the sort of branch, but broadly it’s two parts—top-down, bottom-up.

    Ashley: You know, I think one of the biggest challenges with introducing any kind of fundamental change like Agile or DevOps is sort of crossing that chasm between saying we’re doing it and kind of really truly embracing the spirit or the concepts or the culture of it. Have you found that also to be true?

    Anand: Well, yes there’s a difference. The way I reconcile this difference perhaps is in the following way and David probably has a completely different experience from his vantage point but for me one of the biggest challenges that organizations face or at least let’s say practitioners face, and then I’ll come to organizations—one of the things that I think practitioners are challenged by is in a way thinking that everybody understands things and just gets it. Everyone understands why we have to be Agile. Everyone understands why DevOps is important. And the truth of the matter is not a lot of people understand that and even if they do understand the words they don’t necessarily understand the implications to the rest of the organization.

    So there is a little bit of this need to manage upwards, need to manage outwards to be able to champion the need for that change, the need for these new ways of working. The other thing which I think organizations and practitioners are challenged by is understanding where these new ways of working will succeed and where they would not succeed, and this is some of the things that we do actually talk about within the sort of ITIL 4 library of guidance that digital transformation or these new ways of working are really important, yes, but in a certain context they can be a hindrance rather than a boom, and organizations need to understand where these sorts of things can work. If the work that’s being done is highly predictable or has very long cycles of work and feedback then perhaps trying to break it down might only go so far, but if you try to overengineer or impose an Agile way of working you might actually end up doing more harm than good, even though you’re starting out with good intentions.

    Certain things, to put it bluntly, don’t care how the work is being done. Once you start reaching the level of governance or supplier management or whatever it might be, in a way yes the CIOs today might say, “DevOps is really important, Agile is really important, et cetera,” but if you press them about what exactly is being done – look, I just say based on my conversations with other leaders I understand this is important, but the exact mechanics of how we’re actually going to execute on this I leave to the team who’s actually doing the work. So you’ve got to understand that change is difficult. The further up you go in an organization the harder it is going to be to try to convince somebody about the nitty-gritties that need to change, and you’ve got to be able to understand your audience, understand what they need to hear, understand how to convince them, and that aspect of communication is one of the critical components of organizational change management. It’s one of the places where I see the failure of organizational change management most commonly occur.

    Ashley: Interesting. How about you, David? Share some of your experiences.

    Crouch: You know, I think some of the more challenging but also more interesting from the consultant point of view situations are dealing with some companies that have been around for 80, 100, 150 years, and then I think folks pretty much around the globe would know some of these companies and they’re very mature in some respects. They’ve built their brand over time. But then when they start to suddenly – you know, some of them suddenly wake up and realize, “Oh, wait a second. We have gotten out of touch with what our customers really need, our external customers really need and what they want, and there are competitors that we never even dreamed of before that are entering into our space.”

    They suddenly wake up and realize not only do we need new products and new services but we also need to work in a different way to produce those because customers change their mind a lot, customers are fickle, and I don’t mean that in a bad way. We want to understand what our customers need. And then the challenge becomes, okay, well if we haven’t been doing this right now it’s necessary but not sufficient just to provide training in these areas. Training goes so far. That’s very useful, but then also where are we going to have an area where doing something like Agile software development really makes sense?

    I worked with an organization where any of what are often referred to as systems of record, so these are internal systems that keep the company running, their software team was actually doing a fairly good job because they worked in a relatively Agile way. It wasn’t 100 percent of what you would describe as Agile, but it was pretty darn good. And yet they would often run into roadblocks when they got into the more, for lack of a better term, traditional change management or as we call it these days change engagement processes where they were actually analyzing risk and managing risk all throughout the process because they were doing an Agile approach, and yet sometimes they would have this long delay at the end of the process when they got to their change control board in that company.

    On the other hand, that company almost overnight experienced some significant financial losses and significant layoffs and they very quickly needed to develop some, what we call, systems of engagement, systems and tools that their third-party paying customers would use and would purchase, and in that case, the organization bypassed their internal IT entirely and just went to a third party because the third party really was already doing some version of DevOps in that case for years. Then the question becomes does their existing IT shop still have a reason to be? The answer, in that case, was yes, but it becomes very difficult because if you tried to apply that to the entire organization, particularly overnight, I don’t think it would’ve worked very well.

    Ashley: Oftentimes in those situations, you also see people _____ project if it is an internal group just to try to isolate it and minimize the overall organizational reaction. Some people call them the corporate antibodies that tend to kick out and respond to change.

    Anand: SIt’s interesting you say that, Mitch. What I’ve also seen though is that can also work but it has to be done very carefully because – who was it, which book was it? I think Geoffrey Moore – was it Geoffrey Moore? Yeah, I think Geoffrey Moore wrote an entire book about this phenomenon, a book called “Zone to Win,” and what he was talking about in that book with numerous examples, which I’m sure we’ve all experienced, is that skunkworks, that innovation lab does really well as the innovation lab, but the moment the organization says, “Okay, how do we now scale this new innovative feature, this new innovative product and industrialize it to the point where we can offer it to our thousands, millions of customers,” that’s when it falls down because you’re changing the constraints within which a product exists, if you will. It no longer exists as a small experiment measured in terms of success as an experiment.

    It’s now got to be able to stand up to the measures of being like this industrialized product or service. So if you’re moving the yardsticks, is your product capable of actually bridging that—moving across—I don’t want to use the word bridging the chasm—is the product actually capable of moving across into that industrialized state? And what we’ve also seen through some of the research that went into ITIL 4 as well is a lot of organizations failed to make that bridge between that innovation lab and the industrialized operation.

    Ashley: I appreciate you sharing that too. It could happen very well to the external group that gets formed as well. How do you reintegrate that, repatriate it back into the organization?

    Crouch: And there’s a Harvard professor that famously kind of describes some digital transformation experiments as kind of like attaching a speedboat to an aircraft carrier. You know, they move at different speeds, and although the speed may go really fast, try to get that to translate into the aircraft carrier. It’s probably not going to happen. So it becomes—you know, the challenge is do we do a proof of concept, which sounds like an experiment to me, but if we do that can we really translate that into larger organizational success? And hard to say to come up with a definitive answer, but for larger organizations I think that what I’ve seen is that it’s been very difficult for them to do that successfully, so they have to find other ways of doing it.

    Ashley: There’s a lot of things you can look at. You can look at sort of historical what’s worked, what’s failed in the past, you can look at—I call flow—what is the momentum of the organization? Are we a metrics, are we a finance, are we—what are decisions driven by? One of the things that can also help with that is also alignment. It’s why are you introducing Agile? Other than this being an IT fad, which it’s not just that, but it could be just for that reason.

    Usually, it’s not a great reason to introduce it. You’ve got to have some real true business drivers like your example, Akshay, of we’ve got to respond to these things in market, right? The customers’ expectations have changed. That creates a sense of urgency but also purpose, like what are we trying to do differently and why and then how do we apply this?

    Anand: There’s an additional wrinkle or complication, if you will, for publicly-listed companies as well where they’ve got to report certain metrics, and if you look at any company’s quarterly filings or annual filings where they talk about the year ahead or the quarter ahead, they are using certain types of metrics to communicate their direction and their results to their shareholders, to the market, and you might say, “Look, these measures are no longer relevant for what we’re trying to do or for where we’re trying to go,” but then what does that do to the company’s confidence as a shareholder? The example I cite is when the New York Times tried to move to the digital subscription. You can’t translate print subscribers into digital subscribers or engagement.

    How do you measure the success of the digital engagement because you’re not delivering hard copy newspapers anymore? But the investors were used to seeing those sort of subscriber accounts and subscriber churn and all these sorts of things as forward-looking guidance. So there’s an additional complication from a governance and a reporting mechanism, especially if you’re a publicly listed company, and that’s why people like David have so much fun going into these companies trying to help them figure out, well, it’s not just about changing the ways we’re working; how do we educate our peers, how do we educate our managers, how do we educate our suppliers, how do we educate our consumers, how do we educate our shareholders? There’s all these different stakeholders. It’s not just about changing the way your software development is done. It has massive ripple effects across the ecosystem.

    Crouch: Well, and what’s really interesting about like New York Times is that they were still successful with their print publication, one of the few, when they began their digital transformation. So you have a tremendous amount of courage to be able to say as the leader, “You know what? We’re successful now but this is not going to last. We can see it in the marketplace. So what we actually need to do is intentionally set up a parallel operating model and say we’re going to move in this new direction.”

    And that means that that’s in their case going to siphon away, intentionally siphon away over time the way we do business now and it’s probably going to hurt in the meantime, and we’ll use any of the profits from our existing model to move over to this new model. But gosh, if you’re an investor you have to be pretty nervous about that unless you’re really in it for the long term and you say, “You know what? They’ve seen this and other companies have tried this; how do I know that New York Times is going to be successful when some of their peer publications have not been when they try to do this and for a variety of factors?” It seemed to work for them but it’s pretty daunting.

    Ashley: That’s what I would call the disrupt yourself before you’re disrupted model, right? Because usually it’s much more painful when you’re reacting to someone else who’s coming in and taking market share or whatever. That’s a really good example too of in the print media going into digital, it’s not the same product, it’s not the same consumers, it’s not the same behavior. The market really has changed, or maybe you’re just really going after a different market who would never now buy print but they would just use that example.

    So a lot of it can be driven by those things that we’ve answered about strategy or understanding of the market or shift’s that are occurring and tie those reasons as part of why we need to create software faster or why we need to create it in waves of some smaller increments to be able to respond to really rapid shifts or experiment in market. That’s another great reason to do that.

    Anand: Absolutely.

    Ashley: Well, talk a little bit about—you mentioned the speedboat. I can just imagine it tied to the aircraft carrier, kind of sort of like the fly tied to the string, right? How do you start to introduce speed into a larger organization that has a lot of momentum around the way they’ve been doing things that’s worked for them probably very successfully? How do you start introducing something like speed to say, “This is super-critical and here’s how we’ve got to operate now?”

    Crouch: I think it starts with trying to understand what they’re primarily trying to do. Where does having more speed make the most sense? So it could be we’re in a consumer marketplace and the customers are changing their minds a lot and there may be a way to develop something more quickly to respond to that need or in the perfect world to actually kind of predict and shape that demand for the great digital masters. On the other hand, some companies may say, “Well, maybe we’re not in a strictly business-to-consumer marketplace but we need to increase operational efficiencies in some areas,” and so maybe speed is more important there. And then for a few companies out there it may be both, right?

    It may be if we increase operational efficiencies on the inside of the company we can also drive the outside third party customer, and that’s no easy way to do that, to figure out where speed is going to be appropriate and where doing things quicker might actually get in the way in some cases. Akshay, I don’t know what your experiences have been there.

    Anand: I mean, look, there’s a lot of great organizational change, organization design, literature out there for people who want to read up more about this sort of thing. The Satir change model or Kotter’s Seven Steps or whatever else it might be. In my experience, one of the first steps I’ve seen in any successful change is – to use Kotter’s language – creating a sense of urgency. I guess this is something I remember from my time living in the US. Wasn’t there this sort of tagline that the first step to recovery is admitting you have a problem or something along those lines, right? So it’s kind of like that.

    The first step towards creating a change is to acknowledge that you need to change, because once everyone acknowledges that you need to change you can start to think about, well, what sort of change do we want to have? And this has to be a highly collaborative effort. It’s not just your CXOs locking themselves in a conference room at a resort for two days and whiteboarding this out. This has to be a truly collaborative exercise across the entire organization. There may need to be some constraints that are in place, and what I’ve typically seen is a board of directors might say, “Look, this is where we want to be five years from now,” not necessarily saying how that needs to be achieved but saying this is where we want to be.

    That in combination with creating that sense of urgency says, “Look, if we need to hit this sort of a target we need to be able to develop software faster or close out outages quicker or have a more Agile procurement process or whatever else it might be,” and that’s when you start to identify these sort of strategic and tactical initiatives. Because you’re working collaboratively you’re dispersing that cognitive effort, if you will. So it’s not just somebody from one particular vantage point saying, “As a CFO I think we are giving away too much money so we’re going to be cost-cutting only.” That’s a singular perspective. But once you distribute perspective you can then say, “Right, if we are supposed to meet this we have to change here, we have to change there, we have to modify that,” and so on.

    For me that is the essential first two steps; create that sense of urgency and then distribute the cognition of what needs to change. Create that collaborative series of committees, councils, call it what you will, that will help you identify in different areas what needs to change, and then you start investing to execute on that series of tactical initiatives or programs of change.

    Ashley: When I first learned about change management—David, maybe you can comment on this—we used the phrase “the burning platform.” What’s the burning platform that we’re on that’s forcing us to change or really compelling us to make a change? It can be as dangerous as are we going to exist but also it could be here’s why a two- or three-year strategy makes sense. Let’s not just communicate this strategy; let’s communicate the why.

    What’s changing? What evidence do we have of this? What do we want to get ahead of or prevent ourselves from experiencing? Have you seen that as a successful strategy for helping create that burning platform or at least the sense of it?

    Crouch: I think so. I mean I think the top level of support from the leadership is absolutely critical, but when it comes to defining the problem getting down to as specific as you possibly can. It’s not just a matter of we’re not getting new products out to customers quickly enough or it’s not just a matter of we’re not implementing enough infrastructure changes within a certain period of time. It’s a matter of exactly what about that isn’t going well and why would improving that be made better? In some cases it’s obvious. In some cases, for example, it’s not so obvious, and when people think of Agile and DevOps one of the first things that comes to mind is speed, we’re going to get more speed.

    That’s an important part of it. It’s not the only reason why you might use some of those techniques. To use a simple example, if I need to be at the theater for a show—not these days—but if I have to be there at 7:00 PM and I only live 15 minutes away, driving faster won’t necessarily improve my results. I get there—it still doesn’t start until 7:00 PM. If I get there at 6:00 PM I guess I get a good parking spot. So speed is just one element, but think about with Agile doing things in smaller iterations, controlling the risk, controlling the scope. All of these are considerations in addition to just speed.

    Ashley: What –

    Anand: And the other thing that I think the research shows us—sorry, I didn’t mean to –

    Ashley: No, no. Go right ahead, Akshay.

    Anand: I was going to say one of the other things that our research has shown us as we’ve been working on ITIL 4 and other bits and pieces, even frameworks like managing successful programs and so on, is organizations need to figure out the right duration of their strategy cycle. Now if your company is a two-person company then maybe your strategy cycle is measured in days or weeks, all the way up to if you’re the government of India and you’re trying to manage—not manage, but govern one-point-how-many-ever billion people then your strategy cycle is likely to be measured in years, if not decades. Well, hopefully not decades. So I think the other aspect that we’ve seen, at least in typical organizations, strategy cycles have gone from being sort of this thing that happened every five years to maybe things that are happening every three years because the world’s changing too fast for a five-year cycle to make sense.

    We may see a point where three-year cycles are deemed insufficient and we need to move to two-year or one-year cycles. But that’s the other trend that we’re also seeing, that strategy cycles themselves are reducing.

    Crouch: I’ve seen that a little with managed service providers, for some of the managed service providers. Even one year seems to be an awfully long time for some of them, you know? So there’s this kind of blend between are we talking about strategy or tactics here, but I agree; the timeline is becoming shorter and shorter it seems.

    Ashley: It certainly has, and of course our current COVID situation, I’ve seen some resource. I know some of the sources of this, but some have said, “We’ve accelerated as much as six years in our digital transformation strategies in the last six months.” Probably not everybody has, I imagine. Well, this has been great talking with you both. I know you mentioned, Akshay, Geoffrey Moore’s book “Zone to Win” as a great resource. Any other resources on introducing Agile and DevOps or on change management that you guys can think of might suggest?

    Anand: Well, I think I would be very remiss if I didn’t actually mention ITIL 4, which is the product for which I’m an ambassador.

    Ashley: That was a setup question.

    [Laughter]

    Anand: Easy question, though. So look, there are a couple of really good books in ITIL 4 which talk about this. Two books in particular that I’d like to highlight. The first is “High Velocity IT” and the second is “Digital and IT Strategy.” “Digital and IT Strategy” actually does refer back to some of the content in “High Velocity IT.” “High Velocity IT” is more focused on the tactics and operations aspect of introducing these new ways of working and managing the changes and the implications therein to the rest of the organization. “Digital and IT Strategy” is more about the governance and strategy and the interface to tactics, so it’s about how do you determine what’s the appropriate business model, how do you create a change in your business model and from that create a change in your operating model, from that figure out where Agile and DevOps make sense and where it doesn’t make sense and so on.

    So it’s very much the sort of top-down but the leader’s perspective of things, whereas “High Velocity IT” is the practitioner’s perspective of things. I think those two would definitely be really good. I’d also plug a couple of books like “Team of Teams” I think is really an interesting read, talking about how especially in an organization like the US Army, which you would typically think is very large and flexible and so on, how they were able to create that kind of a change, not necessarily with software development but around ways of working from across multiple teams, across multiple time zones, so that’s also a really good book to check out.

    Ashley: Yeah, I’ve not read that book but I’ve heard it’s good. I think it’s by Stanley McChrystal, right? General McChrystal?

    Anand: That’s right.

    Ashley: Great author. I think he uses his whole military experiences, examples of that, so I’ve heard him speak about it. How about you, David? Any suggestions?

    Crouch: I would just say that, you know, as one of the authors of the ITIL “Digital and IT Strategy” book I’m going to put in a plug for that. I’m just a little bit biased, even though it probably took several years off of my lifespan to help write that—I’m kidding. But no, that’s really talking more about the strategy side of things. I recall reading an article years ago as part of one of his classic books. Michael Porter talked about IT is not strategic, and I struggle with that. I agreed kind of at the time and now I don’t know, and I think if IT is now strategic in this era of digital transformation it really comes down to things like high velocity IT to DevOps to Agile to some of the techniques that we talk about in the “Digital and IT Strategy” book and “High Velocity IT.” You know, I would suggest reading that.

    Anand: There’s probably a couple of other plugs to make just before we go, sorry.

    Ashley: You bet.

    Anand: For those who want to read a little bit deeper into some of the more cutting-edge, I would say, thought leaders, read up on Simon Wardley, Wardley mapping and his whole strategy cycle stuff. He publishes that on his blog. There’s tons and tons of material on his blog, which I think he started condensing to a series of medium articles. But there’s also YouTube talks that he does. Amazing stuff. And the second is Dave Snowden and the Cynefin framework. It’s getting a lot of attention in the Agile space especially, but it’s a decision-making framework. It’s a sense-making and a decision-making framework and it has applications from business strategy to project management and everything in between, and I think that’s another –

    Ashley: Excellent.

    Anand: – at the moment cutting edge but hopefully a soon mass-market approach.

    Ashley: Oh, those are some fantastic resources. And maybe a parting thought for all of us is I don’t know if IT is strategic but I’m pretty sure software is strategic these days that’s driving – really fueling our digital transformation.

    Anand: Absolutely.

    Ashley: So a topic for another conversation. Well, thank you, gentlemen, Akshay Anand, who’s product ambassador of Axelos, and David Crouch, senior advisor Beyond20. Thank you so much for sharing your experience and your thoughts on this topic. I’m sure it’s been really helpful.

    Anand: Thanks for having us, and to the audience, you can find me on Twitter @bloreboy if you care to follow me, and I’d love to hear some feedback either through Mitch or directly, however you’d like. I’d love to know what you thought about what we were talking about today.

    Ashley: Wonderful. Happy to do that. And I’ll pass that along if we do. David, do you want to share any contact information?

    Crouch: Yeah, same here. I’m on LinkedIn. A slightly younger, bow-tied version of myself is on LinkedIn. Feel free to connect with me. Certainly visit our blog at Beyond20. We’re always writing articles, trying to grapple with some of these issues, changing our mind some of the time, but you find a lot of what I write there.

    Ashley: Excellent. Well, we’ll look forward to talking with you both again soon. Take care. Have a great day.

    Anand: Thank you.

    Ashley: You’ve listened to another DevOps Chat podcast. This is Mitch Ashley thanking everyone for joining us today. Be safe, be careful out there.

  • DevSecOps Implementation: Static Analysis

    DevSecOps Implementation: Static Analysis

    One of the things I’ve done for Accelerated Strategies Group recently is looking into DevSecOps toolsets. This is a fun area for me, as development and security fit together well in my mind. Having a separate security group is useful, even necessary in some scenarios, but letting developers crank out code and looking for vulnerabilities only after has always seemed terribly inefficient. Whenever I’ve managed developers, security training is just part of the package I bring. It’s far easier to write it well the first time (the equivalent of shift-left) than rewrite it.

    I won’t name names here—this is work I did for ASG, it is not mine to give away—but I will talk about DevSecOps today, and moving into the future. I’m planning three to four installments for this series, discussing the different aspects of DevSecOps tools.

    For clarity, we’re defining DevSecOps tools as “developer-enabled security tools.” This means that in some way, they help developers improve overall security posture while leaving them to be developers. We all know security has to shift left in every way if it’s to be useful in a DevOps world, but most organizations haven’t seriously started the move to shift security. Hopefully, this series helps.

    Depending upon which financial analyst firm you prefer, projections are that the DevSecOps market will grow 28% to 33% a year for the next few years. While this is good news for DevSecOps vendors, it implies very few organizations have systemically adopted DevSecOps, or there would not be room for that level of growth.

    Whether a traditional team or a DevOps team, the static analysis portion of DevSecOps is (generally) easy to set up and configure and most products offer feedback at both coding time and build time. IDE plugins have come a long way, some even offering advice as-you-code and others allowing developers to run on-demand static scans of their work. The industry as a whole has worked hard to lower the false-positive rates that made most organizations reluctant to deploy them broadly, and prioritization of issues is possible in the best products, making identification and remediation pretty easy. Try a pilot out: Pick a vendor and project, grab the IDE plug-in and check out what they can do. Most do not yet directly support code smells, but I suspect that is coming rapidly, as the systems are already designed to look at source and determine issues. Adding to the collection of possible issues will be a differentiator that quickly turns into a market requirement.

    Things to check for:

    • Language support: Do the languages you use have a scanner? While most of the tools on the market offer support for a breadth of languages, there are differences, so check the corner cases. This truly is a make or break decision—the best product on the market is useless to your org if it doesn’t support the languages you use.
    • IDE integration: This is pretty straightforward, are the IDEs your developers use day in and day out supported? Most of these tools offer command-line variants also, so it is possible that even without support directly in the IDE, the security functionality of a given product dictates that command-line is better. In that scenario, a process to ensure developers are even running scans is a good idea.
    • Vulnerabilities detected: At a minimum, support for whatever is currently in the National Vulnerability Database (NVD) should be required, along with OWASP compliance. Most vendors offer support beyond the NVD, so a bit of research to see what types of vulnerabilities are detected will be in order.
    • Click-through support: With IDE plug-ins, one of the great strengths of scanning is the ability of the developer to simply click on the vulnerability in the results list and go to the offending code. Make certain this works—and works as advertised.
    • False-positive rate: This has historically been a problem, but the industry has brought it down overall. Ask your prospective vendors to share their numbers. Our experience shows low single-digits for a percent, which is far better than the past, but still leaves room for improvement.

    The other part of static analysis is plugging static analysis into the build process. Most of these products have integrations into the most popular build management tools. Scanning the entirety of the application allows the static analysis to see a broader vision of the app and note issues with interactions between portions written by different developers. Reporting for build-time scans is generally at the application/team level, and the best tools will show results of development-time scans side by side with build-time scans if you want them to.

    Things to check for:

    • Language support: Again with the languages. It’s pretty simple here: The core of the application may be built using languages a given vendor supports but critical parts might not. Integration is where that becomes a real issue. Make sure what needs scanning can get scanned.
    • Integrations: Most of these tools offer automated scans through integration with build tools or manual scans that the build manager can kick off. Automation is preferred by a long shot, so check if your tool of choice is integrated into your organization’s build tooling.
    • Depth: We’re not going to get into source composition analysis (SCA) here—that’s for another blog—but we are going to observe that vulnerabilities can come from your source’s sources. How far does the scanning tool trace in a static analysis situation? Is it deep enough to cover a given application’s integrations and inter-relationships? While SCA will detect a lot that static scanning won’t, things such as “unsafe values sent to [insert external API]” should be available in static code analysis.
    • Reporting: This is where you’ll find the most variety in popular tools. Look at their reporting and click-through support to find what best suits your needs.
    • False-positive rate: Same as IDE integrations above—make certain it is low enough for your organization’s comfort.
    • Scan run location: This is becoming a differentiator in the space—do you want your scans run in the cloud? In the data center? Some mix of both? Check your vendors; there are significant differences in the level of support for these options.

    This is just the tip of the iceberg, of course. It is an entire market segment, and static analysis tools have been around a long time. Many organizations have not taken dev-time security seriously yet, even though we all know we need to; I’m just offering a place to get started.

    DevSecOps just makes sense. Static scanning is mature and generally well-integrated. This is a great place to start shifting security left. We get it: Developers don’t want to be security analysts or they’d have filled one of those many security openings. But IDE integration makes these scanning tools more interactive and less intrusive, showing developers how to do it securely as they code. This is great to me because good coding practices will become second nature with real-time, on-the-spot reminders and pointers. It also means that if the developer acts upon the tool’s recommendations, the code is developed more securely the first time and there will be no need to revisit it for security proofing.

    You’re all keeping the lights on in a terribly tough time. DevSecOps attempts to make it more likely that your astounding work isn’t interrupted. It just makes sense to get these tools to protect your organization’s portfolio and your hard work.

  • Survey Sees Faster Shift to Open Source Software

    Survey Sees Faster Shift to Open Source Software

    A survey of 638 IT professionals finds the economic downturn brought on by the COVID-19 pandemic is, like other slumps, driving increased adoption of open source software.

    Conducted by Tidelift, a provider of commercial support and maintenance for open source projects, the survey finds 44% of respondents reporting their organizations are now more likely to use more open source. The survey also finds 42% of organizations have cut their application development budgets this year, with only 10% seeing an increase.

    More than two-thirds said saving time and money is the top reason to use more open source for application development during the downturn (68%) while increasing efficiency of application development and maintenance was cited by almost half of respondents (48%).

    However, reliance on open source software comes at a cost. The top two open source challenges cited are making good decisions about which components and versions to employ and making good decisions about when to upgrade components and frameworks, at 62% each. The third top challenge is identifying security vulnerabilities (49%).

    Only a small fraction (18%) of organizations are extremely confident that their open source components are secure, up-to-date and well-maintained, and slightly fewer (17%) have a formal process for managing open source.

    Tidelift CEO Donald Fischer said the current downturn is not all that much from previous economic slumps that lead to many organizations adopting an “open source first” approach to software. The challenge is many of those organizations don’t always appreciate the total cost of employing open source software. With each new open source project that gets added to the enterprise IT portfolio, the need to rely on external support services becomes more pressing, he said.

    One thing that has changed over the years is more end user organizations are contributing to open source projects, with 83% of respondents reporting their organization contributes to open source projects. Almost half (49%) of organizations now have policies governing employee contributions to open source.

    While many organizations initially view open source software as a way to reduce costs the days when the open source community was primarily focused on providing free alternatives to commercial operating systems and databases are long over. Innovations today are most often driven by open source consortiums such as the Linux Foundation, the Eclipse Foundation, the Apache Software Foundation and the Open Infrastructure Foundation, formerly known as the OpenStack Foundation. The rate at which software can be developed by teams of contributors working on an open source project makes it difficult for an IT vendor working on a single project to keep pace.

    There will, of course, always be some form of commercial software consumed by enterprise IT organizations. However, the percentage of open source software continues to rise with each passing year. The latest economic downturn is only accelerating a trend that was occurring long before the COVID-19 pandemic arrived. The challenge now is finding the best way to manage all that open source software at a time when IT environments are also becoming more complex.

  • Winning DevOps in the Finance Industry

    Winning DevOps in the Finance Industry

    DevOps is making its impact on almost every software-powered company these days. Companies in the finance domain, however, have been perhaps the slowest to embrace DevOps because of concerns regarding governance, security, compliance and regulatory regulations. But it is safe to say that they are currently leading the digital innovation in this era of cloud-native technology.

    Why DevOps for the Finance Industry?

    Higher customer engagement and continuous transactions make the finance industry one of the busiest and naturally requires uninterrupted infrastructure. DevOps implementation helps companies develop fast, fail fast and learn fast so they can fulfill customer expectations and deliver features faster to market than their competitors. Technology in the finance sector has not evolved much, as many companies have assumed it is too risky to embrace DevOps because of the amount of legacy code and old methodologies still in practice. As a result, digital innovation has lagged.

    Challenges of Implementing DevOps in Finance

    Unlike other industries, regulated industries imply several challenges:

    • Strong restrictions on secured networks.
    • Fine-grained audit trails.
    • Strong ACLs models.
    • Full lifecycle governance.
    • Integration with third parties.

    Security and compliance are top concerns in financial companies, and DevOps practices are, in fact, viewed as risk factors to security. Plus, the increased frequency of software releases in DevOps is seen as a threat to governance and regulatory controls.

    Is the Needle Moving Toward DevOps?

    According to the “2020 State of Database DevOps report“ by Redgate, financial services organizations report the highest levels of database DevOps adoption this year, with proportionally higher rates than any other sector. That shows financial sector maturity in terms of digital innovation. 

    Image source credit: Redgate

    Case Studies of Financial Companies and Their DevOps Journey

    Barclays DevOps Adoption

    In 2015, Barclays announced it was adopting DevOps for its digital transformation journey. Barclays processes payments that equate to around 30% of the UK’s gross domestic product. The developers’ morale and quality of code increased because of DevOps and the leadership team at Barclays credited DevOps with significantly reducing the complexity of its codes, which has allowed the company to reduce delivery risk and ultimately, improve the quality of services.

    DevOps the Lunar Way

    Lunar Way’s journey shows you don’t have to be big to use Kubernetes. The company started its cloud-native DevOps journey by splitting its massive monolith application into smaller microservices. To spin up these microservices, Lunar Way used Ansible, Terraform and Jenkins and to deploy these microservices as a whole unit. 

    Then Lunar Way started to experience scaling issues with microservices and didn’t experience any of the microservices benefits. 

    The company started looking for ways to move past this complexity by shifting its focus from machine-oriented to application-oriented architecture. It chose Kubernetes as the abstraction layer along with AWS, not worrying about where the containers are running, and this is how it was able to manage and unlock the velocity of microservices. Lunar Way also chose Kubernetes from a security perspective and to specify how the applications should run. Now the company runs around 80+ microservices in production with the help of Kubernetes. Watch and learn how Lunar Way did it: “Two years in production with Kubernetes.”

    Italy’s Biggest Bank Embraces DevOps

    A conventional bank running its real business on such a young technology? No way, are you kidding me? Nope, I am not kidding. 

    Italy’s banking group, Intesa Sanpaolo, has made this transition. Banks still run their ATM networks on 30-year-old mainframe technology, so embracing the hottest trend and tech is nearly unbelievable. 

    Even though ING, the banking and financial corporation, changed the way the banks were seen by upgrading itself with Kubernetes and DevOps practices very early in the game, there was still a stigma attached to adopting Kubernetes in highly regulated and controlled environments including health care and financial services. The bank’s engineering team came up with an initiative strategy in 2018 to throw away the old way of thinking and started embracing microservices, container architecture and migration from monolithic to multi-tier applications. Today, the bank runs more than 3,000 applications. Of those, more than 120 are now running in production using the new microservices architecture, including two of the 10 most business-critical for the bank. 

    HSBC Going the DevOps Way

    HSBC is changing the way typical banking works with cloud-native technology and DevOps, as an early user of Google’s Cloud Services Platform to provide core banking services to its business customers. 

    HSBC plans to build its all-new business banking service to run on a Kubernetes-managed container infrastructure using Google’s toolset. HSBC also has excellent partnerships with AWS and Microsoft, favoring the multi-cloud strategy. The company’s engineering team believes that moving to a container model under the umbrella of Kubernetes is significant because it means the environment is similar across different clouds. HSBC also hired people from outside of the banking sector because it wanted to change its culture from within. HSBC is said to have invested $10 million so far in its DevOps initiative.

    Capital One’s Agile to DevOps Journey

    Capital One’s Agile journey started in late 2011, with just two teams, which expanded slowly as more teams were trained in Agile development. The developers at Capital One were following the Scaled Agile Framework (SAFe). Initially, since the teams were new to the automation methodologies, integration testing, security testing, unit and performance testing were all done outside of development sprints by separate test teams. Later, once they understood how important it is to have a collaborative culture, they integrated testing into the dedicated DevOps teams and automated it well. Then slowly, they moved all testing into development sprints, adopting a culture of DevOps and uniformity and wiring integration, security and performance testing into a continuous delivery pipeline. As of 2016, the company had more than 700 Agile teams following continuous delivery.

    DevOps at Bank of America

    Bank of America is one of the world’s biggest financial institutions following DevOps practices, and its digital enterprise operations support nearly 4,300 retail financial centers and approximately 16,600 ATMs worldwide. The award-winning digital bank has nearly 38 million active users and approximately 29 million mobile users.

    Automating Security and Compliance

    As DevOps continues to evolve and become mainstream, it will continue to be of interest for many senior managers across financial services. The above case studies show they are already making DevOps a priority. But security and compliance remain barriers slowing the innovation in the financial companies. 

    Organizations should start small and keep security high priority, first using continuous integration and then adding continuous delivery and deployment using the available tools in the market. Over time, the more a company automates, the higher return on investment it can achieve.

    Automating security permissions and controls will help financial companies remove some of their most common software barriers and get their releases out quicker than before, while still maintaining the necessary governance and compliance. 

    Take a Step Forward

    Financial services companies need complex and rigorous software development processes to meet intense corporate and regulatory requirements. Yet, they are expected to release software faster than ever to meet expectations and beat the competition. By starting small and keeping security and compliance in mind, DevOps in finance can be achieved with great success.