“DevOps and Compliance Synergy: Reality or Fiction?”Achieving compliance might be challenging when embracing DevOps due to the rapid pace of development. However, the agility and automation associated with DevOps might actually simplify compliance. When integrating compliance into the SDLC, DevOps teams may have to use different tools or slightly modify part of their processes to adapt to specific requirements. In this episode, Sonya Lowrance (Tricentis), Anna Murray (Rocket Software), Dan Kirsch (Techstrong Research), and Tim Johnson (Cloudbees) join hosts Alan Shimel and Mitch Ashley to discuss how to effectively integrate compliance as part of your DevOps process and the challenges that come with it. Is it possible to deliver high-quality software at the speed of DevOps whilst ensuring security compliance?
Tag: DevOps Unbound
-
EP 22: DevOps Got Me Through It
“DevOps Got Me Through It” – Organizations all around the world went through an overnight digital transformation due to a worldwide pandemic and the shift to remote work. The businesses that embraced DevOps methodologies were able to pivot much faster with their digital strategy, allowing them to drive innovation, speed and adaptability, while providing greater value to their customers.
DevOps has helped businesses thrive during these uncertain times by bringing an agile mindset, tooling and processes, and fostering a collaborative culture that helps teams move faster while delivering high-quality software.
The bottomline is that we couldn’t have survived the impact of the pandemic and wouldn’t have been able to successfully speed up digital transformation without DevOps. Hosts Alan Shimel and Mitch Ashley are joined by Tracy Ragan (DeployHub), Brian Dawson (Linux Foundation) and Stephen Chin (JFrog) as they discuss how DevOps has helped their organizations grow and why DevOps is so essential for business. -
DevOps Got Me Through It – DevOps Unbound EP 22
“DevOps Got Me Through It” – Organizations all around the world went through an overnight digital transformation due to a worldwide pandemic and the shift to remote work. The businesses that embraced DevOps methodologies were able to pivot much faster with their digital strategy, allowing them to drive innovation, speed and adaptability, while providing greater value to their customers.
DevOps has helped businesses thrive during these uncertain times by bringing an agile mindset, tooling and processes, and fostering a collaborative culture that helps teams move faster while delivering high-quality software.
The bottomline is that we couldn’t have survived the impact of the pandemic and wouldn’t have been able to successfully speed up digital transformation without DevOps. Hosts Alan Shimel and Mitch Ashley are joined by Tracy Ragan (DeployHub), Brian Dawson (Linux Foundation) and Stephen Chin (JFrog) as they discuss how DevOps has helped their organizations grow and why DevOps is so essential for business. -
EP 20: DevOps for Low-Code – Friend or Foe?
Low-code development has become increasingly popular since the onset of the COVID-19 pandemic in 2020. Low-code technologies allow organizations to build and customize fully responsive, easy-to-deploy applications with little to no coding. It accelerates digital transformation and empowers citizens developers. In this episode of DevOps Unbound, hosts Alan Shimel and Mitch Ashley are joined by Derya Sousa (Kianda), Andrew Manby (HCL) and Brian van den Brink (Valori) to discuss the benefits of low code and how it is transforming software development. The video is below and a transcript of the conversation follows.
Alan Shimel: Hey, everyone, welcome to another episode of DevOps Unbound. DevOps Unbound is a biweekly panel session on all things DevOps, of all things—hence the name, DevOps Unbound. It is sponsored by our good friends at Tricentis, and many thanks to Tricentis for sponsoring DevOps Unbound.
For those of you who are catching DevOps Unbound for the first time, in addition to our biweekly panel sessions like this, which are pre-recorded, we also do a monthly live roundtable open to the audience, and you are one of the guests at the roundtable where you get to ask questions and steer the panel discussion yourself. So, if you go to DevOpsUnbound.com or check it out on Digital Anarchist, TechStrongGroup.com, or devopsy.kinsta.cloud, you’ll be able to see the schedule and maybe join in one of our live roundtables.
Today, though, is one of our pre-recorded panels—and what a panel we have. Our topic today is low code/no code and many of its aspects and flavors, if you will.
But before we jump into that, I want to introduce you to our panel today—and I thought we lost one there for a second, but we’re okay. Let me first introduce you to Brian van den Brink. Brian, welcome to DevOps Unbound.
Brian van den Brink: Thank you, Alan. Yeah, I’m Brian, I work for Valori, it’s a company in the Netherlands. We completely focus on testing, so we don’t have any developers, it’s really all about testing for us, and I worked there for about eight years now. I started, really, as a tester of test automation, et cetera, but then I also noticed in the assignments I did that it was also important to learn something about requirements, how do you get them ready to be tested.
So, I focused also a few years on business analysis to get going and know what I need for testing, up until, I think, approximately five years ago when I came into contact with OutSystems, and really focusing now on testing solutions specifically for the OutSystems platform and seeing how can you use a low code platform in your benefit as a tester. And yeah, it really opened a lot of opportunities for me, I think, but also for testers, like, how can you work with low code and use it for your testing strategy, so.
Alan Shimel: Excellent, and thank you very much, Brian. Next, let me introduce you to Derya Sousa, and I’ve been trying all through our prelim to get it right, I hope I did, but if not, say it correctly, but please introduce yourself. Okay, three time’s the charm. I’m sorry—go ahead.
Derya Sousa: It’s great to be here on this panel today. Thank you so much for having me, first of all. So, I’m Derya Sousa, I’m the Co-founder and COO of a company called Kianda Technologies, based in Ireland, helping companies across the globe. Kianda is a no code/low code application development platform, and we are the enablers of digital transformation and accelerators of digital transformation for companies.
My background is in IT. I worked as a developer IT consultant across different companies, industries during my career, and I came across the fact of how difficult it is to deliver digital solutions for companies. That’s how the idea of Kianda was born in mid-2016, with the vision of helping people regardless of their technical skills to create the technology they need.
Alan Shimel: Okay, thank you. Thank you for what you’re doing and thank you for joining us today. Our next panel member is Andrew—excuse me, it’s Andrew Manby, correct? Andrew, if you wouldn’t mind introducing yourself?
Andrew Manby: Sure. My name’s Andrew Manby. I head up Product Management for our Digital Solutions Portfolio here at HCL Software. You know, in our portfolio, we have both professional developer low code platforms and also citizen developer low code platforms.
A little bit about me, I’ve been involved in multiple successful software products. Some of them you may know about, a product called DataStage, which we created back in a former company that really, along with Informatica, lit the imagination of the data integration market and have also been involved in other market spaces as well, including bringing thought leadership to IBM around big data analytics.
Alan Shimel: Excellent. And then, last but not least, not necessary a panel member, but a panel member, but he’s also my Co-host for DevOps Unbound, it’s TechStrong Group CTO and TechStrong Research Principal and Co-founder, Mitchell Ashley. Hey, Mitch, how are you?
Mitch Ashley: Hey, Alan, always good to be doing a panel together and for the folks on the panel I know, good to see you again. Folks that I’ve met for the first time, like our audience—good to have you all here. I know we’ll have a great discussion.
Alan Shimel: Absolutely. Thanks. So, guys and gal, let me kick this off. As I was talking about before we went live, I almost view the low code/no code discussion with a BC, Before COVID, time frame, right? Much like ADBC, we have DC and then pre-COVID. I think—and Derya, you’re living proof of this. Low code/no code was alive and well prior to COVID, right? You started in 2016. Low code/no code’s been around, obviously, much longer than that.
But certainly, with the onset of COVID and the pandemic, never have we seen a need to do more, faster, to get apps out better, to expand the base of people who can develop like we have this past going on almost two years now, right, year and a half—well, more than a year and a half, two years. And so, we’ve seen sort of a rapid development in the state of low code/no code.
And Derya, I don’t mean to pick on you, but I’m gonna ask you to kinda kick us off, here. I gotta assume that this is what you’ve seen at your company in particular over the last year and a half to two years.
Derya Sousa: I think to start with, I could tell you, when we founded Kianda, when we would say low code/no code platform, our audience would say, “What do you mean?” Now, we came to a stage that anyone we talk to understands now, most of the people we are talking to, they understand what we do because of this trend and the movement that’s happening with the help of the unfortunate event of the pandemic. But low code/no code was always here, solutions that we had. It started long, long—years ago. You know, we were using Excel sheets, we were using Access, we were using WordPress back then to create our website. So, they are, I think, the initial solutions that we started to use.
But it came to a stage that the surge of the remote working, the surge of the digitalization needs that are growing really dramatically now in the last two years, it’s going into more enterprises, it’s going into more organization level. And what we are seeing, this is a problem that’s not a new problem. Currently, only 0.5 percent of the work population is capable of coding.
So, what was happening before was, you know, business users, anyone working in organizations, they were in need for technical solutions, and they were always waiting for IT teams to go in and create these systems for them. But it’s changing now, there is more inclusivity in creating technologies, empowering people, regardless of them knowing how to code or not. It’s giving an amazing world of opportunities for people who were not able to be part of it, be actively part of digital transformation are now becoming what we call citizen developers to create these technology companies, organizations are in need. And of course, we live in a digital first world now with everything that’s going on around us. Any company who doesn’t try to benefit leverage from these technologies that are rapid application development are gonna stay, unfortunately, behind when it comes to competitive advantages and improving or creating that economical value for the organizations.
Alan Shimel: Agreed. Folks on my panel—thoughts, follow on to that?
Andrew Manby: Yeah, I have a few thoughts. So, you know, as Mitch and I have discussed a couple of times, right—for me, low code or sort of no code has been around for a long period of time. In fact, my father-in-law was programming in dBase, you know, many years ago, right? [Laughter]
Alan Shimel: That’s another name I haven’t heard in a while.
Andrew Manby: Yeah, okay, let’s get Delphi, FoxPro, all of those things out on the table. Let’s do it.
Mitch Ashley: I think I met your father-in-law, so. [Laughter] [Cross talk] dBase.
Andrew Manby: [Laughter] Yeah. So, all of these things have been around for a long period of time, right? And if technology has taught us anything is, they evolve to serve a particular need and that’s because the business was trying to accomplish a particular outcome, right? So, they built something, they’ve effectively had it deployed.
Now, what’s happened now is, you know, we have a lot more tech savvy people in the workforce and, you know, IT is taking a little bit more of a proactive approach. They want a bit more governance, they want a bit more standardization, they also want to be assured of the security and governance of what the business buys, and I think this generation of low code and also no code tools, which I think are two separate things, by the way—
Alan Shimel: Absolutely.
Andrew Manby: – help us to do that. And so, you know, I don’t think it’s new. You know, what’s actually interesting—and I hope we’re gonna get to this, Alan—is the type of things that we’re seeing that people are building with these individual products.
So, I think it was always there beforehand. I think a lot of organizations like, for instance, we have one major airline and there’s a couple of hundred people who are business users building applications on it, right? But typically, you know, to be clear, a lot of them are employee-facing, and they fill the gap between some of those traditional enterprise applications or maybe to some of those SaaS franchises that we often hear about now to get the job done, right? So, I think it was alive and well before, and I think it’s continuing.
Alan Shimel: Absolutely, it’s continuing, but I think it’s changing. And Brian, I think you might serve as a poster child.
Brian van den Brink: I think, actually, if you look a few years ago, like maybe four or five years ago, a lot of companies were maybe starting to work with, for example, health systems with low code platforms, using them as smaller apps, basically, in their organization.
For example, you know, logging hours of your work, whatever, and then starting to grow, like—hey, we can really get to a product really fast, and then other departments of the organization see, hey, they realized an app in a few months, and it’s interesting to maybe expand on it. And then you see, like, a center of excellence within a company start to build apps, and then it grows bigger, and now, I think, currently more and more organizations actually use low code for their core applications as well. So, actually, the most important processes are also done with low code nowadays, and it’s growing like that, because they see the benefits, I guess. And when I see—
Alan Shimel: Well, why shouldn’t they?
Brian van den Brink: – yeah. And what I see in my work as well is that, if you work in a high code environment, it might be difficult for a tester or for a business analyst to say, “Hey, this is how it works” and dive into the product, you know, dive into the code, then you need these technical skills. But I think with low code, it’s easier, for example, for a tester to say, “Is this really what we meant with this piece of software?” and you can really get the dialogue going with your developer, even with your product owner, for example, to see, is it really doing what we want it to do, and how can we improve it, for example? And that’s so much easier in a low code environment, I think, than in a high code environment.
Mitch Ashley: You know, Alan, there’s some—Accelerated Strategies, now TechStrong Research did some research earlier this year on low code/no code and there were some interesting trends in that. And I think part of it was driven by, Derya was talking about the effect of the pandemic; you were, as well. That rose software to prominence in terms of business strategy must-have capabilities. And along with that, sort of low code/no code was sort of viewed by IT as something that happened outside of IT, we don’t really care about it, the business units are doing something, whatever, to it is a mainstream platform.
And I’m not saying it wasn’t before, but I think the embracing of low code/no code as a strategy, I think that improved over the last 18 months, and you can also see it by the need to access data, what information people need to get to, because no application is truly an island. They eventually have to get security, they eventually have to get to data and other systems and the fact that, Andrew, you talk about citizen developers and professional developers both as IT viewing those as legitimate creators of applications.
Are you seeing that trend as well? That certainly bore out in our research.
Andrew Manby: Yeah, so, we have some of our own survey that’s coming out very shortly that we did with Forrester. You know, we actually found that, from a business decision making perspective for the people surveyed, 80 percent of them said that low code was one of the top three IT priorities for the organization, right?
Mitch Ashley: Mm-hmm.
Andrew Manby: You know, 77 percent of the respondents actually said that, you know, agility and velocity were the two most important things to their business in achieving that.
So yeah, I think companies are realizing that. I do think that there’s a couple of other different phenomena, and I think, Mitch, you and I have talked about this when we’ve had a separate conversation. I do think there are the organically grown stuff that Brian was talking about, which is, you know, you provide a citizen development tool, you establish a center of excellence, right, and then you sort of start draining the desire of the organization to fill the gap with building these applications.
But I think there’s also—and maybe you touched on it, Brian—you know, organizations are taking also a top down type approach, right? And they’re saying, what are the important digital touch points I have in my organization and, given the advent of COVID, what do I need to be able to fill the gap or continue my business, right? So, if I’m a retailer, through the early part of COVID, one of the biggest challenges I had was, how do I handle, you know, touchless commerce? How do I pick up the deliveries and still allow my retail department to function, right?
Or say, for instance, I’m a tax office, right, and all of a sudden, you know, the tax function, there’s no people in the office, and what we want people to do is start filing an abeyance request to delay the delivery of the taxes—how are you gonna do that? Or, you know, families are looking for special support, family services support, right? So, those types of things have suddenly become very, very important and low code, whether it was built by the IT organization or it was built by the business has really helped to fill the void. Sometimes, there have been brand new things, and sometimes they’re just really about adapting the business to the new reality.
Alan Shimel: Absolutely. Let me weigh in, here. You know, I think prior to the pandemic, there is a long history of low code as I think, Andrew, you pointed out, right, though it may not have been called low code then. But prior to the pandemic, low code/no code was around, but a lot of people viewed it as, “Look, if I’ve gotta get something done simply, it’s Minecraft, you know, for adults.” It’s easy, you know, for simple—as I think, Brian, you mentioned, it’s good if I wanna track my hours or something like that. It’s a glorified spreadsheet.
And then I think what we’ve seen with the pandemic is, in my mind, we’ve seen three branches of low code/no code, and I think, Andrew and Mitchell, you both mentioned it. One is no code, and I gotta be really honest with you—I got a problem with that. I have the same problem with no code that I have with no ops. I think that’s pure marketing, and I don’t know what you can truly, truly do with just no code. And we can talk about that, feel free when I’m done if you guys wanna jump.
Andrew Manby: I’m just thinking about your Excel spreadsheet, right?
Alan Shimel: Uh huh.
Andrew Manby: You know, Excel is no code until you type in that formula.
Alan Shimel: Right, and then all of a sudden, you better—take it from me, I’m illiterate on Excel formulas, so I know there’s code there.
Mitch Ashley: Because you put in an expression, you’re now coding, right?
Andrew Manby: Yeah, exactly. [Laughter]
Alan Shimel: But when we look at quote-unquote low code, here is the big takeaway with COVID for me is, we have low code for developer teams—and Brian, I’m gonna include testers in there, because testers were already scripting and doing some coding in order to write their test.
So, we have low code for development teams—DevOps teams, whatever you wanna call them, developers—and then we have low code for the quote-unquote citizen developer. These are people who traditionally didn’t do any development other than maybe spreadsheets. And, you know, lo and behold, the business analyst, the person over in Sales or Marketing or in HR, they now have the capability of not having to wait or go through that whole IT process driven formula, but they could use some paint by numbers and relatively easily build a very functional application. Don’t equate citizen developer with simple app. There’s some sophisticated stuff we can do now.
Brian van den Brink: Alan, I wanted to react to that is that, indeed, the citizen developers, but also the IT development teams working with low code.
Alan Shimel: Yep.
Brian van den Brink: But what I do see is that more and more companies choose low code to basically restructure their, how do you say, legacy applications, you know, rebuild it or rethink their software. And I think it’s really crucial to still have—maybe you can work with citizen developers as part of, okay, what are the business requirements we need and how are we gonna implement it, et cetera. But it’s super important, I think, to still have your development skills on par with what you need. Because even with low code, maybe it’s easier to make something, but you still need the same kind of skill set that the developer would have to know how do I structure things, you know, how do I set up my architecture around it. Those kinds of things are still development skills, maybe without coding, but it’s still development skills, I think.
And that’s why I think you always need maybe a senior developer, whatever, in your team to be able to handle those kinds of questions and then supported, I think, of course, by juniors or citizen developers even, but the development skills that you need to build an application are still very much necessary, I think.
Alan Shimel: Agreed. Derya, it looks like you wanted to—
Derya Sousa: I would like to add something, a point to what you said, Alan, to your definitions. So, there are solutions in the market that low code solutions, they generate code for a developer to go in and speed up this process of delivery and make the changes as they receive them.
Alan Shimel: Yeah.
Derya Sousa: There are solutions like what we provide. We don’t generate code, so we call ourselves as our customers call a true no code platform. However, we allow developers to go in and create widgets, create rules and actions that can help them bring that end to end business application to life. And I think that’s important, because there is a good bit of differentiation between them, and from our experience, we work with IT teams as well as Business Departments.
Alan Shimel: Yeah.
Derya Sousa: But we start with IT Departments, because they’ve seen the benefits they can get—
Alan Shimel: The strength, the power.
Derya Sousa: – by using a platform like this—exactly, filling the gaps that they might have. And now, even since the pandemic, we are talking to more public entities for—public hospitals, for example, for government agencies. Even them, they tend to be slower to catch up with innovation and technologies. Even them are looking at low code/no code systems, how they can use their legacy systems, which they’ve been using maybe two decades, but they had no idea how, actually, they could bring a user-friendly interface on top of these systems.
So, that’s where, I think, solutions like rapid application development solutions that come into place and bring a lot of benefits to organizations.
Andrew Manby: Yeah, I would agree that certainly, you know, some of those you think are traditional sort of holdouts, right, typically on the laggards in the adoption curve are certainly doing that. You know, we’ve seen that in state government, we’ve seen that in large tax authorities. We’ve also seen that in the District of Columbia Motor Vehicle Registry. But I think also one of the things that I want to make sure that we add to the conversation, I sort of touched on it is, you know, I think organizations, I’m just thinking about actually the first private distribution only utility in North America is the Vermont Electric Company. And they, a lot of companies want to go through additional transformation, right? I know we’ve all heard that term, right? But they really set out a plan for a number of years to sort of think about how they could not only replace some of the legacy, but really sort of think about, you know, shifting the cheese on what does it mean to digitize the processes that they actually do.
So, I think, for some of the organizations, you know, they’re not only thinking about—okay, how do I put a new front end on an existing system where, you know, it’s been particularly inflexible. But I think also, a lot of them are a lot more, you know, thinking about how do I, you know, move the cheese? So, in the case of the D.C. DMV, how do I, for instance, say, “I don’t want to have people go into a motor vehicle registry to register their vehicle, right? I don’t need them to go into a public office to pay a parking fine or a speeding ticket—what if we just actually changed our processes and put it on one of these things, right?
So, I think a lot of organizations out there are actually thinking about low code in that way, and I think that’s where the conversation shifts from being an IT way and an enabler to also being—alright, you know, the CMO, the COO has a seat at the table and she is saying, “Look, how can we improve the engagement with our customers where our traditional ways of engagement with them have change significantly?” And I see probably equal amounts of companies coming to us and saying, “I’d like to do this,” right? You know, “I want to change the way people buy tires for their cars,” right?
Mitch Ashley: You know, in thinking about this, it seems that sometimes the terms we create don’t always serve us well, right?
Andrew Manby: Right, right.
Mitch Ashley: And, you know, visual programming is what we would’ve called low code maybe 5 or 10 years ago and you know, is a macro low code or no code? And actually, I don’t really care, it’s a macro, it’s just like—you know, is Python a scripting language or a programming language? I really don’t care. It sort of doesn’t matter. I think the end result is, you know, you talked about, Derya, the rapid application development, and I don’t mean the RAD term, but creating applications quickly and expanding the universe of people who can do that across the business is all good. I mean, it’s to the benefit, to the end of the business. And we can create a mess for ourselves, or we can do it well, in a way that it works together.
But I think, hopefully, we get to a point in a year or two where we don’t think about low code/no code, right, it’s just—it’s another way of, it’s an application platform for creating apps. And anybody can do it, because it’s—I mean, whether it’s no code, I don’t care if Trello’s a no code or a low code. I don’t care if Zapier is a low code or a no code, you know, just to pick a few commonly used things. I think we’re about the result we wanna get.
Andrew Manby: Yeah, I agree, Mitch. I mean, I think at the end of the day, you know, we obviously know that technology goes in cycles. You know, certainly, in our portfolio, we have low code that we would sell to a customer for doing that specific task, right, and those are broad sort of use cases.
But, you know, low code or that sort of no code citizen developer construct exists in our digital experience platform and it also exists in some technology we have specifically for the DevOps process, which is a low code version way of them to help orchestrate what they do in the DevOps process. And you think to yourself—well, are they a commercial low code tool? Well, no, but they’re bringing the, enshrining the value of a no code/low code tool to a specific challenge.
You know, maybe that will be the way that this market will evolve over time, but definitely, I would say that where we are right now is, it’s permeating into lots of different places, right? You know, does that mean that, say, is Monday a no code tool? But probably what makes it stickier is the fact that it has some no code type capabilities to it, right? And maybe that’s okay, right, for customizing certain types of applications.
Mitch Ashley: Well, it ventures into the whole workflow space, right? I mean, you know, Atlassian now is positioning themselves as a competitor to Monday and others as a workflow tool, and you could say that’s low code or no code, too. So, it’s really, when we talk about that as a segment of the market, it really is much bigger than just what we might narrowly define as low code.
Alan Shimel: Yeah. So, to me, the definition goes back to something Brian was talking about earlier, and that is—what’s the involvement of IT, right? Even with this citizen developer thing, I see it bifurcating. There are some cases where traditional IT teams are calling in a non-traditional IT person to help with the building, architecting, sizing of an app. And that non-traditional IT person benefits from the team being, using low code/no code tools, right? So, that’s one scenario with the IT Department’s involvement.
Then there’s sort of a shadow low code/no code development kinda operation without IT, right? Where—hey, man, it takes too long, I don’t wanna go through the whole change management process and the architecting, I just need this quickly. And I got the tool to do it and it’s cheap and it’s SaaS an it’s boom, boom, boom and I’m done. “Hey, where’d that come from?” “Eh, I built it.” “You built it?” “Yeah, I built it.” Right? And that’s powerful. I’m not gonna—let’s not kid ourselves. That’s a powerful enabler in an organization when the IT person says, “Hey, wait a second, where’d you get that?” “I built that.” Right?
How do those two jibe? And I think that’s really where the friction, where the rubber meets the road, and what is low code, what is no code, who does it? I think the delineation is, is IT involved or not? And, you know, Derya, you were saying most of your engagements are going through the IT Department. Brian, you seem to be on the IT Department is involved side of the house as well. Andrew, I’ve heard you go both ways here a little bit. But, to me, that’s the key sort of indicator.
Andrew Manby: Well, and Alan, can I ask you a question, then, right?
Alan Shimel: Sure.
Andrew Manby: You know, I tend to think about low code for citizen developers to be a bit like a sandbox, right? You know, I live—where I live, there’s a zoo a mile away, right, and there’s an area there and a kids’ playground, there’s a tent and there’s a sandbox underneath it, right? And it makes it very easy for the family to sit down and observe their kids, right?
So, you know, I think the way that no code is developing from an IT perspective, it’s trust but verify. Can I see what they’re doing? If they get into trouble, I can help them out, or as Derya said, you know, if there’s something that they need, right, maybe they need a specific widget to do a payment or they need something to deliver a package or they need some very specific type of math problem and they build an algorithm in the math lab and they want to bring it in, that’s when IT does that, but also, again, it’s that oversight and governance, right? Trust but verify, but you know, who’s using the application? What data do they have access to? Are they doing Select Star from HR executive salaries, right? Is someone curating those data sources for them so they can do no harm in their developments?
So, I think that’s the place where citizen development is going. I think that’s where organizations want to invest, that’s where they’ll put the COEs. I think the days of, say, my father-in-law building something in dBase will go away from an IT perspective.
Alan Shimel: So, let me use your sandbox analogy. I’ll put forth the proposition to you that most parents, a lot of mommies, take their kids to the sandbox and leave them there because it’s a relatively safe environment and they can concentrate on other things, maybe doing some of their own work or speaking to the other moms or what have you, and they consider it a safe environment to leave their kids, and they only keep half an eye on them versus, you know, being on swings or something. Same thing, here.
Andrew Manby: Exactly, Alan. So, I think, for me, you hit the nail on the head, right? If it’s about IT backlog, you’ve effectively given one strategy to halve or reduce your IT backlog because, with your supervision, right, the moms can go off and do something else which they focus on which really requires their skills.
Alan Shimel: Mm-hmm.
Andrew Manby: But then, you know, as citizen developers in the sandbox, I can get what I need done for the business faster. Isn’t it like a division of—
Alan Shimel: Well, let’s go try it.
Andrew Manby: – a division of labor?
Alan Shimel: It is sort of a division of labor because we’re not talking about kids in sandboxes. But when you have testers doing low code/no code, you know, building quick, being able to testing faster, better—that, to me, is a very different low code/no code use, right? That’s a—
Brian van den Brink: What we often see, for example, with Tricentis, Tricentis is a low code testing tool, and I think it’s important as a company if, for example, I choose low code development for my core applications. Then also, you have to probably think about what does that mean for the other roles within the IT Department like, do we have to think about then, also, a low code testing tool that fits our development platform, for example? Often, I think what I still see happening is that we choose low code for several reasons and the decisions that help that. Also, you can rely on—okay, that’s the same decisions we need for our test tooling, for example. And I think what’s important is that I think what the low code opens is that you can work with a low code platform and use those capabilities for a testing perspective as well.
So, for example, if I test a process, it takes a lot of time to go through all the screens and to test a workflow, for example. And I’m testing a workflow and there’s all kinds of data I need to test specific outcomes of that workflow, try to find solutions in your platform to basically mock that data or to set that data up through a faster way than to going through all those screens, maybe you can use an API or you can leverage something out of the platform to make that faster. And I think that’s where low code testing comes in place is that it’s easier to shift left towards a developer and discuss those kinds of things. Like—okay, we have a requirement we want from this business perspective, what do we need to build that as a developer and what do we need as a tester to test that more efficiently? And I think that’s where the thing comes together is that if we discuss that within a team and then it’s really an IT thing, what do we need from a technology perspective to make everybody’s life easier, basically?
And I think low code, companies use low code, for example, because they have a challenge in finding the technical people to be able to build their apps. And we have similar challenges in testing that it’s difficult to find the technical people to build the technical scripts. So, there, I think we need to make sure, as a company, that you match these challenges in the same way that—okay, low code platforms probably also need low code testers, because you have, you know, you make those decisions for a reason, I guess. And that’s what I see in my work.
Yeah, it opens a lot of opportunities as a tester to be more efficient in your work and to make sure that you keep up the pace with your developers. Because low code development is a lot faster than traditional development, and if you don’t change as a tester as well, you will always be, like, a bottleneck in the process, because you’re not as fast anymore. And then that’s why you need to think of smarter solutions as well.
Alan Shimel: Derya, low code apps need low code testers?
Derya Sousa: Sorry?
Alan Shimel: Brian said something that if you have low code apps, it makes sense to have low code testers.
Derya Sousa: Low code testers—yeah, yeah, absolutely. Just to add to the original conversation about governance and security, I would like to maybe add a couple of points there. I think, again, the solution to what you mentioned, Alan, there is—yes, we deal with IT Departments and, you know, business users are being exposed to these solutions, low code/no code solutions—where is the security, who has access to what, and how we can implement governance. And it sounds like a broad concept, but you know, any organization would need to implement governance, you know, how they operate, what is moss, what is a shoot, what is a can for them to do and now to do things.
And the same approach should be implemented when it comes to employing this type of low code/no code solutions. So, just because the platform is like a LEGO building block or sandbox, as you mentioned, it doesn’t mean that we should forget about everything, it’s too exciting, let’s build whatever we need and, you know, roll it out quickly. I think the same approach as the company would have had in their change management, change control, governance, they all should be now considered the same way for low code/no code approaches, whether it’s IT building it or whether it’s business users building it.
And what we usually see that’s working is when IT and Business Departments work together. So, IT creates some kind of a controlled environment for these non-IT users to go in and create solutions while IT is shadowing them or supervising them, giving them tools they need. But at the end, it’s all a controlled environment where, I’m really the biggest fan of giving people free, innovative solutions that they can freely innovate and create solutions they need, but this should not come without a plan. So, a low code/no code system should not be implemented without a plan.
So, that’s why we are seeing sometimes a struggle that companies go through, because they see the opportunities, they start with one solution and they want to implement so many more in a couple of months’ period, and it can be challenging if there is no plan. It can be challenging if there is no strategy, overarching strategy that looks to different kind of aspects of how to implement such a rapid application development system, yeah.
Brian van den Brink: Yeah, when we have—
Andrew Manby: I probably disagree, because I think there’s different modalities. I would say a lot of citizen developer low code is an either they develop it to prototype it and then hand it to IT once they’ve actually proven it, sometimes they’re there for a period of time so it’s literally like a pop-up booth or a food truck. You know, it arrives, it serves it purpose, and then it’s torn down, right? So, I don’t think you can necessarily have IT be involved in all of them.
I mean, certainly, you have some sort of, you know, diamond team to do that, but I don’t think it all happens like that. I think there’s different ways that, yeah, some of it is definitely planned out through IT and Business to deliver something. Sometimes it’s a cross functional team and it’s driven by the business. But I think part of the attraction of low code, and we mustn’t forget about it is, it’s that organic nature and the empowerment of the people in the business to get their job done.
And sometimes, you know, back to Alan’s point, it’s—you know, you don’t wanna get slowed down by what IT’s doing and as vendors, if we do our job correctly, we give them that safe space to be able to do that, right? Trust but verify.
Brian van den Brink: I think it’s important to—I think, indeed, what it opens, and I agree there is, it allows you to prototype things from a business perspective, but I think what’s important then in the next space is that that prototype is not to go in our production environment, basically. So, you know, you have to make sure that sometimes what we say is that, if you look at old spaghetti code, with low code, it’s easy to make, like, fast-cooking spaghetti, basically. Because, you know, you can make your mistakes faster because it allows you to create things faster, as well.
So, I think, indeed, you start with a prototype, you verify the basic concept, and then at some point, if you really wanna structurally have a core application out of it, it has to go through architecture principles and things like that. But I think that’s what Derya also means, maybe, with the plan is that, once it goes to that, after that first phase, it really goes into that planning phase and how are we gonna structurally broaden this application, for example?
Derya Sousa: Absolutely. Just to add there, so, the applications we build, we deliver, or our customers are delivering are core applications, core business applications, and I don’t see them working well in terms of security and governance if they are just let and there’s no plan or there’s no cooperation with IT teams.
So, from my experience, I’ve seen this and there are, really critical solutions are being built with low code/no code systems, and they are connecting to critical data as well, and especially with security in mind, that we all have to have nowadays. I don’t see this getting to the highest potential without IT being involved.
Andrew Manby: Yeah, I will agree about that, but I would argue, continue to argue that there are a lot of things that need to be put in production in a safe space that have been curated by IT that we just allow the business to run with, right? That’s just the nature of the beast, right?
Alan Shimel: Absolutely
Andrew Manby: And to be honest, I don’t know why you would want to change that.
Alan Shimel: Well, this is a great conversation, but we’re out of time, we’re over time. But, you know, obviously, if you couldn’t tell at home listening to this, there’s a lot more we could—we could’ve spent all day talking and still not cover it all. Maybe we’ll have to re-run it and bring it back for another low code/no code episode, soon.
But we’re gonna need to wrap up this episode of DevOps Unbound. Brian, Andrew, Derya—thank you so much for joining in and contributing to this panel, it was great. Mitchell, as always—good job, man, you know? You’re doing this—
Mitch Ashley: Enjoyed listening to the panel as well as participating. It’s a great group.
Alan Shimel: Absolutely. Many thanks to Tricentis for sponsoring DevOps Unbound. We’ll be back in two weeks with a fresh topic related to DevOps. Hope to see you there and have you join us, but for now, this is Alan Shimel for DevOps Unbound. Have a great day, everyone.
[End of Audio]
-
EP 19: How AI/ML are Used in DevOps Today
Artificial intelligence (AI) and machine learning (ML) help improve the performance of DevOps teams by automating repetitive tasks and eliminating inefficiencies across the SDLC. By using AI, teams can test, code and check software faster and more efficiently. In this episode of DevOps Unbound, Brian Dawson, Judith Hurwitz, Alan Shimel and Mitch Ashley discuss how AI and ML are transforming DevOps and what new use cases await AI and ML in 2021 and beyond. The video is below, followed by a transcript of the conversation.
[Intro Music]
Alan Shimel: Hey, everyone. I’m Alan Shimel, CEO of MediaOps, devopsy.kinsta.cloud, Container Journal, Security Boulevard, and you’re watching DevOps Unbound. DevOps Unbound is sponsored by our friends at Tricentis, so many thanks to them. And DevOps Unbound is a biweekly show where we cover topics of interest to the DevOps audience.
I am the host. My cohost is our CTO and CEO of Accelerated Strategies Group, my friend Mitchell Ashley. Mitchell, welcome.
Mitch Ashley: Thank you. Good to be here as always, and with this illustrious panel.
Shimel: Absolutely. And an illustrious panel it is. We have two panel members joining Mitchell and I today. Let me introduce you to both of them. They’re both great folks in their own right and I’m going to let them introduce themselves.
Let’s start with our friend Judith Hurwitz. Judith, welcome to DevOps Unbound.
Judith Hurwitz: Thank you so much, Alan. It’s a pleasure and an honor to be here.
Shimel: It’s our pleasure.
Hurwitz: So I’m Judith Hurwitz. I’m the coauthor of 10 books. Have been in the industry for 30-plus years. Focus on everything from DevOps, security, manageability in cloud, hybrid cloud, and really looking at how you take technology to transform organization. It’s a complex topic, not easy, but it’s what we’re in the middle of.
Shimel: Absolutely. Then last but certainly not least, our friend Brian Dawson. Hey, Brian. Welcome.
Brian Dawson: Hey, Alan. Thank you. Good to be on with you again. To tell the audience a bit about myself, as I’ve told you before, Alan, I’ve been in software development and delivery for about 30 years. I consider myself a technologist. And even prior to focusing on DevOps I’ve had a focus on optimizing software development and delivery.
Excited to talk about AI. I kind of dabbled and dipped my toes in the space in my time that I spent at the company that is now PlayStation. Things have come a long way since. And during that time, I’ve spent the past 10 years of my career focused on identifying, applying, and spreading DevOps practices, so I’m excited to discuss the two together here with this group.
Shimel: Absolutely. Brian, I’m not sure if you mentioned your present position with the Linux Foundation.
Dawson: I did not actually, because also this organization I’m at today has a big footprint in the AI and ML space. So today I’m with the Linux Foundation. I oversee our developer relations and ecosystem development. And of note, within that is the LF AI & Data Foundation, which houses a number of impactful projects in this space.
Shimel: Thank you. I just thought the folks at LF would welcome that, a little plug for them.
Dawson: Well I appreciate it, thank you.
Shimel: Yep. Alright. So the topic for today’s DevOps Unbound is artificial intelligence and machine learning. How can they help improve performance of DevOps teams? In other words, what’s their role in the DevOps world? Right? And let me preface our conversation by giving you sort of my view of it. DevOps for many people was all about automation.
What can we automate? Let’s automate everything we can. Automate, automate, automate. And obviously in discussions around automation topics such as artificial intelligence and machine learning you would tend to think help with automation, and therefore anything that helps with automation is good for DevOps. And I should also say that we lump AI and ML together like they’re Siamese twins joined at the hip, and they’re not necessarily cojoined.
You can have AI without ML and you can have ML without AI. At least I think so. I’d be interested in your thoughts. But has it lived up to the hype? Will it ever live up to the hype? Was the hype unfounded? Was it unrealistic?
What role has AI and/or ML to this point played on DevOps and what will it play going forward? I think that’s our topic today. Any one of you, if you want to kick it off with your own feelings or respond, go right ahead.
Dawson: Well I’d like to jump in as a troublemaker early on and say I challenge Alan that DevOps is really about automation as kind of the basis for the tie-in. If we look at it, DevOps is really a set of cultural practices and tenets that align DevOps and other software delivery stakeholders around the shared objective of delivering quality software reliably, rapidly, and repeatedly. Now the reliably, rapidly, and repeatedly component absolutely lends itself to some of the benefits that we could mine from AI, but I actually think it’s a great opportunity. If we don’t say DevOps equals automation equals AI, or rather we say DevOps equals a culture aligned around a shared objective and how can AI and ML help support those shared objectives.
Hurwitz: I think you make a great point, Brian. I think one of the most important issues, and you mentioned culture. We have been for, what, 50 years through this development and operations perspective on how we get things done, and the promise of AI is, okay, I can push a button and it will take care of everything, and this promise has been there for, what, 20 years. The reality is it’s just not that simple. There are definitely things that we are doing today and we’re seeing evolve that is definitely helping the developers and the operational professionals.
For example, if you have repeatable functions that happen all of the time, that, for example, if you press a button and a certain task should happen you can probably use a model and collecting massive amounts of data to do that automatically. And that’s a very good use of AI. We’ll work today – they call it MLOps or AIOps, and there’s a good reason why that’s what we’re focused on today, is because you’re looking for predictable patterns and predictable anomalies so you can avoid making stupid mistakes that, oh my god, why did I do that? I knew what the right answer was.
But AI is not a panacea. And I think if we look back even two or three years ago, you had companies that all were saying, “Okay, we have automated, we’ve put AI into DevOps, and all you have to do is press a button and all of your problems are over.” That’s just not reality.
Ashley: There’s the marketing AI and ML and then there’s real AI and ML. Marketing meaning everybody gloms onto the term and uses it in, let’s say, a case statement or an if statement in people’s logic and their software. Yeah, I started doing some work in AI in ’80s and ’90s programming in LISP and PROLOG and doing some corporate education in that area, some expert systems with triage, things like that. So I dabbled in a little bit there and then I was exposed to more recently about five or six years ago, worked with a gentleman named Dr. Bernard _____ who’s one of the experts in the field. And I asked him, “Help me understand why has machine learning taken off as a service.”
He described it as a subset of AI, so that was his model anyway. What he told me really made sense and it kind of helps me understand where can we apply machine learning, and being kind of the most popular part of AI. He said machine learning takes massive amounts of data, and the fact that we have the cloud and all these applications creating data, a lot of the data exhaust, now people are going back and mining that data. So Judith, in your writing, in your books about data analysis and leading up to AI, machine learning is great for that because you can have supervised or unsupervised algorithms, supervised meaning that’s a cat, that’s a cat, that’s not a cat, or unsupervised which is just going to pour through the data and start to look for those patterns and trends and anomalies that you were talking ab out, Judith.
So when people talk about using AI or ML, in our industry I always think about where’s there’s lots of data and can we leverage it in some impactful way? So as I look at products or technologies that claim they’re doing that, that’s at least one sort of criteria to sort of ferret out. Is it real there? Is there something real there? Or is it more spin and fluff to help us – you know, we all have case statements in our spot, right?
Shimel: Absolutely. Actually, Brian, it looked like you were going to say something. I didn’t want to jump on you.
Dawson: Alan, of course I have a lot to say, but nothing in particular. Since I’m off mute, I’ll say I absolutely agree with Mitch and Judith and underscore the points that they made.
Shimel: Yep. You know what? I want to take a moment though and explore. Brian, you disagreed; DevOps isn’t about automation. And I get the whole cultural aspects of DevOps and all of that, but certainly automating as a way to be more efficient, to get more done faster, I think is part and parcel of the DevOps mindset.
Dawson: Yeah. And if I may, without waiting to see where you wanted to go with that, I’d say it is absolutely important, but there’s a line Jez Humble would use in talking about CD. And frankly, I think the book that Dave Farley, Jez Humble and team wrote around continuous delivery is undercelebrated and under referenced. And in talking about that, one of the things he said is continuous delivery, and I’m going to say by extension some aspect of DevOps, don’t require any tools. I can do continuous delivery with a Bash script. The catch is is it’s not the most efficient in pursuit of effective collaboration, of delivery velocity so you can iterate.
And as we’ve talked about in this past established, that controlled feedback loop, that requires things like automation to achieve those goals. Right? I just think sometimes when we talk about what we see going on with AI and ML and those terms we overload a principle or practice with expectations that aren’t core to it, and I worry a bit about that so I wanted to call it out. And sorry, Alan, I don’t know if that’s where you wanted to go with the commentary.
Shimel: No, no, no. I want to go wherever you want to go, Brian.
Dawson: Okay. [Laughs]
Shimel: This is a collective. Judith, what about – I’m sorry, go ahead.
Hurwitz: Oh, yeah. So Alan, I think one of the key issues is the whole area of continuous delivery. In previous generations, you would build an application and it would live pretty much as it was written, could be for 10 years. Today, applications are constantly having to be revised because customers change, partners change, the sources of data change. So unless you have the ability to constantly update and change and modify, you lose out. One of the values of using automation and using AI and machine learning models with lots of data is to support the ability to do this.
You don’t have enough hands and enough brains to anticipate where problems may occur because you’ve changed things. How many times have we seen problems occur because somebody has added a new service to their environment and somebody forgot to change a configuration file? Now that’s not something that you need a massive brain to be able to do, but people get busy and they forget to do simple things.
Ashley: I think, Judith, that’s a really good point, because I wanted to ask you about this in the software creation process. Given things are so dynamic now and constantly changing, understanding just the environment, the infrastructure-as-code all the way up through the application and how much all of that is changing, it seems like stepping in for humans in certain conditions in that environment is a great application of AI. For someone to synthesize all the factors that might go into where a problem exists or what might be causing a problem seems to be where some of those algorithms might be helpful. Do you agree with that? Has that been your experience of what people are thinking about for AI?
Hurwitz: Yeah, definitely. I think what we’re seeing is this is where sort of the human factor comes in. You set it up so that if the printer is turned off don’t send me an alarm, fast, emergency, there’s a problem with the printer. We all have gotten used to that. But when there are problems that you’ve never seen before, that you don’t have data on and the model doesn’t take into account, then the AIOps or MLOps then gives you a message there’s something strange going on. I don’t know how to fix it.
What do you want to do here? And so over time when that appears again, well now you had data this occurred once before. Maybe it was an anomaly twice. But then when you have enough data and enough experience over time then you build that into the model, so the next time that occurs you make a fix. On the other hand, you don’t want the system to say, “Oh, I know what that is,” make a fix, when it turns out no, no, no, you don’t understand the context of what’s going on here. And just because there’s a correlation doesn’t mean that there’s a cause for that.
Shimel: That’s a good point.
Ashley: It’s not always about automating a response to it, right? Everyone in the security world are very cautious or skeptical about those things happening, blocking legitimate traffic. In the financial world that’s a huge issue that we run into. So you point out a good scenario, whereas we see repetitive patterns over time and that can help machine learning algorithms now understand, okay, that’s what that pattern looks like. So you can identify what it is next time instead of it’s just an anomaly.
Dawson: Yeah, it’s interesting, Mitch, that you say that. And going back to the starting topic about AI and ML not really achieving its promise yet, and as you called out getting data as an issue, it’s not necessarily applicable to every space. You need a level of determinism and predictable patterns that you can learn from and build on. And I have for a long time been excited that when we look at CI and CD, when we look at, Alan, the automated workflow of software development and delivery, or even if it’s not automated kind of the standard, there’s a couple of things that you do have.
You’re doing builds, sometimes thousands a day, builds and deployment, getting that out into prop. You could be generating a ton of data. And also in its nature, you’re striving to achieve a level of consistency and repeatability in how you deliver software. And I’ll say I’m not an expert to the level some of the people on this episode are, but I get at a very top level get really excited about the opportunity for AI and ML to help better CI and CD and align with the principles of DevOps. What I see it really helping is reducing cognitive load so developers can focus on coding, innovating, and solving problems while helping ensure that quality and stability that’s difficult to maintain while you’re moving fast.
So just thought I’d call out, and I am curious to see when vendors and others really start to do like what our friend Kohsuke has done with Launchable, and really apply AI and ML to the left-hand side or Dev side of the process to reduce the load and ensure quality and stability.
Shimel: Guys, we’re all lumping AI and ML, AI and ML, AI and ML. Does it have to be AI and ML? In my experience, a lot of what we call AI and ML is a lot more on the ML than it is on the AI. Right? So it is fair to call it truly AI? Is it really ML? And maybe ML is for today and AI is a tomorrow thing. I don’t know.
Hurwitz: So I think there’s definitely a problem of nomenclature here. What we’re really dealing with primarily now is models and modeling data and creating models from data. That’s the reality of today. I think that there is a lot of misuse of the term AI. There have been some absolutely wild predictions. I can’t remember the name of the computer scientist who predicted that he would be able to replicate the human brain –
Ashley: Marvin Minsky.
Hurwitz: – with AI. No, this was past Minsky.
Ashley: Oh, was it after? Okay.
Hurwitz: Yeah. This was like in the last five years.
Ashley: Oh, okay, okay.
Hurwitz: So over time, you’re always going to be dealing with models. You’re never going to get rid of the models. AI is sort of the next – models are a subset of what you eventually achieve with AI. But AI is really a concept that will probably take decades to evolve, and I think one of the problems we have right now is where you have vendors, and I’ve talked to hundreds of them over the years, where they say, “We have an AI application,” because it’s a hot buzz word.
And you can look back and in the history over the last 30 to 40 years and see whatever the hot topic is all of the vendors say we do that. So I think that that’s one of the problems that businesses are facing right now. What is the difference? What does it really do? How does it make your company better? How do you use this technology to be prepared for change?
Ashley: I think that’s a good way to break it down, Judith, too, because early days of AI were about emulating human thinking. We’re still long from – I’m not sure – all my intelligence is artificial, by the way. I acquired all of it. I don’t think I was born with any of it. But that was kind of where it started, and then AI became about expert systems largely. And then algorithmic learning, machine learning algorithms, was really where I think most of the activity is today because of that prevalence of data.
It seems like most organizations are faced with sort of one of three strategies. What do we look for from our vendors and how they may use AI in a meaningful way or machine learning in a meaningful way that’s going to help my business? Do I build models myself from the data, like you were talking about, Judith, to apply to like a financial analysis kind of situation? More of an expert system. Or do I use machine learning algorithms in my own software to do interesting and valuable things that my software can do? And maybe you play in all three of those or a subset of those, but that seems to be the question I think most individuals or organizations are at.
We’re not at a place where we can go build models, but we’re looking for these capabilities either in our own code or in third party products.
Hurwitz: So I heard an interesting story from one of my clients a few years ago, maybe five years ago. He had a client that was very gung-ho about AI. So he went out and he hired five data scientists, paid them $1 million each, gave them their own space, and left them alone. These are the smartest people on the planet. Came back in six months.
He didn’t want to bother them, they’re so smart. Alright. What have you found out? Have you written this application? And they said, “Well, we’ve been discussing this for six months and we have determined that algorithm we’re going to use.” The point is that they worked in isolation, they thought they were the smartest people on the planet, they did not talk to people on the business side about what business problems they had, they didn’t talk to the people who understand corporate data, they didn’t ask them what data do you actually have, what data do you need, they didn’t talk to people who knew the business strategy or the business processes that were in place or needed to change.
So it really is a team sport, and I like Brian’s discussion in the beginning about culture because it really is about sort of hybrid organizations where you have leaders that know a little bit about all of these areas and then a team that’s brought together that can work across these areas.
Shimel: Fair.
Dawson: Judith, can I ask do you have any recommendations for DevOps teams on how to evaluate or investigate where ML is a solution? And I ask this based on observation – we’ll see if we agree – that oftentimes people are looking for a problem to apply the solution to as opposed to for ML as a solution to a problem they’ve identified. I’m just curious if you –
Hurwitz: I violently agree with you. So I think a lot of times people get so enamored with new technology that they look at it as a way to solve all problems. And for a DevOps team I think we’re finally getting to the stage where there’s really reality in DevOps. It’s not the developers who are saying, “That’s not my problem; the operations people have to make this work.” There’s really beginning to be this collaboration between development and operations, and this shift left is definitely becoming real.
So that’s definitely true, but for these teams to be successful they have to have a holistic view of where is your business going? What do they actually need? Why have they come to you and say we’ve go to do something? Is it just because it would be cool to do something and spend money or is there a real rationale behind this that they need? What’s the pain that’s out there that they can solve.
So they have to start with that. They have to start not only collaborating between developers and the operations team but with the business leaders, with the people who understand all of the data, people who understand security. So all of this comes together in very much a holistic pattern.
Shimel: Fair. Fair. Hey, guys, we’re way past halfway through here and I wanted to kind of turn our conversation to the future. We’ve had a discussion on sort of the history of AI and ML and lessons learned, et cetera, but when we look forward, Brian, you mentioned the Linux Foundation has, I don’t know if it’s a daughter foundation or a subgroup dedicated to AI and like technologies. What do we see when we look sort of near-term future? Forget, yes, one day we’ll mimic human brain patterns. Who knows if we’ll be alive by then. But near term, what do you see? What is Linux Foundation planning for?
Dawson: Well, so looking forward, and it’s funny because you brought me in and I started to think about dream the impossible dream of what’s going to happen in the future, I will call out and will mention again I said at the start the Linux Foundation is a parent foundation of what we call a Linux Foundation open source project that hosts other projects. So LF AI & Data, you can call it a sub foundation of Linux Foundation because they host, I believe at this point, 12 graduated projects with about 20 to 30 projects in incubation. So we’re talking 30 to 50 projects under the LF AI & Data umbrella that are all working on various aspects of shared efforts, multiple large commercial organizations as well as standards bodies coming together to drive the future of AI and ML.
What I do see coming out of LF AI & Data Foundation in the short term is standards and foundational implementations, i.e. moving sort of beyond the rudimentary discovery around AI, building packaged or gray box implementations that everybody agrees and collaborates on, which I think will help unlock the less initiated, less expert vendors to begin to deliver truly AI-based differentiating capabilities. So to put that in short, because it was a lot of words, is I think it’s about establishing a foundation for us to start to build on and accelerate our progress within the AI and data space applied to DevOps.
Hurwitz: And so Brian, I think that you’re 100 percent right, and that’s when commercialization really happens is when we have those standards and when everybody agrees to use those foundational services. I think the challenges that we face are how to get the commercial vendors who don’t necessarily want to – they want to give lip service to these standards but they really want you to only use their version of the quote/unquote standard so that customers never leave. So it’s that stickiness factor that they are looking for. So I think it’s a hard journey.
Dawson: Yeah. Well I think, Judith, you actually nailed the reason the Linux Foundation exists frankly, and if we look at Kubernetes as a model, it was hard pressed. Google could’ve easily said, “We’re not going to hand this over to the Linux Foundation. We’re going to dominate this space. We are going to be the modern cloud OS.” But they understood for it to gain traction, for it to grow and truly offer benefit industrywide, they had to hand it over to the Linux Foundation to grow and manage.
They had to bring in Microsoft, they had to bring in Amazon to play in that space. And I see LF AI & Data serving that same role, what I tend to call in some language unlocking innovation, or as our tagline is, decentralized innovation. I would beg to say that that is – we didn’t call that out directly, but that’s one of the challenges that we’re seeing in the AI and data space. If there’s not short-term monetization we’re not going to do it, and if there is then we want to own it. So can we create an impartial playing field for everybody to come in and innovate together, and then build commercial solutions off of that.
Hurwitz: Yeah, it’s a challenge. It really is. I think Kubernetes is a great example. I think data in some ways is more complicated because data is really the crunch.
Dawson: Yeah. Well I know everybody wants to own it. Everybody wants to own the data.
Shimel: No doubt about it.
Ashley: There’s also discussion about AI needs to become an engineering discipline as opposed to this sort of edge specialty. It doesn’t mean that it necessarily is going to be applying everywhere, but I think how we apply AI to certain kinds of problems. It seems to me the two most ripe areas are that highly complex environment. So with more and more infrastructure that’s automated, more things have gone to digital, how do you manage the infrastructure or manage the triage problem solving? And then of course the other areas, people will find _____, continue to find _____ where AI can be applied to gain competitive advantage in a certain domain or space.
And it seems like that’s the trajectory we’re on for quite a while. I don’t think it’s ever going to be AI takes over everything and it’s the thing that replaces DevOps or whatever. But it seems to me kind of a tool. Sorry. Go ahead, Judith.
Hurwitz: No, what I was going to say is very interesting. If you look at healthcare, for example, if you look at the spectrum from I can automate certain DevOps functions that are repeatable, where I can identify patterns, and as I collect more data I can automate more things, you have something like healthcare, which in terms of orders of magnitude of complexity are just huge. And what I’m seeing is a lot of the vendors who think that I’m going to tackle healthcare with AI and we’re going to own the industry, a lot of them are getting out of the business because right now it’s just too hard and it will be too hard for quite a while.
Ashley: That’s true. The medical industry in many ways [laughs] is a hard nut to crack.
Dawson: Yeah. If I may start to dream a little and chime in based on something you said, Mitch, I do see that in the near term we are at a point where we talk about modern software development and delivery, the rapid pace with which we’re delivering change. We’re building on inventions and progress made over decades. Library reuse is very heavy. We are getting to a point where as we build more complex systems we have to figure out how can we outsource sort of maintenance and management, for want of a better word, to use it kind of grossly to solutions like ML?
How can ML help us continue to improve, grow, and build on what we’ve done but manage the scale and complexity? And I think we’ll move from some standardization, foundational blocks, we’ll apply more ML in both the operations and development space to help manage that complexity to maintain stability, but then I also eventually see the next stage being now how do we apply ML and surface it to a developer at the time of a commit? Here’s the expected outcome of this change is, to help provide guardrails to the developer, the end state – or I wouldn’t even say the end state.
I think we’d agree existentially there’s never an end state. We’re at a point where we don’t even have explicit CI/CD pipelines. We can commit code to a repository, the language can be inferred, we can give cues or signals to infer where we want it deployed, how we want it deployed, and I actually see ML and to an extent layers of AI just figuring out what to do with code at rest. So if you flash forward in 10 years and we’re truly applying these to technologies at scale in the cloud, I’d just change a line of code.
That change automatically is delivered to production and AI and ML is helping us do that. So that’s sort of my dream. I don’t hit compile, I don’t hit build, I don’t build out stages and workflows. I just change code and that’s running in a system somewhere.
Hurwitz: So don’t you also want to have the ability to say before you say – you look at what’s happening, and don’t you want to sort of at least at this stage say are you sure you want to do that? Are you sure you want to delete that pile?
Dawson: Yes.
Hurwitz: So until we get to the point – because you have to be able to trust that the system is smart enough to understand if I make this change what cause will that unleash? Because we’re not dealing with a perfect world in DevOps by a long shot.
Dawson: Yeah. I do want that. I want protection. I want it to know where the vulnerabilities are and warn me, and I do think, Judith, that would be a prerequisite into this deploying code at rest or truly applying Dev AI.
Shimel: Fair enough. Hey, guys, we are just about out of time. I think this was a great discussion. If there’s one thing I could take out of it though is that the crystal ball remains cloudy in terms of how this is all going to play out, and we’re going to have to wait for it to kind of come into focus. But it certainly will be an important part of software development, of operations going forward, and it’s an increasingly important part. But for now though we’re –
Ashley: I think the crystal ball is a Magic 8 Ball. Ask again later.
[Laughter]
Shimel: Right, ask again – that’s a good one, Mitch. But for now, we’re going to call an end to this episode of DevOps Unbound. Again, thanks very much to Tricentis for their sponsorship. Thank you so much, Judith, thank you so much, Brian, for appearing here and we hope to see you on a future episode. Mitch, as always, great job riding shotgun with me.
Ashley: Good to partner with you. Absolutely.
Shimel: Absolutely. But this is it for this episode of DevOps Unbound. This is Alan Shimel. Have a great day, and we hope to see you soon on another DevOps Unbound as well as don’t forget every month we do a live roundtable open to you, our audience, with questions. So stay tuned for that as well. Take care, everyone. Bye-bye.
[Outro Music]
[End of Audio]
-
EP 18: Does Standardization Disrupt Innovation?
Although everyone has their own definition of DevOps, it all comes down to creating better software faster, empowering development and operations teams and delivering value to customers. However, there really isn’t a standard approach to DevOps.
Should we standardize DevOps to introduce automation, avoid confusion and speed up the process? Will standardizing DevOps foster or disrupt innovation?
In this episode of DevOps Unbound, hosts Alan Shimel and Mitch Ashley are joined by Jayne Groll from the DevOps Institute, Paul Bruce from Tricentis and Sarah Baker from Airbnb to discuss the pros and cons of standardizing DevOps, the challenges of standardization and possible solutions. The video is below, followed by a transcript of the converstation.
Alan Shimel: Hey, everyone, welcome to another DevOps Unbound. If this is your first time catching a DevOps Unbound, welcome. DevOps Unbound is a biweekly video series where we talk about relevant topics in DevOps. In addition to the biweekly show, we do, it’s every four to six weeks, a live roundtable show where we invite the audience to participate and drive the discussion with online questions and comments and so forth, so we open it up to a live audience. Unfortunately, this show is not one of our live roundtables, it is prerecorded, but we have a great panel and I think you’re gonna enjoy it.
I’m gonna get into today’s topic in a second, but before we do, let me introduce you to today’s panel. First of all, I wanna introduce a business partner of mine and my friend, Jayne Groll. Jayne, why don’t you introduce yourself to the audience, if you don’t mind?
Jayne Groll: Thanks, Alan. Hi, everyone. I’m Jayne Groll, I’m CEO of The DevOps Institute, we’re a global member association and a certification body around topics related to DevOps. I’m really, really excited to be here and I’m hope everyone listening joins us at DevOps Institute and becomes a member.
Shimel: Fantastic. Welcome, Jayne. Joining Jayne, I’m happy to have Paul Bruce. Hey, Paul, welcome, and if you wouldn’t mind, you know, introducing yourself?
Paul Bruce: Yeah, hey, Alan. Good to be back. My name is Paul. In my full time job, I work with Tricentis in a number of different areas. I’m historically a performance and reliability nerd. In my off time, I organize a number of events, help to co-organize volunteer things like DevOps Days Boston, just did o11yfest this year about observability, we had that conversation, I think, with Christine Yen a couple months ago.
Shimel: Good stuff. Mm-hmm.
Bruce: And generally, I’m a people advocate. I had to pull DevOps out of my LinkedIn title because there were too many recruiters that I don’t need, because I’ve got a pretty good job. But also, because people advocate really describes what I care the most about, right? People are some of the hardest part of the DevOps people, process, and technology, really, software. So, I really focus my time on that.
Shimel: That’s great to have you back on, too, Paul—good stuff. Next, we have a first time guest on DevOps Unbound today, a new member of the band, Sarah Baker. Sarah, welcome to DevOps Unbound, if you wouldn’t mind introducing yourself?
Sarah Baker: Thank you. Wonderful. Welcome. My name is Sarah Baker, my pronouns are she and her. I’ve been working in Operations my entire career and I’ve moved into DevOps as something I always—I needed a name for it. I always wanted the people part of Ops and technology and DevOps was that. And I’ve been doing that as soon as I learned the term. And I’ve been moved and started working with Paul Bruce in the IEEE work group 2675, who is trying to create a standard for DevOps.
Shimel: Fantastic.
Baker: I work for—and today, I work as the Manager of Biztech in Airbnb.
Shimel: Thank you, and thanks for joining us. That sounds like a great place for you to do this. And then the last member of our panel today is my co-host and another business partner, a long-time friend as well, Mitch Ashley. Mitch, if you wouldn’t mind introducing yourself?
Mitch Ashley: You bet. Always good to have a good panel on with you, Alan. Mitch Ashley, I’m CEO of Accelerated Strategies, an analyst firm that’s focused on DevOps, and also on cloud native, cyber security, and digital transformation. I come to DevOps both as a practitioner and a vendor leveraging it, so very interested in hearing the perspective of folks on our topic today. So, thanks for joining us, everybody, too.
And by the way, our DevOps Unbound is sponsored by Tricentis. I think you may have mentioned that, right, Alan?
Shimel: No, I think I didn’t, Mitch, but thanks for picking it up for me. [Laughter]
Groll: [Laughter]
Ashley: I was trying to give you credit where credit may not be due.
Shimel: Yeah, no, I don’t wanna take credit when I messed up. Yeah, DevOps Unbound is sponsored by Tricentis, and many thanks to them for their sponsorship. It makes all of this happen.
So, let’s turn to the topic at hand today, guys. You know, when I first started devopsy.kinsta.cloud eight years ago, I met Jayne maybe a year, within a year of that or a year and a half of that. One of the interesting things about DevOps is, there was no manifesto. There was no hard definition of what is DevOps and what is not DevOps, you know, contrary.
And ever since then, it’s been a double-edged sword. In some ways, it’s helped DevOps grow, it’s helped DevOps stay current over the last 8 to 10 years as the world has changed, DevOps has changed along with it. On the other hand, though, we’re still dealing with the, “So, what is DevOps?” Right? “What exactly is DevOps and what exactly is not DevOps?” which is sometimes harder than what is DevOps.
And as we sit here now, August, 2021, there really still is not a quote-unquote standard for DevOps. There is no standard DevOps. It is what you want it to be, right? As I think Andrew Clay Shafer said, “The DevOps you get is the DevOps you deserve,” right? But have we outgrown that stage? You know, that seems like a terrible twos stage of DevOps to some; to others, it’s what kept it cool. Has DevOps matured? Do we need standardization? Will it be something that kills the goose that lays the golden eggs?
We’re gonna discuss that today, and I’m interested in all of your views. Jayne, as the CEO of DevOps Institute, how many people have taken certified DevOps classes and certifications with DevOps Institute? It’s gotta be tens of thousands.
Groll: Yeah, about 25,000 so far. So, it’s a pretty large community.
Shimel: Mm-hmm.
Groll: And if you don’t mind, I wanna kind of just lay the stage, because I’m really interested in the standardization initiative. So, you know, I had a privilege of seeing, going to DevOps Days in 2012 in California. So, if you can imagine, it was in an old, dusty data warehouse, right, and it was at the very, very early stages. And so, one of the reasons DevOps Institute emerged was because there was a lot of really cool things here culturally, from an automation perspective, right, doing things faster, collaboration.
And so, we started up DevOps Institute because we wanted to make sure the humans were represented. And I love that Paul and Sarah in particular both represented people. The problem with that is, at that point, it was very, very open, it was very kind of organic. You know, it was really a group of passionate people that were trying to just change the way IT operates, right, from a people, process, and technology perspective.
What happened after that, though, is that there’s so much noise about DevOps that if I were an enterprise CIO right now, my head would explode, right? Now, is there one right way to do DevOps? No. Is it only gonna get you there with automation? No. You know, is there a very, very strong human element—absolutely.
So, it’s a double-edged sword, Alan, as you said, is that we wanna be able to kind of scale down, right, some of the common language, some of the common definitions. I always say I’m married to a CPA, they have generally accepted accounting principles; maybe someday, we have generally accepted IT principles. But how do we distill the noise, right, and everyone having a different interpretation of how to do DevOps and what it is compared to, let’s get past the, “what is it?” We’ve spent, what, 10 years trying to figure out what it is, to, “how do we do this?” Right?
Bruce: Mm-hmm.
Groll: And it isn’t just one roadway that takes you there. So, I think it’s a challenge. I mean, we introduced foundational certifications around DevSecOps and SRE and DevOps Foundation and DevOps Engineering, really, to create kind of a common vocabulary more than the, “here’s the single way to introduce CI/CD.”
So, I don’t know if that helps kinda set the stage, but I think it’s a really big challenge right now that, if we standardize it, we run into the risk of what happened with ITIL where it couldn’t keep pace, right, and it became religious. I come from the ITIL space, right, so it’s like, open chapter 1, verse 3 and sing a hymn before you introduce change management to how do we then, on the other end of the spectrum, it’s so open and so undefined that it’s really complicated for legacy, I call them heritage organizations, to be able to move from this step to this step.
Bruce: Mm-hmm.
Shimel: The church of ITIL. Paul, what’s your feeling on that?
Bruce: Well, everybody’s on a journey, right? Your personal journey, your professional career, whatever, maybe you go do a walkabout on a particular topic to learn more about that. And so, we started this venture, there was an early round of, “Who wants to be involved in something like this?” in 2016. I answered a call from Bob Aiello and Lynn Carter, who were kind of forming this idea of, there’s a bunch of people who really care deeply who have practitioner practical experience in the field. And the early part of this process was just learning, kinda level setting on what we have done, right?
Jayne mentioned that it’s hard, maybe, to have accepted IT standards, even though there are standards, right, that not every organization applies the same way. I don’t think that invalidates some of the general principles that we know to be true, which is, you know, you ship broken software, you hurt people, you hurt yourself, right? But those are high level. How do we break this apart and start to make sense of the world, as you said, the heritage, Jayne, you said the heritage organizations—what is so wrong, what’s going on that we have these kind of security problems, these performance problems, this inability to be competitive in terms of acceleration and stuff.
And so, I think the purpose of a standard is not necessarily to force people to do something a certain way. I think it’s simply just kind of like, if you’ve ever had the Christmas ornaments or just ornaments box that has the insert that allows for things to be put in careful places and also work together as a unit, right, to provide a normative, not a prescriptive standard, but a normative standard to say, “Look, these processes that you already know? Well, you already followed these from, let’s say, 12207, right?” These processes around supply acquisition, knowledge management, human resources? Right, human resourcing in DevOps is a big problem, at least in the Boston area. Try to hire an SRE—there’s only about 15,000 to 20,000 in the United States, legitimately. There’s a reason for that demand.
So, how do we think about the processes we already know and start to say, how are these things different because there are some accepted principles and practices around DevOps? Now, the practices are not things that you would ever wanna dictate in a standard. This is not the same granularity level as some other standards that dictate how to encode time formats. That’s very important at that level. But the level that we were working on was how to level set across the things that, especially organizations in high compliance and high risk industries, right, risk-averse industries—how do they take the practices and processes that they know from a process standard and start to go, “I understand how to plug, now, what the DevOps principles are into the standard to encourage both of those practices in a more predictable manner?”
So, when we did all this work—it’s taken years, right, of a lot of learning collaboration between a whole host of people I don’t have even time to list out, but a lot of people from a lot of various different aspects of software have a lot of experience in those aspects. You know, it took synthesis and refactoring, and I was up nights and weekends just rewriting some of the statements, you know? And each of these, it’s a line to 12207, so there’s multiple processes, process declarations like quality management, quality assurance, verification and validation.
Like I said, these project enabling processes and then these technical processes, and each of those process sections have outcomes, activities, and tasks. Things that are likely to occur in this area, right, as well as, regardless of how you do this, right, not dictating what the DevOps looks like, it’s outcomes of these things that are auditable, that not compliance but conformance. That you can confirm at various levels to these outcomes to say, if it’s important to get this right, when you hit the piñata, what comes out should be candy, not garbage, right?
So, I’ll pause there, because I wanna give Sarah a chance to kinda provide perspective, because she’s also been working a lot with us, for all those years. [Laughter]
Baker: I think I counted 27 people on the final author list for the IEEE.
Bruce: Yeah.
Baker: And it was many, many weeks of arguing, reading together the exact language. And I agree very much with some parts of what Paul says, and I wanna step back and talk a little bit about how I see, you know, what, to me, was the big shifts. And so, the way we wrote the doc was to kinda look at the 12207 and say, “How does DevOps change it? How has it changed the game? What are the perspectives that you need to have in mind in order to get the outcomes that Paul’s talking about, there?”
And so, when we were writing it, we were writing it to that. How is this different, and to what extent do you wanna consider these areas? It’s a framework of thinking. At its largest level, it’s a framework of, for me, the value chain of delivery of software. And it’s not—and it uses many of the concepts that come out of learnings of industrial engineering, way back in the ‘40s, ‘50s, ‘60s. But it had never really come into a lot of the thinking of software, because software is different. And the more I was working and kind of digging into what are the concepts underneath DevOps, I came quite aligned with a lot of industrial engineering concepts about how things flow and how things move back and what the peaceful aspects are in that.
And so, that—actually, if you read through the standard, you’ll see a lot of those flow pieces and all of the pieces that you then say, “Oh, it’s the feedback loops, yeah, lots of feedback. Collaboration, it’s all there.” And it’s just applying a lot of those learnings into software engineering, to me. And it was stuff I was like, “Oh, it’s so wonderful to have this.”
And the other thing, one of the big learnings that I think I brought into the conversation is how—and Paul will back me up on this—how security for a lot of us changed slightly as we started walking our way through the 2675 work. The terms of risk became a very interesting concept that flowed through the document more than I would have expected.
Bruce: Mm-hmm.
Baker: And so, there was some learning within the totality of our team as we wrote this. It was quite fascinating.
Ashley: I’m really fascinated to hear from both of you that have been working on this, because it’s not something I’ve been involved in. I sorta have a mixed history with standards, because I’ve worked in areas like networking and things like that where standards are very beneficial or really fundamental, but I’ve also learned standards are kind of what you’re gonna do, not how you’re gonna do it. Because we ran a test lab that, even though it says we’re gonna do least cost routing in a network device this way, people implement it differently and they don’t always work together. So, it’s not a nirvana, but it’s also not a pariah, either.
Bruce: Mm-hmm.
Ashley: So, what I wrestle with here is that, I think some of the thinking—and Jayne can speak to this, but innovation happens so quickly now. So many things are created without standards; maybe they evolve into standards, maybe they don’t. And it doesn’t bother me that there’s lots of noise around DevOps, because we’ve all seen the cycles, the hype cycles, right? If it’s worthwhile, everyone and their brother, sister, cousin, uncle will all jump on the bandwidth and call what they’re doing fill in the blank—it’s AI, it’s DevOps, it’s whatever. That’s just part of our industry. And it’ll either fizzle—you know, value stream management, is it a real thing? We’ll see. It’ll fizzle or it’ll happen, right, if it’s a real thing. And DevOps has proven itself not only is it a real thing, but other parts of the organization now see an opportunity to play, to participate, to become part of the software creation process.
Bruce: Mm-hmm.
Ashley: And I think that’s what’s so powerful about DevOps—it’s not about a technology, it’s about how we create software. And if you were gonna start out, you know, Jayne, when you went to that meeting in 2012 when you said, “We’re gonna go create a standard for completely doing software differently because we have cloud and we have the ability to automate and we were gonna change the software architecture,” you couldn’t have known how to do that. But being the industry would’ve, you know, the antibodies would’ve crawled all over all of us and stopped it. So, I think there’s a lot of power in the innovative, organic nature of it.
And I don’t want DevOps to turn into ITIL, [Laughter] because that’s sort of taken on a life of its own, and now it’s trying to catch up and it’s done some good things, too. So, I wrestle with trying to constrain it when I think it’s real power is the innovation behind what we’re doing.
Bruce: Yeah, so, before I—
Baker: If I could take a moment—
Bruce: – yeah, go ahead.
Baker: – I’d like to take a moment and call out something. Part of the problem with ITIL is like, it’s ITIL. And, you know, a gazillion managers said, “ITIL, let us get a hammer and hit you over the head with it.” And DevOps, fortunately doesn’t have that problem. It has, you know, a little fuzzier in its definition, but the great thing about DevOps is that it has that sort of fuzzy edges. You can’t, it’s really hard to beat someone over the head with it. You can try. I’ve seen it happen, you know? Everybody’s a DevOps engineer now, but it’s not necessary.
And what people talk about is not that, but the work, you know? It’s not, “We’re conforming with ITIL,” it’s like, “We’re doing DevOps.” The doing part is the important part, not the compliance part. So, when you say, when you hear people talk about it, they talk about it differently, and that’s the good thing.
I use ITIL, I just don’t tell anybody the word.
Jayne Groll: Which is a shame. It’s such a shame because, you know, I come from the ITIL space, and in the early days, this was not what the intent was to be. It was very much like DevOps in terms of, “Let’s help IT grow up,” right, in terms of some things.
And, you know, to what you’re all saying, you’re absolutely right. A framework, whether it’s a standard, is a guideline, right? It helps you with the journey, as Paul said. And so, if I wanna kinda check, check, check—you know, I’ve worked with some of the ISO standards, right? So, in the ITSM space, there was ISO 20,000, 27,000 for security, right? And it basically, like, in the one or service management says, “You should record all your changes.” It doesn’t say how to record it, it doesn’t—it’s a really great practice, you should do this.
The problem, sometimes, with standards—and I’m sure you know this and it’s a risk—the pure intention is pure. The ability to weaponize it, right, from the auditors and from others is a real risk, right? Because we’ve seen that with other standards as well, where it becomes more about the compliance than it is about how having it being a guiding light. So, I think there’s always a risk there.
And then the other risk is, you know, think about observability, right? So, observability has been trending, what, for a year, maybe two, right? Four—okay, but really mainstreaming, like, talking about observability as kind of a mainstream practice, right? And I will share with you, Mitch, to your point, when we launched DevOps Institute in 2015, we were not welcomed with open arms, right? We were vilified for trying to put some boundaries around this.
Now, you know, move forward seven years and I think we’ve overcome a lot of those and we have some that were detractors or really great supporters. But I think they were concerned that it would become weaponized or commoditized or whatever.
So, I’m just gonna say one last thing because I think it’s important. Commoditization of knowledge, which, this is kind of in support of the standard, right? Commoditization of knowledge where we make sure it’s available to everyone, some people see that as a negative—it’s not. What would happen if the whole world, right—every IT organization, every IT professional decided to take this standard and use it as a guiding light, according to the needs of their organization? They weren’t worried about the audit, they got trained, they shared knowledge—wouldn’t that be amazing? But that’s commoditization.
So, I think the risk of good versus evil on initiatives like this is real, but I also think that distilling it to everybody, kind of like accountants have done, right, is important because IT needs to grow up.
Bruce: Yeah, well—
Ashley: Well, and if you look at the accounting standards, they don’t tell you how to do it, they tell you what to do.
Groll: Right.
Ashley: It isn’t prescriptive in the how very much, so it’s very much like an IEEE standard or whatever.
Bruce: Yep. Well, first off, I forgot to mention when we first started—I speak as an individual contributor. If it isn’t abundantly clear, [Laughter] I don’t speak for the IEEE, and I don’t speak for the other people who have spent years contributing to this meaningfully.
So, when I say certification processes inherit all the exact same things you just mentioned as problems—weaponizing, actually low ROI maybe in some cases—we see these deficiencies and these outcomes that aren’t so great. And if you’re somebody who’s just gonna sit back and say, “I’m not willing to hold myself accountable” to stand up and say, “What does this thing actually mean like a real engineer should, you know, to start to break apart the problem and see what are the components, how does this thing work and how do I want it to work? How does it work best for the people around me as well?” If you’re not willing to do that, then I would call you the nay-sayer, you know what I mean? Like, not any one in particular, but that would be a nay-saying, like, let’s avoid accountability, let’s avoid doing what we know needs to be improved?
Continuous improvement is a huge element that is all throughout the entire standard. It’s one of the things that’s identified as some of the core principles in an ISO study that was done before we started in on this standard to see what are the common themes of people, when they’re really feeling like there is this DevOps mindset, this culture, these practices—what do they mean by it?
And a number of things came out. One was a high degree of automation, right? Okay, well, that’s very tactical. No matter how you do it, right—high degree of automation. Duh, right? How about effective communication and collaboration? How about shifting things and continuous everything? The notion of, that these things are not this one time and then three years later, you get to come back. No, no, no. If it’s a good idea now, maybe don’t bite off a three-year apple. Maybe take a bite, but know that you started in, there’s a continuous improvement process.
I can say, we mentioned earlier about a holy war that might exist when you start getting into what is DevOps and what is not DevOps. Like, I started—the best way I can describe to people, because like I say, half of my day is spent on and off of Slack with people who are actually doing, working this out in a highly statistical, large distribution across the country, you know, and sometimes outside the U.S., too, just time zone supply.
But I can say, for sure, after seeing the complete garbage Dumpster fire where nobody is willing to stand up and say, “Hey, aren’t these things common and standard?” And oh, by the way, not vendors that have something to push, right? I mean, that’s, I think, Jayne, where you were talking about maybe observability is now a thing. You know, the only reason why it’s a thing for the most part is because it’s in our face now because all the vendors are hash tagging the crap out of it.
Same thing happened with DevOps five, six years ago—everybody had to get on the bandwagon. I remember CEOs way back then going, “What is this DevOps thing?” Oh, man. You know, just talk to some of your engineers.
And what I can say, right, I have developed at least an internal personal perspective on what is not DevOps. So, if I can borrow the last 30 seconds of the mic for a second, here are some things I wrote as notes as I’m going back over the standard, re-reading it, going what I violently agree with from both others and myself, we did not have a chance to see this before it went live many times, right?
So, what I got from one of the higher order sections in Section 5 was this feeling of, like, what is not DevOps. And those things are burnout. DevOps is not burnout. You know, you’ve probably been burnt out if you’ve worked in DevOps before, so where is this coming from? Why is this the case? It’s not burnout, it’s not no help. You don’t get any help, go do this thing, you know? It’s not heroics. It’s not superheroes in capes, right? Single points of failure, as I call them. It’s not cliques—which, by cliques, I mean power silos. Huh-uh, there is no such thing as The DevOps Department, people. There’s no rogue decisions. There’s traceability in decisions. There’s accountability in decisions.
Leadership thrash—not easy to, if you’re really adopting this DevOps mindset and way of thinking about how to do things better, leadership thrash doesn’t really fit in. Toil and debt, right? One of my biggest things is, you know, I seek to eradicate toil, which is not easy. [Laughter] Just reducing it a little is nice, but it’s, and it’s probably pragmatic to say so, but really eradicating toil from other people’s plates and from your own, that’s a challenge, right? That’s kind of a life goal. And if we do this stuff together, right, if we say these are not the things we want, the question is how do we move towards something that isn’t those things, right, and that is closer to where we all align and agree, like, this is a better way of delivering software and doing it faster, right?
So, anyway, I’ll pause there.
Ashley: You know, one of the things I’d love to hear from both Sarah and Paul is, there’s always sort of that, “So what?” answer—question, right? “If we did this, what is the result? What’s the impact of it?” And I mean that in a positive way, I’m not being critical.
Bruce: Yep.
Ashley: If this standard was widely adopted, how would it improve? How would it—what things are we wanting to improve and change from where we are today? Can you describe that in any way to kinda give myself and others maybe an idea about that?
Bruce: I just did. People already implemented it.
Baker: Well, at the base, there’s the vocabulary, yeah.
Bruce: Sorry, people are—
Baker: At the base, that is part of it. Sorry. Paul and I always get [Cross talk].
Shimel: Sarah, go ahead. Paul spoke—you go.
Bruce: Yeah, yeah.
Baker: Vocabulary is that first piece, making sure we’re all talking about the same framework. And so that, when we’re exchanging ideas, how did you do it, how did you do it—you know, not any one way to do it is gonna it, but knowing what your options are and talking about it with others and trying something out and figuring out if it works or not in your environment is one way to get everybody on the same playing field. But if you’re not talking about the same thing, you can’t do that.
So, it’s having the ability for a large corpus of options to apply to your situation. To me, that’s one of the biggest things that it provides, and it also gives you ways of interacting with the rest of the business that is not IT, and kind of, “Oh, this isn’t—it’s not just IT. You’re a stakeholder, you’re a customer.” And it includes all of those pieces for the collaboration and really helps support it.
So, to me, that’s what the standard brings. It widens the organization’s perspective a lot, and allows that cooperation to seep into the organization across all of these different pieces. Because to me, that’s how it started as DevOps, DevSecOps, DevOps Observation Ops. I mean, it grows because that’s part of its DNA is to include.
Bruce: Yeah.
Baker: And so, it’s very natural that it has done that and it includes other parts of the organization into that basic thought processes. I totally could see DevOps marketing security—I mean, you see all that. That’s what’s, to me, the real advantage of thinking this way is that it includes the entire org.
Groll: Well, and first of all, utmost respect to Paul, Sarah, and the other 25 people that are working on this, because this is a gargantuan task, right? Trying to encapsulate something that creates a common framework, that has the purest intention and also, you know, if you think about the heart of why DevOps emerged in the first place, it emerged because we were not collaborating together, right?
So, you know, in the early days of IT—I always say I’ve been in IT longer than I really want to admit to—you know, there were 25 people that sat on the same floor, in the same office building, and we all knew each other and we went to happy hour. And then, you know, the tree grew and there were lots of branches and other things. So, you know, giving a central source of truth that everybody can kind of internalize, process, speak the same language—you know, IT, we have so many different languages, we’re like countries that are butt up right next to each other.
And I said, I have so much respect, because this is a gargantuan task. And Sarah, I agree with what you said, the purpose is really to take this need for collaboration and optimizing automation, right, so making automation essentially members of your team, right, and then taking that out into the business because every business is a technology organization, it’s—I’m sure you’re aware of the risks. You know, we’re not telling you anything that you’re not aware of, but as I said, I just have so much respect. Because it’s not gonna be an easy road, right? There are so many different opinions from vendors and from everyone who’s involved in this space that are going to have an opinion or whatever. And I think, you know, that’s something that everybody’s gonna have to deal with.
Again, the only thing I would suggest is build it in a way that it’s nimble—dare I use the word agile, so that—
Bruce: So, this is already done, right?
Groll: Yeah.
Bruce: Like, this is not a draft, this is not an initiative. This has already been done for the past four years and is now an official standard. So, the good news is, to your point, I think, about suggestions about how to do this, we are working on the second version of this draft already.
As soon as it was out, even before it was out, we knew there were things that we just couldn’t—we couldn’t spend the time properly addressing. Like I said before, there was a lot of refactoring, but there was a lot of careful avoidance of not already repeating what was already in a good security standard over here, or in a good risk management thing, or one of the folks that worked with us, Tafline Ramos, works at the ISO, right? And what is it, 29119, right, verification and validation, the processes—that was already stuff that’s already been, and great thinking and not slow, you know? Can fully be put into a modern, unicorn sneeze rainbows and fart sprinkles type approach as well as the large Fortune 100.
So, just to say, this is already—we’ve already put a ton of, it’s already out there, and the easiest way is, I mean, some people might be wondering, “Oh, yeah, but it’s a standard, so now I gotta pay for it?” Some standards, you do; some standards, you don’t. This one, IEEE, they cover their costs with the standards. The easiest way, if you wanted to get a free copy, become part of the new, the next version of this, right? See what we’ve done, poke holes in it, have collaboration, let’s work on it together, because that’s the point of this is to bring a framework of how we are working together in a more effective manner.
Jayne, I’m really sorry I cut you off.
Groll: No, no, you didn’t cut me off. Just a cautionary tale. One of the reasons ITIL kind of lost a lot of its momentum—and I’ll be very frank, I owned an ITIL training company for 15 years and I was a chief evangelist, but it took 8 years between version 3 and version 4. And so, it wasn’t built, it was a monolith.
Bruce: Mm-hmm.
Groll: And I think that was—you know, in the meantime, lots of other things happened, and I think there’s a cautionary tale. I’m not suggesting you’re looking at eight years, please don’t misunderstand, but this market is so—
Bruce: No, it’s probably about three, right?
Groll: Yeah, this market’s pretty fast. Sorry.
Bruce: Our target is about three, between where we officially started and now it’s out and the next revision, at most, would be another three, right, that’s our target. It could be earlier or later than that.
Think about, we’re all citizens of the United States here, right? Or anybody in Canada?
Shimel: Is that a HIPAA violation to answer?
Bruce: Yeah, right.
Shimel: Just kidding, just kidding, just kidding.
Groll: [Laughter]
Ashley: I’m not showing you my vaccine card.
Bruce: I ask because, I ask because—
Shimel: I didn’t wanna go there, but go ahead.
Bruce: – we all know what the U.S. Constitution is and where it came from.
Shimel: Mm-hmm.
Bruce: Do we think that it was perfect and never has to be revised? Absolutely not. That’s what the Amendments are for. And arguably, I people wanna go back to the original version, there are some serious problems with that original version. Same thing with ITIL, same thing—remember with, I think you might have heard about The Agile Manifesto. You know, that was a drunken party with a bunch of white dudes, okay?
Groll: I know. [Laughter]
Bruce: So, yeah, there were some good ideas that were encoded in there, but do they not have to be revised and worked out? Even Dave Thomas says agile is dead, long live agile. Like, what people have taken it to is not the point.
So, the goal of this is to really get a full perspective. To what Sarah was saying, it’s not just—I’ve talked with a lot of the DevOps Days, various different people from DevOps Days, from a number of different, like, The DevOps Enterprise Summit and stuff, and there are some people out there that are like, “No, DevOps is just about developers and operations and it’s about just these things, and if you start getting biz in there—no, no.” And it’s like, what a narrow perspective, because you know what, you had a good idea and then you screwed it up by not including other people, by not realizing there are meta learnings that come out of what you locally did—maybe, arguably, you did well—that apply broadly. And when they do apply broadly, they bring us back to the same premise, which is, we’re all in this together, right? We all win when we all win. As Tim O’Reilly might say, provide more value than you consume, right?
So, that’s where we intend on taking this. It’s not to turn into a bunch of agile consultants again or ITIL consultants, right?
Ashley: [Laughter]
Groll: Framework growers, that’s what we call it.
Shimel: Yep. Guys, I gotta pull the plug on ya, I’m sorry, guys, and ladies. We are out of time. I think a great way to end this, though, is—Sarah, Paul, where can people get more information on the standard from the IEEE?
Bruce: Sure. Google it, Google IEEE 2675, you’ll get to that page if you wanna buy a copy.
Shimel: 2675.
Bruce: I think without being a member, it’s like 99 bucks—so, look, you know, if you wanna buy it for yourself—
Shimel: It’s not breaking the bank.
Bruce: You probably also have, in your organization already somebody’s a member, your Risk and Compliance are probably already members to IEEE.
Baker: Yeah.
Bruce: And again, like I said, the best way to get a free copy is to come work with us on what does the next standard mean. It’s not a huge effort and commitment like Sarah and I have put in. [Laughter]
Baker: [Laughter]
Bruce: You can dip your toes in, see what makes sense, what works for you, and we love to—we include people in that effort.
Shimel: Great. Well, Paul, Sarah, Jayne—thanks for joining us on our panel today. Mitchell, as usual, great job. This is another edition of DevOps Unbound in the books, check it out. Go check out the IEEE standard. We’ll be back in two weeks with a fresh show and a fresh topic. Until then, this is Alan Shimel, be strong, be well—Tech Strong. Take care, everyone. Bye bye.
[End of Audio]
-
EP 17: Lies, Damn Lies and Metrics
In this episode of DevOps Unbound, hosts Alan Shimel and Mitch Ashley are joined by Helen Beal of the DevOps Institute, Viktoria Praschl of Tricentis, Frank Ohlhorst, principal analyst with Accelerated Strategies Group and Alex Hidalgo of Nobl9 to discuss how to use DevOps metrics effectively to improve delivery performance and help teams become high-performing with a culture of continuous improvement. The video is below and a transcript of the conversation follows.
Alan Shimel: Hey, everyone. This is Alan Shimel and you’re watching DevOps Unbound. DevOps Unbound is sponsored by Tricentis and we’re happy always for their sponsorship and helping up put this show together. DevOps Unbound is an every-other-week session with experts from around the world talking about relevant topics to DevOps. In addition to that, about once a month or every six weeks we will do a live roundtable which is open to you, our audience, to come on, come in, ask questions live, and really kind of steer the conversation. We do have another live roundtable coming up. If you check the devopsy.kinsta.cloud website you can register for that. Today’s conversation, though, is prerecorded and unfortunately that means I get to ask all the questions, not you.
But let me introduce you to what I think is an amazing panel we have lined up for us today. Well, actually, before I do that let me tell you what we’re going to talk about today on DevOps Unbound. So, it’s the issue of metrics and kind of the ghost in the machine, the dark side of metrics – lies, damn lies, metrics – whatever you want to call it. How do we measure success or failure? How do we measure progress in things, in areas like DevOps or digital transformation? What does success really look like? Our panel is going to talk about that today. Let me introduce you to them.
First, I’m happy to introduce Frank Ohlhorst. Frank, if you wouldn’t mind saying a few words about yourself?
Frank Ohlhorst: Sure. And I’ll be quick as not to bore people.
Shimel: Okay.
Ohlhorst: I’m Frank Ohlhorst. I’m a journalist, author, and consultant. And I currently work over at DevOps as a contributor, at Security Boulevard as a contributor, and I am also a contributing analyst. And that’s pretty much my story, other than I’ve got 30-plus years – I hate to date myself – 30-plus years in the IT field and I’ve worked both for the federal government, I’ve been a developer, and I’ve put a lot of effort into cybersecurity.
Shimel: Excellent. Thanks, Frank. Next, let me introduce you to one of my friends from the other side of the pond. I’ve known her now for eight years or more. The one and only Helen Beal. Helen, welcome.
Helen Beal: It’s a pleasure to be here as always and spending time with you and the others. So, thank you very much for the opportunity. I’m Helen Beal. I’m Chief Ambassador at DevOps Institute, and therefore one of the co-creators of ADOC, which is the Assessment of DevOps Capability, which I may mention once or twice during today’s session. I’m also chair of the Value Stream Management Consortium, and prior to all of that I was coaching in DevOps in a number of different companies across Europe and the Middle East for several years.
Shimel: Thank you and welcome, Helen. Next up, let me introduce you to Viktoria Praschl. Viktoria, if I mispronounced your last name I apologize. I just realized when I looked at it that I thought I remembered it, but maybe not. Go ahead.
Viktoria Praschl: It was totally correctly pronounced, so you did a good job.
Shimel: Thank you.
Praschl: Hey, everybody. Happy to be here today. I’m Viktoria Praschl. I’m with Tricentis, a software and quality assurance company based out of Austria and Vienna. And I’m responsible for our UP and sales and transformation team where we work a lot with customers on how they transform their organization when it comes to DevOps and quality assurance.
Shimel: Excellent. Thank you, Viktoria. Our last panel member today also is somewhat of a regular; he’s been with us a number of times. And that’s author, expert Alex Hidalgo. Alex, why don’t you give them a little background and welcome?
Alex Hidalgo: Thanks so much, Alan. Hi, my name is Alex Hidalgo. I am the Director of Site Reliability Engineering at Nobl9, which is a startup focused entirely on providing the best possible service level objectives, the best possible SLO platform. I’m also an author of Implementing Service Level Objectives, and I’ve been working in this space for – in some way or another the last two decades or so. My current area of research is really into how to bring reliability lessons from other industries into the tech world and what we can learn from others.
Shimel: Excellent. Thank you, Alex. And then, last but not least, my cohost for DevOps Unbound, my partner in business, Mitch Ashley. Mitchell?
Mitch Ashley: As always, good to be with you, Alan. And I love our panel. You all are fantastic; I’m excited to hear what you say. As Alan mentioned, business partner and cohost. I run Accelerated Strategies Group, an analyst firm that’s focused on DevOps, cloud native, cybersecurity. And also, I serve as CTO with MediaOps, the company that puts this event on. And both a practitioner and product creator in software, software as a service, cybersecurity for a number of years. I’m not going to name or number how many they are, so you’ll have to guess.
Shimel: [Laughs] Okay. I’m not guessing. But anyway. All right. Let’s get into this week’s DevOps Unbound. So, I remember a VC mentor of mine getting up in front of – we had launched a new company, he stood up and said something to the effect that in this brave new world we all live in everything can be measured and we measure everything, and it is based upon those measurements we determine who wins and who loses and by how much.
So, that was 20 years ago, maybe more, a little more than 20 years ago: 22 years ago. I wonder, is it still true? Or have numbers, like everything else, with fake news and all of that, kind of lost their meaning and lost their importance in today’s organizations? Helen, can you –?
Beal: I’m not sure it’s ever been true. From my experience with metrics I think what I’ve always observed is just people fall into one of two camps. So, either they’re not measuring enough, or they don’t have enough metrics, or they have far too many. I think there’s a cognitive load limit on how many metrics we can handle and we probably need about a handful.
I think it’s true that we need to have telemetry everywhere, which is of course a very popular DevOps adage. I think we need to have the facility to be able to get to the metrics that we want, but we have to be able to choose them and be able to think of them as an adaptable framework. So, for example, deployment frequency, of course a very popular metric in DevOps, is very useful if you’re moving from deploying every three months to once a week or more. But once you’re deploying several times a day it’s a much less interesting metric.
The adage I always use is if you can’t measure it you can’t improve it, which is a Peter Drucker phrase. And I do have a problem with that as well because I think you can improve things without measuring them. But the thing is that you’ll never know. And to me that’s the most important thing about measurement: It’s about being conscious about what you’re doing.
Shimel: Guys, any thoughts on that?
Ohlhorst: I think it also comes down to an issue of context. People can measure whatever they want out there and they often do, and they spend so much time measuring that they don’t take into account the context of the measurement. You can measure certain processes in the DevOps process and they really don’t matter when it comes down to it when you’re looking at trying to achieve a particular goal.
Shimel: They don’t. You know what, Alex, we’re going to let you go, and then, Viktoria, you follow up.
Hidalgo: Yeah, I think one of the big problems that I often see is that people end up trying to measure themselves against others. They hear “Oh, at my last company” – or “I heard at Google or whatever they’re able to deploy X number of times a day and let’s aim for that.” But why? Do you need to deploy that many times a day? Is it actually applicable to you or your application or your organization or your customers? Are you actually gaining anything from trying to do things the way you know others are? Every situation is unique. Every company is unique. Every organization is unique. Every application is unique. And that means you really need to take a step back and think about “What is right for us? What do we need to improve upon?”
Praschl: Yeah, that’s what I really wanted to say, basically similar, that you take a step back and you really need to look at what’s your baseline. Where are you and what do you want to achieve? What are the outcomes you’re looking for? And I think it’s very important: The metrics will change over time depending – as you mature your processes and as you mature in your organization you will have to constantly change and adjust your metrics which you are measuring. So, I think the key thing is to really initially take a look: “Where am I? And what are the most important initial metrics where I start with?” And then, constantly, basically changing and improving toward further goals.
Shimel: Great. I think – I’m reminded of the adage “Just because you can doesn’t mean you should.” And I think that’s where we have – and not just in DevOps, frankly, in IT in general. That’s where we run into problems. Because it’s digital almost by definition it’s very easy to grab metrics on all of these digital things, on all these. But just because you can doesn’t mean you should. And just because you can doesn’t mean it’s necessarily indicative. I think, as Helen mentioned, when you’re going to a release a quarter to a release a week, ooh, that’s big news. That’s good metrics. But – and to Alex’s point, not everyone needs to be Google or Amazon and release hundreds of times a day or whatever it is – dozens of times a day. And I think that’s where we get ourselves kind of wrapped around the metrics axle.
Ashley: I think it goes back to the question of why. If you don’t understand why you’re measuring something, then why do it? Because everybody is not going to be on the same page about how you want to utilize that information. I think about the VC quote that – I remember I was there for that too, Alan. And in a financial world winners and losers, how much return on investment, that’s pretty clear. But I also think there’s a lot of fear around numbers because people are afraid that it’s going to be used against them or it’s not going to be understood, what they really mean. In a continuous improvement context it’s really to – that’s an aid to help you get better and know if you’re getting better, and you have to decide if it’s the right things, which you’ve got to put that context, as Frank mentioned.
And we have to be on the same page about what it is we really – what does it mean to put a release into production? Is that really – is that a canary release? Is it a real – across every data center that we are in the cloud? What does it mean to do a release? So, you have to get to that specificity. We’re all in agreement that’s – and that’s important to us because we either want to measure ourselves by it or improve at it.
Beal: I think it’s been implied by a lot of the speakers already, but I just wanted to really want to say explicitly as well that I don’t think metrics shouldn’t be inflicted on people. They should belong to the team. The teams need to generate them themselves. You can set things like KPIs at a higher level and have them connect, but things like OKR should be really generated and owned by the team themselves. And that really speaks to some of the cultural elements at DevOps, which are much harder but not impossible to measure, but things like distributing of autonomy – sorry, distributing authority and having autonomy at the team level to choose and define and improve the work, and that move from leadership from being managers and leadership becoming coaches that are moving impediments out of the way.
I think, Alex, you were inferring this as well, the importance of the team-level metric.
Hidalgo: Yeah. One of the things I’d like to kind of bounce off there is I explicitly try to always use the word “measure” because “measure” means something different then “collect.” I think Helen mentioned earlier it’s good to have telemetry everywhere, and so you have numbers everywhere, you have metrics everywhere, you have time series everywhere. And you run into the multiple comparison problem where you don’t really know what you’re looking at anymore. You run into spurious correlations. Have you ever seen that website where it can track people drowning in pools correlates almost perfectly with the number of Nic Cage movies released each year and things like that? And that’s where you end up when you have too many numbers.
And that’s why I always like to say you measure, because “measure” means you’re taking the important signals out, and you measure flour to bake a cake; you don’t just collect flour. Otherwise, you’re going to not end up with a cake. And it has to be an intentional act, and this kind of – like Helen just said, you can’t inflict metrics on people. You need to let them collect it correctly and measure it correctly.
Ohlhorst: If we look at DevOps from the pipeline’s perspective it starts to become clear maybe where we should be measuring things or why we should be measuring things. And one of the first things that a lot of organizations run into when they implement a DevOps strategy is they run into bottlenecks because they did not consider certain things. So, perhaps the first place to start with measurement is to try to identify bottlenecks and then go from there. This way you’re improving your process while you’re getting some value out of the metrics, and you’re also establishing what type of metrics you need.
Beal: I’m glad Frank brought that up because I really think that – for me, there are two classes of metrics, though, that are the most important and underpin everything. And one thing I will say about the DevOps pipeline is in my opinion it doesn’t start early enough. There’s that fuzzy front end and I think we need to expand it right back to where the idea happens. But my duality of metrics is basically if we measure flow and we measure realization of value we really have a concept of what the customer experience looks like. So, we know how fast we’re able to deliver the things that they need and we’re starting to understand whether we are actually delivering the things that they wanted.
Hidalgo: And I just wanted to add: Frank’s point about bottlenecks is great, but we can get meta about it. We want to make sure we’re very carefully selecting what we’re measuring. And identifying bottlenecks might be a great start but we need to carefully select which bottlenecks we even want to address in the first place. Just because we can see that something slows down somewhere it doesn’t mean that that’s necessarily an area of focus.
Beal: We could be waiting six weeks for a security check and that’s really hard to get that out of the way, but we could maybe – and Viktoria may be having an example around, I don’t know, test automation or something. Viktoria?
Praschl: Yeah, I mean, also test automation and _____, you have tons of metrics you can actually measure just from a curious perspective, and if you don’t measure the right things there as well it will seem as if it’s at a standstill. But I think it’s also very important to – I liked the bottlenecks you just mentioned, Frank, because one is the team level, but then you also want to see how does it look from an end-to-end process? How does my overall customer journey look like? And I see quite often companies, actually, they start and they identify the right metrics on the team level, they get the teams there, but then they are failing on the overall picture to bring everything together. A lot of people struggle with that. That’s what I see.
Ohlhorst: I think the mistake that a lot of the teams make nowadays is they either look at this and they try to relate either DevOps to a waterfall ideology where there’s a beginning and an end and you can measure certain points or they look at DevOps as this rigid structure on how you do things, where the whole idea behind DevOps is it’s flexible and it can be easily modified to meet specific needs. So, I think people run not only into an issue of where “Okay, where do I establish metrics? Why am I doing metrics?” but also how you interpret those metrics. I mean, like we were talking about before, release frequency and things like that are great for what we use as a term is “vanity metrics.” It’s “Hey, we accomplished something.” But when it really comes down to it, what did we accomplish and how did that improve the process? And I think people are losing sight of that.
Beal: It’s been said already, I think, here but I think that focus on outcomes is really, really key. And I think there’s a couple of other things I wanted to bring up. One is that we talk a lot about being data-driven in DevOps and I’m starting to think actually about being more insights-driven. And the other conversation I’ve had recently is about doing less hypothesis-driven development and more impact-driven development, which I think is another really interesting iteration on our ways of working. And it reminds me of when Alex was talking about this idea of having telemetry everywhere, which is wonderful. And so, the idea then is that you can get any metrics you want. And you don’t want all of them but at least you have the ability to choose them. But as Alex pointed out, there is this downside to this, that we are just overwhelmed with data. And I think that’s something that AI can help us with there as well. I mean, it’s just a fact: There’s far too much data in the world for us to cope with. We need somebody or something probably that we’ve created, a machine to help us with it. So, Alex, I wonder what your thoughts were on AI in the context of measuring our journeys.
Hidalgo: You’ve asked a very interesting person about this because I’m actually fairly dubious that a lot of these models will actually help. You need a lot of data, a lot of data, to train a model. And most people actually don’t have that much. We may be talking about “Oh, we have too much data” and “We’re swimming in data” and this and that. Very few people have enough data to properly train models. Can this work in some contexts? I think so. And I think a lot of smart people are working on the problem. But I do think there’s a little bit of an overblown focus on it today because I think it’s a lot of trying to fit a square peg in a round hole. We need to – like everything else, as we’ve been talking about, you need to be careful and you need to think about what outcomes you’re really looking for and then decide “Is AI, is machine learning the right thing here?” Because it – just because it’s a buzzword, just because lots of people want to do it doesn’t mean it’s always going to work. You need to make sure that you’re picking the right tool for the right job.
Ohlhorst: Yeah, and the same can be said with automation when it comes down to it. If you don’t know the possibilities or the probabilities around a process how can you automate it unless you have the data to provide you with that insight?
Hidalgo: Which, I should say, those words just remind me of what we can do with data is statistics. And actually, when you see me flinch a bit about “Oh, should we use ML?” or “Should we use AI?” I think a lot of the places where people are doing ML they’re actually just using stats. And that’s great because statistics can actually compensate for not having enough data. That’s kind of what it does best in some sense. So, I am a huge fan of using models. I’m a huge fan of using math. Just, again, let’s make sure we’re using the right math. Do we need to build an ML model if a simple regression tells us everything we need to know?
Ashley: Yeah, I do agree with you about the AI, machine learning of it. It takes a ton of data, if that’s a measurement, in the data world. I want to go to something that you said, Helen. I really like the word “outcome.” What are the outcomes that – have we achieved an outcome when we’re developing software or creating an experience or something out of the process? Because ultimately, that’s what matters. We can improve the things within it. We can automate the processes within how we create software. But it’s all for a purpose. It’s all for – if we’re not doing a better job of delivering on an outcome, maybe those things we’re working on internally are sort of navel-gazing, if you will, looking internally but it isn’t really that valuable. Let’s work on the things that we can connect to an outcome. That’s what I would propose.
Beal: It’s funny. The other day – it was at DevOps Enterprise Summit and she went up to speak because she used the phrase “time to market,” and I kind of in the Slack chat went “Oh, time to value,” because obviously there’s no point in getting something out the door if it’s useless, whereas with “time to value” at least we’re recognizing that it’s been – giving something useful to something. And then, John Smart corrected me with “time to learning.” And I thought, “Oh, no, actually, it has moved.” We’ve moved from time to market, time to value, and now we’re in time to learning, where we’re recognizing that actually only – something only becomes useful when we’ve seen it have value and we’ve learned from it and decided to do something else.
But yeah, it’s the same as this kind of impact-driven development as well, this idea that we really focus on having an idea about what we’re trying to do, and then, importantly, measuring that it did what we thought it was going to do. And we’ve been traditionally really bad at doing this in the technology world because we’ve been really project-driven. And when we’re project-driven we have huge amounts of requirements which are (a) really difficult to track, the impacts of, because we deliver them all at once and we weren’t doing things like canary testing which does allow us to look at a much more granular level. And the other problem with being project-driven is we’re always under so much pressure to deliver. Our testing always gets squished because our development overruns and our launch date never moves. So, by the time things go live we’re so exhausted, and then we disband the team and put them into other places anyway, so no one ever goes and has a look and says, “Did that really deliver the four million quid that we – or $4 million that we said it would?”
I wonder what other people think about how metrics change as we move from project to product.
Shimel: So, I’ll quickly jump in here on that, and that is – and it’s, Helen, something you mentioned, Alex mentioned. I think what we’re guilty of – and that’s part – in the abstract we talked about the dark side, and it’s not that “Mitchell, I am your father,” but there’s a dark side of these metrics which are that in the absence of facts or in the absence of knowledge we try to fashion metrics to compensate. So, with something like DevOps where there’s a lot of cultural aspects that come into play, there’s a lot of intangibles that come into play, and they’re almost by definition hard then to measure because they’re intangible; they’re not tangible. And so, we compensate by coming up with what some may consider frankly nonsense metrics that make us feel better or make us feel like somehow we are measuring the immeasurable. And I think that is an important part of this whole thing. Measure what you can, but are there things that are just immeasurable? Anyone?
Beal: Yeah, I – sorry, me again for a second. I don’t believe so. As a coach and a consultant for a long time I was discovering clients that felt that their biggest problem with deploying DevOps or implementing DevOps was the cultural side. So, that seems like something that you can’t measure. And it’s true that you can’t really get data out of Jenkins to tell you what’s happening in your culture but there are other ways to do measurement. And I think it was Viktoria – sorry if it wasn’t Viktoria, but somebody said the important of baselining your current state, and that is absolutely essential. And I think people don’t always do that. They kind of dive in to try and get metrics without really kind of going “Okay, this is where we’re starting from figuring it out.”
So, from the human aspect – I mean, talking about ADOC, which I mentioned earlier, the Assessment of DevOps Capability, we have 5 different dimensions in that and each dimension has 12 subtopics, so we have 60 topics in total. And I’m starting to sound like we’ve got way too many metrics, but there’s a lot of subtlety in the way that it’s baselined. And the first dimension is human aspects. And within that we’re really getting a feel across an organization, team by team, where they feel their level of capability is around things like psychological safety, around transformational leadership, around how much autonomy and mastery and purpose they have in their daily role.
So, you’d be surprised, I think, about how possible it is to measure cultural capability without necessarily having a knowledge of how many – or relating it to how many artifacts you’ve got in your artifact repository. There are other ways of doing this. And sorry, I interrupted you, Alex.
Hidalgo: No. I think those are all great points, but we do have to, as with everything else, approach it with a little bit of caution because especially if these are self-reported, if these are numbers that you’re looking at like “Are you happy on your team?” “Do you think your organization is headed in the right direction?” – those normal kind of questions and then asking people yes or no, or do you put it on a sliding bar from one to five? And studies show most people are going to just take a one or five anyway. And that’s not saying this is bad. It’s just something else that’s another caveat. It’s another “Let’s make sure we’re looking at this correctly.” “Let’s understand that some percentage of our employees are just going to click through it because they don’t want to fill out this form.” If it is this self-reporting thing, you know?
Beal: It’s such an important point you make, Alex. And it’s something that we deal with when we engage with organizations. There’s such a lot to talk about, because what you’ve actually identified there isn’t a problem with the tool; it’s a problem with the culture. And if people are not feeling that they are part of the change that the organization is trying to make and they’re not empowered to have their say and their voice and be involved in it then they will skew their answers.
So, we absolutely have to help people understand how important their answers are, what impact they’re going to have on their future and their ability to grow themselves in their organization and make them feel part of the change. But yeah, humans can be a tricky business. But back to your point about data volumes, again, in this kind of assessment environment it is quite important to have enough data to weed out variations from people that maybe don’t care or are skewing for other reasons.
Hidalgo: Which, again, you can do some basis statistics to ensure that you are dropping your outliers. You – to be able to get very useful data out of this stuff, again, you’ve just got to make sure you’re looking at it in the right way.
Shimel: Viktoria?
Praschl: Yeah, I think quite often we just force metrics on teams, so that’s why I like it a lot, what you said, Helen, that you need to also understand the culture. So, where is the team from a metrics perspective? Do they understand why we are measuring? Or are you already in a stage where they say, “Okay, I understand why we are measuring certain things,” but now we need to constantly figure out which metrics we are measuring?
So, I think it goes back to the team and explaining the “why” and then really helping them or work with them together, once you have the outcome, what do we do with that? How can we improve? What do we do with the outcome of the metric? What do we do with the numbers? So, I like it a lot, what you just mentioned on assessing the culture.
Beal: Yeah, and that’s a great point. We baseline but we baseline for a reason, because we – I talk about the improvement culture a lot, so that four-step process where we start by establishing the long-term vision direction, then we establish current state, then we establish our next target state, and then we PDCA or experiment between those two states. So, kind of the finishing part of that first cycle is defining the hypotheses and executing them, and then we start again. And we’re not baselining now; we’re comparing it against our baseline at a time period of three, six months and what’s happened with our experiments in between.
Praschl: Yeah, because I think most companies, they just don’t know. They don’t know where they stand. They don’t know where the team stands. They don’t know where the culture stands. So, that’s the starting point to get everything rolling.
Beal: And it’s a big investment, Viktoria, for people to go down this path not to know how far down the path they’ve got.
Praschl: I think for that you need the executive buy-in as well. So, it’s like making sure that you have from the top also the support that you do that. And then, as you go into this constant learning process, I think it becomes very important there.
Hidalgo: And trying to look at how far down the path you’ve gone, that – it’s going to, again, be different for everyone. Is it more important how far or how far you have to go? And is there even an end to the path? Or is this a process that’s ongoing forever? And just another example of having to be thoughtful about what’s being measured.
Beal: I don’t believe there’s an end to the path, which is why I use words like “capability” over “maturity,” because “maturity” indicates that there is some kind of a horizon that you can reach. But of course, no matter how many more capabilities you’re building yourself we live in a market that’s forever inventing new ways of doing things and new tools to support the things that we’re doing. So, it’s an ever-moving target that we’re trying to respond to.
Ashley: I think there’s a whole nother side of metrics I don’t know if we’ve touched on yet, and that is the metric itself can be valuable and inform us about some things, but I also think the analysis of what’s – why? What are the potential causes? In a quality process you do a Pareto diagram to kind of analyze what you think are potential contributing factors and what maybe is influencing why it’s happening in a certain way to try to get to – if you’re – for example, if you’re doing a process improvement what parts of this – a lot of things go into whatever created that metric; how do we know what we might work on first or doing some experimentation with?
But the ultimate thing is there’s a lot of learning that comes out of that process. It really kind of helps you understand “Wow, I had no idea that’s actually how it was working.” And now that we know that with a little more fidelity we can make some adjustments.
Beal: That’s powerful. I wanted to respond to Alex, actually, and then I forgot – it went out of my mind. But this idea that the team needs the right metrics – and Viktoria and I have both said those metrics should not be inflicted on the team, but the other thing is most organizations are made up of multiple teams, and different teams, like Alex said, have different capability patterns. But it’s very interesting to try and understand, for example, why one team is really good at something like diversity and inclusion or really good at AI ops or something, and not saying that the other teams are bad but trying to learn from those success stories. And I think it’s probably the Phoenix Project that put it, it was making local discoveries and global improvements.
And I think that kind of mindset of looking for the strengths and spreading the strengths across a business is really important when we’re doing things like measuring people on their journey because the – we talked about culturally that people are kind of either scared of this stuff because they’re expecting to get beaten up about it or they’re cynical and they’re not giving true answers. But there’s another side of this, that people are also quite afraid of being assessed or even assessing themselves because somebody’s going to go “Well, you’re shocking at deployment frequency. What are you going to do about it?”
And it’s like we have to change that whole mindset in an organization that they are a dynamic learning organization. It’s all about continuous learning and not about beating people up over the head or removing their annual bonuses or the like, which are things that I’ve seen in various places.
Ohlhorst: Which gets us back to that key point of the “Why are we measuring?” Making sure people on the team know why we are measuring things, not only from a cultural perspective, but also from a value perspective so that they know that by them giving their answers based upon queries that there is value to that, and that is the “why” behind it. And ultimately, the goal is to improve processes or accelerate SDLC or whatever it is, but there is a goal there that is not just random data being thrown out asking “What flavor of gum do you like?”
Praschl: Yeah, I think it’s also –
[Crosstalk]
Praschl: It’s important to get them to understand the “why.” It’s usually a long and not as easy of a process. But once they have this and understand it you have the buy-in and the outcome will be way more worth it if you do that way.
Hidalgo: A thing that I’ve found that can help with this kind of stuff is people underestimate how powerful interviews can be. Instead of just asking people to self-report or just collecting random numbers out of your computer system sit down with people and talk to them. Not for five minutes. Schedule an hour. Sit down, find out what is this person thinking? Make sure you’re talking to the right people, not just in terms of an organizational structure, because you do want to make sure you’re talking to people at all levels of the company and all roles. But also make sure that the people you’re interviewing are comfortable with it in the first place. Not everyone has the personality to want to sit down for an hour.
So, another example: Select carefully. Measure carefully. But I’ve found so much value in just – instead of trying to do this in some – a method that is based around metrics. We’re talking about metrics. Sometimes you don’t need metrics. Sometimes you just need the human interaction and you can walk away with that if you really want to then throw that in that kind of scale and attach a number to it maybe. But you can learn a lot just by talking to people.
Beal: I think the best assessments are always a combination of quantitative and qualitative data. And I think we’re seeing this increasingly in the data world as well. About 80 percent plus of most businesses’ data is unstructured, which by its very nature is that kind of textural response data, whether it’s on social media or in your logs in Splunk. We’re seeing more and more people understanding that it’s not just about what’s in an RTMS. So, I absolutely agree with you, Alex. Yeah.
I think there’s a big advantage, though, which I’m going to point out, around ADOC. So, some organizations are very large, so when you’re spending an hour with people and there’s 8000 people it’s quite impossible to speak with 8000 people. So, actually having an ability to empower everybody to be involved in the change by giving them a platform on which to share their opinions is quite useful. And of course, you can put open text fields in there as well and get those snippets and insights that you would miss if you were just getting a one to five or yes or nos.
I did want to maybe slightly disagree with one thing you said, though, which is about the people that don’t want to talk. It always kind of worries me – and this is the big thing about, I think, diversity and inclusion and things like Heloxi that they are trying to give equal voices to people, is I don’t want us to fall into that trap where we think “Oh, that developer is really quiet and introverted; we won’t ask them what they think because they don’t really like talking.” It’s like they’ve probably got just as much right to their opinion as everyone else and we shouldn’t avoid hearing it because we just think maybe they don’t like chatting as much as other people.
Hidalgo: I totally agree. But from my point of view you’ve just got to make sure that whoever is comfortable with the process – people don’t want to feel singled out. Just like everything else, depending on the person, depending on the culture, depending on the team, they might feel like “Why am I being picked? Is this a bad thing?” So, it’s just looking thoughtful. That’s all I meant. Definitely don’t want to exclude anyone. Definitely want to make sure we’re getting every possible voice. Just want to make sure people feel safe about it.
Beal: Totally. And that psychological safety is so important. I learned this from one of my clients years ago. We didn’t use to do things like participant briefings, and now I think it’s really important to get everybody that’s involved in this type of work and talk to them about what DevOps is and the “why” that we’ve all talked about – so, organization performance and succeeding in a disruptive market and all that kind of stuff – and then give them the comfort about anything that they share is between them and the people that are collecting that information. And they may be quoted but they’ll never be associated with what they said. Their words may be used as an example but they can be completely honest. And actually, they are being asked to be involved because their opinion is so highly valued in the organization and we want to gather their insight, so they shouldn’t feel fearful or that their job is at risk. They should feel honored and hopefully excited to be involved in a process that’s intended to make their future more fun and secure.
Hidalgo: The only other thing I’d add there is – and again, I totally agree; I think we’re actually –
Beal: Finally agreeing.
Hidalgo: – in agreement here. Yeah. Yeah, exactly. But the one other thing I would add is that even if we have a strong suspicion or even have the numbers to prove that our organizations or our teams generally are psychologically safe and we’re building a good culture here and this and that, you don’t know what it was like for people at previous companies. People can carry baggage for a long time. It can take people a long time to adjust, especially as your career goes longer and longer. You have these memories and not all of them are great. And so, even if you think “Ah, our company as a whole, we are scouring very high,” for individuals it may not have anything to do with the current organization at all; they could still be carrying some scars from previous ones.
Beal: Yeah, and there’s –
[Crosstalk]
Shimel: Viktoria?
Praschl: It’s about creating the safe space. So, some probably will take longer to open up, but usually if you see that others are joining and the more comfortable you feel, you also get those people to open up down the road.
Shimel: I think we’re all humans, to borrow a phrase, and we bring our own experiences, outlooks, prejudices to whatever metrics we’re going to measure and to whatever we’re going to be in. And I think we can’t lose sight of that either.
Beal: Right. Those scars that Alex was talking about, I just wanted to talk about one other element of this, which is that the scars are negatives. We’re never badly affected by positive experiences, if that makes sense. So, from a neuroscience perspective we basically train our minds to be fearful. We keep on avoiding – we have an avoidance response that’s trained into us. So, I think I really like to try to use neuroscience when I’m helping people through a DevOps change because there are things we can learn from it, things like neuroplasticity and these different avoidance and approach responses that we have in different scenarios. And actually looking at an organization as a whole is great, looking at different teams and how they compare, but actually understanding, as Alan has just said, that how individual humans operate and in the context of their previous experiences and just how we’re built, because we’re all built differently from _____ anyway, is really important. And that’s really where the transformational leadership comes in and having people that understand how to be coaches and how to understand people at a personal level.
Ohlhorst: Interesting that we’re talking about the process from almost applying a “flight or fight” type of ideology to it, that here we are talking about software development and DevOps in particular and pipelines and all that, and so many people so often forget that there are people involved in the process. And that is where you come up against the most challenges.
Shimel: Yeah. It’s about the humans. Guys, we are – we’re about out of time, though we could probably sit here and talk for the rest of the day. But it wouldn’t – I don’t know if anyone would want to watch, and we all have other things to do. What a great conversation on metrics and DevOps and people and everything else today. Maybe we’ll come back and we’ll run a part two of this or something. But for now I need to pull the plug.
I want to thank our friends at Tricentis for making DevOps Unbound possible with their sponsorship. Alex, Helen, Viktoria, Frank, great conversation, man. Great conversation, men and women. Good stuff. Mitchell, you want to take us home?
Ashley: What I’m struck by is the richness of just the word “metrics” and what all it can mean and the depths of it and aspects, whether it’s the human side of it, whether it’s the “why,” whether it’s the outcome. There’s several – I think there’s several good takeaways there from this, and so I hope it’s helped everyone kind of think about maybe why you’re doing what you’re doing and how you want to lead that going forward.
Shimel: Excellent. All right. This is Alan Shimel for MediaOps, devopsy.kinsta.cloud, here on DevOps Unbound on TechStrong TV in the TechStrong TV network. We’ll see you in about two weeks with another great DevOps Unbound episode, but for now this is it. Thank you, everyone. Thank you for watching. Take care..
[End of Audio]
-
EP 16: How DevOps has Changed Testing Forever
DevOps has changed the game in almost every aspect of testing: who, what, when, where and even why. What’s working, what needs to change immediately, and what challenges—and opportunities—lie ahead?
In this DevOps Unbound episode, hosts Alan Shimel and Mitch Ashley are joined by Abel Wang from Microsoft, Tracy Miranda from CDF, Caroline Wong from Cobalt, Kurt Chase and Ben Simo from Tricentis to share a variety of perspectives on how testing fits into the ever-evolving world of DevOps. The video is below, followed by a transcript.
DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.
Alan Shimel: Hey, everyone. Thanks for joining us on this special edition of DevOps Unbound. You’re watching this, probably, at the Tricentis Virtual Event and we are happy to have you join us, happy that you’re attending the event, and we’ve got what we think is gonna be a great discussion on testing and all of its various facets and what does the future – or near future – hold. We’ve got a great lineup of speakers that I’d like to introduce you to who are on our panel today. Let me first start with – she’s on the top of my screen – is Caroline Wong. Caroline, why don’t you say, “Hello” and introduce yourself?
Caroline Wong: Hello. My name is Caroline Wong. I’m the Chief Strategy Officer at Cobalt. So, at Cobalt, what we do is pen testing as a service. We build a security software platform – like a lot of DevOps companies.
We have data-driven product-based teams. We have – we really value automation and failing fast. I’ve spent 15 years in the information security industry. I wrote a book called Security Metrics. I host a podcast called Humans of InfoSec, and I teach the OWASP Top 10 Learning Path on LinkedIn Learning, and super excited to be here with this group today. Thanks for having me.
Shimel: Thank you, Caroline. Next up, we have Kurt Chase.
Kurt Chase: Hey, everyone. I’m Kurt Chase. I’m currently Head of Release Management and Engineering Services at Tricentis. Engineering Services is a group that supports the engineering function so, we do our own type of testing – testing all the processes and everything we use – to deliver our software at Tricentis. So, prior to Tricentis, I spent some years at Splunk and at AutoDesk. It’s great to be here with all of you today.
Shimel: Fantastic.
Chase: Be here today with all of you.
Shimel: We got what you meant, Kurt. It’s okay. Next up, we have Ben Simo. Ben, why don’t you introduce yourself?
Ben Simo: Hi. I’m Ben Simo. I’m a software tester of 30 years and past president of the Association for Software Testing who has recently stepped into a product management and product research role at Tricentis trying to discover and develop our testing platform of the future and bring new things to Tricentis customers and hopefully introduce new people to Tricentis with new products that enable new ways of testing.
Shimel: Okay. Happy to have you here. Next up, Tracy Miranda. Tracy – well, I’m gonna let you introduce yourself. Go ahead.
Tracy Miranda: Hi, everybody. I’m Tracy Miranda. I’m the executive director at the Continuous Delivery Foundation. It’s an open source organization that’s part of the Linux Foundation, and also home to projects to like Jenkins and Spinnaker, and our members include companies like Netflix and Google. And I have over 20 years of experience in open source so, I love open source and I’ve worked in it as a developer – also in QA and all different facets. So, yeah. Big fan.
Shimel: Thank you, Tracy. Last, but not least, is my co-host on DevOps Unbound – my partner in crime here, Mitchell Ashley. Mitchell, why don’t you introduce yourself?
Mitch Ashley: Thanks, Alan. Yes. I’m CEO of Accelerated Strategies Group, which is an analyst firm that focuses on, in addition to cyber security, digitial transformation, DevOps, and cloud-native kinds of applications, which really matches my background very well. I’ve been both a product creator CEO/CTO as well as running IT, CIO, CSO – that kind of thing. So, currently, I’m also CTO with MediaOps – I guess CSO at MediaOps, too, so, I’m trying to collect as many three or four-letter titles as I can. So far so good.
Shimel: Alrighty. So, first of all, thank you all for coming on today. Secondly, testing – what a wild, mixed-up world we have in testing, right? Just from looking at the folks on our panel today, we have security testing – pen testing, in particular. We have the world of open source and what a factor that is in testing.
We have product testing, right? We have sort of new product testing – taking testing to new heights. And you know, that variety is sort of a double-edged sword, right? Because it’s like jack of all trades, master of none type of thing. There’s so many aspects of testing, right?
Even in our audience here at this virtual event, we have testers who come from, you know, what’s called old line QA testing, right, to DevOps enabled testing to visual testing – testers who, if it’s not an open source testing platform, they won’t use it, to proprietary systems. Through it all, though, a lot of times, all of this testing becomes sort of a red-headed stepchild or a whipping boy, if you will, right, to the whole application development system operations teams. It’s like, at the same time, their best friend and their worst enemy, right? They want everything tested, but they don’t want to give you the time, the budget, or the resources to do it, and then, when something doesn’t work, they said, “Well, we should have tested more, right?” So, how – first of all, do we agree that testing has been sort of the whipping boy of a lot of what we’ve seen go down in app dev and operations? Anybody want to jump on that?
Chase: Well, I think it’s certainly gotten a bad name. I think some of that comes from prior to the digital transformation. A lot of the engineering was waterfall, and testing was always that last stop before the software went out live, you know? Nowadays, I think everyone’s come to realize the speed and scale at which we develop things, you have to do the testing up front at the very beginning. You can’t wait till the end, you know?
You’re gonna be too slow. So, I think, from that perspective, absolutely. I think that’s why testing has really, I think, gotten a bad name, because they’re kind of the blockers to getting anything out into the public domain, whereas nowadays, you have to enable that up front because you want to get it into the public domain yesterday.
Shimel: Yeah. And so, in many ways, to me, that sounds like, well, DevOps has been the best thing that ever happened to testing, right? In many ways, DevOps has been the best thing that’s happened to security testing, right? We’ve certainly seen that shift left, if you will. But has that shift left helped – has that rising tide lifted all boats?
Miranda: I still see –
Shimel: Oh, okay, Tracy. Go ahead.
Miranda: I’m gonna jump in and say one thing that strikes me as funny. Like, we have, in our industry, everybody has embraced the DevOps terms, and then, you start to see all these spinoffs to put emphasis on things like, for security, we have DevSecOps, and you’ve got every variant of something with ops. But when it comes to testing, you know, there’s no TestOps. There’s no new buzzwords and it kind of speaks to how much it’s being neglected. Like, we had a conference on continuous delivery, and we asked for talks of everything in the life cycle, and we got lots and lots of submissions except for testing.
So, again, I think it still is really under-appreciated and just not given the due credit for what it stands for.
Shimel: Very –
Simo: Yeah. And I think some of this comes from, as Alan mentioned, that testing is a very broad and diverse subject, and I’ve often watched people argue about testing that each interpret it to their piece of what they do with testing. And things like CI and some of the DevOps practices have helped us improve with the checking that our software can work – you know, demonstrating that it can work – and there’s a lot of people – if you primarily wear a developer hat, you’d look at that’s what testing is. And we’ve automated that.
There’s been huge improvements in tooling and technology to do that, but sometimes, along with that is we throw out that we need someone involved that’s looking at things from a critical perspective that’s concerned about the safety and efficacy of what we’re delivering that is the, I guess, the one questioning and searching for information that they raise issues that we need to deal with that are reasons to not push things out to production now – so that we can make informed decisions. So, often, I think just that critical side gets thrown out as we improve the other side of testing, which is being able to demonstrate that things can work.
Shimel: Anyone else?
Ashley: You know, Alan, one of the ways you can tell who drew a diagram is to look at what’s in the center. And so, if we think about CICD, we kind of surround testing, but we don’t talk about testing. So, in some ways, it’s elevated, and some ways, it still sort of abstracts it out of DevOps. But my experience has been not really, because most teams know that testing has to get automated as part of that process, ’cause you can’t have this automated checking and automated deployment, then let’s kind of let a bunch of manual things happen. And not that testing hasn’t been automated – it has been – but I think it has elevated, at least in part, I think a good deal.
My experience has been. I haven’t worked in every environment, maybe, where test still isn’t thought of very highly. By the way, the thing that the – it isn’t the last thing in the chain. Documentation is the last thing.
Chase: That’s true. That’s true.
Ashley: Let’s give documentation –
Chase: And release notes. It’s true. I have tech doc’s team, too. You’re right. [Laughs]
Shimel: And that hasn’t really changed much either, though testing has. No. No. But you know what? Caroline, I’d like you to focus in specifically.
I mean, I think about – you know, Mitchell and I co-founded a security company, and we came out with a product in 2003 of just a vulnerability assessment and manager – “VAM”, we called it – and at that point, look, we used to beg people to scan their infrastructure at least once a year, right, if not quarterly. We were really pushing for quarterly. And you know, the idea of scanning and testing applications before you deployed them – unheard of. It was unheard of, right? We didn’t have the security tools or people.
We did that after the fact. Now, that’s not normal anymore, right? Today, at least 85 percent of applications are tested prior to deployment. I don’t know who those other 15 percent are, but that’s a whole nother story. But talk to us.
You’ve lived through this, Caroline. What do you think?
Wong: Sure. So, you know, I think there needs to be an acknowledgment. There’s sort of these like, first order priorities and these second order priorities. When I think about security – and I think some of these concepts apply to testing more broadly – there’s a saying that, “You don’t take a $200.00 fence to protect a $5.00 asset.” I think security is about you have something of value and so, you want to test it – rather, so, you want to secure it.
Testing, I think, is like you made something new and so, you want to test it. You’re not gonna test something unless you have something new. You’re not gonna secure something unless you have something of value. So, it’s okay that security and testing are not sort of the number one thing, you know? Inherently, by their definition, it’s sort of a second order attribute, if you will.
Now, I think, you know, each of us has been through a version of annual organizational budgeting. You know, each organization says, “Okay. Here are the goals for the years. We’ve got limited resources and budget to accomplish those goals. How are we gonna allocated those resources?”
And there are some things that, for a lot of organizations, are very clear. You know, even if I take a step way back and I say, “Okay. Most organizations might think about budget allocations in three buckets. Number one – general and accounting; number two – sales and marketing; number three – research and development.” You know, nowhere in those words is either security or testing.
And that’s okay, you know? I think, Alan, when you share your story about VAM, you know, Vulnerability Assessment and Management, I agree that over the last, I would say, decade-two decades – so, when I started my security career 15 years ago, we were almost trying to convince folks that breaches really happened – that this was really a thing to be worried about. You know, today, everyone knows because it’s on the media headlines every day. This is real. And so, that has been a shift in the perception, I think, in our industry.
There’s also been a big compliance driver. So, the – in the area that I’m very familiar with, security pen testing, a lot of that pen testing is actually driven by external compliance requirements – things that say, “Because of compliance framework A, B, C, you must actually conduct manual pentest on your application, for example, once a year.” You know, doing it more often than that is often because of perhaps what a lot of this group might think of as a really good thing – trying to actually secure your software, trying to actually test your software to ensure it does what you think and what you intend for it to do, but that’s not necessarily the first thing that comes to an executive’s mind when they’re thinking about budgeting, and there’s a variety of reasons for that, right? There’s an external perception thing. I think there’s also a lot of learning, certainly, that the security industry can do to be more approachable and to really understand better what is the software development methodology that we’re engaging with.
What is the specific methodology and behaviors of these individuals? And how can we actually collaborate with them most effectively?
Shimel: Fair enough. I mean, look, I don’t want to be the pessimist in the group or the bluebird of happiness here, but we have had success over the last 5-10 years, Caroline. You’re right. Especially in security, but not just security. We’ve had success in testing across the board.
Tracy, to your point, the fact that we don’t have DevTestOps, I think, is, you know, testament to the fact that we’ve recognized that testing is so critical to the CICD process that it’s part of it. I mean, you can’t have DevOps, you can’t have CICD without a level of continuous automated testing in there – in my mind, anyway. I mean, prove me wrong. But, you know, I think that has been probably one of the biggest successes that the DevOps movement has brought – has been responsible for.
Miranda: Yeah. And I really like Caroline’s analogy with security because, you know, that’s something else we think of as fundamental to DevOps, but it’s sort of elevated itself and, in a relatively short space of time, went from being ignored to being, you know, headline news. And I think there’s a lot of lessons there. And I – like, I will say, I do think testing is going through that journey and we’re probably on the cusp of it kind of really exploding and watching a few players in the industry and companies starting to do really interesting work in testing. I think that’s gonna turn around very shortly.
So, yeah, there’s an exciting space that I think its moment is coming – just like securities has.
Chase: Yeah. I think it’s all about the testing now, quite honestly. I mean, when you look at the future with low-code/no-code, it all becomes question-question mark ops, you know? Any function in a company could actually put something together and run it, and they’re gonna have to test it, you know, and it becomes less about all the engineering and more about the test patterns and things like that that are needed as you get to these future states. So – and speed and scale is gonna continue, I think.
That’s driven everything with the digital transformation. It’s speed and scale. Everything needs to come faster, and we want more of it, you know? And so, I think that really pushes the envelope, and that’s what DevOps has responded to and, I think, addressed very, very well, but it’s not over. We’re gonna continue to scale up and go even faster.
So, we have to be prepared for that, and I think it becomes more about testing in the future, and less about, you know, we’re gonna have these pre-built components – APIs you can piece together however you want – then, testing becomes, I think, almost first order, to Caroline’s point. And security, especially. I think security, we’re seeing that change now. If you’re a SAS company especially, security has to be top in line, ’cause you just can’t put something out there and not have it tested from a security perspective. And, furthermore, with compliance, it’s not just ISO and GovCloud and those programs, but large corporations now are saying, “Hey, if you want software in my ecosystem, these are my rules and I want to see your pentest reports and I want to see this.”
So, I think more and more people are realizing that if the software’s not tested, it’s not worth it. And so…
Simo: And, in my experience, part of adopting a DevOps culture and not just tool set is putting testing at the forefront. Some of the most productive teams that I’ve worked with that deliver some of the highest quality software I’ve seen – testing was center to absolutely everything that everyone on those development teams did.
Ashley: Hm-hmm.
Chase: Yep.
Shimel: Yeah.
Miranda: Yeah. We think – go ahead.
Ashley: I’m just gonna jump in and say – there’s another factor that actually works to our benefit and that is not only to speed through automation and DevOps process, but contemporary software architectures, containerized micro-services, service mesh – we’re releasing smaller bits of code in encapsulated forms, which means, you know, we’re not testing monolithic applications all the time anymore so, we’re not having to run two weeks’ worth of tests just to get one pass through.
Chase: Yeah.
Simo: Right. _____ container solved the “It works on my machine” problem because we can ship the developer’s machine.
Chase: Yep.
Shimel: Yeah. That was always a big problem, right? Tracy, you were gonna say something though.
Miranda: Yeah. No. I think the direction we see it heading is we’ll start breaking the testing monolith to talk about what is unit test, what is system test, and appreciate that is a specialization in itself. And the other things I’m starting to see from some of the end user teams we talk to is this real focus on culture – you know, a continuous quality culture – and seeing that hand in hand as part of the testing movement and again, _____ back to security – you know, the security first culture went hand in hand with that whole emphasis on security so, similarly, the continuous quality culture is starting to be something that – like, the really high performing teas we see pay a lot of attention to.
Shimel: Absolutely. You know, Kurt, you said something there about low-code/no-code and how that could be a game changer. You know, some people refer to that as the rise of the citizen developer, and I’m wondering if maybe what we need is the rise of the citizen tester, right? Is testing too hard?
Chase: Yes. I waste gonna say – the one thing we haven’t touched upon – I think Caroline did – is testing is hard. Security testing is super hard. And when you look at today’s modern environment and a lot of the ISVs – at least the ones I’ve seen and been associated with – you have a legacy of monolithic applications that you have to support that are bringing in a ton of revenue, and, at the same time, you’re transitioning – you know, going through your own digital transformation. So, that adds complexity after complexity in testing, and I do think, in the future, testing – you know, a lot of us, like Ben said, to his point, we have to push testing up – you know, shift left, move it far left as we can.
And that’s gonna continue. I think that has to be the norm. To Caroline’s point, it has to become first order. And I think it’s starting to right now. I really do. Especially the – all the SAS solutions that are out there – you know, shipping code daily, hourly. You need excellent testing to do that and do it with competence.
Shimel: Yeah. I think you do. But we do have to – and it’s funny that we’re talking about making testing easier because – not security testing, right? Security testing, pentesters – they were always sort of the hired guns of the security space, right? There were SAS or – there weren’t companies like Cobalt, right?
The pentesters that I knew growing up – growing up – you know, that I knew coming up in the security space were gun slingers. These were, you know, white hat/gray hat folks who would come in and they’d do a little social engineering and pen testing using some open source tools, and you know, they give you a report. But you know, these were folks who traveled around the world doing that kind of thing. You know, we didn’t have a company like Cobalt that did this as a service, so to speak. So, it’s very different.
But here’s the underlying thing with like, QA kind of testing – and, you know, I don’t mean any disrespect to anyone in our audience today, but for too long, a lot of folks in testing were viewed as failed developers or people who couldn’t be really good developers did testing or maybe testing was a weigh station on the way to becoming a real developer – like a Pinocchio stop or something, right, where you’re not a real boy yet. But that’s not true. There’s also – there’s a whole cadre of professional testers. And you know, in their capable hands, you could do great things. But I guess the question is – does testing have to be so hard that we need professional testing – professional testers to do testing?
Or can we make testing automated enough that you – I’m not saying it has to be the dunces who don’t become developers. That’s ridiculous. But can we broaden the base of people who do testing? And that includes security testing? I’ll throw it out. Ben, you were a big-time guy in the testing industry. Tell us – what do you think?
Simo: So, I think we can expand who’s involved in testing and that more people than those currently doing testing can learn to do testing. I don’t know _____ developers learning to become better testers and test their stuff. But I will say that ultimately, the part of the work that is the testing cannot be automated. Tasks that we do in testing can be automated. So, whether – not matter what hat someone primarily wears, if it’s a developer, you know, writing tests to demonstrate their code works or someone’s using TDD and they’re building some tests and then, building some code that matches it, the testing is the critical evaluation that’s going on in their mind of solving the problem and demonstrating that it’s solved – or evaluating how something might go wrong and coming up with some approach to demonstrate that it may go wrong and check for that.
So, those things, whether you’re doing it automated or the execution is automated or done manually, the actually testing work, I’d argue, is the cognitive work that people are doing. And there is some – that we can never automate away. There are some ways that I think AI can help us. There’s also some ways that may lead us astray. But ultimately, I think we need people to learn to be critical thinkers to understand the technology, to understand the domain of the users and the problems that are being solved, and applying that critical thinking is the testing.
Of course, then, we want to use tools to help us with some of our tests so, when we come into DevOps and CI, executing those demonstrations of the software working is a key part of that. Now, whether we do it as part of our, you know, building to those tests or we create them afterwards, then they can, at the very minimum, act as changed detectors going forward. But there was some testing in people’s minds that was going on that created those.
Shimel: Caroline, what about – okay.
Wong: I agree with quite a lot that Ben is saying. You know, I can’t help but think to myself that – I actually have this sort of visual in my mind, and it’s actually at a circus of a trapeze artists, you know? I think about these people, and they’re sort of like, whirling in the air and they’re doing these like, really incredible feats, you know, and they just do whatever they do, you know? But there’s like, a safety net to catch them if something goes wrong. It also reminds me of a metaphor that is often used in security that says, “We put brakes on a car so that you can go fast.”
You know, when Mitch said earlier, you know, “These days, we’re releasing smaller pieces of code, modules of code” and there’s a way in which, yes, each small piece of code – each module – can be tested and some of that can be automated. We should automate performance testing. We should automate unit testing. But, to Ben’s point, there’s a lot that can’t be automated. There’s a lot of testing that actually depends on things like human judgment and opinion and it actually requires a person’s decision, you know?
So, I think there’s – I think there’s a couple of frameworks for us to consider, right? If the first order is about creating value and making something new and the second order is about testing to make sure it does what you wanted it to do and protecting the value created, there’s also this spectrum that says there’s smaller bite-size stuff that may be a good candidate for automated testing, but, I think it’s super important to recognize that these small pieces of code – these modules that Mitch was talking about – it’s also super important to test what happens when they come together. So, to the extent that security folks need to be able to test chained exploits and business logic flaws, you know, those are exactly the types of things that will never be automated – is what I think, anyway.
Shimel: I don’t disagree. I think security in particular is just there’s some things you’re just not gonna automate. I don’t care what you say. But that’s okay. Anyone else?
Chase: I do want to make a comment on something you said. This was a long time ago now, when I was a dev manager, that I used to sense that when my developers – you know, if you were on my team, if you wanted to really get me mad, treat your tester badly. Treat them, you know, a bit, like you said, like they’re not quite at the level you are. I would be incensed as a manager. You know, that was a long time ago.
Fortunately, I don’t see that as much nowadays. I think that role has been elevated where I don’t feel that at all on my teams, and if I do, that is, again, to this day, I’ll get super mad about that because I think it takes all kinds to develop software today and ship software. Testing role is just as critical as an engineering role, just as critical as the ops’ role, the security role. So, yeah, that was one thing that would drive me nuts ’cause I did sense that and go through that, and this back in, you know, the late ’80s-’90s, if you will, before the year 2000 – I’m dating myself – but it definitely was palpable in some of the large ISVs, you know? And as a manager, that was something that would drive me nuts.
Shimel: Agreed. Agreed. Tracy, open source – look, testing has been inexplicably wound up with open source for as long as I’ve been around, right? Whether it’s security – open source security testing and you know, things like Metasploit and Nessus and Nmap or you know, in QA testing kind of stuff, it’s a rich history of open source testing tools. How do – and part of your mission at CICD is – at the CICD Foundation – is to nurture that, I’m going to assume, correct?
So, how do you guys nurture that? What are you – what do we do to maintain that rich, open source tradition here?
Miranda: Yeah. No. Absolutely. And, as I say, we at the Continuous Delivery Foundation, we want to look across the entire software delivery life cycle, and I think it is safe to say, you know, that testing hasn’t risen to the top. We’re actively – like the board wants it to be a more focused-on topic, the technical committee wants to encourage folks to bring us more projects in open source and with testing, but we’re still having to kind of fight through all the noise around other projects.
And one of the ways we’re approaching it is, for example, our technical oversight committee wants to set out a landscape of the types of projects in each kind of bucket and say, “Okay, look. We need to have something in security. We need to have something in testing that covers these different aspects.” Almost like a call to action to say, “Bring us your testing projects and –” like my favorite saying about testing is, “There’s two key problems in testing – not enough tests and too many tests.” And yeah, I think we’re starting to see some really good frameworks around test generation.
I’d love to see more open source options for that. And Kohsuke Kawaguchi, the famous creator of Jenkins, move on to focus on testing so, that says a lot about the importance of this area getting more open source focused. So, yeah, I’m excited to see what could come out, but we do have to actively encourage folks to come join us and have those conversations and not to, you know, have it assumed it’s all focused on delivery of pipelines for delivering software.
Chase: I think another thing you’re seeing today is a lot of people, when they start developing an app, when they start creating something new, that’s the question they ask up front. “How are we gonna test this? How are we gonna automate it? How are we gonna pentest it?” I think those things, at least, are starting to bubble up a lot stronger in the cycle – you know, early on. ‘Cause I know that’s something we recommend highly – is that if you’re developing something new or need to head down a digital transformation, think about the testing. How are you gonna test that up front, you know? And products are actually being developed with that mindset now.
Simo: And with short iterations, it’s how are we going to test this tomorrow, next week? It can’t be left off for months later to be someone else’s problem. So, people, I think, are learning to deal with those questions about testing from the start of their discussions about what the requirements are.
Chase: The other thing I love – and in Caroline’s world, has been very successful – I know it is specialized skill set, but trying to teach that throughout your troops – you know, throughout the engineering team. Show them what, you know, across _____ scripting errors. What does it look like in code? That’s super successful. That really, really helps make – ’cause security is difficult, you know, no matter how you slice it.
Even when you’re looking at product development – if you want to invest in fixing security defects or add a cool new thing for your users, you know, a lot of times, the cool new thing is gonna win out. But that’s starting to change as well, too. I think people are starting to value security – especially in today’s modern world, you know? It has to be up front, really.
Ashley: Yeah. It’s all about AppSec at the moment. That’s one of the hot topics as well. You know, someone mentioned AI earlier. I just want to offer a perspective about that because we always seem to focus on the shiny new object.
I know I have. I often do. But I think what we’ll see with AI is select applications when it comes to testing. The sort of vertical place is where it’s really well suited. When we say, “Yeah, AI”, we really mean machine learning in most cases, which is machine learning algorithms, and there’s two types – there’s supervised/unsupervised.
Supervised is, “That’s a car. That’s a car. That’s not a car, right? That’s a bug. That’s a bug.”
And unsupervised takes massive amounts of data and intellectual patterns and learnings from that. So, I think that as we’re taking our test automation and our test tools, there may be some of that, and it might be really useful for AI. I wouldn’t get worried about people being replaced as testers. Not in my lifetime, that’s for sure. It is still a – there’s very much a craft to it, just like there is software design.
There is a craft to testing and a skill and a mentality about it that’s unique, just like with security.
Shimel: No doubt. Caroline – oh, go ahead, Ben. I’m sorry. No, no, Ben. Go.
Simo: Yeah. I was gonna say – and with any other sorts of automation – even you know, not learning automation that’s come before AI and machine learning based things, is that it not only has potential, in some cases, to replace some of the mundane work we do, but to enable to the skilled people to focus their work and apply their brain power to the things that really matter. So, I think there’s great potential for AI to help direct people’s attention to the things that maybe should be of concern and help deal with that big question of testing out of infinite possibilities of PaaS and data variations. Which one’s gonna show me the problem? And I think there’s huge potential for AI to help us find that if we’re also aware it may guide our eyes away from something that we do want to see.
Shimel: Caroline, how does that relate to security, though? I mean, is AI helping Cobalt with pen testing? Be honest.
Caroline Wong: So, the type of software that we’re building today is actually workflow and collaboration software. So, what we’re trying to do, is we’re trying to bring security folks and developers and penetration testers all together sort of in one place where they can collaborate. The technology piece, for us, is not about automating defect discovery. It’s not – I mean, if we’re gonna us ML or AI in the future, honestly, what we’re gonna do is we’re gonna use it to say, “What are the characteristics of this organization that wants us to do a test on this particular type of asset? What are the attributes of that asset and how might we match that to our talent pool – something that we do totally manually today?”
I think that’s the opportunity. I don’t think that we’re gonna get to a point where we’re using the AI and the ML for automating the defect discovery itself. Frankly, I think that even if I take a step away from pen testing and I look at the security scans that are in existence today, I think, you know, whether or not they’re referred to as using AI or ML techniques, they’re just simply imperfect, you know? And there’s a super important place for them, but there’s a lot of false positives. There’s a lot of interpretation of data.
And I think that’s always gonna be the case. I think that no matter what, when it comes to humans and machines, the machines can add a lot of value, but I don’t know that when it comes to software development testing and security in particular that there’s every gonna be a complete substitution. I don’t think that – I don’t think that makes sense.
Shimel: Absolutely. So, guys, we’re coming up on the end of our time. I need to wrap up. Tracy, let me say this for you, if you don’t mind – and if I’m wrong, correct me. For you testers out there, the CICD Foundation – or CD Foundation, actually; I keep – we always say “CI” like it’s attached at the hip – but the CD Foundation is looking to hear from the testing community.
Probably some companies – including Tricentis – may want to look into becoming involved with CD Foundation because if you don’t have a seat at the table, you don’t have a right to complain about not being heard. And so, you know, when we look at foundations like Linux Foundation and the CD Foundation, CNCF, we need the testing community – vendors, professionals, folks in it, you know – to be involved and get involved. Be heard and be part of it. So, I’m gonna extend that out there to everyone. But, short of that, though, Caroline, Tracy, Kurt, Ben, and, of course, Mitch, thank you so much for joining us on this special edition of DevOps Unbound.
We hope everyone enjoys the rest of their Tricentis Virtual Summit. This is Alan Shimel for MediaOps, devopsy.kinsta.cloud, Security Boulevard, Container Journal, Tech Strong TV. Have a great day, everyone, and thanks for joining us.
[Musical outro]
[End of Audio]
-
EP 15: Why Testing is so Frustrating
In this episode of DevOps Unbound, hosts Alan Shimel and Mitch Ashley are joined by Adam Kalsey of Tricentis, Brian Dawson of the Linux Foundation and Caroline Wong of Cobalt to discuss why testing is considered the most frustrating bottleneck for developers and engineers. Panelists will explain the benefits of testing early on in the SDLC and how to accelerate the feedback cycle with automated and continuous testing, as well the pros and cons of testing, the value of shift-left testing and how to effectively balance quality and speed. The video is below, followed by a transcript.
DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.
Alan Shimel: Hey, everyone. I’m Alan Shimel, and you’re watching DevOps Unbound. DevOps Unbound is sponsored by our good friends at Tricentis, and it consists of a biweekly, 40, 45 minute video panel on relevant topics within the DevOps sphere, and featuring a panel of experts that rotates every other show, and really great people, and something I personally really enjoy doing.
And then, about once a month or once every six weeks, we actually have a live studio audience—or there is no studio audiences in COVID, actually, but we have a live virtual audience that participates in our show. Today’s show is just our panel, though, without a live audience, and I’m really looking forward to our discussion.
Our discussion today is around testing. You know, with the pressure on developers to deliver code faster, sometimes testing is considered a road bump in the way. Testing is considered slowing down the whole process. There are a lot of developers who’d like to push testing off that plate. Of course, no one wants to go back to testing after we’ve deployed software, primarily, but there is that tension there.
Let me introduce you to our panel today; we’re gonna discuss this in depth. First of all, making his first appearance as a member of The Linux Foundation on our Tech Strong TV network is our old friend, Brian Dawson. Brian, welcome to DevOps Unbound.
Brian Dawson: Thank you, Alan. Thank you, Alan. And it’s fitting that the first panel, first webinar I do since joining The Linux Foundation is with you.
Shimel: It’s our pleasure to have you on. Brian, what’s your position at Linux Foundation now?
Dawson: So, at The Linux Foundation, I am functioning as the VP of Developer Relations and Ecosystem Development. So, just to help, we’re gonna help The Linux Foundation to the development community while driving more of a DevOps opinion into a growing ecosystem of solutions that support open source development.
Shimel: Fantastic. Next up is my good friend, Caroline Wong. Caroline is—oh, she’s an anchor on many of our different shows here at MediaOps and Tech Strong. Caroline, it’s great to see you again. A little background, maybe, for people who aren’t familiar with you?
Caroline Wong: Great to be here. Thank you so much. My name’s Caroline Wong, I’m the Chief Strategy Officer at Cobalt. We’re a pen test as a service company.
I actually started my DevSecOps career 16 years ago leading information security teams at eBay and Zynga. So, in both cases, we were running online operations 24/7 with millions of simultaneous users daily. And now, at Cobalt, we build security software. And, like many DevOps companies, we’ve got data driven, product based teams. We value automation and failing fast, and super excited or today’s conversation.
Shimel: Fantastic. Welcome. Great to have you, Caroline. Our third panel member is another returning DevOps Unbound veteran, and that’s Adam Kalsey of Tricentis. Adam, welcome.
Adam Kalsey: Thanks. Thanks for having me. I lead Developer Relations for Tricentis. I’ve been building web applications since 1995, and have always had a soft spot in my heart for bad applications and broken things. And so, I love to help developers build higher quality software and do it in a more efficient, faster way.
Shimel: Absolutely, and thanks for being on with us today, Adam. Last but not least, in his Florida outfit here, visiting me in Florida, is my Co-host of DevOps Unbound, Mitch Ashley. Mitchell, welcome.
Mitch Ashley: Coming to you from the MediaOps Studio B.
Shimel: Yes.
Ashley: Great to be here. I’m a former product, both security and software SaaS, CTO as well as a CIO, and now I run Accelerated Strategies Group, an analyst firm that focuses on those things.
Shimel: Absolutely. Hey, Mitch, thank you.
So, let’s get started. Adam, you’re with Tricentis, the sponsor of DevOps Unbound, and they’re a testing company, probably the leaders, one of the leaders worldwide in automated testing, continuous testing. Have we put too many straws on the developers’ camel’s back by making them, you know, be more responsible for testing to say we need to test more of our code earlier.
Is it a bridge too far for many developers? Is it making them resent testing? You know, what’s the current state?
Kalsey: I think where we’re failing developers is by handing them lots of more things to do without giving them any—without giving them what tools, and I don’t mean just necessarily software tools, but without giving them the things that they need to do it effectively. If we just pile things on developers and say, “Oh, this is your job now” but we don’t give them the support, we don’t give them the training, we don’t give them the expertise, we don’t give them anything to make that better—sure, it’s just gonna be harder on ‘em.
Shimel: To play devil’s advocate—oh, those poor little testers making a quarter of a million dollars a year each. Do we want ‘em to do testing, too? Right? Is there some of that here, Brian? Or—go ahead, Adam?
Kalsey: No, no, Brian—please, please, jump in.
Shimel: Well, he’s from The Linux Foundation, he now represents these folks. [Laughter]
Dawson: Well, first, not all of them make a quarter of a million dollars a year, right? But I do think what you raise is important when you bring up the investment, right? We are hiring talent with a particular expertise in a particular domain, a particular ability to solve problems and innovate.
And there is an argument to be said, Alan, that, “Do I want to spend this money, this quarter of a million dollars plus 200,000 in RSUs on having somebody write test harnesses?” Now, I think—you know, that can be balanced by what Adam said. Well, first of all, yes, because we want everybody thinking about quality. But you know what, if we want a return on that investment, we wanna free the cognitive load, then you know what? We need to invest in automation, we need to invest in better collaboration and reducing friction from the process.
Shimel: Agreed, agreed. Caroline, you know, it’s been a big—this whole DevSecOps movement and shift left has been about shifting security testing left and getting developers more involved in security testing, right? Yet another camel on the—yet another straw on the camel’s back. Should we be doing that?
Wong: So, I—oh, my Internet is awful. Can you hear me?
Shimel: Yes, we hear you fine.
Wong: Okay, fantastic, awesome. I’ll just—I know that there’s post editing, so it’s okay for me to say that.
Shimel: Mm-hmm.
Wong: So, Alan, I bring a particularly unique perspective to this group in the sense that I actually oversee people at our company, I oversee Human Resources. And I’ve been with the company since we were 10 people, and today we’re 200. And one of the super interesting things that happens is, naturally, as a company grows, what people do, they specialize and then we have to figure out how to communicate more effectively.
And I think that if you’ve got an engineering team and you say, “Okay, now you’re responsible for Development and you’re responsible for Ops, and your responsible for security, then there’s got to be some level, depending on the maturity and the size of the organization and what the organization’s trying to accomplish of specialized roles and protocols for engagement.”
And I think that when, either number one, those roles are not clear to, I think Adam was making a point earlier about security folks need to be providing guidance to engineers where—because engineers are not expected to be security experts, frankly. Security experts are not expected to be engineers. And then, at the point where information is being handed off where the responsibility for taking an action, those, all of those sort of procedures, they have to be clear, otherwise they’re simply not going to happen.
Shimel: Yeah, agreed. I think they have to be clear. But even if we make them as clear as we possibly can, is it just—are we putting too much on developers? I mean, I think fundamentally, that’s the question here, right?
Wong: There’s—
Ashley: Alan, it’s not only—sorry, I’m just gonna jump in. It’s not only quantity of work—is it the right work?
Shimel: Right.
Ashley: Because I know from writing software and being part of software teams for so many years, there’s this thing called unintended consequences or unintended use. Well, you’re not supposed to use it that way, but the person who designed it intended it to be used a certain way, but one of the important functions of tests is not just to uncover bugs and elicit those things or security issues, it’s also to push the boundaries about how you might use the software and essentially stretch it to try to find where it’s gonna break where they can. And oftentimes, that’s a different mentality than or perspective, certainly, than the person that wrote the code.
Dawson: And if I may jump in and say maybe part of the discussion can get to generalist versus specialist, right? There is this idea, Alan, that when you talk about putting a lot in the developer’s hands, this idea that there is this prolific person called the full stack developer that can separate themselves from what they develop to look for security vulnerabilities, to test it, then ultimately deploy and run it, right? And I think we need to give thought to, are we trying to build a bunch of generalists, which doesn’t potentially enable them to do any job the best, or are we enabling specialists to better work together, share information, and crowdsource quality and crowdsource security?
And I’m just curious if any of the other panelists have an opinion on this concept of full stack developers versus, say, needing traditional, lifelong quality experts.
Kalsey: Yeah, I like to say that we decided to create cross-functional teams and we did it by removing all of the functions, and we centralized them all in a single person—and that’s not a cross-functional team. That’s—
Shimel: That’s a cross-functional person.
Kalsey: Yes. Well, and you end up with the jack of all trades, master of none problem. You have to have the specialists on even a cross-functional team. You have to have, there’s a lack of testers, there’s a lack of UX people, there’s a lack of security people. They have to be part of that team. You have to have people that, dedicated folks to do the hard work. A developer may be able, and a generalist may be able to do a lot of the easy things and a lot of the, “Oh, yeah, we’ve done this before or we’ve seen this, and this is a known know, let’s go fix that.”
It’s the hard work of figuring out—what don’t we know, and then having the expertise to be able to know what that hard work is, even, to go after it. And without that, of course the developers are going to be overloaded, they’re being asked to do everything and they don’t know what everything is.
Shimel: Agreed, agreed. I mean, I think, again, security is the poster child for this, right? We’re asking developers to be security people or to make security a priority. Now, you ask most security people four or five years ago, they’d tell you you’re smoking your socks. Developers don’t care about security. They never did. Only we care about security.
It’s changed. It’s changed a little bit. We do think other—no one wants to develop insecure code. But we can’t ask them to be security people—or can we, Caroline?
Wong: I sort of think we have to. I think that, you know, the big, bad thing that no one wants to happen is a security breach, and a security breach cripples an engineering team’s ability to execute on a roadmap. You know, if something big and bad happens, then entire teams have to rush and commit to fixing this problem sometimes for months, you know, time which, of course, could be invested more profitably elsewhere.
I have a quote that I’d love to share from a colleague. Denali Lumma is the VP of Engineering over at Project Ronin, and she also makes a really good point about tech debt. She says, at a certain point, it’s too late, there’s too much tech debt, too much complexity, the code base is too big, and it becomes impossible to do anything about it. What you see there is a business will slowly die because shipping features becomes harder and harder, critical vulnerabilities and bugs are introduced more frequently, and eventually it kills the business.
You know, I often think of software as a living organism, right? Whether you think of it as a tree or as an animal, there is this balance between making sure you’re doing sort of your fun things, your running, but you’re also resting when you need to heal. You’re also—you know, if we’re talking about a human being, you’re also sort of going to the doctor once a year.
Shimel: Agreed.
Wong: Yeah.
Shimel: Agreed. Thoughts?
Kalsey: Quality has to be a mindset, security has to be a mindset, user experience has to be a mindset on the developer time, among everybody there. The user experience person needs to understand and have a mindset of developing and of creating a secure experience and not, “Well, we’re gonna get rid of all the security because that’s hard for people to use.” Everybody has to have that mindset, but you still need the experts and you still need the expertise to be able to do things well. Just having a mindset doesn’t make you fantastic at it. You can say, “Oh, I love to get out and I’ve got an exercise mindset, so I’m gonna go run a triathlon tomorrow.” It’s probably not going to work.
Shimel: Probably not.
Ashley: You know, it’s interesting—we could’ve had this conversation 20 years ago, 10 years ago.
Shimel: Absolutely.
Ashley: You know, why are we still frustrated with tests? What is it with tests that is, you know, it used to be sort of the orphan of the development process, you know, the only thing that happens later than test is documentation. Well, now, [Laughter] it’s still an issue. Why is that? Why are we so frustrated with tests today? I’m curious what folks think.
Dawson: I’m gonna start with one not very complex thought—it’s actually hard.
Shimel: It damn right is.
Ashley: A little more respect—we need a little more respect for tests. [Laughter]
Shimel: Well, but this was like, I remember when we first launched devopsy.kinsta.cloud, a popular theme was that, you know, DevOps would kill the testing star, right? That there would be less QA jobs because everything was gonna be automated continuously and shift left, we wouldn’t need testers—poppycock, right? We have as many testers as we ever had, if not more. It didn’t kill off testing jobs. I think it was Adam that said we’ve made testing part of the DevOps team. So, we have people on the DevOps team who are responsible for the various, you know, styles and types of testing that we do, including security.
I think what’s happened, though, is, these DevOps teams—when I first started devopsy.kinsta.cloud, if you spoke to developers, they’d say, “Oh, this DevOps is all about the Ops.” And then you’d talk to the Ops people and they’d say, “This is DevOps, it’s all about Dev, it’s Dev-centric,” right? And that’s the sign of a good compromise when they’re both pointing the finger at that.
But it has come—at least as far as testing goes, from what I see, it has sort of tilted where, you know, as we’ve shifted testing left, we’ve shifted responsibility left as well, which kinda winds up closer to the Dev, or certainly to the DevOps teams.
However, though, let’s look at the alternative, and I’m gonna ask all three of you—all four of you have been around the block once or twice. My personal feeling is, is that we do a much better job of test coverage today than we did 7 or 10 years ago, and I think DevOps and continuous automated testing is a big reason for that.
Thoughts? Anybody disagree? Show me wrong.
Wong: I will challenge you, Alan.
Shimel: Okay. I was hoping someone did.
Wong: I’ll challenge you, and the reason is because—so, we’re all familiar with the OWASP Top Ten.
Shimel: Very much.
Wong: There have been a handful of versions of the OWASP Top Ten since the first was produced in 2003. I believe we’re gonna see another version later this year. The current version is from 2017, and if you take the 2017 version of the OWASP Top Ten and you put it right next to the 2003 version, it is sort of mind blowing how similar they are.
Shimel: Yeah.
Wong: And what that says to me is, we actually know exactly what the problems are. We know how to find them, we know how to fix them, and we know how to prevent them, but they’re still around, and so I think that’s—
Shimel: They are, but are there as many of them, right?
Wong: Absolutely. [Laughter]
Shimel: You think there’s as many cross-site scripting or buffer overflows than there was 10 or 15 years ago?
Wong: So, there are some categories that have been eliminated. You know, there are some categories that, for which we can define a secure pattern, and we can encourage folks to use that secure pattern, and for which we can actually eliminate some of these things.
But there’s others or which we cannot. You know, I think about—I recently visited my sister’s home, and they have two young children, a 2-year-old and a 4-year-old. And at night, what they do is, they run a Roomba, which is brilliant, you know? If you could spend your weekend playing at the park with your kids instead of vacuuming, of course you’d wanna do that. And the Roomba works because they sort of take the time to get all the toys off the floor and make sure everything’s clear and once they use the Roomba, they clean it. There is this different work involved in effectively using a Roomba that’s different from sort of manually using a vacuum.
And I do think there are these two distinct categories of testing, one category which you could say is known, repeatable, and automatable, and another for which manual testing, I believe, will always need to exist. There are types of security flaws like business logic flaws, race conditions, chained exploits that I don’t believe that in my lifetime, we will be able to figure out how to teach a computer to find these particular types of issues.
Kalsey: And it’s beyond just security testing. I mean, exploratory testing, from a tester, your automated tests are not going to find, “This error message makes no sense in this context. I did this, I expected—I don’t know what this error message is trying to tell me to do.”
Your automated tools aren’t going to find that. All they’re going to be able to find is, “It didn’t work, we sent an error message. We had the things we wanted to put in the error message there.” You can test that your code is doing the right thing, but if you didn’t design it to do it right, you’re not gonna find that with an automated test.
Shimel: Fair.
Dawson: I would tend to say, for the surveys that we’ve issued and I’ve read, your original statement, “Has test coverage improved significantly?”—yes, I think all indications are, we are getting better at automated testing, we are getting better at that pursuit of that thing we call test coverage.
But I think what’s more important is, has the quality of test improved? Are we testing the right things, right, from functional testing to security testing, various shades of security testing? And I’d say (1) yes, we have improved there by sort of the qualitative and quantitative measures that I’ve seen. But I also think, as we truly start to focus more on quality, we start to achieve more coverage, we start to implement better tests, it becomes more clear, as I think Adam and Caroline are speaking to how much more work there is to do, how much other things there are to improve testing on. Pair that with a constantly changing technological landscape, right? We’ve gotta learn new stacks, we’ve gotta figure out how to test new stacks. We’ve got half of our put in in a SaaS platform, but then we deploy on prem, et cetera, et cetera, et cetera. And then you add to that that, at the same time, we’re trying to transform organizations.
So, Adam, you talked about exploratory testing; Caroline, the human element. I think one of the things that pursuit of the full stack Dev, that pursuit of the fully automated processes failed to do in a lot of orgs I speak to is, we talk quality first, but oftentimes, the DevOps transformation doesn’t involve quality engineering first, right? And I think one of the things we need to get better at is engaging with and partnering with Quality and establishing our automated and modern processes.
Kalsey: Yeah. We talk good in the testing industry a lot about shift left and that testing is moving earlier in in the process, and too many organizations have taken that as, “Well, the developers are the ones that are early in the process, so that means the developers do it.” No! Shift your damn testers left. Get them involved in the design phase. Get them involved in talking about the architecture. Get them involved in the entire thing, not shift the task to the people who are earlier in the process. Shift the whole process, please.
Dawson: Or to a script, yeah.
Ashley: I’m gonna take a whole ‘nother tack, and I thought you were gonna go here, Brian, is, I think we need to shift tests down. We need full stack testing. It’s not application testing anymore. Most of the security errors that we have are due to configuration errors. And what Caroline said was absolutely true. We still have SQL injection, cross-site scripting, and overflows may be less so. But why? It’s because it’s not just in the app layer that those things happen, it’s in that software stack. And I know not every tester can be an expert at the stack itself, but maybe that’s, instead of kinda dividing the DAST and SAST and the different kinds of testing that we do and just deciding, can we shift those in one direction is, let’s think about it from a depth perspective, because that’s where a lot of those errors are coming from. It’s not just the app.
Kalsey: Oh, yeah, and the tooling on this is also, can be super fragmented. You need one thing to test your infrastructure code, you need another thing to test your UI, you need another thing to do the security—and they’re not, consolidating those reports and combining them and figuring out what worked and everything is hard work and there’s nothing that does that. This is a place that Tricentis could definitely improve and improve our own tooling here.
And then, once you’ve done that testing, the results are often in a silo. Okay, I found out I had a problem—how do I fix it? How do I go see the data behind what was going on in the system to understand this and solve this? And now I’ve gotta dive into a monitoring tool or my logs or an observability tool instead of just having everything all in one place.
Shimel: So, I think there’s a—I’m sorry, Caroline, you go.
Wong: No worries, and thank you, Alan. So, we’ve talked about shift left, shift right. Mitch is kinda talking about shift up and down. I think there’s another axis, if you will, which is sort of what I’ll call shift in and shift out. So, you can test things in and out of band, and that’s gonna make sort of a wildly different difference in terms of how those processes run.
You know, I think 10 years ago, security people appreciated the power that came with being a gate in the process, being an end band process. You know, today, it actually makes sense, I think, for a lot of security testing to be out of band, but that removes the sort of power and authority of security people to say something like, “You must fix this, otherwise you’re not gonna go live if it’s happening out of band.”
Shimel: Fair. So, I’m getting a little dizzy, we’re going left, right, up, down, in, out. But—
Ashley: It’s a circle. [Laughter]
Shimel: Yeah, you know, shake it all about. But here, fundamentally, though, are we just kinda dancing—no pun intended—are we just dancing around the real issue, which is, we wanna do more pre-deploy, we wanna do more, we want better quality software, right? I think we could all wrap our heads around that idea and vote for that. We all want better quality software. Security people are at a premium. We can’t shift security people left or en masse. Testing people are also at a premium, though we’ve integrated them into the DevOps teams. Developers are at a huge premium, they’re probably our most expensive resource in terms of humans.
What do we need to do to make everyone’s job easier here, to make this function better? Is it a question we’ve gotta automate more? I mean, maybe that’s what it is, if we automate more, or is it a cultural thing, right, where—
Ashley: There’s also test-driven design, right, which is, as you’re designing it, you talk about, you put together your test strategy of, “Okay, we do it this way. Let’s talk about how we are going to test it, start to lay out the framework for how we think this can be validated.” I mean, it’s not the only thing to do, but I think—so, if you start with that in mind, now you’ve got a path to pursue as opposed to, “Here’s a chunk of software, you folks figure out how to test it, right? We built it, you test it.”
Dawson: Yeah, and I’d add, I think there’s no—again, not only is the practice of testing harder than we often like to admit, but the, you know, the path to kind of getting to where we fully wanna be, where we don’t dread testing, complex, hard path, and I’d say it’s the DevOps trinity, you know, Alan, and we all can probably hit a few points on each.
But you have to address people and culture, as Adam brought in earlier. There’s no getting away from, security needs to be a thought for everybody, right? I can install all the locks I want on the front of my door. Actually, over 24 years—today is my anniversary—my wife has programmed me to lock the door when I come in the house without thinking, right? She’s kind of impressed a security culture on me.
You do have to address people and culture, process, practice, tools and technology. Automation alone will not solve it. A mindset alone will not solve it. All of that needs to come to fruition.
Shimel: Mitch, make a note—invite Brian’s wife to our next security panel.
Dawson: [Laughter]
Ashley: I’ve already looked up her e-mail address—thank you, Brian, appreciate it.
Shimel: Happy anniversary, by the way, too, Brian.
Dawson: Thank you, thank you.
Kalsey: This is one of the places where I think what Caroline said is so important, that this in bound, out of bound—if you’re trying to do everything in line, of course, it’s gonna be harder. In shifting that mindset to doing things continually in the background, we already do so much stuff and process automation in the background where we don’t have people check, “Here’s a step in the process.” No, this is going along in the background, dependency watching and that sort of thing and updates.
We can be doing testing in the background as well for a lot of it, and identifying what needs to be in the foreground and what needs to be in the background. Look, I’ve seen CI/CD pipelines that take hours and hours and hours to run because the tests take six hours to execute and to build. That can’t work in any sort of a reasonable DevOps system.
Wong: I think there’s two parts to a solution, and I’m really, frankly, just parroting what Brian said. I think there’s a collaboration piece between the people. You know, to an earlier part of our conversation, you’ve gotta have the folks with those specialized skill sets. Ideally, you’re able to get access to those folks on demand, which historically has not been something that’s been possible, but that’s beginning to happen.
The second thing is—so, you’ve got your people collaborating, you’ve also gotta have your technology integrated. So, for whatever security tool or processors, how do those outputs actually get to the engineers? What we need is bidirectional integrations for things like Dev ticketing systems. We need things like APIs for test results. I think collaboration at the people place and integration at the technology place—both of which we could call processes—then it’s just, it’s really a repeat of what [Audio skips].
Shimel: Mm-hmm. I mean, but wasn’t this one of the promises of DevOps? I mean, I got into DevOps because I thought it was a better way to do security, truth be told. But one of the promises of DevOps is, we’re gonna automate more, so we could go faster with more quality—two out of three ain’t bad, right? And, you know, did we lose that somewhere? Is that no longer a goal, or have we taken that as far as it goes?
Kalsey: We keep trying to solve it with technology instead of by looking at the culture and changing how people think. And you can’t think the same way and apply new technology to it and expect the technology just to magically solve all your problems.
Shimel: Brian, you agree, or Caroline?
Dawson: I absolutely 100 percent agree, and I’ll agree and then I’ll just add a point. And I know I’m being a bit nebulous about it, but there’s a human element to this. At the end of the day, it’s humans developing software for humans, right? Whether it’s the code on your electronic control module in your car or Waze on your phone or the OS, at the end of the day, the final endpoint is a human. And you know what? At least now, and I know we’re improving, there’s no way that we can ensure—and I don’t mean to accept security here, but to test the quality of the human experience, right?
And think about this—at the end of the day, knowing that the language on a screen is self-descriptive and informs me what to do. Testing the fact that, or ensuring that when I submit my credit card number it’s not vulnerable, right? Whether we’re talking API testing, whether we’re talking security testing, all of this at the end of the day boils up to customer or user experience testing.
And I guess what I’m gonna posit is that we can never entirely automate that. And where we’ve hurt ourselves and possibly made a false promise is, we’re gonna create a bunch of full stack engineers, we’re gonna get rid of, away from operations functions, testing functions, and we’re gonna use TDD and automate everything from the start, Gherkin, yadda yadda yadda. And I believe—look, that has enabled us to go faster, but our impedance mismatch, the limit we’ve hit right now is where we’ve failed to better connect people and enable them to collaborate and share knowledge. So, not just security, not just back end, not just data, but the entire customer experience is of quality. Dev can’t do that alone.
Shimel: No, sir.
Dawson: I’m sorry, Alan, did I get, did I—
Alan Shimel: No, no, that’s all—I’m saying no, sir, you’re 100 percent correct, but I know Caroline wanted to say something.
Wong: So, I do have a comment about automation. I think that DevOps has figured out how to do automation when it comes to defect discovery. I think we’re sort of good at that. I think what we’re still trying to figure out is automation when it comes to workflow and process so that, actually, again, to Brian’s point, the collaboration can happen automatically, rather than due to a human who’s involved and making it happen. How do we get the right information to the right people in the easiest, most efficient way at the right time, and how do we make it super easy for them to do their jobs?
I do think that there remains—I do think that the next evolution of automation and DevOps is gonna go from defect discovery automation, which I think we’ve gotten quite good at, to workflow and process automation.
Shimel: Fair enough. Adam, thoughts?
Kalsey: I like it. I mean, identifying what you’re automating and what’s working in that automation is important, but also identifying what’s not being automated or what in your automation is broken. And not broken from the standpoint of, it doesn’t execute or it doesn’t run to completion, but broken in, it’s not solving the problem you wanted it to solve.
Shimel: Fair enough. Guys, we have time for maybe one more question. I’m gonna pose it and throw it out there to you. Is it possible we’ve shifted too far left or that you can shift too far left and we need a little rubber band, here, going the other way, a little pendulum swing? Or you think we just scratched the surface? Caroline, you look like you wanna say something.
Wong: I think that’s a great question. I do think it’s possible to shift too far left, and the reason is because, while we, I think, are all familiar with the statistics and the studies that say if you fix something earlier in the cycle, it’s cheaper—there’s stuff that exists to the right that doesn’t exist yet on the left. Software development is, you know, to sort of a little analogy that we talked about earlier, I think it’s a dance. I think it’s a dance between people. I think software comes about as a result of the behaviors and the actions and the decisions that various folks take together. And security and quality are emergent properties. They’re not Band-Aids, they’re not vitamins, they’re not painkillers. They have been, as a result of this dance. So, there are some things for which, if you try to test it too early, it doesn’t actually exist in its final form yet. If I take my 6-year-old to the doctor and then I don’t when she’s 21, you know, might we have caught something super important as a 6-year-old that we were able to step in and address right away that would’ve been great? Yes, but as a 21-year-old—guess what? She’s gonna have different problems that were not apparent at age six.
Shimel: Spoken like a true mom and parent, right? Parents, we worry about that. Anyone else? Thoughts on that? Can we go too far left, or have we gone too far left?
Dawson: I, Alan, think it just depends. I think we’ve interpreted left different ways. I think it’s a good, catchy term. It’s an important and powerful catchy term, but as I’ve heard others discuss, we sort of overemphasized this idea of left. One, to Caroline’s point, right, we just do everything here and we’re done, good to go, right? But then this idea that we can put everything in Dev? Well, no, I think—I think, for what it’s meant to communicate, shifting left is an apt term.
I think in terms of ensuring that we’re doing everything we can as far left or as early in the process to mitigate issues later—great. Right? But we don’t just shift security left. Right? Like Caroline said, inbound, out of bound. Actually, we need to shift security everywhere. We don’t just shift quality left, we need to shift quality everywhere.
So, I think there are—we need to be ensuring that it’s almost zero trust software delivery, right? At every stage of the process, from concept where you should be sitting down with QA and the security team to capture the requirements to test, to deploy, to run, we need to be thinking about it, also.
Alan, I’d say shift left is good—shift everywhere is really what we meant to be doing, right?
Shimel: It’s better. Great.
Kalsey: Any time we engage in sloganeering, there’s gonna be people that pick the wrong things from the slogan. I mean, “Hey, we have DevOps. That means we don’t need Ops anymore!” And that’s not what it meant, and there’s people that have taken the wrong cues from shift left, and instead of shifting the processes left, they shifted the tasks onto the people who were left already, and that’s not gonna work.
Shimel: Absolutely. Hey, Mitchell, you wanna tie a bow on this?
Ashley: I’ll put a bow on this. You know, there are admittedly many, many things that I don’t know, but one of the things that I do know is that, if a developer says it’s tested, it’s ready and a QA person says it’s tested and it’s ready, five out of six dentists will choose the QA person. I will, too.
So, there’s something there. There’s a reason why we don’t have DevQA or DevTest. We don’t talk about it that way, we talk about DevOps or DevSecOps. QA is extremely valid and necessary, and we can’t live without it, and I think we need to invest in helping, elevating how we test and how we think about our software applications to test in. So, those dentists will be happy.
Shimel: And Crest, the Crest test for QA. Thanks very much, Mitchell. Caroline, Brian, Adam—thanks for joining us on this episode of DevOps Unbound. Great conversation. For our audience out there, do check it out. We have another episode coming up in—well, not next week, the week after, and then I think we have a roundtable the two weeks after that.
But for now, though, we’re gonna wrap. Many thanks again to Tricentis for sponsoring DevOps Unbound. Thank you for watching, thanks to our panel. This is Alan Shimel, we’re out.
[End of Audio]
-
EP 14: What Have We Learned?
The past year-plus has brought disruption and opportunity to businesses and software teams all over the world. Change is inevitable, but 2020 was a year of massive change: businesses pivoted, supply chains were disrupted, customer expectations shifted to contactless services, digital transformation projects accelerated, new digital experiences were created, the move to the cloud jumped into high gear and our ability to create and deliver software was elevated to an essential capability for business.
It’s time to pause and ask ourselves, what did we learn from all of this disruption and change? What lessons can we carry forward? How do those experiences inform how we prepare for 2021 and beyond?
Cornelia Davis from Weaveworks, Kurt Chase from Tricentis and Sanjeev Sharma from Truist join hosts Alan Shimel and Mitch Ashley in this episode of DevOps Unbound to discuss lessons learned, how to overcome the obstacles we’ve faced over the past year and how to improve business processes and make better decisions. The video is below, followed by a transcript of the conversation.
DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.
Alan Shimel: Hey, everyone, this is Alan Shimel, and you’re watching DevOps Unbound. For those of you watching for the first time, DevOps Unbound is a biweekly video series where we tackle interesting topics related to DevOps. Once every four to six weeks, we do a live roundtable with a live audience questions and live audience participation, and we invite you to our next one. You can find that scheduling at DevOpsUnbound.com.
Today, though, is a session just with our panel, and what a panel it is. Let me introduce you to them. First off, let me introduce our friend, Sanjeev Sharma of Truist. Sanjeev, maybe you wanna introduce yourself to the audience.
Sanjeev Sharma: Sure. Thanks, Alan. First of all, thanks. Great to see you all. Hopefully, we will be able to meet in person soon, but thanks for having me on. I am the Head of Automation and Platform Engineering at Truist Bank. Truist, for those of you who might not know, is the new bank that was created last year by the merger of BB&T and SunTrust. I have various hats I wear at Truist, but my background is mostly in the DevOps and cloudware adoption space.
Shimel: Excellent. Thank you, Sanjeev, and welcome. Sanjeev, I should mention, is also the author of The DevOps Adoption Playbook.
Next, we’re really, really pleased and honored to have Cornelia Davis with us. Cornelia is, of course, a pretty well-known thought leader and leader at the DevOps space, but is also at Weaveworks. Cornelia, welcome. If you wanna give a little background?
Cornelia Davis: Sure, yeah. Thank you so much, Alan. It’s so great to be on again. Yeah, I am the CTO at Weaveworks, and you might know Weaveworks as the company that coined the term GitOps, that has been kind of driving that movement. And GitOps is just an element of a broader category, which is really developer platforms, which is what we’re talking about many, many times when we talk about DevOps.
And so, prior to Weaveworks, I’ve been here for about a year and a half, I was at Pivotal, where I worked on Cloud Foundry and Kubernetes based developer platforms, you know, platform teams for enterprises for quite some time. And I’m also the author of a book called Cloud Native Patterns, which is a book aimed at architects and software developers, helping them learn how to build software for this highly changing, highly distributed—constantly changing, highly distributed environment that is the cloud.
Shimel: Absolutely, thank you. It’s a pleasure to have you here. Last but not least on our panel today is our friend Kurt Chase with Tricentis. I should mention Tricentis is a sponsor of DevOps Unbound, and thank you, Tricentis, for sponsoring. But Kurt, why don’t you give them your background?
Chase: Yeah, thanks, Alan. Great to be here today with Cornelia and Sanjeev. Nice to meet both of you. I’m currently running Release Management and Engineering Services at Tricentis, which, Engineering Services is a group that provides cross cutting services primarily for internal engineering, but effectively for the whole corporation as well.
Prior to Tricentis, I spent years at Splunk, again, supporting the engineering function, all the CI/CD builds and releases and things like that. And then prior to that, I spent almost two decades at Autodesk, producing the AutoCAD, multiple releases of the AutoCAD software product.
So, again, it’s great to be here today with all of you. Excited to talk about some of the things we’ve learned over the last year, year and a half—and what a year it’s been, holy cow. [Laughter]
Shimel: Yes, it has. Then let me finally introduce my co-host here at DevOps Unbound, my good friend and partner, Mitchel Ashley. Mitch, why don’t you give a little background?
Mitch Ashley: Thanks, Alan, and of course, I’m excited to have our panel—thank you for joining us, all of you, today.
I’m Mitch Ashley, CEO of Accelerated Strategies, an analyst firm that focuses on DevOps and cloud native, cyber security, digital transformation. I also work with Alan as part of MediaOps, the folks that put on this event, with the CTO of that organization. So, it’s a lot of DevOps practicing and observing what others are doing, so I’m excited to hear what Sanjeev and Cornelia and Kurt have to say, too.
Shimel: Alright. Thank you, Mitchell. And as I mentioned, DevOps Unbound is sponsored by Tricentis, so—many thanks to them.
Today’s episode, folks, as I mentioned off camera, I don’t wanna jinx anyone or anything, but at least here in the U.S., we seem to, at least we hope that we’re seeing this whole COVID life sort of starting to recede in our rearview mirror. We are returning to normal or a new normal, some may say recklessly too quickly, or some may say not quickly enough, but certainly, the return to normal is on, and accelerating.
But, you know, there have been many, many lessons, many, many experiences, some painful—extremely painful—from a business and a humanity point of view over this last 14, 15 months or whatever it’s been. There’s also been a tremendous amount of success, a tremendous amount of acceleration, a tremendous amount of, “Jeez, this really—this worked. We didn’t know if it would work, but it worked,” probably the Internet being the first thing, right? Where would we be with this without an Internet?
But can we—let’s take a moment, and let’s, as I say, hope and pray that this is in our rearview mirror or starting to move to the rearview mirror. What are the lessons? What can we learn, what will we take forward from this? And, you know, I don’t wanna boil the ocean, because I think there’s a lot, but if I asked each of you to give me three things that you really, you think—not just you personally or your organization, but the tech industry has learned, the business world has learned, perhaps society in general, humanity has learned, what would you say some of those things are?
I don’t wanna put anyone on the spot, but Kurt, you always have something to say. I introduced you last, we’ll start with you.
Chase: Sure. I’ll jump right in, and one thing that’s amazing to me, not so much from the technology or engineering or R&D side, let’s talk about business, here—the other aspects of the business that were able to adapt to remote work, that, to me, was pretty incredible.
When COVID really took hold, I wasn’t too worried about software engineers and folks producing software working from home, doing their coding, testing, and all that. The other functions—finance, legal, things that are required to ship your products—seeing those go remote at the pace they did, that was pretty incredible. I think they definitely took advantage of some of the learnings from their individual engineering teams. But to me, that’s one thing that sticks out right away is the speed at which businesses adapted to working remotely and the success they achieved across the functions in the business. That was incredible to me.
Shimel: Agreed. I mean, the speed of adoption was something. Cornelia, did you wanna say something?
Davis: Yeah, sorry, I was muted there. I did wanna chime in, because I agree, and one of the ones that really knocked my socks off was how quickly call centers went remote.
Chase: Yeah. [Laughter]
Davis: Because that isn’t something I would’ve expected. I would’ve expected them to still be kind of really tied into their telephony services and, you know, the phones that are on their desks.
But I spoke with many, many people, you know, various customer service people from the banks and from, you know, the Amazon.com—well, there’s not that much of a call in center there, but from all of these different organizations where, when you chatted with these folks, they were like, “Yep.” Within a week, they were able to move everything, move everybody remote in these traditionally really physical call center based systems. Which means that there had to be some preparation. I don’t know if it was intentional, but there had to be something that allowed them to flip that switch so quickly. Is it just forwarding phones? Is it just VPNs? I don’t know.
Shimel: No, no, it was—so, I have some info on that, if I can. I interviewed a fellow down here in Florida, I think he was out of Tampa, and he operated multiple call centers, like in Costa Rica, which is actually a call center capital, believe it or not, that’s a big business down there. It turned out that the software they used, right, people in call centers, they don’t just—Kurt just doesn’t call someone randomly. Everyone gets assigned their calls or their inbound when a call comes in and gets assigned.
The software they used within the call center itself worked just as well, the person didn’t have to be in the desk next to you, he could’ve been anywhere, it turns out, or she could’ve been anywhere. And so, it worked just as well in a remote thing. So, they still used the software they were using that tracked how many calls they did, who they spoke to, the notes, putting it into the central database. Yes, they had to be cloud-based, right, because there was no one in the call center to kinda run that infrastructure. But the software, it turned out, worked remotely as well as it worked on everyone’s desk in the call center. It was an amazing thing.
And so, what they found—again, and this is gonna be, I think, something that’s gonna be a recurring theme—who needs the physical call center? They don’t need that infrastructure anymore. They could have people working at home just as well.
And so, the more advanced ones—and you’ve probably seen this yourselves—you call someone, they may be very well at home, but they’re giving you the same mediocre service that you get when you call the call center itself, right? [Laughter] But yeah, no—
Chase: And that’s the point. I think with a business, it’s applicable across functions there, it’s amazing. And business learned that they can still be extremely successful, even if everyone’s remote. And having that pool of remote talent—that changes everything. [Cross talk]
Sharma: I think a couple of things to be noted there, right? First of all—sorry, Cornelia, I think I spoke over you. You know, one is, the pendulum also needs to swing back from that extreme.
Chase: That’s the challenge, yeah. [Laughter]
Sharma: Right? Because, while we have people who are able to work from home because the technology is there, from an equity perspective, one has to look at people who cannot work from home who are being forced to.
Chase: Yeah.
Sharma: And we experienced that when we opened up our offices. We are just beginning to open up our offices. June 1st was the first day folks would come back. And I was surprised to see some people had jumped at the opportunity and said, “I want to go back.”
Chase: Yeah.
Sharma: Then when you look under the hood and you talk to them, you realize, you know, they’ve been struggling. They’ve been operating out of a tiny closet, right, because they don’t have the space or because they have toddlers or they’ve been struggling with Internet issues, and it is worse overseas than it is here in the U.S.
So, I think the pendulum needs to swing back to a more normal position, but hopefully, it won’t swing all the way back to where they say everybody has to be on site, on premises, right? So, I think that is definitely, some balance has to happen to recognize—some people do need a workplace, where they can work comfortably.
Chase: You’re absolutely right.
Davis: Yeah.
Chase: I know a story of one individual who was in a one bedroom apartment in San Francisco and after three weeks, he couldn’t handle it. It’s like—I don’t have no space.
Davis: Yeah.
Shimel: It happens. Cornelia?
Davis: And, you know, honestly, I have space, and I have a wonderful office and I’m super comfortable, and I’m also desperate to get back into the office.
Chase: Yeah.
Davis: Because—and I love, Sanjeev, that you called out the equity thing, because that’s something that is so painfully obvious in the whole pandemic is that this has only broadened the inequities that we have across society, and we need to absolutely focus on that.
But then the other lesson, you know, with me, I’m extremely privileged, I live this super comfortable, I never really had to worry about catching COVID, I could just choose to stay home. But humans need humans and, you know, we’re all coming to you on your two-dimensional screen. I am super fortunate that last week, I had the opportunity to have my first face to face with a couple of colleagues in over a year, and it was rejuvenating. I’ve been saying, it’s a 3D work world again instead of a 2D work world.
Shimel: Yeah.
Davis: And so, I mean, humans need humans. But you’re right, Sanjeev, I think that the even bigger thing is that we really, really need to look at the inequities. And in the beginning, in the early parts of the pandemic, without getting too political here, we had all of this narrative around these people who we considered essential workers who were kind of the forgotten, like, taken for granted folks, and they were put on a pedestal, and that, unfortunately, didn’t last that long.
Chase: Yep.
Sharma: Yeah.
Davis: And so, we need to bring that back.
Shimel: And they weren’t paid like they were that essential, either, but that’s [Cross talk]—yeah, it was a pretty low pedestal. But let me, you know what, you bring two things—I’m sorry, Mitch, I just wanna make two quick points.
Number one, when we talk about the inequality, let’s be really clear. This has affected women and mothers especially, more than men, right?
Ashley: Mm-hmm.
Davis: Absolutely.
Shimel: Mothers have had—because not only have they had to work from home, but they’ve had to take care of the kids who are not in school. And it’s very hard being a teacher, a home care provider, and doing your work at work. And so, how many women have had to work out of the workforce because they had to take care of their kids and their kids are home and they didn’t have an option, there was no child care for them. So, I think—
Ashley: And the challenges of re-entering the workforce.
Shimel: Right, and now they’ve gotta re-enter after, you know, months off, and that’s never easy. People wanna know, “Well, why the gap in your résumé?” Well, there’s a gap in my résumé because I have this beautiful child that I’m raising, you know, and how dare you? That’s number one.
Number two, it is a case of the rich get richer, right? Companies that, not only individuals as you say, Cornelia, but companies that were able to leverage this. Man, they accelerated, but they accelerated at the expense of their competitors who couldn’t, right? Think about all the open retail shops in the malls near you and the strips, right?
Davis: And this is a place where I’d love to pull DevOps into the conversation, because we are, of course, as a part of DevOps Unbound. One of the things that I thought was impressive—and I’ve worked with some of these companies. In, you know, my previous life at Pivotal, I worked with The Home Depot, for example. When you had The Home Depot and you had these other organizations that, in a matter of weeks, updated their software so that you could do things like curbside pickup? That right there is DevOps in action.
Shimel: Yep.
Davis: It is the ability to release software, get feedback, release software, and there was no six-week testing cycle that we used to do, or eight-week testing cycle.
Shimel: Mm-hmm.
Davis: It was like, “You know what? We are gonna be testing in production, because we didn’t have time to test this for six weeks.” So, it brings in all of these tools that Kurt, I’m sure, you have day to day experience with, which is these things that allow us to put safety nets in place to be able to release all the time. I mean, that is something where the people who were along their DevOps journey already had a leg up on those that didn’t.
Ashley: You know, Cornelia, I had that exact experience where I couldn’t get a computer, you couldn’t get one shipped to you, they just weren’t there, so I had to build a computer for some video work, and did it solely from parts from Best Buy, by curbside pickup, two Best Buys in town.
Davis: Yeah.
Ashley: So, it literally was, like, a week later, all of a sudden, I could do this, and I was pretty shocked by it.
One of the things I wanna say, though, about what we’re discussing is, the people dimension of it is extremely important. I think there’s a whole other set of lessons for the business. I think we’ve pressed reset on a lot of assumptions about how businesses have to operate, that you have to be in an office. And essentially, we’ve kinda globalized the business by just taking the location out of it now. And yes, there are privileged and underserved and people who aren’t privileged. And those inequities probably are gonna get bigger, not smaller, because I don’t think we’re gonna all slap back into the office immediately.
But what the businesses learned is that lesson you just talked about with Home Depot and you with Best Buy is—you know what? We can decide we’re gonna pick Team or Slack in two days instead of two years. We can get a software release out in a few days if we have to. Talk about constraint based management—you know, we now have this constraint and everything else comes off.
So, I think what it’s done overall, though, is for DevOps and for all of us is, it’s moved software from maybe second or third to number one. Our ability to get software out the door that’s effective, meets what we need it to do, but that is a strategic advantage, maybe even a survival advantage of what we’ve seen happen in this past year.
Chase: Yeah. I think some of it is, you know, businesses like that were forced to do it or die, basically, you know? And I 100 percent agree with Cornelia that I think they look to DevOps and some of the practices that they’re all used to. You know, the collaboration we need across functions to move forward and all those things are key, some of the foundations of automation, you know? That has been key in the DevOps space, and I think they learned, they need to take advantage of automation to do these things as well.
So, yeah, it’s been fascinating to see that. And that was a point I brought up to see how businesses transformed and our world changed. And I think it’s opened up endless opportunities for us moving forward. And to Sanjeev’s point, I hope when that pendulum swings back, corporations take a look, that this is no longer a corporate decision where people work, it has to be more joint. It has to be more personal, because every situation’s different, and if you wanna maximize all your programs in HR, diversity and all those things, you have to take that into account. I don’t think you can do a one size fits all any longer.
I still wonder if all those big office buildings, I worked in downtown San Francisco for many, many years—are we gonna fill those up again, ever? I really wonder now if that’ll ever happen. I could see it not happening, quite honestly.
Ashley: I’m curious, Sanjeev, you’ve been in the middle of this bringing two companies together with this new bank in the middle of all of this. What in God’s name was that like? [Laughter] What are the additional challenges you’ve had for that, or maybe what opportunities has it created?
Sharma: Yeah, so, I think it—you know, I joined in the middle of the merger, right, so we were in the middle of it when I joined. So, what I can tell you in the 10 months I’ve been there, we have shifted a lot of the priorities because of the pandemic, right? A lot of things which we would have done later on, we did first. We were able to do things.
But the flexibility has definitely opened up. I can tell you, between the time I joined and between now, the ability to hire people who are not local to one of our offices has gone through the roof. It used to be an exception, now it’s the norm, right? The last three people I hired, two are in the West Coast and one is in Minneapolis, right, so Midwest. It doesn’t matter anymore, right? The same way there are—the operation has opened up a lot.
And when I look at it broadly, outside of who I’m working with, when I’m talking to other companies where I have friends or colleagues, they are seeing more offshoring happen, right? Because they’re going, “Why do I need to hire somebody in L.A. if they’re never going to see the person ever anyway, does it matter whether he’s in Bangor or Bangalore, right? Let’s put the person, let’s hire the right person at the right place.” So, I think a lot of that has opened up. It has, of course, shifted some of the salaries, also.
So, you know, a lot of things are going on, interesting dynamics are happening, and we’ll see how that all pans out. But from a perspective of, you know, doing the transformation, it is hard when you can’t have people in the same room on a white board and things which could have just drawn on a white board and knocked out in a 1 hour or 90 minute working session, now it’s like 15, you know, Zoom views, WebEx or Teams calls, which could have been, and it’s like, “Where is that white board we made up? Was it on Miro or was it on Mural or was it on this or was it on that? You know, and who can edit it, who can’t edit it?” All those—you know, dang it, if we just had a white board and all of us in the same room, we could have knocked this out in half a day.
So, it is difficult to do certain things remotely. I think there has to be a balance, which needs to happen. And yeah, during a merger just made it more complex, because we are going through change every day, right? People’s networks are changing, people’s—how we are doing things are changing, we are updating stuff. It’s been a very interesting few months and we’re not done yet. You know, the merger isn’t over. We’ve still got a long way to go. So, it’ll be more fun going forward, I think.
Shimel: I think it brings up the issue, though, of how much of this sticks, right? Because, I mean, look what Sanjeev just described. We’ve all had similar experiences, I’m sure, hiring people from all over the world—what difference does it make? They don’t have to come in the office.
Does that stick? Because if it sticks, there are profound implications for society in general. Why live in San Francisco unless you love the view of the Bay. It’s a beautiful bay, right, the bridge is nice, but pay $4,000.00 for a closet-sized one bedroom apartment? Maybe not. I mean, on the other hand, I live in South Florida. Housing market is insane because of people moving here from New York and California and spending a fortune—Colorado, right?
So, does this—that’s one example. Does the ability to work from anywhere, it makes no difference, is that something that sticks around, or do we, as Sanjeev said in the beginning, see a pendulum start to swing back? What do you think?
Chase: Well, for me, like I said, I think elements of it have to exist and remain. I think there needs to be a balance, as Sanjeev said. I no longer think it’ll entirely be a corporate decision where employees are located. I think for some roles and groups in organizations, there needs to be a deeper conversation on where is the best talent, what’s the best way to use them? You know, having people sit—just the commute. The one thing I don’t miss is the commute. Oh, my God, you know, those three hours a day commuting, you know, that, to me, we have to somehow retain that benefit, you know, from working remotely somehow. I’m not sure how.
But that’s a very good question, Alan, I don’t know. It’s like, will those office buildings ever be full again? I don’t know. I’m not sure.
Sharma: On that point, I’ve been remote since 2013. In 2013, I was at IBM. I moved to a global role, and since then, I haven’t had an office. I’ve worked out of this room since then. But I used to travel 150 to 175,000 miles a year.
Chase: Got it, yeah.
Sharma: Right? So, for me, the office was a place to go collaborate, whether it was my office or my company, IBM—
Shimel: And for all of us.
Sharma: – right, or a client’s office, it was not a place to go and have a desk, right? My desk is here, right, my books are here—you know what I mean, right?
Chase: Yeah.
Sharma: I think we should, I would like to see us go to that model where offices become more of a collaboration space rather than a place you have to show up and work alone in a cubicle. What’s the point of that?
Ashley: I agree.
Sharma: Unless somebody needs it, right? That’s different if they don’t have space at home and the need a cubicle to work in—fine, put them in a WeWork next to where they live, right, close to where they live rather than have them commute to the other, you know, 45 minutes one way.
I think that’s the model we should head towards.
Chase: I agree.
Sharma: And, you know, the tools to enable that are there, we’ve proved it. We’ve proved, during the lock down, during the pandemic that the tools are there, but I hope it goes that way. I hope it doesn’t go back to, “You have to show up in this cubicle” even if you are, you know, like the guy from Office Space protecting the stapler and never talking to anybody.
Ashley: The butts in seats model, right? [Laughter]
Chase: We’ve seen some large organizations [Cross talk] make those calls already and say you can be remote forever now, you know, things like that. So, I think that’ll become a bit more selective, but I still hope, when that pendulum does swing back, that the employee has a say, you know, and his voice is a little bit taken more seriously.
Shimel: Or her voice.
Davis: Or hers.
Chase: Or hers—or hers, [Cross talk].
Davis: Exactly, that’s where I’d like to chime—I’d like to chime in there, because Alan teed this up earlier and then, because you had two things, Alan, we followed thread number two.
I wanna come back to thread number one, which is that this has affected women in the workforce, and it’s not just us technologists, but it’s a broad spectrum, and it’s probably even more so in other working arrangements. And so, I love what you’re saying there, Kurt, about it being a partnership between the corporation and the employee and determining what’s best for them. And then Sanjeev, you, I think, just teed up—is that the survival answer for WeWork is that part of that negotiation between the corporation and the employee, she or he, includes having the ability to move into a workplace that is close to your office but allows you that space that you need, for example, not to be on call for child care.
Of course, the other thing I can’t help but say—and I’m very fortunate, I have one son, he’s 26, I’m not tying shoes or going to PTSA meetings anymore—
Shimel: I hope not. [Laughter]
Davis: – so, I wasn’t affected in that way. Yeah, [Laughter] I wasn’t affected in that way in the last year, but I was a working mother and I worked from home that whole time. And I just wanna remind everyone, men and women, that you’re right, women take on that burden, but it doesn’t have to be that way, fathers.
Shimel: Amen.
Chase: Mm-hmm.
Cornelia Davis: So, please pitch in even more than you think, just because it’s been kind of the standard operating procedure. If nothing else, I hope the last year has maybe, in some cases, resulted in a, hey, a need-based way of rebalancing responsibilities. That is something that every single individual, male or female, can participate in kind of these societal expectations and challenging those.
Ashley: I think something happens—
Shimel: You know what? Oh, go ahead, Mitch, I’m sorry.
Mitch Ashley: When things open up, when opportunity happens and things make that shift, it’s different when there’s opportunity there, but when things have shifted and made a shift, they don’t snap back to the old way automatically. It transforms sort of the landscape going forward, and there’s so many things about this—and you mentioned women in the workplace, we also have people of color—and we think about the emphasis of diversity that’s happened over the last year and a half, which is fantastic. And we see so much more bringing to the forefront of women in leadership positions, people of color being involved in security more, whatever it might be. You know, I don’t think that that changes.
So, hopefully, those people who now are getting to participate in a bigger, broader way or have more opportunity to do that, (a) I don’t think this is gonna go away, (b) that’s gonna change things for us, right? You know, maybe, Cornelia—[Laughter] here, I’m speaking about what I don’t know about—but maybe more women in leadership positions make some of those, it’s assumed that the woman at home does the child care or whatever is part of our societal transformation, that it isn’t necessarily an assumption. Hopefully we can get past that and remove that barrier for people so there is more opportunity.
Shimel: So, Mitch, on that front, though, here’s the deal. Number one, I think the move to diversity is not necessarily COVID related, I think it’s just an idea that its time had come, it has been a long time coming, number one.
Ashley: I totally agree.
Shimel: Number two, I think one of the positive things that happened is, it wasn’t just the woman stuck at home taking care of the kid. She was stuck at home taking care of the kid while the husband was trying to carve out a desk space in his bedroom somewhere to try to do a meeting or something, right?
But what often happened, at least among my friends—look, my kids, also, Cornelia, they went away to college even though their college was online, just to get out of the house from their mother and father who wouldn’t let them breathe. But most—like, people I know whose kids were younger, the mother and father were home. They had to split up, delegate duties. You did the wash, I cook the meals, you clean the table, I took the kids here, you took the kids there or entertained them now because you had work to do, I had work to do.
In a best case scenario, it became very much a shared responsibility. Of course, not every scenario or every instance has that best case outcome. There’s a lot of people [Cross talk], you know?
Ashley: I think it’s been a journey to share. It may not be quite as, like, “Okay,” [Cross talk].
Davis: Yeah, yeah.
Shimel: Well, that goes, this is—
Davis: I mean, you’ve painted a fairly ideal picture.
Shimel: That is an ideal picture, you’re right.
Davis: Yeah. You know, this conversation is making me wonder, and I don’t have these stats, but I wonder if anybody else has heard the stats—you’re right, absolutely, that this shift towards more underrepresented folks in leadership positions, women, people of color, and so on, that absolutely started before COVID.
I wonder how COVID has affected that. Have the numbers accelerated, have they decelerated? I think—I suspect the latter, and I’m pretty much an optimist, but I’m also a bit of a realist, and I wonder whether the last year has slowed down progress when it comes to diversity at the leadership level. I don’t know. It’s just a gut feeling. I’d love to see some metrics on that.
Sharma: I can tell you, Cornelia, in IT roles, it has always been very difficult to hire people from minorities, right, underrepresented minorities.
Davis: Yeah, of course.
Sharma: It always has been. I personally believe it has gotten worse, because if you look at especially leadership roles, senior roles—we’re not talking fresh college grads—most of those roles, no matter what anybody says, happen through referrals and connections, through your network.
Davis: That’s right.
Sharma: And that network stalled. That network stalled during the pandemic, right?
Shimel: Yeah, networking, yeah.
Sharma: Right? It is as big as it was then or grew slightly, right, that networking opportunities were gone.
Davis: Yep.
Sharma: No Meetups, no conferences, no lunches, no power breakfasts, no women in technology meet ups—nothing. The network stalled—you only knew the people you knew, right?
Davis: Yep.
Sharma: And I think that hurt, bad. I find when I’m trying to hire, when my peers are trying to hire, we are struggling more to have—see, you can’t hire when you don’t receive the résumés. When there are no résumés coming in, how do you hire, right? And I’m struggling, and if anybody has, if either one of you could—any one of you has any suggestions on how to fix that, I’m all in. Please, I’m all ears.
Davis: I think—
Chase: Yeah, I was gonna say, you pointed out an anti-pattern to what I was gonna say, Sanjeev. I was gonna say, at least with COVID and all the remote work, our talent pool has opened up worldwide, but you’re saying that hasn’t helped in that regard.
It has for me with some of the roles I’m hiring. Now granted, they’re not Director, Senior Director, VP level roles yet. But for the other roles, having the world as my talent, you know, the talent pool versus the United States, that makes it [Cross talk].
Sharma: Well, I look at it a bit differently. I look at it a bit differently, Kurt. You know, hiring—how do I, I wanna make sure I say this properly. Hiring an Indian in India is not hiring a minority.
Chase: Yeah, true. Yeah, true.
Sharma: Hiring an Egyptian in Egypt is not hiring a minority. Hiring a Turkish person in Turkey is not a minority, right?
Shimel: Right.
Sharma: They have to be a minority in where you’re hiring them, that’s the only way it works.
Shimel: Agreed.
Davis: Yeah, I think, [Cross talk] I wanted to comment on your remarks, Sanjeev. I think that you’re spot on. Because there were opportunities, and some of the techniques that we have used in corporations to source diversity candidates or underrepresented candidates, those things went away. So, being able to go to a historically black university or go have your recruiter show up at Women Who Code meet ups, those went away, and those aren’t the same.
Sure, Women Who Code still had meet ups line, but it’s not the same as walking around during the social hour and saying—or, I’ve been at many meet ups where, before they start the program, they say, “Okay, who’s hiring?” And everybody stands up and then, “Who’s looking?” Everybody stands up. “Alright, you folks, talk.” That was a very organic thing that was not just so easy to say, “How do you do that in an online setting?”
You’re right, it’s—I hadn’t even thought of that, but that whole network is the same now as it was a year ago is remarkable, especially when it comes to leadership.
Sharma: Exactly. That’s why hiring from a historically black university, we can still do that today, because the recruiters still know the counselors there, they still know the placement office or whoever does—you know, those networks are still there. It’s just that the network hasn’t grown. The higher you go, right, the air gets, you know, rarer, right, and those jobs are fewer and you are not screening 100 résumés to do that. You’re relying on the network more than often, no matter what anybody else says, right?
And that network stalled. I mean, who have we met at a conference who we didn’t know before? Now, when I go to a meet up and I see somebody I know—
Davis: Yeah, but I think what you’re pointing out—
Sanjeev Sharma: – I am IM-ing that person, not some stranger who I’ve never met, right? Unless there’s something really, some reason to do that. Then—no, no, it is the opposite, it has really hurt. I think we need to fix this.
Davis: Yeah.
Sharma: We need to put this [Cross talk] the moment we come back.
Davis: Yeah, but I wanna suggest that—yeah, but I wanna challenge that assumption, which is that I hire leadership from within my network. Maybe that right there is the thing that we need to work on.
Sharma: Well, I’m with you. We need to work on it. What I’m saying is—that’s what I was saying. I was saying that what we tend to go, tend to—we tend to grow our network to find leadership role people, right? People in leadership.
Davis: Yeah.
Sharma: When you can’t grow your network, you’re reliant on what’s already in your network, right? I mean, otherwise, you’re scouring LinkedIn or some other website or pinging people in your network asking for referrals.
Davis: Yeah.
Sharma: But that network stalled. That network didn’t grow. I can’t point to 50 people I met in the last year who were cool and interesting who I did not know before who were underrepresented minorities have I would have met, had I gone to Reinvent or IBM Think or, you know, DevOps World or KubeCon or whatever. All the people I would have met, I didn’t meet them.
Shimel: So, that’s a whole other thing. Virtual events, right?
Ashley: Mm-hmm.
Shimel: We’ve all spent significant portions of our lives at conferences. Cornelia, that’s where I usually see you is at conferences—Sanjeev, as well. Virtual events are cool. We put on one a month here at MediaOps. Gene Kim does his DevOps Enterprise Summit virtually, people even paid money to attend it.
And all of the virtual events, they try to do a social thing to it. It’s just not very good, frankly. It just—it’s just not the same as being in the hallway or grabbing a drink at the bar or a soda or whatever, you don’t have to be drinking. But, you know, just not—it’s not that, someone said it before, face to face, person to person. I think it was Cornelia, you recently had an in person. It’s not the same, it’s not the same.
I mean, so, different than the three of you, Mitchell and I have had a chance to meet hundreds of new people this year. I do three to four to five of these interviews a day. My day starts with talking to people in Asia or Israel and goes to Europe and through the Americas, all the way to the West Coast. And what I’ve found is that I’ve become a resource for people like you, Sanjeev, who say, “Hey, I’m looking for someone to head up cloud or DevOps or testing” or what have you. And I go through, who have I interviewed recently?
But that’s not the way it’s supposed to be. And the question is—and we’re getting near on time—I wanna bring it back to DevOps and software, right? And, you know, what—
Chase: I think you touched upon it from the beginning, Alan, it’s the Internet. You know, the fact that our technology infrastructure was able to shoulder the burden, that was pretty incredible to me, too. You know, I expected Zoom to go out for weeks, if you will. But I think it demonstrated that the technology infrastructure we have is pretty solid. You know, it allowed us to scale up pretty quickly, pretty massively and, you know, there were outages here and there, but I would’ve expected a lot more.
Shimel: Minimal—minimal. Even counting all the DDoS attacks and the ransomware and all of that nonsense, it was still minimal.
So, we had that infrastructure, but we also had this incredible capability to accelerate, to automate that DevOps kind of has been teaching us for seven, eight, nine years or more now, right? And that really helped. Curbside pickup and the—it’s not just curbside pickup, it’s the back end, obviously, right, that allows you to pull into that parking space and hit the button and say, “I’m here, and this is the car I’m in,” right? That kinda stuff, how many companies and jobs were saved by that kind of capability.
Chase: The other thing for me internally is, I think it also opened up the ability to question everything, just because—at all levels, you know? Whether you’re designing a new feature, adding something to a product, designing a new process—is that the right thing, just because we’ve done it like that? I mean, really question it, because I think it, I mentioned this before. I think the opportunities ________ are endless now, really.
We haven’t even started to take advantage of the things we’ve learned. I believe we’re starting to now, as Sanjeev mentioned, the pendulum coming back. Part of that is some massive learnings that we’ve also had forced upon us, if you will. But I think we’re gonna come through it even stronger and better, for sure.
Davis: Yeah.
Sharma: I think the—
Davis: You know, I wanna, I’d love to chime in on—
Shimel: Go ahead, Cornelia.
Davis: – yeah, I’d love to chime in on that scale. You used the word scale and these systems were able to scale and handle just a tremendous surge in traffic.
Shimel: Oh, yeah.
Davis: And when you said that, it took me immediately back to a DevOps Enterprise Summit where Nicole Forsgren and Jez Humble—and those of you who don’t know them, you should know them.
Chase: Yeah, we know them.
Davis: State of the DevOps Report, State of the DevOps Report, the book Accelerate. I remember this moment where they were on stage, and Jez Humble, in his very humble way, hand in his pockets, says, “Turns out, architecture matters.”
Shimel: Yeah. Right.
Davis: And so, it’s having these systems that are designed to scale, and we find so many enterprises that have said, “Oh, well, you know what? Google scale, Amazon scale, that’s not a problem that I need to face.” Well, surprise—
Chase: Yeah, it is.
Shimel: It is.
Davis: – maybe it is. And so, architecture matters, and so I can’t help, as a software architect, coming back to that and saying, “Yep, there ya go.” [Laughter]
Shimel: Yep, [Cross talk].
Sharma: And I think we’ve seen companies who were—there’s two ways to look at it. One way is to look at it from a scale perspective. The other way is to look at it from a resilience and reliability perspective.
Davis: Right, yeah.
Sharma: Right? Which are two aspects of the same thing. It’s all architecture. And I think companies realize very early that reliability and resilience are critical—resilience to be able to make changes rapidly, right? Today, I have, I don’t have curbside pickup, tomorrow I have. Today, I had 10 people doing this, tomorrow I’m gonna have 200 people doing this.
Ashley: Right.
Sharma: Today, I had a teller sitting in front of her terminal, so the user interface didn’t need to be that cool. You know, the things could hang and the teller could, you know, make sense, make small talk while the system is catching up. Well, nobody’s accepting that on the mobile app when they’re directly dealing with the website or the mobile.
So, I think that whole shift on resilience and reliability are real, are important, and all that goes back to your point, Cornelia, to architecture. That was front and center to people, and I think [Cross talk]—
Chase: Yeah. And I think along with that, Sanjeev, is security. I think security, this really exposed—we have to do more in security. We have to do more, you know?
Ashley: We have to rethink security.
Chase: I think that’s one of the chief learnings.
Shimel: No doubt about that. Guys, I hate to do this to you, but I need to pull the plug on this. We already have people in my waiting room for our next interview.
What a great panel this is. We’re gonna have to rerun this again and go to a part two, because there’s things I wanna talk to about this as well. I spoke to a guy today who started a new, he calls it the mobile cloud for gaming, it’s called Now.gg. Because their research is that our kids, you know, these digital natives, if they don’t get a response in half a second, they’re gone, they’re onto the next thing. And so, what does that mean for your banking apps? What does it mean for we’re doing?
But we need to end this. Sanjeev, Cornelia, Kurt—man, thank you, guys, so much. Thank you, all, so much for making this an amazing panel. Thank all of our viewers out here for watching this; we hope you enjoyed it. Mitchell, I’m gonna let you take it home.
Ashley: Well, I think—what a fascinating group and wonderful things to say. I think the biggest learning for me—and I really appreciate Cornelia and Sanjeev talking about the lost opportunity we had through networking to help people come together and expand their networks. So, let’s invest in that going forward. We need to rebuild those and make that happen. So, I appreciate—I really wanna thank you for helping me understand that better.
Shimel: Absolutely. Many thanks to Tricentis for sponsoring this episode of DevOps Unbound. We’ll be back in two weeks with a fresh episode, a new panel, a new topic, but hopefully it’s just as interesting a conversation.
This is Alan Shimel for MediaOps. Until next time, be safe and be well.
[End of Audio]