devopsy.kinsta.cloud

Category: DevOps Unbound

EP 18: Does Standardization Disrupt Innovation?

Although everyone has their own definition of DevOps, it all comes down to creating better software faster, empowering development and operations teams and delivering value to customers. However, there really isn’t a standard approach to DevOps.

Should we standardize DevOps to introduce automation, avoid confusion and speed up the process? Will standardizing DevOps foster or disrupt innovation?

In this episode of DevOps Unbound, hosts Alan Shimel and Mitch Ashley are joined by Jayne Groll from the DevOps Institute, Paul Bruce from Tricentis and Sarah Baker from Airbnb to discuss the pros and cons of standardizing DevOps, the challenges of standardization and possible solutions. The video is below, followed by a transcript of the converstation.

Alan Shimel: Hey, everyone, welcome to another DevOps Unbound. If this is your first time catching a DevOps Unbound, welcome. DevOps Unbound is a biweekly video series where we talk about relevant topics in DevOps. In addition to the biweekly show, we do, it’s every four to six weeks, a live roundtable show where we invite the audience to participate and drive the discussion with online questions and comments and so forth, so we open it up to a live audience. Unfortunately, this show is not one of our live roundtables, it is prerecorded, but we have a great panel and I think you’re gonna enjoy it.

I’m gonna get into today’s topic in a second, but before we do, let me introduce you to today’s panel. First of all, I wanna introduce a business partner of mine and my friend, Jayne Groll. Jayne, why don’t you introduce yourself to the audience, if you don’t mind?

Jayne Groll: Thanks, Alan. Hi, everyone. I’m Jayne Groll, I’m CEO of The DevOps Institute, we’re a global member association and a certification body around topics related to DevOps. I’m really, really excited to be here and I’m hope everyone listening joins us at DevOps Institute and becomes a member.

Shimel: Fantastic. Welcome, Jayne. Joining Jayne, I’m happy to have Paul Bruce. Hey, Paul, welcome, and if you wouldn’t mind, you know, introducing yourself?

Paul Bruce: Yeah, hey, Alan. Good to be back. My name is Paul. In my full time job, I work with Tricentis in a number of different areas. I’m historically a performance and reliability nerd. In my off time, I organize a number of events, help to co-organize volunteer things like DevOps Days Boston, just did o11yfest this year about observability, we had that conversation, I think, with Christine Yen a couple months ago.

Shimel: Good stuff. Mm-hmm.

Bruce: And generally, I’m a people advocate. I had to pull DevOps out of my LinkedIn title because there were too many recruiters that I don’t need, because I’ve got a pretty good job. But also, because people advocate really describes what I care the most about, right? People are some of the hardest part of the DevOps people, process, and technology, really, software. So, I really focus my time on that.

Shimel: That’s great to have you back on, too, Paul—good stuff. Next, we have a first time guest on DevOps Unbound today, a new member of the band, Sarah Baker. Sarah, welcome to DevOps Unbound, if you wouldn’t mind introducing yourself?

Sarah Baker: Thank you. Wonderful. Welcome. My name is Sarah Baker, my pronouns are she and her. I’ve been working in Operations my entire career and I’ve moved into DevOps as something I always—I needed a name for it. I always wanted the people part of Ops and technology and DevOps was that. And I’ve been doing that as soon as I learned the term. And I’ve been moved and started working with Paul Bruce in the IEEE work group 2675, who is trying to create a standard for DevOps.

Shimel: Fantastic.

Baker: I work for—and today, I work as the Manager of Biztech in Airbnb.

Shimel: Thank you, and thanks for joining us. That sounds like a great place for you to do this. And then the last member of our panel today is my co-host and another business partner, a long-time friend as well, Mitch Ashley. Mitch, if you wouldn’t mind introducing yourself?

Mitch Ashley: You bet. Always good to have a good panel on with you, Alan. Mitch Ashley, I’m CEO of Accelerated Strategies, an analyst firm that’s focused on DevOps, and also on cloud native, cyber security, and digital transformation. I come to DevOps both as a practitioner and a vendor leveraging it, so very interested in hearing the perspective of folks on our topic today. So, thanks for joining us, everybody, too.

And by the way, our DevOps Unbound is sponsored by Tricentis. I think you may have mentioned that, right, Alan?

Shimel: No, I think I didn’t, Mitch, but thanks for picking it up for me. [Laughter]

Groll: [Laughter]

Ashley: I was trying to give you credit where credit may not be due.

Shimel: Yeah, no, I don’t wanna take credit when I messed up. Yeah, DevOps Unbound is sponsored by Tricentis, and many thanks to them for their sponsorship. It makes all of this happen.

So, let’s turn to the topic at hand today, guys. You know, when I first started devopsy.kinsta.cloud eight years ago, I met Jayne maybe a year, within a year of that or a year and a half of that. One of the interesting things about DevOps is, there was no manifesto. There was no hard definition of what is DevOps and what is not DevOps, you know, contrary.

And ever since then, it’s been a double-edged sword. In some ways, it’s helped DevOps grow, it’s helped DevOps stay current over the last 8 to 10 years as the world has changed, DevOps has changed along with it. On the other hand, though, we’re still dealing with the, “So, what is DevOps?” Right? “What exactly is DevOps and what exactly is not DevOps?” which is sometimes harder than what is DevOps.

And as we sit here now, August, 2021, there really still is not a quote-unquote standard for DevOps. There is no standard DevOps. It is what you want it to be, right? As I think Andrew Clay Shafer said, “The DevOps you get is the DevOps you deserve,” right? But have we outgrown that stage? You know, that seems like a terrible twos stage of DevOps to some; to others, it’s what kept it cool. Has DevOps matured? Do we need standardization? Will it be something that kills the goose that lays the golden eggs?

We’re gonna discuss that today, and I’m interested in all of your views. Jayne, as the CEO of DevOps Institute, how many people have taken certified DevOps classes and certifications with DevOps Institute? It’s gotta be tens of thousands.

Groll: Yeah, about 25,000 so far. So, it’s a pretty large community.

Shimel: Mm-hmm.

Groll: And if you don’t mind, I wanna kind of just lay the stage, because I’m really interested in the standardization initiative. So, you know, I had a privilege of seeing, going to DevOps Days in 2012 in California. So, if you can imagine, it was in an old, dusty data warehouse, right, and it was at the very, very early stages. And so, one of the reasons DevOps Institute emerged was because there was a lot of really cool things here culturally, from an automation perspective, right, doing things faster, collaboration.

And so, we started up DevOps Institute because we wanted to make sure the humans were represented. And I love that Paul and Sarah in particular both represented people. The problem with that is, at that point, it was very, very open, it was very kind of organic. You know, it was really a group of passionate people that were trying to just change the way IT operates, right, from a people, process, and technology perspective.

What happened after that, though, is that there’s so much noise about DevOps that if I were an enterprise CIO right now, my head would explode, right? Now, is there one right way to do DevOps? No. Is it only gonna get you there with automation? No. You know, is there a very, very strong human element—absolutely.

So, it’s a double-edged sword, Alan, as you said, is that we wanna be able to kind of scale down, right, some of the common language, some of the common definitions. I always say I’m married to a CPA, they have generally accepted accounting principles; maybe someday, we have generally accepted IT principles. But how do we distill the noise, right, and everyone having a different interpretation of how to do DevOps and what it is compared to, let’s get past the, “what is it?” We’ve spent, what, 10 years trying to figure out what it is, to, “how do we do this?” Right?

Bruce: Mm-hmm.

Groll: And it isn’t just one roadway that takes you there. So, I think it’s a challenge. I mean, we introduced foundational certifications around DevSecOps and SRE and DevOps Foundation and DevOps Engineering, really, to create kind of a common vocabulary more than the, “here’s the single way to introduce CI/CD.”

So, I don’t know if that helps kinda set the stage, but I think it’s a really big challenge right now that, if we standardize it, we run into the risk of what happened with ITIL where it couldn’t keep pace, right, and it became religious. I come from the ITIL space, right, so it’s like, open chapter 1, verse 3 and sing a hymn before you introduce change management to how do we then, on the other end of the spectrum, it’s so open and so undefined that it’s really complicated for legacy, I call them heritage organizations, to be able to move from this step to this step.

Bruce: Mm-hmm.

Shimel: The church of ITIL. Paul, what’s your feeling on that?

Bruce: Well, everybody’s on a journey, right? Your personal journey, your professional career, whatever, maybe you go do a walkabout on a particular topic to learn more about that. And so, we started this venture, there was an early round of, “Who wants to be involved in something like this?” in 2016. I answered a call from Bob Aiello and Lynn Carter, who were kind of forming this idea of, there’s a bunch of people who really care deeply who have practitioner practical experience in the field. And the early part of this process was just learning, kinda level setting on what we have done, right?

Jayne mentioned that it’s hard, maybe, to have accepted IT standards, even though there are standards, right, that not every organization applies the same way. I don’t think that invalidates some of the general principles that we know to be true, which is, you know, you ship broken software, you hurt people, you hurt yourself, right? But those are high level. How do we break this apart and start to make sense of the world, as you said, the heritage, Jayne, you said the heritage organizations—what is so wrong, what’s going on that we have these kind of security problems, these performance problems, this inability to be competitive in terms of acceleration and stuff.

And so, I think the purpose of a standard is not necessarily to force people to do something a certain way. I think it’s simply just kind of like, if you’ve ever had the Christmas ornaments or just ornaments box that has the insert that allows for things to be put in careful places and also work together as a unit, right, to provide a normative, not a prescriptive standard, but a normative standard to say, “Look, these processes that you already know? Well, you already followed these from, let’s say, 12207, right?” These processes around supply acquisition, knowledge management, human resources? Right, human resourcing in DevOps is a big problem, at least in the Boston area. Try to hire an SRE—there’s only about 15,000 to 20,000 in the United States, legitimately. There’s a reason for that demand.

So, how do we think about the processes we already know and start to say, how are these things different because there are some accepted principles and practices around DevOps? Now, the practices are not things that you would ever wanna dictate in a standard. This is not the same granularity level as some other standards that dictate how to encode time formats. That’s very important at that level. But the level that we were working on was how to level set across the things that, especially organizations in high compliance and high risk industries, right, risk-averse industries—how do they take the practices and processes that they know from a process standard and start to go, “I understand how to plug, now, what the DevOps principles are into the standard to encourage both of those practices in a more predictable manner?”

So, when we did all this work—it’s taken years, right, of a lot of learning collaboration between a whole host of people I don’t have even time to list out, but a lot of people from a lot of various different aspects of software have a lot of experience in those aspects. You know, it took synthesis and refactoring, and I was up nights and weekends just rewriting some of the statements, you know? And each of these, it’s a line to 12207, so there’s multiple processes, process declarations like quality management, quality assurance, verification and validation.

Like I said, these project enabling processes and then these technical processes, and each of those process sections have outcomes, activities, and tasks. Things that are likely to occur in this area, right, as well as, regardless of how you do this, right, not dictating what the DevOps looks like, it’s outcomes of these things that are auditable, that not compliance but conformance. That you can confirm at various levels to these outcomes to say, if it’s important to get this right, when you hit the piñata, what comes out should be candy, not garbage, right?

So, I’ll pause there, because I wanna give Sarah a chance to kinda provide perspective, because she’s also been working a lot with us, for all those years. [Laughter]

Baker: I think I counted 27 people on the final author list for the IEEE.

Bruce: Yeah.

Baker: And it was many, many weeks of arguing, reading together the exact language. And I agree very much with some parts of what Paul says, and I wanna step back and talk a little bit about how I see, you know, what, to me, was the big shifts. And so, the way we wrote the doc was to kinda look at the 12207 and say, “How does DevOps change it? How has it changed the game? What are the perspectives that you need to have in mind in order to get the outcomes that Paul’s talking about, there?”

And so, when we were writing it, we were writing it to that. How is this different, and to what extent do you wanna consider these areas? It’s a framework of thinking. At its largest level, it’s a framework of, for me, the value chain of delivery of software. And it’s not—and it uses many of the concepts that come out of learnings of industrial engineering, way back in the ‘40s, ‘50s, ‘60s. But it had never really come into a lot of the thinking of software, because software is different. And the more I was working and kind of digging into what are the concepts underneath DevOps, I came quite aligned with a lot of industrial engineering concepts about how things flow and how things move back and what the peaceful aspects are in that.

And so, that—actually, if you read through the standard, you’ll see a lot of those flow pieces and all of the pieces that you then say, “Oh, it’s the feedback loops, yeah, lots of feedback. Collaboration, it’s all there.” And it’s just applying a lot of those learnings into software engineering, to me. And it was stuff I was like, “Oh, it’s so wonderful to have this.”

And the other thing, one of the big learnings that I think I brought into the conversation is how—and Paul will back me up on this—how security for a lot of us changed slightly as we started walking our way through the 2675 work. The terms of risk became a very interesting concept that flowed through the document more than I would have expected.

Bruce: Mm-hmm.

Baker: And so, there was some learning within the totality of our team as we wrote this. It was quite fascinating.

Ashley: I’m really fascinated to hear from both of you that have been working on this, because it’s not something I’ve been involved in. I sorta have a mixed history with standards, because I’ve worked in areas like networking and things like that where standards are very beneficial or really fundamental, but I’ve also learned standards are kind of what you’re gonna do, not how you’re gonna do it. Because we ran a test lab that, even though it says we’re gonna do least cost routing in a network device this way, people implement it differently and they don’t always work together. So, it’s not a nirvana, but it’s also not a pariah, either.

Bruce: Mm-hmm.

Ashley: So, what I wrestle with here is that, I think some of the thinking—and Jayne can speak to this, but innovation happens so quickly now. So many things are created without standards; maybe they evolve into standards, maybe they don’t. And it doesn’t bother me that there’s lots of noise around DevOps, because we’ve all seen the cycles, the hype cycles, right? If it’s worthwhile, everyone and their brother, sister, cousin, uncle will all jump on the bandwidth and call what they’re doing fill in the blank—it’s AI, it’s DevOps, it’s whatever. That’s just part of our industry. And it’ll either fizzle—you know, value stream management, is it a real thing? We’ll see. It’ll fizzle or it’ll happen, right, if it’s a real thing. And DevOps has proven itself not only is it a real thing, but other parts of the organization now see an opportunity to play, to participate, to become part of the software creation process.

Bruce: Mm-hmm.

Ashley: And I think that’s what’s so powerful about DevOps—it’s not about a technology, it’s about how we create software. And if you were gonna start out, you know, Jayne, when you went to that meeting in 2012 when you said, “We’re gonna go create a standard for completely doing software differently because we have cloud and we have the ability to automate and we were gonna change the software architecture,” you couldn’t have known how to do that. But being the industry would’ve, you know, the antibodies would’ve crawled all over all of us and stopped it. So, I think there’s a lot of power in the innovative, organic nature of it.

And I don’t want DevOps to turn into ITIL, [Laughter] because that’s sort of taken on a life of its own, and now it’s trying to catch up and it’s done some good things, too. So, I wrestle with trying to constrain it when I think it’s real power is the innovation behind what we’re doing.

Bruce: Yeah, so, before I—

Baker: If I could take a moment—

Bruce: – yeah, go ahead.

Baker: – I’d like to take a moment and call out something. Part of the problem with ITIL is like, it’s ITIL. And, you know, a gazillion managers said, “ITIL, let us get a hammer and hit you over the head with it.” And DevOps, fortunately doesn’t have that problem. It has, you know, a little fuzzier in its definition, but the great thing about DevOps is that it has that sort of fuzzy edges. You can’t, it’s really hard to beat someone over the head with it. You can try. I’ve seen it happen, you know? Everybody’s a DevOps engineer now, but it’s not necessary.

And what people talk about is not that, but the work, you know? It’s not, “We’re conforming with ITIL,” it’s like, “We’re doing DevOps.” The doing part is the important part, not the compliance part. So, when you say, when you hear people talk about it, they talk about it differently, and that’s the good thing.

I use ITIL, I just don’t tell anybody the word.

Jayne Groll: Which is a shame. It’s such a shame because, you know, I come from the ITIL space, and in the early days, this was not what the intent was to be. It was very much like DevOps in terms of, “Let’s help IT grow up,” right, in terms of some things.

And, you know, to what you’re all saying, you’re absolutely right. A framework, whether it’s a standard, is a guideline, right? It helps you with the journey, as Paul said. And so, if I wanna kinda check, check, check—you know, I’ve worked with some of the ISO standards, right? So, in the ITSM space, there was ISO 20,000, 27,000 for security, right? And it basically, like, in the one or service management says, “You should record all your changes.” It doesn’t say how to record it, it doesn’t—it’s a really great practice, you should do this.

The problem, sometimes, with standards—and I’m sure you know this and it’s a risk—the pure intention is pure. The ability to weaponize it, right, from the auditors and from others is a real risk, right? Because we’ve seen that with other standards as well, where it becomes more about the compliance than it is about how having it being a guiding light. So, I think there’s always a risk there.

And then the other risk is, you know, think about observability, right? So, observability has been trending, what, for a year, maybe two, right? Four—okay, but really mainstreaming, like, talking about observability as kind of a mainstream practice, right? And I will share with you, Mitch, to your point, when we launched DevOps Institute in 2015, we were not welcomed with open arms, right? We were vilified for trying to put some boundaries around this.

Now, you know, move forward seven years and I think we’ve overcome a lot of those and we have some that were detractors or really great supporters. But I think they were concerned that it would become weaponized or commoditized or whatever.

So, I’m just gonna say one last thing because I think it’s important. Commoditization of knowledge, which, this is kind of in support of the standard, right? Commoditization of knowledge where we make sure it’s available to everyone, some people see that as a negative—it’s not. What would happen if the whole world, right—every IT organization, every IT professional decided to take this standard and use it as a guiding light, according to the needs of their organization? They weren’t worried about the audit, they got trained, they shared knowledge—wouldn’t that be amazing? But that’s commoditization.

So, I think the risk of good versus evil on initiatives like this is real, but I also think that distilling it to everybody, kind of like accountants have done, right, is important because IT needs to grow up.

Bruce: Yeah, well—

Ashley: Well, and if you look at the accounting standards, they don’t tell you how to do it, they tell you what to do.

Groll: Right.

Ashley: It isn’t prescriptive in the how very much, so it’s very much like an IEEE standard or whatever.

Bruce: Yep. Well, first off, I forgot to mention when we first started—I speak as an individual contributor. If it isn’t abundantly clear, [Laughter] I don’t speak for the IEEE, and I don’t speak for the other people who have spent years contributing to this meaningfully.

So, when I say certification processes inherit all the exact same things you just mentioned as problems—weaponizing, actually low ROI maybe in some cases—we see these deficiencies and these outcomes that aren’t so great. And if you’re somebody who’s just gonna sit back and say, “I’m not willing to hold myself accountable” to stand up and say, “What does this thing actually mean like a real engineer should, you know, to start to break apart the problem and see what are the components, how does this thing work and how do I want it to work? How does it work best for the people around me as well?” If you’re not willing to do that, then I would call you the nay-sayer, you know what I mean? Like, not any one in particular, but that would be a nay-saying, like, let’s avoid accountability, let’s avoid doing what we know needs to be improved?

Continuous improvement is a huge element that is all throughout the entire standard. It’s one of the things that’s identified as some of the core principles in an ISO study that was done before we started in on this standard to see what are the common themes of people, when they’re really feeling like there is this DevOps mindset, this culture, these practices—what do they mean by it?

And a number of things came out. One was a high degree of automation, right? Okay, well, that’s very tactical. No matter how you do it, right—high degree of automation. Duh, right? How about effective communication and collaboration? How about shifting things and continuous everything? The notion of, that these things are not this one time and then three years later, you get to come back. No, no, no. If it’s a good idea now, maybe don’t bite off a three-year apple. Maybe take a bite, but know that you started in, there’s a continuous improvement process.

I can say, we mentioned earlier about a holy war that might exist when you start getting into what is DevOps and what is not DevOps. Like, I started—the best way I can describe to people, because like I say, half of my day is spent on and off of Slack with people who are actually doing, working this out in a highly statistical, large distribution across the country, you know, and sometimes outside the U.S., too, just time zone supply.

But I can say, for sure, after seeing the complete garbage Dumpster fire where nobody is willing to stand up and say, “Hey, aren’t these things common and standard?” And oh, by the way, not vendors that have something to push, right? I mean, that’s, I think, Jayne, where you were talking about maybe observability is now a thing. You know, the only reason why it’s a thing for the most part is because it’s in our face now because all the vendors are hash tagging the crap out of it.

Same thing happened with DevOps five, six years ago—everybody had to get on the bandwagon. I remember CEOs way back then going, “What is this DevOps thing?” Oh, man. You know, just talk to some of your engineers.

And what I can say, right, I have developed at least an internal personal perspective on what is not DevOps. So, if I can borrow the last 30 seconds of the mic for a second, here are some things I wrote as notes as I’m going back over the standard, re-reading it, going what I violently agree with from both others and myself, we did not have a chance to see this before it went live many times, right?

So, what I got from one of the higher order sections in Section 5 was this feeling of, like, what is not DevOps. And those things are burnout. DevOps is not burnout. You know, you’ve probably been burnt out if you’ve worked in DevOps before, so where is this coming from? Why is this the case? It’s not burnout, it’s not no help. You don’t get any help, go do this thing, you know? It’s not heroics. It’s not superheroes in capes, right? Single points of failure, as I call them. It’s not cliques—which, by cliques, I mean power silos. Huh-uh, there is no such thing as The DevOps Department, people. There’s no rogue decisions. There’s traceability in decisions. There’s accountability in decisions.

Leadership thrash—not easy to, if you’re really adopting this DevOps mindset and way of thinking about how to do things better, leadership thrash doesn’t really fit in. Toil and debt, right? One of my biggest things is, you know, I seek to eradicate toil, which is not easy. [Laughter] Just reducing it a little is nice, but it’s, and it’s probably pragmatic to say so, but really eradicating toil from other people’s plates and from your own, that’s a challenge, right? That’s kind of a life goal. And if we do this stuff together, right, if we say these are not the things we want, the question is how do we move towards something that isn’t those things, right, and that is closer to where we all align and agree, like, this is a better way of delivering software and doing it faster, right?

So, anyway, I’ll pause there.

Ashley: You know, one of the things I’d love to hear from both Sarah and Paul is, there’s always sort of that, “So what?” answer—question, right? “If we did this, what is the result? What’s the impact of it?” And I mean that in a positive way, I’m not being critical.

Bruce: Yep.

Ashley: If this standard was widely adopted, how would it improve? How would it—what things are we wanting to improve and change from where we are today? Can you describe that in any way to kinda give myself and others maybe an idea about that?

Bruce: I just did. People already implemented it.

Baker: Well, at the base, there’s the vocabulary, yeah.

Bruce: Sorry, people are—

Baker: At the base, that is part of it. Sorry. Paul and I always get [Cross talk].

Shimel: Sarah, go ahead. Paul spoke—you go.

Bruce: Yeah, yeah.

Baker: Vocabulary is that first piece, making sure we’re all talking about the same framework. And so that, when we’re exchanging ideas, how did you do it, how did you do it—you know, not any one way to do it is gonna it, but knowing what your options are and talking about it with others and trying something out and figuring out if it works or not in your environment is one way to get everybody on the same playing field. But if you’re not talking about the same thing, you can’t do that.

So, it’s having the ability for a large corpus of options to apply to your situation. To me, that’s one of the biggest things that it provides, and it also gives you ways of interacting with the rest of the business that is not IT, and kind of, “Oh, this isn’t—it’s not just IT. You’re a stakeholder, you’re a customer.” And it includes all of those pieces for the collaboration and really helps support it.

So, to me, that’s what the standard brings. It widens the organization’s perspective a lot, and allows that cooperation to seep into the organization across all of these different pieces. Because to me, that’s how it started as DevOps, DevSecOps, DevOps Observation Ops. I mean, it grows because that’s part of its DNA is to include.

Bruce: Yeah.

Baker: And so, it’s very natural that it has done that and it includes other parts of the organization into that basic thought processes. I totally could see DevOps marketing security—I mean, you see all that. That’s what’s, to me, the real advantage of thinking this way is that it includes the entire org.

Groll: Well, and first of all, utmost respect to Paul, Sarah, and the other 25 people that are working on this, because this is a gargantuan task, right? Trying to encapsulate something that creates a common framework, that has the purest intention and also, you know, if you think about the heart of why DevOps emerged in the first place, it emerged because we were not collaborating together, right?

So, you know, in the early days of IT—I always say I’ve been in IT longer than I really want to admit to—you know, there were 25 people that sat on the same floor, in the same office building, and we all knew each other and we went to happy hour. And then, you know, the tree grew and there were lots of branches and other things. So, you know, giving a central source of truth that everybody can kind of internalize, process, speak the same language—you know, IT, we have so many different languages, we’re like countries that are butt up right next to each other.

And I said, I have so much respect, because this is a gargantuan task. And Sarah, I agree with what you said, the purpose is really to take this need for collaboration and optimizing automation, right, so making automation essentially members of your team, right, and then taking that out into the business because every business is a technology organization, it’s—I’m sure you’re aware of the risks. You know, we’re not telling you anything that you’re not aware of, but as I said, I just have so much respect. Because it’s not gonna be an easy road, right? There are so many different opinions from vendors and from everyone who’s involved in this space that are going to have an opinion or whatever. And I think, you know, that’s something that everybody’s gonna have to deal with.

Again, the only thing I would suggest is build it in a way that it’s nimble—dare I use the word agile, so that—

Bruce: So, this is already done, right?

Groll: Yeah.

Bruce: Like, this is not a draft, this is not an initiative. This has already been done for the past four years and is now an official standard. So, the good news is, to your point, I think, about suggestions about how to do this, we are working on the second version of this draft already.

As soon as it was out, even before it was out, we knew there were things that we just couldn’t—we couldn’t spend the time properly addressing. Like I said before, there was a lot of refactoring, but there was a lot of careful avoidance of not already repeating what was already in a good security standard over here, or in a good risk management thing, or one of the folks that worked with us, Tafline Ramos, works at the ISO, right? And what is it, 29119, right, verification and validation, the processes—that was already stuff that’s already been, and great thinking and not slow, you know? Can fully be put into a modern, unicorn sneeze rainbows and fart sprinkles type approach as well as the large Fortune 100.

So, just to say, this is already—we’ve already put a ton of, it’s already out there, and the easiest way is, I mean, some people might be wondering, “Oh, yeah, but it’s a standard, so now I gotta pay for it?” Some standards, you do; some standards, you don’t. This one, IEEE, they cover their costs with the standards. The easiest way, if you wanted to get a free copy, become part of the new, the next version of this, right? See what we’ve done, poke holes in it, have collaboration, let’s work on it together, because that’s the point of this is to bring a framework of how we are working together in a more effective manner.

Jayne, I’m really sorry I cut you off.

Groll: No, no, you didn’t cut me off. Just a cautionary tale. One of the reasons ITIL kind of lost a lot of its momentum—and I’ll be very frank, I owned an ITIL training company for 15 years and I was a chief evangelist, but it took 8 years between version 3 and version 4. And so, it wasn’t built, it was a monolith.

Bruce: Mm-hmm.

Groll: And I think that was—you know, in the meantime, lots of other things happened, and I think there’s a cautionary tale. I’m not suggesting you’re looking at eight years, please don’t misunderstand, but this market is so—

Bruce: No, it’s probably about three, right?

Groll: Yeah, this market’s pretty fast. Sorry.

Bruce: Our target is about three, between where we officially started and now it’s out and the next revision, at most, would be another three, right, that’s our target. It could be earlier or later than that.

Think about, we’re all citizens of the United States here, right? Or anybody in Canada?

Shimel: Is that a HIPAA violation to answer?

Bruce: Yeah, right.

Shimel: Just kidding, just kidding, just kidding.

Groll: [Laughter]

Ashley: I’m not showing you my vaccine card.

Bruce: I ask because, I ask because—

Shimel: I didn’t wanna go there, but go ahead.

Bruce: – we all know what the U.S. Constitution is and where it came from.

Shimel: Mm-hmm.

Bruce: Do we think that it was perfect and never has to be revised? Absolutely not. That’s what the Amendments are for. And arguably, I people wanna go back to the original version, there are some serious problems with that original version. Same thing with ITIL, same thing—remember with, I think you might have heard about The Agile Manifesto. You know, that was a drunken party with a bunch of white dudes, okay?

Groll: I know. [Laughter]

Bruce: So, yeah, there were some good ideas that were encoded in there, but do they not have to be revised and worked out? Even Dave Thomas says agile is dead, long live agile. Like, what people have taken it to is not the point.

So, the goal of this is to really get a full perspective. To what Sarah was saying, it’s not just—I’ve talked with a lot of the DevOps Days, various different people from DevOps Days, from a number of different, like, The DevOps Enterprise Summit and stuff, and there are some people out there that are like, “No, DevOps is just about developers and operations and it’s about just these things, and if you start getting biz in there—no, no.” And it’s like, what a narrow perspective, because you know what, you had a good idea and then you screwed it up by not including other people, by not realizing there are meta learnings that come out of what you locally did—maybe, arguably, you did well—that apply broadly. And when they do apply broadly, they bring us back to the same premise, which is, we’re all in this together, right? We all win when we all win. As Tim O’Reilly might say, provide more value than you consume, right?

So, that’s where we intend on taking this. It’s not to turn into a bunch of agile consultants again or ITIL consultants, right?

Ashley: [Laughter]

Groll: Framework growers, that’s what we call it.

Shimel: Yep. Guys, I gotta pull the plug on ya, I’m sorry, guys, and ladies. We are out of time. I think a great way to end this, though, is—Sarah, Paul, where can people get more information on the standard from the IEEE?

Bruce: Sure. Google it, Google IEEE 2675, you’ll get to that page if you wanna buy a copy.

Shimel: 2675.

Bruce: I think without being a member, it’s like 99 bucks—so, look, you know, if you wanna buy it for yourself—

Shimel: It’s not breaking the bank.

Bruce: You probably also have, in your organization already somebody’s a member, your Risk and Compliance are probably already members to IEEE.

Baker: Yeah.

Bruce: And again, like I said, the best way to get a free copy is to come work with us on what does the next standard mean. It’s not a huge effort and commitment like Sarah and I have put in. [Laughter]

Baker: [Laughter]

Bruce: You can dip your toes in, see what makes sense, what works for you, and we love to—we include people in that effort.

Shimel: Great. Well, Paul, Sarah, Jayne—thanks for joining us on our panel today. Mitchell, as usual, great job. This is another edition of DevOps Unbound in the books, check it out. Go check out the IEEE standard. We’ll be back in two weeks with a fresh show and a fresh topic. Until then, this is Alan Shimel, be strong, be well—Tech Strong. Take care, everyone. Bye bye.

[End of Audio]

September 19, 2021
EP 17: Lies, Damn Lies and Metrics

In this episode of DevOps Unbound, hosts Alan Shimel and Mitch Ashley are joined by Helen Beal of the DevOps Institute, Viktoria Praschl of Tricentis, Frank Ohlhorst, principal analyst with Accelerated Strategies Group and Alex Hidalgo of Nobl9 to discuss how to use DevOps metrics effectively to improve delivery performance and help teams become high-performing with a culture of continuous improvement. The video is below and a transcript of the conversation follows.

Alan Shimel: Hey, everyone. This is Alan Shimel and you’re watching DevOps Unbound. DevOps Unbound is sponsored by Tricentis and we’re happy always for their sponsorship and helping up put this show together. DevOps Unbound is an every-other-week session with experts from around the world talking about relevant topics to DevOps. In addition to that, about once a month or every six weeks we will do a live roundtable which is open to you, our audience, to come on, come in, ask questions live, and really kind of steer the conversation. We do have another live roundtable coming up. If you check the devopsy.kinsta.cloud website you can register for that. Today’s conversation, though, is prerecorded and unfortunately that means I get to ask all the questions, not you.

But let me introduce you to what I think is an amazing panel we have lined up for us today. Well, actually, before I do that let me tell you what we’re going to talk about today on DevOps Unbound. So, it’s the issue of metrics and kind of the ghost in the machine, the dark side of metrics – lies, damn lies, metrics – whatever you want to call it. How do we measure success or failure? How do we measure progress in things, in areas like DevOps or digital transformation? What does success really look like? Our panel is going to talk about that today. Let me introduce you to them.

First, I’m happy to introduce Frank Ohlhorst. Frank, if you wouldn’t mind saying a few words about yourself?

Frank Ohlhorst: Sure. And I’ll be quick as not to bore people.

Shimel: Okay.

Ohlhorst: I’m Frank Ohlhorst. I’m a journalist, author, and consultant. And I currently work over at DevOps as a contributor, at Security Boulevard as a contributor, and I am also a contributing analyst. And that’s pretty much my story, other than I’ve got 30-plus years – I hate to date myself – 30-plus years in the IT field and I’ve worked both for the federal government, I’ve been a developer, and I’ve put a lot of effort into cybersecurity.

Shimel: Excellent. Thanks, Frank. Next, let me introduce you to one of my friends from the other side of the pond. I’ve known her now for eight years or more. The one and only Helen Beal. Helen, welcome.

Helen Beal: It’s a pleasure to be here as always and spending time with you and the others. So, thank you very much for the opportunity. I’m Helen Beal. I’m Chief Ambassador at DevOps Institute, and therefore one of the co-creators of ADOC, which is the Assessment of DevOps Capability, which I may mention once or twice during today’s session. I’m also chair of the Value Stream Management Consortium, and prior to all of that I was coaching in DevOps in a number of different companies across Europe and the Middle East for several years.

Shimel: Thank you and welcome, Helen. Next up, let me introduce you to Viktoria Praschl. Viktoria, if I mispronounced your last name I apologize. I just realized when I looked at it that I thought I remembered it, but maybe not. Go ahead.

Viktoria Praschl: It was totally correctly pronounced, so you did a good job.

Shimel: Thank you.

Praschl: Hey, everybody. Happy to be here today. I’m Viktoria Praschl. I’m with Tricentis, a software and quality assurance company based out of Austria and Vienna. And I’m responsible for our UP and sales and transformation team where we work a lot with customers on how they transform their organization when it comes to DevOps and quality assurance.

Shimel: Excellent. Thank you, Viktoria. Our last panel member today also is somewhat of a regular; he’s been with us a number of times. And that’s author, expert Alex Hidalgo. Alex, why don’t you give them a little background and welcome?

Alex Hidalgo: Thanks so much, Alan. Hi, my name is Alex Hidalgo. I am the Director of Site Reliability Engineering at Nobl9, which is a startup focused entirely on providing the best possible service level objectives, the best possible SLO platform. I’m also an author of Implementing Service Level Objectives, and I’ve been working in this space for – in some way or another the last two decades or so. My current area of research is really into how to bring reliability lessons from other industries into the tech world and what we can learn from others.

Shimel: Excellent. Thank you, Alex. And then, last but not least, my cohost for DevOps Unbound, my partner in business, Mitch Ashley. Mitchell?

Mitch Ashley: As always, good to be with you, Alan. And I love our panel. You all are fantastic; I’m excited to hear what you say. As Alan mentioned, business partner and cohost. I run Accelerated Strategies Group, an analyst firm that’s focused on DevOps, cloud native, cybersecurity. And also, I serve as CTO with MediaOps, the company that puts this event on. And both a practitioner and product creator in software, software as a service, cybersecurity for a number of years. I’m not going to name or number how many they are, so you’ll have to guess.

Shimel: [Laughs] Okay. I’m not guessing. But anyway. All right. Let’s get into this week’s DevOps Unbound. So, I remember a VC mentor of mine getting up in front of – we had launched a new company, he stood up and said something to the effect that in this brave new world we all live in everything can be measured and we measure everything, and it is based upon those measurements we determine who wins and who loses and by how much.

So, that was 20 years ago, maybe more, a little more than 20 years ago: 22 years ago. I wonder, is it still true? Or have numbers, like everything else, with fake news and all of that, kind of lost their meaning and lost their importance in today’s organizations? Helen, can you –?

Beal: I’m not sure it’s ever been true. From my experience with metrics I think what I’ve always observed is just people fall into one of two camps. So, either they’re not measuring enough, or they don’t have enough metrics, or they have far too many. I think there’s a cognitive load limit on how many metrics we can handle and we probably need about a handful.

I think it’s true that we need to have telemetry everywhere, which is of course a very popular DevOps adage. I think we need to have the facility to be able to get to the metrics that we want, but we have to be able to choose them and be able to think of them as an adaptable framework. So, for example, deployment frequency, of course a very popular metric in DevOps, is very useful if you’re moving from deploying every three months to once a week or more. But once you’re deploying several times a day it’s a much less interesting metric.

The adage I always use is if you can’t measure it you can’t improve it, which is a Peter Drucker phrase. And I do have a problem with that as well because I think you can improve things without measuring them. But the thing is that you’ll never know. And to me that’s the most important thing about measurement: It’s about being conscious about what you’re doing.

Shimel: Guys, any thoughts on that?

Ohlhorst: I think it also comes down to an issue of context. People can measure whatever they want out there and they often do, and they spend so much time measuring that they don’t take into account the context of the measurement. You can measure certain processes in the DevOps process and they really don’t matter when it comes down to it when you’re looking at trying to achieve a particular goal.

Shimel: They don’t. You know what, Alex, we’re going to let you go, and then, Viktoria, you follow up.

Hidalgo: Yeah, I think one of the big problems that I often see is that people end up trying to measure themselves against others. They hear “Oh, at my last company” – or “I heard at Google or whatever they’re able to deploy X number of times a day and let’s aim for that.” But why? Do you need to deploy that many times a day? Is it actually applicable to you or your application or your organization or your customers? Are you actually gaining anything from trying to do things the way you know others are? Every situation is unique. Every company is unique. Every organization is unique. Every application is unique. And that means you really need to take a step back and think about “What is right for us? What do we need to improve upon?”

Praschl: Yeah, that’s what I really wanted to say, basically similar, that you take a step back and you really need to look at what’s your baseline. Where are you and what do you want to achieve? What are the outcomes you’re looking for? And I think it’s very important: The metrics will change over time depending – as you mature your processes and as you mature in your organization you will have to constantly change and adjust your metrics which you are measuring. So, I think the key thing is to really initially take a look: “Where am I? And what are the most important initial metrics where I start with?” And then, constantly, basically changing and improving toward further goals.

Shimel: Great. I think – I’m reminded of the adage “Just because you can doesn’t mean you should.” And I think that’s where we have – and not just in DevOps, frankly, in IT in general. That’s where we run into problems. Because it’s digital almost by definition it’s very easy to grab metrics on all of these digital things, on all these. But just because you can doesn’t mean you should. And just because you can doesn’t mean it’s necessarily indicative. I think, as Helen mentioned, when you’re going to a release a quarter to a release a week, ooh, that’s big news. That’s good metrics. But – and to Alex’s point, not everyone needs to be Google or Amazon and release hundreds of times a day or whatever it is – dozens of times a day. And I think that’s where we get ourselves kind of wrapped around the metrics axle.

Ashley: I think it goes back to the question of why. If you don’t understand why you’re measuring something, then why do it? Because everybody is not going to be on the same page about how you want to utilize that information. I think about the VC quote that – I remember I was there for that too, Alan. And in a financial world winners and losers, how much return on investment, that’s pretty clear. But I also think there’s a lot of fear around numbers because people are afraid that it’s going to be used against them or it’s not going to be understood, what they really mean. In a continuous improvement context it’s really to – that’s an aid to help you get better and know if you’re getting better, and you have to decide if it’s the right things, which you’ve got to put that context, as Frank mentioned.

And we have to be on the same page about what it is we really – what does it mean to put a release into production? Is that really – is that a canary release? Is it a real – across every data center that we are in the cloud? What does it mean to do a release? So, you have to get to that specificity. We’re all in agreement that’s – and that’s important to us because we either want to measure ourselves by it or improve at it.

Beal: I think it’s been implied by a lot of the speakers already, but I just wanted to really want to say explicitly as well that I don’t think metrics shouldn’t be inflicted on people. They should belong to the team. The teams need to generate them themselves. You can set things like KPIs at a higher level and have them connect, but things like OKR should be really generated and owned by the team themselves. And that really speaks to some of the cultural elements at DevOps, which are much harder but not impossible to measure, but things like distributing of autonomy – sorry, distributing authority and having autonomy at the team level to choose and define and improve the work, and that move from leadership from being managers and leadership becoming coaches that are moving impediments out of the way.

I think, Alex, you were inferring this as well, the importance of the team-level metric.

Hidalgo: Yeah. One of the things I’d like to kind of bounce off there is I explicitly try to always use the word “measure” because “measure” means something different then “collect.” I think Helen mentioned earlier it’s good to have telemetry everywhere, and so you have numbers everywhere, you have metrics everywhere, you have time series everywhere. And you run into the multiple comparison problem where you don’t really know what you’re looking at anymore. You run into spurious correlations. Have you ever seen that website where it can track people drowning in pools correlates almost perfectly with the number of Nic Cage movies released each year and things like that? And that’s where you end up when you have too many numbers.

And that’s why I always like to say you measure, because “measure” means you’re taking the important signals out, and you measure flour to bake a cake; you don’t just collect flour. Otherwise, you’re going to not end up with a cake. And it has to be an intentional act, and this kind of – like Helen just said, you can’t inflict metrics on people. You need to let them collect it correctly and measure it correctly.

Ohlhorst: If we look at DevOps from the pipeline’s perspective it starts to become clear maybe where we should be measuring things or why we should be measuring things. And one of the first things that a lot of organizations run into when they implement a DevOps strategy is they run into bottlenecks because they did not consider certain things. So, perhaps the first place to start with measurement is to try to identify bottlenecks and then go from there. This way you’re improving your process while you’re getting some value out of the metrics, and you’re also establishing what type of metrics you need.

Beal: I’m glad Frank brought that up because I really think that – for me, there are two classes of metrics, though, that are the most important and underpin everything. And one thing I will say about the DevOps pipeline is in my opinion it doesn’t start early enough. There’s that fuzzy front end and I think we need to expand it right back to where the idea happens. But my duality of metrics is basically if we measure flow and we measure realization of value we really have a concept of what the customer experience looks like. So, we know how fast we’re able to deliver the things that they need and we’re starting to understand whether we are actually delivering the things that they wanted.

Hidalgo: And I just wanted to add: Frank’s point about bottlenecks is great, but we can get meta about it. We want to make sure we’re very carefully selecting what we’re measuring. And identifying bottlenecks might be a great start but we need to carefully select which bottlenecks we even want to address in the first place. Just because we can see that something slows down somewhere it doesn’t mean that that’s necessarily an area of focus.

Beal: We could be waiting six weeks for a security check and that’s really hard to get that out of the way, but we could maybe – and Viktoria may be having an example around, I don’t know, test automation or something. Viktoria?

Praschl: Yeah, I mean, also test automation and _____, you have tons of metrics you can actually measure just from a curious perspective, and if you don’t measure the right things there as well it will seem as if it’s at a standstill. But I think it’s also very important to – I liked the bottlenecks you just mentioned, Frank, because one is the team level, but then you also want to see how does it look from an end-to-end process? How does my overall customer journey look like? And I see quite often companies, actually, they start and they identify the right metrics on the team level, they get the teams there, but then they are failing on the overall picture to bring everything together. A lot of people struggle with that. That’s what I see.

Ohlhorst: I think the mistake that a lot of the teams make nowadays is they either look at this and they try to relate either DevOps to a waterfall ideology where there’s a beginning and an end and you can measure certain points or they look at DevOps as this rigid structure on how you do things, where the whole idea behind DevOps is it’s flexible and it can be easily modified to meet specific needs. So, I think people run not only into an issue of where “Okay, where do I establish metrics? Why am I doing metrics?” but also how you interpret those metrics. I mean, like we were talking about before, release frequency and things like that are great for what we use as a term is “vanity metrics.” It’s “Hey, we accomplished something.” But when it really comes down to it, what did we accomplish and how did that improve the process? And I think people are losing sight of that.

Beal: It’s been said already, I think, here but I think that focus on outcomes is really, really key. And I think there’s a couple of other things I wanted to bring up. One is that we talk a lot about being data-driven in DevOps and I’m starting to think actually about being more insights-driven. And the other conversation I’ve had recently is about doing less hypothesis-driven development and more impact-driven development, which I think is another really interesting iteration on our ways of working. And it reminds me of when Alex was talking about this idea of having telemetry everywhere, which is wonderful. And so, the idea then is that you can get any metrics you want. And you don’t want all of them but at least you have the ability to choose them. But as Alex pointed out, there is this downside to this, that we are just overwhelmed with data. And I think that’s something that AI can help us with there as well. I mean, it’s just a fact: There’s far too much data in the world for us to cope with. We need somebody or something probably that we’ve created, a machine to help us with it. So, Alex, I wonder what your thoughts were on AI in the context of measuring our journeys.

Hidalgo: You’ve asked a very interesting person about this because I’m actually fairly dubious that a lot of these models will actually help. You need a lot of data, a lot of data, to train a model. And most people actually don’t have that much. We may be talking about “Oh, we have too much data” and “We’re swimming in data” and this and that. Very few people have enough data to properly train models. Can this work in some contexts? I think so. And I think a lot of smart people are working on the problem. But I do think there’s a little bit of an overblown focus on it today because I think it’s a lot of trying to fit a square peg in a round hole. We need to – like everything else, as we’ve been talking about, you need to be careful and you need to think about what outcomes you’re really looking for and then decide “Is AI, is machine learning the right thing here?” Because it – just because it’s a buzzword, just because lots of people want to do it doesn’t mean it’s always going to work. You need to make sure that you’re picking the right tool for the right job.

Ohlhorst: Yeah, and the same can be said with automation when it comes down to it. If you don’t know the possibilities or the probabilities around a process how can you automate it unless you have the data to provide you with that insight?

Hidalgo: Which, I should say, those words just remind me of what we can do with data is statistics. And actually, when you see me flinch a bit about “Oh, should we use ML?” or “Should we use AI?” I think a lot of the places where people are doing ML they’re actually just using stats. And that’s great because statistics can actually compensate for not having enough data. That’s kind of what it does best in some sense. So, I am a huge fan of using models. I’m a huge fan of using math. Just, again, let’s make sure we’re using the right math. Do we need to build an ML model if a simple regression tells us everything we need to know?

Ashley: Yeah, I do agree with you about the AI, machine learning of it. It takes a ton of data, if that’s a measurement, in the data world. I want to go to something that you said, Helen. I really like the word “outcome.” What are the outcomes that – have we achieved an outcome when we’re developing software or creating an experience or something out of the process? Because ultimately, that’s what matters. We can improve the things within it. We can automate the processes within how we create software. But it’s all for a purpose. It’s all for – if we’re not doing a better job of delivering on an outcome, maybe those things we’re working on internally are sort of navel-gazing, if you will, looking internally but it isn’t really that valuable. Let’s work on the things that we can connect to an outcome. That’s what I would propose.

Beal: It’s funny. The other day – it was at DevOps Enterprise Summit and she went up to speak because she used the phrase “time to market,” and I kind of in the Slack chat went “Oh, time to value,” because obviously there’s no point in getting something out the door if it’s useless, whereas with “time to value” at least we’re recognizing that it’s been – giving something useful to something. And then, John Smart corrected me with “time to learning.” And I thought, “Oh, no, actually, it has moved.” We’ve moved from time to market, time to value, and now we’re in time to learning, where we’re recognizing that actually only – something only becomes useful when we’ve seen it have value and we’ve learned from it and decided to do something else.

But yeah, it’s the same as this kind of impact-driven development as well, this idea that we really focus on having an idea about what we’re trying to do, and then, importantly, measuring that it did what we thought it was going to do. And we’ve been traditionally really bad at doing this in the technology world because we’ve been really project-driven. And when we’re project-driven we have huge amounts of requirements which are (a) really difficult to track, the impacts of, because we deliver them all at once and we weren’t doing things like canary testing which does allow us to look at a much more granular level. And the other problem with being project-driven is we’re always under so much pressure to deliver. Our testing always gets squished because our development overruns and our launch date never moves. So, by the time things go live we’re so exhausted, and then we disband the team and put them into other places anyway, so no one ever goes and has a look and says, “Did that really deliver the four million quid that we – or $4 million that we said it would?”

I wonder what other people think about how metrics change as we move from project to product.

Shimel: So, I’ll quickly jump in here on that, and that is – and it’s, Helen, something you mentioned, Alex mentioned. I think what we’re guilty of – and that’s part – in the abstract we talked about the dark side, and it’s not that “Mitchell, I am your father,” but there’s a dark side of these metrics which are that in the absence of facts or in the absence of knowledge we try to fashion metrics to compensate. So, with something like DevOps where there’s a lot of cultural aspects that come into play, there’s a lot of intangibles that come into play, and they’re almost by definition hard then to measure because they’re intangible; they’re not tangible. And so, we compensate by coming up with what some may consider frankly nonsense metrics that make us feel better or make us feel like somehow we are measuring the immeasurable. And I think that is an important part of this whole thing. Measure what you can, but are there things that are just immeasurable? Anyone?

Beal: Yeah, I – sorry, me again for a second. I don’t believe so. As a coach and a consultant for a long time I was discovering clients that felt that their biggest problem with deploying DevOps or implementing DevOps was the cultural side. So, that seems like something that you can’t measure. And it’s true that you can’t really get data out of Jenkins to tell you what’s happening in your culture but there are other ways to do measurement. And I think it was Viktoria – sorry if it wasn’t Viktoria, but somebody said the important of baselining your current state, and that is absolutely essential. And I think people don’t always do that. They kind of dive in to try and get metrics without really kind of going “Okay, this is where we’re starting from figuring it out.”

So, from the human aspect – I mean, talking about ADOC, which I mentioned earlier, the Assessment of DevOps Capability, we have 5 different dimensions in that and each dimension has 12 subtopics, so we have 60 topics in total. And I’m starting to sound like we’ve got way too many metrics, but there’s a lot of subtlety in the way that it’s baselined. And the first dimension is human aspects. And within that we’re really getting a feel across an organization, team by team, where they feel their level of capability is around things like psychological safety, around transformational leadership, around how much autonomy and mastery and purpose they have in their daily role.

So, you’d be surprised, I think, about how possible it is to measure cultural capability without necessarily having a knowledge of how many – or relating it to how many artifacts you’ve got in your artifact repository. There are other ways of doing this. And sorry, I interrupted you, Alex.

Hidalgo: No. I think those are all great points, but we do have to, as with everything else, approach it with a little bit of caution because especially if these are self-reported, if these are numbers that you’re looking at like “Are you happy on your team?” “Do you think your organization is headed in the right direction?” – those normal kind of questions and then asking people yes or no, or do you put it on a sliding bar from one to five? And studies show most people are going to just take a one or five anyway. And that’s not saying this is bad. It’s just something else that’s another caveat. It’s another “Let’s make sure we’re looking at this correctly.” “Let’s understand that some percentage of our employees are just going to click through it because they don’t want to fill out this form.” If it is this self-reporting thing, you know?

Beal: It’s such an important point you make, Alex. And it’s something that we deal with when we engage with organizations. There’s such a lot to talk about, because what you’ve actually identified there isn’t a problem with the tool; it’s a problem with the culture. And if people are not feeling that they are part of the change that the organization is trying to make and they’re not empowered to have their say and their voice and be involved in it then they will skew their answers.

So, we absolutely have to help people understand how important their answers are, what impact they’re going to have on their future and their ability to grow themselves in their organization and make them feel part of the change. But yeah, humans can be a tricky business. But back to your point about data volumes, again, in this kind of assessment environment it is quite important to have enough data to weed out variations from people that maybe don’t care or are skewing for other reasons.

Hidalgo: Which, again, you can do some basis statistics to ensure that you are dropping your outliers. You – to be able to get very useful data out of this stuff, again, you’ve just got to make sure you’re looking at it in the right way.

Shimel: Viktoria?

Praschl: Yeah, I think quite often we just force metrics on teams, so that’s why I like it a lot, what you said, Helen, that you need to also understand the culture. So, where is the team from a metrics perspective? Do they understand why we are measuring? Or are you already in a stage where they say, “Okay, I understand why we are measuring certain things,” but now we need to constantly figure out which metrics we are measuring?

So, I think it goes back to the team and explaining the “why” and then really helping them or work with them together, once you have the outcome, what do we do with that? How can we improve? What do we do with the outcome of the metric? What do we do with the numbers? So, I like it a lot, what you just mentioned on assessing the culture.

Beal: Yeah, and that’s a great point. We baseline but we baseline for a reason, because we – I talk about the improvement culture a lot, so that four-step process where we start by establishing the long-term vision direction, then we establish current state, then we establish our next target state, and then we PDCA or experiment between those two states. So, kind of the finishing part of that first cycle is defining the hypotheses and executing them, and then we start again. And we’re not baselining now; we’re comparing it against our baseline at a time period of three, six months and what’s happened with our experiments in between.

Praschl: Yeah, because I think most companies, they just don’t know. They don’t know where they stand. They don’t know where the team stands. They don’t know where the culture stands. So, that’s the starting point to get everything rolling.

Beal: And it’s a big investment, Viktoria, for people to go down this path not to know how far down the path they’ve got.

Praschl: I think for that you need the executive buy-in as well. So, it’s like making sure that you have from the top also the support that you do that. And then, as you go into this constant learning process, I think it becomes very important there.

Hidalgo: And trying to look at how far down the path you’ve gone, that – it’s going to, again, be different for everyone. Is it more important how far or how far you have to go? And is there even an end to the path? Or is this a process that’s ongoing forever? And just another example of having to be thoughtful about what’s being measured.

Beal: I don’t believe there’s an end to the path, which is why I use words like “capability” over “maturity,” because “maturity” indicates that there is some kind of a horizon that you can reach. But of course, no matter how many more capabilities you’re building yourself we live in a market that’s forever inventing new ways of doing things and new tools to support the things that we’re doing. So, it’s an ever-moving target that we’re trying to respond to.

Ashley: I think there’s a whole nother side of metrics I don’t know if we’ve touched on yet, and that is the metric itself can be valuable and inform us about some things, but I also think the analysis of what’s – why? What are the potential causes? In a quality process you do a Pareto diagram to kind of analyze what you think are potential contributing factors and what maybe is influencing why it’s happening in a certain way to try to get to – if you’re – for example, if you’re doing a process improvement what parts of this – a lot of things go into whatever created that metric; how do we know what we might work on first or doing some experimentation with?

But the ultimate thing is there’s a lot of learning that comes out of that process. It really kind of helps you understand “Wow, I had no idea that’s actually how it was working.” And now that we know that with a little more fidelity we can make some adjustments.

Beal: That’s powerful. I wanted to respond to Alex, actually, and then I forgot – it went out of my mind. But this idea that the team needs the right metrics – and Viktoria and I have both said those metrics should not be inflicted on the team, but the other thing is most organizations are made up of multiple teams, and different teams, like Alex said, have different capability patterns. But it’s very interesting to try and understand, for example, why one team is really good at something like diversity and inclusion or really good at AI ops or something, and not saying that the other teams are bad but trying to learn from those success stories. And I think it’s probably the Phoenix Project that put it, it was making local discoveries and global improvements.

And I think that kind of mindset of looking for the strengths and spreading the strengths across a business is really important when we’re doing things like measuring people on their journey because the – we talked about culturally that people are kind of either scared of this stuff because they’re expecting to get beaten up about it or they’re cynical and they’re not giving true answers. But there’s another side of this, that people are also quite afraid of being assessed or even assessing themselves because somebody’s going to go “Well, you’re shocking at deployment frequency. What are you going to do about it?”

And it’s like we have to change that whole mindset in an organization that they are a dynamic learning organization. It’s all about continuous learning and not about beating people up over the head or removing their annual bonuses or the like, which are things that I’ve seen in various places.

Ohlhorst: Which gets us back to that key point of the “Why are we measuring?” Making sure people on the team know why we are measuring things, not only from a cultural perspective, but also from a value perspective so that they know that by them giving their answers based upon queries that there is value to that, and that is the “why” behind it. And ultimately, the goal is to improve processes or accelerate SDLC or whatever it is, but there is a goal there that is not just random data being thrown out asking “What flavor of gum do you like?”

Praschl: Yeah, I think it’s also –

[Crosstalk]

Praschl: It’s important to get them to understand the “why.” It’s usually a long and not as easy of a process. But once they have this and understand it you have the buy-in and the outcome will be way more worth it if you do that way.

Hidalgo: A thing that I’ve found that can help with this kind of stuff is people underestimate how powerful interviews can be. Instead of just asking people to self-report or just collecting random numbers out of your computer system sit down with people and talk to them. Not for five minutes. Schedule an hour. Sit down, find out what is this person thinking? Make sure you’re talking to the right people, not just in terms of an organizational structure, because you do want to make sure you’re talking to people at all levels of the company and all roles. But also make sure that the people you’re interviewing are comfortable with it in the first place. Not everyone has the personality to want to sit down for an hour.

So, another example: Select carefully. Measure carefully. But I’ve found so much value in just – instead of trying to do this in some – a method that is based around metrics. We’re talking about metrics. Sometimes you don’t need metrics. Sometimes you just need the human interaction and you can walk away with that if you really want to then throw that in that kind of scale and attach a number to it maybe. But you can learn a lot just by talking to people.

Beal: I think the best assessments are always a combination of quantitative and qualitative data. And I think we’re seeing this increasingly in the data world as well. About 80 percent plus of most businesses’ data is unstructured, which by its very nature is that kind of textural response data, whether it’s on social media or in your logs in Splunk. We’re seeing more and more people understanding that it’s not just about what’s in an RTMS. So, I absolutely agree with you, Alex. Yeah.

I think there’s a big advantage, though, which I’m going to point out, around ADOC. So, some organizations are very large, so when you’re spending an hour with people and there’s 8000 people it’s quite impossible to speak with 8000 people. So, actually having an ability to empower everybody to be involved in the change by giving them a platform on which to share their opinions is quite useful. And of course, you can put open text fields in there as well and get those snippets and insights that you would miss if you were just getting a one to five or yes or nos.

I did want to maybe slightly disagree with one thing you said, though, which is about the people that don’t want to talk. It always kind of worries me – and this is the big thing about, I think, diversity and inclusion and things like Heloxi that they are trying to give equal voices to people, is I don’t want us to fall into that trap where we think “Oh, that developer is really quiet and introverted; we won’t ask them what they think because they don’t really like talking.” It’s like they’ve probably got just as much right to their opinion as everyone else and we shouldn’t avoid hearing it because we just think maybe they don’t like chatting as much as other people.

Hidalgo: I totally agree. But from my point of view you’ve just got to make sure that whoever is comfortable with the process – people don’t want to feel singled out. Just like everything else, depending on the person, depending on the culture, depending on the team, they might feel like “Why am I being picked? Is this a bad thing?” So, it’s just looking thoughtful. That’s all I meant. Definitely don’t want to exclude anyone. Definitely want to make sure we’re getting every possible voice. Just want to make sure people feel safe about it.

Beal: Totally. And that psychological safety is so important. I learned this from one of my clients years ago. We didn’t use to do things like participant briefings, and now I think it’s really important to get everybody that’s involved in this type of work and talk to them about what DevOps is and the “why” that we’ve all talked about – so, organization performance and succeeding in a disruptive market and all that kind of stuff – and then give them the comfort about anything that they share is between them and the people that are collecting that information. And they may be quoted but they’ll never be associated with what they said. Their words may be used as an example but they can be completely honest. And actually, they are being asked to be involved because their opinion is so highly valued in the organization and we want to gather their insight, so they shouldn’t feel fearful or that their job is at risk. They should feel honored and hopefully excited to be involved in a process that’s intended to make their future more fun and secure.

Hidalgo: The only other thing I’d add there is – and again, I totally agree; I think we’re actually –

Beal: Finally agreeing.

Hidalgo: – in agreement here. Yeah. Yeah, exactly. But the one other thing I would add is that even if we have a strong suspicion or even have the numbers to prove that our organizations or our teams generally are psychologically safe and we’re building a good culture here and this and that, you don’t know what it was like for people at previous companies. People can carry baggage for a long time. It can take people a long time to adjust, especially as your career goes longer and longer. You have these memories and not all of them are great. And so, even if you think “Ah, our company as a whole, we are scouring very high,” for individuals it may not have anything to do with the current organization at all; they could still be carrying some scars from previous ones.

Beal: Yeah, and there’s –

[Crosstalk]

Shimel: Viktoria?

Praschl: It’s about creating the safe space. So, some probably will take longer to open up, but usually if you see that others are joining and the more comfortable you feel, you also get those people to open up down the road.

Shimel: I think we’re all humans, to borrow a phrase, and we bring our own experiences, outlooks, prejudices to whatever metrics we’re going to measure and to whatever we’re going to be in. And I think we can’t lose sight of that either.

Beal: Right. Those scars that Alex was talking about, I just wanted to talk about one other element of this, which is that the scars are negatives. We’re never badly affected by positive experiences, if that makes sense. So, from a neuroscience perspective we basically train our minds to be fearful. We keep on avoiding – we have an avoidance response that’s trained into us. So, I think I really like to try to use neuroscience when I’m helping people through a DevOps change because there are things we can learn from it, things like neuroplasticity and these different avoidance and approach responses that we have in different scenarios. And actually looking at an organization as a whole is great, looking at different teams and how they compare, but actually understanding, as Alan has just said, that how individual humans operate and in the context of their previous experiences and just how we’re built, because we’re all built differently from _____ anyway, is really important. And that’s really where the transformational leadership comes in and having people that understand how to be coaches and how to understand people at a personal level.

Ohlhorst: Interesting that we’re talking about the process from almost applying a “flight or fight” type of ideology to it, that here we are talking about software development and DevOps in particular and pipelines and all that, and so many people so often forget that there are people involved in the process. And that is where you come up against the most challenges.

Shimel: Yeah. It’s about the humans. Guys, we are – we’re about out of time, though we could probably sit here and talk for the rest of the day. But it wouldn’t – I don’t know if anyone would want to watch, and we all have other things to do. What a great conversation on metrics and DevOps and people and everything else today. Maybe we’ll come back and we’ll run a part two of this or something. But for now I need to pull the plug.

I want to thank our friends at Tricentis for making DevOps Unbound possible with their sponsorship. Alex, Helen, Viktoria, Frank, great conversation, man. Great conversation, men and women. Good stuff. Mitchell, you want to take us home?

Ashley: What I’m struck by is the richness of just the word “metrics” and what all it can mean and the depths of it and aspects, whether it’s the human side of it, whether it’s the “why,” whether it’s the outcome. There’s several – I think there’s several good takeaways there from this, and so I hope it’s helped everyone kind of think about maybe why you’re doing what you’re doing and how you want to lead that going forward.

Shimel: Excellent. All right. This is Alan Shimel for MediaOps, devopsy.kinsta.cloud, here on DevOps Unbound on TechStrong TV in the TechStrong TV network. We’ll see you in about two weeks with another great DevOps Unbound episode, but for now this is it. Thank you, everyone. Thank you for watching. Take care..

[End of Audio]

August 15, 2021
EP 16: How DevOps has Changed Testing Forever

DevOps has changed the game in almost every aspect of testing: who, what, when, where and even why. What’s working, what needs to change immediately, and what challenges—and opportunities—lie ahead?

In this DevOps Unbound episode, hosts Alan Shimel and Mitch Ashley are joined by Abel Wang from Microsoft, Tracy Miranda from CDF, Caroline Wong from Cobalt, Kurt Chase and Ben Simo from Tricentis to share a variety of perspectives on how testing fits into the ever-evolving world of DevOps. The video is below, followed by a transcript.

DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.

Alan Shimel: Hey, everyone. Thanks for joining us on this special edition of DevOps Unbound. You’re watching this, probably, at the Tricentis Virtual Event and we are happy to have you join us, happy that you’re attending the event, and we’ve got what we think is gonna be a great discussion on testing and all of its various facets and what does the future – or near future – hold. We’ve got a great lineup of speakers that I’d like to introduce you to who are on our panel today. Let me first start with – she’s on the top of my screen – is Caroline Wong. Caroline, why don’t you say, “Hello” and introduce yourself?

Caroline Wong: Hello. My name is Caroline Wong. I’m the Chief Strategy Officer at Cobalt. So, at Cobalt, what we do is pen testing as a service. We build a security software platform – like a lot of DevOps companies.

We have data-driven product-based teams. We have – we really value automation and failing fast. I’ve spent 15 years in the information security industry. I wrote a book called Security Metrics. I host a podcast called Humans of InfoSec, and I teach the OWASP Top 10 Learning Path on LinkedIn Learning, and super excited to be here with this group today. Thanks for having me.

Shimel: Thank you, Caroline. Next up, we have Kurt Chase.

Kurt Chase: Hey, everyone. I’m Kurt Chase. I’m currently Head of Release Management and Engineering Services at Tricentis. Engineering Services is a group that supports the engineering function so, we do our own type of testing – testing all the processes and everything we use – to deliver our software at Tricentis. So, prior to Tricentis, I spent some years at Splunk and at AutoDesk. It’s great to be here with all of you today.

Shimel: Fantastic.

Chase: Be here today with all of you.

Shimel: We got what you meant, Kurt. It’s okay. Next up, we have Ben Simo. Ben, why don’t you introduce yourself?

Ben Simo: Hi. I’m Ben Simo. I’m a software tester of 30 years and past president of the Association for Software Testing who has recently stepped into a product management and product research role at Tricentis trying to discover and develop our testing platform of the future and bring new things to Tricentis customers and hopefully introduce new people to Tricentis with new products that enable new ways of testing.

Shimel: Okay. Happy to have you here. Next up, Tracy Miranda. Tracy – well, I’m gonna let you introduce yourself. Go ahead.

Tracy Miranda: Hi, everybody. I’m Tracy Miranda. I’m the executive director at the Continuous Delivery Foundation. It’s an open source organization that’s part of the Linux Foundation, and also home to projects to like Jenkins and Spinnaker, and our members include companies like Netflix and Google. And I have over 20 years of experience in open source so, I love open source and I’ve worked in it as a developer – also in QA and all different facets. So, yeah. Big fan.

Shimel: Thank you, Tracy. Last, but not least, is my co-host on DevOps Unbound – my partner in crime here, Mitchell Ashley. Mitchell, why don’t you introduce yourself?

Mitch Ashley: Thanks, Alan. Yes. I’m CEO of Accelerated Strategies Group, which is an analyst firm that focuses on, in addition to cyber security, digitial transformation, DevOps, and cloud-native kinds of applications, which really matches my background very well. I’ve been both a product creator CEO/CTO as well as running IT, CIO, CSO – that kind of thing. So, currently, I’m also CTO with MediaOps – I guess CSO at MediaOps, too, so, I’m trying to collect as many three or four-letter titles as I can. So far so good.

Shimel: Alrighty. So, first of all, thank you all for coming on today. Secondly, testing – what a wild, mixed-up world we have in testing, right? Just from looking at the folks on our panel today, we have security testing – pen testing, in particular. We have the world of open source and what a factor that is in testing.

We have product testing, right? We have sort of new product testing – taking testing to new heights. And you know, that variety is sort of a double-edged sword, right? Because it’s like jack of all trades, master of none type of thing. There’s so many aspects of testing, right?

Even in our audience here at this virtual event, we have testers who come from, you know, what’s called old line QA testing, right, to DevOps enabled testing to visual testing – testers who, if it’s not an open source testing platform, they won’t use it, to proprietary systems. Through it all, though, a lot of times, all of this testing becomes sort of a red-headed stepchild or a whipping boy, if you will, right, to the whole application development system operations teams. It’s like, at the same time, their best friend and their worst enemy, right? They want everything tested, but they don’t want to give you the time, the budget, or the resources to do it, and then, when something doesn’t work, they said, “Well, we should have tested more, right?” So, how – first of all, do we agree that testing has been sort of the whipping boy of a lot of what we’ve seen go down in app dev and operations? Anybody want to jump on that?

Chase: Well, I think it’s certainly gotten a bad name. I think some of that comes from prior to the digital transformation. A lot of the engineering was waterfall, and testing was always that last stop before the software went out live, you know? Nowadays, I think everyone’s come to realize the speed and scale at which we develop things, you have to do the testing up front at the very beginning. You can’t wait till the end, you know?

You’re gonna be too slow. So, I think, from that perspective, absolutely. I think that’s why testing has really, I think, gotten a bad name, because they’re kind of the blockers to getting anything out into the public domain, whereas nowadays, you have to enable that up front because you want to get it into the public domain yesterday.

Shimel: Yeah. And so, in many ways, to me, that sounds like, well, DevOps has been the best thing that ever happened to testing, right? In many ways, DevOps has been the best thing that’s happened to security testing, right? We’ve certainly seen that shift left, if you will. But has that shift left helped – has that rising tide lifted all boats?

Miranda: I still see –

Shimel: Oh, okay, Tracy. Go ahead.

Miranda: I’m gonna jump in and say one thing that strikes me as funny. Like, we have, in our industry, everybody has embraced the DevOps terms, and then, you start to see all these spinoffs to put emphasis on things like, for security, we have DevSecOps, and you’ve got every variant of something with ops. But when it comes to testing, you know, there’s no TestOps. There’s no new buzzwords and it kind of speaks to how much it’s being neglected. Like, we had a conference on continuous delivery, and we asked for talks of everything in the life cycle, and we got lots and lots of submissions except for testing.

So, again, I think it still is really under-appreciated and just not given the due credit for what it stands for.

Shimel: Very –

Simo: Yeah. And I think some of this comes from, as Alan mentioned, that testing is a very broad and diverse subject, and I’ve often watched people argue about testing that each interpret it to their piece of what they do with testing. And things like CI and some of the DevOps practices have helped us improve with the checking that our software can work – you know, demonstrating that it can work – and there’s a lot of people – if you primarily wear a developer hat, you’d look at that’s what testing is. And we’ve automated that.

There’s been huge improvements in tooling and technology to do that, but sometimes, along with that is we throw out that we need someone involved that’s looking at things from a critical perspective that’s concerned about the safety and efficacy of what we’re delivering that is the, I guess, the one questioning and searching for information that they raise issues that we need to deal with that are reasons to not push things out to production now – so that we can make informed decisions. So, often, I think just that critical side gets thrown out as we improve the other side of testing, which is being able to demonstrate that things can work.

Shimel: Anyone else?

Ashley: You know, Alan, one of the ways you can tell who drew a diagram is to look at what’s in the center. And so, if we think about CICD, we kind of surround testing, but we don’t talk about testing. So, in some ways, it’s elevated, and some ways, it still sort of abstracts it out of DevOps. But my experience has been not really, because most teams know that testing has to get automated as part of that process, ’cause you can’t have this automated checking and automated deployment, then let’s kind of let a bunch of manual things happen. And not that testing hasn’t been automated – it has been – but I think it has elevated, at least in part, I think a good deal.

My experience has been. I haven’t worked in every environment, maybe, where test still isn’t thought of very highly. By the way, the thing that the – it isn’t the last thing in the chain. Documentation is the last thing.

Chase: That’s true. That’s true.

Ashley: Let’s give documentation –

Chase: And release notes. It’s true. I have tech doc’s team, too. You’re right. [Laughs]

Shimel: And that hasn’t really changed much either, though testing has. No. No. But you know what? Caroline, I’d like you to focus in specifically.

I mean, I think about – you know, Mitchell and I co-founded a security company, and we came out with a product in 2003 of just a vulnerability assessment and manager – “VAM”, we called it – and at that point, look, we used to beg people to scan their infrastructure at least once a year, right, if not quarterly. We were really pushing for quarterly. And you know, the idea of scanning and testing applications before you deployed them – unheard of. It was unheard of, right? We didn’t have the security tools or people.

We did that after the fact. Now, that’s not normal anymore, right? Today, at least 85 percent of applications are tested prior to deployment. I don’t know who those other 15 percent are, but that’s a whole nother story. But talk to us.

You’ve lived through this, Caroline. What do you think?

Wong: Sure. So, you know, I think there needs to be an acknowledgment. There’s sort of these like, first order priorities and these second order priorities. When I think about security – and I think some of these concepts apply to testing more broadly – there’s a saying that, “You don’t take a $200.00 fence to protect a $5.00 asset.” I think security is about you have something of value and so, you want to test it – rather, so, you want to secure it.

Testing, I think, is like you made something new and so, you want to test it. You’re not gonna test something unless you have something new. You’re not gonna secure something unless you have something of value. So, it’s okay that security and testing are not sort of the number one thing, you know? Inherently, by their definition, it’s sort of a second order attribute, if you will.

Now, I think, you know, each of us has been through a version of annual organizational budgeting. You know, each organization says, “Okay. Here are the goals for the years. We’ve got limited resources and budget to accomplish those goals. How are we gonna allocated those resources?”

And there are some things that, for a lot of organizations, are very clear. You know, even if I take a step way back and I say, “Okay. Most organizations might think about budget allocations in three buckets. Number one – general and accounting; number two – sales and marketing; number three – research and development.” You know, nowhere in those words is either security or testing.

And that’s okay, you know? I think, Alan, when you share your story about VAM, you know, Vulnerability Assessment and Management, I agree that over the last, I would say, decade-two decades – so, when I started my security career 15 years ago, we were almost trying to convince folks that breaches really happened – that this was really a thing to be worried about. You know, today, everyone knows because it’s on the media headlines every day. This is real. And so, that has been a shift in the perception, I think, in our industry.

There’s also been a big compliance driver. So, the – in the area that I’m very familiar with, security pen testing, a lot of that pen testing is actually driven by external compliance requirements – things that say, “Because of compliance framework A, B, C, you must actually conduct manual pentest on your application, for example, once a year.” You know, doing it more often than that is often because of perhaps what a lot of this group might think of as a really good thing – trying to actually secure your software, trying to actually test your software to ensure it does what you think and what you intend for it to do, but that’s not necessarily the first thing that comes to an executive’s mind when they’re thinking about budgeting, and there’s a variety of reasons for that, right? There’s an external perception thing. I think there’s also a lot of learning, certainly, that the security industry can do to be more approachable and to really understand better what is the software development methodology that we’re engaging with.

What is the specific methodology and behaviors of these individuals? And how can we actually collaborate with them most effectively?

Shimel: Fair enough. I mean, look, I don’t want to be the pessimist in the group or the bluebird of happiness here, but we have had success over the last 5-10 years, Caroline. You’re right. Especially in security, but not just security. We’ve had success in testing across the board.

Tracy, to your point, the fact that we don’t have DevTestOps, I think, is, you know, testament to the fact that we’ve recognized that testing is so critical to the CICD process that it’s part of it. I mean, you can’t have DevOps, you can’t have CICD without a level of continuous automated testing in there – in my mind, anyway. I mean, prove me wrong. But, you know, I think that has been probably one of the biggest successes that the DevOps movement has brought – has been responsible for.

Miranda: Yeah. And I really like Caroline’s analogy with security because, you know, that’s something else we think of as fundamental to DevOps, but it’s sort of elevated itself and, in a relatively short space of time, went from being ignored to being, you know, headline news. And I think there’s a lot of lessons there. And I – like, I will say, I do think testing is going through that journey and we’re probably on the cusp of it kind of really exploding and watching a few players in the industry and companies starting to do really interesting work in testing. I think that’s gonna turn around very shortly.

So, yeah, there’s an exciting space that I think its moment is coming – just like securities has.

Chase: Yeah. I think it’s all about the testing now, quite honestly. I mean, when you look at the future with low-code/no-code, it all becomes question-question mark ops, you know? Any function in a company could actually put something together and run it, and they’re gonna have to test it, you know, and it becomes less about all the engineering and more about the test patterns and things like that that are needed as you get to these future states. So – and speed and scale is gonna continue, I think.

That’s driven everything with the digital transformation. It’s speed and scale. Everything needs to come faster, and we want more of it, you know? And so, I think that really pushes the envelope, and that’s what DevOps has responded to and, I think, addressed very, very well, but it’s not over. We’re gonna continue to scale up and go even faster.

So, we have to be prepared for that, and I think it becomes more about testing in the future, and less about, you know, we’re gonna have these pre-built components – APIs you can piece together however you want – then, testing becomes, I think, almost first order, to Caroline’s point. And security, especially. I think security, we’re seeing that change now. If you’re a SAS company especially, security has to be top in line, ’cause you just can’t put something out there and not have it tested from a security perspective. And, furthermore, with compliance, it’s not just ISO and GovCloud and those programs, but large corporations now are saying, “Hey, if you want software in my ecosystem, these are my rules and I want to see your pentest reports and I want to see this.”

So, I think more and more people are realizing that if the software’s not tested, it’s not worth it. And so…

Simo: And, in my experience, part of adopting a DevOps culture and not just tool set is putting testing at the forefront. Some of the most productive teams that I’ve worked with that deliver some of the highest quality software I’ve seen – testing was center to absolutely everything that everyone on those development teams did.

Ashley: Hm-hmm.

Chase: Yep.

Shimel: Yeah.

Miranda: Yeah. We think – go ahead.

Ashley: I’m just gonna jump in and say – there’s another factor that actually works to our benefit and that is not only to speed through automation and DevOps process, but contemporary software architectures, containerized micro-services, service mesh – we’re releasing smaller bits of code in encapsulated forms, which means, you know, we’re not testing monolithic applications all the time anymore so, we’re not having to run two weeks’ worth of tests just to get one pass through.

Chase: Yeah.

Simo: Right. _____ container solved the “It works on my machine” problem because we can ship the developer’s machine.

Chase: Yep.

Shimel: Yeah. That was always a big problem, right? Tracy, you were gonna say something though.

Miranda: Yeah. No. I think the direction we see it heading is we’ll start breaking the testing monolith to talk about what is unit test, what is system test, and appreciate that is a specialization in itself. And the other things I’m starting to see from some of the end user teams we talk to is this real focus on culture – you know, a continuous quality culture – and seeing that hand in hand as part of the testing movement and again, _____ back to security – you know, the security first culture went hand in hand with that whole emphasis on security so, similarly, the continuous quality culture is starting to be something that – like, the really high performing teas we see pay a lot of attention to.

Shimel: Absolutely. You know, Kurt, you said something there about low-code/no-code and how that could be a game changer. You know, some people refer to that as the rise of the citizen developer, and I’m wondering if maybe what we need is the rise of the citizen tester, right? Is testing too hard?

Chase: Yes. I waste gonna say – the one thing we haven’t touched upon – I think Caroline did – is testing is hard. Security testing is super hard. And when you look at today’s modern environment and a lot of the ISVs – at least the ones I’ve seen and been associated with – you have a legacy of monolithic applications that you have to support that are bringing in a ton of revenue, and, at the same time, you’re transitioning – you know, going through your own digital transformation. So, that adds complexity after complexity in testing, and I do think, in the future, testing – you know, a lot of us, like Ben said, to his point, we have to push testing up – you know, shift left, move it far left as we can.

And that’s gonna continue. I think that has to be the norm. To Caroline’s point, it has to become first order. And I think it’s starting to right now. I really do. Especially the – all the SAS solutions that are out there – you know, shipping code daily, hourly. You need excellent testing to do that and do it with competence.

Shimel: Yeah. I think you do. But we do have to – and it’s funny that we’re talking about making testing easier because – not security testing, right? Security testing, pentesters – they were always sort of the hired guns of the security space, right? There were SAS or – there weren’t companies like Cobalt, right?

The pentesters that I knew growing up – growing up – you know, that I knew coming up in the security space were gun slingers. These were, you know, white hat/gray hat folks who would come in and they’d do a little social engineering and pen testing using some open source tools, and you know, they give you a report. But you know, these were folks who traveled around the world doing that kind of thing. You know, we didn’t have a company like Cobalt that did this as a service, so to speak. So, it’s very different.

But here’s the underlying thing with like, QA kind of testing – and, you know, I don’t mean any disrespect to anyone in our audience today, but for too long, a lot of folks in testing were viewed as failed developers or people who couldn’t be really good developers did testing or maybe testing was a weigh station on the way to becoming a real developer – like a Pinocchio stop or something, right, where you’re not a real boy yet. But that’s not true. There’s also – there’s a whole cadre of professional testers. And you know, in their capable hands, you could do great things. But I guess the question is – does testing have to be so hard that we need professional testing – professional testers to do testing?

Or can we make testing automated enough that you – I’m not saying it has to be the dunces who don’t become developers. That’s ridiculous. But can we broaden the base of people who do testing? And that includes security testing? I’ll throw it out. Ben, you were a big-time guy in the testing industry. Tell us – what do you think?

Simo: So, I think we can expand who’s involved in testing and that more people than those currently doing testing can learn to do testing. I don’t know _____ developers learning to become better testers and test their stuff. But I will say that ultimately, the part of the work that is the testing cannot be automated. Tasks that we do in testing can be automated. So, whether – not matter what hat someone primarily wears, if it’s a developer, you know, writing tests to demonstrate their code works or someone’s using TDD and they’re building some tests and then, building some code that matches it, the testing is the critical evaluation that’s going on in their mind of solving the problem and demonstrating that it’s solved – or evaluating how something might go wrong and coming up with some approach to demonstrate that it may go wrong and check for that.

So, those things, whether you’re doing it automated or the execution is automated or done manually, the actually testing work, I’d argue, is the cognitive work that people are doing. And there is some – that we can never automate away. There are some ways that I think AI can help us. There’s also some ways that may lead us astray. But ultimately, I think we need people to learn to be critical thinkers to understand the technology, to understand the domain of the users and the problems that are being solved, and applying that critical thinking is the testing.

Of course, then, we want to use tools to help us with some of our tests so, when we come into DevOps and CI, executing those demonstrations of the software working is a key part of that. Now, whether we do it as part of our, you know, building to those tests or we create them afterwards, then they can, at the very minimum, act as changed detectors going forward. But there was some testing in people’s minds that was going on that created those.

Shimel: Caroline, what about – okay.

Wong: I agree with quite a lot that Ben is saying. You know, I can’t help but think to myself that – I actually have this sort of visual in my mind, and it’s actually at a circus of a trapeze artists, you know? I think about these people, and they’re sort of like, whirling in the air and they’re doing these like, really incredible feats, you know, and they just do whatever they do, you know? But there’s like, a safety net to catch them if something goes wrong. It also reminds me of a metaphor that is often used in security that says, “We put brakes on a car so that you can go fast.”

You know, when Mitch said earlier, you know, “These days, we’re releasing smaller pieces of code, modules of code” and there’s a way in which, yes, each small piece of code – each module – can be tested and some of that can be automated. We should automate performance testing. We should automate unit testing. But, to Ben’s point, there’s a lot that can’t be automated. There’s a lot of testing that actually depends on things like human judgment and opinion and it actually requires a person’s decision, you know?

So, I think there’s – I think there’s a couple of frameworks for us to consider, right? If the first order is about creating value and making something new and the second order is about testing to make sure it does what you wanted it to do and protecting the value created, there’s also this spectrum that says there’s smaller bite-size stuff that may be a good candidate for automated testing, but, I think it’s super important to recognize that these small pieces of code – these modules that Mitch was talking about – it’s also super important to test what happens when they come together. So, to the extent that security folks need to be able to test chained exploits and business logic flaws, you know, those are exactly the types of things that will never be automated – is what I think, anyway.

Shimel: I don’t disagree. I think security in particular is just there’s some things you’re just not gonna automate. I don’t care what you say. But that’s okay. Anyone else?

Chase: I do want to make a comment on something you said. This was a long time ago now, when I was a dev manager, that I used to sense that when my developers – you know, if you were on my team, if you wanted to really get me mad, treat your tester badly. Treat them, you know, a bit, like you said, like they’re not quite at the level you are. I would be incensed as a manager. You know, that was a long time ago.

Fortunately, I don’t see that as much nowadays. I think that role has been elevated where I don’t feel that at all on my teams, and if I do, that is, again, to this day, I’ll get super mad about that because I think it takes all kinds to develop software today and ship software. Testing role is just as critical as an engineering role, just as critical as the ops’ role, the security role. So, yeah, that was one thing that would drive me nuts ’cause I did sense that and go through that, and this back in, you know, the late ’80s-’90s, if you will, before the year 2000 – I’m dating myself – but it definitely was palpable in some of the large ISVs, you know? And as a manager, that was something that would drive me nuts.

Shimel: Agreed. Agreed. Tracy, open source – look, testing has been inexplicably wound up with open source for as long as I’ve been around, right? Whether it’s security – open source security testing and you know, things like Metasploit and Nessus and Nmap or you know, in QA testing kind of stuff, it’s a rich history of open source testing tools. How do – and part of your mission at CICD is – at the CICD Foundation – is to nurture that, I’m going to assume, correct?

So, how do you guys nurture that? What are you – what do we do to maintain that rich, open source tradition here?

Miranda: Yeah. No. Absolutely. And, as I say, we at the Continuous Delivery Foundation, we want to look across the entire software delivery life cycle, and I think it is safe to say, you know, that testing hasn’t risen to the top. We’re actively – like the board wants it to be a more focused-on topic, the technical committee wants to encourage folks to bring us more projects in open source and with testing, but we’re still having to kind of fight through all the noise around other projects.

And one of the ways we’re approaching it is, for example, our technical oversight committee wants to set out a landscape of the types of projects in each kind of bucket and say, “Okay, look. We need to have something in security. We need to have something in testing that covers these different aspects.” Almost like a call to action to say, “Bring us your testing projects and –” like my favorite saying about testing is, “There’s two key problems in testing – not enough tests and too many tests.” And yeah, I think we’re starting to see some really good frameworks around test generation.

I’d love to see more open source options for that. And Kohsuke Kawaguchi, the famous creator of Jenkins, move on to focus on testing so, that says a lot about the importance of this area getting more open source focused. So, yeah, I’m excited to see what could come out, but we do have to actively encourage folks to come join us and have those conversations and not to, you know, have it assumed it’s all focused on delivery of pipelines for delivering software.

Chase: I think another thing you’re seeing today is a lot of people, when they start developing an app, when they start creating something new, that’s the question they ask up front. “How are we gonna test this? How are we gonna automate it? How are we gonna pentest it?” I think those things, at least, are starting to bubble up a lot stronger in the cycle – you know, early on. ‘Cause I know that’s something we recommend highly – is that if you’re developing something new or need to head down a digital transformation, think about the testing. How are you gonna test that up front, you know? And products are actually being developed with that mindset now.

Simo: And with short iterations, it’s how are we going to test this tomorrow, next week? It can’t be left off for months later to be someone else’s problem. So, people, I think, are learning to deal with those questions about testing from the start of their discussions about what the requirements are.

Chase: The other thing I love – and in Caroline’s world, has been very successful – I know it is specialized skill set, but trying to teach that throughout your troops – you know, throughout the engineering team. Show them what, you know, across _____ scripting errors. What does it look like in code? That’s super successful. That really, really helps make – ’cause security is difficult, you know, no matter how you slice it.

Even when you’re looking at product development – if you want to invest in fixing security defects or add a cool new thing for your users, you know, a lot of times, the cool new thing is gonna win out. But that’s starting to change as well, too. I think people are starting to value security – especially in today’s modern world, you know? It has to be up front, really.

Ashley: Yeah. It’s all about AppSec at the moment. That’s one of the hot topics as well. You know, someone mentioned AI earlier. I just want to offer a perspective about that because we always seem to focus on the shiny new object.

I know I have. I often do. But I think what we’ll see with AI is select applications when it comes to testing. The sort of vertical place is where it’s really well suited. When we say, “Yeah, AI”, we really mean machine learning in most cases, which is machine learning algorithms, and there’s two types – there’s supervised/unsupervised.

Supervised is, “That’s a car. That’s a car. That’s not a car, right? That’s a bug. That’s a bug.”

And unsupervised takes massive amounts of data and intellectual patterns and learnings from that. So, I think that as we’re taking our test automation and our test tools, there may be some of that, and it might be really useful for AI. I wouldn’t get worried about people being replaced as testers. Not in my lifetime, that’s for sure. It is still a – there’s very much a craft to it, just like there is software design.

There is a craft to testing and a skill and a mentality about it that’s unique, just like with security.

Shimel: No doubt. Caroline – oh, go ahead, Ben. I’m sorry. No, no, Ben. Go.

Simo: Yeah. I was gonna say – and with any other sorts of automation – even you know, not learning automation that’s come before AI and machine learning based things, is that it not only has potential, in some cases, to replace some of the mundane work we do, but to enable to the skilled people to focus their work and apply their brain power to the things that really matter. So, I think there’s great potential for AI to help direct people’s attention to the things that maybe should be of concern and help deal with that big question of testing out of infinite possibilities of PaaS and data variations. Which one’s gonna show me the problem? And I think there’s huge potential for AI to help us find that if we’re also aware it may guide our eyes away from something that we do want to see.

Shimel: Caroline, how does that relate to security, though? I mean, is AI helping Cobalt with pen testing? Be honest.

Caroline Wong: So, the type of software that we’re building today is actually workflow and collaboration software. So, what we’re trying to do, is we’re trying to bring security folks and developers and penetration testers all together sort of in one place where they can collaborate. The technology piece, for us, is not about automating defect discovery. It’s not – I mean, if we’re gonna us ML or AI in the future, honestly, what we’re gonna do is we’re gonna use it to say, “What are the characteristics of this organization that wants us to do a test on this particular type of asset? What are the attributes of that asset and how might we match that to our talent pool – something that we do totally manually today?”

I think that’s the opportunity. I don’t think that we’re gonna get to a point where we’re using the AI and the ML for automating the defect discovery itself. Frankly, I think that even if I take a step away from pen testing and I look at the security scans that are in existence today, I think, you know, whether or not they’re referred to as using AI or ML techniques, they’re just simply imperfect, you know? And there’s a super important place for them, but there’s a lot of false positives. There’s a lot of interpretation of data.

And I think that’s always gonna be the case. I think that no matter what, when it comes to humans and machines, the machines can add a lot of value, but I don’t know that when it comes to software development testing and security in particular that there’s every gonna be a complete substitution. I don’t think that – I don’t think that makes sense.

Shimel: Absolutely. So, guys, we’re coming up on the end of our time. I need to wrap up. Tracy, let me say this for you, if you don’t mind – and if I’m wrong, correct me. For you testers out there, the CICD Foundation – or CD Foundation, actually; I keep – we always say “CI” like it’s attached at the hip – but the CD Foundation is looking to hear from the testing community.

Probably some companies – including Tricentis – may want to look into becoming involved with CD Foundation because if you don’t have a seat at the table, you don’t have a right to complain about not being heard. And so, you know, when we look at foundations like Linux Foundation and the CD Foundation, CNCF, we need the testing community – vendors, professionals, folks in it, you know – to be involved and get involved. Be heard and be part of it. So, I’m gonna extend that out there to everyone. But, short of that, though, Caroline, Tracy, Kurt, Ben, and, of course, Mitch, thank you so much for joining us on this special edition of DevOps Unbound.

We hope everyone enjoys the rest of their Tricentis Virtual Summit. This is Alan Shimel for MediaOps, devopsy.kinsta.cloud, Security Boulevard, Container Journal, Tech Strong TV. Have a great day, everyone, and thanks for joining us.

[Musical outro]

[End of Audio]

August 5, 2021
EP 15: Why Testing is so Frustrating

In this episode of DevOps Unbound, hosts Alan Shimel and Mitch Ashley are joined by Adam Kalsey of Tricentis, Brian Dawson of the Linux Foundation and Caroline Wong of Cobalt to discuss why testing is considered the most frustrating bottleneck for developers and engineers. Panelists will explain the benefits of testing early on in the SDLC and how to accelerate the feedback cycle with automated and continuous testing, as well the pros and cons of testing, the value of shift-left testing and how to effectively balance quality and speed. The video is below, followed by a transcript.

DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.

Alan Shimel: Hey, everyone. I’m Alan Shimel, and you’re watching DevOps Unbound. DevOps Unbound is sponsored by our good friends at Tricentis, and it consists of a biweekly, 40, 45 minute video panel on relevant topics within the DevOps sphere, and featuring a panel of experts that rotates every other show, and really great people, and something I personally really enjoy doing.

And then, about once a month or once every six weeks, we actually have a live studio audience—or there is no studio audiences in COVID, actually, but we have a live virtual audience that participates in our show. Today’s show is just our panel, though, without a live audience, and I’m really looking forward to our discussion.

Our discussion today is around testing. You know, with the pressure on developers to deliver code faster, sometimes testing is considered a road bump in the way. Testing is considered slowing down the whole process. There are a lot of developers who’d like to push testing off that plate. Of course, no one wants to go back to testing after we’ve deployed software, primarily, but there is that tension there.

Let me introduce you to our panel today; we’re gonna discuss this in depth. First of all, making his first appearance as a member of The Linux Foundation on our Tech Strong TV network is our old friend, Brian Dawson. Brian, welcome to DevOps Unbound.

Brian Dawson: Thank you, Alan. Thank you, Alan. And it’s fitting that the first panel, first webinar I do since joining The Linux Foundation is with you.

Shimel: It’s our pleasure to have you on. Brian, what’s your position at Linux Foundation now?

Dawson: So, at The Linux Foundation, I am functioning as the VP of Developer Relations and Ecosystem Development. So, just to help, we’re gonna help The Linux Foundation to the development community while driving more of a DevOps opinion into a growing ecosystem of solutions that support open source development.

Shimel: Fantastic. Next up is my good friend, Caroline Wong. Caroline is—oh, she’s an anchor on many of our different shows here at MediaOps and Tech Strong. Caroline, it’s great to see you again. A little background, maybe, for people who aren’t familiar with you?

Caroline Wong: Great to be here. Thank you so much. My name’s Caroline Wong, I’m the Chief Strategy Officer at Cobalt. We’re a pen test as a service company.

I actually started my DevSecOps career 16 years ago leading information security teams at eBay and Zynga. So, in both cases, we were running online operations 24/7 with millions of simultaneous users daily. And now, at Cobalt, we build security software. And, like many DevOps companies, we’ve got data driven, product based teams. We value automation and failing fast, and super excited or today’s conversation.

Shimel: Fantastic. Welcome. Great to have you, Caroline. Our third panel member is another returning DevOps Unbound veteran, and that’s Adam Kalsey of Tricentis. Adam, welcome.

Adam Kalsey: Thanks. Thanks for having me. I lead Developer Relations for Tricentis. I’ve been building web applications since 1995, and have always had a soft spot in my heart for bad applications and broken things. And so, I love to help developers build higher quality software and do it in a more efficient, faster way.

Shimel: Absolutely, and thanks for being on with us today, Adam. Last but not least, in his Florida outfit here, visiting me in Florida, is my Co-host of DevOps Unbound, Mitch Ashley. Mitchell, welcome.

Mitch Ashley: Coming to you from the MediaOps Studio B.

Shimel: Yes.

Ashley: Great to be here. I’m a former product, both security and software SaaS, CTO as well as a CIO, and now I run Accelerated Strategies Group, an analyst firm that focuses on those things.

Shimel: Absolutely. Hey, Mitch, thank you.

So, let’s get started. Adam, you’re with Tricentis, the sponsor of DevOps Unbound, and they’re a testing company, probably the leaders, one of the leaders worldwide in automated testing, continuous testing. Have we put too many straws on the developers’ camel’s back by making them, you know, be more responsible for testing to say we need to test more of our code earlier.

Is it a bridge too far for many developers? Is it making them resent testing? You know, what’s the current state?

Kalsey: I think where we’re failing developers is by handing them lots of more things to do without giving them any—without giving them what tools, and I don’t mean just necessarily software tools, but without giving them the things that they need to do it effectively. If we just pile things on developers and say, “Oh, this is your job now” but we don’t give them the support, we don’t give them the training, we don’t give them the expertise, we don’t give them anything to make that better—sure, it’s just gonna be harder on ‘em.

Shimel: To play devil’s advocate—oh, those poor little testers making a quarter of a million dollars a year each. Do we want ‘em to do testing, too? Right? Is there some of that here, Brian? Or—go ahead, Adam?

Kalsey: No, no, Brian—please, please, jump in.

Shimel: Well, he’s from The Linux Foundation, he now represents these folks. [Laughter]

Dawson: Well, first, not all of them make a quarter of a million dollars a year, right? But I do think what you raise is important when you bring up the investment, right? We are hiring talent with a particular expertise in a particular domain, a particular ability to solve problems and innovate.

And there is an argument to be said, Alan, that, “Do I want to spend this money, this quarter of a million dollars plus 200,000 in RSUs on having somebody write test harnesses?” Now, I think—you know, that can be balanced by what Adam said. Well, first of all, yes, because we want everybody thinking about quality. But you know what, if we want a return on that investment, we wanna free the cognitive load, then you know what? We need to invest in automation, we need to invest in better collaboration and reducing friction from the process.

Shimel: Agreed, agreed. Caroline, you know, it’s been a big—this whole DevSecOps movement and shift left has been about shifting security testing left and getting developers more involved in security testing, right? Yet another camel on the—yet another straw on the camel’s back. Should we be doing that?

Wong: So, I—oh, my Internet is awful. Can you hear me?

Shimel: Yes, we hear you fine.

Wong: Okay, fantastic, awesome. I’ll just—I know that there’s post editing, so it’s okay for me to say that.

Shimel: Mm-hmm.

Wong: So, Alan, I bring a particularly unique perspective to this group in the sense that I actually oversee people at our company, I oversee Human Resources. And I’ve been with the company since we were 10 people, and today we’re 200. And one of the super interesting things that happens is, naturally, as a company grows, what people do, they specialize and then we have to figure out how to communicate more effectively.

And I think that if you’ve got an engineering team and you say, “Okay, now you’re responsible for Development and you’re responsible for Ops, and your responsible for security, then there’s got to be some level, depending on the maturity and the size of the organization and what the organization’s trying to accomplish of specialized roles and protocols for engagement.”

And I think that when, either number one, those roles are not clear to, I think Adam was making a point earlier about security folks need to be providing guidance to engineers where—because engineers are not expected to be security experts, frankly. Security experts are not expected to be engineers. And then, at the point where information is being handed off where the responsibility for taking an action, those, all of those sort of procedures, they have to be clear, otherwise they’re simply not going to happen.

Shimel: Yeah, agreed. I think they have to be clear. But even if we make them as clear as we possibly can, is it just—are we putting too much on developers? I mean, I think fundamentally, that’s the question here, right?

Wong: There’s—

Ashley: Alan, it’s not only—sorry, I’m just gonna jump in. It’s not only quantity of work—is it the right work?

Shimel: Right.

Ashley: Because I know from writing software and being part of software teams for so many years, there’s this thing called unintended consequences or unintended use. Well, you’re not supposed to use it that way, but the person who designed it intended it to be used a certain way, but one of the important functions of tests is not just to uncover bugs and elicit those things or security issues, it’s also to push the boundaries about how you might use the software and essentially stretch it to try to find where it’s gonna break where they can. And oftentimes, that’s a different mentality than or perspective, certainly, than the person that wrote the code.

Dawson: And if I may jump in and say maybe part of the discussion can get to generalist versus specialist, right? There is this idea, Alan, that when you talk about putting a lot in the developer’s hands, this idea that there is this prolific person called the full stack developer that can separate themselves from what they develop to look for security vulnerabilities, to test it, then ultimately deploy and run it, right? And I think we need to give thought to, are we trying to build a bunch of generalists, which doesn’t potentially enable them to do any job the best, or are we enabling specialists to better work together, share information, and crowdsource quality and crowdsource security?

And I’m just curious if any of the other panelists have an opinion on this concept of full stack developers versus, say, needing traditional, lifelong quality experts.

Kalsey: Yeah, I like to say that we decided to create cross-functional teams and we did it by removing all of the functions, and we centralized them all in a single person—and that’s not a cross-functional team. That’s—

Shimel: That’s a cross-functional person.

Kalsey: Yes. Well, and you end up with the jack of all trades, master of none problem. You have to have the specialists on even a cross-functional team. You have to have, there’s a lack of testers, there’s a lack of UX people, there’s a lack of security people. They have to be part of that team. You have to have people that, dedicated folks to do the hard work. A developer may be able, and a generalist may be able to do a lot of the easy things and a lot of the, “Oh, yeah, we’ve done this before or we’ve seen this, and this is a known know, let’s go fix that.”

It’s the hard work of figuring out—what don’t we know, and then having the expertise to be able to know what that hard work is, even, to go after it. And without that, of course the developers are going to be overloaded, they’re being asked to do everything and they don’t know what everything is.

Shimel: Agreed, agreed. I mean, I think, again, security is the poster child for this, right? We’re asking developers to be security people or to make security a priority. Now, you ask most security people four or five years ago, they’d tell you you’re smoking your socks. Developers don’t care about security. They never did. Only we care about security.

It’s changed. It’s changed a little bit. We do think other—no one wants to develop insecure code. But we can’t ask them to be security people—or can we, Caroline?

Wong: I sort of think we have to. I think that, you know, the big, bad thing that no one wants to happen is a security breach, and a security breach cripples an engineering team’s ability to execute on a roadmap. You know, if something big and bad happens, then entire teams have to rush and commit to fixing this problem sometimes for months, you know, time which, of course, could be invested more profitably elsewhere.

I have a quote that I’d love to share from a colleague. Denali Lumma is the VP of Engineering over at Project Ronin, and she also makes a really good point about tech debt. She says, at a certain point, it’s too late, there’s too much tech debt, too much complexity, the code base is too big, and it becomes impossible to do anything about it. What you see there is a business will slowly die because shipping features becomes harder and harder, critical vulnerabilities and bugs are introduced more frequently, and eventually it kills the business.

You know, I often think of software as a living organism, right? Whether you think of it as a tree or as an animal, there is this balance between making sure you’re doing sort of your fun things, your running, but you’re also resting when you need to heal. You’re also—you know, if we’re talking about a human being, you’re also sort of going to the doctor once a year.

Shimel: Agreed.

Wong: Yeah.

Shimel: Agreed. Thoughts?

Kalsey: Quality has to be a mindset, security has to be a mindset, user experience has to be a mindset on the developer time, among everybody there. The user experience person needs to understand and have a mindset of developing and of creating a secure experience and not, “Well, we’re gonna get rid of all the security because that’s hard for people to use.” Everybody has to have that mindset, but you still need the experts and you still need the expertise to be able to do things well. Just having a mindset doesn’t make you fantastic at it. You can say, “Oh, I love to get out and I’ve got an exercise mindset, so I’m gonna go run a triathlon tomorrow.” It’s probably not going to work.

Shimel: Probably not.

Ashley: You know, it’s interesting—we could’ve had this conversation 20 years ago, 10 years ago.

Shimel: Absolutely.

Ashley: You know, why are we still frustrated with tests? What is it with tests that is, you know, it used to be sort of the orphan of the development process, you know, the only thing that happens later than test is documentation. Well, now, [Laughter] it’s still an issue. Why is that? Why are we so frustrated with tests today? I’m curious what folks think.

Dawson: I’m gonna start with one not very complex thought—it’s actually hard.

Shimel: It damn right is.

Ashley: A little more respect—we need a little more respect for tests. [Laughter]

Shimel: Well, but this was like, I remember when we first launched devopsy.kinsta.cloud, a popular theme was that, you know, DevOps would kill the testing star, right? That there would be less QA jobs because everything was gonna be automated continuously and shift left, we wouldn’t need testers—poppycock, right? We have as many testers as we ever had, if not more. It didn’t kill off testing jobs. I think it was Adam that said we’ve made testing part of the DevOps team. So, we have people on the DevOps team who are responsible for the various, you know, styles and types of testing that we do, including security.

I think what’s happened, though, is, these DevOps teams—when I first started devopsy.kinsta.cloud, if you spoke to developers, they’d say, “Oh, this DevOps is all about the Ops.” And then you’d talk to the Ops people and they’d say, “This is DevOps, it’s all about Dev, it’s Dev-centric,” right? And that’s the sign of a good compromise when they’re both pointing the finger at that.

But it has come—at least as far as testing goes, from what I see, it has sort of tilted where, you know, as we’ve shifted testing left, we’ve shifted responsibility left as well, which kinda winds up closer to the Dev, or certainly to the DevOps teams.

However, though, let’s look at the alternative, and I’m gonna ask all three of you—all four of you have been around the block once or twice. My personal feeling is, is that we do a much better job of test coverage today than we did 7 or 10 years ago, and I think DevOps and continuous automated testing is a big reason for that.

Thoughts? Anybody disagree? Show me wrong.

Wong: I will challenge you, Alan.

Shimel: Okay. I was hoping someone did.

Wong: I’ll challenge you, and the reason is because—so, we’re all familiar with the OWASP Top Ten.

Shimel: Very much.

Wong: There have been a handful of versions of the OWASP Top Ten since the first was produced in 2003. I believe we’re gonna see another version later this year. The current version is from 2017, and if you take the 2017 version of the OWASP Top Ten and you put it right next to the 2003 version, it is sort of mind blowing how similar they are.

Shimel: Yeah.

Wong: And what that says to me is, we actually know exactly what the problems are. We know how to find them, we know how to fix them, and we know how to prevent them, but they’re still around, and so I think that’s—

Shimel: They are, but are there as many of them, right?

Wong: Absolutely. [Laughter]

Shimel: You think there’s as many cross-site scripting or buffer overflows than there was 10 or 15 years ago?

Wong: So, there are some categories that have been eliminated. You know, there are some categories that, for which we can define a secure pattern, and we can encourage folks to use that secure pattern, and for which we can actually eliminate some of these things.

But there’s others or which we cannot. You know, I think about—I recently visited my sister’s home, and they have two young children, a 2-year-old and a 4-year-old. And at night, what they do is, they run a Roomba, which is brilliant, you know? If you could spend your weekend playing at the park with your kids instead of vacuuming, of course you’d wanna do that. And the Roomba works because they sort of take the time to get all the toys off the floor and make sure everything’s clear and once they use the Roomba, they clean it. There is this different work involved in effectively using a Roomba that’s different from sort of manually using a vacuum.

And I do think there are these two distinct categories of testing, one category which you could say is known, repeatable, and automatable, and another for which manual testing, I believe, will always need to exist. There are types of security flaws like business logic flaws, race conditions, chained exploits that I don’t believe that in my lifetime, we will be able to figure out how to teach a computer to find these particular types of issues.

Kalsey: And it’s beyond just security testing. I mean, exploratory testing, from a tester, your automated tests are not going to find, “This error message makes no sense in this context. I did this, I expected—I don’t know what this error message is trying to tell me to do.”

Your automated tools aren’t going to find that. All they’re going to be able to find is, “It didn’t work, we sent an error message. We had the things we wanted to put in the error message there.” You can test that your code is doing the right thing, but if you didn’t design it to do it right, you’re not gonna find that with an automated test.

Shimel: Fair.

Dawson: I would tend to say, for the surveys that we’ve issued and I’ve read, your original statement, “Has test coverage improved significantly?”—yes, I think all indications are, we are getting better at automated testing, we are getting better at that pursuit of that thing we call test coverage.

But I think what’s more important is, has the quality of test improved? Are we testing the right things, right, from functional testing to security testing, various shades of security testing? And I’d say (1) yes, we have improved there by sort of the qualitative and quantitative measures that I’ve seen. But I also think, as we truly start to focus more on quality, we start to achieve more coverage, we start to implement better tests, it becomes more clear, as I think Adam and Caroline are speaking to how much more work there is to do, how much other things there are to improve testing on. Pair that with a constantly changing technological landscape, right? We’ve gotta learn new stacks, we’ve gotta figure out how to test new stacks. We’ve got half of our put in in a SaaS platform, but then we deploy on prem, et cetera, et cetera, et cetera. And then you add to that that, at the same time, we’re trying to transform organizations.

So, Adam, you talked about exploratory testing; Caroline, the human element. I think one of the things that pursuit of the full stack Dev, that pursuit of the fully automated processes failed to do in a lot of orgs I speak to is, we talk quality first, but oftentimes, the DevOps transformation doesn’t involve quality engineering first, right? And I think one of the things we need to get better at is engaging with and partnering with Quality and establishing our automated and modern processes.

Kalsey: Yeah. We talk good in the testing industry a lot about shift left and that testing is moving earlier in in the process, and too many organizations have taken that as, “Well, the developers are the ones that are early in the process, so that means the developers do it.” No! Shift your damn testers left. Get them involved in the design phase. Get them involved in talking about the architecture. Get them involved in the entire thing, not shift the task to the people who are earlier in the process. Shift the whole process, please.

Dawson: Or to a script, yeah.

Ashley: I’m gonna take a whole ‘nother tack, and I thought you were gonna go here, Brian, is, I think we need to shift tests down. We need full stack testing. It’s not application testing anymore. Most of the security errors that we have are due to configuration errors. And what Caroline said was absolutely true. We still have SQL injection, cross-site scripting, and overflows may be less so. But why? It’s because it’s not just in the app layer that those things happen, it’s in that software stack. And I know not every tester can be an expert at the stack itself, but maybe that’s, instead of kinda dividing the DAST and SAST and the different kinds of testing that we do and just deciding, can we shift those in one direction is, let’s think about it from a depth perspective, because that’s where a lot of those errors are coming from. It’s not just the app.

Kalsey: Oh, yeah, and the tooling on this is also, can be super fragmented. You need one thing to test your infrastructure code, you need another thing to test your UI, you need another thing to do the security—and they’re not, consolidating those reports and combining them and figuring out what worked and everything is hard work and there’s nothing that does that. This is a place that Tricentis could definitely improve and improve our own tooling here.

And then, once you’ve done that testing, the results are often in a silo. Okay, I found out I had a problem—how do I fix it? How do I go see the data behind what was going on in the system to understand this and solve this? And now I’ve gotta dive into a monitoring tool or my logs or an observability tool instead of just having everything all in one place.

Shimel: So, I think there’s a—I’m sorry, Caroline, you go.

Wong: No worries, and thank you, Alan. So, we’ve talked about shift left, shift right. Mitch is kinda talking about shift up and down. I think there’s another axis, if you will, which is sort of what I’ll call shift in and shift out. So, you can test things in and out of band, and that’s gonna make sort of a wildly different difference in terms of how those processes run.

You know, I think 10 years ago, security people appreciated the power that came with being a gate in the process, being an end band process. You know, today, it actually makes sense, I think, for a lot of security testing to be out of band, but that removes the sort of power and authority of security people to say something like, “You must fix this, otherwise you’re not gonna go live if it’s happening out of band.”

Shimel: Fair. So, I’m getting a little dizzy, we’re going left, right, up, down, in, out. But—

Ashley: It’s a circle. [Laughter]

Shimel: Yeah, you know, shake it all about. But here, fundamentally, though, are we just kinda dancing—no pun intended—are we just dancing around the real issue, which is, we wanna do more pre-deploy, we wanna do more, we want better quality software, right? I think we could all wrap our heads around that idea and vote for that. We all want better quality software. Security people are at a premium. We can’t shift security people left or en masse. Testing people are also at a premium, though we’ve integrated them into the DevOps teams. Developers are at a huge premium, they’re probably our most expensive resource in terms of humans.

What do we need to do to make everyone’s job easier here, to make this function better? Is it a question we’ve gotta automate more? I mean, maybe that’s what it is, if we automate more, or is it a cultural thing, right, where—

Ashley: There’s also test-driven design, right, which is, as you’re designing it, you talk about, you put together your test strategy of, “Okay, we do it this way. Let’s talk about how we are going to test it, start to lay out the framework for how we think this can be validated.” I mean, it’s not the only thing to do, but I think—so, if you start with that in mind, now you’ve got a path to pursue as opposed to, “Here’s a chunk of software, you folks figure out how to test it, right? We built it, you test it.”

Dawson: Yeah, and I’d add, I think there’s no—again, not only is the practice of testing harder than we often like to admit, but the, you know, the path to kind of getting to where we fully wanna be, where we don’t dread testing, complex, hard path, and I’d say it’s the DevOps trinity, you know, Alan, and we all can probably hit a few points on each.

But you have to address people and culture, as Adam brought in earlier. There’s no getting away from, security needs to be a thought for everybody, right? I can install all the locks I want on the front of my door. Actually, over 24 years—today is my anniversary—my wife has programmed me to lock the door when I come in the house without thinking, right? She’s kind of impressed a security culture on me.

You do have to address people and culture, process, practice, tools and technology. Automation alone will not solve it. A mindset alone will not solve it. All of that needs to come to fruition.

Shimel: Mitch, make a note—invite Brian’s wife to our next security panel.

Dawson: [Laughter]

Ashley: I’ve already looked up her e-mail address—thank you, Brian, appreciate it.

Shimel: Happy anniversary, by the way, too, Brian.

Dawson: Thank you, thank you.

Kalsey: This is one of the places where I think what Caroline said is so important, that this in bound, out of bound—if you’re trying to do everything in line, of course, it’s gonna be harder. In shifting that mindset to doing things continually in the background, we already do so much stuff and process automation in the background where we don’t have people check, “Here’s a step in the process.” No, this is going along in the background, dependency watching and that sort of thing and updates.

We can be doing testing in the background as well for a lot of it, and identifying what needs to be in the foreground and what needs to be in the background. Look, I’ve seen CI/CD pipelines that take hours and hours and hours to run because the tests take six hours to execute and to build. That can’t work in any sort of a reasonable DevOps system.

Wong: I think there’s two parts to a solution, and I’m really, frankly, just parroting what Brian said. I think there’s a collaboration piece between the people. You know, to an earlier part of our conversation, you’ve gotta have the folks with those specialized skill sets. Ideally, you’re able to get access to those folks on demand, which historically has not been something that’s been possible, but that’s beginning to happen.

The second thing is—so, you’ve got your people collaborating, you’ve also gotta have your technology integrated. So, for whatever security tool or processors, how do those outputs actually get to the engineers? What we need is bidirectional integrations for things like Dev ticketing systems. We need things like APIs for test results. I think collaboration at the people place and integration at the technology place—both of which we could call processes—then it’s just, it’s really a repeat of what [Audio skips].

Shimel: Mm-hmm. I mean, but wasn’t this one of the promises of DevOps? I mean, I got into DevOps because I thought it was a better way to do security, truth be told. But one of the promises of DevOps is, we’re gonna automate more, so we could go faster with more quality—two out of three ain’t bad, right? And, you know, did we lose that somewhere? Is that no longer a goal, or have we taken that as far as it goes?

Kalsey: We keep trying to solve it with technology instead of by looking at the culture and changing how people think. And you can’t think the same way and apply new technology to it and expect the technology just to magically solve all your problems.

Shimel: Brian, you agree, or Caroline?

Dawson: I absolutely 100 percent agree, and I’ll agree and then I’ll just add a point. And I know I’m being a bit nebulous about it, but there’s a human element to this. At the end of the day, it’s humans developing software for humans, right? Whether it’s the code on your electronic control module in your car or Waze on your phone or the OS, at the end of the day, the final endpoint is a human. And you know what? At least now, and I know we’re improving, there’s no way that we can ensure—and I don’t mean to accept security here, but to test the quality of the human experience, right?

And think about this—at the end of the day, knowing that the language on a screen is self-descriptive and informs me what to do. Testing the fact that, or ensuring that when I submit my credit card number it’s not vulnerable, right? Whether we’re talking API testing, whether we’re talking security testing, all of this at the end of the day boils up to customer or user experience testing.

And I guess what I’m gonna posit is that we can never entirely automate that. And where we’ve hurt ourselves and possibly made a false promise is, we’re gonna create a bunch of full stack engineers, we’re gonna get rid of, away from operations functions, testing functions, and we’re gonna use TDD and automate everything from the start, Gherkin, yadda yadda yadda. And I believe—look, that has enabled us to go faster, but our impedance mismatch, the limit we’ve hit right now is where we’ve failed to better connect people and enable them to collaborate and share knowledge. So, not just security, not just back end, not just data, but the entire customer experience is of quality. Dev can’t do that alone.

Shimel: No, sir.

Dawson: I’m sorry, Alan, did I get, did I—

Alan Shimel: No, no, that’s all—I’m saying no, sir, you’re 100 percent correct, but I know Caroline wanted to say something.

Wong: So, I do have a comment about automation. I think that DevOps has figured out how to do automation when it comes to defect discovery. I think we’re sort of good at that. I think what we’re still trying to figure out is automation when it comes to workflow and process so that, actually, again, to Brian’s point, the collaboration can happen automatically, rather than due to a human who’s involved and making it happen. How do we get the right information to the right people in the easiest, most efficient way at the right time, and how do we make it super easy for them to do their jobs?

I do think that there remains—I do think that the next evolution of automation and DevOps is gonna go from defect discovery automation, which I think we’ve gotten quite good at, to workflow and process automation.

Shimel: Fair enough. Adam, thoughts?

Kalsey: I like it. I mean, identifying what you’re automating and what’s working in that automation is important, but also identifying what’s not being automated or what in your automation is broken. And not broken from the standpoint of, it doesn’t execute or it doesn’t run to completion, but broken in, it’s not solving the problem you wanted it to solve.

Shimel: Fair enough. Guys, we have time for maybe one more question. I’m gonna pose it and throw it out there to you. Is it possible we’ve shifted too far left or that you can shift too far left and we need a little rubber band, here, going the other way, a little pendulum swing? Or you think we just scratched the surface? Caroline, you look like you wanna say something.

Wong: I think that’s a great question. I do think it’s possible to shift too far left, and the reason is because, while we, I think, are all familiar with the statistics and the studies that say if you fix something earlier in the cycle, it’s cheaper—there’s stuff that exists to the right that doesn’t exist yet on the left. Software development is, you know, to sort of a little analogy that we talked about earlier, I think it’s a dance. I think it’s a dance between people. I think software comes about as a result of the behaviors and the actions and the decisions that various folks take together. And security and quality are emergent properties. They’re not Band-Aids, they’re not vitamins, they’re not painkillers. They have been, as a result of this dance. So, there are some things for which, if you try to test it too early, it doesn’t actually exist in its final form yet. If I take my 6-year-old to the doctor and then I don’t when she’s 21, you know, might we have caught something super important as a 6-year-old that we were able to step in and address right away that would’ve been great? Yes, but as a 21-year-old—guess what? She’s gonna have different problems that were not apparent at age six.

Shimel: Spoken like a true mom and parent, right? Parents, we worry about that. Anyone else? Thoughts on that? Can we go too far left, or have we gone too far left?

Dawson: I, Alan, think it just depends. I think we’ve interpreted left different ways. I think it’s a good, catchy term. It’s an important and powerful catchy term, but as I’ve heard others discuss, we sort of overemphasized this idea of left. One, to Caroline’s point, right, we just do everything here and we’re done, good to go, right? But then this idea that we can put everything in Dev? Well, no, I think—I think, for what it’s meant to communicate, shifting left is an apt term.

I think in terms of ensuring that we’re doing everything we can as far left or as early in the process to mitigate issues later—great. Right? But we don’t just shift security left. Right? Like Caroline said, inbound, out of bound. Actually, we need to shift security everywhere. We don’t just shift quality left, we need to shift quality everywhere.

So, I think there are—we need to be ensuring that it’s almost zero trust software delivery, right? At every stage of the process, from concept where you should be sitting down with QA and the security team to capture the requirements to test, to deploy, to run, we need to be thinking about it, also.

Alan, I’d say shift left is good—shift everywhere is really what we meant to be doing, right?

Shimel: It’s better. Great.

Kalsey: Any time we engage in sloganeering, there’s gonna be people that pick the wrong things from the slogan. I mean, “Hey, we have DevOps. That means we don’t need Ops anymore!” And that’s not what it meant, and there’s people that have taken the wrong cues from shift left, and instead of shifting the processes left, they shifted the tasks onto the people who were left already, and that’s not gonna work.

Shimel: Absolutely. Hey, Mitchell, you wanna tie a bow on this?

Ashley: I’ll put a bow on this. You know, there are admittedly many, many things that I don’t know, but one of the things that I do know is that, if a developer says it’s tested, it’s ready and a QA person says it’s tested and it’s ready, five out of six dentists will choose the QA person. I will, too.

So, there’s something there. There’s a reason why we don’t have DevQA or DevTest. We don’t talk about it that way, we talk about DevOps or DevSecOps. QA is extremely valid and necessary, and we can’t live without it, and I think we need to invest in helping, elevating how we test and how we think about our software applications to test in. So, those dentists will be happy.

Shimel: And Crest, the Crest test for QA. Thanks very much, Mitchell. Caroline, Brian, Adam—thanks for joining us on this episode of DevOps Unbound. Great conversation. For our audience out there, do check it out. We have another episode coming up in—well, not next week, the week after, and then I think we have a roundtable the two weeks after that.

But for now, though, we’re gonna wrap. Many thanks again to Tricentis for sponsoring DevOps Unbound. Thank you for watching, thanks to our panel. This is Alan Shimel, we’re out.

[End of Audio]

August 4, 2021
EP 14: What Have We Learned?

The past year-plus has brought disruption and opportunity to businesses and software teams all over the world. Change is inevitable, but 2020 was a year of massive change: businesses pivoted, supply chains were disrupted, customer expectations shifted to contactless services, digital transformation projects accelerated, new digital experiences were created, the move to the cloud jumped into high gear and our ability to create and deliver software was elevated to an essential capability for business.

It’s time to pause and ask ourselves, what did we learn from all of this disruption and change? What lessons can we carry forward? How do those experiences inform how we prepare for 2021 and beyond?

Cornelia Davis from Weaveworks, Kurt Chase from Tricentis and Sanjeev Sharma from Truist join hosts Alan Shimel and Mitch Ashley in this episode of DevOps Unbound to discuss lessons learned, how to overcome the obstacles we’ve faced over the past year and how to improve business processes and make better decisions. The video is below, followed by a transcript of the conversation.

DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.

<span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_start"></span>

Alan Shimel: Hey, everyone, this is Alan Shimel, and you’re watching DevOps Unbound. For those of you watching for the first time, DevOps Unbound is a biweekly video series where we tackle interesting topics related to DevOps. Once every four to six weeks, we do a live roundtable with a live audience questions and live audience participation, and we invite you to our next one. You can find that scheduling at DevOpsUnbound.com.

Today, though, is a session just with our panel, and what a panel it is. Let me introduce you to them. First off, let me introduce our friend, Sanjeev Sharma of Truist. Sanjeev, maybe you wanna introduce yourself to the audience.

Sanjeev Sharma: Sure. Thanks, Alan. First of all, thanks. Great to see you all. Hopefully, we will be able to meet in person soon, but thanks for having me on. I am the Head of Automation and Platform Engineering at Truist Bank. Truist, for those of you who might not know, is the new bank that was created last year by the merger of BB&T and SunTrust. I have various hats I wear at Truist, but my background is mostly in the DevOps and cloudware adoption space.

Shimel: Excellent. Thank you, Sanjeev, and welcome. Sanjeev, I should mention, is also the author of The DevOps Adoption Playbook.

Next, we’re really, really pleased and honored to have Cornelia Davis with us. Cornelia is, of course, a pretty well-known thought leader and leader at the DevOps space, but is also at Weaveworks. Cornelia, welcome. If you wanna give a little background?

Cornelia Davis: Sure, yeah. Thank you so much, Alan. It’s so great to be on again. Yeah, I am the CTO at Weaveworks, and you might know Weaveworks as the company that coined the term GitOps, that has been kind of driving that movement. And GitOps is just an element of a broader category, which is really developer platforms, which is what we’re talking about many, many times when we talk about DevOps.

And so, prior to Weaveworks, I’ve been here for about a year and a half, I was at Pivotal, where I worked on Cloud Foundry and Kubernetes based developer platforms, you know, platform teams for enterprises for quite some time. And I’m also the author of a book called Cloud Native Patterns, which is a book aimed at architects and software developers, helping them learn how to build software for this highly changing, highly distributed—constantly changing, highly distributed environment that is the cloud.

Shimel: Absolutely, thank you. It’s a pleasure to have you here. Last but not least on our panel today is our friend Kurt Chase with Tricentis. I should mention Tricentis is a sponsor of DevOps Unbound, and thank you, Tricentis, for sponsoring. But Kurt, why don’t you give them your background?

Chase: Yeah, thanks, Alan. Great to be here today with Cornelia and Sanjeev. Nice to meet both of you. I’m currently running Release Management and Engineering Services at Tricentis, which, Engineering Services is a group that provides cross cutting services primarily for internal engineering, but effectively for the whole corporation as well.

Prior to Tricentis, I spent years at Splunk, again, supporting the engineering function, all the CI/CD builds and releases and things like that. And then prior to that, I spent almost two decades at Autodesk, producing the AutoCAD, multiple releases of the AutoCAD software product.

So, again, it’s great to be here today with all of you. Excited to talk about some of the things we’ve learned over the last year, year and a half—and what a year it’s been, holy cow. [Laughter]

Shimel: Yes, it has. Then let me finally introduce my co-host here at DevOps Unbound, my good friend and partner, Mitchel Ashley. Mitch, why don’t you give a little background?

Mitch Ashley: Thanks, Alan, and of course, I’m excited to have our panel—thank you for joining us, all of you, today.

I’m Mitch Ashley, CEO of Accelerated Strategies, an analyst firm that focuses on DevOps and cloud native, cyber security, digital transformation. I also work with Alan as part of MediaOps, the folks that put on this event, with the CTO of that organization. So, it’s a lot of DevOps practicing and observing what others are doing, so I’m excited to hear what Sanjeev and Cornelia and Kurt have to say, too.

Shimel: Alright. Thank you, Mitchell. And as I mentioned, DevOps Unbound is sponsored by Tricentis, so—many thanks to them.

Today’s episode, folks, as I mentioned off camera, I don’t wanna jinx anyone or anything, but at least here in the U.S., we seem to, at least we hope that we’re seeing this whole COVID life sort of starting to recede in our rearview mirror. We are returning to normal or a new normal, some may say recklessly too quickly, or some may say not quickly enough, but certainly, the return to normal is on, and accelerating.

But, you know, there have been many, many lessons, many, many experiences, some painful—extremely painful—from a business and a humanity point of view over this last 14, 15 months or whatever it’s been. There’s also been a tremendous amount of success, a tremendous amount of acceleration, a tremendous amount of, “Jeez, this really—this worked. We didn’t know if it would work, but it worked,” probably the Internet being the first thing, right? Where would we be with this without an Internet?

But can we—let’s take a moment, and let’s, as I say, hope and pray that this is in our rearview mirror or starting to move to the rearview mirror. What are the lessons? What can we learn, what will we take forward from this? And, you know, I don’t wanna boil the ocean, because I think there’s a lot, but if I asked each of you to give me three things that you really, you think—not just you personally or your organization, but the tech industry has learned, the business world has learned, perhaps society in general, humanity has learned, what would you say some of those things are?

I don’t wanna put anyone on the spot, but Kurt, you always have something to say. I introduced you last, we’ll start with you.

Chase: Sure. I’ll jump right in, and one thing that’s amazing to me, not so much from the technology or engineering or R&D side, let’s talk about business, here—the other aspects of the business that were able to adapt to remote work, that, to me, was pretty incredible.

When COVID really took hold, I wasn’t too worried about software engineers and folks producing software working from home, doing their coding, testing, and all that. The other functions—finance, legal, things that are required to ship your products—seeing those go remote at the pace they did, that was pretty incredible. I think they definitely took advantage of some of the learnings from their individual engineering teams. But to me, that’s one thing that sticks out right away is the speed at which businesses adapted to working remotely and the success they achieved across the functions in the business. That was incredible to me.

Shimel: Agreed. I mean, the speed of adoption was something. Cornelia, did you wanna say something?

Davis: Yeah, sorry, I was muted there. I did wanna chime in, because I agree, and one of the ones that really knocked my socks off was how quickly call centers went remote.

Chase: Yeah. [Laughter]

Davis: Because that isn’t something I would’ve expected. I would’ve expected them to still be kind of really tied into their telephony services and, you know, the phones that are on their desks.

But I spoke with many, many people, you know, various customer service people from the banks and from, you know, the Amazon.com—well, there’s not that much of a call in center there, but from all of these different organizations where, when you chatted with these folks, they were like, “Yep.” Within a week, they were able to move everything, move everybody remote in these traditionally really physical call center based systems. Which means that there had to be some preparation. I don’t know if it was intentional, but there had to be something that allowed them to flip that switch so quickly. Is it just forwarding phones? Is it just VPNs? I don’t know.

Shimel: No, no, it was—so, I have some info on that, if I can. I interviewed a fellow down here in Florida, I think he was out of Tampa, and he operated multiple call centers, like in Costa Rica, which is actually a call center capital, believe it or not, that’s a big business down there. It turned out that the software they used, right, people in call centers, they don’t just—Kurt just doesn’t call someone randomly. Everyone gets assigned their calls or their inbound when a call comes in and gets assigned.

The software they used within the call center itself worked just as well, the person didn’t have to be in the desk next to you, he could’ve been anywhere, it turns out, or she could’ve been anywhere. And so, it worked just as well in a remote thing. So, they still used the software they were using that tracked how many calls they did, who they spoke to, the notes, putting it into the central database. Yes, they had to be cloud-based, right, because there was no one in the call center to kinda run that infrastructure. But the software, it turned out, worked remotely as well as it worked on everyone’s desk in the call center. It was an amazing thing.

And so, what they found—again, and this is gonna be, I think, something that’s gonna be a recurring theme—who needs the physical call center? They don’t need that infrastructure anymore. They could have people working at home just as well.

And so, the more advanced ones—and you’ve probably seen this yourselves—you call someone, they may be very well at home, but they’re giving you the same mediocre service that you get when you call the call center itself, right? [Laughter] But yeah, no—

Chase: And that’s the point. I think with a business, it’s applicable across functions there, it’s amazing. And business learned that they can still be extremely successful, even if everyone’s remote. And having that pool of remote talent—that changes everything. [Cross talk]

Sharma: I think a couple of things to be noted there, right? First of all—sorry, Cornelia, I think I spoke over you. You know, one is, the pendulum also needs to swing back from that extreme.

Chase: That’s the challenge, yeah. [Laughter]

Sharma: Right? Because, while we have people who are able to work from home because the technology is there, from an equity perspective, one has to look at people who cannot work from home who are being forced to.

Chase: Yeah.

Sharma: And we experienced that when we opened up our offices. We are just beginning to open up our offices. June 1st was the first day folks would come back. And I was surprised to see some people had jumped at the opportunity and said, “I want to go back.”

Chase: Yeah.

Sharma: Then when you look under the hood and you talk to them, you realize, you know, they’ve been struggling. They’ve been operating out of a tiny closet, right, because they don’t have the space or because they have toddlers or they’ve been struggling with Internet issues, and it is worse overseas than it is here in the U.S.

So, I think the pendulum needs to swing back to a more normal position, but hopefully, it won’t swing all the way back to where they say everybody has to be on site, on premises, right? So, I think that is definitely, some balance has to happen to recognize—some people do need a workplace, where they can work comfortably.

Chase: You’re absolutely right.

Davis: Yeah.

Chase: I know a story of one individual who was in a one bedroom apartment in San Francisco and after three weeks, he couldn’t handle it. It’s like—I don’t have no space.

Davis: Yeah.

Shimel: It happens. Cornelia?

Davis: And, you know, honestly, I have space, and I have a wonderful office and I’m super comfortable, and I’m also desperate to get back into the office.

Chase: Yeah.

Davis: Because—and I love, Sanjeev, that you called out the equity thing, because that’s something that is so painfully obvious in the whole pandemic is that this has only broadened the inequities that we have across society, and we need to absolutely focus on that.

But then the other lesson, you know, with me, I’m extremely privileged, I live this super comfortable, I never really had to worry about catching COVID, I could just choose to stay home. But humans need humans and, you know, we’re all coming to you on your two-dimensional screen. I am super fortunate that last week, I had the opportunity to have my first face to face with a couple of colleagues in over a year, and it was rejuvenating. I’ve been saying, it’s a 3D work world again instead of a 2D work world.

Shimel: Yeah.

Davis: And so, I mean, humans need humans. But you’re right, Sanjeev, I think that the even bigger thing is that we really, really need to look at the inequities. And in the beginning, in the early parts of the pandemic, without getting too political here, we had all of this narrative around these people who we considered essential workers who were kind of the forgotten, like, taken for granted folks, and they were put on a pedestal, and that, unfortunately, didn’t last that long.

Chase: Yep.

Sharma: Yeah.

Davis: And so, we need to bring that back.

Shimel: And they weren’t paid like they were that essential, either, but that’s [Cross talk]—yeah, it was a pretty low pedestal. But let me, you know what, you bring two things—I’m sorry, Mitch, I just wanna make two quick points.

Number one, when we talk about the inequality, let’s be really clear. This has affected women and mothers especially, more than men, right?

Ashley: Mm-hmm.

Davis: Absolutely.

Shimel: Mothers have had—because not only have they had to work from home, but they’ve had to take care of the kids who are not in school. And it’s very hard being a teacher, a home care provider, and doing your work at work. And so, how many women have had to work out of the workforce because they had to take care of their kids and their kids are home and they didn’t have an option, there was no child care for them. So, I think—

Ashley: And the challenges of re-entering the workforce.

Shimel: Right, and now they’ve gotta re-enter after, you know, months off, and that’s never easy. People wanna know, “Well, why the gap in your résumé?” Well, there’s a gap in my résumé because I have this beautiful child that I’m raising, you know, and how dare you? That’s number one.

Number two, it is a case of the rich get richer, right? Companies that, not only individuals as you say, Cornelia, but companies that were able to leverage this. Man, they accelerated, but they accelerated at the expense of their competitors who couldn’t, right? Think about all the open retail shops in the malls near you and the strips, right?

Davis: And this is a place where I’d love to pull DevOps into the conversation, because we are, of course, as a part of DevOps Unbound. One of the things that I thought was impressive—and I’ve worked with some of these companies. In, you know, my previous life at Pivotal, I worked with The Home Depot, for example. When you had The Home Depot and you had these other organizations that, in a matter of weeks, updated their software so that you could do things like curbside pickup? That right there is DevOps in action.

Shimel: Yep.

Davis: It is the ability to release software, get feedback, release software, and there was no six-week testing cycle that we used to do, or eight-week testing cycle.

Shimel: Mm-hmm.

Davis: It was like, “You know what? We are gonna be testing in production, because we didn’t have time to test this for six weeks.” So, it brings in all of these tools that Kurt, I’m sure, you have day to day experience with, which is these things that allow us to put safety nets in place to be able to release all the time. I mean, that is something where the people who were along their DevOps journey already had a leg up on those that didn’t.

Ashley: You know, Cornelia, I had that exact experience where I couldn’t get a computer, you couldn’t get one shipped to you, they just weren’t there, so I had to build a computer for some video work, and did it solely from parts from Best Buy, by curbside pickup, two Best Buys in town.

Davis: Yeah.

Ashley: So, it literally was, like, a week later, all of a sudden, I could do this, and I was pretty shocked by it.

One of the things I wanna say, though, about what we’re discussing is, the people dimension of it is extremely important. I think there’s a whole other set of lessons for the business. I think we’ve pressed reset on a lot of assumptions about how businesses have to operate, that you have to be in an office. And essentially, we’ve kinda globalized the business by just taking the location out of it now. And yes, there are privileged and underserved and people who aren’t privileged. And those inequities probably are gonna get bigger, not smaller, because I don’t think we’re gonna all slap back into the office immediately.

But what the businesses learned is that lesson you just talked about with Home Depot and you with Best Buy is—you know what? We can decide we’re gonna pick Team or Slack in two days instead of two years. We can get a software release out in a few days if we have to. Talk about constraint based management—you know, we now have this constraint and everything else comes off.

So, I think what it’s done overall, though, is for DevOps and for all of us is, it’s moved software from maybe second or third to number one. Our ability to get software out the door that’s effective, meets what we need it to do, but that is a strategic advantage, maybe even a survival advantage of what we’ve seen happen in this past year.

Chase: Yeah. I think some of it is, you know, businesses like that were forced to do it or die, basically, you know? And I 100 percent agree with Cornelia that I think they look to DevOps and some of the practices that they’re all used to. You know, the collaboration we need across functions to move forward and all those things are key, some of the foundations of automation, you know? That has been key in the DevOps space, and I think they learned, they need to take advantage of automation to do these things as well.

So, yeah, it’s been fascinating to see that. And that was a point I brought up to see how businesses transformed and our world changed. And I think it’s opened up endless opportunities for us moving forward. And to Sanjeev’s point, I hope when that pendulum swings back, corporations take a look, that this is no longer a corporate decision where people work, it has to be more joint. It has to be more personal, because every situation’s different, and if you wanna maximize all your programs in HR, diversity and all those things, you have to take that into account. I don’t think you can do a one size fits all any longer.

I still wonder if all those big office buildings, I worked in downtown San Francisco for many, many years—are we gonna fill those up again, ever? I really wonder now if that’ll ever happen. I could see it not happening, quite honestly.

Ashley: I’m curious, Sanjeev, you’ve been in the middle of this bringing two companies together with this new bank in the middle of all of this. What in God’s name was that like? [Laughter] What are the additional challenges you’ve had for that, or maybe what opportunities has it created?

Sharma: Yeah, so, I think it—you know, I joined in the middle of the merger, right, so we were in the middle of it when I joined. So, what I can tell you in the 10 months I’ve been there, we have shifted a lot of the priorities because of the pandemic, right? A lot of things which we would have done later on, we did first. We were able to do things.

But the flexibility has definitely opened up. I can tell you, between the time I joined and between now, the ability to hire people who are not local to one of our offices has gone through the roof. It used to be an exception, now it’s the norm, right? The last three people I hired, two are in the West Coast and one is in Minneapolis, right, so Midwest. It doesn’t matter anymore, right? The same way there are—the operation has opened up a lot.

And when I look at it broadly, outside of who I’m working with, when I’m talking to other companies where I have friends or colleagues, they are seeing more offshoring happen, right? Because they’re going, “Why do I need to hire somebody in L.A. if they’re never going to see the person ever anyway, does it matter whether he’s in Bangor or Bangalore, right? Let’s put the person, let’s hire the right person at the right place.” So, I think a lot of that has opened up. It has, of course, shifted some of the salaries, also.

So, you know, a lot of things are going on, interesting dynamics are happening, and we’ll see how that all pans out. But from a perspective of, you know, doing the transformation, it is hard when you can’t have people in the same room on a white board and things which could have just drawn on a white board and knocked out in a 1 hour or 90 minute working session, now it’s like 15, you know, Zoom views, WebEx or Teams calls, which could have been, and it’s like, “Where is that white board we made up? Was it on Miro or was it on Mural or was it on this or was it on that? You know, and who can edit it, who can’t edit it?” All those—you know, dang it, if we just had a white board and all of us in the same room, we could have knocked this out in half a day.

So, it is difficult to do certain things remotely. I think there has to be a balance, which needs to happen. And yeah, during a merger just made it more complex, because we are going through change every day, right? People’s networks are changing, people’s—how we are doing things are changing, we are updating stuff. It’s been a very interesting few months and we’re not done yet. You know, the merger isn’t over. We’ve still got a long way to go. So, it’ll be more fun going forward, I think.

Shimel: I think it brings up the issue, though, of how much of this sticks, right? Because, I mean, look what Sanjeev just described. We’ve all had similar experiences, I’m sure, hiring people from all over the world—what difference does it make? They don’t have to come in the office.

Does that stick? Because if it sticks, there are profound implications for society in general. Why live in San Francisco unless you love the view of the Bay. It’s a beautiful bay, right, the bridge is nice, but pay $4,000.00 for a closet-sized one bedroom apartment? Maybe not. I mean, on the other hand, I live in South Florida. Housing market is insane because of people moving here from New York and California and spending a fortune—Colorado, right?

So, does this—that’s one example. Does the ability to work from anywhere, it makes no difference, is that something that sticks around, or do we, as Sanjeev said in the beginning, see a pendulum start to swing back? What do you think?

Chase: Well, for me, like I said, I think elements of it have to exist and remain. I think there needs to be a balance, as Sanjeev said. I no longer think it’ll entirely be a corporate decision where employees are located. I think for some roles and groups in organizations, there needs to be a deeper conversation on where is the best talent, what’s the best way to use them? You know, having people sit—just the commute. The one thing I don’t miss is the commute. Oh, my God, you know, those three hours a day commuting, you know, that, to me, we have to somehow retain that benefit, you know, from working remotely somehow. I’m not sure how.

But that’s a very good question, Alan, I don’t know. It’s like, will those office buildings ever be full again? I don’t know. I’m not sure.

Sharma: On that point, I’ve been remote since 2013. In 2013, I was at IBM. I moved to a global role, and since then, I haven’t had an office. I’ve worked out of this room since then. But I used to travel 150 to 175,000 miles a year.

Chase: Got it, yeah.

Sharma: Right? So, for me, the office was a place to go collaborate, whether it was my office or my company, IBM—

Shimel: And for all of us.

Sharma: – right, or a client’s office, it was not a place to go and have a desk, right? My desk is here, right, my books are here—you know what I mean, right?

Chase: Yeah.

Sharma: I think we should, I would like to see us go to that model where offices become more of a collaboration space rather than a place you have to show up and work alone in a cubicle. What’s the point of that?

Ashley: I agree.

Sharma: Unless somebody needs it, right? That’s different if they don’t have space at home and the need a cubicle to work in—fine, put them in a WeWork next to where they live, right, close to where they live rather than have them commute to the other, you know, 45 minutes one way.

I think that’s the model we should head towards.

Chase: I agree.

Sharma: And, you know, the tools to enable that are there, we’ve proved it. We’ve proved, during the lock down, during the pandemic that the tools are there, but I hope it goes that way. I hope it doesn’t go back to, “You have to show up in this cubicle” even if you are, you know, like the guy from Office Space protecting the stapler and never talking to anybody.

Ashley: The butts in seats model, right? [Laughter]

Chase: We’ve seen some large organizations [Cross talk] make those calls already and say you can be remote forever now, you know, things like that. So, I think that’ll become a bit more selective, but I still hope, when that pendulum does swing back, that the employee has a say, you know, and his voice is a little bit taken more seriously.

Shimel: Or her voice.

Davis: Or hers.

Chase: Or hers—or hers, [Cross talk].

Davis: Exactly, that’s where I’d like to chime—I’d like to chime in there, because Alan teed this up earlier and then, because you had two things, Alan, we followed thread number two.

I wanna come back to thread number one, which is that this has affected women in the workforce, and it’s not just us technologists, but it’s a broad spectrum, and it’s probably even more so in other working arrangements. And so, I love what you’re saying there, Kurt, about it being a partnership between the corporation and the employee and determining what’s best for them. And then Sanjeev, you, I think, just teed up—is that the survival answer for WeWork is that part of that negotiation between the corporation and the employee, she or he, includes having the ability to move into a workplace that is close to your office but allows you that space that you need, for example, not to be on call for child care.

Of course, the other thing I can’t help but say—and I’m very fortunate, I have one son, he’s 26, I’m not tying shoes or going to PTSA meetings anymore—

Shimel: I hope not. [Laughter]

Davis: – so, I wasn’t affected in that way. Yeah, [Laughter] I wasn’t affected in that way in the last year, but I was a working mother and I worked from home that whole time. And I just wanna remind everyone, men and women, that you’re right, women take on that burden, but it doesn’t have to be that way, fathers.

Shimel: Amen.

Chase: Mm-hmm.

Cornelia Davis: So, please pitch in even more than you think, just because it’s been kind of the standard operating procedure. If nothing else, I hope the last year has maybe, in some cases, resulted in a, hey, a need-based way of rebalancing responsibilities. That is something that every single individual, male or female, can participate in kind of these societal expectations and challenging those.

Ashley: I think something happens—

Shimel: You know what? Oh, go ahead, Mitch, I’m sorry.

Mitch Ashley: When things open up, when opportunity happens and things make that shift, it’s different when there’s opportunity there, but when things have shifted and made a shift, they don’t snap back to the old way automatically. It transforms sort of the landscape going forward, and there’s so many things about this—and you mentioned women in the workplace, we also have people of color—and we think about the emphasis of diversity that’s happened over the last year and a half, which is fantastic. And we see so much more bringing to the forefront of women in leadership positions, people of color being involved in security more, whatever it might be. You know, I don’t think that that changes.

So, hopefully, those people who now are getting to participate in a bigger, broader way or have more opportunity to do that, (a) I don’t think this is gonna go away, (b) that’s gonna change things for us, right? You know, maybe, Cornelia—[Laughter] here, I’m speaking about what I don’t know about—but maybe more women in leadership positions make some of those, it’s assumed that the woman at home does the child care or whatever is part of our societal transformation, that it isn’t necessarily an assumption. Hopefully we can get past that and remove that barrier for people so there is more opportunity.

Shimel: So, Mitch, on that front, though, here’s the deal. Number one, I think the move to diversity is not necessarily COVID related, I think it’s just an idea that its time had come, it has been a long time coming, number one.

Ashley: I totally agree.

Shimel: Number two, I think one of the positive things that happened is, it wasn’t just the woman stuck at home taking care of the kid. She was stuck at home taking care of the kid while the husband was trying to carve out a desk space in his bedroom somewhere to try to do a meeting or something, right?

But what often happened, at least among my friends—look, my kids, also, Cornelia, they went away to college even though their college was online, just to get out of the house from their mother and father who wouldn’t let them breathe. But most—like, people I know whose kids were younger, the mother and father were home. They had to split up, delegate duties. You did the wash, I cook the meals, you clean the table, I took the kids here, you took the kids there or entertained them now because you had work to do, I had work to do.

In a best case scenario, it became very much a shared responsibility. Of course, not every scenario or every instance has that best case outcome. There’s a lot of people [Cross talk], you know?

Ashley: I think it’s been a journey to share. It may not be quite as, like, “Okay,” [Cross talk].

Davis: Yeah, yeah.

Shimel: Well, that goes, this is—

Davis: I mean, you’ve painted a fairly ideal picture.

Shimel: That is an ideal picture, you’re right.

Davis: Yeah. You know, this conversation is making me wonder, and I don’t have these stats, but I wonder if anybody else has heard the stats—you’re right, absolutely, that this shift towards more underrepresented folks in leadership positions, women, people of color, and so on, that absolutely started before COVID.

I wonder how COVID has affected that. Have the numbers accelerated, have they decelerated? I think—I suspect the latter, and I’m pretty much an optimist, but I’m also a bit of a realist, and I wonder whether the last year has slowed down progress when it comes to diversity at the leadership level. I don’t know. It’s just a gut feeling. I’d love to see some metrics on that.

Sharma: I can tell you, Cornelia, in IT roles, it has always been very difficult to hire people from minorities, right, underrepresented minorities.

Davis: Yeah, of course.

Sharma: It always has been. I personally believe it has gotten worse, because if you look at especially leadership roles, senior roles—we’re not talking fresh college grads—most of those roles, no matter what anybody says, happen through referrals and connections, through your network.

Davis: That’s right.

Sharma: And that network stalled. That network stalled during the pandemic, right?

Shimel: Yeah, networking, yeah.

Sharma: Right? It is as big as it was then or grew slightly, right, that networking opportunities were gone.

Davis: Yep.

Sharma: No Meetups, no conferences, no lunches, no power breakfasts, no women in technology meet ups—nothing. The network stalled—you only knew the people you knew, right?

Davis: Yep.

Sharma: And I think that hurt, bad. I find when I’m trying to hire, when my peers are trying to hire, we are struggling more to have—see, you can’t hire when you don’t receive the résumés. When there are no résumés coming in, how do you hire, right? And I’m struggling, and if anybody has, if either one of you could—any one of you has any suggestions on how to fix that, I’m all in. Please, I’m all ears.

Davis: I think—

Chase: Yeah, I was gonna say, you pointed out an anti-pattern to what I was gonna say, Sanjeev. I was gonna say, at least with COVID and all the remote work, our talent pool has opened up worldwide, but you’re saying that hasn’t helped in that regard.

It has for me with some of the roles I’m hiring. Now granted, they’re not Director, Senior Director, VP level roles yet. But for the other roles, having the world as my talent, you know, the talent pool versus the United States, that makes it [Cross talk].

Sharma: Well, I look at it a bit differently. I look at it a bit differently, Kurt. You know, hiring—how do I, I wanna make sure I say this properly. Hiring an Indian in India is not hiring a minority.

Chase: Yeah, true. Yeah, true.

Sharma: Hiring an Egyptian in Egypt is not hiring a minority. Hiring a Turkish person in Turkey is not a minority, right?

Shimel: Right.

Sharma: They have to be a minority in where you’re hiring them, that’s the only way it works.

Shimel: Agreed.

Davis: Yeah, I think, [Cross talk] I wanted to comment on your remarks, Sanjeev. I think that you’re spot on. Because there were opportunities, and some of the techniques that we have used in corporations to source diversity candidates or underrepresented candidates, those things went away. So, being able to go to a historically black university or go have your recruiter show up at Women Who Code meet ups, those went away, and those aren’t the same.

Sure, Women Who Code still had meet ups line, but it’s not the same as walking around during the social hour and saying—or, I’ve been at many meet ups where, before they start the program, they say, “Okay, who’s hiring?” And everybody stands up and then, “Who’s looking?” Everybody stands up. “Alright, you folks, talk.” That was a very organic thing that was not just so easy to say, “How do you do that in an online setting?”

You’re right, it’s—I hadn’t even thought of that, but that whole network is the same now as it was a year ago is remarkable, especially when it comes to leadership.

Sharma: Exactly. That’s why hiring from a historically black university, we can still do that today, because the recruiters still know the counselors there, they still know the placement office or whoever does—you know, those networks are still there. It’s just that the network hasn’t grown. The higher you go, right, the air gets, you know, rarer, right, and those jobs are fewer and you are not screening 100 résumés to do that. You’re relying on the network more than often, no matter what anybody else says, right?

And that network stalled. I mean, who have we met at a conference who we didn’t know before? Now, when I go to a meet up and I see somebody I know—

Davis: Yeah, but I think what you’re pointing out—

Sanjeev Sharma: – I am IM-ing that person, not some stranger who I’ve never met, right? Unless there’s something really, some reason to do that. Then—no, no, it is the opposite, it has really hurt. I think we need to fix this.

Davis: Yeah.

Sharma: We need to put this [Cross talk] the moment we come back.

Davis: Yeah, but I wanna suggest that—yeah, but I wanna challenge that assumption, which is that I hire leadership from within my network. Maybe that right there is the thing that we need to work on.

Sharma: Well, I’m with you. We need to work on it. What I’m saying is—that’s what I was saying. I was saying that what we tend to go, tend to—we tend to grow our network to find leadership role people, right? People in leadership.

Davis: Yeah.

Sharma: When you can’t grow your network, you’re reliant on what’s already in your network, right? I mean, otherwise, you’re scouring LinkedIn or some other website or pinging people in your network asking for referrals.

Davis: Yeah.

Sharma: But that network stalled. That network didn’t grow. I can’t point to 50 people I met in the last year who were cool and interesting who I did not know before who were underrepresented minorities have I would have met, had I gone to Reinvent or IBM Think or, you know, DevOps World or KubeCon or whatever. All the people I would have met, I didn’t meet them.

Shimel: So, that’s a whole other thing. Virtual events, right?

Ashley: Mm-hmm.

Shimel: We’ve all spent significant portions of our lives at conferences. Cornelia, that’s where I usually see you is at conferences—Sanjeev, as well. Virtual events are cool. We put on one a month here at MediaOps. Gene Kim does his DevOps Enterprise Summit virtually, people even paid money to attend it.

And all of the virtual events, they try to do a social thing to it. It’s just not very good, frankly. It just—it’s just not the same as being in the hallway or grabbing a drink at the bar or a soda or whatever, you don’t have to be drinking. But, you know, just not—it’s not that, someone said it before, face to face, person to person. I think it was Cornelia, you recently had an in person. It’s not the same, it’s not the same.

I mean, so, different than the three of you, Mitchell and I have had a chance to meet hundreds of new people this year. I do three to four to five of these interviews a day. My day starts with talking to people in Asia or Israel and goes to Europe and through the Americas, all the way to the West Coast. And what I’ve found is that I’ve become a resource for people like you, Sanjeev, who say, “Hey, I’m looking for someone to head up cloud or DevOps or testing” or what have you. And I go through, who have I interviewed recently?

But that’s not the way it’s supposed to be. And the question is—and we’re getting near on time—I wanna bring it back to DevOps and software, right? And, you know, what—

Chase: I think you touched upon it from the beginning, Alan, it’s the Internet. You know, the fact that our technology infrastructure was able to shoulder the burden, that was pretty incredible to me, too. You know, I expected Zoom to go out for weeks, if you will. But I think it demonstrated that the technology infrastructure we have is pretty solid. You know, it allowed us to scale up pretty quickly, pretty massively and, you know, there were outages here and there, but I would’ve expected a lot more.

Shimel: Minimal—minimal. Even counting all the DDoS attacks and the ransomware and all of that nonsense, it was still minimal.

So, we had that infrastructure, but we also had this incredible capability to accelerate, to automate that DevOps kind of has been teaching us for seven, eight, nine years or more now, right? And that really helped. Curbside pickup and the—it’s not just curbside pickup, it’s the back end, obviously, right, that allows you to pull into that parking space and hit the button and say, “I’m here, and this is the car I’m in,” right? That kinda stuff, how many companies and jobs were saved by that kind of capability.

Chase: The other thing for me internally is, I think it also opened up the ability to question everything, just because—at all levels, you know? Whether you’re designing a new feature, adding something to a product, designing a new process—is that the right thing, just because we’ve done it like that? I mean, really question it, because I think it, I mentioned this before. I think the opportunities ________ are endless now, really.

We haven’t even started to take advantage of the things we’ve learned. I believe we’re starting to now, as Sanjeev mentioned, the pendulum coming back. Part of that is some massive learnings that we’ve also had forced upon us, if you will. But I think we’re gonna come through it even stronger and better, for sure.

Davis: Yeah.

Sharma: I think the—

Davis: You know, I wanna, I’d love to chime in on—

Shimel: Go ahead, Cornelia.

Davis: – yeah, I’d love to chime in on that scale. You used the word scale and these systems were able to scale and handle just a tremendous surge in traffic.

Shimel: Oh, yeah.

Davis: And when you said that, it took me immediately back to a DevOps Enterprise Summit where Nicole Forsgren and Jez Humble—and those of you who don’t know them, you should know them.

Chase: Yeah, we know them.

Davis: State of the DevOps Report, State of the DevOps Report, the book Accelerate. I remember this moment where they were on stage, and Jez Humble, in his very humble way, hand in his pockets, says, “Turns out, architecture matters.”

Shimel: Yeah. Right.

Davis: And so, it’s having these systems that are designed to scale, and we find so many enterprises that have said, “Oh, well, you know what? Google scale, Amazon scale, that’s not a problem that I need to face.” Well, surprise—

Chase: Yeah, it is.

Shimel: It is.

Davis: – maybe it is. And so, architecture matters, and so I can’t help, as a software architect, coming back to that and saying, “Yep, there ya go.” [Laughter]

Shimel: Yep, [Cross talk].

Sharma: And I think we’ve seen companies who were—there’s two ways to look at it. One way is to look at it from a scale perspective. The other way is to look at it from a resilience and reliability perspective.

Davis: Right, yeah.

Sharma: Right? Which are two aspects of the same thing. It’s all architecture. And I think companies realize very early that reliability and resilience are critical—resilience to be able to make changes rapidly, right? Today, I have, I don’t have curbside pickup, tomorrow I have. Today, I had 10 people doing this, tomorrow I’m gonna have 200 people doing this.

Ashley: Right.

Sharma: Today, I had a teller sitting in front of her terminal, so the user interface didn’t need to be that cool. You know, the things could hang and the teller could, you know, make sense, make small talk while the system is catching up. Well, nobody’s accepting that on the mobile app when they’re directly dealing with the website or the mobile.

So, I think that whole shift on resilience and reliability are real, are important, and all that goes back to your point, Cornelia, to architecture. That was front and center to people, and I think [Cross talk]—

Chase: Yeah. And I think along with that, Sanjeev, is security. I think security, this really exposed—we have to do more in security. We have to do more, you know?

Ashley: We have to rethink security.

Chase: I think that’s one of the chief learnings.

Shimel: No doubt about that. Guys, I hate to do this to you, but I need to pull the plug on this. We already have people in my waiting room for our next interview.

What a great panel this is. We’re gonna have to rerun this again and go to a part two, because there’s things I wanna talk to about this as well. I spoke to a guy today who started a new, he calls it the mobile cloud for gaming, it’s called Now.gg. Because their research is that our kids, you know, these digital natives, if they don’t get a response in half a second, they’re gone, they’re onto the next thing. And so, what does that mean for your banking apps? What does it mean for we’re doing?

But we need to end this. Sanjeev, Cornelia, Kurt—man, thank you, guys, so much. Thank you, all, so much for making this an amazing panel. Thank all of our viewers out here for watching this; we hope you enjoyed it. Mitchell, I’m gonna let you take it home.

Ashley: Well, I think—what a fascinating group and wonderful things to say. I think the biggest learning for me—and I really appreciate Cornelia and Sanjeev talking about the lost opportunity we had through networking to help people come together and expand their networks. So, let’s invest in that going forward. We need to rebuild those and make that happen. So, I appreciate—I really wanna thank you for helping me understand that better.

Shimel: Absolutely. Many thanks to Tricentis for sponsoring this episode of DevOps Unbound. We’ll be back in two weeks with a fresh episode, a new panel, a new topic, but hopefully it’s just as interesting a conversation.

This is Alan Shimel for MediaOps. Until next time, be safe and be well.

[End of Audio]

August 3, 2021
EP 13: Observability and Performance Testing

In this episode of DevOps Unbound, Alan Shimel is joined by Christine Yen of Honeycomb, Paul Bruce of Tricentis and Mitch Ashley from ASG/MediaOps to discuss how to effectively test the performance of your applications and how bringing observability into testing environments helps DevOps teams identify and resolve issues early in the SDLC. The video is below, followed by a transcript of the conversation.

Alan Shimel: Hey, everyone. I’m Alan Shimel, CEO of MediaOps, Editor-in-chief of devopsy.kinsta.cloud, Security Boulevard, Container Journal, and you’re watching DevOps Unbound. DevOps Unbound is a biweekly video series where we have some of the leading lights in our community talk about topics of interest around DevOps. We’re lucky enough to be sponsored, DevOps Unbound, by our friends at Tricentis, so many thanks to Tricentis for their sponsorships. But this is a media ops event.

Let me introduce you to today’s panel and then we’re going to jump into our topic. Let me first introduce the guy with all the robots in his background there: Paul Bruce, with a 404 shirt. Paul, welcome. Why don’t you introduce yourself to our audience?

Paul Bruce: Sure. Thanks, Alan. So, my name is Paul Bruce, as you said. I’m on the nerdy side. I’m a performance and reliability geek. That’s just a place of complexity that I love to – it’s just what – it’s just my bag; it’s just what I like to do. On the other side I’m a community organizer. I work with DevOpsDays Boston and the Boston DevOps Meetup. I am hosting an event along with Liz Fong-Jones this year called o11yfest around observability, a vendor-neutral and also – community-focused but also the open telemetry project-focused two-day event. It’s half days so that we don’t burn people out. And I work with the Tricentis team since the NeoLoad acquisition – again, performance reliability. I was with Neotys and NeoLoad for years now. I really loved that team. And I love my new one, the broader one at Tricentis. So, that’s me.

Shimel: Very cool. Thanks, Paul. And then, joining Paul today is – we’re very lucky to have her – someone I’ve had the pleasure of interviewing more than several times now over this past year of Covid but hopefully soon to see in person, my friend Christine Yen. Christine is CEO of Honeycomb. Christine, welcome.

Christine Yen: Hello. It’s wonderful to be here. Alan, you shared my title, but my background is as a product engineer. Honeycomb, for anyone who isn’t familiar, is a tool who – a tool for observability. And I love working on it and I love talking about this topic because even though I tend more towards the software engineering side I have done my fair share of breaking production, and so this topic is near and dear to my heart for software engineers to do that less and find out what their code really did more quickly.

Shimel: Very cool. Thanks, Christine. It’s great to have you here. And then last but not least is my cohost of DevOps Unbound. We’ve worked together for probably longer than either of us want to admit, but it’s hard to believe we’re both only 25. But CTO here at MediaOps, CEO of the Accelerated Strategies Group, Mitch Ashley. Hey, Mitch. Welcome.

Mitch Ashley: Thank you, Alan. But the best 25 – this year again. Twenty-five again, right?

Shimel: Yeah, 25 again again.

Ashley: Good to be here.

Shimel: Anyway – yep. Thanks, Mitch. So, today’s show, really, we’re going to explore the intersection – or maybe it’s a border between performance, performance testing, and observability. What – how do they play with each other nicely? What is the interaction? What is the relationship? And for those in our audience, you may be looking at it and saying, “Well, observability is kind of almost post-deployment. Hopefully we’re doing some performance testing pre-deployment and then post-deployment.” But how – I don’t know if I see a nexus right away but there’s – in my mind anyway there’s an obvious nexus. There’s an obvious connection.

So, let’s talk about it at a high level. How is observability and performance testing connected? And how can they – how can we get a one plus one equals five or four out of performance testing and observability? Christine, if it’s okay, I’m going to ask you to kick off with that and kind of lay out the framework, the ground rules, the – and we’ll all jump from there.

Yen: Well, laying out the framework and ground rules is a little bit of a big ask for early in the morning, but I’ll do what I can.

Shimel: Okay. I’m sure you’ll do great.

Yen: I’m wearing the wrong shirt for this. I often – I think I’ve come on one of these interviews with my “test in production” shirt, but it’s a phrase that too many folks – it can be interpreted in many ways. What does testing in production mean? There is a version of it that means, well, I’m not going to run tests beforehand; I’m just going to see what breaks in production and try to fix it after it’s impacted users. Obviously, that’s not what any of us mean. And there’s a version of it that means, hey, look, when you write preproduction tests, when you write any sort of tests, what you’re trying to do is you’re trying to compare actuals versus expected. You’re trying to form a hypothesis, observe reality, and then think about that difference. Isn’t that really just a different phrasing of what we’re trying to do when we’re looking at production, where we’re using our logging or monitoring or observability tools to see what actually happened when our code is in production?

And so, I love – I think it’s a huge intersection, a huge overlap between things like performance testing and observability because we are all using the same skill set, we’re all using the same brain sequence, but there’s – people seem to think that observability is – can be more exciting because it’s incidents and downtime and alarms are going off. I don’t think it is. I think it’s all the same. It’s just at different points in the development life cycle. It’s on different timelines. It’s with a different level of urgency. But it’s the same thing we’re all just trying to do to come up with hypotheses, validate them, and learn.

Ashley: I really like, Christine, the way you kind of intersected them together because the environment that we’re doing all of this has changed substantially. It wasn’t that long ago where it was kind of a static flow of writing codes, we’d test it, we’d CI/CD, we’d test it, we’d deploy it into production. And DevOps helped speed that up with automation and tools, and of course tools of automation for testing. But now the software stack is fluid. The infrastructure is fluid. The code is fluid because we may have introduced three new changes to containers, for example, in a cloud-native application since we’ve really had an opportunity to go and debug or determine what the source of that was. We also have dynamic applications that start up and go away, and those things may be long gone by the time we actually look at them. So, I think this key to observability is also giving you insight into an environment that may not exist at the time that happened, and so you need that information to be able to understand “This is really something I need to fix, address” – a bug, a feature, whatever it might be.

Bruce: Yeah, and I think there’s a lot of intersection between putting pressure on systems, whether that’s through actual users and risking revenues and all sorts of things or doing prerelease type testing or testing in production. There’s a – performance is about how does this system perform to expectations? And there are a number of different people with a number of different expectations. There’s the technical expectation, maybe articulated – hopefully articulated in something as legitimate as an SLO or an – and then measuring them with SLIs and stuff. But there’s also user expectation. There’s the CEO expectation. There’s B2B. If you’re in some kind of regulated industry and that SLA is actually part of a legal requirement that becomes really important. Like, we saw that unemployment sites – I don’t mean to pick on that kind of work, but that was a mess. Same thing with various different – trying to get the vaccines scheduled and stuff. And it’s like why are we still suffering from these systems that are out there and they’re good enough maybe for the day to day but have never been exercised, have never been – the concept of that being – that scaling has never really come about for that system.

Okay, well, what about systems that do by default have scalability as a first class citizen? Going back to some of the stuff that Christine said – and actually, Christine, I love – your cofounder Charity has a phrase and I’m going to butcher it, but “It’s never not a good idea to have your glasses on when you go driving.” That idea that if you can’t see what’s going on that’s a big problem.

So, a lot of people traditionally would have used something like “Let’s just throw all our APM tools at production.” And that’s great. But going back to that original thing where I was saying pressure on systems really shows us how well we’re doing engineering, the impact of that pressure matters too. So, as a performance engineer, performance and reliability engineer, I’m constantly not just trying to – as my French team would say – make load on a system, put load, put pressure on a system, but I also care deeply about being able to measure the impact of that.

I’ll pause there because we probably some other questions. But there’s more to go into about that. It’s not an either-or. And it’s not about just production or preproduction. It’s about what’s not a good idea to have visibility. Isn’t it a good idea to have visibility on a thing, whether it’s putting pressure before release and having visibility or doing that in production? We’ve got a lot of customers who do testing – who do load testing in production on production systems. This is not an impossible thing, folks, you know?

Shimel: So, let me try to put this in some kind of context that we can jump on. So, here’s what I say Christine saying, which is – and maybe I’m taking it up a couple notches, getting out of the SLO woods – at the end of the day what do we all want? We all want our apps, our infrastructure to run better. We want to have good predictability, visibility into how things are performing. And whether that is an aspect of performance testing, or you want to call it performance monitoring, or do you want to call it observability, at the end of the day the aim is the same, which is let’s understand how our stuff is running and make sure it’s running the way we think it should be running. So, at a very high level it’s almost equating performance or observability – I mean, it’s all with the same goal, which is let’s make sure our stuff’s running good, as the best we can.

In my mind – and Christine, I’ll defer to you – certainly over the last year, year and a half, maybe two even, observability has taken on – as it’s become more hyped and a real word, a real thing out there, has taken on almost an even bigger mission than that, where it just doesn’t equate with – the goal may be “Hey, let’s make sure things are running the best we can make them run and how we expect them to run,” but there – observability has a whole uber mission of things that people want to do with it or use it for. Paul, you mentioned APM. Kind of observability subsumed APM, AI ops, all of these things. But yet, wait, there’s more. Observability does more than that. It’s become this whole big thing out there. And maybe it – maybe that’s right, maybe that’s wrong. I’m not here to judge.

But that, I think, is really the crux of this. Yeah, we all have the same goals. Is observability – part of what observability does helping the performance engineer? Absolutely. Is there more to observability? I think so. Christine, you’re the observability expert. I’m going to ask you.

Yen: You know when you say a word enough times and it starts to lose its meaning and you’re like “This is starting to sound funny”?

Shimel: Yeah.

Yen: We – I think you’re – there’s – we are definitely approaching that with observability a little bit. And whenever I get to that point I like to pause and remind folks “Let’s look at the – let’s take a deep breath and look at the English word we are using to describe this. It’s not a bucket full of buzzwords. It’s not a bucket full of types of data. It is an ability to see. And it’s the ability to see into our production systems. What do you want to do with that? What can you do with that if you have an improved ability to see?”

Okay, well, now we have the laundry list of all the things that we can do once we have the ability to see into these systems that, as Mitchell mentioned, are so different than they were five or ten years ago.

Shimel: Oh, yeah.

Yen: And that to me is – that zooming out behavior is what re-centers me when it feels like we’re trying to do all the things. And you’re like “Well, it kind of makes sense. I want to have my glasses on when I’m driving.”

Shimel: Yeah. Paul?

Bruce: Christine, you’re definitely right about a buzzword merry-go-round. I have experienced that with “DevOps.” I have experienced that with – oh, man, I can’t believe I’m saying it again but I’ve got to because people will know – “shift left.” These terms that just get all the things. And we go at these things, these different terms, we latch onto them, we try to make it our own. Oh, man, vendors love to do that. We’ll take a word and we’ll make it ours and we’ll try to align it to our product strategy. And at the end of the day it’s like there’s 150,000 different versions of this thing, which as a performance guy I’m kind of like “Well, it’s not bad to have a big sample set,” but the sample set of what’s going on in the delivery chain from good idea to working software that’s making you money is not just in production.

Let’s take an example of Nike, for instance. This is a known thing that they ship the watch, the watch that you open up on Christmas Day, and you go to connect to the internet and because the servers were swamped you don’t have a good Christmas experience. That’s not the only – I’m just picking on… But that kind of situation is a situation where there’s a lot of stuff you can do to prevent obvious faults from getting to the point where that’s the bad experience. And you also need visibility on the day where your production is having problems. And it’s not just the day; it’s every day. Every day you have a certain un-zero amount of events and issues in your production systems. How do you address those things? Well, there are plenty of situations where if you’re only thinking about observability as it only applies to production – which, by the way, production is where – what makes you money. If code is not sitting in production doing something useful for people you are not making money off of it necessarily. And production isn’t just out for consumers. It could be all the different constituents in your larger organization. So, even private systems are production. And so, if something is not out there then it’s not making money.

So, is that the only place where we should pay attention, where we should have good visibility on logs, traces, and metrics, on understanding how the business is impacted, long and complicated queues and delays in these systems? No. I mean, yes, in our edge and final production systems, yes, of course we will want that because we don’t want to be in a bad situation not having the right visibility into problems when they’re happening in production. But how do you think problems come about in production? Well, there’s the emergent ones that we can’t possibly know about. Ooh, there’s just so much time in the day. But then there’s a whole bunch of other stuff. Every line of code is not just an asset; it’s a liability. So, as you’re constantly shipping these things out there’s plenty of things you can do to prevent obvious – and I mean dumb and stupid and obvious things from getting out there, causing toil, waste, risk, and loss of brand, revenue, and all that stuff.

So, I think the observability for me is not just in production. It’s really useful to have classic monitoring in preproduction environments except for that it’s super expensive. Oops. And then, observability, pieces of observability, which Christine would probably be better to go into, all those things that you can do and what you get out of that, and now we can have use cases, that also applies to preproduction situations. That’s why I’m so excited about that open telemetry project and the community over there, this notion of being able to emit information from your systems, to either build it in directly from your systems or put it on sort of after the fact, sort of replacing some of the obvious early agent type stuff that monitoring used to be the only thing that could do. That open telemetry stuff now makes it a standard that we can all benefit from no matter which environment. How about all environments? That would be super nice. And to get tracing and traceability from – all the way from the Web app that somebody is touching all the way down to the database and the 30 different APIs that it touches.

That’s really useful, not only to firefight in production but to see “Am I risking dependencies? Am I go wildly off my architecture diagrams?” You know what I mean? Those things slide. Your conceived architecture diagram is never what’s actually running in production. So, to actually be able to have visibility on that is really important.

Ashley: Well, you know, Paul, the lines between production and, let’s say, tests are blurred, to use a phrase – are really kind of blurred, and oftentimes when we adopt whatever new technologies or approaches we’ll attempt to do what we used to do just using a new technology to do it. So, we’ll test the way we used to. And just what sort of clicked with me with cloud early on was how Netflix was doing their rolling upgrades and being able to flip back and forth. The same is really true for testing and performance testing because your availability to resources generally is not an issue, or not the issue it would be normally in an environment where you’re not in the cloud or at least partially in the cloud. So, there’s a lot of things you can do not just in your current testing environment but to set a pretty elaborate – with resources to do different load tests, performance test scenarios, in addition to what you experience in production.

And I think one of the points that you made is salient, is if you’re paying attention to this you now understand your systems, your applications, your environment enough to identify “Well, we may be able to handle the number of unemployment claims we’re getting today but I know when we reach the next peak these are the three things it’s going to impact. I know those are the things we have to go adjust or fix or whatever it might be.” Because you’ve already instrumented it well enough and you have the information through observability to be able to do that. So, I think that’s a different mindset of how we think about performance testing.

Shimel: Yeah. So, I tell you, I think one of the big breakthroughs of DevOps has been to create our “testing environments” to be more lifelike, to be more real lifelike. I mean, because prior to DevOps, the world I grew up in, the classic thing is the developer developed, gave it to the ops guy, the ops guy puts it up and says, “Wait a second, this stuff don’t run.” And he says, “Hey, developer, this doesn’t run.” And the developer says, “I don’t know, dude. It ran on my machine.” And so, there was that disconnect between what the developer wrote and ran it on or they tested it on versus the real life production environment. And one of the great things about cloud and virtualization and all of this stuff is that we can theoretically really duplicate our production environments and test environments and we can increase loads and we can do all those things theoretically so that there’s no or less surprises when we go live in production. And that – look, that’s – thank you, DevOps. That’s part of what this whole DevOps thing, I think, was.

But I don’t know, is – Christine, is that really – does that dog hunt or is that just DevOps urban myth? You know?

Yen: I think you’re on – right on the ball. I mean, when Charity and I started Honeycomb she came right – that story you told, I was that dev; she was that ops. I did the whole “It works on my machine; I don’t know what you’re talking about.” And when we started Honeycomb we thought honestly we were going to be talking to more of the Charities of the world. We were like “Oh, this is a tool for ops people. They’re the ones who understand the pain.” And as Honeycomb grew and as observability grew it became more and more obvious: “No, this is for the Christines of the world too.” This is for the people who need to understand what happens beyond that testing environment. This is – the boundaries are blurring or – I like to think of it as dev and production are coming closer together because there are more and more things we just can’t test in a test environment. As good as we make the test environment, we just can’t.

Shimel: I totally agree.

Yen: And there are so many things that can be learned – the same – by thinking of production as part of the development environment. With feature flags we can start to take code that is really not quite ready for production but test it out in a safe way and learn from it. And that’s so cool. It’s – there’s so much for developers to be able to learn when everyone kind of gets over this mindset of “Well, there’s a wall between development and production.” There isn’t. There shouldn’t be between people or the software. And that’s where all the exciting parts live.

Shimel: I don’t disagree. I don’t disagree.

Bruce: So, Alan, one thing that you said – you painted sort of the stage so that we could have part of this conversation. I’m not a big fan of setting up huge staging environments. In classic world it was like, okay, well, here’s production and it uses 36 Oracle back-ends and – if we were going to run a proper performance test of course we needed 36 on our side. And before the cloud that was incredibly expensive and you’d fax your IT person and they’d get back six months later with the right hardware. And I think there’s a myth that, well, yeah, you need to be realistic at some point, especially when you’re putting pressure on a system, to answer the final question, which is – what? – is it ready enough for me to be – what? – confident enough that this thing is going to transition like we talk about in standards, the transition process, that that’s actually going to go smoothly? And not in a waterfall way, just in a – to get away from this notion of “It worked on my machine” and now the ops person is putting out fires and doesn’t know how to operate the thing.

But I’m not a big fan on spending a ton of money on this staging system which ultimately even if – even if at a very root level it’s exactly the same as the production system there’s always that jerk in the corner going “Yeah, but it’s not production.” And it’s like “Okay, you have a point but you don’t know what goes on.” And the point is I think almost like a – I’m going to do my idiot thing and pull in a mathematical and then probably get it wrong, a mathematical concept of the limit. You remember limits? And the idea is we need to be able to express are we talking about the very ends of things? No, it’s a continuum.

So, there’s a lot of companies that I work with that actually do have requirements, that they go through a particular amount of testing and particular things before they can transition this to production. They don’t have the luxury of treating production like a dev system. There is a ton of people in the world, and it’s not their dysfunction – that’s not a dysfunction, just because the unicorns can fart rainbows and sneeze glitter and launch whatever they want. I see that happening in Boston all the time in some of the startups, and they do actually put themselves at risk. And even when you don’t do that, when you have an extreme amount of maturity and you’re working with relatively complicated technologies like Kafka things can go wrong pretty quickly. And the question is do you have the people and the process and the technology and the knowledge and the experience and the expertise to handle those when those things do come up in real time?

How do you get there? How do you get to people who know how to use your monitoring and observability tools really effectively? How do you know what those systems, how close to your perceived architecture versus your actual in production, how do you know how much delta there is there? How often do you exercise the process of disaster recovery is one of those other classic questions you might ask yourself.

And it gets back to who would be involved when there is a firefight or a last-minute thing like what we’ve done with some of the companies that I worked with very closely through the NeoLoad side of things, is when the education system, when colleges shut down and they finally said, “We are shutting our doors to the people that are paying us money,” that was a scary and real and significant flag for the rest of our US education system to go “Holy crap. We’re going – we have to provide virtual not as a stopgap for the past few months but this is going to be a serious reality for us for the next year.”

And so, a lot of the vendors that help the education system have the right platforms in place. They go “Holy crap. We really need to test for 10X the size that we’ve ever tested before.” So, they’re testing in production on nights and weekends in various different places and doing it at a smaller pace. But who do they have on the call? Not just the Ghostbusters, but the devs, the ops. There’s people from infrastructure, the DBA teams. They have 20-plus people white-knuckled on the call, like [makes quivering sound] – white-knuckled on the call, and that’s a lot of money. And by the way, you’re asking them to do from 10:00 PM to 2:00 AM to 3:00 AM work. And so, they’re not going to be available for all those fancy meetings, architecture meetings that they should be stakeholders at.

So, when you get to the point where you’re doing those massive things sometimes it’s necessary to go massive. But the vast majority of the exercises, you go to the gym – not so much me, but you go to the gym to exercise. You go and you get on your bike and that’s an exercise. You don’t expect to run a race without exercising first, without knowing that path a little bit before you just run a 5K race. So, why would you expect to do that with your systems and your teams?

And that’s where some of the testing, the lower order performance testing comes into play. It’s systematizing the stuff, making a process that can be executed and re-executed no matter which environment it’s pointing to. And the telemetry that you’re getting out of it aligns in each of those environments. If all of a sudden we have really great telemetry in a pre-prod environment but we don’t have it in prod that would be a big problem. Why was it put in here and it’s not represented in production? What if we need that? Or vice-versa: We’re only doing things in production, and so now there’s only – there’s stuff that’s only in production that has no representation of how it would show up in lower environments.

So, keeping those things in synchronous but also keeping the process and the exercised skills in place is why this isn’t just a big environment at the end kind of a situation.

Shimel: Again, I think “Thank you, DevOps” where we have been able to say, okay, not only do we need those or can we use those testing dev environments, but as Christine said, with things like feature flags and some of the other technologies, processes that we use, it is possible to continue that – the feedback loops and iterations in real time while using the – while in production. And that again is one of the things that allows us to continually update, continually iterate, continually – and feedback loops was kind of an original concept as part of this DevOps thing. We’re going to have those feedback loops and we’re going to iterate and reiterate.

But where we’ve gone with observability with that and with performance testing and these things with that have really just opened the floodgates in how we do these things now. I mean, just think about how – when I first started devopsy.kinsta.cloud eight years ago, I mean, there were the unicorns who were updating multiple times a day. But for many people in the world, for many organizations in the world, look, moving from once a year to twice a year, twice a year to quarterly, quarterly to monthly was huge. It was huge. It’s not so huge anymore. It’s common for people to be updating their code daily or more than once a day now.

I saw – I did an interview this week, a recent survey of developers who they claim that they have sped up their deployments by over 100 percent in the last year and a half. That was – something like 70 percent of developers said that. Think about it. I mean, that’s – a billion here, a billion there, you’re talking real money. Seventy percent of developers say they’re going 100 percent faster. Wow. Wow. What a golden great time to be living. And it’s because of things like observability, DevOps, our better performance testing. Anyway.

Guys, we’re coming up on our time limit here. I wanted to kind of – let’s wrap it up. Let’s – what advice – for our folks listening in, watching in here, what can we give them? What can we tell them about this nexus of observability and performance that will make their life better? Paul, I’m going to – if you don’t mind, I’m going to put you on the hot seat and ask you to go first.

Bruce: Sure. Well, there’s a couple things I’ve been working on for a while and then there’s a couple things that I just want to give a shout out to. I’ll start with – Christine doesn’t get to do the pitch thing but I do. Honeycomb is awesome. You see that thing in play and it’s awesome because the team behind it is awesome. You know what I mean? It’s kind of the Conway’s law situation where – the version of it I hear most is the systems are a result of the teams that build them. So, definitely check that out.

Same thing with some of the other tools around there. But don’t get too bought up into the whole observability – don’t just Google “observability” and look at the first one because most people are paying through the nose for SEO placement there. Really actually look at what’s available, and then also dig into the team, the blog, the spirit behind that thing.

The second thing I’d suggest is we just released – I was one of the members of a working group for a standard on DevOps principles and practices that applies to highly regulated industries. It’s IEEE 2675-2021, which we’re hoping to get adopted in ISO so it’s global as well. And what it does is really align what is – how are each of the important processes that can be applied continuously through many different life cycles as they’re going along? You know, that DevOps eternity symbol means we’re all going to be stuck in hell forever, that thing is happening, all the things are happening all the time but there are precise things that need to go on in highly regulated industries, and even not. So, we’ve been working on that for four years. And you can reach out to me on LinkedIn, on Twitter – although LinkedIn is probably a better way to go – if you want to know more about that.

But essentially it’s a body of work that a lot of people have been putting time and effort into to really say, “Hey, look, this is not about what DevOps is. This is about how to do this in these complicated situations.” And you know me; as a performance geek I tried to – I was involved in QM/QA, verification, validation, risk management process, those kind of things. And I tried to make sure that evidence-based decisions, that – which is a hook to basically say visibility, telemetry, the proper SLOs in place, those are definitely in the standard along with other things.

So, definitely check those two things out: Honeycomb and the IEEE 2675.

Shimel: Excellent, Paul. Christine.

Yen: Thank you for that plug, Paul. The one piece of advice I’d like folks to internalize, we’ve talked quite a bit about – or, I’ve talked a bit about developers in production and developers in observability. An asterisk there is that that is not – the practice – the muscles and the practices should transfer well but there are a lot of things in the DevOps world in production that are, well, in some cases outright hostile to developers and in many cases just kind of scary and unfamiliar. And so, if this is something that’s interesting to you, if you’re like “Ah, yes, I would also like to bring development and production closer together,” take a step back and look for things like that. Do your production tools primarily talk about AWS instances and CPU and memory use? Those may be things that might scare your developers off. How can you make the vocabulary that your production tools talk about more familiar to developers? How can you help them bring the concepts and nouns that they’re used to in testing – customer ID or endpoint or logic, something that feels more familiar to the developer world, how can you have that exist in your production tooling as well so that developers can show up and be like “Oh – oh, this isn’t so different”?

I think that there’s a lot of things that are encoded in certain practices because we assume that a certain persona is the one using them. And those walls are blurring – or those walls are coming down; those edges are blurring. There’s a little bit of work left to do to facilitate it.

Shimel: Absolutely. Hey, Mitchell, you want to bring it home?

Ashley: Yeah, I’m going to wrap it up this way because there’s some great practical advice – and by the way, both companies have fantastic products, so please check them out. I think we have this mental model, this habit of thinking of things in one state. Matter has four states – well, five states if you count Higgs boson – Bose higgson. It’s not just solid. It is also fluid. I mean, think of our systems. Don’t think of it as software tools, all the layers of it. Combined together they operate more like a fluid, in a state that’s changing, constantly moving. Things are happening. And that causes you to rethink developers and production, doing production – doing performance testing in a production environment. And the flow of software through that DevOps tool chain is really part of that – the fluidity of that state.

So, I think we like to think things – of things like “Let’s make everything a solid so it doesn’t change, so we can look at it, observe it, do something with it.” It’s not really that way. That’s not the world we live in today. But if you think about it as something that’s under constant change like a fluid, then I think we have a much better chance of figuring out better ways, whether it’s through technology, new paradigms, new ways of working, of how to make our software the best it can absolutely be.

Shimel: And at the end of the day that’s what we all strive for. Right?

All right. What a great way to end this, then. This is a wrap. I think it’s episode 13 of DevOps Unbound. But it’s a wrap on whatever number this was because it’s fluid anyway. It’s a wrap on this episode of DevOps Unbound. Many thanks to Tricentis for sponsoring us on DevOps Unbound. We will be back in two weeks with another great episode. Stay tuned. We also have a big roundtable coming up this month; check that out.

Paul, hey man, congratulations on joining the Tricentis team as part of the acquisition there.

Bruce: Thank you.

Shimel: Christine, they’ve all said great things about Honeycomb, but well deserved. Say hello to Charity for us as well.

Yen: I will. Thank you.

Shimel: Keep doing what you’re doing. I hope to see you all soon or a couple months from now in person. Maybe we will do a live DevOps Unbound at some time somewhere. But until then, this is Alan Shimel for MediaOps. You’ve just watched DevOps Unbound.

[End of Audio]

August 2, 2021
EP 12: DevOps Unicorns

What should you look for when choosing a DevOps vendor? What criteria are others in your industry using to inform their decisions? How do you choose a vendor that is a good fit for your organization? And how do you know if you made the right choice? In this episode of DevOps Unbound, host Alan Shimel welcomes Larry Maccherone of Comcast, John Willis of Red Hat, Ravyn Manuel of the Smithsonian, Garima Bajpai of Capital Carbon Consulting and Mitch Ashley of Accelerated Strategies Group for a panel on DevOps unicorns. The video is below, followed by a transcript of the conversation.

Alan Shimel: Hello, everyone. Welcome to another edition of DevOps Unbound, where we talk about DevOps, hence the title. Thanks for joining us. This episode is titled DevOps Unicorns: The Quest for End-to-End DevOps.

You know, I’ve been in IT 30-plus years and there’s always been this tension or this pendulum that swings between the one throat to choke versus the best of breed solutions in any area. Today it’s DevOps and software delivery and deployment. But in anything, in the security world, in everything, in hosting, in infrastructure it’s always been a common theme. But it’s really, really becoming acute within the DevOps sector now. And we’ve assembled what we think is a great panel to discuss this issue. Let me introduce you to them and we will go from there.

First of all I want to welcome, well, to a panel for the first time in a long time, my friend Larry Maccherone. Larry, welcome to DevOps Unbound. Give people a little background, if you don’t mind.

Larry Maccherone: Sure. Thanks for having me here, Alan. First of all I want to say so I launched the Dev PsychOps Transformation Program at Comcast and grew that for five years. And I’m a developer at heart, though, so I was sort of the ideal person to take what was actually becoming more common outside of big enterprises, Dev PsychOps, and try to actually do a diverse approach inside of a large enterprise like Comcast. And so all of my experience is around that recently, the last five years or so, around that.

Shimel: Excellent. Next let me introduce you, she’s a repeat visitor to DevOps Unbound. Garima, if I mess up your last name I apologize in advance, but Garima Bajpai. Close?

Garima Bajpai: Almost. Yeah. So I’m Garima Bajpai. I’m based in _____, Canada and I’m the founder for the DevOps Community of Practice here in Canada, which has three chapters: Toronto, Ottawa, and Edmonton. I’m also the organizer of the DevOps Summit in Canada, which happens in November 8th and 9th. And I’m also associated with organizations like DevOps Institute as an ambassador, as well as CD Foundation as an ambassador.

Shimel: Fantastic. And, Garima, I apologize on your last name. My bad. I’m trying to read it and the monitor’s too far away from me without my glasses. But thank you for correcting me, and welcome.

Next up, another good friend of mine, my actually business partner on Digital Anarchist. And though that’s not his full-time gig these days; he’s very busy with his red hat career. The one and only Botchagalupe, John Willis. John, welcome.

John Willis: Thank you, sir. Thank you, Alan. Hey, yeah, John Willis, Botchagalupe on Twitter. My current day job is in a team called Global Transformation Office at Red Hat. I’m sort of blessed to be on a team with Andrew Clay Shafer; Kevin Behr, the co-author of The Phoenix Project; and Jabe Bloom. Just quickly, you know, I’ve been in the industry for 40-plus years, 10 or 11 startups, 12 books, if somebody’s counting. Probably the most notable is the co-author of DevOps Handbook.

Also was involved in basically the starting and building of the original DevOps Days. Was the only American at first DevOps Days and Damon Edwards and I created the first DevOps Days event in the USA.

Shimel: Yep. And that’s only a partial list, John. Thanks for joining us today, man.

And then I want to introduce – she’s off-camera right now, but hopefully when you’re watching this at least you’ll get to see her picture, but she was having some video difficulties, but happy to have her on audio, my friend, Ravyn Manuel. Ravyn, welcome. And if you wouldn’t mind, give people a little background.

Ravyn Manuel: Thank you, Alan. I do appreciate being on your show. It is awesome. It makes me laugh. I am the senior application developer for the National Museum of African-American History and Culture. And I’m also the DevOps engineer for that particular unit. As a developer I implement some of the hands-on interactive that you play with in the museum, and as a DevOps engineer I am in charge of just about everything that’s DevOps, from the cloud solutions and how we – from the application development side, how we’re using cloud and the tools and writing strategies and keeping my leadership informed of all the good things that are out there.

Shimel: Fantastic. Ravyn, welcome and thanks for joining us today.

Last but not least, my co-host here on DevOps Unbound, Mitch Ashley. Hey, Mitchell, welcome. You want to just introduce yourself?

Mitchell Ashley: Great. Great to be here. Man, what a panel. I am so excited to be with these folks on today. Yeah, I head up Accelerated Strategies Group, analysts from focusing on, of course, DevOps and digital transformation, full cybersecurity and cloud. And I also serve as CTO with MediaOps, so heading up the platform strategy and technology, so we used to deliver programs like this.

Like John, I think we’ve been probably in the industry about the same amount of time and have zero books to my credits, so he’s made a lot more progress in those areas, but also a startup person and run a lot of software projects, product, as well as IT groups, as well as on the vendor side. So this will be a fun discussion.

Shimel: Absolutely. I should mention DevOps Unbound is sponsored by our friends at Tricentis. To give you an idea of what a great sponsor they are, they don’t even have anyone on our panel today. But they do enable this show to go forward every other week, so many, many thanks to Tricentis.

So, John, I laid it out, right, it’s the one throat to choke versus best of breed, how many vendors can we support, how do I package solutions that are more than just point solutions, or maybe I just want point solutions. What’s the answer here, John?

Willis: The answer is, Alan, I would say if I had the right answer I’d be on a Grady-White right now in the Gulf of Mexico.

Shimel: Exactly. And I’d be right with you.

Willis: But no, I think this is a great panel. I agree with Mitch; I’m excited about just the discussions we can all have. I think the question – you know, I was thinking just that sort of Chinese curse translation, “May you live in interesting times,” right? And that’s clearly where we are right now. And I think your question starts with a shift, right? Like there was this – talking about you, Mitchell, some of us have been here forever, and like back in the day there was this question of one throat to choke versus should I do an ELA with computer associates versus piecemeal things, right? And probably less than one-percent of the enterprises were do-it-yourselfs.

Now today you will find probably – I’m just going to swag the number, but it’s probably north of 10, maybe 15-percent, maybe even higher of companies that are now being challenged to do it yourself. You know, Kubernetes is a great example. That’s a can opener for companies to make a decision of are my vendors keeping up with what comes from basically sort of upstream technology that I should build myself.

So I think that’s the sort of question we move into, is more about do you build internally to solve your problems, to keep up with the changes in the market and the technology, or do you try to find a pair of vendors or a single vendor – and there really is very few single vendors that can give you the whole scope of what’s going on in modern technology today. But I think that’s the question that I think we’ll have fun just talking about today. _____.

Shimel: I don’t disagree. Panel?

Manuel: So, Alan, this is Ravyn. I agree with you, John. I think one of the things – because that’s a struggle for me and my organization, is we need help with implementing DevOps, but I think the vendors are very tool-focused and the tools are like the third priority in the stack of there. And what I’ve been doing when I’m building the strategy for how we’re going to implement DevOps, I looked, and we’re talking about tools. I look at the standards, because if they’re making standards and the tool has to follow that standard then it’s plug-and-play, you don’t get – like if Docker decides that they’re going to go out of business then you can have another container-type technology to do that.

So when we are looking at benders we actually kind of do a piecemeal, because we’re really unicorns. We’re not a software company, we don’t have software products; we have digital products, and they’re all one-offs. And so it really just depends on what fits for us. So we don’t really go for the one vendor solution; we actually go for who can help us get to where we need based on our business problem.

Bajpai: Excellent. And I wanted to add a few things, because when we drill deeper into this conversation we will find out that there are two sides of this coin. One part is where an organization from a maturity perspective is. And the second thing is what are the vendors and key capabilities you’re looking at. But if you look at both sides of the coin there are three things which are needed to calibrate these DevOps tools in the overall context. The first thing, and this is – I want to reiterate the fact, because I come from a very large complex organization background, that it is very essential to have an enterprise architecture view. So I would suggest and recommend that before even, you know, taking this approach of single vendor or multivendor discussion, we need to kind of break down what are the key capabilities which are needed. So whether it’s a single vendor which can provide you that enterprise IT architecture capability, or it will be your organization who will be doing it. So that’s essential.

The second thing I think which is also essential is orchestration layer. You know, you will have to have that governance of total cost of ownership, compliance, security, identity access management, and that orchestration layer has to be there to stitch all you calibrated DevOps tools. So it’s a question of whether you want to kind of build it in house in that organization or you kind of have a vendor-centric approach where this capability should be or could be brought in.

So these are like two things, and then you start calibrating these DevOps tools. So you don’t have to look at the foundation aspects of compliance, security, governance, total cost of ownership, and the job becomes much more easier. The question is that, you know, how to bring these DevOps tools and enable business value.

Maccherone: So I want to riff off of that orchestration layer idea, but I want to backtrack a little bit on sort of the – we have a binary sort of difference here between unicorn and best of breed. But I think within the unicorn bucket there’s really two major flavors of unicorn that are wildly different, and so I want to sort of bring that out. You have these Frankenstein vendors that are piecing together acquisitions into this monster and those are not pleasant experiences. The only benefit you get out of one vendor in that case is really the contracting benefit of having a single master service level agreement. You’re not getting an integrated product that really allows you to actually smoothly do the workflow, insist in the workflow that developers are using to produce value for your company.

And then you have this other class of unicorns in the DevOps world, but less so in the security world, and I’ll talk about that in a second, which have taken a marketplace approach. And this is not a DevOps vendor, but I think the first vendor to really pull off this marketplace approach was Atlassian. You know, they had one product that was really great, but they said, “Okay, plug into this product” and they created APIs and they created toolkits and they created a whole community. Most of those people did it for free and they just sort of did it for themselves and then they shared it with the community. But some of those people became vendors and sold their product in the Atlassian marketplace, and then Atlassian acquired those. They were already pre-integrated with Atlassian’s other offerings when they got acquired, and so their acquisition unicorn approach doesn’t look like a Frankenstein’s Monster in the end, because they were built with the APIs of the vendor in place.

And I think Github is definitely taking that marketplace approach and pulling it off really well, even into the security space. And so I think that kind of unicorn – and Github to a lesser degree, that kind of unicorn doesn’t suffer from that Frankenstein Monster’s sort of problem. No one in the security space has really pulled off the marketplace approach yet, though. We have some Frankensteins and Frankenstein Monsters over there, but we don’t have a full sort of marketplace approach. And I’m waiting for somebody to do that. I think that’s going to be the big win, somebody _____.

Willis: I’d like to do a whole podcast on that, because that’s all I’ve been thinking about right now, is what is the end-to-end on security. But like if I got into it now it would take up way too much time. So I did want to comment on some of the things. You know, the first, Ravyn, I think the interestingly about standards, I agree, right? The problem we have right now is figuring out which standards are actually going to last, are malleable. I mean that’s one of the things I’ve really been working hard on and looking at like OSCAL versus OVAL versus S-CAT versus, you know, just NIST _____ – in order for – I mean there’s just so many things and they all sort of have a little bit of different approach.

So I agree, the question we have to deal with now in these sort of, you know, may we live in interesting times is it’s like picking sort of an early vendor, like I like this vendor technology, but are they going to be around? Are they going to be acquired by a company that I don’t really want to do business with? I think the standards is just it’s such complicated.

Just the other thing I wanted to say is I think that for vendors, you know, and I’m not _____ ’cause I work for Red Hat. Because I will say right now the thing that like in the grand scheme of technology, if you’re going to go all in a vendor, look around. You know, I’m going to say this. I’d love to be corrected and I’m not saying, you know, Red Hat because I work at Red Hat. Anybody that knows me, I’m not a shill for any vendor I’ve ever worked for. But it’s VMware and Red Hat, right? They can keep up with – because you’ve got to – in this world you have to have people on staff, their full-time job are contributors to an open source project.

And if you think about what’s going on with like Kubernetes and clusters and server list, you know, you think of the two companies that are positioned really well at that sort of scale version to be able to keep up with the complexity and change of technology because they’ve put the resources in. I mean a longer list that you can probably make after that. And I’m not including the cloud providers either, ’cause that’s a different story.

But anyway, those are the two comments I wanted to make on some of the things people talked about.

Ashley: I just wanted to jump in. What’s fascinating about this conversation is how diverse and complex it is. There are a lot of aspects to consider, whether it’s an enterprise architecture like Garima is talking about, John’s talking about, who are the vendors that can pull it off. Larry is too, if you’re talking about this sort of roll-up strategy, but retro-ally integrated strategy.

I think another dimension to consider is every organization has a DNA about how they think and operate when it comes to vendors and software and tools. And while I’m not always an advocate of go with the flow, sometimes when you’re thinking about how do we have something adopted, think about how the organization operates. If you are kind of a large tool vendor, let’s work with, you know, Red Hat or VMware or Broadcom or whoever might be – think about that. That’s oftentimes, you know, save yourself some brain cells and live a little longer.

There’s also the innovation curve, and some of the best ways to ride the innovation curve is the open source route and the vendors that are built around that kind of a strategy. There’s combinations of all those.

So I guess the thing I would add to the consideration of the conversation here is think about how you match up your strategy with the DNA of the organization so you can accomplish as much without killing as few brain cells as possible.

Bajpai: And thank you, Mitch, for bringing up this point of DNA of an organization. I think that this is a pathway of dialogue which we are having. I do believe that the mindset is shifting, and even I see in large legacy enterprises that they have started to realize that, you know, this vendor-based approach is something that needs to shift, and they need to bring people as partners in this context. Because when you think about DevOps it’s basically exploring your opportunities, breaking the barrier for, you know, your technology onboarding, the gaps – the alignment and outcome gaps you have created in the past.

I think this is the opportunity where the organizations can shift and bring more access to innovation through these partners. So I truly believe that there are softer aspects of this conversation as well which we need to kind of consider that traditionally how contracts are _____, you know, how we look at these DevOps tools from a vendor perspective, how can we bring them as partners, what we need to change if we start doing that.

It’s not kind of done overnight, right? So we have to kind of set what kind of modeling practices we need to bring in. And this opens up another door of possibilities. And probably during this conversation we will have some opportunity to discuss these SLA constraints which we have built in the organizations which steer us to a specific vendor or steer us to like a specific window through which we can outsource a lot of capabilities and just, you know, a choke the throat approach, right?

So let’s maybe in the upcoming discussions we also can bring these points and highlight.

Shimel: Ravyn, I know Smithsonian is sort of quasi-government, or it’s not true government. But to Garima’s point about SLA and being able to only deal with chosen vendors like that, it is; it is a real governor on your ability to choose best of breed, on your ability to have choice in how you want to do things. Interested in your experiences around that.

Manuel: It’s been difficult. I’ve been listening to this conversation and it just hit – I don’t know why it just hit me, but the vendors – and I love Red Hat, so I’m not trying to talk Red Hat or anything, and open-ship. But when you get a vendor that you have worked with and you trust, and Smithsonian, we use Red Hat a lot in our OCIO. Red Hat is doing open shift and Kubernetes and they are driving their tools to these specific tools. And for me, as a unit within the Smithsonian, I may not want to use open shift and I may not want to use Kubernetes, but I don’t actually have a choice, right? Because that vendor is a great vendor and they’ve got great tools, but they have chosen for me what I am going to be using, and that makes it really hard for me to say “Does that really fit what I really need? Is that too much?”

Because I can tell you that a lot of the vendors that I talk to, they’re expecting us to use this like suite of products, and it’s too much for what we want to do, and so I’m left with I either go with this vendor and overspend and use tools that I don’t really need, or I have to do it myself because I can’t really find somebody that can customize that for me. So it’s a challenge.

Shimel: Yeah, it sounds like it.

Willis: You know, one of the things I struggle with, you know, I sort of love committees and hate committees, right? Because like I so – HashiCorp has this cluster manager called Nomad, and I’ve actually seen people use it effectively with like incredibly less toil, 80-percent of the features of it, but it’s never going to get any escape velocity. It hasn’t and it won’t.

And I fear for our industry that we’re like so focused on there’s just only one way to do cluster management today where like – so to Ravyn’s point, like whether it’s Red Hat or whatever, it seems like you’re not even able to explore possible emerging containerized cluster management solutions because all of the oxygen in our industry right now is consumed by Kubernetes.

Manuel: Amen.

Shimel: Thoughts on that? I’d like to jump in with some thoughts on that. I mean this is sort of a dream/nightmare scenario. Right? I’ll take it – let’s take it off Kubernetes, let’s go back to PC operating systems. It was great that we had only Windows to develop on, and this way if you were an inept developer you only had to worry about developing on Windows. You had one operating system, you could make your app on it and it worked and you got 90-percent market share with it. What a fantastic thing for app developers. Of course, it gave Microsoft the inside track on Office and everything they did. And from a security point of view what a big, fat juicy payload that was, right, to go attack.

So then we started seeing, for as long as I’m in IT, “This is going to be the year of the Linux desktop.” Right? How long have we heard that? And so, you know, and then of course Mac resurfaced.

But this is we strive for some uniformity. We strive for Kubernetes to be the standard, ’cause it’s open source and it’s run by a foundation, and we want a standard. But then we chafe that, “Hey, we all can’t fit into that same standard. We want to be our own unicorn.” And that’s really what we’re talking about here today. If you go back to early, John, you sold the company to Docker before Kubernetes dominance, right? And there were several cluster management solutions vying. Right? There was Mesosphere and Rancher out there _____.

Willis: Mesosphere and _____, yeah. [Laughs]

Shimel: You remember. Yeah, _____ of these, exactly. So, you know, this is – but this is the world we – you know, to quote Hyman Roth in the Godfather II, “This is the world we chose to live in,” right? We strive for these single kind of standards that one-size-fits-all. And maybe that’s a huge mistake; I don’t know. Maybe it is a mistake. Larry, what do you think?

Maccherone: Yeah, I do see that problem, John, you’re sort of focused on here, that sometimes the de facto standard is not the best technical choice. You can go all the way back to Betamax versus VHS, for those of you who are old enough to remember that, right? VHS won out, but Betamax was considered by everyone to be a superior technology. Why?

You know, so I think if you’re going to take a best-of-breed approach you’re going to have to – when you have a true de facto standard like Kubernetes you don’t have to isolate yourself from it, but when you’re making a bet on which one is going to win out – and John is really good at this; John talks about this all the time. John sort of like doesn’t want to put his chips down on something that’s not going to survive long-term, so he wants to think about what’s going to survive.

So in order to hedge your bet there I think you need this orchestration layer that was talked about earlier. You need to sort of isolate the tool vendor from the way it’s consumed to a certain degree, so you can plug-and-play a little bit as time goes on. And this is what we did at Comcast, we built a layer between all of the tool vendors for security tools, because they don’t have a good unicorn solution there. There is no good unicorn solution there anyway, even if I wanted to. And so we had more than one vendor in every tool category competing internally, and sometimes we had three tool vendors. But from the developer’s perspective it was really easy for them to switch out from one to the other, and they got the same feedback channels, they got the same visualization, it aggregated and scored the same way across all these different vendors.

I think that orchestration layer is really key when you have a best-of-breed approach, and I think you have to build it yourself. I don’t think you can buy it today. I really don’t think you can do that.

Willis: I’ll try to shut up quickly. I just want to make a comment. I think there’s two books that help us understand our world. One is Guns, Germs, and Steel by Jared Diamond, which actually shows you why we get into these predicaments. If you haven’t read it, it’s a fabulous book. And then Start With Why by Simon Sinek is the sort of anti-pattern to that, which I would say like there’s clusters – atomic computed things like sort of serverless or functions or containers will be around for 10, 15 years. Clusters of those things will be around for 10 years. So if you think in those terms instead of thinking about Kubernetes and Docker – and Docker is already slowly dying, then you get that sort of start with why isolation against these sort of Guns, Germs, and Steel trends.

Bajpai: Right. Then I also wanted to kind of bring another aspect of this, which is connected, but I think companies might start looking at it from a scaling perspective. We talk about a lot of tools and how to calibrate these tools in our DevOps pipeline. Of course when you have a large organization you have multiple product lines, and these product lines will choose to be flexible, so they need to have that flexibility to deploy their DevOps pipeline in the way they choose. And that’s the whole power of DevOps, right?

But I also feel that the conversation in this discussion is leading us to a point where we start looking at our technical oversight committee from a governance perspective, where we can stitch the dots, we can actually lay that common foundation which is required in order for people not to kind of rewind the _____ over and over again. So I think that’s the key here.

And of course when we talk about a specific random _____ with huge capabilities it has pros and cons, right? So one pro would be like they can mutually enforce new capabilities. So when they bring one feature in one product it’s mutually enforcing another product, which is a beautiful advantage they bring in. But that also limits our possibility to look beyond what the vendor is bringing in. So we are kind of essentially not exploiting our own innovation cycles to introduce more capabilities around how we want to experiment, how we want to kind of build innovative new features into that life cycle.

So these are kind of good considerations to look at and I truly believe that the technical oversight committee is something that you will see in large organizations a wave kind of starts in a bigger way when it comes to DevOps.

Ashley: And there’s interesting too, when you think about different sizes of companies that are in the market for technologies, you know, Larry mentioned what he did at Comcast or what he was part of that was created as sort of an architecture where tools can be plugged in, unplugged, replaced, etcetera. It sounds simple, but I know that they have an architecture to do that. And some organizations have the wherewithal to create those things. What we see in the data, looking at the market, is in the small business, the kind of sub-200 people or so companies, they have really good high-performance teams that can work to create software or set up their own environments, kind of create a very productive setup for themselves, tool chains, etcetera.

Then on the high end or the large end, the enterprises, you know, they have a lot of resources available to them that they can potentially marshal to figure out how to do some of those things. The mid-tier market, the medium-sized company is often where they’re most challenged to – they want to do what the enterprise can do, but they just don’t have the resources. They don’t have the money or the vendor relationships across the board to do all those things and they’re expected to deliver at almost an enterprise-class level, but they don’t have all of the resources to be able to do that. And that’s often where somebody – the non-Frankenstein vendors that do a good job of kind of delivering a suite of solutions can work in those situations. Not that they can’t work in the others; they can, too.

But depending where you are in that kind of a company, that size of a company, makes a big difference on what strategy you can take.

Maccherone: I think that’s a great point and my recent experience last five years has really been enterprise. But prior to that I was talking to lots of companies, and you’re right, the mid-sized companies are the ones that are sort of in that no-man’s land sometimes, where they need some standardization, they need some consistency, but they don’t have the wherewithal to put it all together. And a vendor unicorn approach might actually be better in those circumstances.

Shimel: Hey, guys, I want to – Ravyn, I’m going to let you in a second. John, the name of the first book that you mentioned – I want to put these books in the notes somewhere.

Willis: Oh yeah. I did a presentation with Josh Corman way back with Guns, Germs, and Steel. Anyway, it’s an amazing book about how civilizations move from sort of – you know, how they go from sort of agriculture and they move into these clusters. If you can see the forest beyond the trees you can see how we get into these like Kubernetes is the only word we can use. Anyway, Jared Diamond, Guns, Germs, and Steel, it’s incredible.

Shimel: All right, we’ll try to get it in there. Ravyn, did you want to say something and I cut you off?

Manuel: No, I don’t think that was me.

Shimel: Garima?

Bajpai: Yeah. I wanted to actually enhance what Larry was saying about midsized companies. This is totally based on my experience and exposure with the Canada DevOps Community of Practice. We are seeing more and more of these midsized companies, not only the consulting companies, as well as the companies who want to build a DevOps capability coming to the community of practice and starting to steer this discussion of evaluation and assessment of tools. And I’m also seeing that the winter side. You know, specifically we are talking about tier two companies here. The winters and coming and exposing the tool capabilities in the community of practice, and they want some support and help for us to kind of steer this conversation “Is this a good tool? Is this a good capability where they are in their context?”

And it’s a very interesting discussion because the technical side is one side of the story. I would also like to kind of emphasize the fact that the questions we ask these vendors and these companies are like, “Okay, do you have a product owner for this product?” Because essentially, you know, what it means is that how accountability is built in a specific tool perspective. I also kind of ensure that we ask them what kind of themes they have, like what kind of cross-functional themes they have, so that we do not miss out on the point of interoperability, integration, automation, all these like are important points.

So I just wanted to mention that how we take that approach from the community of practice perspective when we’re asked to kind of assess.

Willis: You know, I totally agree with all of that. One of the things I was thinking about, I’ve had this luxury of being involved in these different communities. Like I’m in Alan’s community, you know, so I’ve gotten hooked into usually what’s interesting and going on in Alan’s world, which is a pretty big world. I’ve been hooked into Gene’s world pretty heavily, you know, DevOps Enterprise Forum/Committee. And then, you know, I’m one of the co-founders and advisors for the DevOps Days.

And one of the things that I get to see is true leaders. And I don’t want to put all this sort of weight and gravity on like you have to have a great leader to be successful, but like as I think about the people – Larry is a great example. Like, you know, you can talk all day long about what Comcast does. I’ve seen Larry in action. Like that’s the kind of guy you want in your enterprise.

Courtney Kissler now, you know, was at Nike and now she’s moved on to a startup. Ross Clanton now at American Airlines, Heather Mickman. I mean I can go on on the list, but these are people that impact change in larger organizations that where it’s incredibly hard. And there’s just something about that.

Maccherone: There is an immune system at large enterprises, though, that makes it hard for someone like me to actually be hired and be successful. You know, the idea of disruptive change is not something that enterprises are sort of seeking out ever. And so if you’re sort of like grassroots and you’re sort of like ground up and you get a few pilots going, that’s easy; there’s not a lot of resistance. It’s when you get enough momentum that somebody feels like their domain is threatened that you get the political pushback in an enterprise. And, you know, that’s really hard to overcome. Companies have to be very conscious about that. And Comcast has done a great job of that for me in the past, but I see it in the nature of the enterprise immune system essentially.

Ashley: We probably could take any one of those last three comments and turn that into a whole other dialogue. Maybe we can get some help from the folks in the production department for pulling us back together.

Alan, do you want to close this out?

Shimel: One thing’s for sure, is we’ve got a lot more to talk about on this subject. I’d love to – I invite you all back to continue the conversation. But for now, Ravyn, I appreciate you soldiering through this on audio only. If we do it again we’ll make sure to get that video working. Garima, always a pleasure to have you on here; you bring so much. Larry and John, what can I say, it’s a pleasure to see you guys. I can’t wait till we can see each other in person. I know a lot of us have gotten our shots; it won’t be long. But thank you so much for adding so much. Mitchell, fantastic job.

Most of all, hey, thank you people that are watching this, because that’s why we do this, is for you guys to get something out of it, join in the conversation, let us hear what you’re thinking. Many thanks to Tricentis for sponsoring DevOps Unbound.

Until next time, though, this is Alan Shimel, Media Ops for DevOps Unbound. Have a great day, everyone. Bye-bye.

[End of Audio]

July 30, 2021
EP 11: Unburdening Developers

Every app development need seems to ultimately fall to developers to fulfill. Build in security. Shift testing left. Provide users cooler, more customized experiences at cloud speed and scale. And do it all faster, of course. How do we accommodate the rising expectations without burning out developers? What can other team members do to lighten the load and allow developers to focus on the creative work they require to thrive? In this episode of DevOps Unbound, hosts Alan Shimel and Mitchell Ashley are joined by panelists Adam Kalsey of Tricentis, Tracy Ragan of DeployHub and Justin Hutchings of GitHub to talk about unburdening developers. The video is below, followed by the transcript.

Alan Shimel: Hey, everyone. Welcome to another episode of “DevOps Unbound,” sponsored by our friends at Tricentis. We have a good, nice topic I want to talk to today, and you know, I refer to it as the straw on the camel’s back, but it’s about the ever-increasing load that developers seem to be taking on with technologies and frameworks such as DevOps and Agile and so forth that we’re seeing out here. And we’ve got a great panel to explore this. Let me introduce you to them before we go any further.

Let me first of all introduce you to my friend Justin Hutchings. Justin is with GitHub and he’s gonna tell you a little bit about himself. Justin, welcome.

Justin Hutchings: Thanks, Alan. Yeah, I’m Justin Hutchings. I’m a director of product here at GitHub. I work on some of our security features for developers, you know, whether that’s things like static analysis, security testing, or supply chain kind of features for developers in opensource and the enterprise. Glad to be here.

Shimel: Nice to have you here, Justin. Welcome. Secondly, she’s a frequent guest here on “DevOps Unbound” and other MediaOps videos, my friend Tracy Ragan, CEO of DeployHub. Tracy, welcome, maybe a little bit about DeployHub and you for the audience.

Tracy Ragan: Absolutely. Thank you, Alan, and thank you, Tricentis, for having these and always welcoming me on these panels. I like doing them. So yeah, I’m Tracy Ragan, CEO of DeployHub. DeployHub is a hub of metadata for deploying and configuring microservices. So we do automated configuration management so you know what your blast radius looks like for a microservice before you deploy. I’m also on the board of the CD Foundation, and you might see me oftentimes evangelizing around microservice usage.

Shimel: Yes, we do. And then – welcome, Tracy. And then last but not least as a panel member today is Adam Kalsey. Is that pronounced right, Adam?

Adam Kalsey: It is, absolutely, yes.

Shimel: And Adam’s with Tricentis, but why don’t you tell us a little bit, Adam?

Kalsey: Yeah, so I lead developer relations at Tricentis, and we make software to help people make better software, to test it and make sure they have higher-quality software. And my role is to help make developers more awesome by helping them make better software.

Shimel: Thank you and welcome. And then last but certainly not least is my cohost for “DevOps Unbound,” Mitch Ashley, CEO of Accelerated Strategies Group, CTO here at MediaOps. Mitchell, welcome. Great to have you on.

Mitch Ashley: Always good to be here, and having been part of many development teams, I’m excited to talk about this great topic.

Shimel: Great. Let’s dive into it, guys. You know, when I started devopsy.kinsta.cloud about eight years ago, there was this constant push and pull. Was DevOps too developer-centric? It was all about the developers, the ops said. This DevOps thing is just – it’s all about the developers; we’re not getting our fair share. And you spoke to developers and it was, “You know what? DevOps is too ops-centric. It’s not about the ops. It can’t be everything ops. It’s got to be about developers.” And so there was this, you know, friction about was it too developer-centric or not.

Over the years, though, what I’ve seen is an increasing amount of responsibility seems to have shifted left, as we say, right? And we shift left, and when we shift it left, we’re primarily – we’re shifting it to the developers. So things like testing, for instance, right, have become – developers are probably more involved in testing, or at least kicking off testing and this stuff, than they ever were. Security – certainly the whole rise of the DevSecOps movement has been about shifting security left, and in fact many of the most successful security companies in the DevSecOps space have become successful by making security tools for developers. We’re making testing tools for developers. We’re making tools to look at your blast radius for microservices for developers. Developers are deploying, right, as part of their DevOps focus.

When is enough? When do we break the developers’ back? Not to mention the developers are the highest-paid people in the food chain, right? So by giving them everything, are we giving them nothing, right? Are we stunting their productivity? Adam, being that you’re the new guy here and you actually are in charge of developer relations at Tricentis, we’re gonna ask you to kick it off. Are we putting too much on the backs of our developers?

Kalsey: I don’t think so. I think that the promise here is that we make things easier for the developers. I mean, sure, you have more tasks. There’s more things that you’re responsible for. But the overall effort and the amount of work that has to be done ends up shrinking. If you don’t find out about a bug in your software or a security problem or such until you’re in production and customers are using it, now all of a sudden you’ve got way more work to do. It’s harder. It’s more expensive. It’s more difficult. There’s more involved in doing that. And so by moving a lot of these things forward, we’re letting the developers be more proactive in taking care of them. We’re helping them get things fixed sooner. We’re reducing the amount of toil that developers have to do. Yes, there’s more upfront work, but that long-term toil starts to fade away and disappear.

Shimel: Fair enough. I don’t know if you – you know, I don’t know if I’ve ever heard someone say developers are – you know, it’s making it easier or less for them. Justin, what do you think?

Hutchings: You know, I’ve talked to a lot of companies that are the early stages of their DevOps transformation, and I think one of the most common mistakes that these companies make is they fail to recognize that adopting DevOps requires both technical changes as well as cultural changes. You can’t just get rid of the testers and say engineering owns tests and expect the same velocity. You’ve got to build out the infrastructure to support that, you have to do the training, and you also have to recognize your throughput is going to be less while you’re paying down that sort of debt on getting that infrastructure and getting the automation and getting all of those things.

Companies that fail at the DevOps transformation, they go through sort of the motions without understanding the culture and talking about it. You know, years ago when I worked at Microsoft, the Windows team sort of famously had a one-to-one ratio between engineers and – software development engineers and test. These were folks that wrote test automation. And there was a big layoff, thousands of testers got laid off, and they said, “We don’t need them anymore. We have observability tools. This will be great.” And that was the most naïve statement I had ever heard, and it took a year for the organization to fully rebound, to figure out what the right set of responsibilities were for engineering, and how to support those folks effectively on it.

Shimel: Fair. Tracy, in your roles, both at DeployHub and CDF, you’re dealing with both constituencies here, right, the traditional DevOps teams. But let’s face it: You’ve got dev and you’ve got ops in there, and you deal with both. What’s your take from the battlefront?

Ragan: Oh, my goodness. I could write like a 20-page blog on this. I guess that wouldn’t be a blog. It would be a research project. [Laughter]

Shimel: Or a book.

[Crosstalk]

Ragan: Yes! So I think that in my discussions with folks over the last six months, eight months, I am seeing a lot of SREs really stressed out. So I don’t know if it’s the development teams that are really taking the brunt of this, but if we think about our history and how we got here, we’ve always had somebody really important on a development team who always worked in hand with the operations side of the house to find out what happened, to help with the deployments. Even if they’re sitting there watching the operation first and hit the button, they’re the person that’s always taken this responsibility. And I think the DevOps movement was an effort to push more operations tasks to that trusted partner on the development team, and now that role has turned into an SRE.

I would say five years ago we wouldn’t have seen this kind of stress, but because we’re going through this massive tsunami moving from monolithic to microservices, and we’re in a transition phase, we are putting a lot – what I see is we’re putting a lot more stress on the SRE role. Developers are going to the SRE, operations are going to the SRE, and the SRE is just trying to keep up with the underlying technology that’s wobbling and moving. You know, they’re trying to figure out how to stand up environments easier. They’re trying to implement Terraform or Rancher. They’re doing everything that they can right now to try to keep up, but they may be underwater for a bit longer.

So when we think about where the stress point is right now, I think that developers have pretty much automated a lot. You know, we have tools like Tricentis to really improve the testing, but now we’re int his weird phase where we have these SREs that are trying to straddle both worlds. And they’re struggling.

Shimel: Justin, you know, in your role there – and, guys, I don’t mean to call. If you want to say something, feel free to obviously jump on. But, Justin, I’m wondering – you know, GitHub has sort of a unique view with this.

Hutchings: We’ve been trying to push folks towards more automation, more DevOps, you know, DevSecOps with all the security features we have. You know, again I think culture is the key. I look at anytime you have an unfunded mandate, you know, it’s gonna fail. A lot of times organizations will say we’re going to microservices or they’ll say we’re gonna adopt all these security tools, and then they don’t think about how that actually gets done and they don’t recognize the sort of pain and training and all of the things that have to happen to get folks there.

You know, and I think there’s been a lot of advancement here. You look at the state of observability. You look at the state of CI/CD. Things are getting better, but I think, you know, we in the industry that sort of work int his space sometimes paint such a rosy picture that our customers aren’t prepared for the hardness of the journey that lies in front of them. And they’re looking at how do I get to the magical garden of DevSecOps, and the magic garden is there, but it is not a free entry, you know?

Ashley: You know, Justin, to what you were saying, I think along with culture, part of is a role change that’s happening across the whole organization, right? A couple things are happening. We’re doing not just automation that the developers are doing, clearly SREs. Other test automation, infrastructure as code, you know, those are all development that’s happening to support those. It doesn’t say that every person is an expert developer, but there’s some development, scripting, other kinds of skills that come with it.

So I don’t know if we’re so much loading it on the developer as much as we’re elevating our level of automation and kind of interconnected of the process, and we’re helping people evolve and growing their strengths – let’s put it that way – to also include some software development, or more closely working with people that can. I think that’s part of that cultural change, too.

[Crosstalk]

Kalsey: One of the things that’s actually happening there is as things become more development-centric and, oh, now it’s Toad and now it’s – the teams are making exactly the wrong lesson out of this and they’re learning the wrong thing out of it. You know, one of the things that Justin said earlier is, “Well, we got rid of all the software testers because the developers are doing the testing now,” and that’s one of the dumbest things that you can do as an organization, is say, “Well, now that my developers can do this, let’s get rid of all the experts.” And instead of saying, “What is the new role of this expert and how does their role change?” whether, okay, now they learn to code so that they can expose more of this as code, or do they become consultants to the developers and help them do this right and help them understand how to do this right? Could I remove a lot of those pieces of toil and a lot of that extra stuff from them and let them focus on the things that they’re good at – writing the code, building the code, thinking through the problems – and let me focus on the things that I’m good at – thinking about the infrastructure, designing the infrastructure, designing test plans, thinking about the non-happy path?

And so companies – I talk to companies all the time that say, “Well, we’ve moved – all our developers are doing the testing now, so we got rid of our QA department!” And that is – that’s just frankly dumb. Don’t do that.

Ragan: I think it was an outcome of the Agile movement, right? Because we started preaching this idea of bringing everybody into the same room together to do these tasks. So we broke up our testing departments and we spread them out across all of the teams, as well as some of the operations, which is how the SRE title came about. So, you know, first we siloed and then we tried not to silo. But ultimately it is how do you come up with a strong automation standard across the organization? And how are you gonna implement it? If there’s a testing team that has a group of people and each one is assigned to an application team, or if you have a siloed testing team and a siloed operation team, ultimately we’re still pushing the same product through the lifecycle. So how well do we do that? And maybe Agile confused things for us for a while and kind of broke things up and disrupted what we were doing. So now the task is how do we move forward with where we’re at to make it continue to be easier to get things across the pipeline?

And I have to say that one of the biggest challenges I hear from people is that continuous integration works well; continuous delivery from dev, test, through prod doesn’t work so well.

Hutchings: Yeah, and I think, you know, what you hit on there, Tracy, is really important because, you know, when you have an organization where the engineers are the only thing and engineers are undifferentiated and treated as some sort of fungible resource, I think things fall apart. You know, what I’ve seen in successful organizations is they have some subject matter experts, whether that’s SRE or a data team or an infra team, that are building out things that make the rest of the organization able to scale. And that’s a really good, healthy blend where you’ve got experts that you can lean on, but then engineers are empowered to do sort of the last-mile delivery there.

Kalsey: Yeah. When a company says they’re gonna create a cross-functional team and then they eliminate most of the functions, that’s a problem. I mean, you don’t say I’m gonna have a cross-functional team, so we’re going to bring product managers and designers in and, oh, by – now we just don’t need product managers and designers. So why do we do that with operations, with testing, with all of these other things that are important for software creation? You know, eliminating those and just saying, “Well, the developers will do it,” of course that makes for a lot more work for the developer. The way that this needs to work is keep those cross-functional teams and change their role and bring them further – bring them earlier into the developer cycle. Have them involved in design. Have them involved in figuring out what the product is going to do and how it’s going to do it, not just let’s get rid of them. The options aren’t they happen at the very end of the process or we don’t have them at all. It’s let’s bring them up further along in this process.

Shimel: You know, part of it, though – and I don’t know if anyone wants to admit it – is that there’s sort of this bias that everyone else on the IT team aspires to be a developer ’cause that is the ultimate kind of thing. So, for instance, it was this way certainly in testing, that being a tester was a waystation on the way to being a full-blown developer, right? Or maybe a developer who wasn’t that good, well, he could do testing, right, and that testing in and of itself wasn’t a “profession.” It was really just like a junior developer did testing. And, you know, security people, well, they were always the rebels, right? Half of them didn’t have formal training. But you know, operations, the SRE folks, right, how much of a developer heritage or pedigree is there in what you need to be able to do from an SRE perspective? And so when we put together these “cross-functional” teams, it’s almost with the idea that, well, you’re all gonna be developers now, right? You’re gonna reach your lifelong ambition by being in this DevOps team as part of a developer. And we lose out on the professional tester, on the professional SRE, on the professional security person, right, in this rush to all be developers. What do you think about this?

Ashley: Well, I think we did this to ourselves in the DevOps movement. You know, I remember when I first wrote my first article on devopsy.kinsta.cloud about what is DevOps and I was sort of grappling with it myself, the mantra was, “We don’t need operations, we don’t need test, we just need developers. Code or developers writing code will eat the world.” And it wasn’t outsiders that said that; we said that, or subsets of us said that. And I think we recognized after we got into it, no, as Adam points out, those are exceptional skills. Frankly some of the worst testers I’ve worked with were developers. They were terrible at testing their own stuff. They were great developers, right? But some of it is elevating maybe too much the role of the developer to be the rockstar beyond the platform, if you will – and there are fantastic developers. I don’t mean there aren’t. But there are amazing, amazing test people.

And so let me put it to you this way. When I was running product development at StillSecure with you, who did I listen to? The testers. I listened to them and I asked them, “What problems are we finding?” and I could tell if we were ready to ship, not just by the numbers but by their opinion. And that’s why I really relied on, because they saw it from a different perspective than a developer does. So that’s my opinion about it. Now feel free to disagree. Anybody can take me on and say I’m full of baloney.

Hutchings: So I was gonna say, Mitch, you know, have folks heard the story about – what is it – the IBM Black Team that they developed back in the ’80s for testing where they – basically testing was a sort of poorly-regarded function in the organization? And they ended up creating sort of a hit squad that famously would go and just destroy people’s code. And they created a culture around that, and they created sort of an ethos of that being a special place to work. And I think that’s something we need to find with our developers. Like, if we only celebrate the developers that make the shiny features, we’re not gonna be in a good state. We’ve got to celebrate the developers that can destroy other people’s code with tests or can build something that scales really well or that build great observability things. Like, it can’t all be how many features did you ship; it has to be how much you contributed to the whole.

Ragan: Yeah, and I – you know, I think, Justin, that the cultural discussion is one that any team that’s going through this experience right now that we’re describing, that the developers do feel burdened in some way, they should be looking at how they have built that team. And kind of each role should be a specialist role, and they all have to be an equal role. And if you don’t have that, you really haven’t created an Agile team and that team’s not gonna become a high-performance computing team, because it is about equality on the team and the ability for every single person to stand up – tester, the SRE, the junior developer – and be able to communicate problems as they see them in an equitable way and not feel that they are being judged.

And I have learned that from working in an opensource community. I’m really amazed how well the opensource communities work together, ’cause you have people with all kinds of different backgrounds, and it fascinates me to see how much they can accomplish and how well they accomplish it when they’re not all working for the same company and they’re there because they want to be. So how do you create that opensource kind of nurturing culture within a particular team if it’s built especially around Agile, where we’ve taken operations and testing and developers and, you know, other specialists and put them on the team – documentation, requirements gathering – and how do you make it an equal team? That I think is the challenge to keep the burden off of the developers’ backs.

Shimel: You know, and that – paradoxically, that’s also the big attraction to like cloud-native communities, to these opensource communities, because I think people recognize the – and I don’t mean it in a bad way – the kumbaya kind of spirit that these opensource communities foster. GitHub, quite frankly, Justin, right, if you look at the history of GitHub, that’s probably been one of the huge drivers of 52 or whatever it is million accounts on GitHub, is that sense of community, right? How do you build that sense of community interdisciplinary across your teams within an organization? Adam, you know, though the role says DevRel, it’s really TeamRel, right? And you know, it’s got to be part of your mantra there, part of your job description, isn’t it?

Kalsey: Yeah. One of the things that is interesting about when you look at opensource is the person who wrote the documentation is a hero, because nobody wanted to write the documentation. The person who wrote the tests is a hero because, wow, they’re writing the tests, and that was a lot of work and I didn’t want to do that amount of work. But then when we go into a company we say, “Well, we’re gonna hire a couple of QA people to do the testing because they’re cheaper than the developers.” And, well, yeah, maybe if you’re hiring bad QA people they’re cheaper than the developers, but if you’re hiring good testers, they’re as expensive or more expensive than the developers.

And so in the opensource world we look at it and we go, “These people are peers and equals,” and then we come into a company and we say, “They’re second-class citizens, and we’re gonna hire a couple of tech writers ’cause they’re a lot cheaper than having the developers write the documentation, or we’re gonna hire some testers, or we’re gonna hire some ops people ’cause the person to watch the dashboard is way cheaper than having a developer do it.” And we’ve seen it happen with the SRE role where, when we elevate that and that becomes part of the development team, the other – SREs are expensive and hard to find. That same sort of thing I think needs to happen in other roles on a development team in order for them to stop being that second-class – and also to stop having developers be the ones that have to do all of this work. If you look at it and say, “Well, the test writer or the doc writer is a cheaper person and we’ll just hire a couple of them,” it’s more likely to fall on the developer to do the hard work than to hire the good people and have them do the hard work.

Shimel: Fair enough. Mitch, what do you think on that?

Ashley: You know, it’s a really good point, and I think part of what you’re saying, Adam, is it’s a specialty. You think differently to be a fantastic tester, not just testing but the whole test strategy and how to do all the different kinds of testing that’s involved, and then of course how do you automate that and do you do that yourself? Do you work with part of the dev team to do that? Same thing for SRE, right? It takes a different – someone who thinks about it differently. And it’s sort of that cognitive diversity of teams that we value from a diversity standpoint. We value that also because those roles also require specialty skills. You know, someone who’s designing – and, Tracy, you could explain this better than I could, but someone who’s designing a cloud-native architecture and thinking about the microservices and what’s gonna constitute a microservice, how we’re gonna communicate between them and how we’re gonna secure them, that’s a different kind of thinker than how do I break all that stuff. That’s really nice that you did that, but watch how fast I can break that. That’s fun, and that’s where people really enjoy working together. But I think that’s part of where you’re elevating, too, Adam, is you’ve got to value all the skills, and they may be expensive, they may be not quite as much, but they’re needed. They’re part of the process.

Shimel: Interesting. You know, Justin, we’re talking about testers. We’re talking about SREs. We’re talking about developers. Once again, no one’s talking security. Are we just sentenced to be forever the outsiders here? What’s it gonna take?

Hutchings: Most of the security people that I talk to take that as a challenge, you know? We obviously want developers to go and embrace security as part of, you know, their everyday job descriptions, but folks in AppSec and especially in the security research community, they’ve always felt like kind of scrappy outsiders that are going to shine a light on things that other people didn’t notice. And so I think even when we have very security-minded, security-motivated engineering teams, security is a discipline that’s got to stand apart, because we want to have that sort of trusted auditor role or that, you know, lookout for – whether it’s bug bounty reports or stack analysis findings that no one looked at. Like, someone’s got to be watching those, ’cause developers, especially in the world of microservices, they might work on a thing, finish it, and then go somewhere else. But the security world changes all the time. If you walk away from that microservice and all of a sudden we learn about a new category of security problems and no one’s watching it, well, that’s a problem now. And so security has a really special role in terms of being the stewards of all the things, even when they’re not directly accountable for the functionality of it. And I think that’s a special thing that we should celebrate.

Shimel: Okay, we’re special. But you know, as special as we are, yet we’re still seeing security tools for developers, right? That is a bona fide trend in our industry, and at the same time we’re also seeing – when we talk about the load on developers, we’re still seeing an extreme need for more security people, right? We just can’t produce enough qualified security folks, it seems. If that’s the facts, right, there’s no getting around it. What else is there to do but put this load on the developers? How else do we deal with it, automation? I don’t know.

Hutchings: You’re absolutely right, and I think the key for tool spenders like GitHub is to figure out how to give developers enough information that they can be successful but not overwhelm them, because security tools, you know, there’s a risk of false positives. If you give them too many, they will ignore them. And so, like, as a tool vendor, I have to be extremely careful to make sure that I don’t burn out developers on the things I think are important, ’cause they might not find them as important as I do.

Kalsey: The other thing those vendors have to do is make less work, eliminate work for the developer, not cause more, not give them more things to pay attention to, more work to do, more administration to do, but look for ways that you can remove that work through the tools.

Ragan: Yeah, so we have a challenge facing us when it comes to automation right now, and the challenge has to do with these imperative pipelines that we’ve been running for the last ten years. Sometimes you have a Jenkinsfile that is pretty complicated, and we’ve been running them, we copy them, and we use them. It’s imperative.

The problem with it is because it’s imperative. We are seeing a shift now to moving away from these imperative pipelines to where it’s more declarative and event-driven, which means that it should be easier then to take these older – if you had a purely declarative pipeline based on events, think about how easy it would be to say, “Okay, we’ve decided on this set of security requirements to add to sort of the workflow steps for the testing workflow, for the development workflow, and we’re gonna plug those events in.” The reason why we don’t do that now is because it’s too hard. [Laughter] It’s just hard, because we have to take these old imperative pipelines and try to modernize them. And without having that declarative framework, these kinds of tools, even adding continuous testing, even adding continuous deployments all the way to production, it’s hard and it takes a lot of work, and you have to change the tires as you’re going down the freeway in an imperative model. The faster we can get to an event-driven process in our CD pipeline where we can declare them and we can automatically generate what we need to do in a workflow, the sooner we’ll be alleviating some of the pain that the developers might be seeing, the sooner we can bring in better testing, the sooner we can bring in the DevSecOps conversation, and in our case automated configuration management.

This has always been a problem for any of the tools in the CD pipeline to start being plugged in, because the imperative pipelines that we are dealing with on a daily basis, nobody wants to touch them. So we’re gonna have to disrupt that first, and when that gets disrupted and people start moving to something more declarative, we are gonna see an increase in adding all of this kind of automation into the CD pipeline.

Ashley: I’m curious, Tracy. Is it the complexity of the pipelines that – the way they’re defined, is that why people don’t watch to touch them, or –

Ragan: It’s the number.

Ashley: The number, okay.

Ragan: You’ve got – literally I’ve seen people with thousands and thousands of workflows ’cause they’re running one for every single application version. And to look in that and say, “Oh, we’re gonna change all of these,” because it’s not templated so you can’t just change it in one place and it’s just gonna trickle down, because they’re all these very statically scripted processes. And for whatever reason on the distributed side of the house – when I say distributed, I’m talking about everything except the mainframe – we have relied on heavily scripted processes, and we’ve scripted ourselves into a box that we cannot – we’re not Agile. We cannot easily update these pipelines to include security in them.

We talk about it at the CD Foundation often, trying to elevate security as a first-class citizen in the CD pipeline. But that means that everybody’s got to start updating all those CD pipelines, and they’re not templated. They’re all statically scripted so you have to address one at a time. And as I think Justin might’ve said, you’ve got to plan to pay for that. So we’re in a box, we’re in a corner, and we have to somehow get ourselves out of that corner in order to alleviate these kinds of problems. And being Agile at the CD pipeline level means that we’re gonna have to dump these imperative processes.

Hutchings: And I think you’re exactly right there, Tracy. One of the challenges we also see with security is this desire to start slow so that you don’t overwhelm people with too much, and if you don’t have a centralized control plane that everyone pulls from, you know, as you start to introduce these security tools, you can’t dial up the difficult level as you go. You have to hit those distributed CI pipelines every single time you want to go and update the difficulty, and that’s not tenable. I mean, you know, there are services out there – I think GitHub with Actions is doing a pretty okay job with allowing some centralized management, but it’s a spot that we all have to invest in because, you know, that control plane for specialists like security to go and help inject things is just not as mature as it needs to be.

Ragan: And the CD Foundation has started an events working group to start having this discussion. Right now they’re talking about just vocabulary, but I believe that will grow into a bigger project in the future so that we can start thinking about how you can have that event-driven process so that you can dial it as you need to, right, as oppose to having them statically scripted in all of these imperative pipeline processes.

Shimel: So, guys, we’re running low on time. Just one other topic I wanted to bring up. You know, in looking at this problem, though, I’m reminded of when I read The Goal, right, which is kind of the book upon which Gene Kim based his Phoenix Project on. The Goal is about more manufacturing; The Phoenix Project was IT. But really it talks about bottlenecks, resource constraints. And the story of The Goal – and I forgot if it was the first rule, the second rule, whatever it was – is as soon as you fix one or get rid of one problem, the next problem down the line shows itself that maybe you didn’t see because the problem in front of it was hiding it camouflaging it. In trying to make life easier for our developers or take the strain off of that point, are we really just going through this goal process of discovering the next bottleneck? And maybe that’s what – maybe that’s the secret to life. Life is just about going from one bottleneck to the next, right, and discovering what’s downstream from there. There’s something Zen about that. But is that what we are sentenced to for sure, or is there a way out of that?

Kalsey: I like to say that when designing large-scale distributed systems you can’t remove your scalability or your failover problems; you just move them to a different part of the stack. And we do that a lot with development teams as well, is we don’t remove the problem; we just move it to a different part of the process and move it around.

I think one of the ways out of that is one of the tenets of Agile is let the teams make the decisions. Hand them the problem and let them decide how to organize, how to do these things, how to make stuff work. You know, if management is pushing down on development and saying, “You’re now going to be the testers,” of course we’re causing more work and more problems and more things for them, and they’re going to become the bottleneck. But if we let the teams figure out – they’re the ones close to the work, doing the work. Let them make the decisions about how we’re gonna do this and what they need to make it happen.

Shimel: Fair enough. All right –

Hutchings: Yeah, I was gonna say I like the term incrementally correct. You know, Alan, your thinking about bottlenecks is absolutely true. I don’t think there’s ever a world where we will get to 100 percent panacea, but we can hope to get a little bit better all the time, and that’s the journey that we’re all on, right?

Ragan: And failing fast.

Shimel: _____ continuous improvement, and failing fast – like get to know –

Ragan: And allow your developers to fail and not – or your testers or your security. It’s okay to fail, because then you learn what you need to do. Failing fast is part of the learning process.

Shimel: Fantastic. All right, guys, we are out of time. I want to thank all three of you for joining Mitchell and I on this episode of ” DevOps Unbound.” Thanks to Tricentis for sponsoring ” DevOps Unbound,” as always. We will be back in – not next week, the week after with a new, fresh episode, as well as check out – we’ll have a new ” DevOps Unbound Roundtable” open to the public for questions, and you can find out about that at DevOpsUnbound.com or on our Digital Anarchist site or devopsy.kinsta.cloud frankly. Look under – it’ll be under webinars and you’ll find the Roundtable there.

Until then, though, Tracy Ragan, as always, it’s a pleasure to have you on. You bring such enlightenment to the whole discussion. Thank you.

Ragan: Thank you, Alan.

Shimel: Thank you. Justin, again, always nice to have you on one of our panels and discussions here. Many thanks for coming. And, Adam, thank you and good luck in – you know, I know you’re relatively new in your position there at Tricentis but it looks like they made a hell of a hire. Good luck, and come back and visit us again.

Kalsey: Would love to. Thanks a lot.

Shimel: All right. Mitch, you want to take us home?

Ashley: I just wanted to thank our panel as well, and I love the incremental progress idea, and it’s always getting outside perspectives sometimes, too, that’s helpful. So I think we’re here to help each other in this process and not one of us has the answers.

Shimel: Right on. All right, this Alan Shimel for “DevOps Unbound.” We’ll see you on the next show. Bye-bye, everyone.

[Outro music playing]

July 29, 2021
EP 10: Observability & Continuous Improvement

Observability at enterprise scale brings with it additional application life cycle management requirements. Success requires knowing which changes result in improvements and which don’t produce positive results and how that information must make it back to the DevOps team through feedback loops. In this episode of DevOps Unbound, Alan Shimel and Mitch Ashley talk with Kurt Chase, senior director, global release management at Tricentis, Charity Majors, co-founder at Honeycomb and Ravyn Manuel, senior application manager at the Smithsonian as they discuss observability and continuous improvement. The video is below, followed by a transcript of the conversation.

<span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_start"></span>

Alan Shimel: Hey, everyone. I’m Alan Shimel and welcome to another DevOps Unbound. We have a great panel in store for you today here on DevOps Unbound. Our topic is observability at a broad level. We’re going to jump into some of this. Let me first of all introduce you, though.

First I want to introduce you, of course, to my cohost of DevOps Unbound. He’s the CEO of Accelerated Strategies Group, CTO here at MediaOps, my friend Mitchell Ashley. Hey, Mitch. Welcome.

Mitch Ashley: Alan, thanks. Great to be here.

Shimel: All righty. Let me introduce you to our panel for today. We have a great panel. I was so happy that this came together. First, coming at us from the mountains of Virginia by way of the Bronx, Ravyn Manuel. Ravyn, why don’t you introduce yourself and where you work?

Ravyn Manuel: Absolutely. My name is Ravyn Manuel and I work for the National Museum of African American History and Culture, which is part of the Smithsonian. I am the Senior Applications Developer and DevOps Engineer. And thanks for not putting me in West Virginia. I appreciate that.

[Laughter]

Shimel: Okay. I don’t blame you. Next up we have the one and only Charity Majors, CTO and Co-founder at Honeycomb. Charity, a little bit about yourself, maybe introduce to the audience?

Charity Majors: I am an ops nerd. I have been on call since I was 17 racking hardware. I no longer rack hardware anymore. Hardware doesn’t exist, as far as I’m concerned. Instead, I’m trying to save the world through better observability.

Shimel: Excellent. And then, last but not least, we have Kurt Chase of Tricentis, Tricentis Releasing. Kurt, why don’t you introduce yourself?

Kurt Chase: Hi, everyone. My name is Kurt Chase. I’m currently Head of Global Release Management at Tricentis, based in Northern California. Nice to join you all today.

Shimel: Nice to have you here, Kurt. All right. So, team, panel, our topic this week is observability. And given that is the topic, I feel compelled to start with Charity because, Charity, in many ways you and the Honeycomb team are responsible for all this buzz around observability. Tell us about it.

Majors: Yeah, a little bit. When we started the company a little over five years ago it was not a term that was really in use – used in tech. Now – which is not to say that we invented it. Far from it. But it has this long and impressive pedigree in mechanical engineering, specifically control theory. And in control theory observability is the mathematical tool of controllability.

So, we started building this startup, Honeycomb, and we knew that what we were doing wasn’t monitoring. We knew that we weren’t reactively monitoring. We knew that we were – it was about instrumenting the code, getting at it from the inside. But for the first five, six months we couldn’t figure out what to call it. It was really hard to figure out how to talk about what we were doing. And then it was mid-July, midway through the first year when I remember Googling “What is the definition of observability?” And when I read it it’s all about “Can you understand what’s happening on the inside of the system just by looking at it from the outside with no prior knowledge?”

And a lightbulb just started going off. It was like “Ding, ding, ding. That’s what we’re trying to build. That’s exactly –” if you translate that into software terms, what it means is can you explain any system state that – any state that your system can get into without shipping new code to handle that state because shipping that code to handle that state implies that you had prior knowledge. So, it’s all about the unknown unknowns instead of the known unknowns. So, yeah, that’s my story.

Shimel: And you’re sticking with it. So –

Majors: Yeah. I mean, it really encapsulates a lot of the problems that people are increasingly having, that there’s a lot that precedes from that definition of “unknown unknowns,” like you need to have high cardinality, high dimensionality, blah, blah, blah, all this stuff. But those are the things that speak to the problems of microservices and multiple storage systems, et cetera.

Shimel: I get it. So, Ravyn, this is your life. How does this observability and what Charity has describe, the fact pattern or the problem pattern that she came to call observability, how does that in your life, day to day of what you’re doing, how does that kind of manifest itself?

Manuel: Thanks, Charity, for actually providing a definition because I was like “Okay, are we supposed to define our own meaning of observability?” So, you’ve given a baseline. And I like that you said it’s not monitoring because I can actually feel the nuance. There’s a nuance between monitoring and observability. And it’s – what’s really interesting is that that’s a problem that I’m trying to solve today at work because we’ve got – and – because we’ve got the black boxes. So, you’ve got observability but when you have software there’s a lot of black boxes. You’re not supposed to be asking what’s going on inside the code because you’re not supposed to know what’s going on inside the code. But when you get a – you put a signal in – I was in the military, so I did electronic repair – so, you put the signal in and you don’t get the right signal out but you’ve got a black box. How do you know what went wrong there if you can’t actually observe what’s going on inside of there?

It’s one thing to know the state of the software when you are the developer. So, as a developer, yes, I know. But when it comes to being the product owner or the project manager or a stakeholder and you’re trying to figure out what – “Why is this doing this?” or “This is not what I wanted,” actually is how that would come from that end, is “That’s not what I wanted.” So, how to get the stakeholders to get what they want when you can’t actually tell – I can’t tell what the code from my coworker is doing? So, in a way observability is a little bit – it is abstracted from monitoring because monitoring is telling you what it’s doing but it isn’t telling you the state. And so, it’s a problem that I’m trying to solve actually today.

Chase: I was going to jump in here and say from my experience although monitoring, logging, reporting, alerting, they’re not observability, I think they are fundamental foundational pieces that you have to have in place. You can start observability without any of them, but I’ve found that the more you can get your monitoring and reporting and even get some alerting going, it allows you to then start observing the system. And for me, I know at Splunk and now here again at Tricentis I’m really trying to connect all the pieces because it’s not just about the applications, but I find that some of the DevOps engineers, they really help with observability, especially the ones that are super inquisitive, and they don’t just – when a result appears or something happens they don’t just take it at face value. They’re inquisitive. They dig in. They try and find the root cause.

And I think that’s key with observability, is taking all the monitoring, reporting, alerting you have, taking the feedback from your customers, all of that comes together, I think, to really help you start to define those unknown unknowns that Charity brought up. I think it becomes very complicated. And then I think I’d love to talk about how AI and machine learning now can start to learn some of these things, because I think we get really good at repeatable incidences, observing those and then accounting for those in our systems, but it’s the unknown unknowns, how do we deal with those? That’s the real interesting part to me nowadays for sure.

Majors: It is, I think, the interesting part. And I also think that AI ops is mostly bullshit and it will be for the foreseeable future.

Chase: For a while. I agree, yeah.

Majors: But I wanted to circle back to what you said about logging and everything, which is – so, there’s a technical definition for observability which has to do with high cardinality, all this stuff, and then there is just understanding your systems. And I absolutely agree: You need some alerting, you need some monitoring. I think there’s a question is – of – it log files? Is it metrics? That’s kind of irrelevant for just data. What’s important is can you understand what’s happening in your system when you need to understand it?

And I do think it’s important to sort of differentiate the technical definition of observability from just the tendency that we have to, say, observing things because there’s a very rich and mature and well-established set of best practices for monitoring, which are in many cases completely mirror opposite from the best practices for observability.

For example, with monitoring it’s like for years we’ve been saying, “You should not have to stare at graphs all day. The system should inform you when you need to – when there’s something wrong.” Super canonical best practice. With observability it’s exactly the opposite. It’s – we’re saying that most problems that happen are so subtle that you need to be looking for them. You don’t – you can’t page everyone at the threshold – you would drive them nuts if you were like “Oh, every little thing is going wrong. Bing, bing, bing.” You’re just – you would drive people nuts and they would quit. But when you’re an engineer, when you’re working on this endpoint or when you’re trying to solve this problem you should absolutely be looking through, asking questions, iterating on them, and looking for the trails in your observability tool at that very fine grain level.

Chase: I agree.

Shimel: So, I hear what all three of you said and it’s funny how everyone brings their own baggage to this issue.

Majors: Oh, sorry.

Shimel: No, no, it’s not – don’t be sorry. I love it. Here’s, I think, fundamentally – for those people out here who are watching this whole – I’ve got a lot of people out here who are DevOps people, who are ops folks, SREs, cyber folks, and their heads are spinning because I think there’s a little bit of what Kurt said to them, which is they don’t do – they do a shitty job monitoring, quite frankly. We don’t do a great job of it. People started throwing this AI ops word around, and there are a lot of companies raising a –

Majors: Yeah, there are.

Shimel: – crap ton of money around their AI ops stuff. And I never really understood what was real about it and many times it just sounded like another word for monitoring to me. But whatever. There is no AI in it.

But – and now, look, over the last year, year and a half, two years we’ve seen observability come up here and they don’t get the difference. Ravyn, I think you do because quite frankly your black box analogy with electronic in the military, that is – you’ve got to figure out what the unknowns are from the unknowns. It’s not just getting all these signals that are known and then recognizing patterns and putting some automation behind it. That’s not observability. I’m sorry. Not in my mind, anyway.

So, I think number one is we need to clearly define it. And Charity, you guys, you are a major kind of evangelist for this: “What is observability versus AI ops versus ALM,” which is before AI ops, and so forth. And there are SREs who are living this and DevOps teams who are living this every single day out here who haven’t got their heads around it. Ravyn, I always look to you as someone who gets it. Where do you fit on the – where do you kind of see it on the – let’s say there’s a scale of ALM, AI ops, blah, blah, blah, observability. What do you think?

Manuel: I feel like I would be very – most close to observability because, again, all – anything with ops in it, when you start doing DevSecOps and DevOps, all of those ops just become marketing terms after a while. But observability is – because it’s not quite tangible it’s actually more real, because it’s real – because it speaks to exactly what’s coming, what’s going on when you’re trying to get continuous improvement, because you need metrics to show and prove that you’re doing continuous improvement. But how do you get there? What is it – what are the steps to get you there? How are you alerted that you are not improving or that you are going backwards?

And so, when – I feel for me as far as observability I’m at the cusp, because I didn’t actually know that the word existed until I found out about the talk. I didn’t know that that had been defined. So, it’s really cool to know that I can now call it observability. It’s because it’s that thing that I can’t – there isn’t a tool to actually help me – actually help me know what it is that I’m looking at all the time. Not all the time. There are some tools that help me some of the time but not all the time. And it’s the times that I don’t know that’s when we are stuck, because it could be impacting the way – the system performance or the team’s performance.

Shimel: Charity, I think you might have a customer there. It sounds like they need a tool.

Majors: I feel like one of the key differentiators here between observability and AI ops and stuff is what it tries to center. With observability we are not – AI ops are like “Eh, we’ll tell you what to look at. You don’t need to understand your system. We’ll understand your system for you. We’ll tell you what to look at. We’ll tell you what it means. We’ll tell you what’s important.” With observability it’s the opposite. I think that we need to look to solutions that let computers do what computers are good at and let humans do what humans are good at. Computers are good at crunching lots of numbers but they can attach meaning to something. Only human beings can attach meaning to something. If I see a big old spike of red that could be good, that could be bad. You don’t know. There is no meaning until a human has attached them to it.

And with AI ops, AI ops only really works if the corpus of data that you’re training it on doesn’t change. So, if you’re actually shipping software every day you’re fucking up your corpus. This – it’s kind of incompatible with modern software development. I have many thoughts and feelings on AI ops. God bless them, but mostly I think that what they’re trying to do is take the humans out of the driver’s seat. And I understand why, because the C-level, CTOs, CIOs, whatever – there’s something that kind of blew my mind when I realized it a couple years ago: They trust vendors over their own people. Their employees come and go; vendors are forever. So, whenever a vendor comes along it’s like you give them tens of millions of dollars; your people don’t have to hold all of the wisdom in their heads. You can replace them. They’re fungible. The tool will tell you; the tool will be the source of truth.

And I just fundamentally reject – I’ve never seen that work well. I think it’s a shitty way to treat your people. And I would prefer to take the –

Chase: And I would not agree with that statement, quite honestly. I’ve had leads that are –

[Crosstalk]

Shimel: All right. “Mr. Vice President, I’m talking.”

[Laughter]

Sorry.

[Crosstalk]

Majors: – and then using our tools to make them better, to impress them, to give them a better sense of where they sit in time and space in the history and what’s happened before and what they’re – they should be wearing grooves in the system as they wear – as they use it in the [inaudible].

So, I think that it should – if you’re centering a person it’s observability and if you’re centering the – it’s AI ops. Sorry. Go ahead, Kurt.

Shimel: It’s okay. Kurt, your turn.

Chase: No, I was just – in my experience the leads, I’ve had – I’m a senior level director now; it was the same at Splunk – I did not encounter that where my leads trusted vendors more than myself or the people on my team. I was surprised to hear that, quite honestly, Charity. I have not experienced that.

[Crosstalk]

Majors: It’s not universal. But it’s – there – it exists quite a lot.

Chase: Yeah, that’s too bad to hear that. That is seriously wrong and that’s going to go against everything we need to accomplish internally.

Majors: Yeah.

Chase: Yeah, if –

Ashley: Have fun attracting great people with that mental model.

Chase: Yeah, yeah, yeah. And the other thing – I’d love to hear your opinion on this, Charity – we have very complicated systems we’re running. The source code management, the build systems, the agents. I’ve found that it always is very helpful to distill it down into discrete parts, try not to blow the ocean at once. Let’s start with the source code management system. How are we observing that? What are we doing here? And would you suggest that’s a great way for people to get going? I mean, when you try and look at everything – you have to look at everything for sure, but to get started it seems like find some of the most important pieces. Source code management is a great one. The crown jewels. What’s your suggestion there for getting started and –?

Majors: Source code management is important but it’s never – it’s almost – it shouldn’t almost ever be in the critical production path. And when people are rolling something out I always advise them to find something that’s – don’t waste your time instrumenting _____ observability, lights, and backwater to the system. Find the thing that you’re trying to understand and fix now. Find the thing that’s broken. Find the thing that’s causing you pain. Because it should be an order of magnitude better at answering your question, observability should, even than monitoring. I don’t think you can look someone in the eye and ask them to switch up what they’re doing and to use something else unless you can tell them it’s going to be an order of magnitude better than what they’ve got now because the costs of transition are so high.

But with observability you should be able to go from instead of just the aggregate where it’s like “Well, I see a spike but I can’t tell you what it means or how they’re different from –” with observability you should be able to go exactly those rows, in which ways are those events different from all the baseline events, and very quickly go “Oh, I have a much richer view of my system now.” And if – once you’ve got that you should be able to explain way more about what’s going – so, we see this all the time when people are running out – when they’re rolling out Honeycomb. They start rolling – they roll it out to the service and they’re like “Oh, God, there are problems there. Did you know that this is broken? Oh, there –” and we’re like “Yeah, yeah, yeah, but you’re just picking up the rug. It’s been there – it’s been that way for as long as you know.” And they’ll pick up another service and the problems – I’m like “Yeah, but these problems have existed – you just had no idea because we were dealing with [inaudible].”

So, I feel like the way to pick up momentum, the way to incentivize people to get excited about this is to look for the biggest pain point in your system and –

Chase: That’s awesome.

Majors: – then patch that with observability.

Chase: That’s great advice. I agree.

Manuel: Charity, a question. Does – can observability actually take in account process? Because you’re talking about if there’s something wrong in the system itself, but sometimes the thing that’s wrong with the system is the process.

[Crosstalk]

Majors: Well, everybody is not about process. And you’re totally right. That’s often – that is not something that you should ever ignore. A lot of the biggest problems are process problems. But observability is not the tool for that.

Shimel: And that’s good to know too because quite frankly – and Mitchell knows this too – we live in a silver bullet world where people think the latest shiny trinket is the cure-all.

[Crosstalk]

Majors: – everything, you should –

[Crosstalk]

Shimel: No, observability, yeah, it’s not going to cure Covid either.

Majors: No.

Shimel: But –

Chase: And that was the point I was trying to make initially. I think I said it wrong. It’s not that monitoring and reporting, logging aren’t foundational; they’re complementary. You kind of need them all. If you’re running modern stacks – I said that incorrectly at the start – it’s not foundational. There’s – you need them alongside each other.

Shimel: They’re adjacent. They’re not –

Chase: Yeah, absolutely. Absolutely.

Shimel: They’re not – it’s not a pyramid per se.

Chase: Right. Exactly.

Shimel: I think it’s part of your stack horizontally.

Chase: Yes. Yes. And I’ve found –

Shimel: Can I – we haven’t’ – go ahead, Kurt, I’m sorry.

Chase: I was going to say I’ve found with especially new individuals and new people to the DevOps team and the DevOps world, getting them involved in the monitoring and saying, “Hey, this is how we start monitoring,” that’s a great place to start. It’s almost entry level for observability, if you – I’d love your opinion on that, Charity, too.

Majors: Well, I really think that everyone who writes and ships software should be on call for their code, no matter how –

[Laughter]

Chase: Yeah, me too.

[Crosstalk]

Majors: Software ownership. And so, yes, that’s very empowering, I think, to show them how to find the trail, the alert that’s paging them, trace it all the way to the system and fix it. That’s something that as engineers, we feed on that. That’s exciting to get to know how to run your ship better. Yes, absolutely.

Shimel: Well, but – so, you know what? Mitchell and I started a company – or, were two cofounders of a company called StillSecure 20 years ago. And that was one of the things. Mitchell ran the engineering team and that was one of the things he instituted, I remember, was that every developer had to take a turn on the help support desk, level one, answer the phone. “Hello, Mr. Customer.”

Majors: It builds empathy.

Shimel: And try to explain it because they had to answer the alert. And even when they weren’t on that level one our escalation procedure was if it went to level two or three it was the developer who did the level two and three support.

Majors: I will say that, yes, and I do think that every engineer should be on call for their work, but you have to couple that with it’s management’s job to make sure that that doesn’t suck, that it isn’t _____ for them, but it doesn’t just kill you with [inaudible]. It is management’s absolute fucking responsibility to make sure that they’re given the time to – away from project work to fix these problems so that they don’t recur, so that they get enough sleep. I think it’s reasonable to ask any engineer to be woken up two or three times a year for their code. That’s it. Because more than that, you’re burning people out hardcore.

Manuel: Been there, done that.

Shimel: You know, Charity, Ravyn –

[Crosstalk]

M : I have a question. I have a question I’d love to ask all of you. I’m thinking about observability has been elevated because of, in part, the world that we’re living in. Software architecture is changing. We’re dealing with microservices, all these much smaller things that are interconnected to make software. And at the same time as we’re changing the pace at which we release software, not only is it unknown unknowns, it’s like a river – Riven –

Shimel: Ravyn.

Ashley: – we’re in the river of the changing software. So, you’re trying to understand all of these things at the same time. It seems like it’s the right time for observability to be the thing to be working on because ops has to be elevated to the same level of the same kind of architecture and processes that we’re using across the – in the life cycle of software. Do you agree with that or not? It seems to me we’re in the right place, right time.

Majors: Absolutely. I mean, I feel like the first wave of DevOps transformation was all about “Ops people, learn to write code.” And we’re all like “Eh, okay.” But I feel like the pendulum has swung. I think now it’s about “Okay, software engineers, your turn.”

Ashley: You’ve got to do ops.

Majors: I have to learn to write operable services and to learn to maintain them and get called for them and – because what microservices has done, all of that complexity and logic has to be bundled up in the app. The app.

Manuel: Exactly.

Majors: And now it’s like we’ve been thrown to the four directions of the map. And what that means is that so much of understanding and operating your application has been pushed into the realm of operations and systems. It used to be that if all else failed you could attach a debugger to your app and just step through, see what was happening. Well, you can’t do that anymore now because the process died and jumped to another service, and another service. And part – the way that we instrument for observability is all about packing up the context of the application and shipping it along with it from hot to hot, one event per request per service, so that you can slice and dice and reason about your systems in a way that you used to be able to.

Ashley: Doing that really changes the design. I’m thinking about Alan’s story of what I did back at StillSecure. What prompted that was when a customer called in and said, “I’m sitting in – we’re using your project and this dialogue box popped up and it says ‘An unknown error occurred’ and that’s all it says. What do I do?” Well, nobody knows what to do with that. So, we all just said, “Look, everybody does support in that case. Everybody does ops.” And that changed: “Okay, we have to figure this out.” And suddenly that was the problem of “How do we make this easier to support and easier to resolve issues with?” So, it does affect behavior and design or product.

Majors: Yeah, absolutely.

Manuel: I think microservices is – will actually do good for observability because if you do microservices correctly then each service is atomic. It does one thing and one thing only. It does that thing good. So, if the unknown is not taken out of it because you know exactly what this thing is supposed to do –

[Crosstalk]

Majors: Oh, no, honey, the unknown is never taken out of it. The unknown is never –

[Crosstalk]

Manuel: In my ideal world it is.

[Laughter]

There are no more unknowns –

Shimel: In Ravyn’s world, yeah.

Manuel: – in Ravyn’s world. Exactly.

Shimel: Yeah. But I’m going to tell you something. The flipside to this, though, too with microservices, and I think we’re starting to see this now, is can you have too many microservices running? Does it – is the complexity load in there such that observability almost becomes meaningless or impossible?

[Crosstalk]

Majors: I don’t know that complexity can every really be created – no, it typically is not meaningless. It’s your only hope to understand this shit if that’s what you’ve done to yourself. But complexity can only be moved around. It’s just a question of tradeoffs, like which set of tradeoffs – some folks have gone whole hog on microservices and then they’ve found the edges to those design _____. But –

[Crosstalk]

Chase: What I find – I was going to say what I find challenging is both at Splunk – Autodesk was interesting. So, to your point, Alan, we used to make engineers buy bagels. You break things; you have to buy bagels. And so, we had a lot of bagels brought in. But both at Splunk and here at Tricentis we have a world where we’re supporting large enterprise applications and delivery of enterprise apps. We’re also – obviously all the microservices – and trying to apply these techniques across the board is very interesting, very challenging.

[Crosstalk]

Majors: Oh, yeah. There’s no one-size-fits-all.

Chase: There’s not. There’s not. And almost to the point you made, Ravyn, observability almost has to be customized for what you’re observing. It’s – I haven’t had a chance to run Honeycomb yet, but for me my experience has been there’s always those little idiosyncrasies about how we run our ops, how we –

[Crosstalk]

Majors: There’s no such thing as a perfect _____. No one else can do it for you. It can be – it can get you started. It can give you a boost. But at the end of the day somebody’s got to fucking understand your software and roll that into your instrumentation. Nobody else can do that for you, let alone _____.

Chase: And that’s where with AI ops, I won’t say it’s complete shit. I think there is some value there. Some of it is just about automating common occurrences. And maybe that’s not valuable but –

[Crosstalk]

Majors: – it’s an AI but they’re just using that to make – to raise lots of _____.

Chase: Oh, sure. Okay. Yeah.

Manuel: Didn’t work.

Majors: Like, “Oh, I have an AI too.”

Shimel: No, but we – and you’re right, AI is a money term. But ML ops is something that’s caught on a bit, more than, I think, than AI ops. And I think that’s really at the heart of it, which was to recognize repeatable patterns or recognize patterns that we can respond to.

[Crosstalk]

Majors: Although, I would still challenge – I think in far too many cases people are reaching for the complicated solution instead of fixing it at the start. They’re like “Ooh, I’ve got 50 bazillion e-mails so I – and all of that disk space so I need AI to solve it” instead of turning off the fucking e-mails and setting up a different kind of alert or some automatic _____. Just – is this really a problem we need to solve? Or could we solve it better by just not having it?

Shimel: There is that.

Majors: Sorry. That’s my last bitch.

Shimel: Do we need to solve every – and I think Ravyn – or, not Ravyn – Charity, you were getting at it before. Do we need to solve every problem?

Majors: Is it the most important problem? Is this going to be the biggest bang for our buck?

Shimel: Yep. I mean, we used to have a –

[Crosstalk]

Majors: – or the _____. Like, senior engineers write software but very senior engineers don’t need software.

Shimel: Yeah, no, I mean, look, when Mitchell and I were doing the security company we had a product, “Vulnerability Assessment of Management.” VAMP. Internally we called it the Bad News Generator because that’s what it was.

[Laughter]

It was the Bad News Generator. And the problem was we would go to an organization like Ravyn’s and it would pop up a few thousand vulnerabilities, and the security or the ops people would be like “What am I going to do?” And “Don’t worry about 95 percent of them. Let’s just worry about the top five percent.” And it was like – it was employment insurance, was what it was, because there would always be more vulnerabilities for us to find next year.

But that being said, it really was the case where you’re not going to fix everything. We don’t have the time to fix everything even if we wanted to.

[Crosstalk]

Majors: And it’s not really doing a great job of giving you differentiation, letting you know what’s really important and what’s just noise. So much of these tools, they just give you so much noise and it’s almost worse than nothing at all.

Shimel: Well, that’s why with the Bad News Generator people used to – so, Mitchell got the lucky part of designing it. I had to go sell it. That was much harder.

Ashley: I remember the –

Shimel: Much harder.

Ashley: – guy from the federal government called and said, “This thing is spamming everybody in the Defense Department.”

Shimel: Yes, “We have all these vulnerabilities.”

Ashley: “How do I turn it off?”

Shimel: Because we had this automated workflow before we had terms like AI and ML. But it is hard.

Hey, guys, we’re coming down the home stretch here. We probably only have five minutes or so left. I wanted to – let’s look forward now. I’m big into looking forward these days. I’m hoping we’re going to do conferences in person again; we’re going to – I don’t know if we’ll go back to our offices but…

What’s next for observability? How does this continue to evolve?

Majors: What’s next is I think really wrapping traces in as a first, best tool so that – right now too many people are like “I have one tool for metric, one tool for logs, one tool for servability, another tool for –” they’re spending just ____ data again, every time. And it’s worse than that because you can’t actually seamlessly break down a chain of events, so you’re just copy/pasting this idea into that tool or hoping despite – because it seems to line up timewise, it’s actually bees in the logs – you should be paying to store this data once and you should be able to flip back and forth between loads. Are you slicing and dicing? Are you doing aggregations at read time? Are you zooming in? Are you zooming out? Or are you slipping back so you can view it as a logger file by time?

I think that observability tools like us and Lightstep – which are the only two observability tools out there by my technical definition – although, I will say I do think that this might be the year that that begins to change. I know that so many of the big players have been working as fast as they can for years on the back end. They’re trying to catch up to where we sit technically faster than we can get catch up to where they sit in terms of the business. So…

Chase: And I –

[Crosstalk]

Manuel: I like that you said that. Sorry, Kurt. That –

Chase: No, no, go ahead, Ravyn.

Manuel: – Charity – I just wanted to say to Charity –

Chase: No, no, Ravyn, go ahead.

Manuel: – about tracing, because that’s actually the solution that I am actually looking for, that I’m looking for –

[Crosstalk]

Majors: It’s got a lot of rough edges.

Manuel: – for organizations. Yes, to be able to follow –

[Crosstalk]

Majors: But there are lots of problems we can’t solve any other way.

Manuel: Exactly. So, I feel good that I’m doing it inherently, like I know what I’m supposed to be doing.

[Crosstalk]

Majors: You’re on the right path.

Manuel: I think I’ve actually found a way to do that. But yeah, tracing I also think is the way to go. That is the only way you’re going to be able to know, because it’s taking you from point A to point B to point C to point D and you’ll be able to see that all. And there will be no unknowns. [Laughs]

Majors: Oh, honey. You keep saying that.

[Crosstalk]

Manuel: It’s my dream. That’s – let me live my little dream.

[Laughter]

Shimel: Kurt, what do you got?

Chase: Yeah, well, where mind – excuse me – where my mind goes to in the future is what are we doing in the education system to really disseminate this information, how monitoring software is developed, how we monitor it. And then also “observability” is such a loaded term because I know at Splunk they would consider observability something much different, for running a SOC and things like that.

[Crosstalk]

Majors: Oh, no.

Chase: So, I think it – I mean, at its core observability is what it is. But how it’s implemented for different disciplines and how they use it, I think companies would definitely take exceptions with some of the things we’ve said here –

Majors: For sure.

Chase: – and what observability means to them. And I would just say keep that in mind, that it is a huge field and I think we’re going to see further improvements. I love the tracing. And being able to have that in your tool set would be fantastic.

Shimel: Charity, final thoughts before we wrap?

Majors: Oh, I am just enjoying the fact that some people are so interested in this that they’ll show up and argue about it all the time. So…

[Laughter]

Shimel: It’s all good. Hey, you know what? It’s a living. Mitchell, as usual, I’m going to give you the last word.

Ashley: Well, I’ve learned two things today. Don’t call my tool an observability tool or an AI tool unless it really meets the definition, or I’m going to get crucified.

[Laughter]

No. No, what – I mean, I love this conversation. I learned a ton. I always do when we have these – what’s so cool to me is that we’re talking about operations and we’re talking about this part of our world and we’re not talking about better ticketing systems. We’re talking about really meaty, interesting things that help us do our jobs better, do a better job for the company, for the customer. That’s exciting to me. It’s just really cool. I enjoy it very much.

Shimel: Join us for our next show where we’ll be talking about better ticketing systems.

[Laughter]

No.

Chase: And what argument did I miss? I was –

[Crosstalk]

Shimel: No, no, no, no. It’s all good. You know what? I enjoy having really bright, smart people who are willing to express their opinions. And when I knew we had this panel, it was the lineup, I knew there was going to be no shortage of opinions, nor any backing down. So –

Ashley: We’re going to call the next one “The observability mix-up.”

Shimel: Right.

Majors: We’re going to have to call it “Observability AI.”

Chase: I still want to see that sticker –

Shimel: “AI Observability.” All right. Hey –

Chase: We need to see the sticker Charity was holding up. We need to see that.

Majors: Oh.

Shimel: Oh, we never got to that. Yes.

Chase: We have to see that.

Majors: I’ve got a few of those. I’ve got “Release := deploy.”

Shimel: You’ve got a lot of them here.

Majors: I’ve got “Ship dangerously.”

Chase: Yeah, I love that one.

Majors: “Deploy on Fridays.”

Shimel: You know what? You used to be able to get these things on conferences. Now we’re reduced to showing them on Zoom.

Majors: Yeah, you have to.

Chase: Yeah. Exactly.

[Crosstalk]

Majors: “Perhaps this is a good day to deploy.”

Chase: Maybe.

Shimel: Okay, yeah. Perhaps.

Chase: It’s Friday. It’s not a good day.

Manuel: No, Friday is not a good day.

Chase: That’s the one, right there.

Shimel: And there it is: “I deploy from Cron.”

Ashley: That’s the one to end on right there.

Shimel: That’s is. We’re going to wrap it. Hey, you’ve watched devopsy.kinsta.cloud. It’s sponsored by Tricentis. Many thanks to them for sponsoring. Just wanted to quickly mention Kurt is from Tricentis. Charity is from Honeycomb, and it’s Honeycomb.io. Go check that out.

Majors: We have a very generous ____ here.

Shimel: A very free _____. Ravyn, you can start playing with it now and see if it’s something that gives you tracing and so forth. Mitchell, thanks for joining us.

Ashley: You bet.

Shimel: We are back – actually, we are back with a roundtable next, which will be open to the public. Looking forward to having you on for that. But until then, this is Alan Shimel for DevOps Unbound. Have a great day, everyone. We’ll see you soon.

[End of Audio]

July 28, 2021
The Future of Cloud Testing

Roundtable 10: Cloud-native application development has helped organizations stay ahead by delivering faster time to market. The growth in cloud computing has also presented new testing challenges – traditional testing isn’t enough to address security holistically for cloud-native apps.

June 16, 2021