Roundtable 2: SAP and DevOps were born of distinctly different worlds, but those two worlds have now collided. The SAP core, an organization’s SAP customizations, and all the connected applications are growing inextricably intertwined as SAP plays a larger role in digital transformation. What’s needed to make SAP a successful—and integral—part of the DevOps culture and CI/CD pipelines? That’s the challenge our panel of experts will tackle in this roundtable. Join us—with your own questions and concerns—as we explore issues such as:
Category: DevOps Unbound
-
Defeating DevOps Demons and Haunted Systems
Roundtable 3: Anyone who’s frightened by the ghouls, goblins and demons of Halloween has obviously never been in the depths of a DevOps initiative. Even the most successful projects can’t escape the scares that inevitably arise when you transform culture, processes and technologies. But fear not, DevOps Unbound to the rescue! Our experts will help you root out the DevOps demons lurking beneath enterprise DevOps adoptions.
-
EP 6: How to Lead a DevOps Transformation
Is C-level support required to have a successful digital transformation? In the sixth episode of DevOps Unbound streaming broadcast on TechStrong TV and devopsy.kinsta.cloud’s sister site Digital Anarchist, Mitchell Ashley of ASG and Alan Shimel are joined by Eveline Oehrlich of the DevOps Institute, Louise McCarthy of Bain & Company and Mark Settle, author of “Truth from the Valley, A Practical Primer on IT Management for the Next Decade,” to discuss DevOps and digital transformation at the C level.
The video is immediately below, followed by the transcript of the conversation. Enjoy!
Transcript
Alan Shimel: Hi, everyone. Welcome to another edition of DevOps Unbound. DevOps Unbound is brought to us by our friends at Tricentis. So, thanks for their sponsorship. But, you know, they sponsor but we pick the topics. We pick the guests. And we have a lot of fun here talking about relevant topics around DevOps. Today’s topic centers around DevOps and digital transformation at the C level. And let me introduce you to our panel. First of all, we have Louise McCarthy. And, Louise, I’m gonna ask you to kinda introduce yourself with a little background for our audience.
Louise McCarthy: Yes. Thank you, Alan, and hi, everybody. I’m Louise McCarthy. My background more recently is doing large, complex digital transformations mainly for the financial services. And mainly through the eyes of the business rather than the technology, although I have held roles as CFO and I’ve held roles COO and CIO as well.
Shimel: Excellent. Next, we have Mark Settle. Mark, welcome.
Mark Settle: Thank you. So I am a seven-time CIO and I most recently left Okta about a year ago and have been advising startup companies since then. I’m based here in Silicon Valley. And probably my encounters with DevOps are focused on supporting DevOps communities in terms of infrastructure requirements if they have, and also introducing the concept in actually business application support within a conventional kind of an IT group.
Shimel: Thank you and welcome. And then Eveline Oehrlich, Eveline, please.
Eveline Oehrlich: Yes, hello there. Eveline Oehrlich. I’m the chief research director at the DevOps Institute. Before that I spent 13 years as vice president at Forrester Research studying a variety of topics, and DevOps of course was one of them, working with many enterprises and C levels and others to explore the challenges and the fun of DevOps.
Shimel: Very good. I’ve never heard anyone talk about the fun of DevOps but I’m looking forward to discussing that. And then, last but not least, of course is my cohost here on DevOps Unbound, Mitch Ashley of Accelerated Strategies Group. Mitchell?
Mitch Ashley: Good to be here. I’m Mitch Ashley, CEO of Accelerated Strategies. We focus on digital transformation, DevOps, cloud native, and cybersecurity. I’m also a former CIO. Not nearly as experienced as Mark is, but have led transformations and also implementation of DevOps as well as been a CTO, CEO of product companies. So I’ve lived on both sides of DevOps.
Shimel: Great. What a great panel we have today. I’m really looking forward to it. So we’re talking about DevOps at the C level. As I mentioned to you all before we started today’s show, our friend Gary Gruver wrote a book, “Leading the Transformation.” He wrote several books. One of ’em was “Leading the Transformation.” And Gary is a big proponent of what he calls the top-down approach to DevOps and digital transformation. And in Gary’s view – and I’m not saying right or wrong. In Gary’s view, you cannot have a successful digital transformation or DevOps transformation without sponsorship, support, air cover, if you will, from the C level, including the CIO or CTO, depending on the organization. CEO, et cetera.
Let’s start there. Is C-level support absolutely positively required to have a successful transformation? Anybody?
[Crosstalk]
McCarthy: Do you mind if I go first, Alan?
Shimel: Sure.
McCarthy: I would actually say, my recent – I’ve done about five or six transformations and tried all sorts of ways to actually get them going. There’s always blockers on the way and it’s usually the blockers that slows the whole problem down. And the blockers either come from the ground up, from the employees or the people that run the processes today, or you get the blocker in terms of the C-level-suite people who tend to wanna own things, and it’s a big of ego that actually blocks things going that way.
And the most recent transformation was successfully owned by the C-suite. So they drove it. They owned it. They were the stakeholders. But most of the engagement actually came from the employees, from the process owners. So we ran the whole transformation from the process owners actually owning and identifying where the transformation needs to take place. And that really made it hugely successful because you didn’t have people fearing for their own jobs. ‘Cause with digital transformation it always means or often means that there’s gonna be heads that unfortunately are lost in various areas of the – in the transformation. But mainly doing transactional-type work. But if it’s the transactional people that are running the processes, they highlight their pain points and then we work with them to actually change those pain points into something positive.
So, I would arguably say both from top-down, but also if you don’t do it with engagement of the people running the processes today, it won’t work.
Settle: I’ll chime in. I think I have the same answer but I’m gonna state it in a slightly different way. I think the way the question gets asked, it sounds as if the CTO or the CIO’s gonna come down from the mountain with the DevOps tablets and then ____ employees and team members _____. “I’ve been converted and now it’s your turn. We’re all gonna go do this together.” [Laughs]. Most of the time that doesn’t work. For one person to come in – well, take an example where a startup company here that’s been reasonably successful. Maybe they get up to a product team of, I don’t know, a generic team of 500 people or something like that. They hire a new chief product officer or a chief technology officer who says, “We’re gonna do this completely different. I have a vision.” Et cetera. Well, they’re gonna do it but that’s gonna be a slog to kinda get there.
What I’ve seen be more successful is almost like – what’s the right term in a battle when you send the beach guy out? The beach master that’s moving people around. So you really want almost like a fifth column movement down at the grassroots level of people who are genuinely interested in exploring these ____ ways of doing business. And then I think what the C-level person adds are the methodological resources to drive scale. And so many DevOps initiatives fail because they sound interesting; they sound contemporary; people think, “Other people do it so it must be beneficial.”
But then after like 18 months, everybody scratches their head; “Where are the benefits? We thought this was gonna be transformational.” It’s not. The technical people are having fun. They’ve developed this new terminology, et cetera. And if you scratch below the surface, well, guess what? You’re doing DevOps in like five different ways because the little C-groups have all picked the tool they wanted and decided that they’re gonna have a daily scrum twice a week or whatever. They’ve got their own little process set up. And then people say, “Where’s the benefit? This was supposed to be transformational.” So at some point in the early stage of the journey, it’s needed to drive scale and ____ of endorsement. But I think it can almost backfire if it’s launched from the brain of Mark Settle that, “Yea, and verily, I’m gonna lead you to the promised land.”
Shimel: Sort of a Charlton Heston beard I’m picturing on you there, Mark. Eveline, how about you?
Oehrlich: Yeah. I actually would say it should be outcome-driven DevOps. It doesn’t really matter if it is from the C level, from the individual, from the inside-out, from the outside-in, from the top to the bottom, the bottom to the top, from the middle to the whatever. It needs to be outcome-driven. What I mean by that is: Only metrics will make us change. Just like I drive through my town here – this is a little analogy. I drive through my town. They installed one of those things where they measure the speed and if you go over 50 kmh you get a ticket. Well, my husband, even though that thing has been there for two years, my husband manages to get a ticket every three months. But what he does is he intercepts the mail and then he doesn’t tell me that he has gotten that ticket.
So now what I have done to stop those 15-euro tickets every two months – because if you add it up you can go for a nice dinner – I am going to go through the mail before he gets home so I see how many tickets he gets. And he gets little lines. And that metric is on the refrigerator – I should’ve brought a picture of it – and that has actually reduced how he is driving through town.
And so if you go back to the DevOps teams and the digital transformation and you ask the individuals at C level or VP level or practitioner level, “How successful have you been or what’s the challenges around?” It’s easy to point to somebody. But data, if you have data which shows where you are and it maps to where you want to go and you share that amongst all these different individuals, there is no argument. It’s not the guy with the long beard, as Mark would come in, or it is not somebody at the lowest level who could argue with that. We could huddle around it and figure out: What do we need to do?
So I think it’s not really about who – of course we do need some individuals who pays for it. We need some kind of a fund and budget and all of that. But I really think it’s outcome-driven rather than organizational-level-driven.
Settle: Can I just chime in on that one, too? So I would agree 100%. And outside of maybe the CTO or CPO or IT organization, bringing evidence of improved sophistication, operational metrics around the process to almost any other C-level executive – they don’t really care. The CFO – if you show that, I don’t know, the year throughput, the story whatever – in one of my books, it’s like reading a Bible to the cat. The cat will look at you and feign interest. But it has absolutely no idea what you’re talking about. And that’s sorta what you get outside the technical functions, everybody – “But that’s your job. You figure out what needs to get done.”
And, again to your point, it’s more of a business outcome. Like, are we actually getting customer request enhancements through in greater quantity or are we able to respond to a competitor’s new feature set faster than we did at their last result cycle? Or whatever those things are. That’s what people wanna talk about.
McCarthy: Yeah. And – [Crosstalk]
At the end of the day, the outcomes is aligned to the strategy of the organization. So you’ve gotta be doing a transformation for a reason. And there’s usually three or four reasons why you’re doing a transformation. It’s around the customer journey so the customer needs to improve the satisfaction. You’ve gotta costs. You’re trying to grow your revenue. And then the one that’s usually an outliner that’s become ____ at the moment is carbon emissions. So for every one of these paint points that I mentioned earlier, there should be a business case that shows measurement of success against each of those four measures. And that’s what you – bottom-up, you report at top level to the C suite to say, “This is why we’re doing it. This is the measure of success of what we’re doing it.” And even breaking some of the pain points and the business cases down to quick wins. So ones that you can be getting on with relatively quickly that free up cash that then can pay for the more strategy-type things.
Shimel: Yeah. So I think this brings up – all three of you have touched on it. And it brings up what I think is really sort of an entry question, which is – it’s the genesis of digital transformation. Why and how does any organization decide to go through the potential pain, heartache, and change of digitally transforming, of adopting DevOps or agile? Louise, I’m pretty sure we spoke about this – was it at an Emmet Keeffe Insight IGNITE in Iceland that you were at?
McCarthy: Yes.
Shimel: I was there with you and I remember talking to you about this. You know, Eveline, in your day, someone like Mark might go to a Forrester conference and come back and say, “Wow, Eveline Oehrlich from Forrester just presented this presentation study on the benefits of digital transformation and DevOps adoption. And, god darn it, she described us to the T. We really need to do that here.” And maybe that worked then, right, seven years ago, ten years ago? I don’t know if that’s the genesis of digital transformation today. I think a lotta times digital transformation today does start with technical champions lower down whispering in Mark’s ear that, “If we were doing DevOps or agile or whatever, we could really make better customer experiences. We can give customers what they want. We could do it better, cheaper, faster.”
[Crosstalk]
Ashley: I think we have a chasm that we have to cross, and that is: DevOps can be sort of a vanity project for the IT or the development organization, right? It can be a purely self-interest: “We’re doing what we do better.” Which is always a good thing, but at the end of the day does it move the needle for the business? You have to be able to translate that into what Eveline said was the why. Louise also said the why. But what is the real benefit we’re trying to get out of it? And when you have a purpose for doing it, now you have metrics you can put around that. Is it revenue? Is it competitiveness? Is it responsiveness to the market? Is it changing customer expectations and behaviors? What is it? That’s truly – then the development or the IT organization knows that they’re making a difference.
Otherwise how do you know whether you should be working on more releases per hour, per day, or you should be working on just fixing the backlog? How do you know? You don’t other than just sort of your own internal metrics. And I guess it kinda reminds me of that phrase of the – what was it? Belly-button gazing, right? You’re sorta looking at yourself and what you think’s important. But I think that connection – you have to cross that chasm, whether the business is coming to you or you have an idea of what we can do better and how it might help, and then let’s then make that connection.
Shimel: That’s my point. Someone like Mark – Mark, what does it take to push you over the edge and say, “God darn it, we gotta do this; we’re committed”?
Settle: Yeah. Let me _____. One thing’s funny about this whole line of conversation is of course, in the world that I live in out here, there’s no question about doing it. It’s like: Who is the most sophisticated – [Crosstalk] or more sophisticated? So, in the war for talent, that same 500-person organization I referenced – they might look at each other and say, “Who do we know at Google or Facebook, in their engineering group? We know we should be doing this better. We’ve kind of played around with the concept. We’ve hired some lower-level people who’ve said, “You guys are just behind the times here. So we need some real leadership inserted.” So that’s the other side of the coin. In parts of the country where maybe it’s like an accepted practice, there’s almost a competition to be the most sophisticated or get the most benefit.
But the spark plug, the question you’ve asked – I’m thinking about some second-tier markets – and I really hate to call out specific locales, but what the hell? As a consultant, you can talk like that. So if you’re in a place like St. Louis and you’re in an organization that just has done business the same way forever, all the IT management, all the technical management has come up through the ranks kinda doing things the old-fashioned way, quarterly releases, and they don’t keep the same kinda metrics that you would in a DevOps environment, to blow a place like that up – there’d have to be some kind of competitive imperative.
Or, you know, the white knight does show up from time to time, right? So if you think about it, sometimes when the new CPO or the CTO comes in – and I’ve experienced this in my own career – there’ll be a cause or a initiative that I promoted during my entire tenure – the new guy or gal comes in and they get it done like in a year. And I think, “What the heck happened here?”
There was one company where I was trying to get them off of Microsoft Dynamics, get ’em onto Salesforce. There was another company, I was trying to get production systems up into the cloud. And we did every kinda seed activity you can imagine. The cloud one, we sent people up to Redmond for some AWS instruction. We did some prototyping, et cetera. New CIO comes in and says, “We’re going to the cloud,” 12 months later like half the production systems are in the cloud. Like, “How did that happen?”
So sometimes change at the top – which gets us back to where this kinda started – maybe that is the seminal spark plug. And especially if that person’s been brought in because there’s a broader C-level recognition that, “We’re not keeping up with the times; we need to make some kinda change.” So you have that broader support. When people start to whine, they know that the whining’s not gonna get them out of the change that’s about to come.
But for just self-generated change for people that’ve – then I go back to, you know, Mitch, where you said: then you can always have a vanity project which you can point to as evolutionary. Like: “We’ve got a small group. They’re putting their toe in the water. They’re doing some interesting things. So we are keeping up with the times.” But then you just don’t get the scale benefits at the end of the day.
Oehrlich: I think there was – just to add to something to what Mark said, as the ecosystem of our organizations has expanded, we have to think of – well, the travel industry doesn’t exist at this point in time, but think of maybe manufacturing, right? In the area where I am in, I’m in Stuttgart, we have large car manufacturers, and they’re connected to Bosch and others, and the ecosystem is rather broad. And these folks at all levels are collaborating, connecting. Not as much now with where we are, but they have exchanged thoughts and ideas, they go to events, and they learn from each other. At all the different levels. And so I think there is that – as Mark said, to some extent there is this – not really a competition, but the exchange of ideas, which has brought additional adoption of DevOps at all levels, right?
So if I am the C level of Bosch and I go to a genomic car-driving event and I meet the CF C level of the CF industry down in Friedrichshafen and they are telling me about, “Wow, we really improved our reliability scores and we have really punched up our cadence around the product X, Y, and Z,” I go back and I say to my team, “Hey, I just met this team over there. They’re doing something fantastic. Can we start?” It’s that spark, which to some extent happens to cross over in this ecosystem as we start to be much more connected and dependent. I think that’s one point.
The other one is: if you look at the individuals who are today in IT, they don’t look like this anymore. I’m a Cobol, Pascal, HP 3000, JCL person. That’s when I grew up. That’s my heritage in operations. The guys and the girls today in these teams – they are completely different. They want Python. They wanna do sexy things. They wanna do fun things. They wanna show what they can do. And they’re looking for inspirations from their leaders as well, and they want to see that they can make a difference. I think that is also, from a cultural perspective, I think – of course it’s important but I think that’s also a driving force. And that’s not just the individual contributors but also the consultants and the C levels who are – I call them the flip-flop-wearing, chino-looking guys who go to the conferences, which we’re seeing them; I’m like, “Oh, I think I’m overdressed with my suit.” That’s the people who actually make DevOps happen, and they wanna have fun. They wanna do things. They don’t wanna do JCL and things like I did in a boring ops group dying of boredom.
Ashley: I think you’re talking about a couple of things, Eveline. One is the attracting talent, right? People have more selection, economic conditions aside, more selectivity about the kind of work that they wanna do, the companies that they wanna work for. There’s also just: how do you learn to develop software? If DevOps is what you’re coming into, a way of developing software, you’re looking for organizations and you expect that approach. So I think your point is right on. It’s also the audience of the people who are gonna be the adopters or the embracers or the champions of doing this. So that all makes a big difference. The command from on high, as Mark was talking about, the tablets – there’s an expression for that: “This too shall pass,” right? That’s the reaction, usually, of those kinds of initiatives. You have to have an option. And I think you spoke to that really well.
Shimel: Look, if you’re a Silicon Valley cool kid or you work at one of these organizations where flip-flops and jeans are the rule and you do the hair thing and all of that – I don’t do that so well anymore – that’s cool. But, Louise, you’ve done transformations at what we may think of as large, stodgy financial organizations, right? And they’re not Silicon Valley-based and they’re not a jeans and flip-flop culture. So how does it play there?
McCarthy: Actually, funny you were talking about flip-flops, high fives and cappuccino. I worked in government actually for a little while, the tax authority, and that was a time when the government was setting up its ____ global digital authority. And everybody in the government, all of a sudden they change from suits to wearing flip-flops and high fives and cappuccinos, and it was like a world away from the starchy government.
But, anyway, moving on to the financial services, yeah, the financial services – I’ve been lucky enough actually to do transformation in legacy financial services, and the big boys like the HSBCs, for example, which is complex. It’s like turning the Titanic around. And I’ve often written articles around: it’s like moving deck chairs around. The reason why I say that is because automation is the way forward. We talk about automation in many guises. Automation takes out people. It takes out transactional-type activities. But the legacy banks just struggle to do it. So I say it’s like moving the deck chairs around the Titanic because of the legacy system they’ve got. Transformation – they talk about it. They move things around. But nothing real and concrete actually comes out of their transformation, which is really sad. Because there’s great opportunities there in the legacy banks.
And on the other side of the fence, I’m working currently with a contender bank, a digital bank, which is new. It’s new. It’s fresh. It’s green. It’s almost as green as your screen there, Alan. It’s a green ____. And it’s great to be able to take the whole end-to-end customer journey and see it digitalized. And how quickly you can change and transform. And developing and changing technology with that suite, the same thing end to end, it’s just awesome to be able to see how quickly it happens. Whereas the legacy banks are still thinking about planning it, talking about it, cogitating over it, when the small guys are out with a new technology.
And one of the things that I’m often transforming whenever I do my transformation – it’s not transformation, it’s Christmas; it’s for life in my mindset. So it’s around continuously thinking about the innovation and it’s about encouraging these legacy banks to have innovation departments that are constantly thinking about the new technology that’s out there to take them ahead of their competitors. So like these digital banks, if the legacy banks are doing this, they’re actually seeing a way of making it happen quickly. And I think that’s the problem that we’re constantly battling: the bureaucratic slowness of moving Titanic.
Shimel: So clearly the kind of organization you’re at determines, or will have a huge impact on this. Guys, we’re coming on the back side of our time limit here. I wanted to talk a little bit about: okay, we’ve made the decision. We are moving forward with our transformation. We are adopting DevOps and agile and these new ways. How at the sea level do we – short of coming down from the mountain with the two tablets in God’s own writing, how do we translate that to the troops? How do we effect the organization? That this isn’t some passing fancy of the CIO and it’s not gonna pass in the next quarter. This is really the way we’re doing it. How do you make that change down the ranks, get people adopting as champions and being successful with it? Mark?
Settle: You know, it’s not unlike an initiative around security. When you talk about cultural change within an organization, I think leaders don’t always appreciate what a huge impact they have on the people that they lead. You know, people take cues from the way you use your time, the things that you show interest in, things you follow up on. Your time management sends tons of signals to the organization. And some of the operational metrics that I poo-pooed before as being of no interest whatsoever to people outside the technical areas are a great way to kind of keep score. You go back to what Eveline said: If there’s a sense we’re making progress, some of those ____ celebrations can be really important.
I’ll just leave you with a thought. I worked with a guys several years ago who used to say, “What interests my boss fascinates me.” As long as there’s some sustained interest – and to go back to, Alan, what you said before: I can just come back from the Forrester meeting all jazzed up, and then two months later I go to a Gartner meeting and now I’m on a whole different tangent. Now I think it’s automation; we gotta get RP robots in here. That’s gonna blow the business up. And then people in the trenches are like, “This too shall pass. He’ll go to another meeting. But I’m not gonna get my –
[Crosstalk]
Shimel: Come back with some other toy, yeah.
[Crosstalk]
Settle: “He’s so crazy.”
Shimel: Yeah. I used to feel it was whatever was the last book I read. That was the next greatest thing to me, right? ‘Cause I just read that book. But, Eveline, fundamentally – and this is something the DevOps Institute is deeply ingrained in – it also becomes a question of: how do we upskill our workforce? How do we prepare – do we just go out and hire new people? That’s hard. Do we upskill our existing people for this? At the C level, how do we effect that?
Oehrlich: Yeah. So, from our research at the DevOps Institute, we know that most organizations are actually preferring to leverage their existing resources to train them and to upskill them. Now, upskilling of course has its challenges because you don’t just upskill around processes of learning a new way of doing continuous delivery, or it’s not just about upskilling new technical skills. It’s also about human skills. That’s I think the challenge. A lot of organizations are biting themselves into that – or pushing themselves, or actually locking themselves into the corner. They think, “Yeah, let’s give them AWS training. Let’s let them understand what the latest DevOps platform is and give them some language skills around whatever the latest hotshot technology programming language is.”
But it’s also soft skills or human skills which are necessary. People need to understand how to work empathetic, how to collaborate, how to connect, how to cross functional borders, how to understand when a security individual tells me something. So having skills around these different categories is important. And there is investment needed. We see that, unfortunately, 7% of survey participants we had last year said that they have no idea what their upskilling programs are. We know 37% of these individuals – a very large sample size of almost 1,200 – said that, “We don’t have a program in upskilling.” And the rest of them either are planning one or have one. So that’s really unfortunately poor and not enough. So, anybody listening in, get ready to upskill your people. Start thinking about training. Start thinking about exchanging knowledge and allowing them to learn. Give time.
I, for myself – and we’re doing that at the DevOps Institute a little more, and our CEO does that a little as well. I call Friday my learning day. And it doesn’t matter what happens. Unless the CEO wants me there, Friday is my learning day. I learn French. I learn about design thinking. And I’m learning about metrics. That’s it. And we all have to do that. It’s very important.
Shimel: Agreed. Louise, you go in as a consultant. In terms of the skill level of the workforce, do you try to upskill them? Do you supplement it with fresh talent? It’s hard getting fresh talent in. Mark, you know in the Valley they cut each other’s throats for this. How do you do that?
McCarthy: It’s an interesting thing because you’re trying to fly an aircraft while changing the engine in some cases. And the same people _____ transform are the same people that need to drive the aircraft. And also if you call in too much fresh talent, then you would probably put the noise in the joint of the people that are supposed to be helping you run the process anyway.
So the way I’ve done it is I’ve gone in and we’ve identified some highly skilled people that know the processes really well and then supplement them with some more transactional process mappers and people like that. So that those people that run the processes can also see things like: how do they do the process mapping and how do they identify the paint points? How do they convert it into business cases? How do they use it and then convert it into technology?
So on that journey, the people that are running their processes are also being upskilled at the same time into continuous improvement and transformation. But I’d say that there’s also – it’s not about upskilling necessarily on all sorts of things. Because as technology’s changing, there’s less the need for upskilling because a lot of technology is self-serve-type technology now. A lot of the RPA touring now. People can do it like they learn Excel, like they learn Word and things like that. so the upskilling like it used to be, about spending months and months programming and that type of thing – there’s less the need for that. It’s more about the upskilling in terms of continuous improvement. And it’s a cultural change, I think somebody mentioned earlier. So it’s going in and using the resources that’re there, upskilling those people, and slightly supplementing them and doing hands-on job training.
Shimel: Excellent. Mark, Mitchell, I’m gonna give you each a chance to kinda close it out here, and then I think we’re about outta time. Mark, final thoughts?
Settle: Let me just make a final comment along the lines of this last topic. I’ve gone through many of these reskilling things, and seeding that group with new talent I think is essential. I mean, if somebody came into me and said, “We’re gonna go to DevOps and you can have consultant money; the consultants will come in; they’ll be here 90 days, 120 days” – this is an acquired skill. It’s almost an art form at its finest. And the devil is in the details. And the day the consultants leave, everybody’s left to their own devices, and it’d be just like saying, “Let’s just shut down the data center and move everything in the data center up on AWS, and we’ll get some AWS consulting in here, and then they’ll leave after 90 days, and you guys run all the production systems up there in that whole new environment after 90 days of familiarization.” I don’t think too many companies would take that kind of a risk.
So I think people get so carried away with thinking that reskilling can be effective, that people will come to embrace it. And, again, I hate to say this, but in my personal experience, if you reskill a population of people, 20% will be superstars, 20% won’t get it, and 60% will be good enough. But if one of the 60% left, you’d recruit somebody in with a higher skill level than the person that left. So they make forward progress. I’m not denigrating people’s commitment to new things. But you’ve gotta bring in that fresh talent that’s been there and done that. They know what it’s supposed to look like, right? And they can look around a corner and they can see the big fuzzy hole in the road that we’re all gonna fall into here in the next two months.
So that’s my thing.
Shimel: Mitchell, you wanna bring it home?
Ashley: Yeah. I’d like to just wrap with the human element, but maybe from a little different perspective. You know, people react to change in different ways. Your advocates – great, they’re all about it. Your resisters – love them ’cause you can actually work on that. You can help. You can help bring them along. It’s that passive unspoken middle ground, the folks that’re waiting to see: “Before I commit myself – I’m not gonna put myself, my job, my career at risk for this next thing. I don’t know if I can do it.” And the idea of bringing people in – I think one of the main purposes is to show the team, A, it can be done, and, B, they can learn it. So it isn’t: bring the consultants in for 30 days, hand over what they did, and now everybody goes, “Now what do I do?” Right?
So I think if you can de-risk, address the fear of what this means, that: “You’re gonna be able to do it and we’re gonna stick with you to help you get there, help everybody get there. And if there’s some folks that choose not to go along the journey, that’s okay too. But we’re all in this together.” I think you’ve gotta address the human element from a very personal: how does each person feel about this and whether they’re signed on to do it? And that’s gonna guide your success. That’s been my experience.
Shimel: Absolutely. Well, guys, we’re about outta time. Mark, Eveline, Louise, and of course Mitchell, thanks for joining us on this episode of DevOps Unbound. Many thanks to our friends at Tricentis for their sponsorship. Until next time, this is Alan Shimel for DevOps Unbound. Be well, be safe, and we’ll see you soon. Bye-bye.
-
EP 5: Outsourcing DevOps, Oxymoron or Truth
Can you outsource DevOps? And if you can, what are the keys to doing it successfully? In the fifth episode of DevOps Unbound streaming broadcast on TechStrong TV and devopsy.kinsta.cloud’s sister site Digital Anarchist, Mitchell Ashley of ASG and Alan Shimel are joined by Ming Gong of UST Global, Chris Rolls of TTC and Robina Laughlin of Guardian Life for a discussion on outsourcing DevOps.
The video is immediately below, followed by the transcript of the conversation. Enjoy!
Transcript
Alan Shimel: Hey, everyone. I’m Alan Shimel, and welcome to another episode of DevOps Unbound. DevOps Unbound is a series, sponsored by our friends at Tricentis, that explores different topics around DevOps every other week, as well as having a monthly roundtable open to the public. Today’s episode is “Outsourcing DevOps: Oxymoron or Truth?” I’m joined, as usually, by my co-host, Mitchell Ashley, CEO, founder of Accelerated Strategies. Mitchell, welcome. Thanks for coming on.
Mitch Ashley: It’s always awesome to be here. I’m excited about our guest today.
Shimel: So am I. Let me introduce you to our guests. We have Robina. And, Robina, I’m gonna ask you to give your full name, title, company, background. And we’re gonna ask you to kick it off.
Robina Laughlin: Sure. Hi. I’m happy to be here. Thank you. My name’s Robina Laughlin, I work for Guardian Life insurance. Actually Guardian Life and I have been working in the quality assurance space now for about 25 years. I’ve been working for Guardian for eight years. We have a great enterprise quality assurance journey. Most recently my department, my quality assurance department’s been working with integration into a DevOps model. So we’re starting that journey. We started about a year ago. And really happy to be part of the conversation.
Shimel: Thank you, and thanks for joining us, Robina. Secondly, I’m gonna introduce my friend, Ming Gong. Ming, a little background?
Ming Gong: Hey, Alan. How are you? It’s always good to be here. My name is Ming Gong and I work for UST Global, I’m vice president. I run a practice called Digital Agility Platform Solutions. So what we do is we provide all kinds of services and even product to our enterprise customers, things such as DevOps transformation, DevSecOps, digital, cloud transformation, as well as latency modelization. So, we have tons of battle scars as we’re implementing DevOps for our customers.
Shimel: Absolutely. And then, last but certainly not least, Chris?
Chris Rolls: Hi, nice to meet everyone. So my name’s Chris Rolls. I’m from a company called TTC. I’m CEO of Americas. And what we do is we help organizations with quality engineering, continuous testing and also Agile and DevOps transformation. So really happy to be here and excited to hear what everyone else has to say.
Shimel: Absolutely. So, guys, when I started – so we launched devopsy.kinsta.cloud in March 2014. Working on it all through the second half of 2013. At that time it was sorta dogma in DevOps that DevOps wasn’t about the tools; DevOps was about people; DevOps was about culture. And that therefore, to be successful in your DevOps journey, outsourcing was a no-no. You couldn’t be successful and outsource your DevOps because it would destroy – you never got to the nitty-gritty of the culture.
Now, that was seven, eight years ago. A lot has changed. I think we’ve seen some very successful – the world is full of DevOps consultants, large, small and in between, who are helping organizations along their DevOps journey. So I think we have kinda moved beyond the dogma. But have we moved to truth? Can you successfully outsource DevOps? And if you can, what are the keys to doing it successfully? And I’m gonna throw that out to our panel today, and we’re gonna start that as our discussion. Ming, not to pick on you, but you already mentioned you have a lotta scars to prove it.
Gong: Absolutely [laughs].
Shimel: Tell us a little bit about your journey or UST’s journey in this space.
Gong: Yeah. I think you mentioned a couple of things that’s really important, which is, ultimately, I think to do DevOps correctly the entire delivery team needs to live and breathe the DevOps – not only the automation, the tool set, but also the processes as well as how you do things or the whole mindset thing. So that’s the absolutely ultimate goal. I think even over the past few years, even though all the companies all moving towards DevOps, but they are still having a lotta challenges in terms how to take some isolated success in certain areas, trying to cascade that throughout the entire organization. And I think this is where, when we talk about, “Hey, is outsourcing a myth or a reality?” I would say it’s actually somewhere in between.
Because it really depends on where the continuum – when I think about DevOps, I think about DevOps continuum, ___ maturity, where they are. For the company who’s really mature, they are less likely to benefit from outsourcing before the folks who’s just starting the journey or middle of the journey – they definitely can use some help. And not only outsourcing in general but also what part of DevOps to outsource as well. So typically when I work with enterprise customers, they have a centralized organization, like some kind of DevOps COE. But they also have each individual team that needs to be enabled. So where we found there tends to be more success is when we help the COE to – infuse them with the DevOps expertise and help them kinda build up the practice around DevOps.
So I definitely think it’s somewhere in between.
Shimel: Fair. Chris, your organization is similar to Ming’s in that you are helping organizations with their transformation. Has your experience been similar, different?
Rolls: Yeah. I think I’m not gonna shock the world by being a consultancy who sees consultants can help with DevOps transformation, right? I would be outta business if I thought that was the case.
But if I was to answer your direct question, can you outsource DevOps? I would say no. Because DevOps is about a cultural change within the organization as well as the technical practices. Now, similar to what Ming is saying, it’s sort of in the middle because outsourcing and consultancies and other vendors can be a part of the solution. They can make some things more challenging in certain ways because you are introducing certain barriers, as in contractual sort of challenges around it. But it definitely can be part of it and it definitely can help, particularly when organizations are trying to figure out how to do things better.
Shimel: Fair. So, Robina, you look at this sort of 180 degrees different than Ming and Chris do, right? You’re not providing consulting to organizations. You are the organization. What’s your view on this?
Laughlin: Yeah, thanks for having me. This is a really interesting conversation. Some of my thoughts around this is: I think the important thing to understand is: what’s your driver to DevOps? Why are you moving to that model? Why is that model gonna be important to you as an organization or as an enterprise? And when you start to answer that question, you get to different models. For us, we adopted SAFe. We started that journey two or three years ago. So DevOps was not the first thing we tried to conquer out of that transformation. It’s on the tail end of it. And it was always on the strategic road map. However, what we found is that it grew organically based on the improvements from adoption of SAFe and agile. So for the teams to get to the velocity that they wanted to do, it became a necessity as opposed to an aspirational goal, right?
So for us when it became that driver to success, it took on a whole different model. And then when we started figuring out how do we wanna do this with our partners, the vendors that we work with, one of the things that we always hold true to in Guardian is that frameworks, guardrails, models our Guardian culture, and we developed those. We developed them in conjunction with our partners. We’ll seek for collaboration. We’ll seek for guidance. But the model has to come back to fit our culture and how our folks work.
And then once we have the guardrails in place and the operating model, inclusion of an outsourcing model is very successful, I find. But I speak from a QA perspective, and I’ve been on an outsourcing model now for 15 years, right? So I always kinda take the same approach to things in that if you have the operating, the guardrails, and people know what they need and why they’re doing it then outsourcing is a good solution. But you have to have a strong backbone for the company that you’re working for and with to be able to represent that model and the culture.
Shimel: Mitch – oh, Ming, go ahead? Whoever, Chris.
Rolls: Oh, sorry. I think just to drill into that as well, Robina mentioned you’ve gotta understand your drivers for DevOps; what are you actually trying to achieve? I think that a lot of times organizations – they want to do DevOps because that’s what the conferences and the experts say they should do. But also you need to understand: what’s your driver for outsourcing or engaging partners and other vendors, right? So if it’s a cost-saving mechanism, which often it is in this economic climate, is something that people are looking at, right? That can be one component. That can be one reason for outsourcing. You could also be outsourcing to get additional expertise that you don’t have internally. You could be outsourcing ’cause you’re scaling up with a whole another set of products and there are sort of HR reasons for wanting to do that. But you need to understand those drivers.
And I think another thing that Robina said crucially which I see organizations get wrong is: the organization who’s – the business, the incline for us needs to own the cultural change and needs to own the methodology. I come back to: no, you cannot outsource DevOps. You can’t buy that on a platter. You have to invest. You have to make the changes yourself as well.
Laughlin: Yeah. There’s some parallels to that conversation, right? ‘Cause a lot time ago this was the conversation for quality assurance. Like: outsource it; there’s an easy button; all your problems will be solved. I think that, from the experience of that, for me there was never an easy button. It was a lotta work and it was a big partnership. But what we did was, through that partnership, we started bringing in technology that my team hadn’t had a lot of exposure to so our options grew, and I found that there were some easy buttons along the way to build, to run tests against that build. So we’re moving into a model now where the assets of quality assurance can be augmenting what is your DevOps model. And that model has to work for your company and it has to be part of your culture. It is not something that just kinda develop out of a box.
Ashley: We probably need some precision around the term “outsourcing” too. So I think in this context –
Laughlin: That’s a good idea [laughs].
Ashley: – we’re not talking about externalizing it. In other words, put it in a box, ship it somewhere, have someone else do it; give it to us when it’s done. Right? Because we’re transforming the organization. I think it can happen a couple of ways. It can happen early in the journey; I think it also can happen – organizations typically get stuck at some point where we’ve gotten 2, 3 teams there but man-scaling it to a 5,000 or 10,000-person IT organization – that’s hard. It’s easy to do it on a startup basis.
So everything to me is always driven by what the company is about, what its DNA is like. Some companies are very data-drive: We don’t make a decision unless it’s got a graph attached to it or a financial analysis attached to it. Or tied to a strategy. Or whatever it is. And different companies often taken on the personality of the founders if it’s a younger company. But it gets established with it. And if you don’t have that figured out and aligned with how you’re implementing anything – agile, DevOps, SAFe, whatever it might be – you either go against the flow and fight it all the way and fail multiple times and never succeed or you figure out what fits into the flow and the momentum, the DNA of the organization.
That takes knowing people. It takes knowing the organization. It takes commitment by people in the company to do it. So it’s not an outsourced give away the problem. It’s a: I need help because I need expertise, because I’m stuck, because there’s other – we don’t know what we don’t know. And we’re looking to get smart from other people. And we want you to leave someday and help us operate on our own. Maybe help us with a different problem someday. But that’s kind of our goal as an organization.
Gong: Yeah. Mitch, I fully agree with that. And we certainly have – a couple of points you point out is pretty interesting. One is the challenge that we seeing for large enterprise customers: they usually have problem with cascading kinda DevOps excellence across the organization, right? But there’re so many hurdles, I would say. One of them is organization siloes. We have seen so many times where you have one division, usually on the digital side, front end – they have great success because technology stacks tend to be more modern so it’s easier to get on the DevOps train. But then once you start trying to cascade to a legacy side, it just doesn’t work. Either because they don’t share very well because there’s organization siloes, or because the technology stack they develop doesn’t really work on the other side.
So where we see, again – I know you have term about outsourcing can mean many different things. But where we see a lot of support or at least success is when we help kind of the organization where they have a COE, where they establish standards – and I think Robina kind of mentioned this too. It’s very important for company to really establish their true North, the compass, right? The guardrails and things like that. But it’s also important to not have a very heavy central organization either to trying to drive DevOps. Because then you’re missing out on all the distributed innovation happening on the ground level so you can never respond fast enough.
So this is why when we created our model, we usually have a team, or a very small team, help with the COE side of it to help them kinda develop their whole community, the culture, things like that. But then we have very nimble – we call it deep pod. It’s like a five-member crew. They have – consists of DevOps architect and couple engineers. And what they do is they will be kinda parachuted into an area within the enterprise where they need a lotta help or the application that they using is very critical to the business. Will drop in there. Not only helping with engineering, figure out the tech stacks and figure out the DevOps – all the nuances of it. But we also do a lot of coaching to make sure we can train the team to be self-reliant. Once they are, we lift the team up then we drop the deep pod into the next area where they need help. So this is kind of the hybrid model is where we find more success.
Ashley: I’m really curious: Who typically brings in your firm, Ming, your firm, Chris? Maybe, Robina, who brought in someone to bring in help with DevOps. Was that at a very high level? Is it a development team level? I’m thinking larger organizations here. What’s the –
[Crosstalk]
Gong: I think Chris and I probably will share the same answer, which is: all of the above. We have a large DevOps engagement to where the CIO invites in. And we operate starting from the strategy level all the way down to the individual team level. But then we also have, through relationships or whatever case might be, where we insert ourselves into a particular team to help and then grow from there.
Rolls: Yeah. Exactly the same. And if it’s okay I wanna go back to one thing you guys said before, which is having a bit of precision around what we mean by outsourcing. Because I think when I see this topic, I jump to full, functional outsourcing of QA or dev or opportunities. But the reality is: outsourcing also exists on a spectrum.
You know, when you hire a maid to clean your house or to service your car or anything like that, you are outsourcing that function of your personal life. And it’s the same with software development, right? So there’s the really far down this end where it’s just you have a consultant who’s embedded in a project team, and then there’s all the way down to the far end where you do that full-scale outsourcing of a function or a department.
And what I want to be clear on is: the further you get along that continuum, the harder it is. You have to be far more careful about it. And so that’s something that’s really important. And to go back to the question that was just asked: completely the same. Sometimes – and you’re very happy when the CIO or the CEO says, “Come in and help me do this transformation across the organization.” Sometimes it is a QA or dev manager has a specific problem that they want to solve. But I’d also be really interested to hear Robina’s thoughts.
Laughlin: So how did we bring – I’m sorry. Can we –?
Ashley: What level did you bring? Was it the CIO brought in somebody to help you with the ____ or was it on a QA development team?
[Crosstalk]
Laughlin: Oh, sure.
Ashley: How did you start down the getting external help, outsourcing help?
Laughlin: Yeah. So this is a little bit of a parallel run to a DevOps model. So from a QA perspective, what we did was we decided to centralize our enterprise quality assurance organization. And when we did that, certainly we got economies of scale, but I needed a partner or a vendor that would come and work with me. And when I say partner, I truly mean it. Like we have team meetings. They’re all folks that work for me. I don’t distinguish. So you have to be able to have that relationship I think to be able to get success out of outsourcing, if you wanna call it that. Because it’s really partner shipping, in my mind.
So that was the journey that we took. And that’s been underway now for almost eight years. So we have an incredible wealth of assets behind us. And I started stopping and looking at what my partners were doing on the other side of building and deploying, and how are they verifying that building? We’ve been moving this shift-left journey of discovery of quality issues for a long time now, and we’re at the point where we’re in the build stage. And I’m like, “Well, we have great assets that could help verify that a build’s gonna work. What’s the next solution to that?”
So I reached out to my Cognizant folks under my QA umbrella and said, “Hey, if we were to think about extending our assets use, what would we do?” And they were like, “Turn to your DevOps team. Turn to your build and deployment team. Let’s understand what they’re using. They’re using Jenkins. They’re using Bitbucket, example for us. So when we started looking at being able to scale our assets and use them across the enterprise, we built them in Jenkins. We built our actual Selenium code in Bitbucket. So now we have the hooks there and we can – as these teams start to catch up or we start to catch up and we meet in the middle, we then have a problem we’re trying to solve. We are getting builds. They are broken, or when they pass, there’s quality issues. Can we discover them through automation? So that’s the easy button.
And in this case I actually do have the easy button. I have this regression suite. I can slim it down to a smoke test or something, and say, “If we can deploy this and do all of this then, yes, we can get those results.” And then the next phase of it is accountability of teams of what they’re gonna do with those results.
So none of that is technical. That’s all cultural. Except for the hooks in what we built. So that’s kinda been our journey and that’s where I’m looking to go strategically.
Shimel: Excellent. So, guys, I want to bring this down or – we touched on it already but I wanna clarify it. You know, Ming and Chris, you’re in a unique position in that you guys get to see dozens of organizations and how they approach this problem. Where, Robina, your view is Guardian-based, right? This is what you guys have done at Guardian.
Here at devopsy.kinsta.cloud I’m also – I’m in a catbird seat in that I get to speak to many people like Ming and Chris, but I also get to speak to many people like Robina. I get to speak to people like Mitchell. I literally get hundreds of views on this. And I recognize patterns. And what do I do? What’s my superpower, superhero power? I’m good at recognizing patterns. I guess I’m a machine learner. But anyway, there usually is a pattern along DevOps adoption. And it’s similar to what Robina mentioned, which is – when I first launched devopsy.kinsta.cloud, our tagline was “DevOps picks up where agile leaves off.” Right?
And, yeah, because what happened was people were moving the agile development processes probably going on 20 years now. Agile’s been around 20 years. And it was great. It makes you feel old. I’m sorry, Robina. But, yeah, it’s 20 years [laughs]. Don’t feel bad. I watched this “Halt and Catch Fire” about the birth of the PC industry. Yes, that’s what it is. Einstein’s theory of relativity [laughs].
Laughlin: I remember my agile training with Ken. I can’t imagine 20 years –
Shimel: Yeah. Ken, who now of course is Scrum.org, right? Versus Scrum – the other one. Not here to get into that one. That’s another rat’s nest.
Gong: Different podcast [laughs].
Shimel: Yeah. But the thing is: the normal course, if you will, is: yeah, we do some agile. Whether it’s Scrum or SAFe or whatever we’re seeing out there, scale. We wanna start getting into some lean IT stuff. And of course when you start marrying agile and lean, it’s DevOps, right? That’s kinda what DevOps is. And now we’re gonna get into CI/CD pipelines. And of course it’s grown since then. We’ve added SRE on. Continuous testing has become huge. Right? It’s a huge piece of this. So we’ve seen the sprawl, the urban sprawl of DevOps and agile over the last 20 years. Along with it, I think it’s become all but impossible to do at a large organizational scale, at a large enterprise scale, without some outside help.
I think we’ve recognized the enemy. It’s us. We’ve created the need in our large organizations, the way we go about adopting these things, that we can’t do it without outside help.
Laughlin: That’s very true, Alan. I think that’s why you really need the outside help and you need to go to the experts. Because no one size fits all. No one tool solves everything. That’s how I got involved with Tricentis: I went out really early looking for a vendor that would grow with me and was on the same aspirational journey that I was, right? So that’s where I think it really helps in terms of consulting. You need to pick the partner that’s gonna work with you and grow with you. And sees you for your uniqueness, not an industry standard.
Gong: That’s right.
Shimel: Chris, Ming?
Rolls: I think you can’t take a single ___ of technical practices or cultural adoption and apply it to every single organization. So that point, Robina, about making sure that you tailor it for the organizations that you’re working with – or even within an organization, the different departments, the different products that you’re working with, right? There’s a different sort of cultural mode within the digital section versus the enterprise technology section, right? So that’s really important.
And I think that also comes to a thing that I see that often people get wrong when they do outsourcing is that they don’t align the incentives properly. Because ____, we all have different incentives. Even, Robina, if I was partnering with you and we want to work as really good partners, then I have a slightly different set of incentives to grow my company as Guardian does, right? And so we have to find ways to align those incentives. Otherwise you will create a cultural conflict which prevents you from really going as fast as you can. So that’s just something that I see often people get wrong about moving to DevOps, because they have to use outsourcing in particular, engaging firms to help them with that.
Gong: I wanna go back to the point Alan was talking about in terms of the massive talent shortage in industry. And it’s not just DevOps. It’s cloud, everything else, right? Just in general I think we all wish there’s more qualified folks who has those technical skills. And I think that’s one of the reason why a lot of our customers want some help from us. Because they simply cannot find enough talent out there that not only knows the DevOps technology but also has the ability to keep up with a crazy changing industry, right? I mean, you see a massive change even within the last couple years where used to be all the siloed best-in-breed product like Jenkins or whatever else, and then artifactory, whatever.
But now you see a trend where industry’s moving towards more of a kinda end to end all in one type of tools, right? For example, you have the Azure DevOps, GitLab, and the whole – they start offering the whole suite of things. Because they recognize the massive talent short. You can’t take care of the problem until you can also solve the problem on technology side as well. So that’s kind of approach that we’re taking as well, where not only we have on people side we have our deep pods kind of coaching the team so we can get lots and lots of teams to train up to be self-reliant. But on the other side we also develop on platform to kind of try to extract all the complexity of DevOps from the user so the user doesn’t have to be an expert. So we call it democratizing DevOps for everyone, right?
So I think for a company to be successful they need to kinda take approach of kinda both people and the tools combination.
[Crosstalk]
Ashley: ___ kinda just share a personal story in my additional adoption of DevOps when I was a CIO at a company. And it was initiated by me ’cause actually Alan had reached out to me and said, “Hey, Mitchell, there’s this thing called DevOps. You gotta go read about.” I’m like, “Okay, what is it? I’ll go figure it out.” I got hooked onto it.
But the rest of the team hadn’t really caught fire yet of getting onto DevOps. And so as a leader of an organization, it’s tough to say, “Okay, I want us to go this direction; I think this would really be helpful for us” when the team’s not there yet. And really understanding what it’s gonna take to get the organization going and get behind it and how to be successful at it. This was earlier, so you reach out more to individuals that’ve started to build some brand and some experience in doing things like DevOps.
Today I would go to a different kind of a team to say, “Here’s where we are. Here’s the projects that we have going on. Here’s kinda what we’re being asked to do,” right? I think every – maybe not true, but it seems like most IT organizations are always being asked to transform some way, somehow. And sometimes pretty radically. And if you can sort of set where we’re headed and tie it to the why, as you talked about, Robina, the why for the business though, not for IT, that seemed to me to be the biggest thing that everybody could rally around. Not that it was DevOps or agile or it was this. It was: “We wanna go after this market. And our ability to get there is really constrained today. And let’s figure out some ways we can go after that.” And that tended to rally around.
So I think today – I guess my point is: Today we’re at a point where we can hire more expertise. And we wanna de-risk our journey and learn from others. We don’t have to figure it all out ourselves and be the smartest person in the room. Certainly I don’t. And that’s a great environment to be able to bring in help.
Shimel: Robina, I’d like to throw out – there was a time when we used to look at quote/unquote “outsourcing” and consultants, and one of the terms that was thrown around was “body shop,” right? Like, “Bring in some bodies. I’m gonna rent some bodies.” I remember when Mitchell and I were at StillSecure. We’d go down to – we had a big federal practice, and we’d go to a federal facility, whether it’s a lab or a knock or what have you. Eighty percent of the people there were not federal employees. They worked for Grumman Northrop or SAIC – take your pick of the beltway bandits. But it was all body shop. And when you’d speak to the salespeople at these consultants, you’d say, “How’s business?” “I got 50 people placed.” Not what the mission was, what the tools were. They judged their success by how many people they had placed.
I like to think – and maybe I’m naïve; Robina, I’m gonna ask you – that we don’t judge the success of our DevOps outsourcing by how many bodies I’ve got buried at a given location. ‘Cause I think that’s a terrible metric to determine the success of a transformation.
Laughlin: Yeah. That’s a great observation, Alan. You just kinda gave me shivers when you talked about number of bodies. I remember those days. They’re painful to look back on, honestly. There was a time when you needed an army of folks to do things and I don’t necessarily think that’s the case now. For me, our journey was: I needed a partner and I needed someone that was gonna come here and work with me. And through that partnership I also needed to know that I was gonna have a 24-7 shop, and that’s gonna necessitate overlap on both sides, right? Your onshore and your offshore. So I don’t think you can ever just kinda set it, forget it, and throw it away to an offshore team. That’s just not how it works.
And I think the other piece of it is: at the very source of it, you have to remember that these are human beings. They have families. They have lives. They wanna do good work just as much as we wanna do good work. And the way that they’re gonna feel that contribution is being part of the team. And the way they’re gonna feel that sense of that is by being present. And we actually flipped the dynamics. So we’re no longer a headcount shop. We’re an outcome shop. So I measure the success of my organization by the outcome of our testing. And for me, with my partner, if they do it with 2 people or 200, that’s their discretion. So I don’t have to get into any more of these conversations of: “We need two more bodies here. We need eight more bodies there. I’m anticipating 12 bodies.” That just turns into I think an empty use of really valuable strategic time that you have with your partner. And you should be using it a little bit more productively, right?
So that’s how I flipped that dynamic. And I’d be very curious to see how Mitch and Ming feel about it. And Chris.
Ashley: Well, my first reaction was I thought you were gonna talk about “Sopranos” or something. A whole different kind of body [laughing].
Shimel: Bada bing.
[Laughter]
Ashley: You know, I really like how Robina described it. Because it isn’t about budget size and all of those kind of backwards metrics. And I’m sure they’re still used today. To me that’s a measure of how aligned you are with the business. If you’ve shed the financial size, the IT-only-centric metrics to “What are we making an impact?” – there’re a lot of ways we can do a digital transformation project. We could get the contact list services let’s say. Something very relevant today. And it doesn’t have to be with hiring 50 more people ’cause we’re probably never gonna find 50 people to hire anyway. So we’ve gotta find other ways to do it.
But there’s also – there are times where I need these people to fill these roles. And I’m sure the body shop approach still applies in some situations. I doubt it on strategic projects though. That would be my supposition.
Shimel: Chris and Ming, I’d like to hear from you guys on it though. Because you’re on the other side of the equation here. Chris?
Rolls: I hate the term “body shopping.” It’s a terrible, terrible term. And I talk to the team and I say, “We have to have stamped on our forehead, ‘We are not a body shop.’” Because if you’re going in there and your goal as a account manager is to get 50 people on an engagement, you’re not thinking about what that organization needs. You’re not thinking about what the business outcomes are. You’re just thinking, “I can make $10.00, $15.00,” whatever how many dollars it is off each person, “and what’s my commission for that?” That’s a terrible way to think about it. And what’s more, it shows an outdated way of thinking about the people who are going to do this work. People, process, technology – people, in my mind, being the most important part.
If you’re thinking of them and you’re calling them “resources,” which sometimes we all slip into, or you’re calling them “bodies” – that’s not just a warm body in a seat typing some code or testing some software or monitoring the systems. They’re people who have skills who are thinking about things who are trying to do the best work they can to deliver outcomes for the company. And so I hate that term “body shop.” I hope that no one associates that with me and my company. And I think it’s a terrible way to think about delivering software.
Gong: Yeah. I definitely second that. I mean, what Robina just articulated pretty much echoes what pretty much all of our enterprise customers have said. They should absolutely transition from where it’s just TNM type of model before now to it’s all about outcome-based. And that’s what we prefer as well. To Robina’s point, why the customer should care how we deliver it as long the outcome is what they want, right?
We actually created a digital agility transformation manifesto [laughs] in honor of agile manifesto. And one of the top one is actually called “Value over volume,” which is exactly what, Alan, you just talked about in terms of, hey, you should not be measuring success with transformation by the volume of things got done. But it’s more about the value, right? So in this case, instead of measuring, for example, how many DevOps ____ did we create – that’s not important. What’s important is: how much faster are we able to help the team to deliver software to the production faster? What’s the frequency increase? So those things are what matters. So that’s what we absolutely focus on.
Shimel: Excellent. Guys, we’ve got maybe seven or eight minutes and there’s one area we haven’t hit on yet that we’d be kinda negligent if we didn’t hit on it, and that is: what effect has COVID had on all of this? On one level, we could say, “Well, look, everyone’s looking remotely anyway. What difference does it make if they’re an employee or an outsourcer? We’re all remote. We’re all equal in that regard.” The other way is: “How do I graft these people into our culture? How do we make them part of the team when we’ve never shook hands in person, when the only way I ever see you is on a “Brady Bunch“ Zoom window here and you take the top left?” Right? “Can we really integrate in just remote-only settings?”
Ming, Chris, you guys deal with it. You’re dealing with it over and over again. Robina, you are too. Right? Everyone’s remote. How has COVID helped, hurt, hindered, accelerated – how’s it affected this?
Rolls: So, I think one thing is that if we asked people five years ago, “How do you do agile? How do you do DevOps?” you’d say, “Everyone has to be in the same room. They have to be collocated. You’ve gotta have stickies up on the wall.”
Shimel: Chris is kinda sticking on us here. Okay. Go ahead, Chris, I’m sorry. You kinda hung on us. We got stickies on the wall.
Rolls: Yeah. So I was saying in the past people would say, “You have to have everyone in the same room, stickies on the wall, and that’s the only way you can do agile and DevOps.” And I think one thing that has been good with COVID is that it has dissuaded us of that illusion that that is the only way to do things because we have been forced to do it differently. That being said, it is so much harder to create culture, whether it’s in-house or whether it’s outsourced, when you don’t have that face-to-face collaboration. There’s only so many times people can do a virtual happy hour and still really get into it. So you have to be – I’m sure first time everyone’s like, “We’re having five happy hours a week. I’ve got my bottle right next to me.” And you’re two months in and you’re saying, “Geez, I cannot do this anymore. I just wanna go to bed or watch “Sopranos,” right?
But I think that we really need to focus on building that culture and come up with creative ways to create connections between people.
Shimel: Agreed. Ming?
Gong: Well, let me see. Hold on. Give me a sec.
Shimel: Okay [laughs]. You going to your happy hour background?
Gong: Absolutely.
[Laughter]
What you’re seeing here is the – my family and I, we took a hike recently to a Malibu beach, and I took a 30-second video of Malibu and use that as my background. So this is one of the things that I started asking my team to do: trying to personalize this kinda virtual experience. I’m a absolutely people person. So I miss the in-person bonding dearly. But in lieu of that, we basically, every time we have team meetings with our customers or with our own team, we pretty much strongly encourage everyone to get on the video so at least we can see each other. But to Chris’ point, there’s always a silver lining to everything. Because the COVID certainly forced or speed up a lotta things that was gonna happen anyways. So, to that, I think it’s a positive thing. Robina? I’d love to hear your thoughts [laughs].
Laughlin: Sure. What a year we’re all having, huh? Surprised. So, I think in terms of working with your partners through COVID – and we’ve been talking a lot about culture – I think one of the foundation things is to pick a company or a partner that has a very similar culture to you. So then you’re not trying to find middle ground in that. And that should really be a big part of your selection process. You need to be comfortable with the folks that you work with and you need to be living kinda to the same standards I think.
So for me, how COVID affected me was: while we were all scrambling to get ourselves home and reaping the benefits of a long-term plan to be able to have a remote workforce – we learned that through Sandy a long time ago and we had done a lotta changes – I was also working really closely with Cognizant in helping to get all the resources that they needed to get my folks home at offshore, right? So I was as concerned about offshore as I was concerned about onshore.
And now to this day my job is actually pretty high-touch now. And the reason being that we’re so far along, I have people that’re really overworking themselves. And I think that – and I don’t know that it’s deadlines as it is: take pride in craftsmanship, and they’re home and they wanna – this is something for them to do. Or it’s a stress release in a really challenging time where everything’s unpredictable. But I spend more of my time saying, “When was the last time you took a day off? When was the last time you took an afternoon off? You need to recharge. You need to gather yourself.” And that’s for offshore too. So that’s been my challenge this year.
Shimel: Fair. Mitchell, I know you guys at Accelerated Strategies Group did a study on the whole remote thing and COVID. Thoughts?
Ashley: Boy, I could talk for an hour on this. But I’ll just point out a couple things and I’ll point folks to the report. We looked at – software developers already know how to work remotely, right? A lot of them do. And so how has that affected them? How has COVID and the work from home affected them and how have they kind of affected the organization?
So, interestingly enough, across the board – there’s about 350 participants. Across the board, everyone is above 60 percent of who have accelerated digital transformation projects, contact – new forms the delivery and automation. But what I really found interesting was how influential DevOps practices, whether you called it DevOps or not, had started to infiltrate into the organization. So I’m talking about things like doing daily standup meetings or more cross-functional teams, increased use of that, asynchronous communications like a Slack or a team as opposed to being on a phone call or in a meeting. So there’s all these ways of working that, as we were talking before, sort of forced everybody into it. And the teams that had really good software teams working remotely jumped on board. And you saw an increase in the use of that kind of style of work, sometimes facilitated by tools, sometimes just process.
So if you wanna read the whole report – and reading is minimal; it’s all infographics; it’s super easy to consume – go to ACCELST.com. And it’s free to download. And enjoy. And let me know if you have questions. Love to talk to you about it.
Gong: Awesome.
Shimel: Excellent. Guys, we are outta time. We’re actually over time. Robina, Ming, Chris, I wanna thank all of you for joining with us today. This was a great discussion. In the show notes, production notes, we will have links to your websites and any other links you wanna put in there. But all three of you are dealing in the real world with real problems that we’re seeing today and helping organizations solve ’em, whether it’s your own organization or other organizations. So keep up the great work. We may revisit this topic on DevOps Unbound someday, and we’d love to have you back on here. Okay?
Gong: Pleasure.
Shimel: Thank you so much.
[Crosstalk]
Laughlin: Nice meeting you guys. Good luck. Thank you.
Shimel: All right. Mitch, thanks as always, man. This is Alan Shimel for DevOps Unbound. Thanks for joining us. We’ll see you on another DevOps Unbound real soon.
-
EP 4: DevOps and the Mainframe
In this fourth episode of our DevOps Unbound streaming broadcast on TechStrong TV and devopsy.kinsta.cloud’s sister site Digital Anarchist, Mitchell Ashley of ASG and Alan Shimel are joined by Gerta Sheganaku of Tricentis, Rosalind Radcliffe of IBM and Sam Knutson of Compuware to discuss app modernization, mainframes and DevOps. Find out what’s the right way to modernize your applications and how to do app modernization in a Z environment.
The video is immediately below, followed by the transcript of the conversation. Enjoy!
Transcript
Alan Shimel: Hey, everyone. This is Alan Shimel. Welcome to another episode of DevOps Unbound. This is episode four of DevOps Unbound, and then we’ve also done two DevOps Unbound roundtables. So this is actually the sixth DevOps Unbound event that we’re presenting. They’ve all been great, and I think this one is gonna be as good or better than any of ’em.
Today’s topic is app modernization and mainframes, along with DevOps, of course. It’s DevOps Unbound. We’ve got a great panel lined up for you. Let me introduce you.
First of all, we have Sam Knutson, and Sam is the product manager — is a product manager for mainframe and agile DevOps at Compuware, a BMC company. Sam, did I get that right?
Sam Knutson: Pretty much, Alan.
Shimel: Correct me. Get it right for the record.
Knutson: So I lead product management at Compuware. We are a BMC company, and we are focused on enabling agile and DevOps on the IBM z/OS platform.
Shimel: Fantastic. And the lesson I have learned here is I’ll let each of you introduce yourselves. My next guest is the one and only Rosalind Radcliffe. If you want to know about DevOps and mainframes, there’s no better person in the world to go to.
Rosalind, welcome, and why don’t you give them a little bit of your title and background?
Rosalind Radcliffe: Thank you. Glad to be here again. Rosalind Radcliffe, IBM distinguished engineer, responsible for DevOps for enterprise systems. I help clients with their transformation to DevOps for z/OS applications as well as application modernization and bringing z/OS into the rest of the world so you can do your cloud-native development for z/OS as well.
Shimel: Excellent. Then we have – I’m gonna do my best here – Gerta Sheganaku of Tricentis. Gerta, how bad was that?
Gerta Sheganaku: Thank you, Alan. That was perfect, really.
Shimel: Okay. Gerta, your background and title?
Sheganaku: I’m with Tricentis, as you said. I initially started at Tricentis in the customer experience space, where I was leading our customers through their digital transformation journeys, really helping them understand how they can enable testing as a catalyst to really get DevOps working.
Then I have moved into the product organization, where I’m currently leading a team where we’re releasing our first AI-based test automation solution, which is really about tackling the problems that we see out there, especially when it comes to app modernization, and testing of old applications and new applications and really making testing easier.Shimel: Excellent. And then last but not least is my sidekick, well, partner in crime, Mitchell Ashley, CEO of Accelerated Strategies Group and also my co-host here on DevOps Unbound. Mitchell, good to see you.
Mitchell Ashley: Good to join you.
Shimel: All right, guys, today’s topic is app modernization, mainframe, and DevOps. It’s interesting. I’ve been kind of binging on Netflix that AMC show, oh, “Catch Fire”? I forget the name of it now. Anyone else watch this show? It’s an AMC on Netflix.
Knutson: “Halt and Catch Fire”?
Shimel: “Halt and Catch Fire.” You’ve watched it, Sam?
Knutson: I’m familiar with the instruction. I’ve seen the previews for the show.
Shimel: It’s a pretty good show once you get into it. So the company that started in Texas bets everything ’cause they’re tired of paying the timeshare people, and they buy an IBM, what is it, 3030, that’s based in Daly, San Francisco, or Daly, California, near San Francisco. They literally have to move the whole company to where the mainframe is versus moving the mainframe to where the company is. Well, it’s TV. But that was the late ’80s, right, and we don’t live in those worlds anymore.
Mainframe applications are running on your phone, right, or at least the front ends of them are, on your tablets, on your computers. The mainframe doesn’t necessarily fill up – this was a whole basement that was one mainframe. It was pretty funny. The guy was sliding cards in and stuff like that. If you get a chance, watch the show. It’s an interesting, interesting look back.
Certainly the modern mainframe has come into the world of DevOps. Rosalind, I think the work you have done with IBM has led the market with that. Quite frankly, Sam, the work Compuware has done has led the market in agile and DevOps on mainframes. How would you assess where we are currently in mainframes in the world of DevOps?
Radcliffe: You mean where the mainframe is or where people are? I think those are two very different things.
Shimel: You’re right. But what we’re not moving the whole company to the computer anymore, thank God. But let’s talk about where is the mainframe, and then we’ll talk about where the people are.
Radcliffe: So you’ve got to remember, the mainframe itself, IBM Z, is the most modern machine out there. The IBM Z platform has evolved every few years for the last – ever since it was invented, and we keep evolving the machine.
So the machine itself is the most modern system, and we’ve evolved the operating system. We’ve evolved things such that open source tools work there. We’ve done everything to remove the barriers from the system so that you can do exactly the same kinds of things, whether or not you’re doing z/OS development or another environment. And the Z hardware itself, we all know it runs – well, we all know on this call it runs Linux on Z.
And so it’s a very large Linux server. It runs containers, and it runs OpenShift. So from a systems standpoint, it provides – there’s nothing keeping it out of the processes and practices. There’s nothing separating it from a systems standpoint. It’s the way people interact with the systems. It’s their existing applications that make it harder.
Shimel: Sam?
Knutson: I think Rosalind makes some great points. The current generation of IBM mainframes, the z15, is an engineering marvel, and the compilers and the operating system produce this amazingly efficient platform. It is the most efficient hyperconverged system in the world today.
But I think the elephant in the room is the word that you used a minute ago when you said “modernization.” We alluded to this a little bit in the way that the practices that people and businesses have today. Well, modernization doesn’t mean moving off platform, because we already talked, and we could go on for an hour, about how great a hardware and software platform this is for running business computing. It is amazingly efficient, especially when you look at the large-scale, global business applications that the Fortune 1000 run, which is where the platform has its largest footprint.
But the perception is that modernization is moving your application into a cloud-native application off the platform, hosted by another provider, or rewriting it in Java, and nothing could be further from the truth. The COBOL compiler alone is a perfectly paired piece of engineering with that platform, and it lets you wring every last ounce of efficiency out of it.
But people need to get past that confusion and say, “What if I could work on the platform in the same way I’m working on every other platform?” It’s that normalization. We’ve kind of coined the phrase “mainstreaming,” “mainstreaming the mainframe” to make it different in syntax only, and Rosalind alluded to this. It’s being able to work on this platform in the same way that you work with any other platform. So when you think about DevOps, then you have a DevOps toolchain that looks like other toolchains, that you can connect all of the ecosystem partners that you’re probably already using in distributed development.
People are definitely the problem, people and practices and culture. More than three-quarters, in fact 78%, of customers say it would be useful if they could update their mainframe applications more frequently than they do today. So there is an aspiration to change and to get better outcomes. You can see what they want. They want to be able to deliver value to customers more frequently, and that’s just one of the improvements that they want to make. But far too many people are not making progress on that.Shimel: Gerta, so you’re not a mainframe specialist, but certainly app modernization and testing of this. I’m wondering if you have any kind of information regarding people using Tricentis to test mainframe-based applications. How big a market is that?
Sheganaku: That’s a very good question. One major thing that you always have to keep in mind, as we just said, especially when talking about app modernization, it is such a wide word, and it can mean so many things, too, to different people. It’s definitely something that’s challenging for our customers looking at their current system landscape. You see customers that are having everything in their landscapes from mainframe applications to also often very legacy mainframe applications that are trying to move onto more cloud-based environments. That is something that we see very often.
Mainframes are still there, but the thing is that people really want to abstract away from the mainframe. Also thinking about where we are today, I think an important question we have to ask ourselves is why is everyone so into app modernization? Why is everyone into migration into the cloud currently?
If you think about it, nowadays every company basically is becoming a software company, and not every company has the talent and the resources to maintain those mainframe applications because at the end of the day, honestly I struggle finding COBOL developers nowadays out there. I went to the Technical University of Vienna, and we definitely did not learn anything around COBOL there. It’s really hard to find the talent nowadays.
When you look at it, basically every company, be it in retail, be it in banking, be it in the energy sector, you now want to have connected systems and so on. Really, everyone is going more towards being a software company. And with that, for people it’s becoming more and more increasingly important to have really good platform-as-a-service offerings, maybe going into serverless or software-as-a-service in the next step, but really having not really to deal with the underlying maybe mainframes and having a way of abstracting and using simpler technologies and only caring about the applications and the data that they’re running on those systems.Shimel: I’ll be honest with you, I don’t know if I agree with that. So here’s the premise of this, right? When we talk about app modernization, I’d love to live in a world where I could throw out an old app ’cause I want to have a new app. But we don’t live in that world, right, and no one lives in that world. That’s why we call it “app modernization.” We tend to modernize the apps we already have.
And so when we look at – I think out of the Fortune 100, 98 are on mainframes. Or it’s some number like that, Sam or Rosalind, if you know. I don’t see 97-and-a-half of them throwing their mainframes out, right? As a matter of fact, they’re probably building more on mainframes. And so what’s happening is we’re seeing whether you want to call it hybrid cloud, VoLTE cloud, whatever you want to call it, we’re seeing where people can build maybe a front end for stuff in the cloud. But if that data is important, if security is important, if scalability is important, dollar for dollar, people who use mainframes know that that’s their best bang for the bank. Mitchell, I’ll let you chime in.
Ashley: I have to jump in. So what’s interesting is we kind of do this to ourselves with the terminology that we use. “Mainframe” is a word for a cabinet used to house the computers the original IBM computers were stored in.
If I said, “Hey, 71% of Fortune 500 companies rely on high-volume transaction-processing platforms. Ten out of the ten world’s top insurers, 23 out of the 25 world’s top airlines, 92% of the top 100 banks rely on high-volume transaction-processing platforms,” you would say, “Wow, that’s pretty cool.”
“And it runs Linus, and it runs containers, and you can connect it to all kinds of applications from tablets and phones to wherever your application may be running, on cloud or on premises.”
You’d say, “Wow, tell me about that. I’d like to learn about that.” That’s what mainframes are.
So it’s interesting. It’s not going away. We do this to ourselves, and I say the analysts have done this to us, too, right? Today it’s all about this next shiny object, the squirrel moment. It is so vital to businesses today, and moving it into how we create software with DevOps and processes like that and tools to support it is essential for businesses’ both economic recovery, survival, and future. Digital strategies can’t happen without it for many, many, many companies.
Okay, did I do a good job, Rosalind and Sam?Knutson: Mitch, those were great stats, and I think you did a great job quoting ’em. I’d boil that down to simply saying the worldwide economy and worldwide civilization runs on the mainframe. And I like the word “mainframe.” It has a little swagger. I don’t think it’s ever gonna go out of style.
Ashley: Nostalgia.
Knutson: And renaming the platform could make it cooler. It simply gets things done. We haven’t renamed cars yet. Well, maybe we’ll rename them to “highly efficient transport platforms,” but I think we’ll still be calling ’em “cars” when they have electric engines powered by portable fusion reactors. But customers see this as a long-term platform. BMC does an annual mainframe survey, and the 2020 edition is gonna launch soon. I can tell you that 90% of the customers see the mainframe as a platform for growth and long-term applications. Sixty-eight percent of ’em expect the size of that compute platform to grow, MIPS to grow. Sixty-seven percent of those organizations have the majority of their data on the mainframe. They trust it because, as you said, it’s highly available, scalable, secure.
And to go back to Gerta’s key point about younger workers, that is absolutely essential. But what’s going on is that people are addressing that because – I think Rosalind and I both alluded to this – the syntax of COBOL isn’t Aramaic. It isn’t some incomprehensible script.
If you put modern tooling around this, if you put modern process and a great, supportive culture that gives that great – you have a great developer experience, you have a great developer culture, then learning the syntax of COBOL is no more difficult than learning the syntax of Perl or Rust or Go. So we see that customers who see the platform long-term are making the appropriate investments. In the responses to that survey I mentioned this year, younger mainframe workforce definitely figures in, 43% less than five years on platform. Sixty-eight percent were under 49 years old, and 60% of that new generation see this as a growing platform.
That really matches the experience that I’ve seen in Compuware’s journey is that we hired out of college and we have been hiring out of college, and those next-gen developers are more than a third of our workforce now in Detroit. Many of them did not know COBOL, and many of our applications are written in assembler. They certainly didn’t learn assembler in college. But what they did have was a strong bent for learning new technologies. Many of them came with computer science degrees. They had learned how to learn. They were what Gartner calls “polyglot programmers,” one, two, three, many in learning languages.
So whether we needed to learn assembler or COBOL or C, it didn’t matter as long as we provided that supportive culture, the vehicles for learning, and we gave them meaningful work early on in their employment and made sure that they understood the vision, mission, and goals of the firm and how they were contributing to something that matters. I think our customers are having similar success in onboarding next-gen developers by doing some of these same things.Radcliffe: I think it’s interesting, the statement, “I can’t find COBOL developers,” because when I was hired in to work in this industry, nobody assumed you knew the language or the system you were going to work. That was the thing you learned. You learned how to develop, and then you came in and learned the company’s culture and the company’s tool choice.
By using modern tools, modern practices, all of these things, the things they have learned in college on how to do, they’ve already got. And then the only thing they’re doing is learning COBOL.
As I always joke, it’s just English. I know that works really well in the US and in parts of the world, and in other parts it’s, “Okay, it’s still a foreign language.” But it’s not a difficult language. It’s easy to understand. PL/I, it’s not difficult to understand. It’s just another language, and developers can learn other languages.
I am kind of with the discussion about “mainframe” is not the word we should always use, and we really need to focus on how do we talk about our platform. I like the term “mainframe,” too. I still use it. But can we talk about our platform as this platform that provides this valuable, high-transaction server? It’s that big server as part of your pipeline. It’s not that old thing. It’s that big new thing that runs all this work, and now let’s focus on the applications that – Honestly, the best thing IBM ever did was a program you compiled in 19-whatever and still works today, and the worst thing IBM ever did was that program that you compiled in 19-whatever and still works today. It’s the best and the worst of the platform. You don’t have to change. It still just works.
And if we can help companies understand if they start to change some of those, they can do the automated testing. They can do all those things. They can get more value out of their platform. They can help refactor and be able to move at the speed they need to.Shimel: Go ahead, Gerta.
Sheganaku: These are all amazing points. I hope no one got me wrong at the time beginning with what I wanted to say. I really like what Rosalind just said, to talk about the underlying platform that is really enabling. But the point that I really wanted to make is that most companies are now becoming software companies. We don’t have that knowledge center of IBM, where there are so many smart people and you can learn from people and really develop that amazing platform that you offer other companies.
But when I look at our customers, it’s usually self-built kind of mainframes that are sitting there that are not maintained that don’t apply current recent security standards. Those customers, what you see them do often is they all hear, “The cloud is the next thing. We all have to go to the cloud.”
They all do this one big mistake that they just take whatever they have and they basically do the lift-and-shift approach, bring it to the cloud, and hope that everything’s gonna work. Of course that’s not gonna work at all. Also from a cost perspective that is insane, thinking about how much cloud resources actually cost you if you don’t do all the optimizations and so on. So what you see people do is kind of a step-by-step learning approach where they are taking things from not-so-secure systems, bringing them over to other systems, then starting to refactor the applications, and giving more and more also responsibility to the development teams.
Once they do that, you see the development teams really getting that DevOps approach, really caring about the infrastructure, and so on. For the developers, of course it’s easier if they have container-based applications that they can simply deploy and so on. But of course underlying it’s very, very important that you have a secure and great platform that is enabling all of that.Shimel: I think that’s a great case for why you’d want to use a mainframe or whatever you call it, ’cause it’s probably more secure than the cloud is, I think, and it’s been proven. But here’s an interesting thing I’ll tell you here I see in devopsy.kinsta.cloud, right? This is devopsy.kinsta.cloud, mind you. And it was surprising to me. Our biggest article last year on devopsy.kinsta.cloud, most views, most comments, “The Beauty Of the COBOL Programming Language” by Bob Reselman.
I don’t know, it had half a million or something, maybe more, a crazy amount of views. Most of the comments were by COBOL programmers in India because there they recognize that there’s such a need for COBOL programmers that you have a lot of young people who are studying COBOL because they know it’s basically a guaranteed job for them, right? They come out and they start working right away.
Here’s another interesting thing, and my friends in Tricentis will notice – will appreciate this. What do you think was another huge area of interest in devopsy.kinsta.cloud? Running SAP in a DevOps world, right? We don’t think of SAP as DevOps-enabled or maybe not even agile.
We did a roundtable a week or two ago on SAP in DevOps. I had to cut the questions off, and these were meaty questions: “How do I do this and that in this and that release?” It wasn’t just pie in the sky. They were very how-to, practical, “We’re running SAP, and we’re moving towards agile and DevOps and we’re doing these things.” So while everybody – we always rush to the shiny new trinket and the hot thing in our industry, I think the fact is certain things endure. As Rosalind said, it’s both their greatest and worst attribute. It is.
But let’s talk a little bit about app modernization ’cause I think Gerta alluded to it. Mitchell, I’ll throw it out at you. No one wants to do shift and lift or lift and shift or whatever. It doesn’t work in 2005 when the cloud first started, and it doesn’t work now. But how do we do app modernization right in a Z environment? Sam, Compuware has made a – it was the reason for your being in a lot of these last couple years, right, something Chris O’Malley always harps on. How do you do modernization right in a mainframe world like this?Knutson: First, you do it on platform because – I can’t say that enough – when you see people try to avoid touching the mainframe apps that live on the mainframe, they do unnatural things from an architectural point of view, and they create these horrible kludges that are not robust, that are not scalable, that are not the most efficient implementation, and that aren’t easy to maintain. So you want to bring those practices on platform. You want to provide developers with helpful editors, with quality tools. You want to tap into the automation pipeline. So build a CD pipeline that goes right onto your pipeline so that you give developers a great developer experience, but you keep the business logic and the data on the platform.
That goes to a number of things. When you talked about security, something that I learned a long time ago, the more copies of data you have, inherently the less secure it is, because typically when mainframe data is moved off the mainframe, the security controls that are on the platform are not replicated.
So if you can take that core business logic, that digital DNA that a business has inscribed over 20, 30, 40, 50 years, continue to leverage that, and of course leverage systems off platform. Compuware really blazed this trail of using cloud and mainframe together with two-platform IT. We think these things are incredibly complementary. But what you don’t need to do is replicate that data all over the place. You don’t need to necessarily have a lot of on-prem systems. The term “hybrid cloud” was used, and that’s a nice, modern way to talk about that. We say “two-platform IT” because these things will pair very, very well.
And the cloud, by the way, is an absolutely great place to host commodity applications. If you talk about your company’s e-mail system, I probably cannot run Microsoft Office better than Microsoft can with Office 365 in the cloud, and they can provide all the quality, the service, the latest updates. It’s a good solution. Can I do invoicing better than a cloud provider like NetSuite or Oracle? Probably not.
But what I want to do is focus on those applications that give me unique business advantage, the things that I wouldn’t find in Amazon, Google, or Microsoft’s application catalog. Those applications give me a competitive advantage, and in updating them I increase that competitive advantage so I can provide new features, new capability. I want to take that on-platform application. I want to bring to it things like automated testing. Make it part of a CD pipeline. Give my developers a great set of tools and give them a great culture of development to help them help the firm be successful.
That is the best approach to modernization I know of, and of course I work very hard to provide those things, including program understanding, so that developers can really get into code that they may not be familiar with, because when I talked about a younger mainframe workforce, that’s a good thing, but it also means that I don’t have people who have apprenticed with an application for ten years before being allowed to do important changes.
You have people who are new to your firm, and they’re being tasked to do important work. So they need to be able to understand an application that may be older than they are quickly enough to inside a two-week agile sprint make a change, test it. Hopefully there’s already automated test present, but if there is not, they want to write some new automated tests and deliver that all as working, demonstrable code at the end of that sprint because that’s the way the world works today. So if you do that, you’re blazing down the road towards application modernization. It’s easy to talk about, but it boils down to just hard work.Ashley: Alan, I think to build on what we’ve said here, I look at it as three elements to the strategy here, and this is just also sharing my own personal experience with doing this. First is the people and the culture and getting our heads around new processes and new ways or additional ways of doing things. That’s what DevOps is about.
Second is processes and tools, which Sam was just eloquently talking about. I think the third obstacle to app modernization is tackling the monolith because a lot of our applications were built differently, that we’re looking at how do we modernize. Well, why do you want to modernize? Sometimes it’s because we have to figure out a way to get in there and better maintain it ’cause it’s just too scary to tackle. The people that built it aren’t there anymore.
Another reason, though, and I think the way to tackle that monolith part of it, is looking at our applications and saying what functionality and what business capabilities in the apps are most important to having the most flexibility or improving the speed at which we can improve that software to respond to and support the business?
If we’re living in a world of changing how we’re delivering services into contactless, we’re living in a world where supply chains are going from inventory to on-demand, whatever it may be, examine those applications. Pick out where those elements are. And those are good candidates for going and saying, “Let’s carve out that piece and look at some new or alternative designs for how we can more easily maintain or more quickly develop capabilities in that part of the software.”
It doesn’t mean you move it off the mainframe. It may stay there. It may be a combination of things. But that’s also part of app modernization is how we create software and the tools that we do it and sometimes how do we take on really big problems but divide ’em into much smaller pieces that we can do something about.Radcliffe: The application modernization of a Z application does unfortunately involve understanding it. And so having those tools for application understanding is the first key, but then actually understanding the business is the other key. And so it’s a combination of application modernization tools and understanding the business domains you’re trying to go after, really thinking about what are the focus areas and understanding what you need to run.
Lots of discussion about microservices, and I always question, “Do you really want a microservice, or do you want a service? What level is the right componentization for this function?” And I always use a banking example because this one’s the easiest. “I need my credit card swipe to happen, and I only have a certain number of nanoseconds, whatever it is, to make that happen.”
I’m not gonna make that a thousand microservices. I’m gonna make it a service. That service has to run fast. It has to be efficient. So it needs to be written as not a monolith, but a service itself. Where is that service? Is it bundled in the middle of something else that’s doing a whole bunch of other function? Can I strip it out? I need to look at the applications. And so using application understanding and the skills we still have today, because in many companies they actually still have a few people left that wrote those apps, not maybe for much longer, but they have ’em now. Let’s start to look.
Use application understanding, the next generation, and learn from each other. I think the next generation, the newer developers, have a lot of skills in a lot of the new tools, and the people my age have the experience in the apps. So let’s bring these people together to help build this new culture, to help bring this together. Let’s take advantage of both sides and the tooling because even if they helped write the application, they still don’t know the whole thing anymore. It’s a monolith.
Bring it together, and take advantage of these people. Use the people, both sides of the people, and bring all those skills together with the tooling to actually help you understand the business domains, the functions that you want to do, and split up that monolith. Maybe you’re splitting it up into a bunch of COBOL programs. It doesn’t matter. You’re providing the business value that you need to in a more maintainable, manageable way.Shimel: Fair. Guys, we’re running closer to the edge on time. I wanted to touch on the issue, though, ’cause it’s what Gerta does a lot of, and that’s around testing, right? As we’re modernizing these mainframe applications, we’re keeping them on the mainframe. Maybe we’re hybridizing them, if that’s a word, putting front end here and backend there or whatever. We’re modernizing the applications.
Testing is an important part of it, as it is always. Anything special in particular that we’re doing around the testing of the modernization of mainframe applications that perhaps is different than – Gerta, go ahead.
Sheganaku: When you think about testing, really just the whole topic of app modernization and also testing can be split into so many different bits and pieces. The one bit is really the architecture that we’re speaking about, where is everything hosted.
Are we dealing with a monolith versus are we splitting everything up into service-oriented architectures or microservices or whatever we want to call those things, really making sure that we shift testing more to the left, that we make sure that the components are tested and such.
Then the other side is really around the infrastructure, so really thinking about do you have those physical servers or do you have now more VMs, cloud-based environments? Do you have containers and so on where you’re hosting your _____ [break in audio] making sure that the intersections are tested properly?
And then another point that’s also very important is really also the process side of things, so the delivery side. Are you having more of a waterfall environment or are you going more towards an agile DevOps environment? And all those things also go hand-in-hand.
The typical approach would be especially you would have some more legacy type of applications that are running in the background that are really the backbone of your business, and then you would have more flexible applications built on top of that. For each of those you need different testing approaches.
Something that we saw here, especially when we say “shift testing to the left,” was that while it was so far really possible to shift testing to the left when it comes to doing unit testing, doing API testing, and so on, it was always very hard when it came to end-to-end testing, especially across systems, when you have different systems, older systems, newer systems all working together, because there you really didn’t have that concept of just defining your tests up front and once everything is ready you just run your tests. It was really very difficult.
Especially now I am focusing on the space of UI testing currently. In the UI space it was basically impossible to do that. That is also something where AI approaches can help in the future because if you look at the applications that we’re interacting with, no matter what technology is used in the background, no matter what type of programming language we’re using, if we’re having something in the cloud or on a server, it doesn’t really matter. How we interact with the applications as humans hasn’t really changed so much over time.
When we looked at banking systems from 20 years ago and made an analysis, they look just the same like today. The applications that we work with on a day-to-day basis, although they move from one platform to the other, the technology keeps changing, bits and pieces keep changing, the way we interact with it really doesn’t change so much. So there it really helps to bring in AI-based approaches that help you take that human perspective when performing those tests to really be platform-independent.Shimel: Excellent. Sam, Mitchell, Rosalind, any thoughts on testing and app modernization with the mainframe?
Ashley: I would just jump in and say it’s a great place to start, not to say you’re starting from scratch, but maybe you are, that whole CI/CD testing. As Gerta was saying, you can use cross-platform approaches, right? It doesn’t have to be one set of tools for this, one set of tools for that, et cetera.
It can be, but there are some advantages to leveraging that knowledge base that Rosalind was talking about. You’ve been doing testing. Now how can you take that to the next level, whether it’s getting quality up or speeding up how quickly you can do releases, whatever the driver might be? As Gerta talked about, those are some fantastic approaches to take.Shimel: Rosalind?
Radcliffe: Yeah, testing is one of the ways that we see people getting value fast, and it in some ways takes less cultural change. It’s easier maybe to get them to do automated testing, and automated testing is actually what makes you faster. You need that ability to test the system.
Shifting left using virtual services, shifting left by virtualizing the whole system, allowing you to test your application in the build process, and doing unit test really does help in the mainframe space to get that feedback. It helps in every space, actually. But in the mainframe space, we seem to have forgotten this. I don’t know why. When I started, we did it. Okay, 20 years later people forgot. Getting to that unit testing, that automated unit testing, and making sure we’re doing that virtual testing, whatever it is to get that application transaction and really have that capability, and there’s lots of tools out there to do the testing.
And the other part is it has to be cross-platform ’cause most Z applications are not Z-only. So open source tools like the Galasa framework that help you test across distributed and Z matter. Things like test data as a service matter. Use optimizations, looking at AI for data and understanding what data should you feed this.
Multiple people have capabilities around that to look at the right data to test to make sure you get that fast feedback all the way at the beginning and get that developer that input so they aren’t waiting that six months of testing, that late-stage testing that is the reason why everybody says the mainframe is so slow because they spend too much time at the end testing. Let’s shift that all and bring it to the beginning, and then there are people delivering weekly into production because they have automated testing in place.Shimel: Absolutely. Sam, I’m gonna give you the last word, and then I’ve gotta wrap it up.
Knutson: I think automated testing, and you have to underscore “automated,” is critically important. It’s one part of this. It’s not about the old, “Fast, cheap, good: Pick two.” People need to continuously improve quality, velocity, and efficiency on the platform, and automated testing is a key part of that. You will get measurable results, and you have to measure the results. So you have to put automated tests in place, you have to put quality aids in place, and then you have to measure that.
If there’s one thing that we probably didn’t spend time today on that I think is a worthy thought for people to put front and center, it’s you’ve gotta measure from the very beginning what you’re doing, because any effort like this is going to entail some amount of spend in terms of resources. It is a worthy spend. It is the natural evolution of development for the platform.
But you have to treat your developers like the high-performance athletes that they are. You have to give them continuous feedback. So use an analytics platform where you can understand what’s going on with your development process, what’s going on with your testing, so that you can see the results.
When I looked in-house, what we were able to see is that as we evolved our development approach from waterfall to agile and we put these automated tests in place and we gave our developers a great developer experience, continually improving their tooling and our toolchain, we were able to get measurable results, like 25% more trapped defects that were caught before they ever escaped into customer hands.
We were able to get twice as much code from individual developers and deliver at the speed of business. We didn’t get those results if we didn’t make the changes, including automated testing, including a great developer experience.
None of it is optional. You don’t get a hall pass on any of this. Pick one and start, but you really have to think – this is a journey, and you’re gonna have to address all of these if you’re gonna get the outcomes that you want and be the most competitive that you want.
I’ve seen customers getting amazing results that are measurable by outside firms where they’ve done studies and been able to see more than 467% ROI from implementing automated testing and building net-new applications on the mainframe. So this works. I’ve seen the proof points. I’ve seen the customer case studies. I would just urge people who have been sitting on the sidelines to get in because the competition absolutely is modernizing the way they build applications on the mainframe.Shimel: That was a strong finish, Sam. Thank you very much. Good stuff. Hey, Sam Knutson, VP of product at Compuware, a BMC company. Rosalind Radcliffe, Distinguished Engineer, IBM Enterprise Systems. Gerta Sheganaku from Tricentis on product for AI testing. And of course the one and only Mitch Ashley.
This is Alan Shimel. Many thanks to Tricentis, by the way, for sponsoring DevOps Unbound today. This is Alan Shimel. We’ll be back in about another two weeks with another great DevOps Unbound. Stay tuned, but for now that’s it. Take care. Bye-bye. -
EP 3: Open Source and DevOps
In the third episode of our DevOps Unbound streaming broadcast on TechStrong TV and devopsy.kinsta.cloud’s sister site Digital Anarchist, Alan Shimel is joined by Kevin Dunne of Tricentis, Tracy Miranda of CDF and Alon Girmonsky of UP9 to answer the question: Can we have DevOps without open source?
The video is immediately below, followed by the transcript of the conversation. Enjoy!
Transcript
Alan Shimel: Hi everyone. Welcome to another episode of DevOps Unbound. DevOps Unbound is sponsored by Tricentis, so many thanks to them. But it’s our show here at MediaOps and we try to explore, you know, just some of the areas around DevOps and not just dev and ops but DevOps as a framework for the way things are being done, the way application and software is being made and deployed today, the way it’s being hosted, the way it’s being operated and maintained and observed, and so many other aspects of how we’re doing technology.
It airs every other week and this episode is episode number three of DevOps Unbound. Hope you’ve enjoyed the first two.
With that let me introduce to our panel today. First of all, well, it may not be right or left on your screen, because so it makes no sense to tell you how it is on my screen, but let me first start off with my friend, Tracy Miranda. Tracy is the executive director of the Continuous Delivery Foundation, which is a Linux Foundation and a sister organization to Cloud Native Computing Foundation. Got that all out of the way. Tracy, welcome.
Tracy Miranda: Hi everybody and thanks for having me Alan.
Shimel: My pleasure to have you on Tracy.
Joining Tracy and I today, I have Kevin Dunne, who has been on our TechStrong TV in the past. Kevin is with Tricentis and runs their open-source communities, as well as some of the open-source projects that are now under their management, some by acquisition over the years, some they’ve developed themselves. But Kevin, welcome to DevOps Unbound.
Kevin Dunne: Thank you, Alan. Happy to be on and glad you’re having us back with you again.
Shimel: That’s our pleasure.
Then last by not least is my friend, Alon Girmonsky. Alon is the CEO/founder of UP9, that’s U-P-9. He’s going to tell us about that when I ask everyone to introduce themselves here in a moment, but Alon was also the founder/CEO of BlazeMeter, a very well known open source or player in the open-source testing world acquired by CA, which of course became Broadcom.
So, today’s episode ladies and gentlemen is around DevOps, cloud native and open source more importantly, most importantly, and how important open source is to this whole DevOps movement and this whole — Someone today, I had someone from Deloitte Cloud on and I was doing an interview called it a “revolution” that we’re seeing, especially since COVID, right? IT has undergone a revolution of cloud transformation, digital transformation, acceleration and so much of it is made possible by open source. So that’s our subject today.
Before we hop into it I wanted to give each of you like 30 seconds, just in case our audience wanted more than the cursory kind of introductions I did. Why don’t we start with you Kevin? Give them who you are, your background and we’ll jump through all three of us from there.
Dunne: Yeah, that’s great. My name is Kevin Dunne. I’m with Tricentis as Alan mentioned. My main focus is on our open source and sort of community products. I’ve been with Tricentis for seven years. Prior to Tricentis I was at QASymphony, which merged with Tricentis about two years ago. So very passionate about open source, it’s been sort of a new area that Tricentis has been getting into.
There was a lot of work around Flood, which was built on top of JMeter and Gatling and now Tricentis has acquired TestProject which is built on top of Selenium and Appium and most recently SpecFlow which is an open source project based on cucumbers.
So I’m really excited to see how these tools are helping to kind of shape the Tricentis roadmap and sort of broaden the community in the open source supports, so very exciting projects going on so far.
Shimel: Alon, how about you?
Alon Girmonsky: Hey everyone and Alan thanks for inviting me. I’m a big fan both of Tricentis and obviously the CD Foundation. Alon Girmonsky as you mentioned my current company UP9, which I have the privilege of being cofounder in, deals with microservices and cloud native in respect to testing. Testing is something that is becoming harder and harder to do, especially when you know you talk about service testing and API testing. So UP9 provides a solution for somewhat autonomous, mostly automatic you know service testing.
As you mentioned prior to that my previous company, BlazeMeter, was all about load testing and stress based on the open-source tool JMeter. You know I had the pleasure over the past 10 years to be involved in the open-source community and understand or at least try to understand what’s the place of open source and why is it so important specifically for developers. I’m very happy to have the opportunity to discuss this in this forum.
Shimel: Last, but not least Tracy how about you?
Miranda: Tracy, I’m the executive director of the Continuous Delivery Foundation or CDF, which is an open-source foundation with the mission to improve the world’s capacity to deliver software with security and speed. The CDF is home to the open-source project Jenkins, JenkinsX, Tekton and Spinnaker.
Before that I was leading open-source community at CloudBees. In general I’m a 15-year veteran of open source, so I love all things open source, open innovation and building communities.
Shimel: Excellent. So folks, here’s my opening premise and we’ll let — it will be our jumping off point.
I also have a long involvement in the open source world. I first got involved in open-source security related projects 2002, 2001 and we were blessed in security to have some amazing open-source tools at our fingertips. Products like Snort, which was developed by Marty Roesch, who then started Sourcefire, acquired by Cisco, was the de facto standard for intrusion detection. The NMap and Nessus scanners, Renaud Deraison, who’s cofounder over at Tenable Network Security and Ron Gula, who started Tenable with Nessus. NMap has been out there. There were many others, ClamAV and so many commercial companies actually sprang up just by using that open-source software under the covers putting a UI on it and stuff.
So I’ve seen open source. I remember talking to CIOs and I remember speaking to you know a DOD agencies about their use of open source and they would tell me that they don’t have any open source, they don’t allow open source and then you would go look at what their developers were working on and it was all open source, you know it was Linux and everything else that they had, just didn’t bother telling them about. So I’ve seen open source become from a gorilla-type of operation in many environments, to the dominant form of software and software development today.
When I got into DevOps it became really apparent to me and this is my jumping-off point, we wouldn’t have DevOps without open source. I just don’t think — I think so much of what we can consider not only the tools of DevOps, so specific open-source projects, but the culture of DevOps, the collaboration, the sharing of knowledge of DevOps is based upon that open-source mindset, on the open-source culture that you couldn’t have DevOps without open source.
Tracy, you ran the open-source community of CloudBees first and only ED at CDF, part of Linux, what’s your feeling on that? Why don’t you kick us off?
Miranda: Yeah so definitely as a big fan of open source I’ve seen how it’s been really instrumental to DevOps. I think you know one of the things that open source does really well, like sometimes people focus just on the code, but it is really all about that collaboration, how people work together. I think at the heart of DevOps really what it’s trying to get at is building bridges, getting people to work together even if they are in slightly different spaces, whether you know it’s DevOps COPs who are trying to bring security practitioners closer to developers or ops people closer to developers. I think open source really captures that spirit of collaboration and that’s why it’s become almost synonymous with DevOps.
Shimel: Kevin, Alon, what do you think?
Dunne: Yeah, I mean I would definitely agree with that. I would say one other thing that you know of the many benefits of open source, right, is kind of just the transparency and visibility. I think you know a lot of companies were finding like DevOps in the CI/COVID process especially is kind of integral to business success, right? So a lot of companies are very hesitant to put a tool in place that is going to be that critical to their success unless they’re able to you know fully understand what it does, you know any limitations that it has, you know extended where needed and you know kind of have that ability to build on top of it as well.
I think way too many companies kind of got burnt a bit by you know solutions that were completely closed source, didn’t allow customization and things like that. Then you know they kind of get stuck when you know something breaks. So they reach the limits and the vendor is not able to you know update that for them. So I think that was one of the big things about you know the whole Jenkins’ community, right, as you just see how many plug-ins and other you know ways people come up with of extending the tool that you just would never see from a closed-source product, right. So I think that’s been something as well as just kind of like the ability to have the transparency into how something works and be able to extend that to fit your own needs is pretty critical to DevOps success I think.
Shimel: Yeah, I, I agree. Alon, you more than I think even Tracy and Kevin I mean you built a business with a successful exit on an open-source kind of business model if you will. You know what’s your view on… Is, can, is it a chicken and egg? Can you have DevOps without open source?
Girmonsky: I think if we go back even in more than a decade there was open source and there were commercial offerings, right? There was an inherent conflict between the two.
I think that if you fast forward for today it’s about product market fit. I think open source needs to be part of any product as long as it involves developers, maybe even more than developers, but developers as Kevin kind of mentioned we talked about transparency I think developers need to have control. They cannot rely on someone else to have a black box. So anything from transparency, having control, not having dependency. You know if you think about speed to market the more control you have your ability to change quickly you know the faster you can go.
So what we’ve learned over you know the past decade the developers they need to have the open source not because it’s free. I think that’s the last aspect of open source. It was in the past it was free and a poor man’s solution. Today you want to have open source because you want to maintain control, visibility, transparency and reduce the dependency on other vendors. That’s kind of my key learnings you know from my experience.
Shimel: Yeah. I mean that reminds me of something I’ve always been taught and heard about open source and that is look with open source it’s free as in freedom and free as in beer, right? It doesn’t cost, it’s free as in beer, right, free beer for everyone. But really though that was a big driver, you’re right Alon that was a huge driver and remains a huge driver that people can just pick up this software and run with it. And if at some point they want to buy an enterprise license or an enterprise version or go to a vender that has a supported version they can pay for it, but they can use it for free today. But more importantly than that is the freedom as in freedom, right? In that it gives them the transparency, it gives them the ability to see exactly what it is they’re getting themselves into as Kevin pointed out.
But to me that’s almost turned the original problem on its face. Tracy you’ve been in these, in this open-source world longer, as long enough, right? In that originally that was also one of the big shortcomings of open source, right, is that everybody, everybody sees the source code, hackers or whoever will more easily exploit that source code or can break in or can see what’s going on. And whom am I going to call, Ghostbusters, right, for support and to fix this thing, right?
And so how can I as an enterprise you know based my business on open source? Is it reliable enough, secure enough, supportable enough? But yet we seem to have overcome that. One of the reasons I think, Tracy, is because of the advent of foundations like the CDF, right? So talk about that if you will a little bit.
Miranda: Yes, yeah, no absolutely. Like I think well first of all I want to make it clear like you can still have DevOps with a proprietary CI/CD stack and there’s plenty of offerings out there. But I think the key thing where open source plays a big role is it’s really in driving the innovation across the industry, making it so people can leverage it in different ways, combine it and get that interoperability to you know produce things that solve their problem.
And the way foundations have really helped is dealing with some of the issues or you know sometimes there have been misconceptions with open source, but sometimes they are genuine problems, which you know fall in the gaps of well maybe the developers on the projects want to focus on the code and there are other things we can do across multiple projects to help you know whether it’s focusing on security or focusing on building communities that are diverse. So open-source foundations have just added that extra layer to give the industry confidence that these are trusted projects that are being well sustained and people can rely on them to either build products or use in their own CI/CD stacks and really just kind of push, push things further I a good, positive direction for the whole industry.
Shimel: I’m going to comment or not disagree, but I wanted to distinguish and clarify something you said there Tracy and that is yes there are closed source or commercial or whatever you want to call it CI/CD solutions out there. But even they and we report on them at devopsy.kinsta.cloud right, even they rely on open-source integrations, open-source components.
I mean you look at Alon with what you’re doing with UP9, with UP9, with what you did in BlazeMeter. Kevin when he talked about the three or four different open-source kind of based tools that Tricentis is doing, they’re all based on open source. When you go under the covers they’re closed source but they’re based on open source, things like Selenium and these kinds of things.
So you know even in the CI/CD space and I’m not here to name names, but we know the closed source you know CI/CD players. I don’t think they’re product works well if you didn’t have an open source kind of framework that they plug into. I think it’s especially true in the testing world. I think Alon you’re living proof of that, right? Could you have had a BlazeMeter without having some of those great open source? Kevin all of the projects you mentioned are based on open source, right? Can you have any of them without open source? It’s a real you know …
And guys don’t wait for me to ask your name, feel free to jump in here.
Dunne: I think it’s a good point. It’s becoming more of the model like Alon was you know mentioning is it’s kind of you stand on the back of giants a lot of times, right? So I think you know you see a lot of the at least in the testing space, right, BlazeMeter, Sauce Labs, BrowswerStack, you know a lot of these bigger companies they’re succeeding by basically doing what you were talking about with the security tools 10, 20 years ago, right?
Take an open source community and kind of you know build something on top of that, right, that provides value, but still is kind of using this known currency of this open-source project. I think people like that because you know by the time they get to buying that commercial solution they’ve got a lot of confidence in the open-source components that have reached a you know critical kind of juncture where they’re bought into those tools enough that they’re willing to invest in something you know commercial to help support it and you’re solving real problems, right.
You mentioned you know free and open source being free, but you know everyone knows one of the big challenges with open source is just the amount of time that you have to invest a lot of times to customizing it, deploying it, you know patching it, upgrading it, dealing with security concerns. And so you know a lot of these vendors come in and they solve a real problem, but I think it is a little bit of a you can put the cart before the horse, right?
The people need to adopt the open-source tool, see that it works, feel the pain of you know what it’s like to manage it on their own and then you know usually graduate to these commercial solutions. I think when people come and try to present the commercial solution upfront without that open-source community you know it kind of fails because you haven’t proven the value yet, right? And it’s a big jump for people to make and that’s you know been a good thing in the market having open source. It’s kind of put the power in the hands of the users that you know users get to call the shots first and the user expects you’re going to give me some value in this open-source tool before you start to charge me an arm and a leg, right?
Miranda: I like the analogy, free as in free puppy, so we can have it, but you still have to take care of it, feed it, and keep it from pooping on the carpet.
Dunne: Exactly yeah.
Girmonsky: Yeah.
Dunne: And that takes a lot.
Shimel: Alon?
Girmonsky: I think I’ve learned a good concept a couple of years ago, kind of the open core concept. It wasn’t there ten years ago, but then you understand okay this is what, what we’ve been doing all along. The ability to open your core, the core of your platform, the core of your offerings. So you know your user will have more flexibility. Maybe the use case is a bit different than what you had in mind initially, right, by having an open core or an open-source project that you’re basing on, right, you allow the users to build their own use cases and then bring value that again the product market fit is in my view better than anything that has close and this is what you can do with it and that’s it.
Shimel: Agreed. So Tracy you know the CDF is a great model for the kinds of stuff we’re all three of us are talking about, right? You have in essence this four main properties or projects that CDF manages. Each one of them is closely tied to one or more commercial entities that solutions are based upon, right? But yet the community revolves around the CDF. By the way it’s the same thing in the CNCF, right, where you have Kubernetes and you have you know a lot of Kubernetes service providers that manage Kubernetes, but the community lives around CDF.
What are you seeing here? Like where is that handoff? Where is the you know passing the baton from that open-source user to the commercial entity?
Miranda: Yeah, no I think that’s a great question. What we see, well certainly what I’ve seen over the last 15 years is it’s an ever-changing line if you like. So when we talk about innovation you know we have these open-source projects and we don’t want people to be reinventing the wheel over and over again. Like take something like pipeline orchestration. You know when it gets to a point where that’s pretty ubiquitous then that’s a good thing to have as an open source you know kind of standardized in one project.
Then we can move up the stack into new innovations and vendors can lead that charge of take open-source projects and try to build things on top of that that add new things to the industry. As those become more and more ubiquitous then you know it probably eventually will get an open-source solution and then people will sort of move onto the next thing.
So I think it’s always kind of this tide as such where the more kind of proprietary things will test out the beating edge and where they can get value for using some things that haven’t been done and then open-source projects become like the infrastructure, the roads, and the bridges and things like that.
Shimel: Fair, guys any comment on that?
Dunne: Yeah I think that’s true. It’s kind of tough though puts vendors in a tough position, right, where it’s like you know you have to – it really raises the bar, right on delivering innovation, right, because you know all of the… It’s kind of a commoditization, right, that you know things that have been in the market for 10, 20 years you know the market, the commercial market for those sort of erodes and you know it gets replaced by open-source tools. And so it’s always kind of playing that game of being the you know next hand-up on the baseball bat.
You know it’s just like your always – if you stop innovating there’s… I think that’s why you see you know the SAS model being really popular with a lot of these software companies is just they need to keep innovating, right, and you can’t just like you know expect to sell a tool, a licensed tool and you know have a customer use that for ten years and collect maintenance from them. Like these are – all the tools that are you know demanding good budget from companies are really you know evolving quite quickly. So it’s really put a lot of pressure on vendors to innovate, otherwise they’re going to become stale and replaced by open source pretty quickly, which is a good thing for the uses of these tools I think at the end of the day.
Shimel: I also think that’s to Alon’s point that’s why OpenCore is such a great model for this, right?
Dunne: Yep.
Shimel: Because you can’t – you can keep innovating at your core and then have your freemium, premium, whatever you want to call feature set kind of revolving around that core and keeping people, “Okay now that’s why I should pay money.”
Dunne: Right.
Shimel: So –
Miranda: And I think –
Shimel: Go Tracy.
Miranda: And I think we’ll increasingly see companies kind of use that to change the dynamic in the market and then they can use open source almost as a weapon to drive commoditization by saying, “Okay well this thing that was a premium feature for a number of vendors, we’re going to open source our vision” and you know suddenly change the dynamics. So it is real interesting because, yes, I think SAS services will make a big difference and everybody in the industry has to be on their toes constantly.
Girmonsky: Absolutely. By the way a way to compliment this is think of you know we used to live in an industry where the big vendors provide point tools, value, do something that you haven’t done before. Today, with developers it’s very hard to think of having an infrastructure that is not open source, right? So let’s assume the entire infrastructure is open source whether by vendors, by developers, or by big cloud like you know Netflix, Amazon and others. Then the next layer is efficiency and productivity. Then the commercial vendors they can provide impact by, you know, this is what I have done in BlazeMeter. It wasn’t — BlazeMeter wasn’t a load-testing tool. I think Flood was the same. It’s JMeter was the testing tool.
Dunne: Yep.
Girmonsky: BlazeMeter back then was all about making JMeter more efficient and the developers more productive. So suddenly LoadRunner wasn’t like, okay there is an alternative to what was de facto commercial tool, okay it’s the open-source tool, but as you said open source on its own it’s pretty tough. So you know here come the vendors that make our lives more productive and use of open source more efficiently.
Shimel: Agreed. Kevin, Alon I’m going to put guys on a hot seat here for a second. You know what CNCF has been wildly successful in so many of the projects that they’ve incubated and now graduated and have been tremendous successes. Tracy, CDF is only what about two years old now?
Miranda: Yeah, not even.
Shimel: Yep, they’ve got four pretty amazing CD tools that are kind of de factor standards, right, that people are gravitating around. With such a strong heritage of open source and testing and continuous testing, automation, why haven’t we seen sort of the equivalent of the CDF in the testing world? Right? Why hasn’t the Linux Foundation or some other, Eclipse or Cloud Foundry or something or a new one sprung up that says, “Hey, we’re going to take these individual projects you know out of hands of individual company and do a community thing”? It may be a case of what, don’t break what’s, don’t fix what’s not broken, right, because it seems to be working the way it is. But why don’t we see a foundation for open-source testing?
Dunne: That’s a good question. I mean I think you know if I had to guess the main reason probably is just that you know testers are sort of developers, but sort of testers, right, and like an automated tester kind of falls in between the gap of maybe both and some view themselves as developers, some view themselves as testers.
I feel like it’s starting to happen. If you look at some of these tools like Cypress for example. You know the newer tools that are emerging definitely they’re communities have much more of like a developer feel around them and there’s much more structures of the open source contributions and whatnot.
But you know historically it was definitely very fragmented and you know the load testers all kind of rallied around JMeter and Gatling and you know the web testers around Selenium and the mobile testers around Appium. There’s really no common thread. I guess it’s kind of more like more siloed, right, versus my guess would be you know these tools like Kubernetes and you know Spinnaker or Jenkins, I mean those are ubiquitous, right? I mean everyone whether they’re building you know a web app you know in Java or in you know C# or whatever can use these tools versus I think a lot of the testing tools because they’re kind of like a thought you know into the decision that’s made after you make some decisions oftentimes on your kind of app infrastructure. That’s why they’re sort of thought of as maybe – they were historically thought of as more of an afterthought and the communities got more fragmented as a result of that.
But I think that is starting to change. I mean we know and we talked about these results on the last time I was on with you, Alan, you know that testers are becoming more and more like developers and becoming more technical, right? So I think the tools you know are going to demand to change this community structure, but it will take time but I think we’ll start to see that over the next few years my guess.
Shimel: Alon?
Girmonsky: Yeah I think that if you look at the industry, the open-source industry for past second decade you know prior to CNCF there as Apache, right.
Shimel: Yeah.
Girmonsky: Yeah and it was kind of you know in many ways the collection of open-source tools that if you’re in Apache open-source tool you’re probably in good. And now we see it’s not about open sources, it’s about what you do. Right? CNCF started mostly as Kubernetes and you know other tool and now the CD Foundation I think the focus is kind of all you know on the development lifecycle, as opposed to point tools and use cases it’s about the entire flow.
Actually I feel very comfortable. You know if testing finds its right position under CD because in CD you build, you test, right, these are the operations you do day-in-day-out. I would hesitate from having too many because it will kind of defocus many people, “I may do this,” “I may do that,” “Is it different?” Having like you know a family to belong to which is the continuous delivery continues deployment maybe in the future. I think it’s actually you know not a bad direction to take.
Miranda: Yes, absolutely and I think you’ve hit spot-on like CD continues deliveries is a process and I think that process does include testing and although we started off with these tools, which are all pipeline orchestration I’d love to you know CDF to future more open-source testing tools and certainly a lot of the conversations we have around interoperability testing comes up a lot. You know people are like, “Why isn’t there a standardized way to show pipeline results, the test runs” and why don’t we strive for that? That’s you know a problem that lots of end users highlight. So yeah, I’m all for you know having CDF be a place where we can have more of the testing community come together, seed, expand beyond kind of the traditional developer or and just make folks feel included in that community. So yeah, we should definitely talk. I like the direction [crosstalk].
Shimel: So you heard, you heard it here first on DevOps Unbound. Alon, Kevin you have her contact info.
Dunne: Oh yeah.
Shimel: Let’s see some testing projects being part of CDF, right? Let’s bring it together.
Alon, the next line of questioning starts with you, but Kevin and Tracy feel free to jump in. So Alon you’ve been there, done that, you did BlazeMeter as you mentioned. You build it on JMeter and some other open-source stuff, but you didn’t want to duplicate what JMeter did, you were supplementing it and building on top of it.
Your present company, UP9 is new, not a lot of people are as obviously familiar with it as BlazeMeter. But I remember interviewing you when we first started talking about UP9 and you said, “Look this is for now, right, this isn’t what BlazeMeter…you know we took a lot of lessons that we learned and a lot of things that we saw, but this is for today.” What makes it for today, especially from an open-source perspective, right, that was different than when let’s say BlazeMeter you know first started with JMeter.
Girmonsky: Yeah. So BlazeMeter dealt with a problem that was irrelevant for 2010 or 2012, it was again it was kind of moving to the cloud, the Web-scale system, but mostly around service oriented architecture, right, and WebUp monolith of the time. At this time let’s say 2020 and going forward we’re talking about cloud native and containerized system. The industry took an approach to build the system that scales horizontally, right, with microservices. When you have than a few microservices you bring in Kubernetes or Docker Swarm or ECS sometimes, right, to orchestrate.
But this entire new tech stack it is far more complex than it was n the past and still you need to test. You have you know you call it not reliability, you have to have reliability case. You know you develop software you don’t want to develop faults that become failures, right? So and again it’s far more scalable in the past. If we used to do one service, now we have 200 services, dependencies, individual roadmaps, what do you do?
So UP9 takes care of it. UP9 provides a way to first test each and all microservices, like to provide the complete test coverage, but also do it you know use machine learning to offload a lot of the work from the developers.
Now to connect this with open source again I’m a strong believer that it’s not this or that, it has to be both. You have to start with open source because you first need to onboard your users, right? The only way to onboard them for them to have confidence is by giving them open-source tools and not because it’s free or because I want to make a name for myself in the open-source community, because that’s the only way to start you know the operation and then you make their lives, the developers’ lives more efficient.
So this is UP9, UP9 start with open core solution that enables testing microservices in a Kubernetes ECS Docker kind of environment, but making it no matter how many services you may have, how many endpoints, whether it’s thousands of endpoints and hundreds of services you know you get things going pretty quickly, because you know this is how you would do things these days.
Shimel: Understood. Kevin, Tricentis I mean Alon’s example applies directly to Tricentis, right? You have three or four projects that you’ve spoken about, acquisitions that you manage, you know you’re managing these open-source communities and project, but yet there’s, you know there’s commercial or closed-source Tricentis. Where is the integration, where’s the overlap, where’s the leverage from one to the other?
Dunne: Yeah it’s a good question. I mean I think we’re kind of starting to see two different I would say maybe like communities of practice you know coming out of large IT organizations, right? You have definitely what Alon was talking about which is like kind of people building these I would call “systems of engagement,” right, which are customer-facing apps, that you know have all of these challenges around how to deal with containerization. So that’s where we really target a lot of our open-source tools, right, and open-source friend tools so that those people who really developing these web apps, mobile apps, you know web services in-house. But we’re also seeing that you know there’s this other large community as well, which is kind of all the systems of record, which should be maybe your main-frame systems, your package apps, like SAP, Salesforce, you know Workday.
And some of those communities, like some companies are starting to really start to try to do CI/CD and things like that around Salesforce and some of these more cloud-friendly apps, but for the most part a lot of those communities that kind of work differently they have different expectations, they usually are less technical and a lot of times would actually prefer sometimes a closed-source tool to kind of match the closed-source ecosystem that they’re supporting.
So that’s where I think you know when you look at it you kind of wonder, “Well hey how, why does Tricentis have both closed-source and open-source tools?” But I think you’ve got to have the right tool for the job and the right tool for the user, right? And so we see that you know you can-to-go and bring a highly technical tool to you know a nontechnical user that’s trying to test you know a very legacy package application. And vice versa you can’t bring you know a commercial tool a lot of times to a developer that wants to be you know able to write code and you know implement it into their DevOps pipeline and things like that and force them to take that approach. So rather than trying to you know have one or the other group suffer we’re sort of trying to bring together a platform which provides you know different tools for the different approaches.
Shimel: Tracy how does that jive with what you see with some of the pipeline tools and projects and companies in the CI/CD world with CDF?
Miranda: Yep, no I think it’s really interesting and it was great to hear that distinction. So when look at cloud native and particularly Kubernetes and things sitting on top of that, I think the state is today that you still have to be highly technical to stitch a system together and to make it work and to keep up with all the updates. So it’s far, far away from being used very effectively by folks who are less technical or folks who want to be a bit more removed and just get a certain job done, but still take advantage of you know all the benefits of these containerized cloud-native systems.
So I think where the open source has to evolve there is to make that a lot easier. We talk a lot about kind of the developer experience and what that looks likes, because today like it’s really confusing if you had to come in new and figure out you know what tool to start with, how to put it together, take Kubernetes itself it’s got hundreds of distributions, which one do you even start? And then there’s a lot of confusing information as well for how you make decisions.
I think there’s a lot we can do in open source to drive some clarity on you know what are you trying to achieve, how do you get there, and how do you make the tools work together effectively and not leave it as an exercise to the user to figure out what goes with what.
Shimel: Agreed, agreed. Guys were coming … You know I could sit and talk about this with the three of you for the rest of the day, but we’re coming up on 45 minutes, which is kind of our cutoff here. Tracy you’ve heard what Alon and Kevin have said. So first of all guys I’m going to hold it to you. I’d like to see you reach out to Tracy and start and I’ll help, I’ll introduce some of my other friends in the open source and testing world and you have my commitment Tracy on that. I’ll do an e-mail intro. I’d love to see some testing projects become part of CDF. I’d like to see it expand beyond the pipelines to the real tools that folks are using in these things.
But you know Tracy CDF as you mentioned is less than two years old. We just last week we had KubeCon CNCF, Virtual Con, whatever I think it was called, what’s in store for CDF? What should our listeners be looking for?
Miranda: Yes, yeah, no, we are having our first big cdCon which will be virtual. It’s a first stand-alone event we have for bringing all the communities together. Because of COVID you know we’ve had to pivot quickly to going virtual. So you know it’s not going to be the same as in person, but it is going to be a really good way to bring people together and get that interaction.
So cdCon which October 7th and 8th and registrations are open now. That will be yeah just a really good way to come in and figure out what is the state of continuous delivery today and lots of different tracks on specialist topics and cloud native and of course lots and lots of open source.
Shimel: Don’t miss it. So that cdCon, October 7th and 8th, highly, highly recommend it. I’m sorry my Apple watch was ringing. There we had to shut that off quick.
Thanks Tracy. People can get information at that at CDF?
Miranda: Yes, as cd.foundation and find details there.
Shimel: Great. Kevin, news around Tricentis open-source projects and products.
Dunne: Yep, yeah. I think for us you know like Tracy mentioned we’ve got to go virtual this fall as well, but we’re having our annual Accelerate conference this time online. We had one that was focused in kind of May on more of the US area, but we’re having more of a global one to come in October. So we’re going to be showing off a lot of exciting new stuff about a lot of the tools that I mentioned like test projects, SpecFlow, Flood. A lot of new integrations kind of between those tools and the core platforms. So starting to develop that story that I mentioned about sort of one platform to fit various different approaches, whether that’s more of the technical users or the nontechnical users. So that’s definitely one to check out. I’m just checking the date. I think it’s the 21st and 22nd of October is when we’re planning that for it I believe.
Shimel: It’s going to be a crazy October of virtual events. Alon, how about you and UP9?
Girmonsky: Yeah, you know and the UP9 is a new startup. We started almost two years ago, but we are working diligently to complete our software. We already released our first version, but you know the work, still a lot of work ahead of us. So I hope to show you a lot of new cool things down the road. Now we are heads down you know building and building.
Shimel: Excellent. Guys I want to thank all three of you for being on DevOps Unbound with us today. I wish we had more time to dive into this. You know there’s so much. Even the idea of having a CDF versus let’s say five or 10 years ago if I told you we were going to have something called a “CDF” you wouldn’t believe me, right? You would think, “What do we need this for?” And you know so we’ve come so far in DevOps and continuous delivery and Kubernetes and — well Kubernetes itself is open source, but this whole revolution honestly around open source and community it’s what makes, it fuels so much of what we are seeing. You know where would we be in COVID without a lot of what we’ve, open source has enabled us to do you know using technology? So I do think it is, it is an integral, it’s a precursor to having the DevOps world that we live in today is open-source software.
So, thanks for joining us. We have to have you back on. I’m serious about testing tools in CDF though guys. I think that’s some, long, long overdue. Tracy, thank you and best of luck now that you’ve moved fulltime into the CDF. I know you’ll make great things happen there. Kevin, Alon, best of luck to both of your companies, UP9 and Tricentis. This is Alan Shimel for MediaOps, devopsy.kinsta.cloud. Thank you all. Thank you for watching. We’ll see you on the next DevOps Unbound, soon. Bye-bye everyone.
-
EP 2: DevSecOps and AppSec
In this second episode of our DevOps Unbound streaming broadcast on TechStrong TV and devopsy.kinsta.cloud’s sister site Digital Anarchist, Mitchell Ashley of ASG and Alan Shimel are joined by Caroline Wong, CSO at Cobalt.io; Andrew Van Der Stock, executive director at OWASP; and Dr. Grigori Melnik, CPO at Tricentis, to discuss DevSecOps and application security.
The video is immediately below, followed by the transcript of our conversation. Enjoy!
Transcript
Alan Shimel: Hey, everyone. This is Alan Shimel, and you’re watching DevOps Unbound. This is episode two of DevOps Unbound, and on our show, which airs biweekly with a monthly roundtable open to the public, we explore topics in DevOps. You know, if it’s DevOps-related, it’s fair game. Today’s topic is DevSecOps, and I realize that DevSecOps has become sort of a mouthful, intrinsically linked into AppSec now and everything we do around that.
We’re gonna explore today about really, security—whose job is it, anyway? Right? Who, ultimately—there’s a lot more developers, DevOps testers than there are security people, and we increasingly rely on them to make sure our applications are secure, are being developed and released and maintained as securely as possible. And we’ve got a great panel of experts to talk about this today.
I want to quickly mention, though, this is episode two; episode one aired two weeks ago, and we will be doing our monthly roundtable, so stay tuned on that.
But let me now introduce you to our panel. Joining me first is Founder and CEO of Accelerated Strategies Group, my partner and friend, Mitch Ashley. Mitchell, welcome.
Mitchell Ashley: Thank you, Alan. Great to be on with you and with this great panel.
Shimel: Yep. The rest of the panel, I’m gonna just say their name and who they work or, and if you can give a little bit of your background. Let’s first start with Andrew van der Stock. Andrew, welcome, and if you can introduce yourself.
Andrew van der Stock: Good day. My name is Andrew van der Stock, I’m the Executive Director of OWSP. I was recently appointed to that role, and I’ve been doing application security since 1998. When we first talked about application security, it was called mobile security, because, well, it was applets and ActiveX controls that were running in browsers. So, it was a rather exciting and very different time.
Shimel: That took me back—yes, it did. Next, the one and only Caroline Wong. Caroline, you want to introduce yourself?
Caroline Wong: Hi, Alan. Thank you so much for having me on today. My name is Caroline Wong, I am the Chief Strategy Officer at Cobalt.io. We’re a company that builds security software. I’ve been in infosec now for 15 years, starting out at eBay and Zynga, some of the original DevOps companies, if you will. I also do a podcast called Humans of Infosec, and I’m a LinkedIn learning instructor for the OWASP 10. Almost 10 years ago now, I published a book called Security Metrics: A Beginner’s Guide.
Shimel: Very cool. And a fantastic speaker, too, I may add. I love watching Caroline present. And then last but not least, a holdover from our first episode of DevOps Unbound, Dr. Grigori Melnik, Chief Product Officer at Tricentis. Grigori, welcome.
Grigori Melnik: Thank you, Alan. Thanks or having me back and hello, everybody, wherever you are in the world. Yes, as Alan said, I’m the Chief Product Officer of Tricentis, the continuous testing platform and company that’s building the testing tooling for the future. And previously, I’ve been at Splunk, so this is where, actually, a lot of my kind of security mindset has been forged; also, MongoDB and Microsoft. So, good to be here with this panel
Shimel: Excellent. So, let me start things off and then we’ll be off and running. Andrew, as the Executive Director at OWASP, right, there was a time where, clearly, the OWASP member, the user, the person who was gonna, who you were existing for was the security professional who was in charge of, you know, as you called it, mobile app security and then, of course, appsec has become all the rage probably for the last at least 12 years, right? Appsec is, a lot of people say if it’s not appsec it don’t mean a thing, when it comes to security.
How has the rise of DevOps, DevSecOps as some call it, some not, how has that kinda changed who you serve, or who does—not you, personally, obviously, but OWASP—who do you guys cater to?
van der Stock: So, that’s actually really interesting. I had a conversation with Mark Curphey, the Founder of OWASP, a little while ago as I was appointed. And the original thing the OWASP folks worked on was the developer guide, which, it’s a developer guide or developers. And the reality is, is that we missed. We stopped talking to developers. We became very, very internally focused, and we’ve converted the vast majority of the infosec folks to the fact that apps are the firewall. I’ve done almost all my pen tests through firewalls and over SSL. So, the reality is, is that the application has always been the firewall. We should have been talking to developers much earlier than this.
My personal opinion is that the thing that OWASP has to do right now is to refocus on talking to developers and having tools and information they can actually use in their toolchain. If we’re constantly just having that internal echo chamber conversation, we will fail. And it’s really important to me that we actually start doing more stuff like the OWASP dependency check which can be pipelined, GitHub actions and things like that. These are where we need to start focusing, not on .pdfs, as the person who does the OWASP Top 10 and the developer guide and the ASVS, .pdfs are probably what we’re famous for, but the reality is, developers don’t know of it, they don’t use it.
I really want us to get involved in Stack Exchange. The number of people who post to Stack Exchange, just to copy and paste stuff, and that information is just wrong? It’d be really great or people like the volunteers and the chapters and the communities to get involved in things like Stack Exchange and other places where developers hang out.
Shimel: Excellent. I didn’t realize that the original focus was developers. Caroline, you’re in a funny place because you’re Chief Strategy Officer at a security tools company and of course, as you mentioned, you’re a LinkedIn—not professor, LinkedIn teacher or whatever the exact term is around appsec and OWASP. How does that, you know, what Andrew said, how does that jive with what you guys, where you guys see the world.
Wong: Yeah, so, you know, when I started my infosec career at eBay and Zynga, in both cases, these were companies that were running online operations 24 by 7, with millions of simultaneous users daily. And so, it was always clear to us that we had to get it right when it came to talking with the developer.
A couple of years before I began my career in infosec in 2003, the first version of the OWASP Top 10 was released. And right now, I wanna say we’re on version 6 or so. Andrew has been spearheading this project for a long time and we’re actually looking at the next revision coming out in February, 2021. If I look at the most recent version, which is 2017 RC2, and I compare that to the original OWASP Top 10 that came out in 2003, one of the things that’s very striking to me is that at least half of the stuff is the same.
So, that says to me that, on some level, some of these basic security fundamentals that I think appsec practitioners understand are not either being implemented by developers and where is that disconnect? So, I actually think that, as an industry, appsec tends to focus on sort of the hot and sexy things, which is, like, finding cool stuff and really interesting sort of sophisticated breaks.
But I think that, if you were to look at, for example, the past year or so, you know, if you look at the 10 largest data breaches, I’d be willing to bet you that the majority of those occurred because of something that had to do with, like, a Security 101, basic kind of thing gone wrong. And that tells me that we’ve been really focused on technology, and actually, the hard part is getting the people and the process to work out together.
So, that’s something that I think a lot about, and I think it really speaks to the core of what we’re addressing today, which is—whose job is it? You know, a security person can find all the vulnerabilities that he or she wants. Until that security person is effective at communicating what the security vulnerability is to a developer who can actually fix it and convincing that individual that it’s a priority enough to actually spend time remediating, then the code will never be more secure.
So, it’s always needed to be a partnership, and I think that, you know, these days, because of things like the transition of software development methodologies from waterfall to agile to DevOps, I think that culturally, we’re beginning to see some changes.
van der Stock: I just want to highlight one of those choices that you said.
Shimel: Oh, I’m sorry—go ahead, Andrew.
van der Stock: Realistically, the thing that I see as the OWASP Top 10 leader is, it’s self-referential. The things that people pay attention to is the things in the OWASP Top 10. And, in fact, CWE is, the MITRE Group are about to release the CWE Top 25 in a few weeks’ time, and it’ll look like the last one in 2011. And this is really sad, because people are actually concentrating on these.
What we really need is for developers and security people to work hand in hand, and I would like to see the security folks stop delivering .pdfs and start delivering, you know, pull requests. If you can fix the problem, you don’t have to convince anyone. If you actually start talking in developers’ terms and actually convince the product managers that this is the right thing to do, we’re gonna be a lot more successful than just saying, “Look, your baby is very ugly.”
Ashley: Andrew, bless your heart—bless your heart. [Laughter] That’s something I’ve advocated for. It’s not that, necessarily, security people have to become the expert software developers, but I think we’ve sort of done this to ourselves by thinking security as a penetration activity, right, something that we’re testing from the outside. I mean, now we’re talking about very porous software architectures and API security and Kubernetes container security. It’s a much more different world than even 5 or 10 years ago.
So, what I’ve advocated for—and it’s a little bit scary at times, but advocated for security people to learn software architecture, whether they learn how to code or not. But they need to understand what’s going on inside the m&m as well as the outer hard shell, if you will. And, you know, one way to do that is to bring software people into security groups. You don’t have to go learn it all yourself, you can learn from other people and bring in skills, as well.
But to your point, you know, it’s not an after the fact thing, it’s gotta be right there in the flow and the process. If it gets fixed as soon as you find it because it’s a priority—great. If it’s a day to a week, month later—forget about it, it goes into the prioritization process and it may be months or quarters before it gets fixed.
Shimel: Grigori, we’re all looking at you, man.
Ashley: [Laughter]
Melnik: Yeah, I was ready to jump in to talk to Caroline.
Shimel: Go ahead.
Melnik: Because what she said absolutely resonated with me, and then what my other esteemed colleagues just added to it, you know, rings the bell. Because, you know, I always looked at the security—and again, for a very, very long time, I was focusing predominantly on the developer audience, ________ engineer and then the development teams, platforms, tools, and all of that. And, you know, kind of designing for security something that’s been a focus of the teams that I was part of. Starting from, you know, as early as back at the beginning of the century with the Microsoft, if you remember the early days, having the disastrous year 2000, 2001 with lots of worms and breaches and —
Ashley: [Cross talk] virus —
Melnik: And then, you know, the bagman CTO, Craig Mundie had the internal initiative that later turned into the actual company-wide letter from Bill Gates, Bill G. himself. And that was the beginning of what’s known as the Trustworthy Computing initiative, which was company-wide. Again, we’re talking about tens and tens of thousands of people had been trained regularly, not just once, but on a regular basis on the fundamentals of security, on design with security. The whole life cycle, we even didn’t talk about SDLC, software development life cycle, we’re talking about secure software development life cycle and all the practices that came from it.
And it actually beared the fruit, right? Because in general, you look at the history and how much more secure, you know, Windows and Microsoft has become—that’s actually, you know, the results of those early initiatives, right? So, by the same token, I think that when we’re talking about security and whose job is it anyway, we need to start from the ground up. And maybe even before the junior developer, maybe even at school, right, if you think about software engineering schools, boot camps, whatever format you are learning your code in, it’s super important to be introduced to those secure practices, to be aware of the patterns and anti-patterns, right?
Like, I think ________ and ACM, the official ones that define the curriculum, they had included, for years, the topic of software security in there, but realistically, how many graduates of computer science degree has actually dived somewhat deeply in it, right? It’s amazing that now it’s been so many, what, two, three decades that we talk about, you know, design for security notions and trustworthy computer initiatives, and we still see what Caroline said, the Top 10 breaches—you still see something as basic as a SQL injection attack, right? Cross-site scripting—all that stuff that is almost, you know, I view it as part of developer hygiene, right? It’s just something that you do fundamentally.
It’s like, these days, it’s not something that developers, we don’t even have this conversation, should developers write their own tests or not? Well, by the same token, there should be no conversation, should developers think about threat modeling, should developers think about, you know, various mitigation strategies? Should they get involved more with the security specialists when it comes to defining or going through the release gate. And yet—and yet, when you look around, those secure practices still hasn’t made into, you know, the mainstream as much as I would’ve liked it to happen.
So, I don’t know what you guys think if there are some other ways to—it just feels like people start looking at it only when the breach occurs and the you’re facing a massive litigation, right? It’s like—but it’s too late, right? We need to think about all this ahead of time. And, of course, everything that OWASP is doing and other organizations and kind of advocating for security, promoting the secure mindset, this whole movement of DevSecOps, I absolutely welcome it. Because anything that we can do to make developers and everybody on the team to think about security early on as opposed to the afterthought is a good thing.
Ashley: I have a question or the panel. You know, these vulnerabilities have been around, as you mentioned, Cynthia and Andrew—
Shimel: Caroline, excuse me.
Ashley: – Caroline, sorry—forever. And you know, it’s not like we don’t know what they are. You know, we have really smart ID environments that will bring up the objects and what all the methods and parameters are. Isn’t there something we can do, either through AI or just in the tool side so when we’re creating software we have fewer of those basic vulnerabilities, is that—
Shimel: Spoken like a real security person! I think that sounds too magic bullet to me, but Andrew, go ahead.
van der Stock: I want to give Caroline some time to answer this, because there’s actually some things I wanna talk about education underneath from Grigori’s talk before. But fundamentally, we need to take the OWASP Top 10 to frameworks and say, “These are the things you must not have—end of story, done. You must not have these.” And then the developers don’t have to think about those other things, and then they can use the developer tools like OWASP Cornucopia, which is like the elevation in privilege game that Microsoft came up with.
Do you know what? The reason why we do this again and again and again is, we allow people to use insecure frameworks and they’re not given the guidance they need. If we had engineering students who didn’t know about, you know, wind loads and things like that, we’d never have a bridge that stood up. Yet, we allow our computer science and software engineering students to graduate without taking one single semester of security.
Shimel: Amen.
van der Stock: That’s why we established an education committee before, and we have to get people to actually do it. The ACM syllabus is okay, but it is focused on crypto, and we don’t need a lot of crypto folks, we need a lot of people who know how to engineer security properly. Now, I want to hand it over to Caroline.
Wong: Thank you, Andrew. And, you know, Mitch, I love your question, because I think that one of the things that security leaders must think a lot about is what combination of machines and humans to bring into their application security programs.
So, anyone who’s only using people is clearly missing out on efficiencies that can be found on machines or implemented, as Andrew was saying, with patterns and pre-written things that people should just be using, HSDS. And anyone that’s only using machines is, on the other side of the coin, missing out on entire classes of vulnerabilities, you know?
So, I think that, ideally, you know, security professionals are working with development teams to use scanners in order to find low hanging fruit and then using that information to provide context for analyzing the risk that’s posed by those issues. I think it’s absolutely a hand in hand sort of thing, even if one of those hands is a computer hand. And, you know, kinda the more that we can do via automation and by simply using secure libraries, using secure frameworks, you know, that allows the sort of precious and often more expensive human technical skills to be focused on things that can’t be found by machines. Finding things like business logic bypasses, race conditions, chained exploits—those are the kinds of things that people should really be focused on. You know, not things that can be solved by, again, simply, in quotes, because to get folks to actually do it turns out to be a whole, another challenge in and of itself, but I really do think it’s very much the mix of both.
Shimel: Agreed.
Ashley: I love the framework idea, too, Andrew, and thank you, Caroline—sorry to mispronounce your name earlier. [Laughter]
Wong: Just think of “Sweet Caroline” like Neil Diamond and you’ll never forget it.
Ashley: There you go. What a wonderful way to remember your name. [Laughter] You know, I wonder, too, if you layer on top of putting into the framework, security people love sort of standards like a NIST framework or something like that. If there was something that, “Okay, all of our frameworks must meet this,” right, then that’s gonna get kinda into the process and the psyche of the organization to say, “Okay, we’re only using software that meets blah, whatever that is,” it seems like that’s one more element that would help with the adoption of standardization of that. Is that practical for something like that to happen.
van der Stock: So, I’ve really been pushing for the idea of the generally accepted security principles, the equivalent to GAAP in the accounting world. There are some fundamentals that we should be trying to do.
The very, very, very, very first NIST special publication had the OWASP Top 10 in it. It was written in 1976.
Ashley: Wow.
van der Stock: It was classified.
Shimel: Really …
van der Stock: When it was—yeah. When it was unclassified and I found it, I found it as I was preparing the OWASP Top 10 2017. The only thing that we don’t have in the OWASP Top 10 is race conditions. Those guys had race conditions in the first one in 1976 when there was just mainframes and they only had one processor and they did things in batch. And quite frankly, for them to think that far forward just shows how early those folks were thinking. However, I think we need to make sure that we give the developers time to think about the desire for security.
I just want to touch back on what Grigori said before, and I think this is where strategy comes into it. When Microsoft, when Bill Gates wrote that memo, they then followed it up with action—executive action. They reset the Longhorn build. Windows 2003 server release 2 is a very different beast in terms under the head than the original 2003. And what was released as Vista was very, very different to what XP was, and that then created Windows 7.
So, quite frankly, it was—yes, we need support from the top, and we need people at the bottom to understand what this really means. But the fact that they had executive support means that things like SQL server just doesn’t have remotely exploitable bugs today. You know, that is a really good outcome, and it’s because of executive support.
Shimel: Agreed, agreed. So, you know, we live in a world right now where, quite frankly, without modern applications that we’ve been surviving on these past months with COVID and everything, imagine how ugly it would be without them. But do we really—do we really have that executive support, maybe not to the level of Bill G. and Microsoft and the Trustworthy initiative. But Caroline, at Cobalt, you’re dealing with dozens of organizations, right? Andrew, you know, at OWASP, hundreds of organizations. Grigori, Tricentis has how many customers that you guys are providing—hundreds as well.
Do you really see that top-down support at most organizations, or was that Microsoft thing sort of a catching a firefly in a jar kind of thing, lightning in a bottle, right? You’re not—we just don’t see that, unfortunately.
Wong: So, I’ll tell you what. A lot of times, when I think about security, I kind of go back to, like, the principal concept that the only reason security matters is when you have value to protect. And so, these days, naturally, a lot of value is moving from the physical to digital worlds—cyber security is more important.
I’d actually really like to challenge something that Mitch and Andrew put out with regards to sort of an all-encompassing security framework that says everyone’s gotta do this. Having worked on the ISO 27034 standard, I’ll actually ask if anyone on this panel has even heard of the ISO 27034 standard. Cool, so that’s great, Alan. [Laughter]
van der Stock: So, I actually tried to provide advice to that when it was coming out, and I couldn’t because I was part of the national organization and that’s the reason it failed.
Wong: You know, there are a lot of different things, right, and I think that the standards route is one that we’ve been trying. And I think that sometimes it works better than others. What’s really gonna work is the market route.
So, when Bill Gates wrote that memo, my understanding is that Bill did not do it for any sort of noble pursuit. He did it because he recognized that if Microsoft was going to continue building security vulnerabilities into their products, that they were gonna lose customers, that they were gonna lose market share.
And so, what’s kind of cool that I see happening when I talk to our customers like VeriFone and MuleSoft and health care organizations and HelpSpot and financial organizations is that, due to the proliferation of cloud and SaaS, today it’s not like you have a CIO who goes and plays golf and has steak dinners with their one big IT software vendor. You know, now, anyone in an organization with a credit card and an Internet connection—buy something and it becomes shadow IT.
So, the whole concept between these business relationships of trust has been transformed. And what you see is tons of software companies using tons of other software companies. So, every software company is also a buyer of other software products and services, and that business transaction requires a level of trust, where I’m seeing customers asking for things like a penetration test in order to prove that there’s some adequate level of security before a business engagement begins.
So, my perspective is actually that frameworks are cool and great, education is cool and great. And I think it’s really gotta be the market that drives the power and the adoption and the prioritization of security.
Ashley: You know, Caroline, there was also—thinking back to the Microsoft example Andrew was talking about—I think an important factor in that was Apple, because they were marketing very heavily as a secure —
Shimel: That was it, yeah. Microsoft was afraid of Apple.
Ashley: So, they have competition, you know? And I hate to use the Vista word, but you know, there was some pretty crappy software that had a lot of security vulnerabilities in Microsoft’s code, and that’s—it was the market, as you said, but it was also competition. Because that’s when Apple was doing pretty well or at least rising up in the laptop market.
Maybe there’s some group of vendors, companies that band together and say, “We’ve got better software because it’s more secure” and that now becomes more attractive. I don’t know, something like that might happen, too.
van der Stock: So, we just got involved with the—sorry, just really quickly. We just got involved with the Linux Foundation and we just established the Open Source Security Foundation, the Open SSF. And it’s early days yet, but I see that as actually being way more successful than, for example, ISO 27000. Not 27034, which has got very low adoption, but 27000 bakes in a risk-based approach which says the business needs to only do that that it thinks is right and can’t really hurt it. And unfortunately, that then becomes whack-a-mole, whereas the Open SSF works from the other way around that I think, you know, with the participating members of GitHub, Google, IBM, Red Hat, and others, and obviously OWASP, we’re gonna be able to do a lot more there than I think in a traditional standard-setting body.
Shimel: I wasn’t aware of that organization, Andrew. That sounds fantastic. We’re gonna look into it here and see what we can do to help get behind that—really, really good stuff.
Grigori, I know you wanted to talk and I’m gonna give you a chance before I jump in.
Melnik: Oh, no, I was just gonna kind of sum up—so, we talked about the importance of frameworks, we talked about the importance of market forces driving these secure initiatives, now the new organizations that have been put in place. But I am still convinced that it’s the developer’s mindset that needs to do sort of the most in order for us to be, you know, winning in this space.
So, doing whatever it takes to take them, take us, the developers, the responsibility for security. It seems like somebody else’s job—“Oh, I have this broadsec group that, you know, that looks at the metrics and the static scans and then occasionally does some kind of a penetration test and exercises or whatever and then sends me a bunch of reports, many of which I’ll try to justify as by design.” No. It’s like, as I’m doing my job, day in and day out, to be thinking about the security, to be thinking about, “Okay, what are the different ways how it can be breached?” and again, making it part of my daily routine, I think, is important.
I can tell you, internally, at Tricentis, I kind of, I’ve been in the process of launching this Trustworthy Computer Initiative, because we have opportunity to improve in this space, big time. But I can tell you, even with some easy things where, you know, we do things like hackathons, right? Not everybody participates in the hackathon, right? So, what we did at MongoDB, I actually loved it, and that actually did require a little bit of effort, but we had our Product Security person put together this type of competition, kind of catch the flag where you would actually have this increasing set of security challenges where you would have to go and break in or identify all these different holes and then, of course, reflect on that.
And actually, that raised awareness of many, many, many, many developers. So, I’m just thinking about what can we do with, you know, the small wins, small initiatives, not necessarily at the level of defining the major standards and pulling the big initiatives, but in this kind of a baby steps to move the needle, I think, is important. The other point —
Shimel: I got—oh, I’m sorry, go ahead, Grigori. I was just trying to get to my next place in line, but you go.
Melnik: Ah, okay. No, the other point I was going to make is that I also think that, for the future of the Internet becoming safer and securer and all of our systems, the other dynamics or the other initiative that I’ve seen happening and getting more and more popular, this whole collaborative testing or collaborative secure, you know, that companies like Hacker One is doing by engaging the white hat hackers in kind of—
Shimel: [Cross talk] and stuff.
Melnik: – inviting them. In fact, inviting them to go and launch the attack, but for the purpose of learning and raising awareness. And even, you know, if you look at the Department of Defense and Pentagon, inviting hackers to hack into the Pentagon systems, I mean, if you think about it, 20 years ago, that wasn’t heard of, right? But what they’re doing there is that, actually, they’re making sure that you get all these diverse perspectives, that you’re not only looking through the narrow angle of your Security Department, but this broader community at large participating; i.e., I think that there’s some hope at the end of that tunnel with the right level of engagement with those collaborative testers, white hat testers that the overall security posture of the Internet, of the world will be improved.
Shimel: I think bug bounties have certainly been a game changer. I don’t think anyone disputes that. But I’m gonna tell you something I’ve learned in my 20 plus years in security. And that is, security, as much as we may want to think of it as not a different animal than many other disciplines within IT and in business in general, and market forces—market forces rule, right? So, security becomes important when the business decides security is important, right?
I remember, you know, having a conversation—I did a podcast with the CEO of MongoDB and Couchbase and Rich Mogull, who I think most of us on the panel probably know Rich. And this is 10, 12 years ago on Network World, I was writing, and I asked those two CEOs, I said, “Does NoSQL stand for no security? Right, because we don’t see a lot of security in the”—this was a long time ago. And they both said they agreed, they said, “Alan, we will build security in, more security into our products when our customers demand it. Right now, customers are more enthralled with having this new type of non-relational database, NoSQL, you know, database that allows us to be more flexible, faster, scalable and all that, we’ll get to security, right?
A couple months later, MongoDB had a pretty decent breach. Microsoft is a great point. Apple was starting to kick their butt, and the word on the street was, even though Microsoft gave you free AV, maybe, you didn’t need AV on Apple. Remember those commercials? “I don’t need AV, I run a Mac.
And so, it’s over and over. When we make security important, you know, Grigori, you guys are the leading continuous testing provider in the world. When your customer starts saying, “Grigori, we’ve got to test for security as well, because that’s part of our quality, right, that’s the Q in QA and we need Tricentis to give us those tests and you go to companies like a Cobalt or you go to the organizations that Andrew’s talking about and stop building that in there, because your customers demand it. Not because it’s the right thing to do, not because it’s a nice thing to do, because it’s demanded, then you’ll see how quick developers take security seriously.” It’s the market.
Melnik: And it is happening. Alan, it is happening. The customers are, absolutely, they are, you know, demanding this. They want to see the evidence and they want to see, they want to have the tools and, again, everything from security testing and static analysis, but a lot smarter, you know, augmented with ML to dynamic analysis and actor-based models and, you know, penetrations and all the way to compliance, right, and a lot of other things.
And you’re absolutely right. I mean, with, back to the days of MongoDB when the initial releases were not very secure, because the idea—they were optimizing for the wrong thing. And this is what I fear was ________. They’re optimizing for adoption. It’s like—get me as many users as possible. How do you get as many users as possible? You lower the entry barrier. Now, if you lower the entry barrier, then automatically, that means that you’re sacrificing some security, right? Just like, again, allowing and opening up the ports and local hosts with MongoDB and all of it in the old days, that’s what had happened. Now, all of that has been tightened. You know, you had the security of trust, you have the security in transit, they’re doing all kinds of phenomenal stuff with encryption, fill level encryption, all of it, but it took like a decade to get to it, right?
When you are at the beginning, you’re putting something in there and you’re trying to win in that race to get more of the magic number of MAUs, right, the monthly active users, a lot of companies and a lot of startups, unfortunately, tend to sacrifice security for that speed to market and user acquisition, and then bites them later, right? So, again, if there is some way to raise the awareness and to think about it early on so that you don’t end up with the unpleasant situations later on, I think that’s something that could change the world.
Shimel: Agreed. Panel, we are, we’re getting near the end of our time, and we have such shy wallflowers here, no one wants to talk. [Laughter] So, I know how long it’s gonna take us to wrap up. I’m gonna give you each a closing, if you will. We’re practicing for the debates, here. We’re gonna give you each a chance to leave the audience with the most important thing you want them to leave. And Andrew, also—is it called the Open Source Security Foundation, OSSF?
van der Stock: Open SSF, yeah.
Shimel: Open SSF?
van der Stock: So, the Linux Foundation—yeah, the Linux Foundation announced it.
Shimel: Is there a website yet for that?
van der Stock: Go to the Linux Foundation first, and then you’ll find it. It’s one of their initiatives. I don’t know that there’s a special site for it.
Shimel: Alright, love to hear more about that. But Mitchell, do you wanna start off and we’ll—your screen’s different than mine, but on my screen, it goes Mitchell, Grigori, Andrew, and we’ll let Caroline bring it home.
Ashley: Okay, great. I’ll make mine brief. I’m actually very optimistic, and I mean this with all sincerity, that listening—people like Caroline and Grigori and Andrew, they’re some pretty brilliant people that are thinking about this, and all of you had a different but also shared some perspectives. So, I’m hopeful that we’re gonna make some progress here in the near future, so thank you for working on this problem. That’s what I have to offer.
Shimel: Thanks, Mitchell. Grigori?
Melnik: So, I wish I actually wore a different shirt today. I had a T-shirt from Splunk and, you know, they have some clever slogans there, but one of them said, “Ignorance is not a bliss,” and that’s my message to all of the developer community. You know, ignorance is not a bliss. Learn security, get the security mindset embedded, you know, from the get-go, from as early days as possible. The engineering managers think about it accordingly and embed security into your teams early on.
And then what Andrew said earlier, I absolutely sign up for it, and I love that this whole notion on the side of the security specialists, security experts—when you’re coming in, don’t send me .pdf reports of, you know, 275 pages and 722 vulnerabilities. Pull requests for the win. Pull request wins today. Pull requests, you know, really, really work. Everything else is just talk. So, again, if security specialists can get more into that written and be more embedded and more attuned to what the developers want and need, I think, jointly, again, we will definitely make a difference, so—thanks.
Shimel: Thank you, Grigori. Andrew?
van der Stock: So, many, many years ago, we wrote the OWASP Developer Guide and then I think the OWASP Top 10, and I realized telling people what not to do was exactly the wrong approach. We need to tell people how to do what they want to do better to enable secure business.
So, the application security verification standard, which I work on, is a set of tests. It’s literally written for developers. It’s literally written to be testable. And this is actually, I think, really important. We need to change the mindset from a negative, “you shouldn’t do that” to a positive, “how about doing it this way?” Because sometimes the developers have a better idea altogether and we just don’t know. But many security folks have the mindset, “I know better than you.” But they’re not software developers, they don’t know.
We need to have a conversation together, and I want to make sure that we’re bringing in developers into the conversation and not treating them as, you know, people who don’t know that much. We don’t know that much, either, and I would like for people to have that bidirectional conversation in a positive, not negative fashion.
Shimel: Fantastic. Caroline—it’s all yours.
Wong: Alright, so, the thing that I’d like to leave folks with today is, I’d invite folks to check out the 2020 State of Pen Testing Report, which I wrote and co-authored with my colleague, Vanessa Sauter. One of the topics that we dive into pretty deep in this report is sort of the relative capability of machines and humans to find different kinds of web app security vulnerabilities. In addition to drawing upon more than 2,500 penetration tests, the data of which exists in the Cobalt platform—and Andrew, I did connect with your colleague, Eric, so we’re gonna be donating some data to the next OWASP Top 10 project.
But in addition to that, we also surveyed more than 100 practitioners in security development, operations, and product across a wide range of industries. And the really good news that I’d like to leave folks with today is that the majority of respondents, in fact, 78%, reported a strong relationship between the security and engineering teams, and we do expect that to grow in the future.
Shimel: I sincerely hope so. Guys, you know, someone much smarter than me once told me that when you want to impress people and you get in a meeting, you know you got a real great group when everyone there is smarter than you. [Laughter] So, thank you, all. I really felt like we had an amazing—what amazing brain power on this one.
I want to thank you, all, for joining and contributing. We will try to put in the notes on this one some of the links to a lot of the sites that were mentioned, reports that were mentioned, et cetera. We’d love to have you back on, you know, maybe at some point in the next month or two, we’ll do a roundtable on this topic, we’ll add another person or two. Caroline, I know I had spoken to Chenxi, actually, Chenxi Wang about coming on, so I would—because I like listening to Chenxi talk about security. But anyway, and we’ll open it up to—yep, we’ll open it up to the public for questions and it’ll be a great time.
But for now, guys, thank you so much. Andrew, best of luck with the new ED position there, and I know they picked the right man for this job, so make us proud, keep making us proud. Dr. Grigori, we’ll see you soon, hopefully on another DevOps Unbound. Okay, and Mitchell, as always—thanks so much. We’ll be in touch.
This is Alan Shimel. Many thanks to Tricentis for actually sponsoring DevOps Unbound, couldn’t do it without their sponsorship and help, and we’ll see you in two weeks with another episode of DevOps Unbound. Have a great day, everyone.
-
EP 1: A Brief History of Testing
DevOps’ sister site Digital Anarchist has launched a bi-weekly video series and companion monthly roundtable on DevOps, Automation, CI/CD and testing. Featuring leaders in all of these areas, we explore the challenges and issues software delivery and IT teams face every day. How do we go faster, smarter with better quality? DevOps Unbound!
The video is immediately below, followed by the transcript of our conversation. Enjoy!
Transcript
Alan Shimel: Hey everyone. Welcome to the very first episode of DevOps Unbound. I’m your host, Alan Shimel and joining us for this initial episode of DevOps Unbound, I’m proud and happy to introduce you to James Bach and Grigori Melnik. Grigori, James, welcome to DevOps Unbound.
Grigori Melnik: Hey everyone.
Shimel: Before we jump into things, let me give each of you an opportunity to kind of introduce yourselves, your backgrounds, your positions, etcetera, so that our audience kind of knows a little bit, if they don’t already, who you both are. So, Grigori, you’ve already been a guest in our show, so if you don’t mind, I’m going to ask James to go first. James? Introduce yourself. Go ahead.
James Bach: My name is James Bach. I’m a consulting software tester. I started as a developer in my teen years after I dropped out of high school. I’m the proudest high school dropout you will ever meet. I later wrote a book about dropping out of high school and my particular self-education method, but my big opportunity came when Apple computer hired me to be a test manager, and at the time, I was the youngest manager in the R&D division of Apple computers.
So that’s kind of something I’m proud of. And I fell in love with testing and I decided to devote my career to testing. I studied the epistemology of testing, the mathematics of testing, the sociology of testing. I am a bit obsessed with testing.
Shimel: To say the least.
Bach: And I have a book about that called “Lessons Learned in Software Testing” with a couple of coauthors. And now I teach and consult.
Shimel: Excellent. And you know what, high school dropout or not, you’ve done pretty darn good, so congratulations and not saying it’s an example for our younger people in the audience to necessarily emulate. Of course, James grew up –
Bach: My son did better than me. He never went to high school, so I’m –
Shimel: Really?
Bach: He’s already outdone me.
Shimel: Absolutely. Very good. Grigori? Why don’t you tell them a little bit about yourself?
Melnik: Sure. So I’m Grigori Melnik and I am the chief product officer of Tricentis, focusing on building the testing tooling of the future, testing tools of the future. Previously, I guess I spent almost two and a half decades focusing on the developer audience and building all kinds of platforms, Microsoft and Splunk and MongoDB and you know, always been kind of customer obsessed in the sense of developer obsessed quite a bit, making sure that whatever we were focusing on would help drive the productivity and alleviate some of the pains that developers in our target audience had.And while a few of those things that I’m particularly proud of, my team at Microsoft was actually the first one that built the very first auto-scaling for what was back known as Windows Azure, now Microsoft Azure, but also focused on common patterns and practices in implementing the crosscutting components that represent crosscutting concerns that later would make into the .NET framework.
So again, a lot of work identifying those patterns, capturing them, externalizing them, testing them out and then helping drive their use. When it comes to testing, I first kind of discovered it in my days in academia when I was doing research and teaching at university, and it was quite interesting because testing was predominantly viewed through the lens of formal method groups. It wasn’t really viewed as an intellectual activity of discovery, as an intellectual activity of exploring or experiencing something.
And those were actually the days when I’ve encountered the group of testers from the context driven school of testing, including James and then we kind of kept in touch ever since, even with me gravitating away from the testing more into the developer world and now doing like the full loop of coming back to testing and I have to tell you, I felt, when the opportunity came, I felt that the world of testing, at least from the perspective of how the discipline itself has advanced would be way further developed in the last decade than I found after kind of leaving it for a while.
So I’m now on a mission of kind of reigniting that whole discipline and putting it more on a pedestal and giving it more view and empowerment and leveling up all around. So, yeah.
Shimel: Excellent. So, thank you both, gentlemen. And I’m Alan Shimel, I’m the editor in chief, CEO for MediaOps, the folks behind devopsy.kinsta.cloud, Container Journal, Security Boulevard, TechStrong TV, et cetera. And just quickly before we jump into today’s agenda, DevOps Unbound though, today’s focus is on testing. It’s not just about testing. It’s really about DevOps kind of unbound, setting us free, being able to leap tall buildings and bound into space, sort of freeing ourselves of gravity, whatever you want. It’s about speed. It’s about acceleration. Right? It’s about automation in a lot of ways.
And each – we’re going got be doing these every other week and each episode will focus on particular aspects of this unbound nature, if you will. Today’s is testing. And then once a month, we’re also going to have a roundtable with about six or eight people of Grigori and James’ stature and we’ll open it up to the public for live audience questions and discussion. So I’m really looking forward to that. DevOps Unbound is sponsored by our friends at Tricentis and we thank them for it and with that out of the way, let’s jump into things.
So, Grigori, I wanted to start off – obviously, you and James take very different paths to our – and both of you have been very successful in your lives and careers. So it just goes to show, there’s more than one way to skin the cat. Right?
But, let’s take a look – look, I’m probably older than both of you, though not that much I’m sure, but let’s look at the history, the historical context of where you guys come from to be sitting here today and how that, you know, vis-à-vis kind of the whole testing industry, the whole testing, you want to call it a profession, the guild, the whatever you want to call it, right, how has this evolved over the years and let’s put some context in with timeframes. James, when you first went to Apple without saying you’re old, what kind of – what were the timeframes for that?
Bach: Well, I am old.
Shimel: So am I.
Bach: But you’re older, which is nice. It’s nice to be a young guy in the room again.
Shimel: Uh-huh. I can’t tell you how well that makes me feel, but go ahead.
Bach: First, 1987, May 21, 1987 was the first day that I ever professionally tested something as a tester. Before that, I was doing videogame development and I had done informal testing, and I hated testing because testing just got in the way of me shipping my product. So, I didn’t like testing until it became my job and then I realized I’m perfect for testing ’cause I love complaining. And I’m much more interested in finding how things don’t work than to figure out how to make them work.
Shimel: That’s interesting. You know, my background’s in cyber, back then we called it InfoSec, and in InfoSec, most of the people got into it ’cause they liked to break things. Right? Yeah. Same thing. Grigori, how does that timeframe-wise and everything else, how does that match up with your own experience?
Melnik: So I like to actually build things. _____ you guys who are breaking things. And occasionally I break things and actually this also speaks about this overall I guess optimism of most developers who think that what they write is perfect and always underestimate the amount of buggy code that we put in there.
But to me, if I think about my first day when I actually had my job as an engineer, actually, it was September, I think 5, 1990, so a few years after James and at that time, I was actually writing a C-library for a PC converting the data from mainframe into the visual format and I had to deal with all kinds of quirks of Fortran and the 80 characters limit and the asterisks for, you know, for bringing – the continuation of the line, all kinds of exciting stuff. But back to the beginning, to make sure that I don’t go too far on the tangent, we actually didn’t have, for a very, very long time – again, I’m giving you my angle into the testing, if you will.
We actually didn’t have a dedicated thinking about testing as an activity, as a pursuit of excellence. It was just something that you kind of were doing and then at some point later, you’ve heard and learned that oh wait, look, there’s a QA department, apparently, right? And they’re the ones that were mostly viewed as gatekeepers or police of some sort. Right?
And then again, if I think about again, once I got more into the research and academia and studying testing specifically because, you know, I’m probably one of the very few people who has testing in the title of my PhD dissertation. But, I remember looking back and thinking, “But where does this come from?” Right? Where did actually software engineering come from?” Right? And of course, this is where the history of the conference deck, you know, after the _____ _____ where all the stalwarts of at that time computer science and well, not even computer science, mathematics departments because that’s who were doing all the programming back then at the point the software engineer discipline and later on the association of debugging, right?
It was the infamous story of how the moth was found in Harvard University’s scientific lab and the term bugging, you know, was coined. I almost think like the – in the early days, it was very much associated with this era of debugging. Right? Then later on, the era of the ability to demo, the fact that the software was matching the stack or the requirements, I think it almost feels like there were at least a decade when people were fixated and some would argue that it still continues.
Shimel: What decade was that?
Melnik: Hmm?
Shimel: What decade are you referring to?
Melnik: The debugging or the – the debugging I’d say –
Shimel: Are you talking about the 70s? Are you talking about the 60s? Are you talking about the 50s?
Melnik: I’m talking about 50s and 60s. I’m talking about the 50s and 60s as almost the era of debugging the way I viewed it. Then, after that, the era of more like 70s and 80s where you were focusing more on matching the spec and demo and that the software would actually approve it. And this is where this big, big, big focus was on formal methods and trying to even mathematically prove that a software worked according to the spec.
And then after that, I feel like the next era was the era of breaking or, you know, something that probably is still continuing. And again, all these elements I feel are present in today’s profession of testing, but the later years, the last 20 years was everything that I was kind of studying, watching from the sidelines, getting myself into. I like to call it the era of enlightenment, if I may. I know that’s super pompous of me, but this is where I view more testing in the way how James and other members of the context drew in school of testing had formulated it which is, you know, lighting the way for the decision-maker, not for the developer necessarily, not for – again, it could be for the developer, but somebody who at the end of the day makes the decision whether what is being produced is acceptable or not, or if it’s good enough to ship. Is it, you know, what risks associated there and informing me as a decision maker about all these different factors and then leaving that decision up to me.
So the testers are not the police. They’re not the quality gate. They’re not _____ anything. It’s really up to the stakeholder who is commissioning all this effort and who makes the ultimate decision about releasing something, putting something into production.
Bach: I’d like to offer some historical notes on this.
Melnik: Please.
Bach: Okay, first, are you aware what the first book was on the subject of software testing, the first known book? A little quiz for your Grigori?
Melnik: Is it the Myers book?
Bach: It’s before that. It’s Program Test Methods from 1972 by Bill Hetzel. Now that book was a report on a conference a testing conference which was the Chapel Hill conference, which happened in ’72. That seems to have been a historical watershed event in testing. Before that, the first chapter on software testing that I’ve been able to find was written in 1961 by my teacher, Jerry Weinberg, who wrote it when he was 25 years old. It’s a brilliant chapter on testing.
What’s fascinating to me is how differently Weinberg talks about testing in 1961 compared to how they talked about testing in 1972, although there is a reference in the 1972 book back to Weinberg, praising Weinberg’s attitude about testing. It’s fascinating to read that because the book in 1961, it’s Computer Programming Fundamentals and the chapter about testing was written by Jerry, he told, although it said it’s co-written, the book’s co-written with Herbert Leeds and Jerry Weinberg, but Jerry wrote the chapter on testing.
And Jerry talked about testing as a matter of imagination, not automation, as a matter of being suspicious and critical, as a fundamentally unalgorithmic process. And he develops this beautiful story in his chapter about a programmer who’s sure that he’s found the last bug and then finds out that he’s wrong. And then he adds another check for that.
And then he’s sure now he’s found the last bug and then he sees something else go wrong. And Jerry’s conclusion is look, this is just, we’re working on complex systems and this is always going to be like this. Now, Jerry was on the first project in 1958 that ever had a dedicated test team.
Now, the reason why he knows that is that in 1961, IBM held a conference of all the computer programmers in the entire world of which there were 400, apparently, and Jerry said his team went to that conference. He was at IBM at the time, and he said everyone thought they were crazy ’cause they had a dedicated group of testers and no one else did.
They all were just testing their own stuff. So it seemed like the idea of having a dedicated test team caught on and then in 1972, the way they talked about testing was all about test cases, automation, formality and that kicked off the era, I think, of the factory, the test factory. We can have technology and we can have procedures that make it so you don’t rely on human thinking.
So this goes against what Jerry Weinberg was preaching, not only back in ’61, but all through his career, I became a student of Jerry’s in around 1989 and took all his classes in the mid-nineties and he became a mentor to me. That’s why I’m a real acolyte of Jerry Weinberg, because he always put the human in the center of everything. But, testing – when I joined testing, all the testing textbooks were about how you have to define test cases and if possible, automate them.
And I felt like that was a fool’s idea. I think, well, the problem is is that there’s too many things to define and far from being unbound, we’ve become super bound by all of these documents and all of this code. And so if you want to be unbound, we have to find a way to temper that. We have to find a way to interact with our tools so that we’re not constrained too much by them. And that, in the nineties, the urge to unbind ourselves from that waterfall world and that heavy documentation world created agile, but it also created the context driven school of software testing, which is a humanist approach to software testing.
And that was in the mid-nineties that we started creating that. And then DevOps and agile continued to sort of take over the world except that it seems to me that agile in a lot of places has lost it’s humanist foundation and it’s turned into tools, tools, tools. And so if we’re going to really talk about Unbound, DevOps Unbound, it seems to me that we have to be specific about what is it we’re unbinding. I hope what this means is it’s people being unbound, using DevOps as a tool rather than –
Shimel: But not just a tool. I’m sorry. I didn’t mean to jump in. But not just – you know what, James, we don’t want to make the same mistake that Agile made. And you just said, one of the problems with Agile is we went from humanist to tools, right, too much of a reliance on tools. Let’s not make the same mistake in DevOps. I think that’s something I’ve advocate for for a while too is we can’t lose the human in DevOps.
Bach: Actually, I want to hear you say more about that, Alan, if you’re willing to say a little more about that.
Shimel: I’m not shy. I’ll give you my two cents. Look, I started devopsy.kinsta.cloud. I cover this place. I cover this area pretty heavily and I have for seven or eight years now. And I think there’s been this – and I don’t want to say agile failed ’cause agile is far from a failure. But there has been a reliance on tools and maybe overly processed. When we think about things like ITIL and ITSM versus DevOps, let’s say, right, we get in – we tend to point fingers at ITIL. Excuse me and ITSM _____ too much process, too heavy, too much of that, that whole change management thing.
But there are good things there. Nothing’s all bad and all good. The world’s not black and white. It’s grey. Right? And so there are things we take out of ITIL and things and ITSM that we put into DevOps. Btu DevOps, to me, anyway, and it’s just my humble opinion, is fundamentally about people and teams and culture, not the tools- there’s a lot of tools we could use and there’s overlap in tools and what’s the best tool for any given job.
We could fight about it till the cows come home, but at it’s nitty-gritty, DevOps is making sure that devs and ops and tests and security and these people work with a higher degree of communication, right, with a higher degree of cooperation getting jobs done faster. And we do that by adding things like automation to the mix. But automation for automation’s sake alone is not enough. Right? And to do automation, yes, you’ve got to use tools sometimes to make that automation work. I get it. But if we lose the fundamental aspect that DevOps is, at its core, about humans, I think we all – we lose what DevOps is and I don’t hold myself out to be as accomplished as either one of you, but I –
Bach: I still love you, Alan. I love what you just said. I think this is – if you can hold on to what the principle and the feeling behind what you just said, we can do anything with tools and we’ll be okay. We’ll be okay. If we’re talking to each other, if we’re caring about each other, I keep seeing people losing that and of course, any technology and any methodology, when it’s applied sociopathically, will become abuse. And so we just – those of us who can, shall do – should do whatever we can do keep it non-abusive and keep us talking. And then I think DevOps can be a wonderful thing.
Shimel: Absolutely. I’m sorry to take us down that. Go ahead, Grigori.
Melnik: No, it’s super encouraging to hear what you said, Alan and again, I’m a very big believer into the greatness of human mind, but it feels to me that in this kind of history of our industry, the pendulum keeps swinging back and forth, right? remember the whole infatuation with the case tools, you know, back in the eighties and nineties and then later on, there was this whole big movement of the software factories, like the term was actually software factors where the idea was that you can go and abstract away at some high level and then the loop, y, the tools will generate all the code and everything for you and make it all highly maintainable, all that, and of course that failed too.
But I think out of this desire for some reason to do – to come up with the sort of boosters. So I don’t know, some kind of the way of equipping the humans to deal with the inherently complex problems where today there’s no better mechanism of dealing with them other than human brains. So this whole vision of, you know, computers writing computer code, I mean, that’s been promised in the first era of AI, not today, but I’m talking about the sixties, right, and seventies were all –
Shimel: Yeah, yeah, I mean Spielberg does pictures.
Melnik: Then it kind of died out. Right? Same thing again, you know, having the computers entirely come up with the test strategy, the thinking about testing automatically, entirely automatically, I think is – I think it’s utopical, at least if you think about what the tools are capable of doing today and for a very, very long time in the future. I think the human will remain to be at the center of this activity, while again, all the tooling is great because tooling gives you additional powers and additional ways of perhaps automating the most rudimentary, the most, you know, boring elements of the job and lets you actually free up time to think about new theories, new hypotheses, new models, new approaches –
Bach: But you’ve got to consider different kinds of tools. Like, have you noticed the difference between let’s say a tool like a programmer’s IDE. When I use a programmer’s IDE, I don’t feel oppressed. I feel empowered. I feel like I’m riding a magic carpet and I have laser cannons mounted on it. I feel wonderful. But when I use most testing tools, I feel oppressed, like I have just a few possible things I can do and if I have any –
Shimel: Like you’re castrated.
Bach: that’s outside of what the test tool designer has thought that I might want to do for testing, then I’m not allowed to do that. This is what I feel like when I encounter something like Cucumber, which is very restricted and the Gherkin language, it’s very restricted. It’s stopping me from going on flights of imagination. And yet, when I use the tools that programmers have created for programmers, unsurprisingly, programmers have created for programmers tools that make them feel powerful and free and able to do all kinds of wonderful things. I would like to bring the same sense of freedom to testing that programmers already enjoy. That’s part of what we’re doing with the transformation work we’re doing at Tricentis.
Shimel: Yeah. I love that.
Melnik: In all fairness, not all of the programmer’s tools are that liberating. Some of them feel like you are operating in a straightjacket too, but in general, at least there are some examples, there are instances of tooling that are widely accepted, widely adopted today that are supporting and kind of accelerating your abilities to do your job without constraining you. That’s for sure.
And you know, the piece that James was alluding to, this is actually the new initiative, that we are driving inside Tricentis, you know, about mentioning this whole new platform of the test tools for the future where we are building something akin to the IDE, we’re actually not calling it IDE, we’re talking ITE, integrated testing environment where actually it’s the human being, the human tester who is in the very, very center of the ITE, and all these other snippets and utilities and services and everything around the human tester, around the human that manages to support the individual in the way, wherever his line of thought, line of exploration, line of test execution may lead to. Okay?
So yeah, we’re kind of envisioning this to be as a super configurable kind of morphing tool, almost like the chameleon style. We’re depending on the context and the environment, the different capabilities of the ITE, the integrated testing environment will represent themselves and then the human will actually pick whether this is actually something that they want to take advantage or not as opposed to again being forced into a specific kind of mold or one way of doing testing. And I don’t think – we don’t think actually anybody has attempted something of this caliber, specifically in the space of test tooling before. That’s why we’re so enthusiastic and so passionate about this project. Aren’t we, James?
Bach: Yeah. I also don’t think that anyone is going to be able to copy is easily because the only way that they could actually compete with us is by hiring someone who loves testing, which test tool companies hate doing because testers are troublemakers. I still can’t really – it’s hard for me to fathom that Grigori really knows what he’s done by hiring me because I’m a constant gadfly.
Melnik: [Laughs]
Bach: I’m constantly complaining about the state of test tools in the world and at Tricentis, frankly, and I want things to be fundamentally better, but that means we have to fundamentally change how we think about what is a tester. Which might get us into the role of testing a bit, but the – to me a tester is a puzzle solver, a problem finder. I’m not a button pusher and I’m not a verifier. I don’t verify things. Tools can verify things. I’m not looking to prove that something works because I can’t prove that anything works. What I can do is collect evidence and then draws inferences from that evidence with a critical mindset.
And of course, lots of tools are collecting facts about the products that I’m testing and those – that can be automated and so we’re pulling in these facts. And we can say, “We got this output and we expected to get this output, so apparently there’s no problem here as far as we can tell so far.” But then the tester layers onto that a critical mindset saying, “But how do we know and how can we be fooling ourselves?”
And so I’m bringing that mentality into Tricentis and I’m really surprised so far they’ve been actually quite welcoming of me, but for years and years, I’ve tried to help other tool companies with this and basically, you know what reaction I get? They basically say, “Well, our customers just want to have automation that just pushes a bunch of buttons and we don’t really care about test design. We don’t really care about empowering testers. So that’s not going to help us make any money, James, so we’re really not interested in supporting the skilled testers that you keep talking about.” So Tricentis has a different kind of vision that I think is hard to match.
Shimel: There’s also a little bit of you can’t make wine before it’s time. Right? And I think right, I think you need the right macro conditions, that things like DevOps have brought about and the whole progress -0 we’ve gone back here now to the fifties and sixties and all the way to the present time.
Certainly the world we live in today may be a – I want to think so anyway, we’re a little bit more open to the idea of saying, “Look yeah, we can automate some stuff. We can have – we don’t need a human to just push buttons. Right? We have tools that can go push the buttons for us, but we need humans to do blank. We need humans to do some of this stuff and so we need to enable and empower them to do that within the context of let’s automate what should be automated, let’s say. But we still need those humans in there.
You know, when I was growing up, much like you guys, right, the only animal in the world that used tools was humans. Isn’t that – that’s what I learned in school, right, that’s what separated us from animals, one of the things, right, was our ability to use tools. Well, then we found out chimpanzees use tools and other monkeys use tools and we still were okay with that ’cause they were primates. Now we found birds use tools, right, to crack nuts and stuff like this.
So tool use in itself does not connotate one’s intelligence or higher being. We need to – so to me, that’s the ultimate case. Right? It’s not about what tools we use. It’s about how we use our brains. And I think that’s apropos to what we see in testing today. Yes, we could come up with tools, but it’s creating an environment for people to think in and use their brains in and apply it that I think will ultimately provide us the highest payback, if you will. Right?
Bach: It’s interesting what you just said about you’re implying that the history of how we got to where we are now is important, that we have to have a certain set of – a certain historical context, technological context in order to take the next leap. And I think one thing in which that’s really true is we now have decades as an industry, decades and decades of experience trying to eliminate people from engineering and we can’t do it. I remember, maybe you do, there were these advertisements in Byte Magazine in the eighties about case tools and all your marketing people can just define your product and you don’t need developers anymore because it automatically generates all the code.
And I remember sitting there at Apple Computer reviewing case tools, advertisements that had been sent to me, thinking, well, could I buy some of these tools and actually use them to help with the testing? And it turned out, no. You couldn’t. It was just all – it was all hype and nothing. And so I shrugged. And so a lot of – and even today, now we have AI, AI is the new case tool and the AI will do all this stuff for you. Then it turns out the AI is biased and racist and confused and if you put kitten ears on a whale, it says, “That’s a kitten.”
Shimel: Yep. And it’s no more than ML, the machine learning anyway, but you know, yes, we’ve been – we have been, you know, chasing this white whale for most of my adult life and at the same time, if it was a nation, the fastest growing nation on earth are developers. There’s 40-something million is the estimate growing what, 20 percent a year or something like that. So I think it’s time we stop probably chasing that white whale and move on to empowering people.
Bach: And one way to do that, I suggest a definition of programmer or perhaps developer. And the definition I use is a developer is someone who stands at the gateway between the world of the machines and the world of humans. And the developer’s purpose is to operate competently in the world of people and then operate competently in the world of machines.
And if you don’t have that person standing at the gateway, then you don’t have a connection between the world of humans and the world of machines. You just have people kind of praying to the god of technology and hoping that good things happen. And so no matter how complicated technology gets, no matter how much AI we have, we still have the need to operate competently in the world of people and the developers will always live right at that threshold. And testers need to live there too and not get mowed under by automation.
Melnik: Yep.
Shimel: And you know what guys, we’re almost out of time, but this also brings up something that I thought we might be able to get to on this show, but we’re running out of time, and that is that this caste system, if you will, caste system of developers being a higher caste than the tester who’s higher than the, what we call today the SRE, the ops person or what have you. And this whole IT stack of humanity, I think we need to examine that too with the develop – when we’re asking the developer to do the security and the testing, is he any better than the tester? Is there a difference? But we’re going to save that one I think for another show, if you guys would like to come back on, and we can explore that a bit.
Melnik: _____ back, but I categorically reject that whole hierarchy and I also do not like the whole, you know, notion when people say, “Hey, but the tools and everything that would really help with the low skilled tester.” That’s another notion that kind of gets _____ _____ –
Bach: No, we reject that.
Melnik: So annoying and, you know, whether it’s development or testing or any other, you know, data science modeling, the UX design, it’s the discipline, it’s something that requires dedicated effort of studying, perfecting, thinking about the way how people do that work and it requires intellect. That’s the most important thing. You know, using some very powerful tools doesn’t give you the license to turn off your brain and I’m happy to come back on the show and talk about the different career letters and progressions because again, it’s going to take us a longer –
Shimel: That’s a whole other show.
Bach: There’s a lot to –
Shimel: _____ and vocal about that.
Bach: say about that. But I just want to note that this whole issue has been studied in great depth in another field, in the field of the sociology of science. For instance, in physics, theorists are in the higher caste and experimentalists are in the lower caste. So if you’re an engineer, you’re in the lower caste and they’ve studied this. How do these people cooperate? How do they assign prestige to each other? It’s a well-known phenomenon in many fields of human endeavor.
Melnik: And the same in math, right, the theoretical math versus applied math, the theoretical math seems to always think that they’re the gods and they’re the ones that are, you know, creating something that, you know, will take centuries for the world to truly appreciate and then you guys are in the applied math solving the problems of today.
Shimel: Exactly. Guys, we’re out of time, but I’m going to hold this – I’m going to hold you both to it. We’re going to come back on – not in the next show because I think we have something scheduled, but in one of the ones coming up, we’ll have you both back on, or maybe this’ll serve as part of our first roundtable in about a month from now when we’ll open it up to the audience as well. But for now, Grigori, James, thank you. Yes
Melnik: Thanks, Alan.
Shimel: James, thank you.
Gentlemen, it was a pleasure. This is Alan Shimel. You just watched the very first DevOps Unbound. We’ll be back in two weeks with a fresh panel, fresh topics, stay tuned. Thanks again to Tricentis for sponsoring and thank you for watching. Bye-bye.
-
DevOps Unbound: How DevOps has Changed Testing Forever
DevOps has changed the game in almost every aspect of testing: who, what, when, where and even why. What’s working, what needs to change immediately, and what challenges—and opportunities—lie ahead? Hosts Alan Shimel and Mitch Ashley are joined by Abel Wang from Microsoft, Tracy Miranda from CDF, Caroline Wong from Cobalt, Kurt Chase and Ben Simo from Tricentis to share a variety of perspectives on how testing fits into the ever-evolving world of DevOps. The video is below, followed by a transcript.
DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.
Alan Shimel: Hey, everyone. Thanks for joining us on this special edition of DevOps Unbound. You’re watching this, probably, at the Tricentis Virtual Event and we are happy to have you join us, happy that you’re attending the event, and we’ve got what we think is gonna be a great discussion on testing and all of its various facets and what does the future – or near future – hold. We’ve got a great lineup of speakers that I’d like to introduce you to who are on our panel today. Let me first start with – she’s on the top of my screen – is Caroline Wong. Caroline, why don’t you say, “Hello” and introduce yourself?
Caroline Wong: Hello. My name is Caroline Wong. I’m the Chief Strategy Officer at Cobalt. So, at Cobalt, what we do is pen testing as a service. We build a security software platform – like a lot of DevOps companies.
We have data-driven product-based teams. We have – we really value automation and failing fast. I’ve spent 15 years in the information security industry. I wrote a book called Security Metrics. I host a podcast called Humans of InfoSec, and I teach the OWASP Top 10 Learning Path on LinkedIn Learning, and super excited to be here with this group today. Thanks for having me.
Shimel: Thank you, Caroline. Next up, we have Kurt Chase.
Kurt Chase: Hey, everyone. I’m Kurt Chase. I’m currently Head of Release Management and Engineering Services at Tricentis. Engineering Services is a group that supports the engineering function so, we do our own type of testing – testing all the processes and everything we use – to deliver our software at Tricentis. So, prior to Tricentis, I spent some years at Splunk and at AutoDesk. It’s great to be here with all of you today.
Shimel: Fantastic.
Chase: Be here today with all of you.
Shimel: We got what you meant, Kurt. It’s okay. Next up, we have Ben Simo. Ben, why don’t you introduce yourself?
Ben Simo: Hi. I’m Ben Simo. I’m a software tester of 30 years and past president of the Association for Software Testing who has recently stepped into a product management and product research role at Tricentis trying to discover and develop our testing platform of the future and bring new things to Tricentis customers and hopefully introduce new people to Tricentis with new products that enable new ways of testing.
Shimel: Okay. Happy to have you here. Next up, Tracy Miranda. Tracy – well, I’m gonna let you introduce yourself. Go ahead.
Tracy Miranda: Hi, everybody. I’m Tracy Miranda. I’m the executive director at the Continuous Delivery Foundation. It’s an open source organization that’s part of the Linux Foundation, and also home to projects to like Jenkins and Spinnaker, and our members include companies like Netflix and Google. And I have over 20 years of experience in open source so, I love open source and I’ve worked in it as a developer – also in QA and all different facets. So, yeah. Big fan.
Shimel: Thank you, Tracy. Last, but not least, is my co-host on DevOps Unbound – my partner in crime here, Mitchell Ashley. Mitchell, why don’t you introduce yourself?
Mitch Ashley: Thanks, Alan. Yes. I’m CEO of Accelerated Strategies Group, which is an analyst firm that focuses on, in addition to cyber security, digitial transformation, DevOps, and cloud-native kinds of applications, which really matches my background very well. I’ve been both a product creator CEO/CTO as well as running IT, CIO, CSO – that kind of thing. So, currently, I’m also CTO with MediaOps – I guess CSO at MediaOps, too, so, I’m trying to collect as many three or four-letter titles as I can. So far so good.
Shimel: Alrighty. So, first of all, thank you all for coming on today. Secondly, testing – what a wild, mixed-up world we have in testing, right? Just from looking at the folks on our panel today, we have security testing – pen testing, in particular. We have the world of open source and what a factor that is in testing.
We have product testing, right? We have sort of new product testing – taking testing to new heights. And you know, that variety is sort of a double-edged sword, right? Because it’s like jack of all trades, master of none type of thing. There’s so many aspects of testing, right?
Even in our audience here at this virtual event, we have testers who come from, you know, what’s called old line QA testing, right, to DevOps enabled testing to visual testing – testers who, if it’s not an open source testing platform, they won’t use it, to proprietary systems. Through it all, though, a lot of times, all of this testing becomes sort of a red-headed stepchild or a whipping boy, if you will, right, to the whole application development system operations teams. It’s like, at the same time, their best friend and their worst enemy, right? They want everything tested, but they don’t want to give you the time, the budget, or the resources to do it, and then, when something doesn’t work, they said, “Well, we should have tested more, right?” So, how – first of all, do we agree that testing has been sort of the whipping boy of a lot of what we’ve seen go down in app dev and operations? Anybody want to jump on that?
Chase: Well, I think it’s certainly gotten a bad name. I think some of that comes from prior to the digital transformation. A lot of the engineering was waterfall, and testing was always that last stop before the software went out live, you know? Nowadays, I think everyone’s come to realize the speed and scale at which we develop things, you have to do the testing up front at the very beginning. You can’t wait till the end, you know?
You’re gonna be too slow. So, I think, from that perspective, absolutely. I think that’s why testing has really, I think, gotten a bad name, because they’re kind of the blockers to getting anything out into the public domain, whereas nowadays, you have to enable that up front because you want to get it into the public domain yesterday.
Shimel: Yeah. And so, in many ways, to me, that sounds like, well, DevOps has been the best thing that ever happened to testing, right? In many ways, DevOps has been the best thing that’s happened to security testing, right? We’ve certainly seen that shift left, if you will. But has that shift left helped – has that rising tide lifted all boats?
Miranda: I still see –
Shimel: Oh, okay, Tracy. Go ahead.
Miranda: I’m gonna jump in and say one thing that strikes me as funny. Like, we have, in our industry, everybody has embraced the DevOps terms, and then, you start to see all these spinoffs to put emphasis on things like, for security, we have DevSecOps, and you’ve got every variant of something with ops. But when it comes to testing, you know, there’s no TestOps. There’s no new buzzwords and it kind of speaks to how much it’s being neglected. Like, we had a conference on continuous delivery, and we asked for talks of everything in the life cycle, and we got lots and lots of submissions except for testing.
So, again, I think it still is really under-appreciated and just not given the due credit for what it stands for.
Shimel: Very –
Simo: Yeah. And I think some of this comes from, as Alan mentioned, that testing is a very broad and diverse subject, and I’ve often watched people argue about testing that each interpret it to their piece of what they do with testing. And things like CI and some of the DevOps practices have helped us improve with the checking that our software can work – you know, demonstrating that it can work – and there’s a lot of people – if you primarily wear a developer hat, you’d look at that’s what testing is. And we’ve automated that.
There’s been huge improvements in tooling and technology to do that, but sometimes, along with that is we throw out that we need someone involved that’s looking at things from a critical perspective that’s concerned about the safety and efficacy of what we’re delivering that is the, I guess, the one questioning and searching for information that they raise issues that we need to deal with that are reasons to not push things out to production now – so that we can make informed decisions. So, often, I think just that critical side gets thrown out as we improve the other side of testing, which is being able to demonstrate that things can work.
Shimel: Anyone else?
Ashley: You know, Alan, one of the ways you can tell who drew a diagram is to look at what’s in the center. And so, if we think about CICD, we kind of surround testing, but we don’t talk about testing. So, in some ways, it’s elevated, and some ways, it still sort of abstracts it out of DevOps. But my experience has been not really, because most teams know that testing has to get automated as part of that process, ’cause you can’t have this automated checking and automated deployment, then let’s kind of let a bunch of manual things happen. And not that testing hasn’t been automated – it has been – but I think it has elevated, at least in part, I think a good deal.
My experience has been. I haven’t worked in every environment, maybe, where test still isn’t thought of very highly. By the way, the thing that the – it isn’t the last thing in the chain. Documentation is the last thing.
Chase: That’s true. That’s true.
Ashley: Let’s give documentation –
Chase: And release notes. It’s true. I have tech doc’s team, too. You’re right. [Laughs]
Shimel: And that hasn’t really changed much either, though testing has. No. No. But you know what? Caroline, I’d like you to focus in specifically.
I mean, I think about – you know, Mitchell and I co-founded a security company, and we came out with a product in 2003 of just a vulnerability assessment and manager – “VAM”, we called it – and at that point, look, we used to beg people to scan their infrastructure at least once a year, right, if not quarterly. We were really pushing for quarterly. And you know, the idea of scanning and testing applications before you deployed them – unheard of. It was unheard of, right? We didn’t have the security tools or people.
We did that after the fact. Now, that’s not normal anymore, right? Today, at least 85 percent of applications are tested prior to deployment. I don’t know who those other 15 percent are, but that’s a whole nother story. But talk to us.
You’ve lived through this, Caroline. What do you think?
Wong: Sure. So, you know, I think there needs to be an acknowledgment. There’s sort of these like, first order priorities and these second order priorities. When I think about security – and I think some of these concepts apply to testing more broadly – there’s a saying that, “You don’t take a $200.00 fence to protect a $5.00 asset.” I think security is about you have something of value and so, you want to test it – rather, so, you want to secure it.
Testing, I think, is like you made something new and so, you want to test it. You’re not gonna test something unless you have something new. You’re not gonna secure something unless you have something of value. So, it’s okay that security and testing are not sort of the number one thing, you know? Inherently, by their definition, it’s sort of a second order attribute, if you will.
Now, I think, you know, each of us has been through a version of annual organizational budgeting. You know, each organization says, “Okay. Here are the goals for the years. We’ve got limited resources and budget to accomplish those goals. How are we gonna allocated those resources?”
And there are some things that, for a lot of organizations, are very clear. You know, even if I take a step way back and I say, “Okay. Most organizations might think about budget allocations in three buckets. Number one – general and accounting; number two – sales and marketing; number three – research and development.” You know, nowhere in those words is either security or testing.
And that’s okay, you know? I think, Alan, when you share your story about VAM, you know, Vulnerability Assessment and Management, I agree that over the last, I would say, decade-two decades – so, when I started my security career 15 years ago, we were almost trying to convince folks that breaches really happened – that this was really a thing to be worried about. You know, today, everyone knows because it’s on the media headlines every day. This is real. And so, that has been a shift in the perception, I think, in our industry.
There’s also been a big compliance driver. So, the – in the area that I’m very familiar with, security pen testing, a lot of that pen testing is actually driven by external compliance requirements – things that say, “Because of compliance framework A, B, C, you must actually conduct manual pentest on your application, for example, once a year.” You know, doing it more often than that is often because of perhaps what a lot of this group might think of as a really good thing – trying to actually secure your software, trying to actually test your software to ensure it does what you think and what you intend for it to do, but that’s not necessarily the first thing that comes to an executive’s mind when they’re thinking about budgeting, and there’s a variety of reasons for that, right? There’s an external perception thing. I think there’s also a lot of learning, certainly, that the security industry can do to be more approachable and to really understand better what is the software development methodology that we’re engaging with.
What is the specific methodology and behaviors of these individuals? And how can we actually collaborate with them most effectively?
Shimel: Fair enough. I mean, look, I don’t want to be the pessimist in the group or the bluebird of happiness here, but we have had success over the last 5-10 years, Caroline. You’re right. Especially in security, but not just security. We’ve had success in testing across the board.
Tracy, to your point, the fact that we don’t have DevTestOps, I think, is, you know, testament to the fact that we’ve recognized that testing is so critical to the CICD process that it’s part of it. I mean, you can’t have DevOps, you can’t have CICD without a level of continuous automated testing in there – in my mind, anyway. I mean, prove me wrong. But, you know, I think that has been probably one of the biggest successes that the DevOps movement has brought – has been responsible for.
Miranda: Yeah. And I really like Caroline’s analogy with security because, you know, that’s something else we think of as fundamental to DevOps, but it’s sort of elevated itself and, in a relatively short space of time, went from being ignored to being, you know, headline news. And I think there’s a lot of lessons there. And I – like, I will say, I do think testing is going through that journey and we’re probably on the cusp of it kind of really exploding and watching a few players in the industry and companies starting to do really interesting work in testing. I think that’s gonna turn around very shortly.
So, yeah, there’s an exciting space that I think its moment is coming – just like securities has.
Chase: Yeah. I think it’s all about the testing now, quite honestly. I mean, when you look at the future with low-code/no-code, it all becomes question-question mark ops, you know? Any function in a company could actually put something together and run it, and they’re gonna have to test it, you know, and it becomes less about all the engineering and more about the test patterns and things like that that are needed as you get to these future states. So – and speed and scale is gonna continue, I think.
That’s driven everything with the digital transformation. It’s speed and scale. Everything needs to come faster, and we want more of it, you know? And so, I think that really pushes the envelope, and that’s what DevOps has responded to and, I think, addressed very, very well, but it’s not over. We’re gonna continue to scale up and go even faster.
So, we have to be prepared for that, and I think it becomes more about testing in the future, and less about, you know, we’re gonna have these pre-built components – APIs you can piece together however you want – then, testing becomes, I think, almost first order, to Caroline’s point. And security, especially. I think security, we’re seeing that change now. If you’re a SAS company especially, security has to be top in line, ’cause you just can’t put something out there and not have it tested from a security perspective. And, furthermore, with compliance, it’s not just ISO and GovCloud and those programs, but large corporations now are saying, “Hey, if you want software in my ecosystem, these are my rules and I want to see your pentest reports and I want to see this.”
So, I think more and more people are realizing that if the software’s not tested, it’s not worth it. And so…
Simo: And, in my experience, part of adopting a DevOps culture and not just tool set is putting testing at the forefront. Some of the most productive teams that I’ve worked with that deliver some of the highest quality software I’ve seen – testing was center to absolutely everything that everyone on those development teams did.
Ashley: Hm-hmm.
Chase: Yep.
Shimel: Yeah.
Miranda: Yeah. We think – go ahead.
Ashley: I’m just gonna jump in and say – there’s another factor that actually works to our benefit and that is not only to speed through automation and DevOps process, but contemporary software architectures, containerized micro-services, service mesh – we’re releasing smaller bits of code in encapsulated forms, which means, you know, we’re not testing monolithic applications all the time anymore so, we’re not having to run two weeks’ worth of tests just to get one pass through.
Chase: Yeah.
Simo: Right. _____ container solved the “It works on my machine” problem because we can ship the developer’s machine.
Chase: Yep.
Shimel: Yeah. That was always a big problem, right? Tracy, you were gonna say something though.
Miranda: Yeah. No. I think the direction we see it heading is we’ll start breaking the testing monolith to talk about what is unit test, what is system test, and appreciate that is a specialization in itself. And the other things I’m starting to see from some of the end user teams we talk to is this real focus on culture – you know, a continuous quality culture – and seeing that hand in hand as part of the testing movement and again, _____ back to security – you know, the security first culture went hand in hand with that whole emphasis on security so, similarly, the continuous quality culture is starting to be something that – like, the really high performing teas we see pay a lot of attention to.
Shimel: Absolutely. You know, Kurt, you said something there about low-code/no-code and how that could be a game changer. You know, some people refer to that as the rise of the citizen developer, and I’m wondering if maybe what we need is the rise of the citizen tester, right? Is testing too hard?
Chase: Yes. I waste gonna say – the one thing we haven’t touched upon – I think Caroline did – is testing is hard. Security testing is super hard. And when you look at today’s modern environment and a lot of the ISVs – at least the ones I’ve seen and been associated with – you have a legacy of monolithic applications that you have to support that are bringing in a ton of revenue, and, at the same time, you’re transitioning – you know, going through your own digital transformation. So, that adds complexity after complexity in testing, and I do think, in the future, testing – you know, a lot of us, like Ben said, to his point, we have to push testing up – you know, shift left, move it far left as we can.
And that’s gonna continue. I think that has to be the norm. To Caroline’s point, it has to become first order. And I think it’s starting to right now. I really do. Especially the – all the SAS solutions that are out there – you know, shipping code daily, hourly. You need excellent testing to do that and do it with competence.
Shimel: Yeah. I think you do. But we do have to – and it’s funny that we’re talking about making testing easier because – not security testing, right? Security testing, pentesters – they were always sort of the hired guns of the security space, right? There were SAS or – there weren’t companies like Cobalt, right?
The pentesters that I knew growing up – growing up – you know, that I knew coming up in the security space were gun slingers. These were, you know, white hat/gray hat folks who would come in and they’d do a little social engineering and pen testing using some open source tools, and you know, they give you a report. But you know, these were folks who traveled around the world doing that kind of thing. You know, we didn’t have a company like Cobalt that did this as a service, so to speak. So, it’s very different.
But here’s the underlying thing with like, QA kind of testing – and, you know, I don’t mean any disrespect to anyone in our audience today, but for too long, a lot of folks in testing were viewed as failed developers or people who couldn’t be really good developers did testing or maybe testing was a weigh station on the way to becoming a real developer – like a Pinocchio stop or something, right, where you’re not a real boy yet. But that’s not true. There’s also – there’s a whole cadre of professional testers. And you know, in their capable hands, you could do great things. But I guess the question is – does testing have to be so hard that we need professional testing – professional testers to do testing?
Or can we make testing automated enough that you – I’m not saying it has to be the dunces who don’t become developers. That’s ridiculous. But can we broaden the base of people who do testing? And that includes security testing? I’ll throw it out. Ben, you were a big-time guy in the testing industry. Tell us – what do you think?
Simo: So, I think we can expand who’s involved in testing and that more people than those currently doing testing can learn to do testing. I don’t know _____ developers learning to become better testers and test their stuff. But I will say that ultimately, the part of the work that is the testing cannot be automated. Tasks that we do in testing can be automated. So, whether – not matter what hat someone primarily wears, if it’s a developer, you know, writing tests to demonstrate their code works or someone’s using TDD and they’re building some tests and then, building some code that matches it, the testing is the critical evaluation that’s going on in their mind of solving the problem and demonstrating that it’s solved – or evaluating how something might go wrong and coming up with some approach to demonstrate that it may go wrong and check for that.
So, those things, whether you’re doing it automated or the execution is automated or done manually, the actually testing work, I’d argue, is the cognitive work that people are doing. And there is some – that we can never automate away. There are some ways that I think AI can help us. There’s also some ways that may lead us astray. But ultimately, I think we need people to learn to be critical thinkers to understand the technology, to understand the domain of the users and the problems that are being solved, and applying that critical thinking is the testing.
Of course, then, we want to use tools to help us with some of our tests so, when we come into DevOps and CI, executing those demonstrations of the software working is a key part of that. Now, whether we do it as part of our, you know, building to those tests or we create them afterwards, then they can, at the very minimum, act as changed detectors going forward. But there was some testing in people’s minds that was going on that created those.
Shimel: Caroline, what about – okay.
Wong: I agree with quite a lot that Ben is saying. You know, I can’t help but think to myself that – I actually have this sort of visual in my mind, and it’s actually at a circus of a trapeze artists, you know? I think about these people, and they’re sort of like, whirling in the air and they’re doing these like, really incredible feats, you know, and they just do whatever they do, you know? But there’s like, a safety net to catch them if something goes wrong. It also reminds me of a metaphor that is often used in security that says, “We put brakes on a car so that you can go fast.”
You know, when Mitch said earlier, you know, “These days, we’re releasing smaller pieces of code, modules of code” and there’s a way in which, yes, each small piece of code – each module – can be tested and some of that can be automated. We should automate performance testing. We should automate unit testing. But, to Ben’s point, there’s a lot that can’t be automated. There’s a lot of testing that actually depends on things like human judgment and opinion and it actually requires a person’s decision, you know?
So, I think there’s – I think there’s a couple of frameworks for us to consider, right? If the first order is about creating value and making something new and the second order is about testing to make sure it does what you wanted it to do and protecting the value created, there’s also this spectrum that says there’s smaller bite-size stuff that may be a good candidate for automated testing, but, I think it’s super important to recognize that these small pieces of code – these modules that Mitch was talking about – it’s also super important to test what happens when they come together. So, to the extent that security folks need to be able to test chained exploits and business logic flaws, you know, those are exactly the types of things that will never be automated – is what I think, anyway.
Shimel: I don’t disagree. I think security in particular is just there’s some things you’re just not gonna automate. I don’t care what you say. But that’s okay. Anyone else?
Chase: I do want to make a comment on something you said. This was a long time ago now, when I was a dev manager, that I used to sense that when my developers – you know, if you were on my team, if you wanted to really get me mad, treat your tester badly. Treat them, you know, a bit, like you said, like they’re not quite at the level you are. I would be incensed as a manager. You know, that was a long time ago.
Fortunately, I don’t see that as much nowadays. I think that role has been elevated where I don’t feel that at all on my teams, and if I do, that is, again, to this day, I’ll get super mad about that because I think it takes all kinds to develop software today and ship software. Testing role is just as critical as an engineering role, just as critical as the ops’ role, the security role. So, yeah, that was one thing that would drive me nuts ’cause I did sense that and go through that, and this back in, you know, the late ’80s-’90s, if you will, before the year 2000 – I’m dating myself – but it definitely was palpable in some of the large ISVs, you know? And as a manager, that was something that would drive me nuts.
Shimel: Agreed. Agreed. Tracy, open source – look, testing has been inexplicably wound up with open source for as long as I’ve been around, right? Whether it’s security – open source security testing and you know, things like Metasploit and Nessus and Nmap or you know, in QA testing kind of stuff, it’s a rich history of open source testing tools. How do – and part of your mission at CICD is – at the CICD Foundation – is to nurture that, I’m going to assume, correct?
So, how do you guys nurture that? What are you – what do we do to maintain that rich, open source tradition here?
Miranda: Yeah. No. Absolutely. And, as I say, we at the Continuous Delivery Foundation, we want to look across the entire software delivery life cycle, and I think it is safe to say, you know, that testing hasn’t risen to the top. We’re actively – like the board wants it to be a more focused-on topic, the technical committee wants to encourage folks to bring us more projects in open source and with testing, but we’re still having to kind of fight through all the noise around other projects.
And one of the ways we’re approaching it is, for example, our technical oversight committee wants to set out a landscape of the types of projects in each kind of bucket and say, “Okay, look. We need to have something in security. We need to have something in testing that covers these different aspects.” Almost like a call to action to say, “Bring us your testing projects and –” like my favorite saying about testing is, “There’s two key problems in testing – not enough tests and too many tests.” And yeah, I think we’re starting to see some really good frameworks around test generation.
I’d love to see more open source options for that. And Kohsuke Kawaguchi, the famous creator of Jenkins, move on to focus on testing so, that says a lot about the importance of this area getting more open source focused. So, yeah, I’m excited to see what could come out, but we do have to actively encourage folks to come join us and have those conversations and not to, you know, have it assumed it’s all focused on delivery of pipelines for delivering software.
Chase: I think another thing you’re seeing today is a lot of people, when they start developing an app, when they start creating something new, that’s the question they ask up front. “How are we gonna test this? How are we gonna automate it? How are we gonna pentest it?” I think those things, at least, are starting to bubble up a lot stronger in the cycle – you know, early on. ‘Cause I know that’s something we recommend highly – is that if you’re developing something new or need to head down a digital transformation, think about the testing. How are you gonna test that up front, you know? And products are actually being developed with that mindset now.
Simo: And with short iterations, it’s how are we going to test this tomorrow, next week? It can’t be left off for months later to be someone else’s problem. So, people, I think, are learning to deal with those questions about testing from the start of their discussions about what the requirements are.
Chase: The other thing I love – and in Caroline’s world, has been very successful – I know it is specialized skill set, but trying to teach that throughout your troops – you know, throughout the engineering team. Show them what, you know, across _____ scripting errors. What does it look like in code? That’s super successful. That really, really helps make – ’cause security is difficult, you know, no matter how you slice it.
Even when you’re looking at product development – if you want to invest in fixing security defects or add a cool new thing for your users, you know, a lot of times, the cool new thing is gonna win out. But that’s starting to change as well, too. I think people are starting to value security – especially in today’s modern world, you know? It has to be up front, really.
Ashley: Yeah. It’s all about AppSec at the moment. That’s one of the hot topics as well. You know, someone mentioned AI earlier. I just want to offer a perspective about that because we always seem to focus on the shiny new object.
I know I have. I often do. But I think what we’ll see with AI is select applications when it comes to testing. The sort of vertical place is where it’s really well suited. When we say, “Yeah, AI”, we really mean machine learning in most cases, which is machine learning algorithms, and there’s two types – there’s supervised/unsupervised.
Supervised is, “That’s a car. That’s a car. That’s not a car, right? That’s a bug. That’s a bug.”
And unsupervised takes massive amounts of data and intellectual patterns and learnings from that. So, I think that as we’re taking our test automation and our test tools, there may be some of that, and it might be really useful for AI. I wouldn’t get worried about people being replaced as testers. Not in my lifetime, that’s for sure. It is still a – there’s very much a craft to it, just like there is software design.
There is a craft to testing and a skill and a mentality about it that’s unique, just like with security.
Shimel: No doubt. Caroline – oh, go ahead, Ben. I’m sorry. No, no, Ben. Go.
Simo: Yeah. I was gonna say – and with any other sorts of automation – even you know, not learning automation that’s come before AI and machine learning based things, is that it not only has potential, in some cases, to replace some of the mundane work we do, but to enable to the skilled people to focus their work and apply their brain power to the things that really matter. So, I think there’s great potential for AI to help direct people’s attention to the things that maybe should be of concern and help deal with that big question of testing out of infinite possibilities of PaaS and data variations. Which one’s gonna show me the problem? And I think there’s huge potential for AI to help us find that if we’re also aware it may guide our eyes away from something that we do want to see.
Shimel: Caroline, how does that relate to security, though? I mean, is AI helping Cobalt with pen testing? Be honest.
Caroline Wong: So, the type of software that we’re building today is actually workflow and collaboration software. So, what we’re trying to do, is we’re trying to bring security folks and developers and penetration testers all together sort of in one place where they can collaborate. The technology piece, for us, is not about automating defect discovery. It’s not – I mean, if we’re gonna us ML or AI in the future, honestly, what we’re gonna do is we’re gonna use it to say, “What are the characteristics of this organization that wants us to do a test on this particular type of asset? What are the attributes of that asset and how might we match that to our talent pool – something that we do totally manually today?”
I think that’s the opportunity. I don’t think that we’re gonna get to a point where we’re using the AI and the ML for automating the defect discovery itself. Frankly, I think that even if I take a step away from pen testing and I look at the security scans that are in existence today, I think, you know, whether or not they’re referred to as using AI or ML techniques, they’re just simply imperfect, you know? And there’s a super important place for them, but there’s a lot of false positives. There’s a lot of interpretation of data.
And I think that’s always gonna be the case. I think that no matter what, when it comes to humans and machines, the machines can add a lot of value, but I don’t know that when it comes to software development testing and security in particular that there’s every gonna be a complete substitution. I don’t think that – I don’t think that makes sense.
Shimel: Absolutely. So, guys, we’re coming up on the end of our time. I need to wrap up. Tracy, let me say this for you, if you don’t mind – and if I’m wrong, correct me. For you testers out there, the CICD Foundation – or CD Foundation, actually; I keep – we always say “CI” like it’s attached at the hip – but the CD Foundation is looking to hear from the testing community.
Probably some companies – including Tricentis – may want to look into becoming involved with CD Foundation because if you don’t have a seat at the table, you don’t have a right to complain about not being heard. And so, you know, when we look at foundations like Linux Foundation and the CD Foundation, CNCF, we need the testing community – vendors, professionals, folks in it, you know – to be involved and get involved. Be heard and be part of it. So, I’m gonna extend that out there to everyone. But, short of that, though, Caroline, Tracy, Kurt, Ben, and, of course, Mitch, thank you so much for joining us on this special edition of DevOps Unbound.
We hope everyone enjoys the rest of their Tricentis Virtual Summit. This is Alan Shimel for MediaOps, devopsy.kinsta.cloud, Security Boulevard, Container Journal, Tech Strong TV. Have a great day, everyone, and thanks for joining us.
[Musical outro]
[End of Audio]