Author: Alan Shimel

  • EP 12: DevOps Unicorns

    EP 12: DevOps Unicorns

    What should you look for when choosing a DevOps vendor? What criteria are others in your industry using to inform their decisions? How do you choose a vendor that is a good fit for your organization? And how do you know if you made the right choice? In this episode of DevOps Unbound, host Alan Shimel welcomes Larry Maccherone of Comcast, John Willis of Red Hat, Ravyn Manuel of the Smithsonian, Garima Bajpai of Capital Carbon Consulting and Mitch Ashley of Accelerated Strategies Group for a panel on DevOps unicorns. The video is below, followed by a transcript of the conversation.

    Alan Shimel: Hello, everyone. Welcome to another edition of DevOps Unbound, where we talk about DevOps, hence the title. Thanks for joining us. This episode is titled DevOps Unicorns: The Quest for End-to-End DevOps.

    You know, I’ve been in IT 30-plus years and there’s always been this tension or this pendulum that swings between the one throat to choke versus the best of breed solutions in any area. Today it’s DevOps and software delivery and deployment. But in anything, in the security world, in everything, in hosting, in infrastructure it’s always been a common theme. But it’s really, really becoming acute within the DevOps sector now. And we’ve assembled what we think is a great panel to discuss this issue. Let me introduce you to them and we will go from there.

    First of all I want to welcome, well, to a panel for the first time in a long time, my friend Larry Maccherone. Larry, welcome to DevOps Unbound. Give people a little background, if you don’t mind.

    Larry Maccherone: Sure. Thanks for having me here, Alan. First of all I want to say so I launched the Dev PsychOps Transformation Program at Comcast and grew that for five years. And I’m a developer at heart, though, so I was sort of the ideal person to take what was actually becoming more common outside of big enterprises, Dev PsychOps, and try to actually do a diverse approach inside of a large enterprise like Comcast. And so all of my experience is around that recently, the last five years or so, around that.

    Shimel: Excellent. Next let me introduce you, she’s a repeat visitor to DevOps Unbound. Garima, if I mess up your last name I apologize in advance, but Garima Bajpai. Close?

    Garima Bajpai: Almost. Yeah. So I’m Garima Bajpai. I’m based in _____, Canada and I’m the founder for the DevOps Community of Practice here in Canada, which has three chapters: Toronto, Ottawa, and Edmonton. I’m also the organizer of the DevOps Summit in Canada, which happens in November 8th and 9th. And I’m also associated with organizations like DevOps Institute as an ambassador, as well as CD Foundation as an ambassador.

    Shimel: Fantastic. And, Garima, I apologize on your last name. My bad. I’m trying to read it and the monitor’s too far away from me without my glasses. But thank you for correcting me, and welcome.

    Next up, another good friend of mine, my actually business partner on Digital Anarchist. And though that’s not his full-time gig these days; he’s very busy with his red hat career. The one and only Botchagalupe, John Willis. John, welcome.

    John Willis: Thank you, sir. Thank you, Alan. Hey, yeah, John Willis, Botchagalupe on Twitter. My current day job is in a team called Global Transformation Office at Red Hat. I’m sort of blessed to be on a team with Andrew Clay Shafer; Kevin Behr, the co-author of The Phoenix Project; and Jabe Bloom. Just quickly, you know, I’ve been in the industry for 40-plus years, 10 or 11 startups, 12 books, if somebody’s counting. Probably the most notable is the co-author of DevOps Handbook.

    Also was involved in basically the starting and building of the original DevOps Days. Was the only American at first DevOps Days and Damon Edwards and I created the first DevOps Days event in the USA.

    Shimel: Yep. And that’s only a partial list, John. Thanks for joining us today, man.

    And then I want to introduce – she’s off-camera right now, but hopefully when you’re watching this at least you’ll get to see her picture, but she was having some video difficulties, but happy to have her on audio, my friend, Ravyn Manuel. Ravyn, welcome. And if you wouldn’t mind, give people a little background.

    Ravyn Manuel: Thank you, Alan. I do appreciate being on your show. It is awesome. It makes me laugh. I am the senior application developer for the National Museum of African-American History and Culture. And I’m also the DevOps engineer for that particular unit. As a developer I implement some of the hands-on interactive that you play with in the museum, and as a DevOps engineer I am in charge of just about everything that’s DevOps, from the cloud solutions and how we – from the application development side, how we’re using cloud and the tools and writing strategies and keeping my leadership informed of all the good things that are out there.

    Shimel: Fantastic. Ravyn, welcome and thanks for joining us today.

    Last but not least, my co-host here on DevOps Unbound, Mitch Ashley. Hey, Mitchell, welcome. You want to just introduce yourself?

    Mitchell Ashley: Great. Great to be here. Man, what a panel. I am so excited to be with these folks on today. Yeah, I head up Accelerated Strategies Group, analysts from focusing on, of course, DevOps and digital transformation, full cybersecurity and cloud. And I also serve as CTO with MediaOps, so heading up the platform strategy and technology, so we used to deliver programs like this.

    Like John, I think we’ve been probably in the industry about the same amount of time and have zero books to my credits, so he’s made a lot more progress in those areas, but also a startup person and run a lot of software projects, product, as well as IT groups, as well as on the vendor side. So this will be a fun discussion.

    Shimel: Absolutely. I should mention DevOps Unbound is sponsored by our friends at Tricentis. To give you an idea of what a great sponsor they are, they don’t even have anyone on our panel today. But they do enable this show to go forward every other week, so many, many thanks to Tricentis.

    So, John, I laid it out, right, it’s the one throat to choke versus best of breed, how many vendors can we support, how do I package solutions that are more than just point solutions, or maybe I just want point solutions. What’s the answer here, John?

    Willis: The answer is, Alan, I would say if I had the right answer I’d be on a Grady-White right now in the Gulf of Mexico.

    Shimel: Exactly. And I’d be right with you.

    Willis: But no, I think this is a great panel. I agree with Mitch; I’m excited about just the discussions we can all have. I think the question – you know, I was thinking just that sort of Chinese curse translation, “May you live in interesting times,” right? And that’s clearly where we are right now. And I think your question starts with a shift, right? Like there was this – talking about you, Mitchell, some of us have been here forever, and like back in the day there was this question of one throat to choke versus should I do an ELA with computer associates versus piecemeal things, right? And probably less than one-percent of the enterprises were do-it-yourselfs.

    Now today you will find probably – I’m just going to swag the number, but it’s probably north of 10, maybe 15-percent, maybe even higher of companies that are now being challenged to do it yourself. You know, Kubernetes is a great example. That’s a can opener for companies to make a decision of are my vendors keeping up with what comes from basically sort of upstream technology that I should build myself.

    So I think that’s the sort of question we move into, is more about do you build internally to solve your problems, to keep up with the changes in the market and the technology, or do you try to find a pair of vendors or a single vendor – and there really is very few single vendors that can give you the whole scope of what’s going on in modern technology today. But I think that’s the question that I think we’ll have fun just talking about today. _____.

    Shimel: I don’t disagree. Panel?

    Manuel: So, Alan, this is Ravyn. I agree with you, John. I think one of the things – because that’s a struggle for me and my organization, is we need help with implementing DevOps, but I think the vendors are very tool-focused and the tools are like the third priority in the stack of there. And what I’ve been doing when I’m building the strategy for how we’re going to implement DevOps, I looked, and we’re talking about tools. I look at the standards, because if they’re making standards and the tool has to follow that standard then it’s plug-and-play, you don’t get – like if Docker decides that they’re going to go out of business then you can have another container-type technology to do that.

    So when we are looking at benders we actually kind of do a piecemeal, because we’re really unicorns. We’re not a software company, we don’t have software products; we have digital products, and they’re all one-offs. And so it really just depends on what fits for us. So we don’t really go for the one vendor solution; we actually go for who can help us get to where we need based on our business problem.

    Bajpai: Excellent. And I wanted to add a few things, because when we drill deeper into this conversation we will find out that there are two sides of this coin. One part is where an organization from a maturity perspective is. And the second thing is what are the vendors and key capabilities you’re looking at. But if you look at both sides of the coin there are three things which are needed to calibrate these DevOps tools in the overall context. The first thing, and this is – I want to reiterate the fact, because I come from a very large complex organization background, that it is very essential to have an enterprise architecture view. So I would suggest and recommend that before even, you know, taking this approach of single vendor or multivendor discussion, we need to kind of break down what are the key capabilities which are needed. So whether it’s a single vendor which can provide you that enterprise IT architecture capability, or it will be your organization who will be doing it. So that’s essential.

    The second thing I think which is also essential is orchestration layer. You know, you will have to have that governance of total cost of ownership, compliance, security, identity access management, and that orchestration layer has to be there to stitch all you calibrated DevOps tools. So it’s a question of whether you want to kind of build it in house in that organization or you kind of have a vendor-centric approach where this capability should be or could be brought in.

    So these are like two things, and then you start calibrating these DevOps tools. So you don’t have to look at the foundation aspects of compliance, security, governance, total cost of ownership, and the job becomes much more easier. The question is that, you know, how to bring these DevOps tools and enable business value.

    Maccherone: So I want to riff off of that orchestration layer idea, but I want to backtrack a little bit on sort of the – we have a binary sort of difference here between unicorn and best of breed. But I think within the unicorn bucket there’s really two major flavors of unicorn that are wildly different, and so I want to sort of bring that out. You have these Frankenstein vendors that are piecing together acquisitions into this monster and those are not pleasant experiences. The only benefit you get out of one vendor in that case is really the contracting benefit of having a single master service level agreement. You’re not getting an integrated product that really allows you to actually smoothly do the workflow, insist in the workflow that developers are using to produce value for your company.

    And then you have this other class of unicorns in the DevOps world, but less so in the security world, and I’ll talk about that in a second, which have taken a marketplace approach. And this is not a DevOps vendor, but I think the first vendor to really pull off this marketplace approach was Atlassian. You know, they had one product that was really great, but they said, “Okay, plug into this product” and they created APIs and they created toolkits and they created a whole community. Most of those people did it for free and they just sort of did it for themselves and then they shared it with the community. But some of those people became vendors and sold their product in the Atlassian marketplace, and then Atlassian acquired those. They were already pre-integrated with Atlassian’s other offerings when they got acquired, and so their acquisition unicorn approach doesn’t look like a Frankenstein’s Monster in the end, because they were built with the APIs of the vendor in place.

    And I think Github is definitely taking that marketplace approach and pulling it off really well, even into the security space. And so I think that kind of unicorn – and Github to a lesser degree, that kind of unicorn doesn’t suffer from that Frankenstein Monster’s sort of problem. No one in the security space has really pulled off the marketplace approach yet, though. We have some Frankensteins and Frankenstein Monsters over there, but we don’t have a full sort of marketplace approach. And I’m waiting for somebody to do that. I think that’s going to be the big win, somebody _____.

    Willis: I’d like to do a whole podcast on that, because that’s all I’ve been thinking about right now, is what is the end-to-end on security. But like if I got into it now it would take up way too much time. So I did want to comment on some of the things. You know, the first, Ravyn, I think the interestingly about standards, I agree, right? The problem we have right now is figuring out which standards are actually going to last, are malleable. I mean that’s one of the things I’ve really been working hard on and looking at like OSCAL versus OVAL versus S-CAT versus, you know, just NIST _____ – in order for – I mean there’s just so many things and they all sort of have a little bit of different approach.

    So I agree, the question we have to deal with now in these sort of, you know, may we live in interesting times is it’s like picking sort of an early vendor, like I like this vendor technology, but are they going to be around? Are they going to be acquired by a company that I don’t really want to do business with? I think the standards is just it’s such complicated.

    Just the other thing I wanted to say is I think that for vendors, you know, and I’m not _____ ’cause I work for Red Hat. Because I will say right now the thing that like in the grand scheme of technology, if you’re going to go all in a vendor, look around. You know, I’m going to say this. I’d love to be corrected and I’m not saying, you know, Red Hat because I work at Red Hat. Anybody that knows me, I’m not a shill for any vendor I’ve ever worked for. But it’s VMware and Red Hat, right? They can keep up with – because you’ve got to – in this world you have to have people on staff, their full-time job are contributors to an open source project.

    And if you think about what’s going on with like Kubernetes and clusters and server list, you know, you think of the two companies that are positioned really well at that sort of scale version to be able to keep up with the complexity and change of technology because they’ve put the resources in. I mean a longer list that you can probably make after that. And I’m not including the cloud providers either, ’cause that’s a different story.

    But anyway, those are the two comments I wanted to make on some of the things people talked about.

    Ashley: I just wanted to jump in. What’s fascinating about this conversation is how diverse and complex it is. There are a lot of aspects to consider, whether it’s an enterprise architecture like Garima is talking about, John’s talking about, who are the vendors that can pull it off. Larry is too, if you’re talking about this sort of roll-up strategy, but retro-ally integrated strategy.

    I think another dimension to consider is every organization has a DNA about how they think and operate when it comes to vendors and software and tools. And while I’m not always an advocate of go with the flow, sometimes when you’re thinking about how do we have something adopted, think about how the organization operates. If you are kind of a large tool vendor, let’s work with, you know, Red Hat or VMware or Broadcom or whoever might be – think about that. That’s oftentimes, you know, save yourself some brain cells and live a little longer.

    There’s also the innovation curve, and some of the best ways to ride the innovation curve is the open source route and the vendors that are built around that kind of a strategy. There’s combinations of all those.

    So I guess the thing I would add to the consideration of the conversation here is think about how you match up your strategy with the DNA of the organization so you can accomplish as much without killing as few brain cells as possible.

    Bajpai: And thank you, Mitch, for bringing up this point of DNA of an organization. I think that this is a pathway of dialogue which we are having. I do believe that the mindset is shifting, and even I see in large legacy enterprises that they have started to realize that, you know, this vendor-based approach is something that needs to shift, and they need to bring people as partners in this context. Because when you think about DevOps it’s basically exploring your opportunities, breaking the barrier for, you know, your technology onboarding, the gaps – the alignment and outcome gaps you have created in the past.

    I think this is the opportunity where the organizations can shift and bring more access to innovation through these partners. So I truly believe that there are softer aspects of this conversation as well which we need to kind of consider that traditionally how contracts are _____, you know, how we look at these DevOps tools from a vendor perspective, how can we bring them as partners, what we need to change if we start doing that.

    It’s not kind of done overnight, right? So we have to kind of set what kind of modeling practices we need to bring in. And this opens up another door of possibilities. And probably during this conversation we will have some opportunity to discuss these SLA constraints which we have built in the organizations which steer us to a specific vendor or steer us to like a specific window through which we can outsource a lot of capabilities and just, you know, a choke the throat approach, right?

    So let’s maybe in the upcoming discussions we also can bring these points and highlight.

    Shimel: Ravyn, I know Smithsonian is sort of quasi-government, or it’s not true government. But to Garima’s point about SLA and being able to only deal with chosen vendors like that, it is; it is a real governor on your ability to choose best of breed, on your ability to have choice in how you want to do things. Interested in your experiences around that.

    Manuel: It’s been difficult. I’ve been listening to this conversation and it just hit – I don’t know why it just hit me, but the vendors – and I love Red Hat, so I’m not trying to talk Red Hat or anything, and open-ship. But when you get a vendor that you have worked with and you trust, and Smithsonian, we use Red Hat a lot in our OCIO. Red Hat is doing open shift and Kubernetes and they are driving their tools to these specific tools. And for me, as a unit within the Smithsonian, I may not want to use open shift and I may not want to use Kubernetes, but I don’t actually have a choice, right? Because that vendor is a great vendor and they’ve got great tools, but they have chosen for me what I am going to be using, and that makes it really hard for me to say “Does that really fit what I really need? Is that too much?”

    Because I can tell you that a lot of the vendors that I talk to, they’re expecting us to use this like suite of products, and it’s too much for what we want to do, and so I’m left with I either go with this vendor and overspend and use tools that I don’t really need, or I have to do it myself because I can’t really find somebody that can customize that for me. So it’s a challenge.

    Shimel: Yeah, it sounds like it.

    Willis: You know, one of the things I struggle with, you know, I sort of love committees and hate committees, right? Because like I so – HashiCorp has this cluster manager called Nomad, and I’ve actually seen people use it effectively with like incredibly less toil, 80-percent of the features of it, but it’s never going to get any escape velocity. It hasn’t and it won’t.

    And I fear for our industry that we’re like so focused on there’s just only one way to do cluster management today where like – so to Ravyn’s point, like whether it’s Red Hat or whatever, it seems like you’re not even able to explore possible emerging containerized cluster management solutions because all of the oxygen in our industry right now is consumed by Kubernetes.

    Manuel: Amen.

    Shimel: Thoughts on that? I’d like to jump in with some thoughts on that. I mean this is sort of a dream/nightmare scenario. Right? I’ll take it – let’s take it off Kubernetes, let’s go back to PC operating systems. It was great that we had only Windows to develop on, and this way if you were an inept developer you only had to worry about developing on Windows. You had one operating system, you could make your app on it and it worked and you got 90-percent market share with it. What a fantastic thing for app developers. Of course, it gave Microsoft the inside track on Office and everything they did. And from a security point of view what a big, fat juicy payload that was, right, to go attack.

    So then we started seeing, for as long as I’m in IT, “This is going to be the year of the Linux desktop.” Right? How long have we heard that? And so, you know, and then of course Mac resurfaced.

    But this is we strive for some uniformity. We strive for Kubernetes to be the standard, ’cause it’s open source and it’s run by a foundation, and we want a standard. But then we chafe that, “Hey, we all can’t fit into that same standard. We want to be our own unicorn.” And that’s really what we’re talking about here today. If you go back to early, John, you sold the company to Docker before Kubernetes dominance, right? And there were several cluster management solutions vying. Right? There was Mesosphere and Rancher out there _____.

    Willis: Mesosphere and _____, yeah. [Laughs]

    Shimel: You remember. Yeah, _____ of these, exactly. So, you know, this is – but this is the world we – you know, to quote Hyman Roth in the Godfather II, “This is the world we chose to live in,” right? We strive for these single kind of standards that one-size-fits-all. And maybe that’s a huge mistake; I don’t know. Maybe it is a mistake. Larry, what do you think?

    Maccherone: Yeah, I do see that problem, John, you’re sort of focused on here, that sometimes the de facto standard is not the best technical choice. You can go all the way back to Betamax versus VHS, for those of you who are old enough to remember that, right? VHS won out, but Betamax was considered by everyone to be a superior technology. Why?

    You know, so I think if you’re going to take a best-of-breed approach you’re going to have to – when you have a true de facto standard like Kubernetes you don’t have to isolate yourself from it, but when you’re making a bet on which one is going to win out – and John is really good at this; John talks about this all the time. John sort of like doesn’t want to put his chips down on something that’s not going to survive long-term, so he wants to think about what’s going to survive.

    So in order to hedge your bet there I think you need this orchestration layer that was talked about earlier. You need to sort of isolate the tool vendor from the way it’s consumed to a certain degree, so you can plug-and-play a little bit as time goes on. And this is what we did at Comcast, we built a layer between all of the tool vendors for security tools, because they don’t have a good unicorn solution there. There is no good unicorn solution there anyway, even if I wanted to. And so we had more than one vendor in every tool category competing internally, and sometimes we had three tool vendors. But from the developer’s perspective it was really easy for them to switch out from one to the other, and they got the same feedback channels, they got the same visualization, it aggregated and scored the same way across all these different vendors.

    I think that orchestration layer is really key when you have a best-of-breed approach, and I think you have to build it yourself. I don’t think you can buy it today. I really don’t think you can do that.

    Willis: I’ll try to shut up quickly. I just want to make a comment. I think there’s two books that help us understand our world. One is Guns, Germs, and Steel by Jared Diamond, which actually shows you why we get into these predicaments. If you haven’t read it, it’s a fabulous book. And then Start With Why by Simon Sinek is the sort of anti-pattern to that, which I would say like there’s clusters – atomic computed things like sort of serverless or functions or containers will be around for 10, 15 years. Clusters of those things will be around for 10 years. So if you think in those terms instead of thinking about Kubernetes and Docker – and Docker is already slowly dying, then you get that sort of start with why isolation against these sort of Guns, Germs, and Steel trends.

    Bajpai: Right. Then I also wanted to kind of bring another aspect of this, which is connected, but I think companies might start looking at it from a scaling perspective. We talk about a lot of tools and how to calibrate these tools in our DevOps pipeline. Of course when you have a large organization you have multiple product lines, and these product lines will choose to be flexible, so they need to have that flexibility to deploy their DevOps pipeline in the way they choose. And that’s the whole power of DevOps, right?

    But I also feel that the conversation in this discussion is leading us to a point where we start looking at our technical oversight committee from a governance perspective, where we can stitch the dots, we can actually lay that common foundation which is required in order for people not to kind of rewind the _____ over and over again. So I think that’s the key here.

    And of course when we talk about a specific random _____ with huge capabilities it has pros and cons, right? So one pro would be like they can mutually enforce new capabilities. So when they bring one feature in one product it’s mutually enforcing another product, which is a beautiful advantage they bring in. But that also limits our possibility to look beyond what the vendor is bringing in. So we are kind of essentially not exploiting our own innovation cycles to introduce more capabilities around how we want to experiment, how we want to kind of build innovative new features into that life cycle.

    So these are kind of good considerations to look at and I truly believe that the technical oversight committee is something that you will see in large organizations a wave kind of starts in a bigger way when it comes to DevOps.

    Ashley: And there’s interesting too, when you think about different sizes of companies that are in the market for technologies, you know, Larry mentioned what he did at Comcast or what he was part of that was created as sort of an architecture where tools can be plugged in, unplugged, replaced, etcetera. It sounds simple, but I know that they have an architecture to do that. And some organizations have the wherewithal to create those things. What we see in the data, looking at the market, is in the small business, the kind of sub-200 people or so companies, they have really good high-performance teams that can work to create software or set up their own environments, kind of create a very productive setup for themselves, tool chains, etcetera.

    Then on the high end or the large end, the enterprises, you know, they have a lot of resources available to them that they can potentially marshal to figure out how to do some of those things. The mid-tier market, the medium-sized company is often where they’re most challenged to – they want to do what the enterprise can do, but they just don’t have the resources. They don’t have the money or the vendor relationships across the board to do all those things and they’re expected to deliver at almost an enterprise-class level, but they don’t have all of the resources to be able to do that. And that’s often where somebody – the non-Frankenstein vendors that do a good job of kind of delivering a suite of solutions can work in those situations. Not that they can’t work in the others; they can, too.

    But depending where you are in that kind of a company, that size of a company, makes a big difference on what strategy you can take.

    Maccherone: I think that’s a great point and my recent experience last five years has really been enterprise. But prior to that I was talking to lots of companies, and you’re right, the mid-sized companies are the ones that are sort of in that no-man’s land sometimes, where they need some standardization, they need some consistency, but they don’t have the wherewithal to put it all together. And a vendor unicorn approach might actually be better in those circumstances.

    Shimel: Hey, guys, I want to – Ravyn, I’m going to let you in a second. John, the name of the first book that you mentioned – I want to put these books in the notes somewhere.

    Willis: Oh yeah. I did a presentation with Josh Corman way back with Guns, Germs, and Steel. Anyway, it’s an amazing book about how civilizations move from sort of – you know, how they go from sort of agriculture and they move into these clusters. If you can see the forest beyond the trees you can see how we get into these like Kubernetes is the only word we can use. Anyway, Jared Diamond, Guns, Germs, and Steel, it’s incredible.

    Shimel: All right, we’ll try to get it in there. Ravyn, did you want to say something and I cut you off?

    Manuel: No, I don’t think that was me.

    Shimel: Garima?

    Bajpai: Yeah. I wanted to actually enhance what Larry was saying about midsized companies. This is totally based on my experience and exposure with the Canada DevOps Community of Practice. We are seeing more and more of these midsized companies, not only the consulting companies, as well as the companies who want to build a DevOps capability coming to the community of practice and starting to steer this discussion of evaluation and assessment of tools. And I’m also seeing that the winter side. You know, specifically we are talking about tier two companies here. The winters and coming and exposing the tool capabilities in the community of practice, and they want some support and help for us to kind of steer this conversation “Is this a good tool? Is this a good capability where they are in their context?”

    And it’s a very interesting discussion because the technical side is one side of the story. I would also like to kind of emphasize the fact that the questions we ask these vendors and these companies are like, “Okay, do you have a product owner for this product?” Because essentially, you know, what it means is that how accountability is built in a specific tool perspective. I also kind of ensure that we ask them what kind of themes they have, like what kind of cross-functional themes they have, so that we do not miss out on the point of interoperability, integration, automation, all these like are important points.

    So I just wanted to mention that how we take that approach from the community of practice perspective when we’re asked to kind of assess.

    Willis: You know, I totally agree with all of that. One of the things I was thinking about, I’ve had this luxury of being involved in these different communities. Like I’m in Alan’s community, you know, so I’ve gotten hooked into usually what’s interesting and going on in Alan’s world, which is a pretty big world. I’ve been hooked into Gene’s world pretty heavily, you know, DevOps Enterprise Forum/Committee. And then, you know, I’m one of the co-founders and advisors for the DevOps Days.

    And one of the things that I get to see is true leaders. And I don’t want to put all this sort of weight and gravity on like you have to have a great leader to be successful, but like as I think about the people – Larry is a great example. Like, you know, you can talk all day long about what Comcast does. I’ve seen Larry in action. Like that’s the kind of guy you want in your enterprise.

    Courtney Kissler now, you know, was at Nike and now she’s moved on to a startup. Ross Clanton now at American Airlines, Heather Mickman. I mean I can go on on the list, but these are people that impact change in larger organizations that where it’s incredibly hard. And there’s just something about that.

    Maccherone: There is an immune system at large enterprises, though, that makes it hard for someone like me to actually be hired and be successful. You know, the idea of disruptive change is not something that enterprises are sort of seeking out ever. And so if you’re sort of like grassroots and you’re sort of like ground up and you get a few pilots going, that’s easy; there’s not a lot of resistance. It’s when you get enough momentum that somebody feels like their domain is threatened that you get the political pushback in an enterprise. And, you know, that’s really hard to overcome. Companies have to be very conscious about that. And Comcast has done a great job of that for me in the past, but I see it in the nature of the enterprise immune system essentially.

    Ashley: We probably could take any one of those last three comments and turn that into a whole other dialogue. Maybe we can get some help from the folks in the production department for pulling us back together.

    Alan, do you want to close this out?

    Shimel: One thing’s for sure, is we’ve got a lot more to talk about on this subject. I’d love to – I invite you all back to continue the conversation. But for now, Ravyn, I appreciate you soldiering through this on audio only. If we do it again we’ll make sure to get that video working. Garima, always a pleasure to have you on here; you bring so much. Larry and John, what can I say, it’s a pleasure to see you guys. I can’t wait till we can see each other in person. I know a lot of us have gotten our shots; it won’t be long. But thank you so much for adding so much. Mitchell, fantastic job.

    Most of all, hey, thank you people that are watching this, because that’s why we do this, is for you guys to get something out of it, join in the conversation, let us hear what you’re thinking. Many thanks to Tricentis for sponsoring DevOps Unbound.

    Until next time, though, this is Alan Shimel, Media Ops for DevOps Unbound. Have a great day, everyone. Bye-bye.

    [End of Audio]

  • EP 11: Unburdening Developers

    EP 11: Unburdening Developers

    Every app development need seems to ultimately fall to developers to fulfill. Build in security. Shift testing left. Provide users cooler, more customized experiences at cloud speed and scale. And do it all faster, of course. How do we accommodate the rising expectations without burning out developers? What can other team members do to lighten the load and allow developers to focus on the creative work they require to thrive? In this episode of DevOps Unbound, hosts Alan Shimel and Mitchell Ashley are joined by panelists Adam Kalsey of Tricentis, Tracy Ragan of DeployHub and Justin Hutchings of GitHub to talk about unburdening developers. The video is below, followed by the transcript.

    Alan Shimel: Hey, everyone. Welcome to another episode of “DevOps Unbound,” sponsored by our friends at Tricentis. We have a good, nice topic I want to talk to today, and you know, I refer to it as the straw on the camel’s back, but it’s about the ever-increasing load that developers seem to be taking on with technologies and frameworks such as DevOps and Agile and so forth that we’re seeing out here. And we’ve got a great panel to explore this. Let me introduce you to them before we go any further.

    Let me first of all introduce you to my friend Justin Hutchings. Justin is with GitHub and he’s gonna tell you a little bit about himself. Justin, welcome.

    Justin Hutchings: Thanks, Alan. Yeah, I’m Justin Hutchings. I’m a director of product here at GitHub. I work on some of our security features for developers, you know, whether that’s things like static analysis, security testing, or supply chain kind of features for developers in opensource and the enterprise. Glad to be here.

    Shimel: Nice to have you here, Justin. Welcome. Secondly, she’s a frequent guest here on “DevOps Unbound” and other MediaOps videos, my friend Tracy Ragan, CEO of DeployHub. Tracy, welcome, maybe a little bit about DeployHub and you for the audience.

    Tracy Ragan: Absolutely. Thank you, Alan, and thank you, Tricentis, for having these and always welcoming me on these panels. I like doing them. So yeah, I’m Tracy Ragan, CEO of DeployHub. DeployHub is a hub of metadata for deploying and configuring microservices. So we do automated configuration management so you know what your blast radius looks like for a microservice before you deploy. I’m also on the board of the CD Foundation, and you might see me oftentimes evangelizing around microservice usage.

    Shimel: Yes, we do. And then – welcome, Tracy. And then last but not least as a panel member today is Adam Kalsey. Is that pronounced right, Adam?

    Adam Kalsey: It is, absolutely, yes.

    Shimel: And Adam’s with Tricentis, but why don’t you tell us a little bit, Adam?

    Kalsey: Yeah, so I lead developer relations at Tricentis, and we make software to help people make better software, to test it and make sure they have higher-quality software. And my role is to help make developers more awesome by helping them make better software.

    Shimel: Thank you and welcome. And then last but certainly not least is my cohost for “DevOps Unbound,” Mitch Ashley, CEO of Accelerated Strategies Group, CTO here at MediaOps. Mitchell, welcome. Great to have you on.

    Mitch Ashley: Always good to be here, and having been part of many development teams, I’m excited to talk about this great topic.

    Shimel: Great. Let’s dive into it, guys. You know, when I started devopsy.kinsta.cloud about eight years ago, there was this constant push and pull. Was DevOps too developer-centric? It was all about the developers, the ops said. This DevOps thing is just – it’s all about the developers; we’re not getting our fair share. And you spoke to developers and it was, “You know what? DevOps is too ops-centric. It’s not about the ops. It can’t be everything ops. It’s got to be about developers.” And so there was this, you know, friction about was it too developer-centric or not.

    Over the years, though, what I’ve seen is an increasing amount of responsibility seems to have shifted left, as we say, right? And we shift left, and when we shift it left, we’re primarily – we’re shifting it to the developers. So things like testing, for instance, right, have become – developers are probably more involved in testing, or at least kicking off testing and this stuff, than they ever were. Security – certainly the whole rise of the DevSecOps movement has been about shifting security left, and in fact many of the most successful security companies in the DevSecOps space have become successful by making security tools for developers. We’re making testing tools for developers. We’re making tools to look at your blast radius for microservices for developers. Developers are deploying, right, as part of their DevOps focus.

    When is enough? When do we break the developers’ back? Not to mention the developers are the highest-paid people in the food chain, right? So by giving them everything, are we giving them nothing, right? Are we stunting their productivity? Adam, being that you’re the new guy here and you actually are in charge of developer relations at Tricentis, we’re gonna ask you to kick it off. Are we putting too much on the backs of our developers?

    Kalsey: I don’t think so. I think that the promise here is that we make things easier for the developers. I mean, sure, you have more tasks. There’s more things that you’re responsible for. But the overall effort and the amount of work that has to be done ends up shrinking. If you don’t find out about a bug in your software or a security problem or such until you’re in production and customers are using it, now all of a sudden you’ve got way more work to do. It’s harder. It’s more expensive. It’s more difficult. There’s more involved in doing that. And so by moving a lot of these things forward, we’re letting the developers be more proactive in taking care of them. We’re helping them get things fixed sooner. We’re reducing the amount of toil that developers have to do. Yes, there’s more upfront work, but that long-term toil starts to fade away and disappear.

    Shimel: Fair enough. I don’t know if you – you know, I don’t know if I’ve ever heard someone say developers are – you know, it’s making it easier or less for them. Justin, what do you think?

    Hutchings: You know, I’ve talked to a lot of companies that are the early stages of their DevOps transformation, and I think one of the most common mistakes that these companies make is they fail to recognize that adopting DevOps requires both technical changes as well as cultural changes. You can’t just get rid of the testers and say engineering owns tests and expect the same velocity. You’ve got to build out the infrastructure to support that, you have to do the training, and you also have to recognize your throughput is going to be less while you’re paying down that sort of debt on getting that infrastructure and getting the automation and getting all of those things.

    Companies that fail at the DevOps transformation, they go through sort of the motions without understanding the culture and talking about it. You know, years ago when I worked at Microsoft, the Windows team sort of famously had a one-to-one ratio between engineers and – software development engineers and test. These were folks that wrote test automation. And there was a big layoff, thousands of testers got laid off, and they said, “We don’t need them anymore. We have observability tools. This will be great.” And that was the most naïve statement I had ever heard, and it took a year for the organization to fully rebound, to figure out what the right set of responsibilities were for engineering, and how to support those folks effectively on it.

    Shimel: Fair. Tracy, in your roles, both at DeployHub and CDF, you’re dealing with both constituencies here, right, the traditional DevOps teams. But let’s face it: You’ve got dev and you’ve got ops in there, and you deal with both. What’s your take from the battlefront?

    Ragan: Oh, my goodness. I could write like a 20-page blog on this. I guess that wouldn’t be a blog. It would be a research project. [Laughter]

     Shimel: Or a book.

    [Crosstalk]

    Ragan: Yes! So I think that in my discussions with folks over the last six months, eight months, I am seeing a lot of SREs really stressed out. So I don’t know if it’s the development teams that are really taking the brunt of this, but if we think about our history and how we got here, we’ve always had somebody really important on a development team who always worked in hand with the operations side of the house to find out what happened, to help with the deployments. Even if they’re sitting there watching the operation first and hit the button, they’re the person that’s always taken this responsibility. And I think the DevOps movement was an effort to push more operations tasks to that trusted partner on the development team, and now that role has turned into an SRE.

    I would say five years ago we wouldn’t have seen this kind of stress, but because we’re going through this massive tsunami moving from monolithic to microservices, and we’re in a transition phase, we are putting a lot – what I see is we’re putting a lot more stress on the SRE role. Developers are going to the SRE, operations are going to the SRE, and the SRE is just trying to keep up with the underlying technology that’s wobbling and moving. You know, they’re trying to figure out how to stand up environments easier. They’re trying to implement Terraform or Rancher. They’re doing everything that they can right now to try to keep up, but they may be underwater for a bit longer.

    So when we think about where the stress point is right now, I think that developers have pretty much automated a lot. You know, we have tools like Tricentis to really improve the testing, but now we’re int his weird phase where we have these SREs that are trying to straddle both worlds. And they’re struggling.

    Shimel: Justin, you know, in your role there – and, guys, I don’t mean to call. If you want to say something, feel free to obviously jump on. But, Justin, I’m wondering – you know, GitHub has sort of a unique view with this.

    Hutchings: We’ve been trying to push folks towards more automation, more DevOps, you know, DevSecOps with all the security features we have. You know, again I think culture is the key. I look at anytime you have an unfunded mandate, you know, it’s gonna fail. A lot of times organizations will say we’re going to microservices or they’ll say we’re gonna adopt all these security tools, and then they don’t think about how that actually gets done and they don’t recognize the sort of pain and training and all of the things that have to happen to get folks there.

    You know, and I think there’s been a lot of advancement here. You look at the state of observability. You look at the state of CI/CD. Things are getting better, but I think, you know, we in the industry that sort of work int his space sometimes paint such a rosy picture that our customers aren’t prepared for the hardness of the journey that lies in front of them. And they’re looking at how do I get to the magical garden of DevSecOps, and the magic garden is there, but it is not a free entry, you know?

    Ashley: You know, Justin, to what you were saying, I think along with culture, part of is a role change that’s happening across the whole organization, right? A couple things are happening. We’re doing not just automation that the developers are doing, clearly SREs. Other test automation, infrastructure as code, you know, those are all development that’s happening to support those. It doesn’t say that every person is an expert developer, but there’s some development, scripting, other kinds of skills that come with it.

    So I don’t know if we’re so much loading it on the developer as much as we’re elevating our level of automation and kind of interconnected of the process, and we’re helping people evolve and growing their strengths – let’s put it that way – to also include some software development, or more closely working with people that can. I think that’s part of that cultural change, too.

    [Crosstalk]

    Kalsey: One of the things that’s actually happening there is as things become more development-centric and, oh, now it’s Toad and now it’s – the teams are making exactly the wrong lesson out of this and they’re learning the wrong thing out of it. You know, one of the things that Justin said earlier is, “Well, we got rid of all the software testers because the developers are doing the testing now,” and that’s one of the dumbest things that you can do as an organization, is say, “Well, now that my developers can do this, let’s get rid of all the experts.” And instead of saying, “What is the new role of this expert and how does their role change?” whether, okay, now they learn to code so that they can expose more of this as code, or do they become consultants to the developers and help them do this right and help them understand how to do this right? Could I remove a lot of those pieces of toil and a lot of that extra stuff from them and let them focus on the things that they’re good at – writing the code, building the code, thinking through the problems – and let me focus on the things that I’m good at – thinking about the infrastructure, designing the infrastructure, designing test plans, thinking about the non-happy path?

    And so companies – I talk to companies all the time that say, “Well, we’ve moved – all our developers are doing the testing now, so we got rid of our QA department!” And that is – that’s just frankly dumb. Don’t do that.

    Ragan: I think it was an outcome of the Agile movement, right? Because we started preaching this idea of bringing everybody into the same room together to do these tasks. So we broke up our testing departments and we spread them out across all of the teams, as well as some of the operations, which is how the SRE title came about. So, you know, first we siloed and then we tried not to silo. But ultimately it is how do you come up with a strong automation standard across the organization? And how are you gonna implement it? If there’s a testing team that has a group of people and each one is assigned to an application team, or if you have a siloed testing team and a siloed operation team, ultimately we’re still pushing the same product through the lifecycle. So how well do we do that? And maybe Agile confused things for us for a while and kind of broke things up and disrupted what we were doing. So now the task is how do we move forward with where we’re at to make it continue to be easier to get things across the pipeline?

    And I have to say that one of the biggest challenges I hear from people is that continuous integration works well; continuous delivery from dev, test, through prod doesn’t work so well.

    Hutchings: Yeah, and I think, you know, what you hit on there, Tracy, is really important because, you know, when you have an organization where the engineers are the only thing and engineers are undifferentiated and treated as some sort of fungible resource, I think things fall apart. You know, what I’ve seen in successful organizations is they have some subject matter experts, whether that’s SRE or a data team or an infra team, that are building out things that make the rest of the organization able to scale. And that’s a really good, healthy blend where you’ve got experts that you can lean on, but then engineers are empowered to do sort of the last-mile delivery there.

    Kalsey: Yeah. When a company says they’re gonna create a cross-functional team and then they eliminate most of the functions, that’s a problem. I mean, you don’t say I’m gonna have a cross-functional team, so we’re going to bring product managers and designers in and, oh, by – now we just don’t need product managers and designers. So why do we do that with operations, with testing, with all of these other things that are important for software creation? You know, eliminating those and just saying, “Well, the developers will do it,” of course that makes for a lot more work for the developer. The way that this needs to work is keep those cross-functional teams and change their role and bring them further – bring them earlier into the developer cycle. Have them involved in design. Have them involved in figuring out what the product is going to do and how it’s going to do it, not just let’s get rid of them. The options aren’t they happen at the very end of the process or we don’t have them at all. It’s let’s bring them up further along in this process.

    Shimel: You know, part of it, though – and I don’t know if anyone wants to admit it – is that there’s sort of this bias that everyone else on the IT team aspires to be a developer ’cause that is the ultimate kind of thing. So, for instance, it was this way certainly in testing, that being a tester was a waystation on the way to being a full-blown developer, right? Or maybe a developer who wasn’t that good, well, he could do testing, right, and that testing in and of itself wasn’t a “profession.” It was really just like a junior developer did testing. And, you know, security people, well, they were always the rebels, right? Half of them didn’t have formal training. But you know, operations, the SRE folks, right, how much of a developer heritage or pedigree is there in what you need to be able to do from an SRE perspective? And so when we put together these “cross-functional” teams, it’s almost with the idea that, well, you’re all gonna be developers now, right? You’re gonna reach your lifelong ambition by being in this DevOps team as part of a developer. And we lose out on the professional tester, on the professional SRE, on the professional security person, right, in this rush to all be developers. What do you think about this?

    Ashley: Well, I think we did this to ourselves in the DevOps movement. You know, I remember when I first wrote my first article on devopsy.kinsta.cloud about what is DevOps and I was sort of grappling with it myself, the mantra was, “We don’t need operations, we don’t need test, we just need developers. Code or developers writing code will eat the world.” And it wasn’t outsiders that said that; we said that, or subsets of us said that. And I think we recognized after we got into it, no, as Adam points out, those are exceptional skills. Frankly some of the worst testers I’ve worked with were developers. They were terrible at testing their own stuff. They were great developers, right? But some of it is elevating maybe too much the role of the developer to be the rockstar beyond the platform, if you will – and there are fantastic developers. I don’t mean there aren’t. But there are amazing, amazing test people.

    And so let me put it to you this way. When I was running product development at StillSecure with you, who did I listen to? The testers. I listened to them and I asked them, “What problems are we finding?” and I could tell if we were ready to ship, not just by the numbers but by their opinion. And that’s why I really relied on, because they saw it from a different perspective than a developer does. So that’s my opinion about it. Now feel free to disagree. Anybody can take me on and say I’m full of baloney.

    Hutchings: So I was gonna say, Mitch, you know, have folks heard the story about – what is it – the IBM Black Team that they developed back in the ’80s for testing where they – basically testing was a sort of poorly-regarded function in the organization? And they ended up creating sort of a hit squad that famously would go and just destroy people’s code. And they created a culture around that, and they created sort of an ethos of that being a special place to work. And I think that’s something we need to find with our developers. Like, if we only celebrate the developers that make the shiny features, we’re not gonna be in a good state. We’ve got to celebrate the developers that can destroy other people’s code with tests or can build something that scales really well or that build great observability things. Like, it can’t all be how many features did you ship; it has to be how much you contributed to the whole.

    Ragan: Yeah, and I – you know, I think, Justin, that the cultural discussion is one that any team that’s going through this experience right now that we’re describing, that the developers do feel burdened in some way, they should be looking at how they have built that team. And kind of each role should be a specialist role, and they all have to be an equal role. And if you don’t have that, you really haven’t created an Agile team and that team’s not gonna become a high-performance computing team, because it is about equality on the team and the ability for every single person to stand up – tester, the SRE, the junior developer – and be able to communicate problems as they see them in an equitable way and not feel that they are being judged.

    And I have learned that from working in an opensource community. I’m really amazed how well the opensource communities work together, ’cause you have people with all kinds of different backgrounds, and it fascinates me to see how much they can accomplish and how well they accomplish it when they’re not all working for the same company and they’re there because they want to be. So how do you create that opensource kind of nurturing culture within a particular team if it’s built especially around Agile, where we’ve taken operations and testing and developers and, you know, other specialists and put them on the team – documentation, requirements gathering – and how do you make it an equal team? That I think is the challenge to keep the burden off of the developers’ backs.

    Shimel: You know, and that – paradoxically, that’s also the big attraction to like cloud-native communities, to these opensource communities, because I think people recognize the – and I don’t mean it in a bad way – the kumbaya kind of spirit that these opensource communities foster. GitHub, quite frankly, Justin, right, if you look at the history of GitHub, that’s probably been one of the huge drivers of 52 or whatever it is million accounts on GitHub, is that sense of community, right? How do you build that sense of community interdisciplinary across your teams within an organization? Adam, you know, though the role says DevRel, it’s really TeamRel, right? And you know, it’s got to be part of your mantra there, part of your job description, isn’t it?

    Kalsey: Yeah. One of the things that is interesting about when you look at opensource is the person who wrote the documentation is a hero, because nobody wanted to write the documentation. The person who wrote the tests is a hero because, wow, they’re writing the tests, and that was a lot of work and I didn’t want to do that amount of work. But then when we go into a company we say, “Well, we’re gonna hire a couple of QA people to do the testing because they’re cheaper than the developers.” And, well, yeah, maybe if you’re hiring bad QA people they’re cheaper than the developers, but if you’re hiring good testers, they’re as expensive or more expensive than the developers.

    And so in the opensource world we look at it and we go, “These people are peers and equals,” and then we come into a company and we say, “They’re second-class citizens, and we’re gonna hire a couple of tech writers ’cause they’re a lot cheaper than having the developers write the documentation, or we’re gonna hire some testers, or we’re gonna hire some ops people ’cause the person to watch the dashboard is way cheaper than having a developer do it.” And we’ve seen it happen with the SRE role where, when we elevate that and that becomes part of the development team, the other – SREs are expensive and hard to find. That same sort of thing I think needs to happen in other roles on a development team in order for them to stop being that second-class – and also to stop having developers be the ones that have to do all of this work. If you look at it and say, “Well, the test writer or the doc writer is a cheaper person and we’ll just hire a couple of them,” it’s more likely to fall on the developer to do the hard work than to hire the good people and have them do the hard work.

    Shimel: Fair enough. Mitch, what do you think on that?

    Ashley: You know, it’s a really good point, and I think part of what you’re saying, Adam, is it’s a specialty. You think differently to be a fantastic tester, not just testing but the whole test strategy and how to do all the different kinds of testing that’s involved, and then of course how do you automate that and do you do that yourself? Do you work with part of the dev team to do that? Same thing for SRE, right? It takes a different – someone who thinks about it differently. And it’s sort of that cognitive diversity of teams that we value from a diversity standpoint. We value that also because those roles also require specialty skills. You know, someone who’s designing – and, Tracy, you could explain this better than I could, but someone who’s designing a cloud-native architecture and thinking about the microservices and what’s gonna constitute a microservice, how we’re gonna communicate between them and how we’re gonna secure them, that’s a different kind of thinker than how do I break all that stuff. That’s really nice that you did that, but watch how fast I can break that. That’s fun, and that’s where people really enjoy working together. But I think that’s part of where you’re elevating, too, Adam, is you’ve got to value all the skills, and they may be expensive, they may be not quite as much, but they’re needed. They’re part of the process.

    Shimel: Interesting. You know, Justin, we’re talking about testers. We’re talking about SREs. We’re talking about developers. Once again, no one’s talking security. Are we just sentenced to be forever the outsiders here? What’s it gonna take?

    Hutchings: Most of the security people that I talk to take that as a challenge, you know? We obviously want developers to go and embrace security as part of, you know, their everyday job descriptions, but folks in AppSec and especially in the security research community, they’ve always felt like kind of scrappy outsiders that are going to shine a light on things that other people didn’t notice. And so I think even when we have very security-minded, security-motivated engineering teams, security is a discipline that’s got to stand apart, because we want to have that sort of trusted auditor role or that, you know, lookout for – whether it’s bug bounty reports or stack analysis findings that no one looked at. Like, someone’s got to be watching those, ’cause developers, especially in the world of microservices, they might work on a thing, finish it, and then go somewhere else. But the security world changes all the time. If you walk away from that microservice and all of a sudden we learn about a new category of security problems and no one’s watching it, well, that’s a problem now. And so security has a really special role in terms of being the stewards of all the things, even when they’re not directly accountable for the functionality of it. And I think that’s a special thing that we should celebrate.

    Shimel: Okay, we’re special. But you know, as special as we are, yet we’re still seeing security tools for developers, right? That is a bona fide trend in our industry, and at the same time we’re also seeing – when we talk about the load on developers, we’re still seeing an extreme need for more security people, right? We just can’t produce enough qualified security folks, it seems. If that’s the facts, right, there’s no getting around it. What else is there to do but put this load on the developers? How else do we deal with it, automation? I don’t know.

    Hutchings: You’re absolutely right, and I think the key for tool spenders like GitHub is to figure out how to give developers enough information that they can be successful but not overwhelm them, because security tools, you know, there’s a risk of false positives. If you give them too many, they will ignore them. And so, like, as a tool vendor, I have to be extremely careful to make sure that I don’t burn out developers on the things I think are important, ’cause they might not find them as important as I do.

    Kalsey: The other thing those vendors have to do is make less work, eliminate work for the developer, not cause more, not give them more things to pay attention to, more work to do, more administration to do, but look for ways that you can remove that work through the tools.

    Ragan: Yeah, so we have a challenge facing us when it comes to automation right now, and the challenge has to do with these imperative pipelines that we’ve been running for the last ten years. Sometimes you have a Jenkinsfile that is pretty complicated, and we’ve been running them, we copy them, and we use them. It’s imperative.

    The problem with it is because it’s imperative. We are seeing a shift now to moving away from these imperative pipelines to where it’s more declarative and event-driven, which means that it should be easier then to take these older – if you had a purely declarative pipeline based on events, think about how easy it would be to say, “Okay, we’ve decided on this set of security requirements to add to sort of the workflow steps for the testing workflow, for the development workflow, and we’re gonna plug those events in.” The reason why we don’t do that now is because it’s too hard. [Laughter] It’s just hard, because we have to take these old imperative pipelines and try to modernize them. And without having that declarative framework, these kinds of tools, even adding continuous testing, even adding continuous deployments all the way to production, it’s hard and it takes a lot of work, and you have to change the tires as you’re going down the freeway in an imperative model. The faster we can get to an event-driven process in our CD pipeline where we can declare them and we can automatically generate what we need to do in a workflow, the sooner we’ll be alleviating some of the pain that the developers might be seeing, the sooner we can bring in better testing, the sooner we can bring in the DevSecOps conversation, and in our case automated configuration management.

    This has always been a problem for any of the tools in the CD pipeline to start being plugged in, because the imperative pipelines that we are dealing with on a daily basis, nobody wants to touch them. So we’re gonna have to disrupt that first, and when that gets disrupted and people start moving to something more declarative, we are gonna see an increase in adding all of this kind of automation into the CD pipeline.

    Ashley: I’m curious, Tracy. Is it the complexity of the pipelines that – the way they’re defined, is that why people don’t watch to touch them, or –

    Ragan: It’s the number.

    Ashley: The number, okay.

    Ragan: You’ve got – literally I’ve seen people with thousands and thousands of workflows ’cause they’re running one for every single application version. And to look in that and say, “Oh, we’re gonna change all of these,” because it’s not templated so you can’t just change it in one place and it’s just gonna trickle down, because they’re all these very statically scripted processes. And for whatever reason on the distributed side of the house – when I say distributed, I’m talking about everything except the mainframe – we have relied on heavily scripted processes, and we’ve scripted ourselves into a box that we cannot – we’re not Agile. We cannot easily update these pipelines to include security in them.

    We talk about it at the CD Foundation often, trying to elevate security as a first-class citizen in the CD pipeline. But that means that everybody’s got to start updating all those CD pipelines, and they’re not templated. They’re all statically scripted so you have to address one at a time. And as I think Justin might’ve said, you’ve got to plan to pay for that. So we’re in a box, we’re in a corner, and we have to somehow get ourselves out of that corner in order to alleviate these kinds of problems. And being Agile at the CD pipeline level means that we’re gonna have to dump these imperative processes.

    Hutchings: And I think you’re exactly right there, Tracy. One of the challenges we also see with security is this desire to start slow so that you don’t overwhelm people with too much, and if you don’t have a centralized control plane that everyone pulls from, you know, as you start to introduce these security tools, you can’t dial up the difficult level as you go. You have to hit those distributed CI pipelines every single time you want to go and update the difficulty, and that’s not tenable. I mean, you know, there are services out there – I think GitHub with Actions is doing a pretty okay job with allowing some centralized management, but it’s a spot that we all have to invest in because, you know, that control plane for specialists like security to go and help inject things is just not as mature as it needs to be.

    Ragan: And the CD Foundation has started an events working group to start having this discussion. Right now they’re talking about just vocabulary, but I believe that will grow into a bigger project in the future so that we can start thinking about how you can have that event-driven process so that you can dial it as you need to, right, as oppose to having them statically scripted in all of these imperative pipeline processes.

    Shimel: So, guys, we’re running low on time. Just one other topic I wanted to bring up. You know, in looking at this problem, though, I’m reminded of when I read The Goal, right, which is kind of the book upon which Gene Kim based his Phoenix Project on. The Goal is about more manufacturing; The Phoenix Project was IT. But really it talks about bottlenecks, resource constraints. And the story of The Goal – and I forgot if it was the first rule, the second rule, whatever it was – is as soon as you fix one or get rid of one problem, the next problem down the line shows itself that maybe you didn’t see because the problem in front of it was hiding it camouflaging it. In trying to make life easier for our developers or take the strain off of that point, are we really just going through this goal process of discovering the next bottleneck? And maybe that’s what – maybe that’s the secret to life. Life is just about going from one bottleneck to the next, right, and discovering what’s downstream from there. There’s something Zen about that. But is that what we are sentenced to for sure, or is there a way out of that?

    Kalsey: I like to say that when designing large-scale distributed systems you can’t remove your scalability or your failover problems; you just move them to a different part of the stack. And we do that a lot with development teams as well, is we don’t remove the problem; we just move it to a different part of the process and move it around.

    I think one of the ways out of that is one of the tenets of Agile is let the teams make the decisions. Hand them the problem and let them decide how to organize, how to do these things, how to make stuff work. You know, if management is pushing down on development and saying, “You’re now going to be the testers,” of course we’re causing more work and more problems and more things for them, and they’re going to become the bottleneck. But if we let the teams figure out – they’re the ones close to the work, doing the work. Let them make the decisions about how we’re gonna do this and what they need to make it happen.

    Shimel: Fair enough. All right –

    Hutchings: Yeah, I was gonna say I like the term incrementally correct. You know, Alan, your thinking about bottlenecks is absolutely true. I don’t think there’s ever a world where we will get to 100 percent panacea, but we can hope to get a little bit better all the time, and that’s the journey that we’re all on, right?

    Ragan: And failing fast.

    Shimel: _____ continuous improvement, and failing fast – like get to know –

    Ragan: And allow your developers to fail and not – or your testers or your security. It’s okay to fail, because then you learn what you need to do. Failing fast is part of the learning process.

    Shimel: Fantastic. All right, guys, we are out of time. I want to thank all three of you for joining Mitchell and I on this episode of ” DevOps Unbound.” Thanks to Tricentis for sponsoring ” DevOps Unbound,” as always. We will be back in – not next week, the week after with a new, fresh episode, as well as check out – we’ll have a new ” DevOps Unbound Roundtable” open to the public for questions, and you can find out about that at DevOpsUnbound.com or on our Digital Anarchist site or devopsy.kinsta.cloud frankly. Look under – it’ll be under webinars and you’ll find the Roundtable there.

    Until then, though, Tracy Ragan, as always, it’s a pleasure to have you on. You bring such enlightenment to the whole discussion. Thank you.

    Ragan: Thank you, Alan.

    Shimel: Thank you. Justin, again, always nice to have you on one of our panels and discussions here. Many thanks for coming. And, Adam, thank you and good luck in – you know, I know you’re relatively new in your position there at Tricentis but it looks like they made a hell of a hire. Good luck, and come back and visit us again.

    Kalsey: Would love to. Thanks a lot.

    Shimel: All right. Mitch, you want to take us home?

    Ashley: I just wanted to thank our panel as well, and I love the incremental progress idea, and it’s always getting outside perspectives sometimes, too, that’s helpful. So I think we’re here to help each other in this process and not one of us has the answers.

    Shimel: Right on. All right, this Alan Shimel for “DevOps Unbound.” We’ll see you on the next show. Bye-bye, everyone.

    [Outro music playing]

  • EP 10: Observability & Continuous Improvement

    EP 10: Observability & Continuous Improvement

    Observability at enterprise scale brings with it additional application life cycle management requirements. Success requires knowing which changes result in improvements and which don’t produce positive results and how that information must make it back to the DevOps team through feedback loops. In this episode of DevOps Unbound, Alan Shimel and Mitch Ashley talk with Kurt Chase, senior director, global release management at Tricentis, Charity Majors, co-founder at Honeycomb and Ravyn Manuel, senior application manager at the Smithsonian as they discuss observability and continuous improvement. The video is below, followed by a transcript of the conversation.

    Alan Shimel: Hey, everyone. I’m Alan Shimel and welcome to another DevOps Unbound. We have a great panel in store for you today here on DevOps Unbound. Our topic is observability at a broad level. We’re going to jump into some of this. Let me first of all introduce you, though.

    First I want to introduce you, of course, to my cohost of DevOps Unbound. He’s the CEO of Accelerated Strategies Group, CTO here at MediaOps, my friend Mitchell Ashley. Hey, Mitch. Welcome.

    Mitch Ashley: Alan, thanks. Great to be here.

    Shimel: All righty. Let me introduce you to our panel for today. We have a great panel. I was so happy that this came together. First, coming at us from the mountains of Virginia by way of the Bronx, Ravyn Manuel. Ravyn, why don’t you introduce yourself and where you work?

    Ravyn Manuel: Absolutely. My name is Ravyn Manuel and I work for the National Museum of African American History and Culture, which is part of the Smithsonian. I am the Senior Applications Developer and DevOps Engineer. And thanks for not putting me in West Virginia. I appreciate that.

    [Laughter]

    Shimel: Okay. I don’t blame you. Next up we have the one and only Charity Majors, CTO and Co-founder at Honeycomb. Charity, a little bit about yourself, maybe introduce to the audience?

    Charity Majors: I am an ops nerd. I have been on call since I was 17 racking hardware. I no longer rack hardware anymore. Hardware doesn’t exist, as far as I’m concerned. Instead, I’m trying to save the world through better observability.

    Shimel: Excellent. And then, last but not least, we have Kurt Chase of Tricentis, Tricentis Releasing. Kurt, why don’t you introduce yourself?

    Kurt Chase: Hi, everyone. My name is Kurt Chase. I’m currently Head of Global Release Management at Tricentis, based in Northern California. Nice to join you all today.

    Shimel: Nice to have you here, Kurt. All right. So, team, panel, our topic this week is observability. And given that is the topic, I feel compelled to start with Charity because, Charity, in many ways you and the Honeycomb team are responsible for all this buzz around observability. Tell us about it.

    Majors: Yeah, a little bit. When we started the company a little over five years ago it was not a term that was really in use – used in tech. Now – which is not to say that we invented it. Far from it. But it has this long and impressive pedigree in mechanical engineering, specifically control theory. And in control theory observability is the mathematical tool of controllability.

    So, we started building this startup, Honeycomb, and we knew that what we were doing wasn’t monitoring. We knew that we weren’t reactively monitoring. We knew that we were – it was about instrumenting the code, getting at it from the inside. But for the first five, six months we couldn’t figure out what to call it. It was really hard to figure out how to talk about what we were doing. And then it was mid-July, midway through the first year when I remember Googling “What is the definition of observability?” And when I read it it’s all about “Can you understand what’s happening on the inside of the system just by looking at it from the outside with no prior knowledge?”

    And a lightbulb just started going off. It was like “Ding, ding, ding. That’s what we’re trying to build. That’s exactly –” if you translate that into software terms, what it means is can you explain any system state that – any state that your system can get into without shipping new code to handle that state because shipping that code to handle that state implies that you had prior knowledge. So, it’s all about the unknown unknowns instead of the known unknowns. So, yeah, that’s my story.

    Shimel: And you’re sticking with it. So –

    Majors: Yeah. I mean, it really encapsulates a lot of the problems that people are increasingly having, that there’s a lot that precedes from that definition of “unknown unknowns,” like you need to have high cardinality, high dimensionality, blah, blah, blah, all this stuff. But those are the things that speak to the problems of microservices and multiple storage systems, et cetera.

    Shimel: I get it. So, Ravyn, this is your life. How does this observability and what Charity has describe, the fact pattern or the problem pattern that she came to call observability, how does that in your life, day to day of what you’re doing, how does that kind of manifest itself?

    Manuel: Thanks, Charity, for actually providing a definition because I was like “Okay, are we supposed to define our own meaning of observability?” So, you’ve given a baseline. And I like that you said it’s not monitoring because I can actually feel the nuance. There’s a nuance between monitoring and observability. And it’s – what’s really interesting is that that’s a problem that I’m trying to solve today at work because we’ve got – and – because we’ve got the black boxes. So, you’ve got observability but when you have software there’s a lot of black boxes. You’re not supposed to be asking what’s going on inside the code because you’re not supposed to know what’s going on inside the code. But when you get a – you put a signal in – I was in the military, so I did electronic repair – so, you put the signal in and you don’t get the right signal out but you’ve got a black box. How do you know what went wrong there if you can’t actually observe what’s going on inside of there?

    It’s one thing to know the state of the software when you are the developer. So, as a developer, yes, I know. But when it comes to being the product owner or the project manager or a stakeholder and you’re trying to figure out what – “Why is this doing this?” or “This is not what I wanted,” actually is how that would come from that end, is “That’s not what I wanted.” So, how to get the stakeholders to get what they want when you can’t actually tell – I can’t tell what the code from my coworker is doing? So, in a way observability is a little bit – it is abstracted from monitoring because monitoring is telling you what it’s doing but it isn’t telling you the state. And so, it’s a problem that I’m trying to solve actually today.

    Chase: I was going to jump in here and say from my experience although monitoring, logging, reporting, alerting, they’re not observability, I think they are fundamental foundational pieces that you have to have in place. You can start observability without any of them, but I’ve found that the more you can get your monitoring and reporting and even get some alerting going, it allows you to then start observing the system. And for me, I know at Splunk and now here again at Tricentis I’m really trying to connect all the pieces because it’s not just about the applications, but I find that some of the DevOps engineers, they really help with observability, especially the ones that are super inquisitive, and they don’t just – when a result appears or something happens they don’t just take it at face value. They’re inquisitive. They dig in. They try and find the root cause.

    And I think that’s key with observability, is taking all the monitoring, reporting, alerting you have, taking the feedback from your customers, all of that comes together, I think, to really help you start to define those unknown unknowns that Charity brought up. I think it becomes very complicated. And then I think I’d love to talk about how AI and machine learning now can start to learn some of these things, because I think we get really good at repeatable incidences, observing those and then accounting for those in our systems, but it’s the unknown unknowns, how do we deal with those? That’s the real interesting part to me nowadays for sure.

    Majors: It is, I think, the interesting part. And I also think that AI ops is mostly bullshit and it will be for the foreseeable future.

    Chase: For a while. I agree, yeah.

    Majors: But I wanted to circle back to what you said about logging and everything, which is – so, there’s a technical definition for observability which has to do with high cardinality, all this stuff, and then there is just understanding your systems. And I absolutely agree: You need some alerting, you need some monitoring. I think there’s a question is – of – it log files? Is it metrics? That’s kind of irrelevant for just data. What’s important is can you understand what’s happening in your system when you need to understand it?

    And I do think it’s important to sort of differentiate the technical definition of observability from just the tendency that we have to, say, observing things because there’s a very rich and mature and well-established set of best practices for monitoring, which are in many cases completely mirror opposite from the best practices for observability.

    For example, with monitoring it’s like for years we’ve been saying, “You should not have to stare at graphs all day. The system should inform you when you need to – when there’s something wrong.” Super canonical best practice. With observability it’s exactly the opposite. It’s – we’re saying that most problems that happen are so subtle that you need to be looking for them. You don’t – you can’t page everyone at the threshold – you would drive them nuts if you were like “Oh, every little thing is going wrong. Bing, bing, bing.” You’re just – you would drive people nuts and they would quit. But when you’re an engineer, when you’re working on this endpoint or when you’re trying to solve this problem you should absolutely be looking through, asking questions, iterating on them, and looking for the trails in your observability tool at that very fine grain level.

    Chase: I agree.

    Shimel: So, I hear what all three of you said and it’s funny how everyone brings their own baggage to this issue.

    Majors: Oh, sorry.

    Shimel: No, no, it’s not – don’t be sorry. I love it. Here’s, I think, fundamentally – for those people out here who are watching this whole – I’ve got a lot of people out here who are DevOps people, who are ops folks, SREs, cyber folks, and their heads are spinning because I think there’s a little bit of what Kurt said to them, which is they don’t do – they do a shitty job monitoring, quite frankly. We don’t do a great job of it. People started throwing this AI ops word around, and there are a lot of companies raising a –

    Majors: Yeah, there are.

    Shimel: – crap ton of money around their AI ops stuff. And I never really understood what was real about it and many times it just sounded like another word for monitoring to me. But whatever. There is no AI in it.

    But – and now, look, over the last year, year and a half, two years we’ve seen observability come up here and they don’t get the difference. Ravyn, I think you do because quite frankly your black box analogy with electronic in the military, that is – you’ve got to figure out what the unknowns are from the unknowns. It’s not just getting all these signals that are known and then recognizing patterns and putting some automation behind it. That’s not observability. I’m sorry. Not in my mind, anyway.

    So, I think number one is we need to clearly define it. And Charity, you guys, you are a major kind of evangelist for this: “What is observability versus AI ops versus ALM,” which is before AI ops, and so forth. And there are SREs who are living this and DevOps teams who are living this every single day out here who haven’t got their heads around it. Ravyn, I always look to you as someone who gets it. Where do you fit on the – where do you kind of see it on the – let’s say there’s a scale of ALM, AI ops, blah, blah, blah, observability. What do you think?

    Manuel: I feel like I would be very – most close to observability because, again, all – anything with ops in it, when you start doing DevSecOps and DevOps, all of those ops just become marketing terms after a while. But observability is – because it’s not quite tangible it’s actually more real, because it’s real – because it speaks to exactly what’s coming, what’s going on when you’re trying to get continuous improvement, because you need metrics to show and prove that you’re doing continuous improvement. But how do you get there? What is it – what are the steps to get you there? How are you alerted that you are not improving or that you are going backwards?

    And so, when – I feel for me as far as observability I’m at the cusp, because I didn’t actually know that the word existed until I found out about the talk. I didn’t know that that had been defined. So, it’s really cool to know that I can now call it observability. It’s because it’s that thing that I can’t – there isn’t a tool to actually help me – actually help me know what it is that I’m looking at all the time. Not all the time. There are some tools that help me some of the time but not all the time. And it’s the times that I don’t know that’s when we are stuck, because it could be impacting the way – the system performance or the team’s performance.

    Shimel: Charity, I think you might have a customer there. It sounds like they need a tool.

    Majors: I feel like one of the key differentiators here between observability and AI ops and stuff is what it tries to center. With observability we are not – AI ops are like “Eh, we’ll tell you what to look at. You don’t need to understand your system. We’ll understand your system for you. We’ll tell you what to look at. We’ll tell you what it means. We’ll tell you what’s important.” With observability it’s the opposite. I think that we need to look to solutions that let computers do what computers are good at and let humans do what humans are good at. Computers are good at crunching lots of numbers but they can attach meaning to something. Only human beings can attach meaning to something. If I see a big old spike of red that could be good, that could be bad. You don’t know. There is no meaning until a human has attached them to it.

    And with AI ops, AI ops only really works if the corpus of data that you’re training it on doesn’t change. So, if you’re actually shipping software every day you’re fucking up your corpus. This – it’s kind of incompatible with modern software development. I have many thoughts and feelings on AI ops. God bless them, but mostly I think that what they’re trying to do is take the humans out of the driver’s seat. And I understand why, because the C-level, CTOs, CIOs, whatever – there’s something that kind of blew my mind when I realized it a couple years ago: They trust vendors over their own people. Their employees come and go; vendors are forever. So, whenever a vendor comes along it’s like you give them tens of millions of dollars; your people don’t have to hold all of the wisdom in their heads. You can replace them. They’re fungible. The tool will tell you; the tool will be the source of truth.

    And I just fundamentally reject – I’ve never seen that work well. I think it’s a shitty way to treat your people. And I would prefer to take the –

    Chase: And I would not agree with that statement, quite honestly. I’ve had leads that are –

    [Crosstalk]

    Shimel: All right. “Mr. Vice President, I’m talking.”

    [Laughter]

    Sorry.

    [Crosstalk]

    Majors: – and then using our tools to make them better, to impress them, to give them a better sense of where they sit in time and space in the history and what’s happened before and what they’re – they should be wearing grooves in the system as they wear – as they use it in the [inaudible].

    So, I think that it should – if you’re centering a person it’s observability and if you’re centering the – it’s AI ops. Sorry. Go ahead, Kurt.

    Shimel: It’s okay. Kurt, your turn.

    Chase: No, I was just – in my experience the leads, I’ve had – I’m a senior level director now; it was the same at Splunk – I did not encounter that where my leads trusted vendors more than myself or the people on my team. I was surprised to hear that, quite honestly, Charity. I have not experienced that.

    [Crosstalk]

    Majors: It’s not universal. But it’s – there – it exists quite a lot.

    Chase: Yeah, that’s too bad to hear that. That is seriously wrong and that’s going to go against everything we need to accomplish internally.

    Majors: Yeah.

    Chase: Yeah, if –

    Ashley: Have fun attracting great people with that mental model.

    Chase: Yeah, yeah, yeah. And the other thing – I’d love to hear your opinion on this, Charity – we have very complicated systems we’re running. The source code management, the build systems, the agents. I’ve found that it always is very helpful to distill it down into discrete parts, try not to blow the ocean at once. Let’s start with the source code management system. How are we observing that? What are we doing here? And would you suggest that’s a great way for people to get going? I mean, when you try and look at everything – you have to look at everything for sure, but to get started it seems like find some of the most important pieces. Source code management is a great one. The crown jewels. What’s your suggestion there for getting started and –?

    Majors: Source code management is important but it’s never – it’s almost – it shouldn’t almost ever be in the critical production path. And when people are rolling something out I always advise them to find something that’s – don’t waste your time instrumenting _____ observability, lights, and backwater to the system. Find the thing that you’re trying to understand and fix now. Find the thing that’s broken. Find the thing that’s causing you pain. Because it should be an order of magnitude better at answering your question, observability should, even than monitoring. I don’t think you can look someone in the eye and ask them to switch up what they’re doing and to use something else unless you can tell them it’s going to be an order of magnitude better than what they’ve got now because the costs of transition are so high.

    But with observability you should be able to go from instead of just the aggregate where it’s like “Well, I see a spike but I can’t tell you what it means or how they’re different from –” with observability you should be able to go exactly those rows, in which ways are those events different from all the baseline events, and very quickly go “Oh, I have a much richer view of my system now.” And if – once you’ve got that you should be able to explain way more about what’s going – so, we see this all the time when people are running out – when they’re rolling out Honeycomb. They start rolling – they roll it out to the service and they’re like “Oh, God, there are problems there. Did you know that this is broken? Oh, there –” and we’re like “Yeah, yeah, yeah, but you’re just picking up the rug. It’s been there – it’s been that way for as long as you know.” And they’ll pick up another service and the problems – I’m like “Yeah, but these problems have existed – you just had no idea because we were dealing with [inaudible].”

    So, I feel like the way to pick up momentum, the way to incentivize people to get excited about this is to look for the biggest pain point in your system and –

    Chase: That’s awesome.

    Majors: – then patch that with observability.

    Chase: That’s great advice. I agree.

    Manuel: Charity, a question. Does – can observability actually take in account process? Because you’re talking about if there’s something wrong in the system itself, but sometimes the thing that’s wrong with the system is the process.

    [Crosstalk]

    Majors: Well, everybody is not about process. And you’re totally right. That’s often – that is not something that you should ever ignore. A lot of the biggest problems are process problems. But observability is not the tool for that.

    Shimel: And that’s good to know too because quite frankly – and Mitchell knows this too – we live in a silver bullet world where people think the latest shiny trinket is the cure-all.

    [Crosstalk]

    Majors: – everything, you should –

    [Crosstalk]

    Shimel: No, observability, yeah, it’s not going to cure Covid either.

    Majors: No.

    Shimel: But –

    Chase: And that was the point I was trying to make initially. I think I said it wrong. It’s not that monitoring and reporting, logging aren’t foundational; they’re complementary. You kind of need them all. If you’re running modern stacks – I said that incorrectly at the start – it’s not foundational. There’s – you need them alongside each other.

    Shimel: They’re adjacent. They’re not –

    Chase: Yeah, absolutely. Absolutely.

    Shimel: They’re not – it’s not a pyramid per se.

    Chase: Right. Exactly.

    Shimel: I think it’s part of your stack horizontally.

    Chase: Yes. Yes. And I’ve found –

    Shimel: Can I – we haven’t’ – go ahead, Kurt, I’m sorry.

    Chase: I was going to say I’ve found with especially new individuals and new people to the DevOps team and the DevOps world, getting them involved in the monitoring and saying, “Hey, this is how we start monitoring,” that’s a great place to start. It’s almost entry level for observability, if you – I’d love your opinion on that, Charity, too.

    Majors: Well, I really think that everyone who writes and ships software should be on call for their code, no matter how –

    [Laughter]

    Chase: Yeah, me too.

    [Crosstalk]

    Majors: Software ownership. And so, yes, that’s very empowering, I think, to show them how to find the trail, the alert that’s paging them, trace it all the way to the system and fix it. That’s something that as engineers, we feed on that. That’s exciting to get to know how to run your ship better. Yes, absolutely.

    Shimel: Well, but – so, you know what? Mitchell and I started a company – or, were two cofounders of a company called StillSecure 20 years ago. And that was one of the things. Mitchell ran the engineering team and that was one of the things he instituted, I remember, was that every developer had to take a turn on the help support desk, level one, answer the phone. “Hello, Mr. Customer.”

    Majors: It builds empathy.

    Shimel: And try to explain it because they had to answer the alert. And even when they weren’t on that level one our escalation procedure was if it went to level two or three it was the developer who did the level two and three support.

    Majors: I will say that, yes, and I do think that every engineer should be on call for their work, but you have to couple that with it’s management’s job to make sure that that doesn’t suck, that it isn’t _____ for them, but it doesn’t just kill you with [inaudible]. It is management’s absolute fucking responsibility to make sure that they’re given the time to – away from project work to fix these problems so that they don’t recur, so that they get enough sleep. I think it’s reasonable to ask any engineer to be woken up two or three times a year for their code. That’s it. Because more than that, you’re burning people out hardcore.

    Manuel: Been there, done that.

    Shimel: You know, Charity, Ravyn –

    [Crosstalk]

    M : I have a question. I have a question I’d love to ask all of you. I’m thinking about observability has been elevated because of, in part, the world that we’re living in. Software architecture is changing. We’re dealing with microservices, all these much smaller things that are interconnected to make software. And at the same time as we’re changing the pace at which we release software, not only is it unknown unknowns, it’s like a river – Riven –

    Shimel: Ravyn.

    Ashley: – we’re in the river of the changing software. So, you’re trying to understand all of these things at the same time. It seems like it’s the right time for observability to be the thing to be working on because ops has to be elevated to the same level of the same kind of architecture and processes that we’re using across the – in the life cycle of software. Do you agree with that or not? It seems to me we’re in the right place, right time.

    Majors: Absolutely. I mean, I feel like the first wave of DevOps transformation was all about “Ops people, learn to write code.” And we’re all like “Eh, okay.” But I feel like the pendulum has swung. I think now it’s about “Okay, software engineers, your turn.”

    Ashley: You’ve got to do ops.

    Majors: I have to learn to write operable services and to learn to maintain them and get called for them and – because what microservices has done, all of that complexity and logic has to be bundled up in the app. The app.

    Manuel: Exactly.

    Majors: And now it’s like we’ve been thrown to the four directions of the map. And what that means is that so much of understanding and operating your application has been pushed into the realm of operations and systems. It used to be that if all else failed you could attach a debugger to your app and just step through, see what was happening. Well, you can’t do that anymore now because the process died and jumped to another service, and another service. And part – the way that we instrument for observability is all about packing up the context of the application and shipping it along with it from hot to hot, one event per request per service, so that you can slice and dice and reason about your systems in a way that you used to be able to.

    Ashley: Doing that really changes the design. I’m thinking about Alan’s story of what I did back at StillSecure. What prompted that was when a customer called in and said, “I’m sitting in – we’re using your project and this dialogue box popped up and it says ‘An unknown error occurred’ and that’s all it says. What do I do?” Well, nobody knows what to do with that. So, we all just said, “Look, everybody does support in that case. Everybody does ops.” And that changed: “Okay, we have to figure this out.” And suddenly that was the problem of “How do we make this easier to support and easier to resolve issues with?” So, it does affect behavior and design or product.

    Majors: Yeah, absolutely.

    Manuel: I think microservices is – will actually do good for observability because if you do microservices correctly then each service is atomic. It does one thing and one thing only. It does that thing good. So, if the unknown is not taken out of it because you know exactly what this thing is supposed to do –

    [Crosstalk]

    Majors: Oh, no, honey, the unknown is never taken out of it. The unknown is never –

    [Crosstalk]

    Manuel: In my ideal world it is.

    [Laughter]

    There are no more unknowns –

    Shimel: In Ravyn’s world, yeah.

    Manuel: – in Ravyn’s world. Exactly.

    Shimel: Yeah. But I’m going to tell you something. The flipside to this, though, too with microservices, and I think we’re starting to see this now, is can you have too many microservices running? Does it – is the complexity load in there such that observability almost becomes meaningless or impossible?

    [Crosstalk]

    Majors: I don’t know that complexity can every really be created – no, it typically is not meaningless. It’s your only hope to understand this shit if that’s what you’ve done to yourself. But complexity can only be moved around. It’s just a question of tradeoffs, like which set of tradeoffs – some folks have gone whole hog on microservices and then they’ve found the edges to those design _____. But –

    [Crosstalk]

    Chase: What I find – I was going to say what I find challenging is both at Splunk – Autodesk was interesting. So, to your point, Alan, we used to make engineers buy bagels. You break things; you have to buy bagels. And so, we had a lot of bagels brought in. But both at Splunk and here at Tricentis we have a world where we’re supporting large enterprise applications and delivery of enterprise apps. We’re also – obviously all the microservices – and trying to apply these techniques across the board is very interesting, very challenging.

    [Crosstalk]

    Majors: Oh, yeah. There’s no one-size-fits-all.

    Chase: There’s not. There’s not. And almost to the point you made, Ravyn, observability almost has to be customized for what you’re observing. It’s – I haven’t had a chance to run Honeycomb yet, but for me my experience has been there’s always those little idiosyncrasies about how we run our ops, how we –

    [Crosstalk]

    Majors: There’s no such thing as a perfect _____. No one else can do it for you. It can be – it can get you started. It can give you a boost. But at the end of the day somebody’s got to fucking understand your software and roll that into your instrumentation. Nobody else can do that for you, let alone _____.

    Chase: And that’s where with AI ops, I won’t say it’s complete shit. I think there is some value there. Some of it is just about automating common occurrences. And maybe that’s not valuable but –

    [Crosstalk]

    Majors: – it’s an AI but they’re just using that to make – to raise lots of _____.

    Chase: Oh, sure. Okay. Yeah.

    Manuel: Didn’t work.

    Majors: Like, “Oh, I have an AI too.”

    Shimel: No, but we – and you’re right, AI is a money term. But ML ops is something that’s caught on a bit, more than, I think, than AI ops. And I think that’s really at the heart of it, which was to recognize repeatable patterns or recognize patterns that we can respond to.

    [Crosstalk]

    Majors: Although, I would still challenge – I think in far too many cases people are reaching for the complicated solution instead of fixing it at the start. They’re like “Ooh, I’ve got 50 bazillion e-mails so I – and all of that disk space so I need AI to solve it” instead of turning off the fucking e-mails and setting up a different kind of alert or some automatic _____. Just – is this really a problem we need to solve? Or could we solve it better by just not having it?

    Shimel: There is that.

    Majors: Sorry. That’s my last bitch.

    Shimel: Do we need to solve every – and I think Ravyn – or, not Ravyn – Charity, you were getting at it before. Do we need to solve every problem?

    Majors: Is it the most important problem? Is this going to be the biggest bang for our buck?

    Shimel: Yep. I mean, we used to have a –

    [Crosstalk]

    Majors: – or the _____. Like, senior engineers write software but very senior engineers don’t need software.

    Shimel: Yeah, no, I mean, look, when Mitchell and I were doing the security company we had a product, “Vulnerability Assessment of Management.” VAMP. Internally we called it the Bad News Generator because that’s what it was.

    [Laughter]

    It was the Bad News Generator. And the problem was we would go to an organization like Ravyn’s and it would pop up a few thousand vulnerabilities, and the security or the ops people would be like “What am I going to do?” And “Don’t worry about 95 percent of them. Let’s just worry about the top five percent.” And it was like – it was employment insurance, was what it was, because there would always be more vulnerabilities for us to find next year.

    But that being said, it really was the case where you’re not going to fix everything. We don’t have the time to fix everything even if we wanted to.

    [Crosstalk]

    Majors: And it’s not really doing a great job of giving you differentiation, letting you know what’s really important and what’s just noise. So much of these tools, they just give you so much noise and it’s almost worse than nothing at all.

    Shimel: Well, that’s why with the Bad News Generator people used to – so, Mitchell got the lucky part of designing it. I had to go sell it. That was much harder.

    Ashley: I remember the –

    Shimel: Much harder.

    Ashley: – guy from the federal government called and said, “This thing is spamming everybody in the Defense Department.”

    Shimel: Yes, “We have all these vulnerabilities.”

    Ashley: “How do I turn it off?”

    Shimel: Because we had this automated workflow before we had terms like AI and ML. But it is hard.

    Hey, guys, we’re coming down the home stretch here. We probably only have five minutes or so left. I wanted to – let’s look forward now. I’m big into looking forward these days. I’m hoping we’re going to do conferences in person again; we’re going to – I don’t know if we’ll go back to our offices but…

    What’s next for observability? How does this continue to evolve?

    Majors: What’s next is I think really wrapping traces in as a first, best tool so that – right now too many people are like “I have one tool for metric, one tool for logs, one tool for servability, another tool for –” they’re spending just ____ data again, every time. And it’s worse than that because you can’t actually seamlessly break down a chain of events, so you’re just copy/pasting this idea into that tool or hoping despite – because it seems to line up timewise, it’s actually bees in the logs – you should be paying to store this data once and you should be able to flip back and forth between loads. Are you slicing and dicing? Are you doing aggregations at read time? Are you zooming in? Are you zooming out? Or are you slipping back so you can view it as a logger file by time?

    I think that observability tools like us and Lightstep – which are the only two observability tools out there by my technical definition – although, I will say I do think that this might be the year that that begins to change. I know that so many of the big players have been working as fast as they can for years on the back end. They’re trying to catch up to where we sit technically faster than we can get catch up to where they sit in terms of the business. So…

    Chase: And I –

    [Crosstalk]

    Manuel: I like that you said that. Sorry, Kurt. That –

    Chase: No, no, go ahead, Ravyn.

    Manuel: – Charity – I just wanted to say to Charity –

    Chase: No, no, Ravyn, go ahead.

    Manuel: – about tracing, because that’s actually the solution that I am actually looking for, that I’m looking for –

    [Crosstalk]

    Majors: It’s got a lot of rough edges.

    Manuel: – for organizations. Yes, to be able to follow –

    [Crosstalk]

    Majors: But there are lots of problems we can’t solve any other way.

    Manuel: Exactly. So, I feel good that I’m doing it inherently, like I know what I’m supposed to be doing.

    [Crosstalk]

    Majors: You’re on the right path.

    Manuel: I think I’ve actually found a way to do that. But yeah, tracing I also think is the way to go. That is the only way you’re going to be able to know, because it’s taking you from point A to point B to point C to point D and you’ll be able to see that all. And there will be no unknowns. [Laughs]

    Majors: Oh, honey. You keep saying that.

    [Crosstalk]

    Manuel: It’s my dream. That’s – let me live my little dream.

    [Laughter]

    Shimel: Kurt, what do you got?

    Chase: Yeah, well, where mind – excuse me – where my mind goes to in the future is what are we doing in the education system to really disseminate this information, how monitoring software is developed, how we monitor it. And then also “observability” is such a loaded term because I know at Splunk they would consider observability something much different, for running a SOC and things like that.

    [Crosstalk]

    Majors: Oh, no.

    Chase: So, I think it – I mean, at its core observability is what it is. But how it’s implemented for different disciplines and how they use it, I think companies would definitely take exceptions with some of the things we’ve said here –

    Majors: For sure.

    Chase: – and what observability means to them. And I would just say keep that in mind, that it is a huge field and I think we’re going to see further improvements. I love the tracing. And being able to have that in your tool set would be fantastic.

    Shimel: Charity, final thoughts before we wrap?

    Majors: Oh, I am just enjoying the fact that some people are so interested in this that they’ll show up and argue about it all the time. So…

    [Laughter]

    Shimel: It’s all good. Hey, you know what? It’s a living. Mitchell, as usual, I’m going to give you the last word.

    Ashley: Well, I’ve learned two things today. Don’t call my tool an observability tool or an AI tool unless it really meets the definition, or I’m going to get crucified.

    [Laughter]

    No. No, what – I mean, I love this conversation. I learned a ton. I always do when we have these – what’s so cool to me is that we’re talking about operations and we’re talking about this part of our world and we’re not talking about better ticketing systems. We’re talking about really meaty, interesting things that help us do our jobs better, do a better job for the company, for the customer. That’s exciting to me. It’s just really cool. I enjoy it very much.

    Shimel: Join us for our next show where we’ll be talking about better ticketing systems.

    [Laughter]

    No.

    Chase: And what argument did I miss? I was –

    [Crosstalk]

    Shimel: No, no, no, no. It’s all good. You know what? I enjoy having really bright, smart people who are willing to express their opinions. And when I knew we had this panel, it was the lineup, I knew there was going to be no shortage of opinions, nor any backing down. So –

    Ashley: We’re going to call the next one “The observability mix-up.”

    Shimel: Right.

    Majors: We’re going to have to call it “Observability AI.”

    Chase: I still want to see that sticker –

     Shimel: “AI Observability.” All right. Hey –

    Chase: We need to see the sticker Charity was holding up. We need to see that.

    Majors: Oh.

    Shimel: Oh, we never got to that. Yes.

    Chase: We have to see that.

    Majors: I’ve got a few of those. I’ve got “Release := deploy.”

    Shimel: You’ve got a lot of them here.

    Majors: I’ve got “Ship dangerously.”

    Chase: Yeah, I love that one.

    Majors: “Deploy on Fridays.”

    Shimel: You know what? You used to be able to get these things on conferences. Now we’re reduced to showing them on Zoom.

    Majors: Yeah, you have to.

    Chase: Yeah. Exactly.

    [Crosstalk]

    Majors: “Perhaps this is a good day to deploy.”

    Chase: Maybe.

    Shimel: Okay, yeah. Perhaps.

    Chase: It’s Friday. It’s not a good day.

    Manuel: No, Friday is not a good day.

    Chase: That’s the one, right there.

    Shimel: And there it is: “I deploy from Cron.”

    Ashley: That’s the one to end on right there.

    Shimel: That’s is. We’re going to wrap it. Hey, you’ve watched devopsy.kinsta.cloud. It’s sponsored by Tricentis. Many thanks to them for sponsoring. Just wanted to quickly mention Kurt is from Tricentis. Charity is from Honeycomb, and it’s Honeycomb.io. Go check that out.

    Majors: We have a very generous ____ here.

    Shimel: A very free _____. Ravyn, you can start playing with it now and see if it’s something that gives you tracing and so forth. Mitchell, thanks for joining us.

    Ashley: You bet.

    Shimel: We are back – actually, we are back with a roundtable next, which will be open to the public. Looking forward to having you on for that. But until then, this is Alan Shimel for DevOps Unbound. Have a great day, everyone. We’ll see you soon.

    [End of Audio]

  • JFrog, Vdoo Securing SecOps

    JFrog, Vdoo Securing SecOps

    Earlier this summer, JFrog acquired Vdoo to deliver end-to-end continuous security from development to device—is this what DevSecOps looks like? JFrog CEO Shlomi Ben Hami and Natenel Davidi, CEO of Vdoo, speak with Alan Shimel about what the acquisition means and how this will change the way organizations approach DevSecOps. The video is below, followed by a transcript.

    Announcer: This is Digital Anarchist.

    Alan Shimel: Hey, everyone. Thanks for joining us here on a special segment of Tech Strong TV. I am really happy to be joined by two gentlemen, one coming to us all the way from Israel, Netanel Davidi from a company called Vdoo, and for those who don’t know, Vdoo is a Hebrew word meaning we understand. What?

    Netanel Davidi: Making sure.

    Shimel: Making sure. And then my second guest needs no introduction to our audience, it’s my friend, Shlomi Ben Hami. Shlomi, of course, is CEO and one of the Co-founders of JFrog, and he’s coming to us, I’m gonna assume, from the swamp in Silicon Valley today?

    Shlomi Ben Haim: You are correct. Great to see you again.

    Shimel: Shlomi, welcome. Good to see you, my friend. It looks like you’ve been in the sun a little—a little tanned. I’m glad to see they’re letting you out a little bit.

    But guys, we’ve got some big news to break today. Nati, you’re a new person—okay, I was gonna say, who wants to break the big news, but up to you, Shlomi.

    Ben Haim: Yes, I think I’ll go first and obviously, Nati—

    Davidi: Okay.

    Ben Haim: – will have far more information to share with all of you. Nati Davidi and the Vdoo team are joining JFrog. We are announcing the acquisition of this wonderful company that will complete our SecOps solution, not just for the DevOps and SecOps pipeline, but all the way to the edge, whatever the edge is, whether it’s a device, a data center, or any other destination.

    And Alan, you’ve heard me so many times saying that DevOps without security is half-baked and crippled, so the solution that we are now building together will take us another leap forward toward our vision of liquid software, and becoming the company behind all the software updates in the world. So ,we are obviously very, very excited to join Nati and the team to the JFrog family.

    Shimel: Congratulations. Thank you, Shlomi. Nati, as we say here in Boca Raton, mazel tov to you and the whole Vdoo team. So, tell our audience—and our audience, Nati, you should know is a very cyber savvy audince. Let’s hear a little bit about the Vdoo story. We know JFrog, but give us a little of the Vdoo story.

    Davidi: Sure. So, it all started almost four years ago when we were an initial team of people coming from endpoint security and a space of software vulnerability expert mitigation. Our previous company, we dealt with it as part of Palo Alto Network, they acquired my previous company, Cyvera, and another part of the group coming from embedded security and the space of reverse engineering and vulnerability research. And we joined together to cope with what we believed three or four years ago to be the biggest problem today in the security space, which is the connected device and embedded systems.

    We call it that way more technically because, clearly, there are very many other good companies out there doing more network security for IoT. We claimed and are still claiming that you need to take care of the source of the problem, which are the manufacturers and the creators of such devices, of such embedded systems, and for that, we established Vdoo.

    So, we did initially, we focused on that, and very quickly, we came up with a platform for product security, focused specifically on embedded systems and IoT, that allows developers and product security stakeholders to analyze automatically any kind of connected device. But when I say analyze, it’s not from the network point of view, but rather analyze the software stack in order to identify any security issue, any of the main buckets like third party issues, issues with your own code, the first party proprietary cloud that you are writing, issues with whatever in between, productization issues, configuration issues, hardening issues, architectural issues, things around cryptography, around how you keep your keys, how you configure things.

    So, we said we want to be the first player that allows everything, truly everything. Look at all of the aspects of security clearly, of course, supply chains, threats within it, and say to the manufacturers of devices, “Take this, and this will be your product security officer in a box, and a product security team in a box.” And it worked well.

    Now, the interesting thing is that, in order to do it, we had only one applicable approach—analyzing binaries. Why? Because in the embedded space, everything is around binaries. You have images of real time operating systems, third party compiled binaries and files and packages that you cannot really see into them what’s going on. You get them in the form of drivers from Qualcomm or AUTOSAR, whatever known library, and the manufacturers need to look into those, and they don’t have the source code. They have to be able to analyze them.

    So, we focused on binaries, and with that, again, we came with the platform and very quickly without even us knowing, our customers started to use us for purposes other than embedded security analysis. They started to analyze web applications, mobile applications, containers. Because when dealing with embedded, which is considered to be the most complicated and advanced or difficult things to analyze automatically, doing the rest is usually more about gaining more data but not developing more technology.

    So, we adapted, and we allowed our customers to analyze the applications that are talking with the devices and the web services that are talking with the devices. And we added more and more capabilities, and by then, we said, “Wait, why focus in on embedded systems if, through the binary analysis approach, we can serve the entire industry, we can provide security for whatever software artifact is out there.”

    And the next thought was, we just need to find the right partner who will deal with binaries. [Laughter] And the answer was, there is one such pioneer, which is known to everyone, which is JFrog.

    So, I asked Shlomi kindly to meet, to consider collaboration, so we can enjoy the many thousands of customers and huge community of JFrog and tell them, “No, use us to analyze your repository on top of Xray. Xray brings great capabilities, we enhance it with more capabilities for zero day analysis, configuration analysis. So, let’s do it together.”

    And that’s how our discussion started and today, we see where it is going and we are very, very excited about it.

    Shimel: What a great story. So, did you get the space two floors from them in order to meet, or were you already there? [Laughter] I’m kidding, I’m kidding.

    Seriously, I mean, my background is—look, I cofounded a company that really pioneered what we used call NAC, right, Network Access Control, and we did it for the U.S. government, for the DoD, right? So, this was not embedded systems, these were every endpoint network device coming on via, you know, 2.1x, we would test configuration vulnerability, et cetera. I know what a tough job that is.

    Once you go into the world of embedded systems, though, it’s like—anything goes, right? As you said, you’ve got Qualcomm drivers, there so much black box stuff. Knowing what’s secure and not is—I can’t even imagine, you know, how you do that. But now, once you’re taking binaries and artifacts, you’re right, there is only one company for you to go to, you know, it’s this man’s company. So, what a great match.

    Shlomi, he mentioned, Nati mentioned Xray. Xray’s a good product. But it’s a very specific product, right? If you’re downloading an artifact from the repository, it’s gonna check that as you’re doing it. This, of course, opens a whole different world of checking downloads and components and so forth. What’s the plan?

    Ben Haim: So, I’ll first start by saying, Alan, that I’m sitting here thinking that this would be probably the only interview in the world that you will have two CEOs that are so excited about binaries. [Laughter]

    Shimel: Yeah. [Laughter]

    Ben Haim: I’m enjoying it.

    Shimel: I think you’re right—you’re right, you win that.

    Ben Haim: So, listen—when we started, we built Artifactory to be the universal repository for all types of artifacts. And very fast, we identified the software package, the binary, as the first level citizen in your DevOps pipeline. There is no other asset that is more important. And why? Not because JFrog thinks so, but because of the fact that this is what you bring from outside, this is what you build inside the organization. This is what you test, this is what you promote, this is what you deploy, and this is the incremental update that you want to see at the end of the day on the edge.

    Now, with this in mind and the liquid software vision to be the company that changed the way software is being updated in the world, it was clear to us that SecOps cannot end on securing your repository. That’s great, that’s a great value. Xray can give you the composition analysis, the dependencies graph, secure your binaries—that’s great.

    But what happens when the Product Manager needs to take a binary? This is where the trust is broken. And we can—you know, every SecOps company, and you met all of them and you know all of them, Alan—every SecOps company will tell you, “We want to kind of build the bridge between developers and security guys.” It will not happen because you don’t use the same currency.

    And when it comes to binaries, both Vdoo and JFrog speak the same language, identify the same pains, and take it very seriously, not just on the development side or then on the testing side or then for the Product Manager side—all the way, all the way to the update, all the way to the deployment to production. So, obviously, we were very excited about what Vdoo had to offer, and it was again, just like everything else in JFrog, a binary question—are we competing or buying?

    And I’m very happy and very pleased to get to this moment now, speaking with you, knowing that the Vdoo guys are joining JFrog. Now, why this is so important from a culture point of view—and Alan, you know me well. You know how culture and the synergy is also about the people. We are bringing approximately 100 people that thinks and program and build their products like hackers. They know how the threat looks like. We are joining them through our hundreds of people that think like developers. What are the chances that this will not bring a better solution, a better blend to the market?

    Shimel: Sure.

    Ben Haim: And with that, I’m taking the next leap forward to fulfill the liquid software vision.

    Shimel: Agreed, agreed. And you know, frankly, it couldn’t come at a better time, right? Because wherever you are in the world today, and look, I’ve been in security 25 years—never has security gotten the attention, real attention, everyone always gave lip service, but real attention now. Whether it’s the supply chain or the ransomware or the whatever the attack du jour is, we are recognizing that cyber, cyber security represents probably one of the most serious threats to our continued transformation, right?

    And we saw this past year, unfortunately, without digital transformation and the move to a connected world software, where would we be, right? We would be in a lot worse shape than we were. And cyber security is the biggest threat to this. We need—we need to stop taking and do more doing, right, when it comes to building more secure software, right? And if people are building software like Shlomi and the team say, liquid software, right, without versions, it’s gotta be secure. And not only, it has to be secure, people have to be confident that it’s secure, right? Because that’s the biggest inhibitor—that’s the inhibitor, the lack of confidence.

    Davidi: And you can see this coming clearly, finally, from the regulators starting to draft bills around it, and you see demand from the end users—not necessarily consumers, but everyone, hospitals, manufacturers, they start to telling, “you want to buy from you, I want your software to be secure.” This is a demand, this is a requirement for RFPs, for RFIs. We see it everywhere. So, this is going to be a boon, because so far, the cyber security space was highly focused on the after the fact security, on the enterprise, care about, take care of it from the very beginning all the way to the network. And this is the only—

    Shimel: No, this builds up material, right, Nati? I’m sure you’re familiar, right? And there’s EEO, the bills of material order. Look, if this thing—you know, why did it take so long? I don’t know, but if this thing catches on that you need to have a bill of material for every software out there, for every app out there, people will, you know—go ahead.

    Ben Haim: I will just add to it, Alan, it also makes sense the way it happened. Look at the digital transformation. We are now facing the second wave of it, and we started to realize, everyone, every organization started to realize that being fast is not good enough. But you have to be fast first ________.

    Shimel: Yeah.

    Ben Haim: And then to build the security solution around it. So, developers delivered. They started to be faster than the security team, and now, security is catching up. But if you look at 99.9 security companies, they are not selling value, they are selling fear. They are telling you what will happen, what will happen if you will not do that. And developers hate that. Developers want to build, they want to have the freedom of choice and to be faster. And the security tools that are not focusing on the right pain have to be sold as fear and not as value. And this is the change that JFrog and Vdoo will bring to the market. Putting ________ aside, we focus on the community pain, as we did with Artifactory, as we did with Xray, as we announced just three weeks ago at swampUP with JFrog distribution. How can you distribute if it’s not secure all the way to the edge?

    Shimel: Okay. We’re running low on time, but let me bring together a little—I need some nuts and bolts, here.

    So, the deal was announced on Tuesday, today’s Thursday. Is it closed yet, was it—without getting into, I know you’re not gonna discuss the particulars, but is the deal closed? If not, when is it supposed to close? When are the Vdoo people, when are these hackers joining the developers, chocolate and peanut butter? When are we gonna see it?

    Ben Haim: Yeah, we hear this question again and again, first from our team, what are the first signs that we would like to see and what are our other plans?

    So, obviously, the definitive agreement is signed, and the closing is supposed to be in the next few weeks. We already have both sides, teams standing ready to merge the products, and the first step will be merging the infrastructure, making sure that this wonderful research that Vdoo is leading with vulnerabilities and database is also merging to Xray so our users can start enjoying that. We are thinking about the product philosophy, the hybrid cloud and on prem—all cloud, all regions, cloud native ready, the unified solution that will be scalable for our users. We have over 6,000 customers that are waiting for something like that.

    So, obviously, we will take the first leap and merge the assets and the low hanging fruit into Xray. And then, during 2022, we will build a unified platform that will include Vdoo’s capabilities into the JFrog platform.

    Shimel: Excellent. Excellent. Nati, just a question for you. I mean, obviously, all the people at Vdoo will stay on at JFrog. For people who may be very familiar with JFrog but have not heard of Vdoo, what’s the Vdoo website?

    Davidi: It’s Vdoo, V-D-O-O dot com.

    Shimel: Excellent. Alright. You can go find out more about it there. As I said in the beginning, Nati—congratulations, mazel tov on…you know, I learned from a mentor once, it’s kinda like a boat. The best two days are the day you start the company and the day you sell your company, so congratulations on that, it’s a beautiful thing.

    Congratulations to you, my friend, Shlomi, right? This is a piece of the puzzle that was missing and it’s important, and it’s gonna be more important every day going forward, so—good stuff.

    Ben Haim: It’s important, it’s a very, very exciting market for us. It’s increasing our addressable markets, and more than everything, we’re bringing over almost 100 people that have the same DNA and the green blood just like JFrog.

    Shimel: Look, getting 100 security experts at a shot—that alone might be worth it, because they’re hard to come by, my friend. Alright, Nati, congratulations. Shlomi Ben Haim, CEO—thank you, as always, my friend.

    This is Alan Shimel, we’re on Tech Strong TV. We’re gonna take a break. We’ll be back in a moment.

    [End of Audio]

  • Shopify Gives Devs More Creative Freedom

    Shopify Gives Devs More Creative Freedom

    Shopify’s Unite 2021 conference was held on June 29, 2021, and the e-commerce platform had some big, developer-centric news to share. Alan Shimel spoke with Shopify’s Glen Coates about the developer-first business model that is giving developers the most opportunity to build their businesses with a new 0% revenue share model, and how developers can leverage as much creative freedom as they want, from low-code capabilities in Online Store 2.0 to “max-code” custom storefronts. The video is below, followed by a transcript of the conversation.

    Alan Shimel: Hi, everyone. Welcome to another TechStrong TV segment. My guest this segment is Glen Coates. Glen is with Shopify. Glen, welcome to TechStrong TV.

    Glen Coates: Thanks, Alan. It’s great to be here, man.

    Shimel: It’s great to have you on. Glen, no pressure, but I think you’re the first person from Shopify who has been on TechStrong TV.

    Coates: There you go. _____ [inaudible due to audio distortion] it’s going to be all up from here, like you’re starting at the right place.

    Shimel: We’re starting at a low bar.

    [Crosstalk]

    ‒ it only gets better. We’ll see. Anyway, Glen, I think our audience doesn’t need an introduction to Shopify. We’ve all bought things online through Shopify storefronts and so forth, but why don’t we talk a little bit about you, your background, your position with Shopify?

    Coates: Right. I’ve been at Shopify a couple of years now. Before Shopify, I founded a startup called Handshake, and Handshake was a wholesale-focused e-commerce platform, so very much like Shopify but not focused on direct-to-consumer selling, just on wholesale. Handshake was acquired by Shopify about two years ago, so I’ve been at Shopify since the acquisition. In my time in the last couple of years at Shopify, I initially started out focused on wholesale, obviously, given my background. I spent most of last year focused on Shopify’s checkout, which is, obviously, a pretty important part of Shopify, and then about six months ago, _____ _____ product for what is called Shopify Core.

    If you’re a merchant, Shopify Core is like what’s in the box when you buy Shopify. It’s the online store, it’s the checkout, and it’s the back office admin that lets you set up and run your business. If you’re a developer, Shopify Core is the core developer platform, so the APIs, the dev tools, the docs, all the things that let you build apps and themes and work on the Shopify platform. Yeah, and, obviously, it’s a big responsibility and a really exciting role to be able to help the world of entrepreneurs start and try their business ideas.

    Shimel: Absolutely. Glen, just to give a little color, how many different Shopify merchant stores are there, whether it’s sheer number, percentage of commerce, or what have you?

    Coates: Yeah. Today, Shopify powers almost two million merchants. I mean, obviously, the last year has been kind of an accelerant to businesses going online, and that’s both been new businesses, people trying businesses for the first time, and it has also been traditional businesses going digital more quickly than they might have otherwise. But yeah, today, Shopify powers almost two million merchants worldwide, and it wasn’t that long ago that that was only a million. Even at massive scale, even being the world’s largest commerce platform in terms of number of merchants, it’s still growing very quickly, which I think just goes to the pace at which the world of business is going online, and how many people want some help with that.

    Shimel: Got it. Excellent. Thanks for all that info. If you don’t mind, let’s turn to the topic of today’s conversation, and I’m going to let you explain it, if it’s okay.

    Coates: Sure. Next week, it is Shopify Unite, which is our annual conference. Over the years, Shopify Unite has been focused on a combination of merchant and developer topics, but this year we’re really doubling down on developers, and really making it a developer-centric conference. The reason for that is sort of going back to the question you asked me a second ago, is Shopify was founded in Canada, and, if you go back five, six, seven years, Shopify’s mission has always been to make commerce better for everyone. But, if you go back half a decade, it was mainly North American merchants mainly selling apparel, mainly direct-to-consumer, which that’s a big market but it’s not the world of commerce.

    Over the last few years, Shopify has expanded considerably into new industries, new countries, new ways of welling, so wholesale as well as retail, and so we’re now in a much more diverse world of commerce, different businesses doing different things. As we sort of march down the road and try to fulfill that mission of making commerce better for everyone, that everyone is becoming a lot more diverse as we expand. Shopify can’t build all the solutions for everyone, all those different niches. While we have a great core product that includes many of the out-of-the-box things that people need to start and run a digital business, it’s not everything. We need the help of the world of developers to help build those solutions for the markets for the spaces that they are experts in to complement what is in Shopify Core, and really deliver everything that merchants need to successfully start, run, and scale a business.

    This year, we’re really focused on developers and helping them get onboard with our platform. There are really three main legs to that, I would say. The first one is mainly technical. I think for your audience, some of those details are important, like Shopify really is expanding to be PlanetScale commerce infrastructure, really becoming the power lines, the plumbing, the electricity grid that runs the world of commerce. So, we’ve done some things to really scale up our infrastructure that if you’re a developer building on Shopify, you can be confident your stuff is going to run at global scale without you really having to do anything. Our checkout can handle seven times the load that it could before. We’re now at greater than 30,000 checkouts per minute on a single shop.

    It’s interesting what’s happening to the world of commerce, like the world of social selling has made, and the way algorithms reinforce themselves in terms of distribution to drive traffic towards things that are trending. It’s become more important than ever that a merchant who blows up, whether it’s because of them being re-shared by an influencer or whatever it is, that they can capture that demand in the moment. Traffic is basically getting more spiky on the Internet. When things trend, they trend fast, and so Shopify already was the highest throughput checkout on the Internet but it’s now 7x higher, so merchants should never worry about not being able to make the most of those moments when they’re trending becoming more and more important.

    Shimel: It’s hyperscale, is what it is, Glen. We’re seeing it not only in merchants and commerce, but across everything. The Internet is big business, it is worldwide business, and especially with COVID, right? If you haven’t built your infrastructure to hyperscale, you’ve got trouble.

    Coates: Right.

    Shimel: That’s what digital transformation is about. That’s what the world is demanding right now, hyperscale.

    Coates: I imagine the readers, viewers of your site are not strangers to the world of what it takes to scale websites, and I think a lot of what Shopify tries to do is say, “Well, listen. We’ve figured that out for you. You’re a merchant with an idea. You’re going to come up with a new shoe or a new candle,” or whatever it is. Entrepreneurship is a risky endeavor, not everyone succeeds, but, when you succeed, like we really want you to be able to just capture the whole thing. The checkout works at one checkout a minute when you’re just starting out, and then, on your biggest days, it goes to 30,000 and you just don’t have to do anything. It just works, which is great. Kind of piece number one of Unite this year is Shopify really is PlanetScale infrastructure for commerce, and we already were the leaders and we’ve kind of gone up another multiple from there. Point two is really like, “As a developer, what things can I build on your platform? What are the opportunities for me to innovate, like where can I do that, and why is that exciting?”

    We’ve made some major updates to our online store, to our themes platform, which is obviously every merchant basically runs an online store with a theme on it. There is a new themes platform that allows developers to properly separate basically data and presentation into different layers in a way that makes themes more modular and extensible and kind of upgradable in the future. We have this powerful new data infrastructure that allows merchants and developers to essentially build custom data models inside a shop, and not just be confined to the out-of-the-box data model that ships with Shopify. There is now a much more powerful customization framework for both theme developers and app developers to integrate custom content into themes without having to hack the theme by hand and try to take it off the upgrade path, which is really great for merchants. We have also recognized that the Shopify online store is based on a web templating language called Liquid, which is similar to the templating languages that all developers are basically familiar with.

    We are also announcing, if you want to, you choose your own stack. If you want to build your website using React or something else that’s not our out-of-the-box stack, we’re also making available a new platform codenamed Hydrogen that will allow developers to basically choose their own stack, build online stores using whatever technologies they want, but ‒ and this is the crucial part ‒ still leverage a global infrastructure and hosting in scale and, “I don’t have to think about it. It’s just going to work,” on the Shopify platform, which is really great and really unique. Last but not least, I mentioned earlier that I spent last year on checkout. We are opening up for the first time the ability for developers to build checkout apps. Checkout is the piece of Shopify that it really matters. It’s where the money changes hands. It has to scale, it has to be super reliable ‒

    Shimel: Super secure.

    Coates: Super secure, super trusted, right, and trusted not just in the technical security sense of the word, but even just in the emotional sense of the word, like when I’m a buyer and I’m buying from an online store, when I see this checkout, does it feel familiar? Does it feel like something I can rely on? Am I comfortable handing over my money? Checkout is really where, in the _____ journey, the emotions shift from at the top of the funnel it’s kind of, “Okay, your brand, it’s really different. It’s exciting. I want to see differentiation. I want to be challenged and see new things.” But, when you’re handing over your money, the emotional tone is different, right? You’re not looking for excitement, you’re looking for predictability, right?

    Shimel: Yeah.

    Coates: Historically, Shopify’s checkout has been less customizable than the actual storefront pages higher in the funnel, and so now we’re opening up a new extensions framework for checkout that allows developers to build in apps that securely extend the function of Shopify’s checkout without compromising on the trust, the scalability, the speed, and all those kind of baseline, crucial things.

    Shimel: Very cool.

    Coates: That’s really cool. The last thing I’ll mention is we’ve talked about, “Okay, as a developer, if I build on your platform, is it just going to scale to the world?” The answer is yes. “Okay, so then what am I going to build?” Okay, so there are a bunch of new programming surface areas that we’re opening up for people to build apps and themes into, and then the last piece is, “What’s in it for me? Where are the dollars? If I start building your platform, am I actually going to make any money as a developer?” One of the things we’ve announced is _____ economic opportunities for developers. One is we’re cutting the rev share on apps in the Shopify app store to zero for the first million dollars in revenue per year, and, after the first million dollars, we’ve actually cut our rev share from 20 percent, which was already one of the lower rev shares in the industry, down to 15 percent. We’re really trying to help developers who have an idea for an app or an idea for a theme get that off the ground, no rev share on that first million bucks, and we really just want to help developer entrepreneurs get going as easy as we try to help merchant entrepreneurs get going ‒

    [Crosstalk]

    Then, the second one is that the Shopify theme store has historically not been open for submissions from any developer who wants to build a theme. There has been a sort of a curated set of themes there until now, but we’re actually opening up the theme store, so developers who want to build a theme and sell it to merchants will now have that opportunity that they didn’t before, which is also a huge market that is new.

    Shimel: Look, I think emphasizing the developer and the developer interaction in Shopify is huge. Glen, this is all great. I didn’t want to stop you when you were talking, but of course we’ve pre-recorded this show and it’s now playing after Unite has already taken place, I believe, on the 29th.

    Coates: Right.

    Shimel: But, for people who maybe want to see a replay of it or maybe get the highlights, is there a place we could send them to check it out?

    Coates: Yeah, if they just go unite.shopify.com, they’ll be able to watch the keynote and then dive into the other sessions that were part of the event and dig into whatever they would like.

    Shimel: Fantastic. Glen, thanks for a great recap on what happened at Unite and what’s going on at Shopify. Please come back on and keep us posted. You were wrong. You set a high bar. It’s going to be tough for people to beat, so come in and please keep us up to date on what’s going on at Shopify, especially with this renewed focus on developers and development options on the platform.

    Coates: I appreciate your time, man. Thanks for having me.

    Shimel: All right. Glen Coates on Shopify, about Unite, the Shopify conference here on TechStrong TV. We’re going to take a break and we’ll be right back with another guest.

  • The Future of Cloud Testing

    The Future of Cloud Testing

    Roundtable 10: Cloud-native application development has helped organizations stay ahead by delivering faster time to market. The growth in cloud computing has also presented new testing challenges – traditional testing isn’t enough to address security holistically for cloud-native apps.

  • Live from JFrog Swamp Up 2021 – The Top 4 DevOps Headlines of 2025

    Live from JFrog Swamp Up 2021 – The Top 4 DevOps Headlines of 2025

    We’re reporting live from swampUP. If you want to join this great session and many others you can still grab your free pass here.

     

  • Live From JFrog SwampUP 2021 – CodeCraft – BMW’s Driver of ‘Digital Car’ Software Development

    Live From JFrog SwampUP 2021 – CodeCraft – BMW’s Driver of ‘Digital Car’ Software Development

    We’re reporting live from swampUP. If you want to join this great session and many others you can still grab your free pass here.

  • ADOC: Assessing DevOps Culture

    ADOC: Assessing DevOps Culture

    The Assessment of DevOps Capabilities (ADOC) is crowdsourced, vendor-neutral and designed for individuals, teams and organizations who want to baseline their current DevOps state, identify the next target state, gain insights into how to improve their organization and team capabilities and measure and accelerate continuous improvement during their DevOps journey.

    The assessment models five DevOps dimensions – the human aspects, process and frameworks, functional composition, intelligent automation and technology ecosystems – to help you gauge your DevOps maturity and identify areas for improvement. In this TechStrongTV interview, Alan Shimel talks with Garima Bajpai, Louise Cermak and Brendan O’Reilly about reshaping culture to get the most out of any DevOps journey. The video is below, followed by a complete transcript.

    Announcer: This is Digital Anarchist.

    Alan Shimel: Hi, everyone. Thanks for joining us on another segment here for TechStrong TV. This is a special segment panel for TechStrong that I’m gonna be conducting, here. I’m happy to have you join us. I’m also really happy to have three interesting people join us. Let me introduce you to them. And if I mispronounce people’s names, I apologize; we do the best we can here on TechStrong TV, right?

    But we’re gonna start with Garima, is it Bajpai? Fair? Good? Okay. Garima, why don’t you introduce yourself?

    Garima Bajpai: Thank you, Alan. I’m Garima Bajpai from Ottawa, Canada. I am the Founder for the DevOps Community of practice here in Canada, which has three chapters, Toronto, Edmonton, and Ottawa. I’m also the organizer for the DevOps Summit which happens in Canada November 8th and 9th, and I am Ambassador with DevOps Institute, as well as an Ambassador with the CD Foundation. So, today, we are going to have a very interesting dialogue on DevOps assessments. So, I can’t wait to get started, right?

    Shimel: Yeah, thank you, thank you. Thank you for all you’re doing for the DevOps community. Next up, let me introduce you to Brendan O’Reilly. Brendan, welcome—a little bit about yourself?

    Brendan O’Reilly: Sure. So, my company is called Daysha DevOps. We’ve been working with the DevOps Institute now for about five or six years, predominantly delivering their training. And I myself am a consultant, and like the other speakers here today, really looking forward and excited about what’s happening with ADOC, because I see a big opportunity for our customers to get deeper into their DevOps journeys, to progress those journeys.

    Shimel: Fantastic, and thank you. And then last, but certainly not least is, we have Louise Cermak—Louise, did I get that right?

    Louise Cermak: Almost, yeah.

    Shimel: Well, say it right, because you’re entitled to have your name said right, it’s my fault. So, please, how do we pronounce your name?

    Cermak: Thanks, Alan, it’s “Chermock.”

    Shimel: “Chermock,” okay.

    Cermak: Yeah, so, I’m Louise Cermak. I’m a Principal Consultant with Catapult CX. We build software ourselves, and we also consult in agile and DevOps and have been doing so for sort of 10, 15 years. And we’re excited for ADOC, because actually, having the combination of being engineers as well, we can really understand the capability and how to apply it.

    Shimel: Absolutely. Well, all three of you, thanks for joining us. So, a couple of you have mentioned ADOC, right, and this is something that our friends at the DevOps Institute announced, I guess it was last week or the week before now. And ADOC, for our audience who may not be familiar, stands for Assessment of DevOps Capabilities. And this is a new assessment model that the folks at DevOps Institute have been working on.

    Look, you know, I know they’ve been working on it for over a year already. It’s been primarily driven by Helen Beal, who is a—Helen is a very well-known DevOps person in the community, but she’s also the Founder of the DevOps Institute Ambassadors, and has been a DevOps consultant herself for many, many years, as long as I’m doing devopsy.kinsta.cloud.

    And then Helen was really helped with this by Eveline Oehrlich, who’s a DevOps Research Director at DevOps Institute, a former Forrester VP. And then I also know that she’s been helped a lot with people like the three of you, who are real world DevOps consultants who are helping organizations every single day with their DevOps transformations and capabilities.

    So, before we jump into ADOC, though, let’s take it up a notch and talk about DevOps assessments in general, right? Are they—why are they important? Why are they important, and if anyone wants to give us sort of a brief history of evolution or time around these assessments and how they’re used, let’s say, by consultants and/or end user organizations.

    Garima, I know you have a lot of experience, so if you don’t mind, I’m gonna ask you to kick it off, and then have the rest of the panel chime in.

    Bajpai: Right, and I actually term DevOps transformations as a journey towards remaking culture. So, it has been largely a complex journey, because of two factors, actually. The first factor is that the complexity brings along with, you know, a lot of focus on how you can upskill people, how can you actually transform your knowledge based and skill set to a journey which is basically based on experimentation, on learning, and learning from failures, right?

    And the second aspect is that there is no such literature available. You know, there is a lot of talk about DevOps transformation, but the literature is in the making. So, there is limited academic support as well as support from a best practices perspective how to do this transformation, right? So, these assessments come very—it is very crucial, because it will help you assess where you are, and every organization will have unique challenges, right?

    So, it is important to assess where you are in your journey and what you can do. You know, it’s not like a traditional transformation journey of three to five years where you are relying on a lot of targets ________ models which were already defined. It’s continuously evolving, right? So, you will have to have a view on what you can do today and how you can incrementally, in short cycles, transform.

    So, what happens in this process is that, you know, assessment becomes more crucial and essential, because there are discrete efforts. There are efforts in short cycles, and this assessment and measurement will help you course correct, will help you understand where you are heading to in your journey and what is needed as next steps to ensure that you incrementally build value, right?

    So, that’s why, I think, these assessments are extremely important. We will talk along this conversation how ADOC can actually help you on this journey as well as what are the major impediments and challenges organizations have and what should they look out for.

    Shimel: Fair.

    Cermak: I think, over the last few years where we’ve been helping organizations, what we’ve noticed is that many of them are trying to do their own assessments, and I’ve been into so many organizations where people have got Excel spreadsheets and they’re trying to measure how they’re performing.

    Because, you know, Garima’s right—there’s a lot of learning and experimentation. A lot of it is self-learning, which is what we encourage, right, from a DevOps culture perspective. But people wanna know that they’re going in the right direction, so they start with their own spreadsheets, fill them in, try and measure their capabilities. And a tool like ADOC helps organizations have a much more structured way of doing that, and obviously, it’s more scalable than an Excel spreadsheet, but people wanna know that they’re doing the right things.

    Shimel: Agreed.

    O’Reilly: Just a little bit about the history, perhaps Alan—you know, ADOC is brand new, but there was a progenitor in the form of DORA, if you recall.

    Shimel: Sure, sure.

    O’Reilly: Yeah, so, that was with Jez Humble and Nicole Forsgren. So, when they produced their State of DevOps report, it was a means by which organizations could form views about what good looked like, so you have your release and then so on.

    And it definitely raised people’s curiosity about—can we do this, can we measure? And of course, one of the principles with any DevOps transformation is to be data-driven. And so, the 180-odd questions that are included in the enterprise edition are very much about trying to capture and turn subjective opinion into something quantitative, a good first guess. And I think DORA has been successful, it’s now part of the Google organization, I believe, but this is a nice independent offer from the DevOps Institute.

    We’re really looking forward, here at Daysha, to taking that to our clients. I see some really interesting use cases. Some of the people that I’m talking to right now are just starting in a CTO role, and they want to baseline what they’re doing.

    Shimel: Absolutely.

    O’Reilly: So, they can look—you know, six months from now, they can say, “How much have we done?” They wanna get an understanding of what have they got in-house right now and then look at it again to see if they’ve been effective in those changes that Garima was—really well-made point by Garima—nice, short, sharp, Plan Do Check Act type changes that are gonna take place over the next six months.

    Shimel: I mean, for me [Cross talk]—

    Bajpai: Right, I mean, I also want to focus on two specific aspects of ADOC, which stands out. So, a lot of people have actually understood that you cannot remake culture without people-centricity, right? So, that’s a given. And we also understand that emerging technology can be a tool in your toolbox when you’re trying to kind of transition from X state to Y state. And a lot of focus has been placed on processes and frameworks.

    What ADOC has been standing out for me is, they have actually reflected to other different things, which are equally essential. The first thing is functional silos. And if you look at large enterprises, what they struggle with, like, what has been in the past is that we have large, functional organizations and units with their own business goals and objectives. And when these people are trying to [Audio skips] towards ________-centricity, towards DevOps capabilities. That’s a large challenge, because you know, it’s not easy to actually break down the functional silos, burn all the ships so that there is no, you know, way back. It has to be carefully done so that we have a possibility of course correction. That’s the first important point.

    The second important point which stands out for me is automation. And when I look at automation, I see that data driven approach towards DevOps and how that will ensure that we automate good processes, we automate the capabilities which are creating value, not only end to end, but also within the practices, right? So, that’s—those two things, I think, put together with people, technology, and process would actually make a kind of, a full house for, you know, doing this assessment in one way rather than looking at it in pockets.

    Shimel: Agreed. In my mind, you know, Brendan, you mentioned the DORA folks, and of course, Nicole has moved on from Google, she’s over at GitHub, and I don’t know if Jez is even with Google anymore, frankly. But really, with DORA, it wasn’t necessarily a consultant’s tool, though consultants used it.

    I think one of the interesting things with ADOC and what I know about how it came about is, it really was designed for consultants to use as part of their consulting practice to help you collectively help your clients get a baseline, a roadmap to show where you’re strong, where you’re weak and going forward so that that consultant can be more helpful, more impactful and, frankly, more efficient, which translates to less dollars, right, that clients waste in kind of circling around to figure out where the ground zero is.

    So, I would just point out that, now, that’s one use of ADOC is as a consultant’s tool. There’s also the community version for individual organizations and large enterprises can do it on their own, they don’t necessarily need a consultant. But it really was designed with that in mind, and I think it’s gonna be interesting to see, you know, DevOps especially, it was one of the knocks on DevOps, right? And I’m devopsy.kinsta.cloud, I’ve heard this from day one—it’s squishy, there’s no manifesto, like in agile. There’s no real definition. It’s, how do you put your finger on it? You know, it seems like a unicorn or a mythical beast or something.

    Well, with ADOC, a lesson I learned early in my career in technology was, everything can be measured, and we measure everything, right? And ADOC kinda gives us a little bit of that certainty, at least as opposed to where you are in DevOps. I’m wondering, have you guys actually had a chance yet to try ADOC out with real world clients and what some of the results or findings were?

    Cermak: Yeah, we’re doing that right now with a client and we found that it was much easier to prioritize the activities that they needed to do, so we could align the activities with their roadmap and make sur that they were incrementing in the right direction to address some of the challenges that they had with their own customers.

    So, we took a very customer-based approach to prioritization of the activities, and they could really see a difference in the way they were able to engage the customer, deliver the features much faster, get the features agreed properly and through improving their continuous delivery pipeline, for example, and their lead time to change would improve and stability and reliability. So, we were able to prioritize exactly the things that were important right then to them with their customer base.

    Shimel: That’s great. Anybody else?

    O’Reilly: Well, just to echo, I personally haven’t had experience with the product just yet, but I have had that experience of walking the corridors with a spreadsheet, meeting people, capturing data—yeah, and that’s a really slow, inefficient way to capture a picture that evolves very, very quickly. So, I’m excited to do it this way, which is just way more efficient and lower cost to the client.

    And I think in terms of the consulting add-on, it’s that experience of having been through several transformations that allows us to analyze the results once they’re produced. And then, exactly like Louis has pointed out—prioritize, right? So, what changes can we make that will fit in with the natural rhythm of this company if they’ve got, if they’re not yet delivering on a regular basis? So, that kinda stuff I think is where the consulting value add is.

    Shimel: Interesting.

    Cermak: And the other thing that’s quite nice is, you can get the marginal gains as well, because you don’t have to tackle everything at once. You can improve one metric and then go and do the next one and the next one and continually improve, which I think the teams that we’re working with have really appreciated that, because they can see an uptick quite quickly, they get motivated, they wanna change more, they want to get better, and they keep going and keep going and getting those marginal gains.

    Bajpai: Right, and this is actually an extremely good point, because it gives us a perspective and this is something which I actually have experienced myself. I went to experience ADOC myself and, you know, I wanted to bring my team in as well. But what has happened in this journey, I have realized that organizations have overestimated the potential of short-term gains. And when I say that, often, people do not realize that whatever transition or transformation from a DevOps perspective they are doing, they are looking at short-term gains and probably losing out on the long-term potential of, you know, larger, creating larger value, long-term value, right?

    So, this is also a reflection that when you take this kind of an assessment, you would potentially have two aspects—what you can do in short-term, and what you achieve through several initiatives put together incrementally and how that outcome would look like in long-term, right? So, that is where these assessments would become very crucial for organizations, at especially big organizations.

    Shimel: Agreed, agreed. With all three of you being involved in consulting, I’m gonna ask you to take off your consultant hats for a second and for end users, end user organizations out here watching this, is ADOC of use to them without a consultant?

    Cermak: Oh, definitely, yeah. You know, we do build software, and we use those metrics ourselves, and we’re constantly monitoring those dials and—okay, we do some consulting around this stuff as well, but actually, we’ve got customers that we need to deliver to, and for us, being on top of all of that all the time is really helpful.

    So, anyone taking the assessment would be able to identify changes that they could make in their own engineering. That’s straightforward. It’s really set out very easily to understand.

    Shimel: Very good.

    O’Reilly: Alan—

    Shimel: Go ahead, Brendan.

    O’Reilly: Yeah, so, there’s a friend of mine who’s a lawyer, and pretty much any time I ask him a question, the answer is, “It depends.” [Laughter] So, in answer to your question about whether end using organizations can benefit from this or that consultant, I would say it depends. It depends on the mindset in the organization, the appetite for learning, the appetite for change, and so forth.

    But for sure, even just to kind of get started and to understand where they’re at, there’s huge value here without a consultant just to get a picture of where you’re at.

    Shimel: Agreed. And I think that’s part of this, too, and I think that’s also where there’s the—I forget if they call it the enterprise edition, which is the full, I forget how many questions there are already, hundreds of questions, versus the community edition, which I think is just, like, 30 questions or something like that. But even the community edition, you know, I think brings value to an end user organization who may just want to—you know, let me stick a flag in the ground as a marker and then see where we go from there.

    Guys, we’re about out of time, but I want to thank you, all three of you, for joining us. I should mention that if you’re interested in ADOC, right, Assessment of DevOps Capabilities, you can find out a lot more about it at the DevOps Institute site, which is DevOpsInstitute.com, and I know that there’s licenses for consultants, end user organizations. We’ve also launched the membership association, so as a member of the association, I think you get some use of it as well.

    But for now—Louis, Brendan, Garima—thank you so much for joining us on TechStrong TV. Good luck, and keep us posted with ADOC.

    Cermak: Thanks, Alan—pleasure.

    Bajpai: Thank you.

    O’Reilly: Thanks, Alan.

    Shimel: Thank you. Alright, we’re gonna take a break and we’ll be right back with our next guest.

    [End of Audio]

  • EP 9: The Road to GitOps

    EP 9: The Road to GitOps

    GitOps takes DevOps best practices for development and applies them to infrastructure automation. In short, GitOps delivers a hermetic, immutable deployment. So, is GitOps replacing DevOps?

    Managing infrastructure as code through GitOps can be a game changer for organizations. But, when is a GitOps approach best used or not a fit? In this DevOps Unbound episode, I’m joined by Tracy Ragan of DeployHub, Parag Doshi of Tricentis, Brendan O’Leary of GitLab and Mitch Ashley of Accelerated Strategies Group to discuss these questions, the impact of GitOps on the CI/CD market and where the future of GitOps is heading in 2021.

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript

    Alan Shimel: Hello, everyone. This is Alan Shimel, CEO at MediaOps. And you’re watching another episode of DevOps Unbound. Let me introduce you to our panel for this week’s episode, and we’ll jump right into our subject matter. First of all, we have the newcomer to DevOps Unbound, but he’s not a newcomer to our audience, my good friend Brendan O’Leary, senior DevOps evangelist at GitLab, a governing board member for CNCF. Hey, Brendan. Welcome.

    Brendan O’Leary: Alan, nice to see you, as always, and happy to be here.

    Shimel: Thank you for being here, Brendan. Next up we have our good friend and a semi regular here on many of our video segments at MediaOps, Tracy Ragan. Tracy is CEO of DeployHub, as well as the community-elected board member of the CD Foundation. Hey, Tracy. Welcome.

    Tracy Ragan: Thank you. Thank you so much, Alan, for having me here today. I always love doing these.

    Shimel: Absolutely. We love having you. Our third panel member today comes to us from Tricentis, who’s also the sponsor for DevOps Unbound, and it’s Dr. Parag Doshi. Parag runs engineering, VP engineering at Tricentis. Parag, welcome.

    Parag Doshi: Hello, Alan, everyone. Nice to be here. Thanks for having me.

    Shimel: Thank you. Thanks for being on. And then last but not least, my cohost for DevOps Unbound, Mitch Ashley. Mitchell is the CEO/founder of Accelerated Strategies Group as well as CTO at MediaOps. Hey, Mitchell, thanks for coming on today.

    Mitch Ashley: Absolutely, Alan. And thank you for all of our panelists for joining us. You are all fantastic. We enjoy having you.

    Shimel: You know, just before we jump into it, the continuing evolution of COVID times, we’ve done so much with our backgrounds, [Laughter] on our videos, right? Looking here at Tracy’s snazzy DeployHub background and Mitchell’s background and Parag’s – and Brendan’s old-school, man. Brendan’s giving us – if I’m not mistaken, he used to be down in the basement and now he moved up. I think he displaced one of the kids or something.

    O’Leary: Yeah, I moved upstairs.

    Shimel: Yeah, yeah. And that’s a real background, so good for Brendan.

    Ashley: And he’s got an MV7 microphone there. Nice job there, Brendan, on the mic.

    O’Leary: Oh, thanks. Yeah.

    Shimel: Yeah.

    O’Leary: All, you know, post COVID things to make it work, Alan. I wanted to have the background, the real background to make it work. But that was also a COVID change.

    Shimel: It’s authentic. Yeah. [Laughter] Crazy stuff. Anyway, guys, today’s show is around one of the hottest topics in DevOps – GitOps. Right? And I’ve actually heard from people, you know, that GitOps is the new DevOps or GitOps is somehow replacing DevOps, and I think DevOps has outgrown the – it’s going to be replaced by something.

    It’s transcended to a different level of existence, if you will. But certainly GitOps is on a lot of people’s lips. It’s not only on their lips, it’s in their development plans and how they are delivering software to the cloud, to cloud native, and all over. But is the right move for every situation? Is it a panacea?

    When is GitOps a good thing to use? When is it maybe not a good thing to use? Right? How much does it change sort of the classic CI/CD pipeline that we’ve started to finally wrap our heads around in DevOps? Stuff like that. So I am going to – Brendan, of course, GitOps, GitLab, they both have Git in it.

    So we’re going to make you – I’d like you to start off. Let’s start with some definitions. What’s GitOps mean to you?

    O’Leary: That’s a great question, and I think one that’s still probably [Laughter] unsettled, right? Much like defining DevOps can sometimes be seen as unsettled. GitOps came out of the cloud native community, and so a lot of folks maybe restrict the concepts of GitOps to say it only exists in a Kubernetes world. Because that’s kind of where this approach came from. But we’ve seen folks employ GitOps principles to all kinds of different infrastructure, automation, from old-school VMs to containers to Kubernetes clusters as well.

    So when I think of GitOps, I think it kind of requires three things. One, infrastructure as code. Right? You said that in the beginning. That’s kind of like table stakes to getting in. It’s something we’ve been working on for a long time. But really defining as code in Git – the Git part of GitOps – what our infrastructure’s going to look like, and then keeping that configuration stored in that code.

    And maybe the desired state is stored as code or not, but at least the way we get to that desired state is. Second, I think it’s merge request. So we call it merge request. You’ll hear other folks call them pull requests. But the concept of using the point where someone is proposing a change as the central place where teams collaborate on.

    Is this the change we want to do? Is this going to help solve the problem we’re trying to solve for? Right? Bringing that concept to the world of infrastructure is hugely beneficial, because it’s how our developers have already been working for years and years. And making that merge request, kind of that central source of truth for approvals and security checks and all those other things is really critical.

    And then last, I think it’s – the third thing is CI/CD, right? Of course. So the idea of GitOps is automating that infrastructure, update with a very simple Git workflow. And so you really do have to have embraced and adopted continuous integration and continuous delivery really in order for the whole package of GitOps to make sense for you.

    Shimel: Pretty darn good. Comments, add-ons, thoughts? Parag, Tracy?

    Ragan: I would only add one thing here. If you say what is GitOps, I like to shorten it down. I think what Brendan just defined is all accurate, but it’s operations by pull request. The goal of GitOps, what GitOps delivers is a hermetic, airtight, immutable deployment. And that is in essence what we’re trying to accomplish by using Git as the place where we declare and define what we’re going to push out to or have pulled out to the cluster.

    So it’s the concept of a hermetic deployment, something airtight that no human touched. Don’t you dare touch that and then deploy it out, because of we don’t know what you did. [Laughter] So the whole idea is to create a hermetic process.

    Shimel: I love that. That was concise and right on.

    Doshi: I have a similar viewpoint. I think of GitOps as a combination of infrastructure as code plus event-driven architecture. Right? And in this case we’re talking about merge requests or pull requests or whatever terminology you want to use, depending on the flavor of Git you love. But the way I think of this as a former physicist is entropy is there. It’s too hard to keep track of everything. Humans can’t remember everything.

    We need to be able to describe, as application developers have been doing for the past couple decades, in code the versions of things, the documentation associated with everything. And then the automation that we’ve had in CI/CD, which people think of, oh okay, that’s developers pushing their code out, well there’s no reason why we have to stop at application level. It needs to be the whole thing, because application and infrastructure must stay in sync at all times. So by putting a description of both the application, the infrastructure, and within the surrounding environment, all three of those into Git, you’re bringing the best practices of AppDev into the whole world together so they’re all synchronized.

    Ashley: Yeah. I like that description too, Parag, because in a way that’s breaking down silos that we still have around the code build and deployment process. And it’s almost like if you think back when we started using code repositories and, okay, everybody put their source code in one place, okay, let’s expand that to artifacts and other things. Well, now we’re doing the same thing of let’s abstract at one more level and make the whole thing as cradle to grave, if you will, for getting code out into product. Whether it’s any part of that stack, it’s a GitOps process.

    So to your point, Tracy, about it’s an immutable object, it’s something that we know from beginning to end, it’s all been in one environment and we happen to have manual processes that have just been automated that potentially are weak or could be broken.

    Doshi: I’ll add one more comment on top of Branden’s about – I think it was around – it came out of cloud native computing foundation, but really the way I look at containers is it doesn’t matter that’s a container or VM. We’ve just moved a control plane up to a higher level, but the same principle’s apply. So there’s no reason why you cannot use these same principles in a container type of environment or a non-container environment. The only difference I would say with containers is we can be very surgical, fine grain, down to each individual pot. So there’s a little advantage with cloud native architectures to be very fine grained.

    O’Leary: Yeah, I agree. And I think that’s important for folks to separate the concept of GitOps, the principles that we’ve all talked about. And I agree with the other panelists and how they’ve talked about it. But separate those principles from the technology, right? Because it’s not always true that Kubernetes is going to be the right technology for you to deploy every workload you have in, but still this concept of GitOps.

    And I really liked Tracy’s sealed container – well, overloading the word container there. [Laughter] But that sealed deployment that doesn’t have a lot of human – or doesn’t have any human interaction. That is what gets us to what Mitch was talking about, which is we want to take this isolated, people working in isolation, the kind of cave-like nature that operations typically used to have where you had to have this specialized knowledge that one person knew. Oh, wait. How do we deploy that service?

    Oh, we’ve got to call up Alan. He’s the one that knows how to do that. [Laughter] Right? You want to get rid of that and make it all, what Tracy said, which is like a hermetically-sealed deployment that we know we can reproduce every time.

    Ragan: Yeah. So GitOps was a brilliant way to leverage existing tools and habits to create Chef and Puppet. That is basically what was done. I mean, it has a Puppet kind of master architecture. You have a GitOps operator that’s running on your cluster. It does a pull based on what’s in Git. It very much took the basic concept of Chef and Puppet and leveraged existing tools that people already had, some open source, GitOps operators out there, and they were off and running. And that is the brilliance and beauty of GitOps. I mean, I have a lot of issues with GitOps, and I’m sure we’ll share them, but its brilliance is in that simplicity.

    Shimel: The irony though, Tracy, is that many people will tell you that Kubernetes killed Chef and Puppet, right? And we’ve seen Chef get sold maybe for less than money raises in valuation previously.

    Ragan: No, I’m just talking about the basic concepts.

    Shimel: No, no, I get it. But if we look at the concepts behind what Chef and Puppet did, which was automate that deployment, deployment automation like that, and then with Kubernetes it sort of negated the need for that, but yet we kind of snuck that in the back door. Except we put it in Git to use in Kubernetes. So the irony of that is pretty profound, right?

    Ragan: Well, it’s all about the container.

    Ashley: Right, right.

    Shimel: Yeah. You’re right. Maybe it’s because Chef and Puppet – [Crosstalk]

    Ragan: The containers simplified, yes.

    Shimel: – didn’t figure out how to work in the container. Look, for that matter Jenkins too. Right? We went to Jenkins X because of all of that and everything else.

    Ashley: That’s a really great example, yeah.

    Shimel: But what I’m reminded is of the old Who song, right? “Meet the new boss, same as the old boss.”

    Ragan: Exactly.

    Shimel: And you still have that old boss. With that being said, here’s another thing, and Parag, I’m going to ask you to kick off as I take us a direction here, and that is GitOps, yes, it’s for deployment. What about testing? I said this is all about testing. We’re doing our testing in Git. Is that the – that stops, right? Is that better? Worse? Is that the way it should be, always, sometimes, never?

    Doshi: So I don’t know if I’d call it testing in Git, but I would say now Git becomes a source of truth. So I think what Tracy said is right, that it’s just that same sort of automation of infrastructure but now moved into making Git the centerpiece of where you define this. So the Git part is unchanged, but the testing is now event driven. Right? So because you’ve merged in code, you’ve committed code or gone to do a pull request, that can trigger testing.

    And we call this continuous testing. And I think the evolution of this can even go as far as our test definitions themselves should also be put into Git. That’s the next evolution that we’ll have. I don’t believe it’s really commonplace just yet, but when that happens then, we are trying to describe as much as possible into Git as a repository and use the principles of version control added with the event-driven concepts.

    So when a developer does X or when an operator is in environment Y and something needs to change, that whole Git workflow can invite continuous testing to say do I allow that to merge back in? If you want to have an approval, you could do those kinds of things. So to me, incorporating continuous testing, and the day that we put test definitions into Git as well, we’ve gone quite a bit further of completing this vision and really making DevOps real. Right? Because DevOps isn’t just releasing really fast, deploying really fast. It’s deploying fast with confidence and ease. That’s what we need.

    Shimel: Thoughts on that, panel?

    Ashley: It sure seems like a natural evolution to include the test environment. And as you say, Parag, it’s probably not commonplace yet today. A question for you about that; do you think that there’s incentive with the folks that are taking care of the testing, whether they be the QA team or developers writing tests, things like that? Do you think they have the right incentives to move it into Git or will it be something else that causes it to happen?

    Doshi: I think as long as people have the incentive of delivering value as quickly as possible with the best meantime to recover, the lowest defect leakage rate, as long as we do break down those silos, as you said. If the testing team is, okay, well, I don’t understand what the developers do, just throw it over the wall and then I do my test, that’s not going to work. This is talking about dev testers, the full stack developers. And as long as everybody’s motivation is right, that we’re trying to deliver value faster and faster, that we will not have confidence – that a VP will not say let my applications go out – you know, they’re only going to look at portfolios of apps and say what amount of testing has been done? What amount of security checks have been done?

    That’s another thing. The environment. Okay? That’s the biggest entropy of all. We have the least control over the environment. The same code, the same infrastructure in different environments behaves differently. Until we get that environment description also into Git, coupled with the testing model themselves that says here’s how we test it in that environment or that application or this infrastructure pattern, then DevOps becomes real. Until then, this is all talk.

    O’Leary: Yeah, I agree. I think it’s critical, and it’s something that GitOps, again, this concept of GitOps, I think is exciting to a lot of people that have been trying to make DevOps real for so long, because it enables that in a way. Right? We see kind of the light at the end of the tunnel, I would say. And if that is all brought into Git and all made into something that can deploy independent of any human interaction, well now you have the ability to do continuous testing. You have the ability to stand up an identical environment, the golden goose or [Laughter] golden egg of a, hey, we want an environment that’s exactly like production to really test it and really hit with security tests and know that when we do deploy to production we’re going to have that confidence.

    So if you’re able to continue to iterate on the environment, the infrastructure, just like you’re iterating on the code, it enables you to then say, okay, now we can not only have the code to review and maybe some static code analysis that happens and some testing that happens, and then maybe some human testing that happens, now we’ve also got an entire environment spun up automatically that mirrors production, where we can have a high level of confidence that it’s going to work when it really ships.

    Shimel: So guys, you’re telling me DevOps hasn’t been real quite yet. I feel like I’m living in the Matrix, kind of living in a simulation. Tracy, I sense you’re Agent Smith here, though. You have a different view. Maybe GitOps isn’t the panacea it is or it’s been projected here. What’s your view?

    Ragan: So no hate mail, okay? No hate mail. [Laughter]

    Because I know there’s a lot of enthusiasts out there, and rightfully so. But let me just level set the way I see where we’re headed and where GitOps can’t take us. GitOps is imperative. It’s scripted. And it works really well – so think about how most people use GitOps.

    You have a code repository that you keep all your source code in. You make a change to your source code. You check that in just the same as you always did. You also have an environment repository, and what lives in your environment repository is your Kubernetes deployment YAML file. And that is where you define all of the stuff that we’ve been talking about and that’s what the GitOps operator looks at and pulls across.

    Now if you’re in a Kubernetes environment and you have containerized your application – this works great, it really does – and you don’t have a whole lot of environments – you have three environments dev, test, and prod – because remember, in a Kubernetes YAML file there’s a couple of things that are important to the GitOps operator. One is the tag of the container. So once you register your container image you get a SHA, and that makes it a unique identifier. And you have to update your YAML script with that SHA.

    Now dev, test and prod may have different values, configuration values. So you have to branch it. So you branch dev, test, and prod. So now you have three versions of that YAML file to manage your container deployment. And that’s pretty doable for most organizations today, but it is still imperative. Everything’s being defined by the script for each of those environments.

    Now let’s go and expand it and scale it. Let’s scale it to a microservice environment. Every microservice has its own YAML file because it should be independently deployed. And they should be shared, because that’s the beauty of a service-oriented architecture. They should be shared across teams. So what you do is as an application team, you’re going to make your coding changes in your coding repository and then you’re going to have different YAML files for the different services.

    Now let’s say you’re a store. I use the candy store and hipster store and the clothing store. They all use the card service. Well, they’re all going to put their card service YAML file into their environment repository, and that’s going to get deployed to their own clusters and name spaces. So you have a scaling issue, and it’s because of the imperative nature of GitOps.

    We are starting to move and have more conversations about declarative and events and catalogs, where this stuff is going to be stored and generated at runtime. So for example, instead of having all of these scripts, you’re going to have central places to go to say what’s the key value pair for this environment? I’m going to pull this registry information that has the SHA, I’m going to go get my key value pairs, and that way you don’t have to have so many scripts and you’re not managing – it’s onerous.

    It really can become onerous. We have customers who have 200 microservice and 20 clusters. That’s a lot of scripts. And if you’re an SRE and you want to update your microservice and you want to provide it to everybody, you’re going to have to figure out every single application environment repository that has it in there for every single environment and make those corrections. So it doesn’t scale as well as it should. Now I know that their CNCF has a group, a working group, around GitOps, and they’re starting to think about GitOps version two, and they’re going to have to address these kinds of core problems around an imperative process.

    Shimel: Love it. Brendan?

    O’Leary: Yeah. I mean, I like that concept and I think you’re right, Tracy, that the ability to manage the scale is definitely something you have to be prepared for if you’re thinking about GitOps, because if you just take it at its face value you end up with all those scripts and it could make the problems you have worse rather than fixing some of the problems you hope it would come into and solve. And so I think in a cloud native environment, again, it maybe pairs a little bit more simply, but I think that’s why again it’s great to think about the concept of focusing on that developer-centric approach and the experience that they’re going to have when it comes to shipping something out, to your point of if I want to ship an update to my microservice that impacts everyone, I don’t want to have to go through 78 repositories and update that everywhere.

    And so having GitOps do the infrastructure management in the same version control system as application development is great, but you also have to have a plan in place for how does that scale to the scale that you’re really going to build to? If that the scale of developers, if that’s the scale of services, if that’s the scale of environments. And so I think it’s important to have tooling and an approach in place that accounts for that as well.

    Shimel: Parag, you were going to say something I think before we –

    Doshi: Yeah. Sure. So I love what Tracy described there, and getting into the specifics of the proliferation of YAML files, and especially as we break down microservices. So this is the whole concern with microservices. What you took as monolith and you described as one big black box, now you’ve broken it down. And so now we have the proliferation of all these things. Right?

    So I like things like Helm to help to reduce that and parameterize all these things. But fundamentally with GitOps, the concepts are there. Like I said, infrastructure as code with an event-driven type of approach version control. What I like about CNCF and containerization is the purpose of containers inherently – and Branden can keep me honest on this – is really to make the environment completely separated from your solution, your application, your infrastructure. Right? So today’s industry, the best approach we have is containers or serverless, which is also part of CNCF, so that we have things like Docker files and YAMLs that allow us to reduce the entropy of the environment as we can as much as possible and describe them declaratively in things like GAML files.

    And I do agree with Tracy 100%, that now it becomes, okay, well now I’ve got all these different configs, pulled that out of my code. Even with service mesh technology and CNCF we’re pulling out things like security and rate limiters between every single microservice. All that’s taken out of the logic, the business logic, on everybody doing security different ways. Now it becomes a configuration issue. How do I – if I want to change a configuration, do I need to change 100 different YAML files or config maps? And I think the technology’s evolving.

    Help is just one example. And then I’ll associate that with all the continuous delivery. You all know Spinnaker and a bunch of different technologies in this area, Octopus and so forth, that can help.

    Shimel: Yep. Mitch, did you want to say something? Otherwise, I want to – I’ve got another –

    Ashley: All I can add to the conversation is version control across all of this also gets very complex. [Laughter] I think you all –

    Shimel: But isn’t that what Git and Repositories are supposed to help us with anyway?

    Ashley: It is, but now you’re versioning all of your scripts along with all the environments and all of the variables of what you’re trying to script.

    Ragan: And it’s not relational.

    Ashley: Yeah.

    Ragan: You know, when you have – what’s the blast radius of my microservice? I’m going to push it out because my hipster store needs the new card service. If you’re using an SOA architecture that means that you have one card service and everybody’s using that version and that’s the best way to move forward because you eliminate the sprawl. If you put it all in its own namespace, you have multiple versions of the card service. So now you have a problem with the card service and you have to go update all the namespaces that’s using that card service.

    So you have to have a relational model to be able to track what applications or what namespaces are using what microservices. And unfortunately, Git is not relational. We create relations using tags, using different repositories, but that becomes redundant.

    Ashley: Great example.

    Shimel: Fair enough. So let me –

    Doshi: Tracy, you talked about the structure.

    Shimel: Oh, go ahead, Parag.

    Doshi: I was just going to follow up and ask a question to Tracy. I think what you’re saying there is Git is just a big version control repository. If we can provide some extra structure and relationships around all these artifacts that we’re storing there we might have better control.

    Ragan: And that’s what you call catalog.

    Doshi: Yeah.

    Ragan: Right? So I think we’re starting – when we really look at a service-oriented architecture and we look at – let’s say, for example – let’s push ourselves farther into the future. And Brendan brought this up around service mesh. And I believe this can happen and will happen. I believe service mesh will be enhanced on.

    Right now you can say do a deployment and deploy out to 10% of my users. I think service mesh will start looking at active directory and say deploy out to my developers. So now we have one cluster and we have immutable services out there. We put a new service out there, it gets requests routed to the developers, they do their testing on it, once it’s good it’s rerouted to their testing community.

    Then it’s routed to a certain group of their production community. Which means that we don’t have deployments across multiple clusters. We’re going to start pushing into a one big cluster, which sounds a lot like the mainframe. [Laughter] But we’ve come full circle. And as Alan said, the song, right? Meet the new boss, same as the old boss. [Laughter]

    Shimel: Right. We won’t get fooled again. That is true. There is certainly some truth to that. I wanted to spend a little time talking about the impact of GitOps on the CI/CD market. We’ve written in devopsy.kinsta.cloud several articles, the gist of which was DevOps is the symptom of disease. CI/CD is the cure, if you will. Right? And we’ve built this whole DevOps industry around CI/CD solutions.

    And I mean look, GitLab’s a great example. They had a great Git product to begin with. But it was their CI/CD solution that kind of moved them into that end-to-end DevOps, if you will. Right? GitOps I think has profoundly changed that market because it’s profoundly changed the way we do CI/CD. Well, that and Kubernetes, right?

    Kubernetes I think was the initiator of it. What does it mean for the market? What does it mean for our audience out here? And we have a technical audience, 60% of which are practitioners, 30% are managers or higher, and 10% are just, I don’t know, vendors. But what does it mean for them out there?

    What does it mean? I mean, these CI/CD tools that they’ve gotten used to, that they’ve grown up with a bit over these last years. What does it mean for them?

    O’Leary: Well, Yeah. I think that we love to say CI/CD together. I mean, I love to say CI/CD together, right? That’s now we name our product. But those concerns have always really been separate and never really been completed merged and married. Like it’s great to try and think about them as a single unit – CI slash CD, right?

    But I think the reality is that CI is something that has been around for a really long time, probably 20-plus years if not more, but CD is something that a lot of organizations still struggle with for any number of reasons, whether that’s a technical reason, whether that’s an organizational structure reason, whether that’s a risk aversion reason. The idea of doing continuous deployment is something that really is very dependent on your organizational structure, who your end users are, and the, yes, to some degree like you said, Alan, the tooling that you choose. And what we’ve seen in traditional CI/CD tooling is this concept of applying what we do in CI, which is scripting based on an event, running through a number of scripts.

    That is what then becomes CD. Well, now we’re going to push with scripts out to the environment. And so GitOps, you noticed in my definition I didn’t say that a pull was necessary. I think you can do push-based GitOps if you have infrastructure as code, you have MRs at the point of change control, and you have the ability to do CI/CD. But I think most people when they hear GitOps think about this another new concept, which is this concept of pulling. Right? We’ve heard Tracy talk about that.

    And so having an agent, an active agent inside of your infrastructure that is constantly ensuring that your desired state is the actual state in the world – that’s what Kubernetes brought to us, that’s what – and then that’s what pull-based GitOps brings to us. And so that’s why for us we’ve been doing CD for years, like you said, Alan, and we’re doing mostly push-based. But as our customers’ needs expanded in GitOps we’ve adopted a Kubernetes operator and now we have our own Kubernetes operator, and you’re seeing this more and more in all the tools, that runs inside of the cluster and pulls changes in.

    And that methodology, while – again, credit Tracy – sounds a lot like Puppet, [Laughter] it still is something that was the alure of Kubernetes for a lot of people. Hey, I want to describe my desired state and have something else worry about that desired state always being true regardless of if I push something wrong once. I want this desired state to always be correct. And so I think that’s how we’re going to see a huge change in the tool space. Right?

    To Parag’s point, we’re talking about Spinnaker. That’s something that Netflix has been using for a long time, but no one ever thought that they would be deploying like Netflix. Well, now it’s a huge deal and lots of people are talking about it.

    Shimel: We are.

    O’Leary: And they are, right? And so it’s very interesting.

    Shimel: Mitchell, it sounds like an analysis question.

    Ashley: Well, I was thinking about that very much, and I appreciate Brendan’s perspective on it. Because any time you have a major paradigm change, you have early adopters and then you have a long tail of how that adopts out. And you’re talking about the Netflix. People are now starting – how long have we been talking about Netflix as the example? As long as we’ve had DevOps and continuous deployment.

    And I think we have a bit of the same challenge here of instead of still having layers of our stack and different responsibilities or processes or things that go into creating environments, whether it’s from code or not, when we say it’s all going to be based on a pull and it’s all going to happen in its own environment, if you will, that’s a bit of a big pill to swallow for a lot of organizations, especially in complex applications. I’m not saying it’s not going to happen, but I think it’s going to be slow to adopt. And I’m sorry if I’m going to get a bunch of hate mail.

    It’s a great concept, but I do think it’s going to be more difficult to adopt than even containers were for breaking down applications. Not that it’s not a good idea or the right idea, it’s just instead of software architectures, modern software architectures being something the developers now get their mind around with containers and microservices and fabrics, meshes, now you’re talking about different parts of the organizations going into infrastructure and ops. And that’s all getting compacted together and you’ve got multiple parts of the organization getting behind that paradigm shift. So that’s my view of it.

    Will we see things in the market around GitOps? Of course we will. But I don’t think we’re going to see this tipping point that will happen rapidly. That’s my assessment.

    Shimel: Okay. So it seems like you and Tracy have formed some sort of coalition here, and –

    Ragan: [Laughter]

    Ashley: I always go to Tracy when I need something _____.

    Shimel: Right. Send the hate mail that way.

    Ashley: _____ Brendan and Parag, too. [Laughter]

    Shimel: Right.

    Ragan: I want to say that the CI/CD pipeline can adopt GitOps quite efficiently. Think what Brendan pointed out. When you think about CI/CD you think everything’s a push. Right? And you add a script that you’ve executed that your CI/CD server, your orchestration server, your job scheduler said, “Hey, there’s a step here.” Then that step goes out and pushes an update to production environment.

    How that’s going to – because GitOps is a pull methodology it doesn’t have a promotion methodology. And what really we’re going to see is for folks using GitOps is we’re going to see the CI/CD process do the – create the pull request. And there will be an approval. If you want a pull request to go into production, there’s going to have to be probably a manual step to approve that pull request, and then once it gets there the CI/CD pipeline creates it. Now the challenge will be having the CI/CD pipeline automatically update all of those declarative scripts. [Laughter]

    Because there will have to be a different one based on the environment. But that is totally doable and it doesn’t change that. But what it does do, it says, hey, CI/CD. You need to have a hermetic process. It needs to be airtight. You need to have a single source of truth.

    What that is ultimately will still be called GitOps, even if it’s not Git that is that single source of truth. It’ll be like we reach for a Kleenex, not a tissue. [Laughter]

    Doshi: Let me back up what Tracy just said, and maybe take the view of – I think you said 30% of the listeners are looking at this from a business angle. And your question also sounded like what does this mean as our industry, to our industry? So today we’ve got, I don’t know, 170 different products, if you will, and tools around – and Brendan can probably say, no, it’s now 250 or something [Laughter] – around DevOps and all the new different concepts.

    Shimel: Closer to 700.

    Doshi: Oh, okay. My bad. [Laughter]

    I was a little behind. So we have so many things here, and there’s a reason for that too, right? So we can’t just say, hey, let me take a cloud provider’s full CI/CD DevOps vision and hope that we’re going to impose one way of doing things, because this is not all worked out. The moment you bring the developer mindset to other layers of the computer stack – infrastructure, environment, operations – it becomes so complex. And I think at an industry what it’s going to do is it’s going to actually spur a lot of opportunity, a lot more innovation.

    I don’t know if that means that 750 will go to 1,050, but there’s so much room for different companies that will do things differently – push, pull, I need an approval before production due to a segregation of duties. No, no, no. I want to approve before I even commit the code or at least run the security static analysis, the dynamic analysis, the container image scanning. Everybody will have different ways of doing things, and I think it’s going to spur on lots of opportunities for a lot of companies.

    Shimel: Fair enough. A giant petri dish. Guys, I’m going to need to pull the plug on us here because we’re up against our time limit. But what a great conversation. For our audience, in a couple of weeks – and I don’t have the date in front of me, I apologize – we’re actually going to do an interact roundtable on this subject. Some of our panel members will be back for that.

    And we’re going to allow you to come online and ask questions and comment and join the conversation on this. I think it’s a great conversation. So be on the lookout for that and come check us out. But in the meantime, Tracy, as always, you know, you bring so much to this. Thank you so much. Thank you so, so, much.

    Ragan: Oh, it’s my pleasure. Absolutely my pleasure.

    Shimel: Yes. Brendan, always a pleasure to have you on, my friend.

    O’Leary: Yeah.

    Shimel: Look forward – also, Brendan does a show with us here on TechStrong TV every month. It’s called The Lab. And he discusses – you know, the 22nd of every month is a new GitLab release and Brendan gives us the lowdown on each month’s release. It comes out once a month. It’s called The Lab, and it’s available here on TechStrong TV. So Brendan, thanks so much for that.

    O’Leary: Yeah. Thanks, Alan, for having me. I always learn so much from these panels. So hopefully we all learned something today, but I really enjoy doing them.

    Shimel: Always a pleasure. And you add as much as you learn, Brendan. Thank you. Parag, one question for you. You mentioned earlier that you were a former physicist. Is there such a thing as a former physicist? Isn’t it more like I think, therefore I am?

    Doshi: Yes.

    Shimel: Once a physicist, always a physicist?

    Doshi: It certainly is Hotel California. You’re absolutely right. [Laughter]

    Shimel: Yep. But thank you so much for joining us. Always great to have you here and bringing some great perspective to everything. I appreciate it.

    Doshi: Thank you.

    Shimel: Mitchell, what can I say? We wrapped another one, Mitch. Good stuff.

    Ashley: Yeah, we did. And I’m just so glad that Parag didn’t bring quantum physics into this, because I’d be in trouble.

    Shimel: That could be another show we’ll do.

    Doshi: [Inaudible comment]

    Shimel: Quantum computing. But, no, anyway, that’s going to wrap up this episode of DevOps Unbound. We hope you’ve enjoyed it. This is every other week we do a fresh DevOps Unbound, so be on the lookout. Every month we do the roundtable with the audience, so be on the lookout for that as well. This is Alan Shimel for MediaOps. You’ve – oh, before we go, many thanks to Tricentis for sponsoring DevOps Unbound. We couldn’t do it without them. So thank you all very much. We’ll see you on another episode.