Category: DevOps Unbound

  • Digital Transformation – Escaping the Event Horizon

    Digital Transformation – Escaping the Event Horizon

    Roundtable 9: Those who survive, and even thrive, during disruption successfully escape the event horizon, creating innovative new business, product and operational models. They also leverage and harness technology in new and innovative ways through digital transformation. Not escaping the event horizon leaves companies who languish in survival mode, or worse.

  • The Human Side of DevOps – Upskilling for the Digital Transformation

    The Human Side of DevOps – Upskilling for the Digital Transformation

    Roundtable 8: DevOps is all about people, process and tools. People, or humans, are critical to DevOps success. Just as we invest in tools and build processes, software leaders recognize it is vital to invest in DevOps skills for their people and software teams.

  • EP 9: The Road to GitOps

    EP 9: The Road to GitOps

    GitOps takes DevOps best practices for development and applies them to infrastructure automation. In short, GitOps delivers a hermetic, immutable deployment. So, is GitOps replacing DevOps?

    Managing infrastructure as code through GitOps can be a game changer for organizations. But, when is a GitOps approach best used or not a fit? In this DevOps Unbound episode, I’m joined by Tracy Ragan of DeployHub, Parag Doshi of Tricentis, Brendan O’Leary of GitLab and Mitch Ashley of Accelerated Strategies Group to discuss these questions, the impact of GitOps on the CI/CD market and where the future of GitOps is heading in 2021.

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript

    Alan Shimel: Hello, everyone. This is Alan Shimel, CEO at MediaOps. And you’re watching another episode of DevOps Unbound. Let me introduce you to our panel for this week’s episode, and we’ll jump right into our subject matter. First of all, we have the newcomer to DevOps Unbound, but he’s not a newcomer to our audience, my good friend Brendan O’Leary, senior DevOps evangelist at GitLab, a governing board member for CNCF. Hey, Brendan. Welcome.

    Brendan O’Leary: Alan, nice to see you, as always, and happy to be here.

    Shimel: Thank you for being here, Brendan. Next up we have our good friend and a semi regular here on many of our video segments at MediaOps, Tracy Ragan. Tracy is CEO of DeployHub, as well as the community-elected board member of the CD Foundation. Hey, Tracy. Welcome.

    Tracy Ragan: Thank you. Thank you so much, Alan, for having me here today. I always love doing these.

    Shimel: Absolutely. We love having you. Our third panel member today comes to us from Tricentis, who’s also the sponsor for DevOps Unbound, and it’s Dr. Parag Doshi. Parag runs engineering, VP engineering at Tricentis. Parag, welcome.

    Parag Doshi: Hello, Alan, everyone. Nice to be here. Thanks for having me.

    Shimel: Thank you. Thanks for being on. And then last but not least, my cohost for DevOps Unbound, Mitch Ashley. Mitchell is the CEO/founder of Accelerated Strategies Group as well as CTO at MediaOps. Hey, Mitchell, thanks for coming on today.

    Mitch Ashley: Absolutely, Alan. And thank you for all of our panelists for joining us. You are all fantastic. We enjoy having you.

    Shimel: You know, just before we jump into it, the continuing evolution of COVID times, we’ve done so much with our backgrounds, [Laughter] on our videos, right? Looking here at Tracy’s snazzy DeployHub background and Mitchell’s background and Parag’s – and Brendan’s old-school, man. Brendan’s giving us – if I’m not mistaken, he used to be down in the basement and now he moved up. I think he displaced one of the kids or something.

    O’Leary: Yeah, I moved upstairs.

    Shimel: Yeah, yeah. And that’s a real background, so good for Brendan.

    Ashley: And he’s got an MV7 microphone there. Nice job there, Brendan, on the mic.

    O’Leary: Oh, thanks. Yeah.

    Shimel: Yeah.

    O’Leary: All, you know, post COVID things to make it work, Alan. I wanted to have the background, the real background to make it work. But that was also a COVID change.

    Shimel: It’s authentic. Yeah. [Laughter] Crazy stuff. Anyway, guys, today’s show is around one of the hottest topics in DevOps – GitOps. Right? And I’ve actually heard from people, you know, that GitOps is the new DevOps or GitOps is somehow replacing DevOps, and I think DevOps has outgrown the – it’s going to be replaced by something.

    It’s transcended to a different level of existence, if you will. But certainly GitOps is on a lot of people’s lips. It’s not only on their lips, it’s in their development plans and how they are delivering software to the cloud, to cloud native, and all over. But is the right move for every situation? Is it a panacea?

    When is GitOps a good thing to use? When is it maybe not a good thing to use? Right? How much does it change sort of the classic CI/CD pipeline that we’ve started to finally wrap our heads around in DevOps? Stuff like that. So I am going to – Brendan, of course, GitOps, GitLab, they both have Git in it.

    So we’re going to make you – I’d like you to start off. Let’s start with some definitions. What’s GitOps mean to you?

    O’Leary: That’s a great question, and I think one that’s still probably [Laughter] unsettled, right? Much like defining DevOps can sometimes be seen as unsettled. GitOps came out of the cloud native community, and so a lot of folks maybe restrict the concepts of GitOps to say it only exists in a Kubernetes world. Because that’s kind of where this approach came from. But we’ve seen folks employ GitOps principles to all kinds of different infrastructure, automation, from old-school VMs to containers to Kubernetes clusters as well.

    So when I think of GitOps, I think it kind of requires three things. One, infrastructure as code. Right? You said that in the beginning. That’s kind of like table stakes to getting in. It’s something we’ve been working on for a long time. But really defining as code in Git – the Git part of GitOps – what our infrastructure’s going to look like, and then keeping that configuration stored in that code.

    And maybe the desired state is stored as code or not, but at least the way we get to that desired state is. Second, I think it’s merge request. So we call it merge request. You’ll hear other folks call them pull requests. But the concept of using the point where someone is proposing a change as the central place where teams collaborate on.

    Is this the change we want to do? Is this going to help solve the problem we’re trying to solve for? Right? Bringing that concept to the world of infrastructure is hugely beneficial, because it’s how our developers have already been working for years and years. And making that merge request, kind of that central source of truth for approvals and security checks and all those other things is really critical.

    And then last, I think it’s – the third thing is CI/CD, right? Of course. So the idea of GitOps is automating that infrastructure, update with a very simple Git workflow. And so you really do have to have embraced and adopted continuous integration and continuous delivery really in order for the whole package of GitOps to make sense for you.

    Shimel: Pretty darn good. Comments, add-ons, thoughts? Parag, Tracy?

    Ragan: I would only add one thing here. If you say what is GitOps, I like to shorten it down. I think what Brendan just defined is all accurate, but it’s operations by pull request. The goal of GitOps, what GitOps delivers is a hermetic, airtight, immutable deployment. And that is in essence what we’re trying to accomplish by using Git as the place where we declare and define what we’re going to push out to or have pulled out to the cluster.

    So it’s the concept of a hermetic deployment, something airtight that no human touched. Don’t you dare touch that and then deploy it out, because of we don’t know what you did. [Laughter] So the whole idea is to create a hermetic process.

    Shimel: I love that. That was concise and right on.

    Doshi: I have a similar viewpoint. I think of GitOps as a combination of infrastructure as code plus event-driven architecture. Right? And in this case we’re talking about merge requests or pull requests or whatever terminology you want to use, depending on the flavor of Git you love. But the way I think of this as a former physicist is entropy is there. It’s too hard to keep track of everything. Humans can’t remember everything.

    We need to be able to describe, as application developers have been doing for the past couple decades, in code the versions of things, the documentation associated with everything. And then the automation that we’ve had in CI/CD, which people think of, oh okay, that’s developers pushing their code out, well there’s no reason why we have to stop at application level. It needs to be the whole thing, because application and infrastructure must stay in sync at all times. So by putting a description of both the application, the infrastructure, and within the surrounding environment, all three of those into Git, you’re bringing the best practices of AppDev into the whole world together so they’re all synchronized.

    Ashley: Yeah. I like that description too, Parag, because in a way that’s breaking down silos that we still have around the code build and deployment process. And it’s almost like if you think back when we started using code repositories and, okay, everybody put their source code in one place, okay, let’s expand that to artifacts and other things. Well, now we’re doing the same thing of let’s abstract at one more level and make the whole thing as cradle to grave, if you will, for getting code out into product. Whether it’s any part of that stack, it’s a GitOps process.

    So to your point, Tracy, about it’s an immutable object, it’s something that we know from beginning to end, it’s all been in one environment and we happen to have manual processes that have just been automated that potentially are weak or could be broken.

    Doshi: I’ll add one more comment on top of Branden’s about – I think it was around – it came out of cloud native computing foundation, but really the way I look at containers is it doesn’t matter that’s a container or VM. We’ve just moved a control plane up to a higher level, but the same principle’s apply. So there’s no reason why you cannot use these same principles in a container type of environment or a non-container environment. The only difference I would say with containers is we can be very surgical, fine grain, down to each individual pot. So there’s a little advantage with cloud native architectures to be very fine grained.

    O’Leary: Yeah, I agree. And I think that’s important for folks to separate the concept of GitOps, the principles that we’ve all talked about. And I agree with the other panelists and how they’ve talked about it. But separate those principles from the technology, right? Because it’s not always true that Kubernetes is going to be the right technology for you to deploy every workload you have in, but still this concept of GitOps.

    And I really liked Tracy’s sealed container – well, overloading the word container there. [Laughter] But that sealed deployment that doesn’t have a lot of human – or doesn’t have any human interaction. That is what gets us to what Mitch was talking about, which is we want to take this isolated, people working in isolation, the kind of cave-like nature that operations typically used to have where you had to have this specialized knowledge that one person knew. Oh, wait. How do we deploy that service?

    Oh, we’ve got to call up Alan. He’s the one that knows how to do that. [Laughter] Right? You want to get rid of that and make it all, what Tracy said, which is like a hermetically-sealed deployment that we know we can reproduce every time.

    Ragan: Yeah. So GitOps was a brilliant way to leverage existing tools and habits to create Chef and Puppet. That is basically what was done. I mean, it has a Puppet kind of master architecture. You have a GitOps operator that’s running on your cluster. It does a pull based on what’s in Git. It very much took the basic concept of Chef and Puppet and leveraged existing tools that people already had, some open source, GitOps operators out there, and they were off and running. And that is the brilliance and beauty of GitOps. I mean, I have a lot of issues with GitOps, and I’m sure we’ll share them, but its brilliance is in that simplicity.

    Shimel: The irony though, Tracy, is that many people will tell you that Kubernetes killed Chef and Puppet, right? And we’ve seen Chef get sold maybe for less than money raises in valuation previously.

    Ragan: No, I’m just talking about the basic concepts.

    Shimel: No, no, I get it. But if we look at the concepts behind what Chef and Puppet did, which was automate that deployment, deployment automation like that, and then with Kubernetes it sort of negated the need for that, but yet we kind of snuck that in the back door. Except we put it in Git to use in Kubernetes. So the irony of that is pretty profound, right?

    Ragan: Well, it’s all about the container.

    Ashley: Right, right.

    Shimel: Yeah. You’re right. Maybe it’s because Chef and Puppet – [Crosstalk]

    Ragan: The containers simplified, yes.

    Shimel: – didn’t figure out how to work in the container. Look, for that matter Jenkins too. Right? We went to Jenkins X because of all of that and everything else.

    Ashley: That’s a really great example, yeah.

    Shimel: But what I’m reminded is of the old Who song, right? “Meet the new boss, same as the old boss.”

    Ragan: Exactly.

    Shimel: And you still have that old boss. With that being said, here’s another thing, and Parag, I’m going to ask you to kick off as I take us a direction here, and that is GitOps, yes, it’s for deployment. What about testing? I said this is all about testing. We’re doing our testing in Git. Is that the – that stops, right? Is that better? Worse? Is that the way it should be, always, sometimes, never?

    Doshi: So I don’t know if I’d call it testing in Git, but I would say now Git becomes a source of truth. So I think what Tracy said is right, that it’s just that same sort of automation of infrastructure but now moved into making Git the centerpiece of where you define this. So the Git part is unchanged, but the testing is now event driven. Right? So because you’ve merged in code, you’ve committed code or gone to do a pull request, that can trigger testing.

    And we call this continuous testing. And I think the evolution of this can even go as far as our test definitions themselves should also be put into Git. That’s the next evolution that we’ll have. I don’t believe it’s really commonplace just yet, but when that happens then, we are trying to describe as much as possible into Git as a repository and use the principles of version control added with the event-driven concepts.

    So when a developer does X or when an operator is in environment Y and something needs to change, that whole Git workflow can invite continuous testing to say do I allow that to merge back in? If you want to have an approval, you could do those kinds of things. So to me, incorporating continuous testing, and the day that we put test definitions into Git as well, we’ve gone quite a bit further of completing this vision and really making DevOps real. Right? Because DevOps isn’t just releasing really fast, deploying really fast. It’s deploying fast with confidence and ease. That’s what we need.

    Shimel: Thoughts on that, panel?

    Ashley: It sure seems like a natural evolution to include the test environment. And as you say, Parag, it’s probably not commonplace yet today. A question for you about that; do you think that there’s incentive with the folks that are taking care of the testing, whether they be the QA team or developers writing tests, things like that? Do you think they have the right incentives to move it into Git or will it be something else that causes it to happen?

    Doshi: I think as long as people have the incentive of delivering value as quickly as possible with the best meantime to recover, the lowest defect leakage rate, as long as we do break down those silos, as you said. If the testing team is, okay, well, I don’t understand what the developers do, just throw it over the wall and then I do my test, that’s not going to work. This is talking about dev testers, the full stack developers. And as long as everybody’s motivation is right, that we’re trying to deliver value faster and faster, that we will not have confidence – that a VP will not say let my applications go out – you know, they’re only going to look at portfolios of apps and say what amount of testing has been done? What amount of security checks have been done?

    That’s another thing. The environment. Okay? That’s the biggest entropy of all. We have the least control over the environment. The same code, the same infrastructure in different environments behaves differently. Until we get that environment description also into Git, coupled with the testing model themselves that says here’s how we test it in that environment or that application or this infrastructure pattern, then DevOps becomes real. Until then, this is all talk.

    O’Leary: Yeah, I agree. I think it’s critical, and it’s something that GitOps, again, this concept of GitOps, I think is exciting to a lot of people that have been trying to make DevOps real for so long, because it enables that in a way. Right? We see kind of the light at the end of the tunnel, I would say. And if that is all brought into Git and all made into something that can deploy independent of any human interaction, well now you have the ability to do continuous testing. You have the ability to stand up an identical environment, the golden goose or [Laughter] golden egg of a, hey, we want an environment that’s exactly like production to really test it and really hit with security tests and know that when we do deploy to production we’re going to have that confidence.

    So if you’re able to continue to iterate on the environment, the infrastructure, just like you’re iterating on the code, it enables you to then say, okay, now we can not only have the code to review and maybe some static code analysis that happens and some testing that happens, and then maybe some human testing that happens, now we’ve also got an entire environment spun up automatically that mirrors production, where we can have a high level of confidence that it’s going to work when it really ships.

    Shimel: So guys, you’re telling me DevOps hasn’t been real quite yet. I feel like I’m living in the Matrix, kind of living in a simulation. Tracy, I sense you’re Agent Smith here, though. You have a different view. Maybe GitOps isn’t the panacea it is or it’s been projected here. What’s your view?

    Ragan: So no hate mail, okay? No hate mail. [Laughter]

    Because I know there’s a lot of enthusiasts out there, and rightfully so. But let me just level set the way I see where we’re headed and where GitOps can’t take us. GitOps is imperative. It’s scripted. And it works really well – so think about how most people use GitOps.

    You have a code repository that you keep all your source code in. You make a change to your source code. You check that in just the same as you always did. You also have an environment repository, and what lives in your environment repository is your Kubernetes deployment YAML file. And that is where you define all of the stuff that we’ve been talking about and that’s what the GitOps operator looks at and pulls across.

    Now if you’re in a Kubernetes environment and you have containerized your application – this works great, it really does – and you don’t have a whole lot of environments – you have three environments dev, test, and prod – because remember, in a Kubernetes YAML file there’s a couple of things that are important to the GitOps operator. One is the tag of the container. So once you register your container image you get a SHA, and that makes it a unique identifier. And you have to update your YAML script with that SHA.

    Now dev, test and prod may have different values, configuration values. So you have to branch it. So you branch dev, test, and prod. So now you have three versions of that YAML file to manage your container deployment. And that’s pretty doable for most organizations today, but it is still imperative. Everything’s being defined by the script for each of those environments.

    Now let’s go and expand it and scale it. Let’s scale it to a microservice environment. Every microservice has its own YAML file because it should be independently deployed. And they should be shared, because that’s the beauty of a service-oriented architecture. They should be shared across teams. So what you do is as an application team, you’re going to make your coding changes in your coding repository and then you’re going to have different YAML files for the different services.

    Now let’s say you’re a store. I use the candy store and hipster store and the clothing store. They all use the card service. Well, they’re all going to put their card service YAML file into their environment repository, and that’s going to get deployed to their own clusters and name spaces. So you have a scaling issue, and it’s because of the imperative nature of GitOps.

    We are starting to move and have more conversations about declarative and events and catalogs, where this stuff is going to be stored and generated at runtime. So for example, instead of having all of these scripts, you’re going to have central places to go to say what’s the key value pair for this environment? I’m going to pull this registry information that has the SHA, I’m going to go get my key value pairs, and that way you don’t have to have so many scripts and you’re not managing – it’s onerous.

    It really can become onerous. We have customers who have 200 microservice and 20 clusters. That’s a lot of scripts. And if you’re an SRE and you want to update your microservice and you want to provide it to everybody, you’re going to have to figure out every single application environment repository that has it in there for every single environment and make those corrections. So it doesn’t scale as well as it should. Now I know that their CNCF has a group, a working group, around GitOps, and they’re starting to think about GitOps version two, and they’re going to have to address these kinds of core problems around an imperative process.

    Shimel: Love it. Brendan?

    O’Leary: Yeah. I mean, I like that concept and I think you’re right, Tracy, that the ability to manage the scale is definitely something you have to be prepared for if you’re thinking about GitOps, because if you just take it at its face value you end up with all those scripts and it could make the problems you have worse rather than fixing some of the problems you hope it would come into and solve. And so I think in a cloud native environment, again, it maybe pairs a little bit more simply, but I think that’s why again it’s great to think about the concept of focusing on that developer-centric approach and the experience that they’re going to have when it comes to shipping something out, to your point of if I want to ship an update to my microservice that impacts everyone, I don’t want to have to go through 78 repositories and update that everywhere.

    And so having GitOps do the infrastructure management in the same version control system as application development is great, but you also have to have a plan in place for how does that scale to the scale that you’re really going to build to? If that the scale of developers, if that’s the scale of services, if that’s the scale of environments. And so I think it’s important to have tooling and an approach in place that accounts for that as well.

    Shimel: Parag, you were going to say something I think before we –

    Doshi: Yeah. Sure. So I love what Tracy described there, and getting into the specifics of the proliferation of YAML files, and especially as we break down microservices. So this is the whole concern with microservices. What you took as monolith and you described as one big black box, now you’ve broken it down. And so now we have the proliferation of all these things. Right?

    So I like things like Helm to help to reduce that and parameterize all these things. But fundamentally with GitOps, the concepts are there. Like I said, infrastructure as code with an event-driven type of approach version control. What I like about CNCF and containerization is the purpose of containers inherently – and Branden can keep me honest on this – is really to make the environment completely separated from your solution, your application, your infrastructure. Right? So today’s industry, the best approach we have is containers or serverless, which is also part of CNCF, so that we have things like Docker files and YAMLs that allow us to reduce the entropy of the environment as we can as much as possible and describe them declaratively in things like GAML files.

    And I do agree with Tracy 100%, that now it becomes, okay, well now I’ve got all these different configs, pulled that out of my code. Even with service mesh technology and CNCF we’re pulling out things like security and rate limiters between every single microservice. All that’s taken out of the logic, the business logic, on everybody doing security different ways. Now it becomes a configuration issue. How do I – if I want to change a configuration, do I need to change 100 different YAML files or config maps? And I think the technology’s evolving.

    Help is just one example. And then I’ll associate that with all the continuous delivery. You all know Spinnaker and a bunch of different technologies in this area, Octopus and so forth, that can help.

    Shimel: Yep. Mitch, did you want to say something? Otherwise, I want to – I’ve got another –

    Ashley: All I can add to the conversation is version control across all of this also gets very complex. [Laughter] I think you all –

    Shimel: But isn’t that what Git and Repositories are supposed to help us with anyway?

    Ashley: It is, but now you’re versioning all of your scripts along with all the environments and all of the variables of what you’re trying to script.

    Ragan: And it’s not relational.

    Ashley: Yeah.

    Ragan: You know, when you have – what’s the blast radius of my microservice? I’m going to push it out because my hipster store needs the new card service. If you’re using an SOA architecture that means that you have one card service and everybody’s using that version and that’s the best way to move forward because you eliminate the sprawl. If you put it all in its own namespace, you have multiple versions of the card service. So now you have a problem with the card service and you have to go update all the namespaces that’s using that card service.

    So you have to have a relational model to be able to track what applications or what namespaces are using what microservices. And unfortunately, Git is not relational. We create relations using tags, using different repositories, but that becomes redundant.

    Ashley: Great example.

    Shimel: Fair enough. So let me –

    Doshi: Tracy, you talked about the structure.

    Shimel: Oh, go ahead, Parag.

    Doshi: I was just going to follow up and ask a question to Tracy. I think what you’re saying there is Git is just a big version control repository. If we can provide some extra structure and relationships around all these artifacts that we’re storing there we might have better control.

    Ragan: And that’s what you call catalog.

    Doshi: Yeah.

    Ragan: Right? So I think we’re starting – when we really look at a service-oriented architecture and we look at – let’s say, for example – let’s push ourselves farther into the future. And Brendan brought this up around service mesh. And I believe this can happen and will happen. I believe service mesh will be enhanced on.

    Right now you can say do a deployment and deploy out to 10% of my users. I think service mesh will start looking at active directory and say deploy out to my developers. So now we have one cluster and we have immutable services out there. We put a new service out there, it gets requests routed to the developers, they do their testing on it, once it’s good it’s rerouted to their testing community.

    Then it’s routed to a certain group of their production community. Which means that we don’t have deployments across multiple clusters. We’re going to start pushing into a one big cluster, which sounds a lot like the mainframe. [Laughter] But we’ve come full circle. And as Alan said, the song, right? Meet the new boss, same as the old boss. [Laughter]

    Shimel: Right. We won’t get fooled again. That is true. There is certainly some truth to that. I wanted to spend a little time talking about the impact of GitOps on the CI/CD market. We’ve written in devopsy.kinsta.cloud several articles, the gist of which was DevOps is the symptom of disease. CI/CD is the cure, if you will. Right? And we’ve built this whole DevOps industry around CI/CD solutions.

    And I mean look, GitLab’s a great example. They had a great Git product to begin with. But it was their CI/CD solution that kind of moved them into that end-to-end DevOps, if you will. Right? GitOps I think has profoundly changed that market because it’s profoundly changed the way we do CI/CD. Well, that and Kubernetes, right?

    Kubernetes I think was the initiator of it. What does it mean for the market? What does it mean for our audience out here? And we have a technical audience, 60% of which are practitioners, 30% are managers or higher, and 10% are just, I don’t know, vendors. But what does it mean for them out there?

    What does it mean? I mean, these CI/CD tools that they’ve gotten used to, that they’ve grown up with a bit over these last years. What does it mean for them?

    O’Leary: Well, Yeah. I think that we love to say CI/CD together. I mean, I love to say CI/CD together, right? That’s now we name our product. But those concerns have always really been separate and never really been completed merged and married. Like it’s great to try and think about them as a single unit – CI slash CD, right?

    But I think the reality is that CI is something that has been around for a really long time, probably 20-plus years if not more, but CD is something that a lot of organizations still struggle with for any number of reasons, whether that’s a technical reason, whether that’s an organizational structure reason, whether that’s a risk aversion reason. The idea of doing continuous deployment is something that really is very dependent on your organizational structure, who your end users are, and the, yes, to some degree like you said, Alan, the tooling that you choose. And what we’ve seen in traditional CI/CD tooling is this concept of applying what we do in CI, which is scripting based on an event, running through a number of scripts.

    That is what then becomes CD. Well, now we’re going to push with scripts out to the environment. And so GitOps, you noticed in my definition I didn’t say that a pull was necessary. I think you can do push-based GitOps if you have infrastructure as code, you have MRs at the point of change control, and you have the ability to do CI/CD. But I think most people when they hear GitOps think about this another new concept, which is this concept of pulling. Right? We’ve heard Tracy talk about that.

    And so having an agent, an active agent inside of your infrastructure that is constantly ensuring that your desired state is the actual state in the world – that’s what Kubernetes brought to us, that’s what – and then that’s what pull-based GitOps brings to us. And so that’s why for us we’ve been doing CD for years, like you said, Alan, and we’re doing mostly push-based. But as our customers’ needs expanded in GitOps we’ve adopted a Kubernetes operator and now we have our own Kubernetes operator, and you’re seeing this more and more in all the tools, that runs inside of the cluster and pulls changes in.

    And that methodology, while – again, credit Tracy – sounds a lot like Puppet, [Laughter] it still is something that was the alure of Kubernetes for a lot of people. Hey, I want to describe my desired state and have something else worry about that desired state always being true regardless of if I push something wrong once. I want this desired state to always be correct. And so I think that’s how we’re going to see a huge change in the tool space. Right?

    To Parag’s point, we’re talking about Spinnaker. That’s something that Netflix has been using for a long time, but no one ever thought that they would be deploying like Netflix. Well, now it’s a huge deal and lots of people are talking about it.

    Shimel: We are.

    O’Leary: And they are, right? And so it’s very interesting.

    Shimel: Mitchell, it sounds like an analysis question.

    Ashley: Well, I was thinking about that very much, and I appreciate Brendan’s perspective on it. Because any time you have a major paradigm change, you have early adopters and then you have a long tail of how that adopts out. And you’re talking about the Netflix. People are now starting – how long have we been talking about Netflix as the example? As long as we’ve had DevOps and continuous deployment.

    And I think we have a bit of the same challenge here of instead of still having layers of our stack and different responsibilities or processes or things that go into creating environments, whether it’s from code or not, when we say it’s all going to be based on a pull and it’s all going to happen in its own environment, if you will, that’s a bit of a big pill to swallow for a lot of organizations, especially in complex applications. I’m not saying it’s not going to happen, but I think it’s going to be slow to adopt. And I’m sorry if I’m going to get a bunch of hate mail.

    It’s a great concept, but I do think it’s going to be more difficult to adopt than even containers were for breaking down applications. Not that it’s not a good idea or the right idea, it’s just instead of software architectures, modern software architectures being something the developers now get their mind around with containers and microservices and fabrics, meshes, now you’re talking about different parts of the organizations going into infrastructure and ops. And that’s all getting compacted together and you’ve got multiple parts of the organization getting behind that paradigm shift. So that’s my view of it.

    Will we see things in the market around GitOps? Of course we will. But I don’t think we’re going to see this tipping point that will happen rapidly. That’s my assessment.

    Shimel: Okay. So it seems like you and Tracy have formed some sort of coalition here, and –

    Ragan: [Laughter]

    Ashley: I always go to Tracy when I need something _____.

    Shimel: Right. Send the hate mail that way.

    Ashley: _____ Brendan and Parag, too. [Laughter]

    Shimel: Right.

    Ragan: I want to say that the CI/CD pipeline can adopt GitOps quite efficiently. Think what Brendan pointed out. When you think about CI/CD you think everything’s a push. Right? And you add a script that you’ve executed that your CI/CD server, your orchestration server, your job scheduler said, “Hey, there’s a step here.” Then that step goes out and pushes an update to production environment.

    How that’s going to – because GitOps is a pull methodology it doesn’t have a promotion methodology. And what really we’re going to see is for folks using GitOps is we’re going to see the CI/CD process do the – create the pull request. And there will be an approval. If you want a pull request to go into production, there’s going to have to be probably a manual step to approve that pull request, and then once it gets there the CI/CD pipeline creates it. Now the challenge will be having the CI/CD pipeline automatically update all of those declarative scripts. [Laughter]

    Because there will have to be a different one based on the environment. But that is totally doable and it doesn’t change that. But what it does do, it says, hey, CI/CD. You need to have a hermetic process. It needs to be airtight. You need to have a single source of truth.

    What that is ultimately will still be called GitOps, even if it’s not Git that is that single source of truth. It’ll be like we reach for a Kleenex, not a tissue. [Laughter]

    Doshi: Let me back up what Tracy just said, and maybe take the view of – I think you said 30% of the listeners are looking at this from a business angle. And your question also sounded like what does this mean as our industry, to our industry? So today we’ve got, I don’t know, 170 different products, if you will, and tools around – and Brendan can probably say, no, it’s now 250 or something [Laughter] – around DevOps and all the new different concepts.

    Shimel: Closer to 700.

    Doshi: Oh, okay. My bad. [Laughter]

    I was a little behind. So we have so many things here, and there’s a reason for that too, right? So we can’t just say, hey, let me take a cloud provider’s full CI/CD DevOps vision and hope that we’re going to impose one way of doing things, because this is not all worked out. The moment you bring the developer mindset to other layers of the computer stack – infrastructure, environment, operations – it becomes so complex. And I think at an industry what it’s going to do is it’s going to actually spur a lot of opportunity, a lot more innovation.

    I don’t know if that means that 750 will go to 1,050, but there’s so much room for different companies that will do things differently – push, pull, I need an approval before production due to a segregation of duties. No, no, no. I want to approve before I even commit the code or at least run the security static analysis, the dynamic analysis, the container image scanning. Everybody will have different ways of doing things, and I think it’s going to spur on lots of opportunities for a lot of companies.

    Shimel: Fair enough. A giant petri dish. Guys, I’m going to need to pull the plug on us here because we’re up against our time limit. But what a great conversation. For our audience, in a couple of weeks – and I don’t have the date in front of me, I apologize – we’re actually going to do an interact roundtable on this subject. Some of our panel members will be back for that.

    And we’re going to allow you to come online and ask questions and comment and join the conversation on this. I think it’s a great conversation. So be on the lookout for that and come check us out. But in the meantime, Tracy, as always, you know, you bring so much to this. Thank you so much. Thank you so, so, much.

    Ragan: Oh, it’s my pleasure. Absolutely my pleasure.

    Shimel: Yes. Brendan, always a pleasure to have you on, my friend.

    O’Leary: Yeah.

    Shimel: Look forward – also, Brendan does a show with us here on TechStrong TV every month. It’s called The Lab. And he discusses – you know, the 22nd of every month is a new GitLab release and Brendan gives us the lowdown on each month’s release. It comes out once a month. It’s called The Lab, and it’s available here on TechStrong TV. So Brendan, thanks so much for that.

    O’Leary: Yeah. Thanks, Alan, for having me. I always learn so much from these panels. So hopefully we all learned something today, but I really enjoy doing them.

    Shimel: Always a pleasure. And you add as much as you learn, Brendan. Thank you. Parag, one question for you. You mentioned earlier that you were a former physicist. Is there such a thing as a former physicist? Isn’t it more like I think, therefore I am?

    Doshi: Yes.

    Shimel: Once a physicist, always a physicist?

    Doshi: It certainly is Hotel California. You’re absolutely right. [Laughter]

    Shimel: Yep. But thank you so much for joining us. Always great to have you here and bringing some great perspective to everything. I appreciate it.

    Doshi: Thank you.

    Shimel: Mitchell, what can I say? We wrapped another one, Mitch. Good stuff.

    Ashley: Yeah, we did. And I’m just so glad that Parag didn’t bring quantum physics into this, because I’d be in trouble.

    Shimel: That could be another show we’ll do.

    Doshi: [Inaudible comment]

    Shimel: Quantum computing. But, no, anyway, that’s going to wrap up this episode of DevOps Unbound. We hope you’ve enjoyed it. This is every other week we do a fresh DevOps Unbound, so be on the lookout. Every month we do the roundtable with the audience, so be on the lookout for that as well. This is Alan Shimel for MediaOps. You’ve – oh, before we go, many thanks to Tricentis for sponsoring DevOps Unbound. We couldn’t do it without them. So thank you all very much. We’ll see you on another episode.

  • EP 8: Testing Is Not a Monolith

    EP 8: Testing Is Not a Monolith

    With DevOps comes implementation of CI/CD, automation and testing. Is this sufficient to achieve “shift left testing”? Or does testing need to shift further left? Do we need new or additional tests for acceptance (ATDD) or behavioral testing (BDD)?

    In this episode of DevOps Unbound, we are joined by Christine Fisher of NAIC, Leandro Melendez (aka Señor Performo) of Qualitest, Ken Pugh of Ken Pugh Inc., and Mitch Ashley of Accelerated Strategies Group to explore the state of testing in DevOps and CI/CD pipelines, and where testing can evolve to keep pace with software development and delivery.

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript

    Alan Shimel: Hey, everyone. Thanks for joining us on another edition of DevOps Unbound. We have a really great DevOps Unbound topic to explore today, and that is that testing is not a monolith. A lot of people, especially people who aren’t familiar with testing, just say testing—QA, testing—and we think of, oh, it’s a test. And we know there’s different kinds of tests, but do we really know all the different kinds of tests that need to be done, as we’re developing and deploying and managing and monitoring software applications and so forth? I don’t think so. 

    I think, really, we could cater this show to those of you out there who do know, but I think today, we want to talk a little bit to those who don’t know as well and let’s try to get you educated on what some of the various aspects of testing are that you may not be familiar with. They’re important, nevertheless, and how they influence what you see and the apps you use and what we may see going forward.

    I’ve got a great panel to introduce you to today. I’m gonna allow them each to introduce themselves. Let me first start off with Leandro Melendez, I can’t say it with that Spanish roll of the tongue that you can, Leandro, I apologize. And Leandro is known in the industry as well as a name—I’m gonna let, I don’t wanna steal your thunder. Leandro, introduce yourself.

    Leandro Melendez: Thank you very much, and don’t worry, you made a great pronunciation of my name, way better than what I’m used to, so that was pretty good.

    Shimel: Okay.

    Melendez: Yeah, repeating, I’m Leandro Melendez, a performance test manager from Qualitest Group, and as well, I’m known on the internets as Señor Performo, as you all said, Performo with strong R’s.

    And well, we provide several services to—my specialty is performance testing, integrating it right now with full continuous integration DevOps and agile methodologies. But as well—and you mentioned it very well earlier—it starts to permeate in other areas of testing when you start to do it holistically and at a large scale, and thinking on all of the steps on the SDLC. So, we will talk a little bit more about it.

    The last thing about me, as I said, Señor Performo, on the internets, I have a YouTube channel, I try to help educate people on especially performance testing, but anything that concerns testing and methodologies, agile, DevOps, and all that, I’m very happy to help provide information as well as hear whoever gives me a chance to speak and share the knowledge, I’m very happy to.

    Shimel: We’re honored to have you here, Leandro, thank you. Next, we’re gonna introduce Ken Pugh. Ken is—well, Ken, you introduce yourself. Go ahead, I’m sorry.

    Ken Pugh: That’s okay. Hi, I’m Ken Pugh. I’ve been in the programming software development field for about two-fifths of a century, and in agile for about a fifth of a century, and doing everything from gathering requirements to final testing.

    I’ve got a book out called “Lean-Agile Acceptance Test-Driven Development: Better Software Through Collaboration,” and I’ve been teaching, training, and emphasizing acceptance test-driven development-slash-behavior driven development for the past 15 years.

    Shimel: Fantastic. And for our fraction challenged friends in the audience, a fifth of a century is about 20 years, so that’ll give you a little idea.

    Next up, we have Christine Fisher. Christine, welcome, introduce yourself, please.

    Christine Fisher: Thank you. My name is Christine Fisher, I manage a team of BAs, QAs, and UX at the National Association of Insurance Commissioners. My team and I have been on a journey with behavior-driven development over the last two and a half years, and as we get ready to talk about testing here, my big secret is, I’ve actually never been a tester, I just have managed some. But with our jump into DevOps, I felt very strongly about advocating for what our role should be and how the work should change to keep up with the other changes that were happening with DevOps, which helped lead us to BDD.

    Shimel: Fantastic. Pre-COVID times, we’d say, “But I did stay in a Holiday Inn Express last night,” but we can’t even say that any more.

    Last but not least is my co-host and anchor on our show, Mitchell Ashley. Mitchell, why don’t you—quick introduction?

    Mitch Ashley: I don’t know what fractions I’ve worked, but I do know that I used that joke last night and my family had never heard it, so—so much for the Holiday Inn Express joke. [Laughter] 

    But, no, I’m an old software person, started in software, and have done lots of things around both cloud as well as, I’ve also been on the vendor side, so network security and SaaS services. So, testing people are some of the most unique and, in my world, beloved people, because those are the folks I really look to to say, “How are we doing? Where are we?” And I think even more so in the DevOps time, so I’m excited to explore this with you all. So, thanks for joining us.

    Shimel: Fantastic. So, first of all, let’s assume that our whole audience aren’t test experts, they’re more DevOps generalists, there’s a good smattering of cybersecurity folks, cloud native, infrastructure, ops, and we’re throwing out some ADD—behavior-driven, BDD—you know that they may not be familiar with.

    So, why don’t we do a quick kinda definition? And we’re gonna ask the non-tester, if you will, Christine, if you wouldn’t mind leading off with what are these references we’ve made to these types of testing?

    Fisher: Sure. So, when we’re talking about behavior-driven development, we’re talking about understanding the behavior of the end user and understanding what the workflows are, why they are trying to accomplish that task, and working on testing, understanding the requirements, testing those requirements in that way, not just writing a requirement, developing a requirement, testing a requirement without any true understanding of what the person who’s going to use the software on the end does.

    And when I talk about not being a tester, more on the functional side with requirements as a BA, things like that, that was always the interesting part to me. And then sometimes, we would test something and it would be tested correctly, but we still missed the mark. And so, having the whole team understand what the end user wants to do, putting it in the common language, that’s really important for behavior-driven development.

    Shimel: Agreed, agreed. Ken, do you want to add anything?

    Pugh: Yeah. So, one of the big things in behavior-driven development or acceptance test driven development is having the triad, and Christine sort of made a reference to it. The customer, developer, and tester get together prior to implementation, create the scenarios, the scenarios become the tests, and the developers are given those tests to run against their code, and those tests are the agreed upon, shared understanding of how the system should work.

    Shimel: Got it. Señor Performo—is that correct, yeah? Sí?

    Melendez: Very well said, yeah. [Laughter]

    Shimel: Señor Performo, how does acceptance-driven, behavior-driven development figure into your performance testing kind of regimen, if you will?

    Melendez: Well, as I mentioned earlier, these type of concepts permeate in all the testing practices, if I could say, and very well as it was said here, I like to call it the three amigos, that you put them together and even before you start rolling anything, drafting anything, you get the requirements well together with what you are going to be testing and what has to be marked as passed before you even—or a developer—can mark a task as completed. And I might have said it wrong, a developer, because in general, you’re supposed to be a team, and everyone is responsible for each element.

    And this comes together with performance metrics, where, in general, most of the people, when they think about performance, their mind goes right away to load tests, when I have to put hundreds of users to slam a system, where, if you see it from the beginning as well as some of these requirements when you generate them that, I don’t know, I’ll give an example, that the text field has to just allow a certain given of characters or be this color, rounded—little definitions like that, it should have included performance metrics as well inside of it. So that, when you are generating any piece of software or functionality, it’s measured there and then. 

    I have to say it’s an issue that I found often in the old waterfall days. I would arrive to do a load test after six months of development and start to automate some things that I’m like—this, hey, everybody, this is visibly slow. Just, I’m a single guy. I don’t need to load test it, why no one caught this before? 

    And those are the most general problems, in terms—I think, as well, in functionality, some things that come from even unit tests, low level tiers where you could catch many of those bugs and performance SLIs that are not met and that’s how it integrates a lot with performance. I always preach that you need to start to think about it as early as possible. And as Christine mentioned, getting the people together, what are the goals, what is the user gonna require, and what are the tasks? Because not everything should respond super fast and not everything should or can have a good response. I mean, we all wish that a—I don’t know, monthly billing period can be closed in three seconds, but that’s just for that response time from a click.

    So, we need to filter that out depending on what is the user needing, willing to endure, to call it in a way, and put it together, and this permeates as well. It’s some sort of functionality like—yeah, I need this to respond in this period of time for me to be comfortable using it, for it to be useful, functional.

    Shimel: Comments, thoughts from the team?

    Pugh: So, as you were mentioning that, I don’t know if you’re familiar with healthcare.gov.

    Melendez: [Laughter] Oh, yeah.

    Pugh: I think they should’ve had a few performance tests done well before they released it. It seems like—I don’t know, that it was like, maybe the day before, they went, “I guess we’re gonna have a lot of people on this.” [Laughter] 

    Melendez: Yeah, and I think, and to paint and explain what I was saying, with a single element that doesn’t perform well, that doesn’t work well, even from the initial tier, they should have created these requirements, saying, I don’t know, “When I log in and enroll, healthcare starts the process when I click this. It should respond this fast, it should not do, I don’t know, this many full table scans, connections to the database,” things that you can go way, way low level in the IT tier, and others and it’s just like, “Yeah, the user should not wait more than five seconds on that click”—a single user, one person.

    I am almost sure that most of the problems that they had was that they didn’t start to tune that, and they allowed the developers to check in all that awesome code, to call it euphemistically. And that’s when you start to scale up problematic things is when everything derails. And yeah, healthcare.gov, aside from not doing good load testing, they had multiple issues at the single piece level, and that’s where it is, push it left. Start to think at a unit level, single item requirements, when you have the three amigos together talking about these things, take everything into account and don’t let your little Post-its to be marked as done and move to the next area of your board without covering some of those basic requirements, which most people doesn’t think about or include them. Or whenever, and this is a little biddity or tiddity, when the developer is working on it, should have in mind all those requirements—functional, performance, security. 

    And that’s where I say, all this permeates to all the testing areas and other metrics when other areas of testing are doing their tests—manual, functional, automated, even security—my goal is to start to gather performance metrics so that everything happens together and we don’t have to wait until the end for big load tests to figure out that healthcare.gov will have what happened, sadly. [Laughter] 

    Pugh: So actually [Cross talk]—no, so, actually, you were talking about something that I’ve, in fact, I’ve done a few blogs on this, but it’s been around for a while is the testing matrix. I.e., that shows, since we’re talking about testing and it’s not just about functionality, it describes all the different sorts of tests, from the functionality tests which are on one side of the matrix, to the quality tests which are on the other side, including performance, security, usability and so forth. And that you’re testing—I mean, ATDD and BDD tend to concentrate on the functionality, because we’re gonna be talking with the customers, that’s what you want.

    But you ought to have all those other tests, the performance tests as you were mentioning, already established ahead of time—100,000 users, how fast should it be, and so forth. Oh, and in fact, just the usability you were talking about—how long should it be after I click that I get to the next page? And those things could be established ahead of time, and if a developer goes up and it’s not meeting it, he knows to fix it. That’s, we don’t have to—

    Melendez: And I would say here, not only established, but worked on from day one. Because as you mentioned, even some usability, security—those are to be checked when everything is assembled, and there are many techniques that we can start to embrace, starting out with automations at different tiers.

    As I said, I’m a big proponent or supporter of—yes, have unit testing, automated everywhere, have the coverage according to the pyramid. Don’t focus so much on manual, front end, or after the fact, leaving everything for the second Friday of the sprint when everyone wants to leave and here, tester, there you go—start our meeting. That’s where everyone has to start to work together and pushing it left.

    Since day one, start thinking about performance, security, functionals. And even some have told me, “Yeah, but there are things that, as you said, usability, that cannot be done until the end.” Well, you can start creating mocks, you can start figuring how things should work and start to work them. And, again, I think the mindset of keeping testing phased, waterfall ways—we need to start to work everything on par and get it rolling, yeah, I totally agree.

    Fisher: I’d like to speak to that point a little bit with DevOps and with the testing. We’ve been talking about a team and we’ve been talking about getting people involved, and what I was seeing in my organization when we first started talking about DevOps is, everybody is an engineer. We can all do these jobs, things like that, right? And jobs might change, there’s room for everybody, but what you’re doing might change a little bit. But that can be really scary to hear, and especially if you are a manual tester who really doesn’t have an inclination or desire to learn automation—and that’s okay. Like, that’s a different conversation, but I am always a proponent of manual testers.

    So, one of the reasons that really pushed me toward that is hearing that conversation, seeing people on my team being told that their job might change, but we don’t know how, you’re an engineer, you can do what everybody else on this team does, but that’s not true. Not everybody has those skills.

    And so, making sure that the team is involved, and when you can have these conversations, BDD, all of these things that we’re talking about now, right, are getting the testers involved in the conversation earlier, and that makes them part of the team. And we have taken our BAs, we’ve taken manual QA, we’ve taken our automation QA—we get them all involved in that conversation. We make sure the developers know where they are in the process, we make sure the developers are adding to the conversation and part of that process.

    And it was really important to me in that DevOps environment to get out in front of that and help define that role when somebody is saying, “We don’t”—you know, again, your role might change, but we’re not quite sure how—I wanted to help my team define that and put us in a good position so we weren’t just sitting there to have, God forbid, a developer come down a few years later and tell us, “No, you do it like this.” We wanted to put ourselves into that position and having those conversations, I think, is such an important part of that and changes some of that dialogue a little bit about we can all do the job. We can’t, but we all have important jobs to do to make it work.

    Ashley: You know, Christine, I think about just an analogy—robotics for surgery. I still want that doctor in there guiding what’s happening, right? We don’t need any bad things happening. [Laughter] Well, same with software. You know, some of the worst folks at testing software are developers, right? You’re just too close to it, even if you’re writing the automated tests.

    To your point, and I think several of you have made this, is, it’s an opportunity for us to (a) collaborate much more closely together rather than testing to be the thing right before documentation at the very, very, very end of the release, right, which is what it used to be. But also, what do we automate? It doesn’t help to automate the wrong tests—let’s get the right things in there. And where is the user in, the customer in performance, in functionality, in speed?

    By the way, one of the things that kind of baffled some development teams is when I would say, “Speed is quality.” They’re like, “What do you mean, speed is quality?” I’m like—well, if Netflix takes three seconds to start your video, okay, that’s good. If it’s pausing for three seconds every five minutes, that’s bad, right?

    So, speed makes a difference in terms of the perception of the experience. So, it is really kind of a harmonious effort where we have to bring all of that together.

    Melendez: And now that you mention it, I’m gonna play a little bit devil’s advocate towards and against at the same time manual. Some forums that I participate in here in Mexico talk about that a lot, like—hey, too much automation. Are we manual testers gonna be out of a job and the machines will replace us, and all those paranoid ways of speaking.

    And I generally explain to them with analogies—I like, a lot, to use analogies. In New York City, before there were any automobiles, cars, there were lots of horses, and horses produce a lot of stuff that was dropped on the streets, and there was a job needed for that, to pick up some of the horse. And I’m sorry if this is an analogy of manual to what the horse produces, but the point that I’m trying to do is that once cars became available, that job decreased considerably. But we still have places where there’s people that work with the horses, maintain them, brush them, and do all those tasks which, as many cars as we may have, we won’t ever have a horse polishing machine or anything like that. There are things that, truly, a human being is the only person that can do it really well—manual testers are very good for those tasks.

    I think, on the other hand, there are too many things that are tried to be done with manual testing. I have seen in many organizations that there are tasks and tests that could be pushed left, that could be started to be done before automated programmatically and leave whatever is truly necessary and where the manual testers are going to be the rock stars and the only ways to do it allow them to.

    So, I think—yes, I would suggest most manual testers to start to pick up a little bit on technical and developing, yadda yadda. I think in the modern days and the upcoming times, everyone should know some sort of development, that should be in schools. And we’re moving into a technological world at a pace that whoever doesn’t pick it up might be a bit handicapped. But, on the other hand, the skills of a good manual testers, when you have that clinical eye to say, “Hey, I see the error” or you don’t even have to touch the application; let’s say, most probably, it accepts fields that it shouldn’t in that field. And most of the times they are right, and you need that input.

    So, on both sides, they won’t be replaced, but they should get used to using the tools as they are. Technology is not here to replace them but to empower them, and they should be able to use it, I think.

    Fisher: Skills can be—excuse me. Skills can be enhanced, absolutely, and I agree with what you say. Nobody wants another meeting on their schedule, but on my team, we have added what we call a test strategy meeting and we get  the BA and the manual and the automation QAs in the room and we talk about each of our stories, and should we automate it, what are we looking for manually? And they took that further than when we first started, and they came up with a scoring guide to say, you know, what is our level of effort with this, what’s the repeatability for the user?

    Those types of things, and they’ve realized that those conversations are so important and you’re exactly right—the manual testers aren’t doing the same job, but now that we’re able to really give them the time to pull those skills out, the critical thinking and what you were mentioning, you know, looking at this field, we are getting so much more quality from our tests, from both sides, and it’s really been a fantastic change.

    Melendez: Yeah, that’s great, and optimizing and, in a way, calling it where you get the best bang for your buck, where to use those efforts, how to use them, prioritizing. And even some things that—hey, this is not a big impact if we get a defect here, we won’t even pay attention to it manually, whoever made it, or some others that this is critical if we get a problem that has to be done manually with two pairs of human eyes, capable eyes, to identify it, or we can let the machine do it.

    I mean, there are balancing on testing, and I think falling into the cognition mental fallacy of, it’s called man with a hammer—someone learns how to use a hammer well and will see nails everywhere and try to hammer them. There are multiple other tools that we can use when it’s needed, when it’s more optimal, and I think it’s, as you say, prioritize, distribute, and make the best effort with each one of them.

    Pugh: So, just getting into that, one of the things in the matrix and one thing I always suggest is that exploratory testing. You take your manual testers and turn them into exploratory testers—finding those things that the functionality tests are not gonna cover.

    I always give an example, for those who don’t know exploratory testing, of—you’re on the web, you’re ordering an item. You press the submit button and the screen comes up and says, “Do not hit the back button. Do not refresh this page.” And the exploratory tester goes, “Yeah, back, forward, back, forward, refresh, refresh—maybe I’ll get it for free. I don’t know!”

    Ashley: [Laughter]

    Pugh: There’s no way of telling. [Laughter]

    Shimel: We live for those moments—we get it.

    Pugh: [Laughter] 

    Ashley: Yeah, don’t make any buttons red, they’ll be pressed, for sure.

    Shimel: Right, yeah. But let me kinda take our conversation a little bit of a way, here. You know, we’re 25, 30 minutes in. We haven’t spoken a lot about DevOps, which is interesting, right? Especially in light of our original conversation off camera.

    You know, when we talk about behavioral-driven testing and ADD testing and performance testing, these testing disciplines all existed before DevOps, right? Ken, you’re doing them for a fifth of a century or more, right?

    Pugh: [Laughter] 

    Shimel: That certainly predates what we call DevOps. And I was here when I started devopsy.kinsta.cloud back in 2013-’14. There were a lot of testers who were running around like, you know, Henny Penny, “The sky is falling,” that they were gonna be out of a job, they were gonna be obsolete overnight. And, of course, that hasn’t happened.

    Christine, you said that DevOps was kind of a driving factor in why you took the baton, here, why you took this position, why you feel it’s important. I’d like to bring it back, our conversation, a little bit to DevOps. How has DevOps changed BDD, ADD, right? Automation? Absolutely, right, but automation is not necessarily DevOps, right? We automated. We’ve mentioned a little shift left, you know, pushing it further, not pushing it further on the developers’ shoulders, because as, Ken, you said, we don’t meet the new boss same as the old boss, right? We don’t need that. 

    But how can we be more efficient, better at testing? Is it just a matter of automation, or is there something else there? Christine, I’m gonna ask you first, because it’s something that I think is near and dear to your mission.

    Fisher: Sure. And I think for me, really, it is the conversations, how long has QA sat outside of the development organization, right? I mean, we can bring them in if we talk about agile and scrum and things like that—even, again, before DevOps, we can have a tester embedded in that team, but we still have lots of organizations that don’t do that.

    Realizing that quality, not testing, that quality is the responsibility of the entire team, and that everybody needs to contribute to that conversation, we talk about breaking down the silos and everything with DevOps and we’re all in this together, and that includes testing. And for me, that’s been the biggest part. We can talk about BDD or whatever we want to talk about, however we want to test something. The most important part to me is that the testers are at the table and that they’re part of that conversation.

    Shimel: And I think that’s inherently part of the DevOps cultural kinda mantra is that everyone should have a seat at the table. Ken?

    Pugh: So, one of the things, getting back to the DevOps, is the fact that every time you have a test failure after something has been actually developed and put into the pipeline, it means you’ve gotta loop back. It means that you are slowing down the flow of business value through that pipeline.

    So, the reason we want to create all these tests first is so that we pass them, and when it goes into deployment or staging, wherever you put it, that you don’t get a test failure. You don’t have to triage everything. And so, that’s why creating all of the tests first that the developers can be running while they’re developing their code or in their development environment is so important.

    Now, I’d just like to add one more little phrase that I always use or talk about is—requirements should have a test for it, right? Pretty straightforward. How do you know whether the requirement is actually being met? But every test is a requirement. If you cannot ship a system with a failing test, then that test is a requirement the system must meet.

    One of the things that developers always complain about is not having enough requirements, unclear requirements. The test that you create prior to implementation—BDD, ATDD or however you get ‘em—are the requirements. And now the developers have the test that they can go against those requirements, make it past, and then the flow through the pipeline doesn’t stop. It doesn’t, you don’t have a blockage because of failure later on.

    Shimel: Fair. Comments on that?

    Melendez: I think catching those errors, defining them, and being a bit more synergistic, to call it, in a way, it’s crucial. You very well mentioned all the testing practices, areas, and disciplines that exist come from waterfall and even before.

    But I think a shift that was forced by DevOps and the agile methodology, continuous releases forced us to think in a different way. And I don’t know if it was the chicken or the egg where we started to see that we needed to throw down these silos and as well that the communication channels and capabilities to observe what was happening. Because as a tester, 15 years ago, I remember, you would just receive something finished, you had no idea, and somehow, you had some peculiarly written requirements that you had to write tests for and try to figure out, but everything was already packed up, backed up, ready to ship. And you were just trying to stop, like, a meteorite that was about to hit, and you were like—I don’t even have control. I detect something, I send it before a production release, and I have this box that I detected. I’m sending them somewhere, probably the developers were even gone. 

    All those situations that we experienced before on any testing area that now that everything is happening continuously, fast, we have feedback, we have—we don’t own only the development side, the testing side, the operations side, but we own all the loop, we’re involved, and hopefully, we have communication with each one of those areas. And I would say not communication that the lines that divided us are blurry now. We are just multi-capped T-shaped people that can jump into, “Hey, now, this is a deal with development on Ops, I understand that this is happening, we have logs. We have different release methodologies.”

    I think DevOps and the openness that is happening as well as, from my perspective performance that I have some tools now that allow me to see into the code, to monitor it, telemetry—all these fun things that we can work with have allowed us to evolve and to integrate as, it was very well said, from clear requirements, clearly knowing what is needed. If we detect something wherever, not only before releasing to production, but now it’s in Ops, there’s a bug that was not caught and we catch it right away, we roll back or we release, quickly, a fix—those things were forced, given those things. Or I don’t know if forced—again, I don’t know if it’s a chicken and an egg situation, what came first. If now we have more visibility and capabilities, we start to release more often and communicate development and operations.

    But all this synergy has to happen and is needed for us to truly embrace DevOps, have quality in it, and keep releasing often with quality.

    Ashley: Alan, I have a question I’d like to pose to the team about DevOps. We don’t do DevOps just for DevOps’ sake, right, there’s usually some drivers behind it. And when you think about the conditions we’re in today, economic conditions, changing markets, rebuilding our supply chains, contactless services—there’s a great example—in days and a few weeks, businesses had to respond. Point being, one of the things businesses are looking for from their technology, from their software, is agility—the ability respond quickly, experiment, go after opportunity, defensive, whatever it might be.

    From a testing standpoint, how do you think that’s changed how you respond and how you do the work that you do, knowing that there are situations you have to react fast, and how has that changed how you work? Any thoughts on that?

    Melendez: Well—sorry, go on. I have played too much today. [Laughter]

    Pugh: Well, let me just give an example of a current situation. There is one place that I was consulting with—their manual testing effort took 5,000 hours to release a product. You’re not gonna be very agile if it takes you 5,000 hours, that’s 50 testers working over a month, and I don’t even know what they did, if they found a defect, whether they restarted again.

    So, the whole concept in being agile is, you need, as we talked about all the automation around everything, because if I’m gonna make a little change to something, I wanna make sure I haven’t broken anything else. So, I need the automation that matches exactly what the requirements are so that, then, I can run it through the pipeline. If I don’t have the automation, I have to pause for three days for manual testing, I’m not gonna get things out pretty quickly.

    Fisher: And to build onto that point with BDD, you can see so clearly where you have a failure in your test. When I joined my current organization, they had just really started with automation, and there wasn’t a lot of direction there—which is fine, that is not any kind of a criticism. But those tests got so complicated and only the people who were writing the tests knew what was happening in them and where those failures were and troubleshooting a failure could take just as long as troubleshooting the code.

    So, taking something like BDD, we’re gonna build on the agility, we wanna make it open to the team, the manual tester can find out where that failure was, a developer can find out where that failure was, and then we can very easily test to see—test the test to find out if it was a problem with the test or if it’s a problem in the code and then send it to the appropriate place to get it addressed. I think that that piece has just been a huge improvement for what’s been happening.

    Melendez: And I want to add, as well, from what you mentioned, what’s changed from the old waterfall days to how we are working now with the new trends and everything—I think the first problem that I personally experienced, and I fought against this agile B stuff that started to happen. “Why don’t you allow me to still do automated, big-ass load tests every, before every release? How am I going to be able to keep up with this if I have constant releases every two weeks?”

    And here, I think the biggest thing is a change of mindset, because many of us—and again, I said this myself, I was indoctrinated into how the waterfall days and silo-based release cycles happened. And I cannot even call it cycles, it was just like the big bang release and that was it and you’re out. And you used to think that way. You had to do big cycles of automation that you could execute that took hundreds and hundreds of hours, people involved. And now you need to think about, how can I tackle this now that it’s supposed to happen so often? 

    But a big change is that it’s not so massive any more. We are not doing a whole full release that we need to do multiple regressions here and there and everywhere to make sure. Now we can just say, “Hey, this sprint, I’m just gonna release this tiny, little, little feature that we can just quickly test. Something breaks, something happens, we are sure that our team, ourselves included, can switch direction, detect where is the issue, and implement the fix. And probably tomorrow, if we are super efficient, super quick, and doing things differently, we can recover from it.

    As you mentioned earlier, when there’s tons of tests that you need to execute and there’s an error that you detect, it’s there. There’s a bug, but we want to release every week, every two sprints—sometimes even modern mindsets can be, “Well, let it go to production. It’s gonna pester us for a day or two, but we know that we can quickly apply a fix, release it. Let it pass. It will cause a tiny bit of problems, and even change the course, change the direction if you need it, like, hey, that’s a feature we don’t need, we can take it off.”

    So, all that mindset changing on how you approach testing with this has to change. It’s very different, and I work with lots of customers where I jump in and I’m like, “Yeah, why are you still doing these things that are a decade, two decades old where all this DevOps, continuous, agile is already happening and you’re not even trying to do agile, but you have this big bang release, huge pieces of functionality that you want to push, that’s not gonna help you. And you need to be paying attention constantly and enable that you can recover. Your testing needs to be aimed at enabling that tiny functionality tests everywhere and just relevant to what you are doing.” It’s a complete switch of mind of what was testing before and what it is today.

    Ashley: Fascinating.

    Shimel: Nice. Go ahead, Mitchell, I’m sorry.

    Ashley: Just that idea of being able to release software quickly, right, instead of 6 months or 12 months—that’s a mind boggling shift, right there. Sorry, Alan, you were taking—

    Shimel: No, no, I’m actually saying I think we’re about out of time. I was gonna ask us to wrap up—I was gonna wrap up, but you know what, Mitchell, I gave you the last word, that’s great.

    Hey, guys, as I said earlier on, the time does go really quick. We’re out of time for this episode. I’d love to have you all back, though, and maybe we can continue this conversation. You know, I find it fascinating as someone who doesn’t come from the testing world, right, that testing in its infinite variety has been so profoundly impacted by DevOps and agile and these technologies. But rather than being left behind, it’s been accelerated, right? Through automation and speed and, quite frankly, more focus, in many ways, DevOps has been one of the best things that’s ever happened to testing, right?

    And that’s the thought I want to leave you with out there or DevOps Unbound. This is Alan Shimel, we’ll see you on our next show. We have a great roundtable coming up, by the way. Check DevOpsUnbound.com for all the latest. Thanks, everyone. Thank you to our panel. Mitchell, as always, thank you. Have a great day.

    Pugh: Bye.

    Fisher: Thank you.

    Melendez: Gracias.

  • EP 7: Accessibility and Usability

    EP 7: Accessibility and Usability

    In episode 7, we discuss strategies for incorporating accessibility and usability into DevOps and software creation processes.

  • Is GitOps Replacing DevOps?

    Is GitOps Replacing DevOps?

    Roundtable 7: Much of what we think of as DevOps is happening now within GitOps. Is GitOps the next stepping stone from DevOps? Will organizations go directly to GitOps? What does this mean for organizations working to implement CI/CD?

  • Avoiding SolarWinds-Like Threats: The Impact Across DevOps Teams

    Avoiding SolarWinds-Like Threats: The Impact Across DevOps Teams

    Roundtable 6: The SolarWinds and FireEye compromise demonstrated the devastating effectiveness of a supply chain cybersecurity attack. How should CISOs, cybersecurity organizations and application delivery professionals prepare? Will suppliers be required to disclose or validate supply chain security measures?

  • Day 2 DevOps

    Day 2 DevOps

    Roundtable 5: Most enterprises are either contemplating or have already begun their DevOps adoption/digital transformation, but once started, the next logical question is “What’s next?” Welcome to Day 2 DevOps, now…what’s next?

  • The Reality of Testing in DevOps

    The Reality of Testing in DevOps

    Roundtable 4: Our panel discusses the true current state of the art for testing and test automation when utilizing DevOps. The discussion will include addressing questions such as; What is the state of art? Is that achievable by most organizations? Are we getting all from testing in DevOps we could be getting? What can we do to make that state of the art achievable by more organizations?

  • DevEverythingOps: Tackling the Top Challenges Together

    DevEverythingOps: Tackling the Top Challenges Together

    Roundtable 1: We won’t bore you with all the reasons DevOps requires alignment across business, development, testing, security, and of course operations. That’s established. But the real question is how to coordinate all these elements at the enterprise level—beyond the high-performing DevOps native teams and across the ones with vastly different cultures, application stacks, and compliance requirements. Join us in this first DevOps Unbound Roundtable to learn hear from the leaders across all aspects of BizDevTestSecOps.