Category: Continuous Delivery

I recently had the chance to sit down and speak with Steve Hazel, CTO and co-founder of Sauce Labs. With so much being written and talked about the role of quality assurance (QA) and testing in a continuous integration/continuous delivery (CI/CD) world, it is good to hear from someone like Steve who is an actual thought leader in the space. Steve tells us what he thinks the limits of automated testing are, the future of QA and more. Have a listen for yourself to find out, or follow along with the printed transcript below!

 

Alan Shimel: Hi, everyone. Alan Shimel, devopsy.kinsta.cloud here, for another DevOps Chat. Today’s DevOps Chat’s guest—that was a tough one—is Steven Hazel, co-founder, CTO at Sauce Labs. Steven, welcome to DevOps chat.

Steve Hazel: Hi, Alan. Glad to be here.

Shimel: Thank you. Glad to have you here, actually. So Steven, there might be some people listening who maybe aren’t familiar with Sauce Labs. So, before we start, why don’t we, if you don’t mind, give our listeners a quick overview? Tell us about Sauce Labs.

Hazel: Yes. At Sauce Labs, we do automated testing for both web and mobile apps. We run a cloud service that gives our customers access to a wide range of platforms, whether that’s operating systems and browsers for web testing, or mobile devices and emulators and simulators for mobile testing. You can get access to all the different platforms you need to test on, and you can get access to a lot of platforms at once to run your tests in parallel to get through a large test suite rapidly, which is really what people need for today’s continuous integration and continuous delivery workflows.

Shimel: Got it. Of course, Steven, Sauce Labs was closely linked with around Selenium and some of the open-source testing tools that are out there today, right?

Hazel: That’s right. Selenium is the dominant web-testing technology. In mobile testing, there’s a variety of technologies, but we support Appium, which also uses the web driver protocol that underlies Selenium. We really think that fitting in with those open-source tools is the smart way to go. It fits with what people are doing.

It gives people a lot of choices about how they want to do their testing, what kind of languages and frameworks they want to use on their end. So, we just plug right into Selenium tests, if you’ve already got Selenium tests, or Appium tests. You can just change the config and point them at Sauce Labs, and they’ll run unmodified on our systems.

Shimel: Got it. Steven, we were talking before we started recording today, I was telling you about a recent article we had over on devopsy.kinsta.cloud. The gist of it was basically that DevOps was the death knell for QA. I told you I didn’t agree with that and you agreed with me. You said you didn’t agree with it, either. But for our audience—they’ll respect you a lot more than they respect me—why don’t you agree with that, Steven?

Hazel: Yeah. I think that to suggest that there’s no longer a role for manual QA when you take on a DevOps process is just not in line with the facts. If you look at what we see large organizations doing, they’re not dropping manual QA for automation. They’re changing the mixture of what they have manual testers doing by quite a lot, but they still have manual testing teams. You still need a manual testing team.

So, what I see happening, as you start to take on automation, your testing coverage improves dramatically because you can afford to do a lot more testing when you’re automating than was ever possible with a manual testing team. But automated testing, the classic example is exploratory testing. You’re just not gonna get the kind of creative, exploratory walk-through of your app with an automated test that a manual tester can do. I think there’s still a role for QA organizations, beyond just manual testing, in being the other side of the conversation, right?

A lot of people have the concern that if developers are writing tests, it’s the inmates running the asylum. The exact developers writing the software are the ones deciding whether it’s high quality, and you want another perspective in there. You want a perspective that’s maybe closer to your end user. So, I think that the other side, beyond just manual testing, that you see with QA teams continuing to have a role is in deciding what cases should be tested, whether you have good test coverage, encouraging good testing practices, even when you are automating your tests.

Shimel: Absolutely. Really, Steven, I guess the boogeyman in the room around the friction with DevOps and QA is automation. There’s some schools of thought that says, “Well, we can automate everything. The more we automate, the better we are.” There’s another school of thought that says, “Well, you can automate as much as you can and automation’s not a bad thing, but there are just gonna be some things that are not automatable, if that’s a word now, or may never be automated. Where do you go in on that continuum?

Hazel: Well, being someone who’s pushing forward automated testing, I want to see automation go as far as we possibly can. I think that that’s very far. That said, I think that the way the mixture of tests changes as you adopt automation is maybe a little bit counterintuitive. You might imagine, well, we’ll automate some percentage of our manual test cases, maybe 80 percent or 90 percent. Maybe we could get to 100 percent if you’re a zealot for automation.

But actually, what you see is, it’s not that you’re automating a percentage of what was there in manual testing. You’ll actually have more automated test cases than you were able to do with manual testing. So, if you just looked at the number of test cases, your automated cases are gonna way outnumber the original count of manual test cases just because you can, right? You can now cover a lot more edge cases, a lot more scenarios because of the power of test automation. So, that’s a real advantage to test automation.

Shimel: Sure.

Hazel: At the same time, certainly there are things, visual testing, there’s some interesting approaches out there, but I don’t think we’ve yet seen a way of making sure that the app looks right, especially as it changes. You really need a human being involved when there are deliberate visual changes to make sure that they came out right. I think it’s those things where automation really can’t give you an answer about, does it look right? Does it act in a way that’s gonna be acceptable to somebody?

I think you still need a human being in there at some point, checking those things, but it’s a different process. It’s no longer this phase gate before every change is released that you used to have. It’s now a process that can be made asynchronous from the bulk of your development and release process.

Shimel: Absolutely. If you don’t mind, Steven, a little bit of time on, you mentioned a little bit, but what do you think of the kinds of tests that lend themselves well to automation? Certainly, a visual test of an interface, different arts. What are the sweet spots for automated testing?

Hazel: I think certainly at the bottom of the testing pyramid, unit testing brings tremendous value. It’s something that’s very easy to automate in a way that’s completely deterministic, very fast. Gives you really quick feedback as a developer as you’re making changes to the code on whether you’ve broken something. But the coverage with unit testing, the coverage of functionality that may be important to your end users, it’s really hard to achieve. It’s really hard to achieve good, solid feedback from your unit tests that you haven’t broken anything anywhere.

Functional testing, the next layer up, which is what really we focus on, gives you that complete end-to-end check. You’re actually testing the features of your product and you’re making sure that end to end, with all the various components involved, they work properly. That’s more of a challenge for automation, but we’re seeing, I think, a majority of development organizations out there seeing success automating functional tests and real value from having that be something where you can run through hundreds of test cases in minutes and get feedback to your team. So, that works quite well for automation.

I think it’s that next level up, as you start to get into the types of tests where you’re looking at, what’s an acceptable experience for the end user? User acceptance testing as distinguished from functional testing, where you’re really starting to look for, does it look right? Does it act right? Is the response time throughout the entire process acceptable?

You can zoom in and measure a lot of these things very specifically, but getting the answer to, “Overall, though, was it a good experience?” That’s the kind of thing where it’s really tough to cover that with automation.

Shimel: Yeah, agreed. Steven, if you don’t mind, I’m an old security guy prior to DevOps. One of the things within the security community that we’ve really pushed around is the idea of automated security testing shifting left in a DevOps environment and allowing security testing to be done in an automated fashion at a CI/CD-type of level. Where do you see that in this whole discussion?

Hazel: Security testing isn’t what we focus on at Sauce Labs, so –

Shimel: Right, I know.

Hazel: I’m not an expert in this, but it is something that I follow and care a lot about. I think I see a lot going on with static analysis, with some of these tools that can help to constrain the types of mistakes that you could have in a program, type systems and things along those lines. I think really that’s very badly needed in making code secure. Some of the issues we’ve seen come up over the last few years, things like Heartbleed, the Glibc bug, really those kinds of things aren’t acceptable, aren’t something that we can continue to have happen, and I think the solution really is fully automated tools, like static analysis tools and type systems, that can lock down the code and just constrain the types of mistakes that could possibly happen.

Shimel: Sure. You know, Steven, I see it as very analogous to the QA situation, which is I think a lot of security people get very territorial around job security because they’re afraid somehow that doing automated security testing earlier in the process—shifting left, so to speak—is somehow gonna affect their livelihoods, or make them obsolete.

Hazel: I can see that. We see that among QA folks, for sure. To me, that’s a shortsighted view. If the world is getting more intense security problems, more intense quality problems, we’re shipping more software, you just can’t solve it 100 percent by throwing bodies at the problem. I think that what’s gonna happen is not that those people’s careers are in jeopardy; instead, as a society, we’re just gonna do more and better security and quality testing than we’ve been able to do in the past.

Shimel: Yeah, agreed. Steven, we’re running a little low on time, so let me get just one or two more questions. Sauce Labs has been, in my opinion anyway, for what it’s worth, somewhat of a DevOps darling in terms of really encompassing—you’re living the DevOps model. You’re working with open-source tools, testing and automation, enabling automation. Sauce Labs is a player in the DevOps ecosystem. Where do you see it going?

Hazel: Well, toward the enterprise is one big trend we’re seeing. Earlier on in the history of Sauce Labs, a lot of our customers were essentially software companies, companies that their primary focus is technology. Now, a lot of our customers are retailers, financial institutions—companies that you don’t think of as primarily a software company, but all of these companies have apps, they have complex web experiences. They’re really adopting DevOps in a big way. They’re moving to what is, for them, a very different process than what they’ve been doing and allows them to ship better software faster, with more complexity than they were able to handle before.

Shimel: Absolutely. Steven, we are low on time. I’m gonna ask you the final question, which I ask most all of my guests here on DevOps Chats. That is, if you had to recommend one book to our audience that they should read, a must-read for them, over the next six months or a year, what would that book be?

Hazel: Oh, wow. A book recommendation. I think the obvious choice in a conversation about DevOps is “The Phoenix Project.” If you haven’t read it, you’ve got to read it. It’s a quick, enjoyable read, but it really gets the concept across better than probably anything else is going to. It’s an obvious recommendation, but I still think it’s worth it.

Shimel: Sometimes the obvious is good. I will just add to that, if you already have read “The Phoenix Project,” Gene Kim’s next book, I think co-written with John Willis and Jez Humble and someone else that I’m blanking on – oh, darn it, the guy who started DevOps, who coined the phrase. Patrick. Patrick Debois. Their new book, I think it’s called “The DevOps Handbook,” will be out, I want to say, in the fall. Not a sequel to “The Phoenix Project,” but another good book to look at.

Anyway, Steven Hazel, CTO, co-founder of Sauce Labs, thanks for being this episode’s guest on DevOps Chat, and continued success with Sauce Labs.

Hazel: Thanks a bunch. Thanks for having me.