devopsy.kinsta.cloud

Tag: continuous testing

  • Accelerating Software Testing

    Accelerating Software Testing

    Yves Junqueira is the CEO of YourBase, a software testing accelerator for large codebases. He talks with Alan Shimel about the company, accelerating software testing and the impact on developer productivity. The video is below, followed by a transcript of the conversation.

    Alan Shimel: Hey, everyone. Thanks for joining us here on another segment for our TechStrong TV. My guest this segment is a new company, first-time guest on our TechStrong TV, Yves – and I’m gonna really mispronounce his last name, Junqueira. I’m supposed to have that rolling R, and I just don’t do rolling R’s. Yves, I’m sorry, man. Welcome to TechStrong TV.

    Yves Junqueira: Thank you, Alan. It’s great to be here. You did it good.

    Shimel: Not so good. You know what? In case your mom or your family’s watching, say your name correctly for ’em.

    Junqueira: Yves Junqueira.

    Shimel: Junqueira. Okay. We’re gonna call you Yves, and we go from there.

    Junqueira: Sounds perfect.

    Shimel: So, looking from your background, are you up in Seattle, Toronto? Toronto?

    Junqueira: That’s Seattle.

    Shimel: Seattle. Yes, the mountains. And you know what? Welcome. I’m gonna let you give a little introduction to your company, and you’re a cofounder of it, but go ahead.

    Junqueira: Gladly, yeah. Our company’s called YourBase. We build developer tools for very large code bases. Specifically, we are a software-testing accelerator for large code bases. I used to work at Google for almost ten years, and my cofounder, John, was a principal engineer at Amazon, at AWS. And, yeah, we’re excited to – I’m very excited to be here, Alan.

    Shimel: Absolutely. Well, I’m excited to hear about what you’re doing. So you’re a code accelerator?

    Junqueira: We are a software-test accelerator, yeah.

    Shimel: Software-testing accelerator, excuse me. What exactly is that?

    Junqueira: Yeah, so it’s very important when you’re building software that you actually know that the software is working. You usually have very large teams that’re creating code, and you don’t want one developer to be stepping on each other – on the other’s toes. And so you need a lot of testing to make sure that when you make changes to the software, it’s still working. But as you’re adding new features, you’re adding new tests, and you add new features, and you add new tests, and over time the number of tests in your code base can grow very, very quickly, and it can become a very big problem for developer productivity and developer happiness, because it can take a very long time to actually test code. If you’re making even a small change sometimes in a large code base, it can take hours to ensure that that change works, to run all the tests. So that’s what we help with.

    Shimel: So you’re automating testing. Is that fair?

    Junqueira: We automate the test selection of testing. So, yeah, we help run the tests, but most importantly we help select what are the right tests that are safe but that have to be run for each change. Without having to test the entire world, we use a process called – we use tracing of code to build a dependency graph to then select the right test to run and then just make things more productive for developers.

    Shimel: Sure. Of course, our audience is a very technical audience, familiar with QA and testing and developing. So you’re running a trace route on the code as it’s entered into Git or – because it’s pre-deployment code, right?

    Junqueira: That’s right. It’s pre-deployment code, yeah.

    Shimel: So as the code is put into Git or some other sorta repository, you select what’re the right tests to run and run it on the code. So this all takes place within the repository like a Git-ops kind of thing?

    Junqueira: Yeah. It can take place in the repository connected with a CI, with a continuous-integration tool, or it can also happen during local development. You can have multiple developers working at the same time on the same code base but perhaps making slight – different code changes. And with your base, they are testing their code but also using their colleagues’ testing to – so that they don’t have to repeat the test that their colleagues already ran for the same code base. So they only have to test exact the small changes that they made on their code.

    Shimel: I love it. Sounds great. And then I don’t wanna use the buzzwords, but is it some sort of AI/ML that is picking what’s the right code – what’s the right test to run for this particular code for these particular changes?

    Junqueira: That’s an important question. It’s not AI. We use just simple data structure and algorithms that we collect when doing tracing. And this is actually important, because if we were using just AI or any sort of statistical approach, we would only get a limited confidence that we are testing the right things. So imagine that you’re a financial institution, right, or a hedge fund, where tests are very important. If you don’t catch certain bug, it can cause millions of dollars in losses, right, in lost revenue, for example. So it’s very important that you actually detect the correct regressions in your code, and if we were using just a simple AI approach, we would only, let’s say, have 90 percent confidence that we were running the right tests in your code. That is not enough. So our customers need assurance. They need full confidence, so we use an approach that is a little bit more intrusive, let’s say, because it’s detecting every single function called that is happening during the tests. So it has a little bit of a performance impact during the initial test. But on the other hand, it just provides full confidence rather than being 90 percent confidence that an AI approach would have.

    Shimel: Got it. I mean, big companies using this, small companies, anyone? Who’s the target user here?

    Junqueira:Fortunately, we can scale from small to very large code bases. So we have customers that are very small, just small startups that are getting started but that really want to have the fastest possible development process. But we also have very large code bases of very large companies, public companies, Fortune 500s, that want to keep moving fast. It’s difficult. Imagine if you have a software product that really needs to evolve quickly. Customers these days demand that software developers – that software companies evolve their products quickly. The customer needs are changing fast, and so they need to be able to make software changes very quickly and do it at – have high velocity with the most productivity possible. So what we allow those companies to have is even though they have a very large code base, the larger companies, they can be almost as productive or as productive as if they had a small code base, and this is really magical. This is something that we’re very proud of providing to these companies.

    Shimel: Absolutely. You know what? People always wanna know this, and I realize it might be coming outta left field. Interesting. You worked at Google ten years. Your cofounder’s an AWS guy. How did that come together?

    Junqueira: Yeah. It’s actually a funny story. We met at a random coffee shop. After I had left Google, I was working at random coffee shops here in the outskirts of Seattle, and I had my laptop with me with a bunch of developer stickers there like Linux and Python and GOAL like developers often have. And John had the same, and so we just looked at each other’s laptop. We were at this coffee shop here, and we – okay, we should chat, ’cause we us a lot of the same technologies. So we started talking, and over time I invited John to come to my weekly table games at my house. I host a game night every week, and John started coming, and I started talking about my project, and he was fascinated by it, and he said that software testing and developer productivity are a huge thing for him. And over time, he decided to join. I’m very lucky that he joined me at YourBase.

    Shimel: Love it. And then talk a little bit here about going to market. How’s it been so far?

    Junqueira: Yeah. We spent a long time perfecting the technology. This is a very tricky software. If you get things wrong, you may end up – your customers may end up shipping bugs. My customer’s gonna end up shipping bugs that should have been caught by the tests. But if I don’t decide to run a test or if I don’t give them right insights, they might actually introduce bugs and be less reliable than they were before. But we started just last month. We launched our first public beta now, and we’re very happy about that. So we have a number of design partners that are working with us on this initial release, and people – customers are very happy. The feedback has been fantastic. So last month we launched it to the outside world, and we made a public beta. Now people can just download it from just going to the website, yourbase.io. Anyone can download and test it. We currently support Python as – for code bases. Our technology has to be implemented on a language-by-language basis. So for now we support Python, but we’re going to support Ruby sometime soon. And we’re very excited. The reception from people has been amazing. It’s really fun to be at this stage of the company, where we actually are ready to talk about it to the outside world.

    Shimel: Excellent, man. It is. Having founded more than one company, more than four companies, five companies, it is an exciting time, and I wish you nothing but great luck and success with this. I want to get into the kinds of testing you’re doing, functional testing, unit testing. Is it selenium kinda stuff? What are the kinds of tests you’re doing there?

    Junqueira: That’s a great question. We are very good at tests that don’t cross process boundaries for now. This can be sometimes unit tests and functional tests. We don’t currently support UI tests yet, in part because there are good companies out there that’re already serving this market, and there was just nobody that was focused on the backend and in unit testing and functional testing. But this is a big space, and we’re just getting started. We have feedback from folks that we – they want support for more things, so that’s something that we’re excited to work on.

    Shimel: Excellent, man. Yves, I wanna thank you for coming on and telling folks about YourBase here on TechStrong TV. Come back. Keep us posted on your progress, and best of luck.

    Junqueira: Thank you so much, Alan. It was great to be here, and, yes, absolutely, I would love to come back.

    Shimel: My pleasure. Say your name for us one more time so I don’t – just after I mangle it, at least they got it right.

    Junqueira: Yves Junqueira.

    Shimel: Junqueira. Yves, thank you, man.

    Junqueira: Thank you, Alan.

    Shimel: Oh, the website, yourbase.com?

    Junqueira: Yourbase.io.

    Shimel: IO, yourbase.io, Y-O-U-R-B-A-S-E-dot-I-O. This is Alan. We’re gonna take a break. We’ll be right back with our next guest.

    [End of Audio]

  • The Pace of Change Requires Better Testing

    The Pace of Change Requires Better Testing

    The COVID-19 pandemic in 2020 created a pressing need to give people additional ways to do their jobs and handle the tasks of day-to-day life via mobile devices and forced the growth of app innovation at a never-before-seen rate. Thanks to new application development technologies and agile processes along with open source and embedded libraries, organizations are delivering greater functionality, faster, and for multiple devices (think web, mobile, tablet, etc.). And while a never-ending stream of new apps and updates has been great for consumers, it’s proven to be a pain point for many enterprises whose testing processes aren’t up to speed, literally and figuratively.

    While test automation is an essential component of agile development, it’s not addressed in published agile methodologies such as Scrum and Kanban. Of course, testing tools and methodologies have changed along with modern development practices, but they haven’t always kept up with today’s rapid-fire pace of AppDev, and the past 18 months have only served to elevate the need for improved testing. For companies to keep up with and move beyond their competition, testing must be conducted in parallel with development.

    Automated and Continuous Testing: Analogous, not Synonymous

    There are many ways to test software, including manual, user interface (UI), automated and continuous testing (CT), but most enterprises today employ some form of automated testing and, increasingly, continuous testing. All too often, people throw the terms automated and continuous around as if they are synonymous, but they aren’t. So, before we delve into the merits of one over the other, it’s important to have a clear understanding of what each of the terms means.

    Automated testing refers to conducting software tests programmatically with little to no human interaction. It measures actual outcomes in relation to predicted outcomes, and, as it’s less time-consuming to execute than manual testing, has become the norm for modern enterprises. Continuous testing, meanwhile, uses automated tests to provide uninterrupted feedback at every stage of the software development life cycle and, therefore, plays a key role in DevOps.

    Automated and continuous testing work hand-in-hand to ensure software quality, cost-savings and speed time to market and should be conducted in lockstep with development. Whereas it’s possible to have one without the other, it’s far from advisable, however.

    The Upside of Continuous Testing

    Continuous testing takes automation to the next level and, as such, holds several advantages over automated testing alone.

    • Early and actionable feedback at each development stage means issues can be addressed sooner—before they evolve into larger, more complex issues.
    • Agile teams’ velocity is measured by how much they can complete in a sprint. Because of this, a team’s testing focus is narrowed to the sprint they are immersed in, meaning that problems affecting functionality previously deployed often aren’t discovered until later in the development life cycle, or they are discovered by users after deployment.
    • An automated and continuous approach means more problems are identified and fixed before deployment.
    • Release cycles are accelerated as organizations can respond to market changes faster and in real-time.
    • Correcting problems early reduces rework, which translates into cost and time savings.
    • Continuous integration requires continuous testing.
    • CT tests not only the parts of the application that have changed but also how those updates work within the app’s entire functionality.
    • Unlike automated testing, CT requires no human intervention.

    What’s the Holdup?

    Despite numerous benefits, CT isn’t as widespread as one would hope. One of the reasons for the disconnect comes down to investment, whether it’s hiring the right people, training or tools.

    Perhaps the greatest stumbling block comes at the hiring stage. CT requires a specific set of technical skills coupled with a natural aptitude for test development and a mindset focused on resiliency. Beyond investing in staff, there are upfront (and ongoing) automation costs to consider. Unfortunately, it’s not as if you can implement CT and never touch it again—CT becomes its own software delivery pipeline that must be developed, maintained and enhanced to ensure it doesn’t become obsolete.

    Continuous Testing: What Could go Wrong?

    As the pace of AppDev quickens, so does the imperative to integrate it into DevOps. Failure to do so has serious implications, the most obvious of which is that without CT, companies fail to test as much and as frequently as they should. Regression tests aren’t done as deeply, for instance, so when a new app or update is released, it’s more likely that something will break. Further, without the right skill sets, teams have a tendency to examine whatever is new and fail to see how it fits into the whole. Moreover, without proper testing, you may inadvertently introduce bugs or discover too late that functionality doesn’t work as planned. Relying on manual and slow, fragile UI testing will only hinder progress in organizations that don’t adapt.

    Negative consequences go beyond the confines of an agile team or enterprise as poor software design and implementation feed into the all-important issue of customer (dis)satisfaction. As everyone knows, unhappy customers aren’t loyal customers, and that translates to a lower bottom line. Another equally serious (perhaps more serious, depending on your perspective) ramification is the potential impact on data security, exposing organizations to a very different but significant risk posture.

    Thanks to DevOps, ideas are moving from deployment to business value at lightning speed. It stands to reason, then, that testing also must be faster than ever.

    When you consider that what you see today is the least amount of change you’re going to experience and that tomorrow is guaranteed to bring more, it should be abundantly clear why continuous testing is so important: The less testing you do, the more problems will occur downstream.

    Businesses develop software for a reason—to get business value. If your app isn’t working, you aren’t reaping its full value. So, instead of asking yourself if you should implement continuous testing, you need to be asking yourself why you haven’t done it already. By failing to invest in CT now, you will undoubtedly pay for it later.

  • A New DevOps Test Environment Platform is Needed

    A New DevOps Test Environment Platform is Needed

    Test environment management is bottlenecking DevOps transformations and is driving the need for a new test environment management platform solution. On-demand test environments in which developers and testers do not have to wait for a shared environment are sorely needed.

    The evidence is clear. In my book Engineering DevOps I indicated that finding and setting up test environments is a primary source of bottlenecks in DevOps value streams.

    In my February 2021 webinar with Plutora, I presented Best Practices for Developing Test Management Environments. I indicated that orchestrating and managing test environments is often a primary source of delays and waste within value streams.

    The State of Testing Report 2020 by PractiTest indicates that DevOps practices increased 57%; from 26% in 2017 to 41% in 2020. The report revealed that 46% of testing has shifted to using CI/CD processes, 55% are actively involved in shifting testing left and 36% are testing in production. This is clear evidence that DevOps testing is on the rise.

    Wolfgang Platz’s December 2019 article Why Software Testing Remains a Bottleneck indicates that testing is responsible for more delays than any other part of the development process – accounting for 52% of the bottlenecks in the development process. And  53% of respondents to the World Quality Report 2020-21 indicated that test environments are not available when needed, 67% lack the capability to book and manage test environments and 71% lack visibility of available test environments.

    An article by Velocity echoes the problems with pre-production environments. “With the rise of microservices and cloud-native applications, environments are becoming more complex. Often you need to wait or compete to get access to develop on a specific microservice, or can spend time waiting to deploy to a shared staging environment only to realize something already changed, and you’re back to square one.”

    The Continuous Testing Report – 2020 by Capgemini defines test environment management (TEM) as managing infrastructures, test data and applications to support testing on demand. These environments are one of the greatest impediments to continuous testing. In this report, 36% of respondents said they are spending more than half their time building and managing test environments; 44% of respondents indicate they spend weeks of their time searching, managing and generating test data.

    Test environments should be regarded as functions to be spun up or destroyed at will, with virtualization, infrastructure-as-code (IaC)  practices, service virtualization, containerization and improved test data management.

    Given all the above evidence, it is clear that a new approach to DevOps test environment management is needed.

    The following are examples of benefits that DevOps test environment management platforms need to offer:

    • Compatible with current tools:
      • Automatically map and configure application infrastructure to develop and test multiple services in an environment that runs just like it would in production. This needs to be fully agnostic to cloud and tech stacks, such as Kubernetes and other environments.
      • The platform needs to work on applications in parallel without disrupting others, with full access to data and third-party dependencies.
    • Automatic configuration and maintenance:
      • The platform needs to automatically configure and update any changes within the production infrastructure’s mapping in real-time so ephemeral environments will always be synced with what is in production. Instead of having to spend time making sure any changes are configured, the platform will ensure the right environment that replicates production, without having to wait.
    • Data and third-party dependencies:
      • The platform needs to load the relevant data and third-party resources that are vital parts of the environment to test, work and debug against, so test environments will, as closely as possible, mimic working in production.
    • Team collaboration:
      • The platform needs the capability to share ephemeral test environments with other stakeholders so they get visibility and can collaborate for faster feedback cycles. This is also important for collaborating for debugging.

    What This Means

    DevOps testing is on the rise, and test environments are a major bottleneck for DevOps testing. A new DevOps test environment management platform is needed to reduce the bottlenecks. On-demand test environments in which developers and testers do not have to wait for a shared test environment are needed. Such a solution would improve agility and reduce wasteful orchestration bottlenecks in test, improve quality and risk management through comprehensive and automated testing and enable continuous improvement by providing more intelligent insights and feedback.

  • EP 15: Why Testing is so Frustrating

    EP 15: Why Testing is so Frustrating

    In this episode of DevOps Unbound, hosts Alan Shimel and Mitch Ashley are joined by Adam Kalsey of Tricentis, Brian Dawson of the Linux Foundation and Caroline Wong of Cobalt to discuss why testing is considered the most frustrating bottleneck for developers and engineers. Panelists will explain the benefits of testing early on in the SDLC and how to accelerate the feedback cycle with automated and continuous testing, as well the pros and cons of testing, the value of shift-left testing and how to effectively balance quality and speed. The video is below, followed by a transcript.

    DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.

     

    Alan Shimel: Hey, everyone. I’m Alan Shimel, and you’re watching DevOps Unbound. DevOps Unbound is sponsored by our good friends at Tricentis, and it consists of a biweekly, 40, 45 minute video panel on relevant topics within the DevOps sphere, and featuring a panel of experts that rotates every other show, and really great people, and something I personally really enjoy doing.

    And then, about once a month or once every six weeks, we actually have a live studio audience—or there is no studio audiences in COVID, actually, but we have a live virtual audience that participates in our show. Today’s show is just our panel, though, without a live audience, and I’m really looking forward to our discussion.

    Our discussion today is around testing. You know, with the pressure on developers to deliver code faster, sometimes testing is considered a road bump in the way. Testing is considered slowing down the whole process. There are a lot of developers who’d like to push testing off that plate. Of course, no one wants to go back to testing after we’ve deployed software, primarily, but there is that tension there.

    Let me introduce you to our panel today; we’re gonna discuss this in depth. First of all, making his first appearance as a member of The Linux Foundation on our Tech Strong TV network is our old friend, Brian Dawson. Brian, welcome to DevOps Unbound.

    Brian Dawson: Thank you, Alan. Thank you, Alan. And it’s fitting that the first panel, first webinar I do since joining The Linux Foundation is with you.

    Shimel: It’s our pleasure to have you on. Brian, what’s your position at Linux Foundation now?

    Dawson: So, at The Linux Foundation, I am functioning as the VP of Developer Relations and Ecosystem Development. So, just to help, we’re gonna help The Linux Foundation to the development community while driving more of a DevOps opinion into a growing ecosystem of solutions that support open source development.

    Shimel: Fantastic. Next up is my good friend, Caroline Wong. Caroline is—oh, she’s an anchor on many of our different shows here at MediaOps and Tech Strong. Caroline, it’s great to see you again. A little background, maybe, for people who aren’t familiar with you?

    Caroline Wong: Great to be here. Thank you so much. My name’s Caroline Wong, I’m the Chief Strategy Officer at Cobalt. We’re a pen test as a service company.

    I actually started my DevSecOps career 16 years ago leading information security teams at eBay and Zynga. So, in both cases, we were running online operations 24/7 with millions of simultaneous users daily. And now, at Cobalt, we build security software. And, like many DevOps companies, we’ve got data driven, product based teams. We value automation and failing fast, and super excited or today’s conversation.

    Shimel: Fantastic. Welcome. Great to have you, Caroline. Our third panel member is another returning DevOps Unbound veteran, and that’s Adam Kalsey of Tricentis. Adam, welcome.

    Adam Kalsey: Thanks. Thanks for having me. I lead Developer Relations for Tricentis. I’ve been building web applications since 1995, and have always had a soft spot in my heart for bad applications and broken things. And so, I love to help developers build higher quality software and do it in a more efficient, faster way.

    Shimel: Absolutely, and thanks for being on with us today, Adam. Last but not least, in his Florida outfit here, visiting me in Florida, is my Co-host of DevOps Unbound, Mitch Ashley. Mitchell, welcome.

    Mitch Ashley: Coming to you from the MediaOps Studio B.

    Shimel: Yes.

    Ashley: Great to be here. I’m a former product, both security and software SaaS, CTO as well as a CIO, and now I run Accelerated Strategies Group, an analyst firm that focuses on those things.

    Shimel: Absolutely. Hey, Mitch, thank you.

    So, let’s get started. Adam, you’re with Tricentis, the sponsor of DevOps Unbound, and they’re a testing company, probably the leaders, one of the leaders worldwide in automated testing, continuous testing. Have we put too many straws on the developers’ camel’s back by making them, you know, be more responsible for testing to say we need to test more of our code earlier.

    Is it a bridge too far for many developers? Is it making them resent testing? You know, what’s the current state?

    Kalsey: I think where we’re failing developers is by handing them lots of more things to do without giving them any—without giving them what tools, and I don’t mean just necessarily software tools, but without giving them the things that they need to do it effectively. If we just pile things on developers and say, “Oh, this is your job now” but we don’t give them the support, we don’t give them the training, we don’t give them the expertise, we don’t give them anything to make that better—sure, it’s just gonna be harder on ‘em.

    Shimel: To play devil’s advocate—oh, those poor little testers making a quarter of a million dollars a year each. Do we want ‘em to do testing, too? Right? Is there some of that here, Brian? Or—go ahead, Adam?

    Kalsey: No, no, Brian—please, please, jump in.

    Shimel: Well, he’s from The Linux Foundation, he now represents these folks. [Laughter]

    Dawson: Well, first, not all of them make a quarter of a million dollars a year, right? But I do think what you raise is important when you bring up the investment, right? We are hiring talent with a particular expertise in a particular domain, a particular ability to solve problems and innovate.

    And there is an argument to be said, Alan, that, “Do I want to spend this money, this quarter of a million dollars plus 200,000 in RSUs on having somebody write test harnesses?” Now, I think—you know, that can be balanced by what Adam said. Well, first of all, yes, because we want everybody thinking about quality. But you know what, if we want a return on that investment, we wanna free the cognitive load, then you know what? We need to invest in automation, we need to invest in better collaboration and reducing friction from the process.

    Shimel: Agreed, agreed. Caroline, you know, it’s been a big—this whole DevSecOps movement and shift left has been about shifting security testing left and getting developers more involved in security testing, right? Yet another camel on the—yet another straw on the camel’s back. Should we be doing that?

    Wong: So, I—oh, my Internet is awful. Can you hear me?

    Shimel: Yes, we hear you fine.

    Wong: Okay, fantastic, awesome. I’ll just—I know that there’s post editing, so it’s okay for me to say that.

    Shimel: Mm-hmm.

    Wong: So, Alan, I bring a particularly unique perspective to this group in the sense that I actually oversee people at our company, I oversee Human Resources. And I’ve been with the company since we were 10 people, and today we’re 200. And one of the super interesting things that happens is, naturally, as a company grows, what people do, they specialize and then we have to figure out how to communicate more effectively.

    And I think that if you’ve got an engineering team and you say, “Okay, now you’re responsible for Development and you’re responsible for Ops, and your responsible for security, then there’s got to be some level, depending on the maturity and the size of the organization and what the organization’s trying to accomplish of specialized roles and protocols for engagement.”

    And I think that when, either number one, those roles are not clear to, I think Adam was making a point earlier about security folks need to be providing guidance to engineers where—because engineers are not expected to be security experts, frankly. Security experts are not expected to be engineers. And then, at the point where information is being handed off where the responsibility for taking an action, those, all of those sort of procedures, they have to be clear, otherwise they’re simply not going to happen.

    Shimel: Yeah, agreed. I think they have to be clear. But even if we make them as clear as we possibly can, is it just—are we putting too much on developers? I mean, I think fundamentally, that’s the question here, right?

    Wong: There’s—

    Ashley: Alan, it’s not only—sorry, I’m just gonna jump in. It’s not only quantity of work—is it the right work?

    Shimel: Right.

    Ashley: Because I know from writing software and being part of software teams for so many years, there’s this thing called unintended consequences or unintended use. Well, you’re not supposed to use it that way, but the person who designed it intended it to be used a certain way, but one of the important functions of tests is not just to uncover bugs and elicit those things or security issues, it’s also to push the boundaries about how you might use the software and essentially stretch it to try to find where it’s gonna break where they can. And oftentimes, that’s a different mentality than or perspective, certainly, than the person that wrote the code.

    Dawson: And if I may jump in and say maybe part of the discussion can get to generalist versus specialist, right? There is this idea, Alan, that when you talk about putting a lot in the developer’s hands, this idea that there is this prolific person called the full stack developer that can separate themselves from what they develop to look for security vulnerabilities, to test it, then ultimately deploy and run it, right? And I think we need to give thought to, are we trying to build a bunch of generalists, which doesn’t potentially enable them to do any job the best, or are we enabling specialists to better work together, share information, and crowdsource quality and crowdsource security?

    And I’m just curious if any of the other panelists have an opinion on this concept of full stack developers versus, say, needing traditional, lifelong quality experts.

    Kalsey: Yeah, I like to say that we decided to create cross-functional teams and we did it by removing all of the functions, and we centralized them all in a single person—and that’s not a cross-functional team. That’s—

    Shimel: That’s a cross-functional person.

    Kalsey: Yes. Well, and you end up with the jack of all trades, master of none problem. You have to have the specialists on even a cross-functional team. You have to have, there’s a lack of testers, there’s a lack of UX people, there’s a lack of security people. They have to be part of that team. You have to have people that, dedicated folks to do the hard work. A developer may be able, and a generalist may be able to do a lot of the easy things and a lot of the, “Oh, yeah, we’ve done this before or we’ve seen this, and this is a known know, let’s go fix that.”

    It’s the hard work of figuring out—what don’t we know, and then having the expertise to be able to know what that hard work is, even, to go after it. And without that, of course the developers are going to be overloaded, they’re being asked to do everything and they don’t know what everything is.

    Shimel: Agreed, agreed. I mean, I think, again, security is the poster child for this, right? We’re asking developers to be security people or to make security a priority. Now, you ask most security people four or five years ago, they’d tell you you’re smoking your socks. Developers don’t care about security. They never did. Only we care about security.

    It’s changed. It’s changed a little bit. We do think other—no one wants to develop insecure code. But we can’t ask them to be security people—or can we, Caroline?

    Wong: I sort of think we have to. I think that, you know, the big, bad thing that no one wants to happen is a security breach, and a security breach cripples an engineering team’s ability to execute on a roadmap. You know, if something big and bad happens, then entire teams have to rush and commit to fixing this problem sometimes for months, you know, time which, of course, could be invested more profitably elsewhere.

    I have a quote that I’d love to share from a colleague. Denali Lumma is the VP of Engineering over at Project Ronin, and she also makes a really good point about tech debt. She says, at a certain point, it’s too late, there’s too much tech debt, too much complexity, the code base is too big, and it becomes impossible to do anything about it. What you see there is a business will slowly die because shipping features becomes harder and harder, critical vulnerabilities and bugs are introduced more frequently, and eventually it kills the business.

    You know, I often think of software as a living organism, right? Whether you think of it as a tree or as an animal, there is this balance between making sure you’re doing sort of your fun things, your running, but you’re also resting when you need to heal. You’re also—you know, if we’re talking about a human being, you’re also sort of going to the doctor once a year.

    Shimel: Agreed.

    Wong: Yeah.

    Shimel: Agreed. Thoughts?

    Kalsey: Quality has to be a mindset, security has to be a mindset, user experience has to be a mindset on the developer time, among everybody there. The user experience person needs to understand and have a mindset of developing and of creating a secure experience and not, “Well, we’re gonna get rid of all the security because that’s hard for people to use.” Everybody has to have that mindset, but you still need the experts and you still need the expertise to be able to do things well. Just having a mindset doesn’t make you fantastic at it. You can say, “Oh, I love to get out and I’ve got an exercise mindset, so I’m gonna go run a triathlon tomorrow.” It’s probably not going to work.

    Shimel: Probably not.

    Ashley: You know, it’s interesting—we could’ve had this conversation 20 years ago, 10 years ago.

    Shimel: Absolutely.

    Ashley: You know, why are we still frustrated with tests? What is it with tests that is, you know, it used to be sort of the orphan of the development process, you know, the only thing that happens later than test is documentation. Well, now, [Laughter] it’s still an issue. Why is that? Why are we so frustrated with tests today? I’m curious what folks think.

    Dawson: I’m gonna start with one not very complex thought—it’s actually hard.

    Shimel: It damn right is.

    Ashley: A little more respect—we need a little more respect for tests. [Laughter]

    Shimel: Well, but this was like, I remember when we first launched devopsy.kinsta.cloud, a popular theme was that, you know, DevOps would kill the testing star, right? That there would be less QA jobs because everything was gonna be automated continuously and shift left, we wouldn’t need testers—poppycock, right? We have as many testers as we ever had, if not more. It didn’t kill off testing jobs. I think it was Adam that said we’ve made testing part of the DevOps team. So, we have people on the DevOps team who are responsible for the various, you know, styles and types of testing that we do, including security.

    I think what’s happened, though, is, these DevOps teams—when I first started devopsy.kinsta.cloud, if you spoke to developers, they’d say, “Oh, this DevOps is all about the Ops.” And then you’d talk to the Ops people and they’d say, “This is DevOps, it’s all about Dev, it’s Dev-centric,” right? And that’s the sign of a good compromise when they’re both pointing the finger at that.

    But it has come—at least as far as testing goes, from what I see, it has sort of tilted where, you know, as we’ve shifted testing left, we’ve shifted responsibility left as well, which kinda winds up closer to the Dev, or certainly to the DevOps teams.

    However, though, let’s look at the alternative, and I’m gonna ask all three of you—all four of you have been around the block once or twice. My personal feeling is, is that we do a much better job of test coverage today than we did 7 or 10 years ago, and I think DevOps and continuous automated testing is a big reason for that.

    Thoughts? Anybody disagree? Show me wrong.

    Wong: I will challenge you, Alan.

    Shimel: Okay. I was hoping someone did.

    Wong: I’ll challenge you, and the reason is because—so, we’re all familiar with the OWASP Top Ten.

    Shimel: Very much.

    Wong: There have been a handful of versions of the OWASP Top Ten since the first was produced in 2003. I believe we’re gonna see another version later this year. The current version is from 2017, and if you take the 2017 version of the OWASP Top Ten and you put it right next to the 2003 version, it is sort of mind blowing how similar they are.

    Shimel: Yeah.

    Wong: And what that says to me is, we actually know exactly what the problems are. We know how to find them, we know how to fix them, and we know how to prevent them, but they’re still around, and so I think that’s—

    Shimel: They are, but are there as many of them, right?

    Wong: Absolutely. [Laughter]

    Shimel: You think there’s as many cross-site scripting or buffer overflows than there was 10 or 15 years ago?

    Wong: So, there are some categories that have been eliminated. You know, there are some categories that, for which we can define a secure pattern, and we can encourage folks to use that secure pattern, and for which we can actually eliminate some of these things.

    But there’s others or which we cannot. You know, I think about—I recently visited my sister’s home, and they have two young children, a 2-year-old and a 4-year-old. And at night, what they do is, they run a Roomba, which is brilliant, you know? If you could spend your weekend playing at the park with your kids instead of vacuuming, of course you’d wanna do that. And the Roomba works because they sort of take the time to get all the toys off the floor and make sure everything’s clear and once they use the Roomba, they clean it. There is this different work involved in effectively using a Roomba that’s different from sort of manually using a vacuum.

    And I do think there are these two distinct categories of testing, one category which you could say is known, repeatable, and automatable, and another for which manual testing, I believe, will always need to exist. There are types of security flaws like business logic flaws, race conditions, chained exploits that I don’t believe that in my lifetime, we will be able to figure out how to teach a computer to find these particular types of issues.

    Kalsey: And it’s beyond just security testing. I mean, exploratory testing, from a tester, your automated tests are not going to find, “This error message makes no sense in this context. I did this, I expected—I don’t know what this error message is trying to tell me to do.”

    Your automated tools aren’t going to find that. All they’re going to be able to find is, “It didn’t work, we sent an error message. We had the things we wanted to put in the error message there.” You can test that your code is doing the right thing, but if you didn’t design it to do it right, you’re not gonna find that with an automated test.

    Shimel: Fair.

    Dawson: I would tend to say, for the surveys that we’ve issued and I’ve read, your original statement, “Has test coverage improved significantly?”—yes, I think all indications are, we are getting better at automated testing, we are getting better at that pursuit of that thing we call test coverage.

    But I think what’s more important is, has the quality of test improved? Are we testing the right things, right, from functional testing to security testing, various shades of security testing? And I’d say (1) yes, we have improved there by sort of the qualitative and quantitative measures that I’ve seen. But I also think, as we truly start to focus more on quality, we start to achieve more coverage, we start to implement better tests, it becomes more clear, as I think Adam and Caroline are speaking to how much more work there is to do, how much other things there are to improve testing on. Pair that with a constantly changing technological landscape, right? We’ve gotta learn new stacks, we’ve gotta figure out how to test new stacks. We’ve got half of our put in in a SaaS platform, but then we deploy on prem, et cetera, et cetera, et cetera. And then you add to that that, at the same time, we’re trying to transform organizations.

    So, Adam, you talked about exploratory testing; Caroline, the human element. I think one of the things that pursuit of the full stack Dev, that pursuit of the fully automated processes failed to do in a lot of orgs I speak to is, we talk quality first, but oftentimes, the DevOps transformation doesn’t involve quality engineering first, right? And I think one of the things we need to get better at is engaging with and partnering with Quality and establishing our automated and modern processes.

    Kalsey: Yeah. We talk good in the testing industry a lot about shift left and that testing is moving earlier in in the process, and too many organizations have taken that as, “Well, the developers are the ones that are early in the process, so that means the developers do it.” No! Shift your damn testers left. Get them involved in the design phase. Get them involved in talking about the architecture. Get them involved in the entire thing, not shift the task to the people who are earlier in the process. Shift the whole process, please.

    Dawson: Or to a script, yeah.

    Ashley: I’m gonna take a whole ‘nother tack, and I thought you were gonna go here, Brian, is, I think we need to shift tests down. We need full stack testing. It’s not application testing anymore. Most of the security errors that we have are due to configuration errors. And what Caroline said was absolutely true. We still have SQL injection, cross-site scripting, and overflows may be less so. But why? It’s because it’s not just in the app layer that those things happen, it’s in that software stack. And I know not every tester can be an expert at the stack itself, but maybe that’s, instead of kinda dividing the DAST and SAST and the different kinds of testing that we do and just deciding, can we shift those in one direction is, let’s think about it from a depth perspective, because that’s where a lot of those errors are coming from. It’s not just the app.

    Kalsey: Oh, yeah, and the tooling on this is also, can be super fragmented. You need one thing to test your infrastructure code, you need another thing to test your UI, you need another thing to do the security—and they’re not, consolidating those reports and combining them and figuring out what worked and everything is hard work and there’s nothing that does that. This is a place that Tricentis could definitely improve and improve our own tooling here.

    And then, once you’ve done that testing, the results are often in a silo. Okay, I found out I had a problem—how do I fix it? How do I go see the data behind what was going on in the system to understand this and solve this? And now I’ve gotta dive into a monitoring tool or my logs or an observability tool instead of just having everything all in one place.

    Shimel: So, I think there’s a—I’m sorry, Caroline, you go.

    Wong: No worries, and thank you, Alan. So, we’ve talked about shift left, shift right. Mitch is kinda talking about shift up and down. I think there’s another axis, if you will, which is sort of what I’ll call shift in and shift out. So, you can test things in and out of band, and that’s gonna make sort of a wildly different difference in terms of how those processes run.

    You know, I think 10 years ago, security people appreciated the power that came with being a gate in the process, being an end band process. You know, today, it actually makes sense, I think, for a lot of security testing to be out of band, but that removes the sort of power and authority of security people to say something like, “You must fix this, otherwise you’re not gonna go live if it’s happening out of band.”

    Shimel: Fair. So, I’m getting a little dizzy, we’re going left, right, up, down, in, out. But—

    Ashley: It’s a circle. [Laughter]

    Shimel: Yeah, you know, shake it all about. But here, fundamentally, though, are we just kinda dancing—no pun intended—are we just dancing around the real issue, which is, we wanna do more pre-deploy, we wanna do more, we want better quality software, right? I think we could all wrap our heads around that idea and vote for that. We all want better quality software. Security people are at a premium. We can’t shift security people left or en masse. Testing people are also at a premium, though we’ve integrated them into the DevOps teams. Developers are at a huge premium, they’re probably our most expensive resource in terms of humans.

    What do we need to do to make everyone’s job easier here, to make this function better? Is it a question we’ve gotta automate more? I mean, maybe that’s what it is, if we automate more, or is it a cultural thing, right, where—

    Ashley: There’s also test-driven design, right, which is, as you’re designing it, you talk about, you put together your test strategy of, “Okay, we do it this way. Let’s talk about how we are going to test it, start to lay out the framework for how we think this can be validated.” I mean, it’s not the only thing to do, but I think—so, if you start with that in mind, now you’ve got a path to pursue as opposed to, “Here’s a chunk of software, you folks figure out how to test it, right? We built it, you test it.”

    Dawson: Yeah, and I’d add, I think there’s no—again, not only is the practice of testing harder than we often like to admit, but the, you know, the path to kind of getting to where we fully wanna be, where we don’t dread testing, complex, hard path, and I’d say it’s the DevOps trinity, you know, Alan, and we all can probably hit a few points on each.

    But you have to address people and culture, as Adam brought in earlier. There’s no getting away from, security needs to be a thought for everybody, right? I can install all the locks I want on the front of my door. Actually, over 24 years—today is my anniversary—my wife has programmed me to lock the door when I come in the house without thinking, right? She’s kind of impressed a security culture on me.

    You do have to address people and culture, process, practice, tools and technology. Automation alone will not solve it. A mindset alone will not solve it. All of that needs to come to fruition.

    Shimel: Mitch, make a note—invite Brian’s wife to our next security panel.

    Dawson: [Laughter]

    Ashley: I’ve already looked up her e-mail address—thank you, Brian, appreciate it.

    Shimel: Happy anniversary, by the way, too, Brian.

    Dawson: Thank you, thank you.

    Kalsey: This is one of the places where I think what Caroline said is so important, that this in bound, out of bound—if you’re trying to do everything in line, of course, it’s gonna be harder. In shifting that mindset to doing things continually in the background, we already do so much stuff and process automation in the background where we don’t have people check, “Here’s a step in the process.” No, this is going along in the background, dependency watching and that sort of thing and updates.

    We can be doing testing in the background as well for a lot of it, and identifying what needs to be in the foreground and what needs to be in the background. Look, I’ve seen CI/CD pipelines that take hours and hours and hours to run because the tests take six hours to execute and to build. That can’t work in any sort of a reasonable DevOps system.

    Wong: I think there’s two parts to a solution, and I’m really, frankly, just parroting what Brian said. I think there’s a collaboration piece between the people. You know, to an earlier part of our conversation, you’ve gotta have the folks with those specialized skill sets. Ideally, you’re able to get access to those folks on demand, which historically has not been something that’s been possible, but that’s beginning to happen.

    The second thing is—so, you’ve got your people collaborating, you’ve also gotta have your technology integrated. So, for whatever security tool or processors, how do those outputs actually get to the engineers? What we need is bidirectional integrations for things like Dev ticketing systems. We need things like APIs for test results. I think collaboration at the people place and integration at the technology place—both of which we could call processes—then it’s just, it’s really a repeat of what [Audio skips].

    Shimel: Mm-hmm. I mean, but wasn’t this one of the promises of DevOps? I mean, I got into DevOps because I thought it was a better way to do security, truth be told. But one of the promises of DevOps is, we’re gonna automate more, so we could go faster with more quality—two out of three ain’t bad, right? And, you know, did we lose that somewhere? Is that no longer a goal, or have we taken that as far as it goes?

    Kalsey: We keep trying to solve it with technology instead of by looking at the culture and changing how people think. And you can’t think the same way and apply new technology to it and expect the technology just to magically solve all your problems.

     Shimel: Brian, you agree, or Caroline?

    Dawson: I absolutely 100 percent agree, and I’ll agree and then I’ll just add a point. And I know I’m being a bit nebulous about it, but there’s a human element to this. At the end of the day, it’s humans developing software for humans, right? Whether it’s the code on your electronic control module in your car or Waze on your phone or the OS, at the end of the day, the final endpoint is a human. And you know what? At least now, and I know we’re improving, there’s no way that we can ensure—and I don’t mean to accept security here, but to test the quality of the human experience, right?

    And think about this—at the end of the day, knowing that the language on a screen is self-descriptive and informs me what to do. Testing the fact that, or ensuring that when I submit my credit card number it’s not vulnerable, right? Whether we’re talking API testing, whether we’re talking security testing, all of this at the end of the day boils up to customer or user experience testing.

    And I guess what I’m gonna posit is that we can never entirely automate that. And where we’ve hurt ourselves and possibly made a false promise is, we’re gonna create a bunch of full stack engineers, we’re gonna get rid of, away from operations functions, testing functions, and we’re gonna use TDD and automate everything from the start, Gherkin, yadda yadda yadda. And I believe—look, that has enabled us to go faster, but our impedance mismatch, the limit we’ve hit right now is where we’ve failed to better connect people and enable them to collaborate and share knowledge. So, not just security, not just back end, not just data, but the entire customer experience is of quality. Dev can’t do that alone.

    Shimel: No, sir.

    Dawson: I’m sorry, Alan, did I get, did I—

    Alan Shimel: No, no, that’s all—I’m saying no, sir, you’re 100 percent correct, but I know Caroline wanted to say something.

    Wong: So, I do have a comment about automation. I think that DevOps has figured out how to do automation when it comes to defect discovery. I think we’re sort of good at that. I think what we’re still trying to figure out is automation when it comes to workflow and process so that, actually, again, to Brian’s point, the collaboration can happen automatically, rather than due to a human who’s involved and making it happen. How do we get the right information to the right people in the easiest, most efficient way at the right time, and how do we make it super easy for them to do their jobs?

    I do think that there remains—I do think that the next evolution of automation and DevOps is gonna go from defect discovery automation, which I think we’ve gotten quite good at, to workflow and process automation.

    Shimel: Fair enough. Adam, thoughts?

    Kalsey: I like it. I mean, identifying what you’re automating and what’s working in that automation is important, but also identifying what’s not being automated or what in your automation is broken. And not broken from the standpoint of, it doesn’t execute or it doesn’t run to completion, but broken in, it’s not solving the problem you wanted it to solve.

    Shimel: Fair enough. Guys, we have time for maybe one more question. I’m gonna pose it and throw it out there to you. Is it possible we’ve shifted too far left or that you can shift too far left and we need a little rubber band, here, going the other way, a little pendulum swing? Or you think we just scratched the surface? Caroline, you look like you wanna say something.

    Wong: I think that’s a great question. I do think it’s possible to shift too far left, and the reason is because, while we, I think, are all familiar with the statistics and the studies that say if you fix something earlier in the cycle, it’s cheaper—there’s stuff that exists to the right that doesn’t exist yet on the left. Software development is, you know, to sort of a little analogy that we talked about earlier, I think it’s a dance. I think it’s a dance between people. I think software comes about as a result of the behaviors and the actions and the decisions that various folks take together. And security and quality are emergent properties. They’re not Band-Aids, they’re not vitamins, they’re not painkillers. They have been, as a result of this dance. So, there are some things for which, if you try to test it too early, it doesn’t actually exist in its final form yet. If I take my 6-year-old to the doctor and then I don’t when she’s 21, you know, might we have caught something super important as a 6-year-old that we were able to step in and address right away that would’ve been great? Yes, but as a 21-year-old—guess what? She’s gonna have different problems that were not apparent at age six.

    Shimel: Spoken like a true mom and parent, right? Parents, we worry about that. Anyone else? Thoughts on that? Can we go too far left, or have we gone too far left?

    Dawson: I, Alan, think it just depends. I think we’ve interpreted left different ways. I think it’s a good, catchy term. It’s an important and powerful catchy term, but as I’ve heard others discuss, we sort of overemphasized this idea of left. One, to Caroline’s point, right, we just do everything here and we’re done, good to go, right? But then this idea that we can put everything in Dev? Well, no, I think—I think, for what it’s meant to communicate, shifting left is an apt term.

    I think in terms of ensuring that we’re doing everything we can as far left or as early in the process to mitigate issues later—great. Right? But we don’t just shift security left. Right? Like Caroline said, inbound, out of bound. Actually, we need to shift security everywhere. We don’t just shift quality left, we need to shift quality everywhere.

    So, I think there are—we need to be ensuring that it’s almost zero trust software delivery, right? At every stage of the process, from concept where you should be sitting down with QA and the security team to capture the requirements to test, to deploy, to run, we need to be thinking about it, also.

    Alan, I’d say shift left is good—shift everywhere is really what we meant to be doing, right?

    Shimel: It’s better. Great.

    Kalsey: Any time we engage in sloganeering, there’s gonna be people that pick the wrong things from the slogan. I mean, “Hey, we have DevOps. That means we don’t need Ops anymore!” And that’s not what it meant, and there’s people that have taken the wrong cues from shift left, and instead of shifting the processes left, they shifted the tasks onto the people who were left already, and that’s not gonna work.

    Shimel: Absolutely. Hey, Mitchell, you wanna tie a bow on this?

    Ashley: I’ll put a bow on this. You know, there are admittedly many, many things that I don’t know, but one of the things that I do know is that, if a developer says it’s tested, it’s ready and a QA person says it’s tested and it’s ready, five out of six dentists will choose the QA person. I will, too.

    So, there’s something there. There’s a reason why we don’t have DevQA or DevTest. We don’t talk about it that way, we talk about DevOps or DevSecOps. QA is extremely valid and necessary, and we can’t live without it, and I think we need to invest in helping, elevating how we test and how we think about our software applications to test in. So, those dentists will be happy.

    Shimel: And Crest, the Crest test for QA. Thanks very much, Mitchell. Caroline, Brian, Adam—thanks for joining us on this episode of DevOps Unbound. Great conversation. For our audience out there, do check it out. We have another episode coming up in—well, not next week, the week after, and then I think we have a roundtable the two weeks after that.

    But for now, though, we’re gonna wrap. Many thanks again to Tricentis for sponsoring DevOps Unbound. Thank you for watching, thanks to our panel. This is Alan Shimel, we’re out.

    [End of Audio]

  • Swinging the Pendulum of IT Back to QA

    Swinging the Pendulum of IT Back to QA

    I was reading Nuria Manuel’s excellent article about quality assurance (QA), and it got me thinking about the wide variety of ‘pendulums’ that impact IT. I call them pendulums because they follow the rule of “every action has an equal and opposite reaction,” and they work on fundamental truths of IT. For example, the one that has probably been going on the longest is centralization/decentralization of IT and its assets. No sooner is everything consolidated and cost savings realized than adaptability concerns and needs of individual business units start the process of decentralization. Then, once decentralization is realized, everyone is painfully aware of the chaos that fifteen miniature IT departments, each going their own way, can cause … and centralization begins again.

    Quality assurance is another such pendulum. Much like soldiers in peacetime, most people don’t recognize the value of QA efforts when things are going well. And when they’re not going well, then it is considered a QA failure—assuming there is a true QA function. Otherwise, it becomes the impetus to launch a formalized QA function … until things are going well and budgets are tight. The problem with QA is that for years, timelines, budgets and Agile management styles have all conspired to swing that pendulum away from a formal QA function. Oh, companies pay the price, but they quite often don’t see it as a cry for a valid QA function, and just treat this particular issue as a one-off that they should make sure doesn’t happen again. Unfortunately, issues in production have become both commonplace and accepted because of this. The only issues in production that seem to actually generate angst are ones that keep users from accessing the product. Anything short of that and workarounds or delayed fixes are seen as okay.

    But the apps you are rush-building today will still be around in 10 years—or, at least, some of them will. Some of them won’t for a variety of reasons–including quality issues. Those that are around in 10 years are going to have to be maintained. QA now can save you over the years, so why all the resistance to it? QA might delay the release of new app. It is understandable that delays are to be avoided if possible, but so are bugs, so maybe constraining how long QA can delay the release and building that number right into the schedule would be a decent compromise.

    That way, QA is looking at the most important/used parts of the app, but not getting buried in the weeds (where there might be bugs, but the ROI for addressing them probably isn’t there). The app will be delivered on time because QA’s time was built into the schedule. This does require that we not allow the traditional and abhorrent practice of shortening the QA window to make up for developers not meeting deadlines. Let us be very clear about this: when developers are rushed because they missed deadlines, that is precisely the time that you want QA to look very closely at their code before release. The last thing you should do is less testing in this scenario. The absolute last thing.

    This is completely doable in your environment. The only really difficult part is holding the line on QA. Business owners are going to want to meet their original timeline, but cutting any phase of testing to get there is just a bad idea, so the struggle will be to hold the line. And figure out why devs missed the deadline in the first place so that the issue can be avoided for the next project.

    You all are rocking it. Make sure all that sweet code-and-deploy work is not undermined by quality issues! Go ahead, implement a QA policy and make it an actual policy. Keep rocking it, get accolades from the rest of us. Maybe even from your employer. Maybe. But definitely from us. Because you’re killing it, month after month.

  • Big Gains Are Made In The ‘Last Mile’ of DevOps

    Big Gains Are Made In The ‘Last Mile’ of DevOps

    A new study by mabl found a correlation between complete, fully automated test coverage and a significant increase in customer happiness and DevOps maturity. In contrast, those with moderate DevOps maturity and nearly automated testing coverage tend to experience a sharp decline in end-user satisfaction.

    Mabl recently released its 2021 State of Testing in DevOps Report, a study that highlights the state of DevOps adoption and quality testing throughout the software industry. According to Dan Belcher, co-founder, mabl, the study results clearly show that true business returns are made in the last mile of DevOps adoption.

    I recently met with Belcher to hear his viewpoint on the major takeaways. Below, we’ll explore these and other insights from the 2021 State of Testing in DevOps Report.

    State of DevOps Maturity

    Nearly one in four teams have a mostly mature DevOps methodology and automated process. Still, a significant portion, 52%, are still aspiring or striving to fully embrace a mature DevOps model. A small percentage, 11%, have a fully automated DevOps software development pipeline in place. The bulk of teams are still using manual deployment processes and disparate frameworks, limiting the potential for full automation.

    Interestingly, non-technical barriers remain the most common inhibitor to realizing a completely automated development pipeline, found the report. Out of these obstacles, reluctance to change was the most common barrier, cited by 29% of respondents. Other challenges to DevOps adoption include budgetary constraints, DevOps not being a company priority and internal politics.

    When added up, non-technical roadblocks account for 82% of the problems that hinder complete DevOps adoption. On the one hand, this depicts the cultural challenges associated with introducing DevOps in an organization. On the other, it could signify the wealth of DevOps packages and open source tooling available to manage areas like automated deployment and version control. The report found GitHub, Jenkins, Gitlab, Azure Pipelines and BitBucket among the five most commonly used DevOps Tools.

    Although it’s growing rapidly, continuous delivery is still an emerging trend; almost half of developers still deploy code less than once a month. Only 15% of teams are deploying code daily or multiple times per day. This echoes findings from CDF’s State of Continuous Delivery Report, which pegged this figure at 11%.

    Fully Automated Testing

    Testing is a critical component of the software life cycle. With the need to ensure functional applications and high-grade security, automated testing is helping teams discover more errors earlier in the development process. According to the report, the most popular test types include UI or functional tests, unit tests, regression tests, integration tests and end-to-end tests. Non-performance tests and visual testing ranked lower. Naturally, the teams that have implemented fully automated pipelines are automating more types of these tests.

    In general, it appears test automation aligns with DevOps maturity. The report found that those with fully automated testing experienced significant net customer experience gains. For example, 72% of teams that were automating regression testing indicated their customer happiness is pretty good or amazing, whereas this figure sank to 6% at companies not automating regression testing. And 80% of teams with high test coverage report high customer happiness, whereas only 30% of teams with low test coverage report high customer happiness. Nonetheless, much has yet to be fully automated—similar to the state of continuous deployment, the bulk of teams rely on a combination of automatic and manual testing approaches.

    In terms of tooling preferences, Selenium ranked as the most popular option for functional testing, with 41% of developers using it. The report also found migration toward Cypress, Smartbear and other functional testing tools. Interestingly, the report also uncovered a noticeable lack of adoption when it came to automated API testing—only about one out of three are automating their API tests. Postman, SoapUI, internal tools and ReadyAPI are among the most popular tools for running manual API tests.

    As I’ve covered before, APIs are surging in use across digital transformation initiatives. Web APIs are an increasingly common mechanism for delivering data to partners and consuming third-party software-as-a-service (SaaS). Full API testing coverage will be essential to audit the entire end-to-end digital experience. Low-code API testing is one way to reduce barriers to auditing the highly technical area of integration.

    Key Takeaways

    DevOps success is a ‘last mile’ problem. There is much variance in DevOps maturity and testing coverage. It’s pretty flat until you reach that last point, when breakthrough results occur. “Success in DevOps is a ‘last mile’ problem,” said Belcher. “The difference between 90% and 100% of the way there is dramatic.”

    Quality really matters. Though much tech discussion focuses on increasing delivery speed, code quality is just as important. “This isn’t about speed at all costs, it’s about innovation with velocity and quality,” said Belcher. “Quality engineers are just about as key to success as developers writing code.”

    DevOps success is less about technology and more about people. As the report found, the majority of the roadblocks to DevOps maturity are non-technical. “The things holding us back, as an industry, are more about leadership and figuring out our processing than introducing more technology,” said Belcher.

    Breaking down role silos is still fundamental to DevOps. Some see DevOps as instrumental to opening collaboration between roles. Simultaneously, there is the idea that new tooling dashboards will arm individual developers with complete ownership to develop, test and deploy software (known as full cycle development).

    Belcher believes that the full cycle momentum has been misinterpreted to mean developers alone have full accountability. Instead, he frames developers as a pitcher on a baseball team; leading the game, but utterly incapable without the rest of the team. Similarly, disparate roles should be working together, said Belcher. Reaching that ‘last mile’ of DevOps is not only about CI/CD or full cycle development; it’s about working together as a group.

    I’ve only scratched the surface of many findings from The State of Testing in DevOps. The report queried 600 quality engineers and software development professionals and offers loads of additional insights on the state of DevOps, testing, CI/CD and more. You can view the full results here.

  • Bringing Continuous Testing Into the Software Delivery Pipeline

    Bringing Continuous Testing Into the Software Delivery Pipeline

    For any highly competitive industry that relies on a smooth and stable user experience, it is vital that their development teams accelerate release cycles, lower operating costs and ultimately reduce risks. Continuous testing is one way companies can achieve these objectives.

    As a subset of software testing, continuous testing (CT) is valuable because it uncovers and addresses different problems at various stages of the development journey that other methods don’t. Most other testing methodologies require a large staff; CT doesn’t. CT leverages in-sprint automation, including end-to-end tests across the front and back end of the technology to cut costs and facilitate rapid delivery without adding more pressure on internal teams. Moreover, as part of a larger software delivery pipeline, CT is positioned closer to the beginning to execute tests early, often and comprehensively to realize cohesive release goals.

    CT introduces quality assurance with constant iterations of testing and actionable feedback at each stage of development to evaluate business risk coverage and ensure the best user experience and customer satisfaction. As the name implies, CT helps developers and product managers detect and fix critical bugs not only in the early stages of development, but continuously throughout the software development life cycle (SDLC), ensuring that errors in the code do not get unintentionally released into the wild.

    As software companies accelerate their release frequency, continuous testing sets a precedent for digital tools development and the quality assurance process.

    Benefits of Continuous Testing

    Continuous testing plays an integral role in the CI/CD pipeline, is part of the overall set of DevOps best practices and the information it provides is invaluable to development teams. The benefits of CT are also far-reaching, with positive impacts on the entire organization, including:

    Better Company-Wide Feedback

    Continuous testing facilitates feedback to all involved teams and stakeholders. For example, when a bug gets detected in code, ideally, the only team notified should be the team responsible for the part of the code with the bug. Without CT to automatically detect bugs and notify those responsible for fixing errors, triaging bugs could be a full team effort or, worse, a dedicated team’s main function. This approach is very inefficient and wastes the time of the engineers whose code is bug-free. The feedback provided from CT cuts down on these unproductive scenarios and allows stakeholders to make immediate and informed decisions.

    Higher Quality Code and User Experience

    A company that effectively uses CT should expect higher quality code and a more stable user experience. Stability measures application health and user experience; it is negatively affected by session-ending events like crashes. With CT, businesses will give their development teams better awareness of product quality and performance while also significantly improving overall business metrics by fixing broken apps faster.

    CT Breaks Down Siloed Teams

    Not only will CT enhance quality assurance, it similarly works to improve the development of digital tools through the breakdown of siloed teams. When teams become siloed, they inadvertently keep processes and knowledge to themselves, resist cross-functional collaboration and work in isolation toward their own goals. Predictably, separate groups fail to share insights or ideas, leading to stagnation and decreasing innovation. However, by leveraging CT, siloed teams can become more collaborative in development, testing and operations, increasing overall business efficiency.

    CT Enables Objective Evaluation

    Finally, CT enables an organization to pause and objectively assess if it is aligning efforts with its specific business goals. As mentioned before, CT uses automation to provide timely feedback on business risks, reducing the chances of a digital application or tool failing at launch. Likewise, dev teams and managers can more easily meet their business goals and optimize the value of a release. Additionally, objective assessments are helpful for critical decisions – as CT gives actionable data for decision-makers to support their choices.

    Today, consumers have come to expect error-free experiences within their software applications, with the most stable of these applications driving conversion, engagement and retention. Applying a continuous testing model to a software development pipeline will ultimately result in applications being delivered faster and at higher levels of quality than ever before, with an enhanced end-user experience. As companies combine agile development with continuous testing, they will optimize each aspect of the development life cycle and their larger business goals.

  • Continuous Testing Practices Gap Assessment

    Continuous Testing Practices Gap Assessment

    Continuous testing is an essential part of DevOps. To get the maximum value from continuous testing, it is important to use recommended best practices, inclusive of people, processes and technologies. A gap assessment is a great way to efficiently evaluate an organization’s practices for DevOps and determine a strategy for improvement. Gap analysis provides valuable input for formulating a strategy and a roadmap to improve continuous testing.

    My SlideShare, Gap Survey, Assessment and Analysis For Continuous Testing, explains how my continuous testing gap assessment tool greatly speeds up the gap survey, gap assessment and gap analysis process for continuous testing.

    It is important to clarify a few definitions for this blog because I have found that different people and organizations use these terms differently. In this blog, and in my consulting work, I use the following definitions.

    • Gap survey is a discovery tool used to collect information about the current state of practices for a topic such as continuous testing.
    • Gap assessment is a process to determine differences (gaps) between current practices and recommended practices for a topic. Gap assessment is also used to refer to the entire collection of activities in this list.
    • Gap assessment workshop is a meeting in which the gap assessment is validated prior to gap analysis.
    • Gap analysis is the process of determining priorities for reducing gaps found during a gap assessment.
    • Gap assessment tool is software that helps perform gap surveys, gap assessments and gap analysis.

    I define and use a seven-step process for gap assessments.

    The steps are as follows:

    1. Pick a topic to analyze. The same processes and tools can be applied equally well to almost any topic, such as continuous testing.
    2. Determine recommended practices for the topic. Something that is worth doing a gap assessment for, such as continuous testing, will normally be fairly complex with tens if not hundreds of practices. Categorize the practices into groups (I call the groups “pillars”) to make the list of practices more understandable and meaningful. To do this requires some research and discovery work. For example, I have a database of practices and pillars for DevOps, continuous testing, DevSecOps and SRE as well as some other topics.
    3. Enter the practices from step two into a gap assessment tool. The tool can be a simple survey tool you create, a commercial survey tool or something custom-made.
    4. Determine who needs to be included in the gap survey.
    5. Collect data using the gap survey.
    6. Perform a gap assessment.
    7. Perform a gap analysis.

    For the remainder of this blog, I will explain in detail the seven steps for continuous testing.

    Step One: Pick a Topic to Analyze

    In this blog, the topic is continuous testing.

    Step Two: Determine Recommended Practices for Continuous Testing

    For an example continuous testing assessment, I used the 15 practices categories described in this three-part series:
    Continuous Testing Practices Part 1
    Continuous Testing Practices Part 2
    Continuous Testing Practices Part 3

    The 15 practice category pillars are:

    • Continuous testing tenets
    • Leadership
    • Culture
    • Test strategy and plans
    • Test management
    • Test automation
    • Test tools
    • Test infrastructure management
    • Test scripts
    • Test results
    • Security tests
    • Pre-flight testing
    • Integration testing
    • Pre-production testing
    • Test in production

    Step Three: Enter the Practices from Step Two Into a Gap Assessment Tool

    A gap assessment tool, pre-loaded with sample continuous testing practices, can be found in a file called “Continuous Test Practices Assessment Tool v2. It can be downloaded for free from one of the resource pages found on my website Engineering DevOps.

    You can edit the practices categories and add/delete practices from each category if you prefer to make changes.

    Step Four: Determine Who to Include in the Gap Survey

    For the gap analysis to be comprehensive, people in roles that are affected by continuous testing practices need to be surveyed, or at least represented, to ensure their perspectives are included. The following are example roles that typically are included.

    • Business leaders – because they influence culture and budgets for continuous testing.
    • Developers – because they need to design in accordance with continuous testing practices.
    • Project owners – because they influence product work priorities.
    • QA testers – because they need to test in accordance with continuous testing practices.
    • Ops – because continuous testing practices affect operations.
    • Security – because continuous testing affects security practices.

    The survey can be conducted for an individual application, a group of applications, or all the applications in the enterprise. However, it is important that the gap assessment and gap analysis be performed on the organizational segment that is being targeted for improvement.

    Step Five: Collect Data Using the Gap Survey

    A gap survey should allow each surveyed person to enter an importance level score, a practice level score and comments for each practice. All this information is essential for the gap assessment and gap analysis. In the gap survey included within my gap assessment tool, survey respondents are asked to score the importance of each practice as one of 0=Not relevant, 1=not Important, 2=Nice to have, 3=Important, 4=Very important or 5=Critical. Practice level choices are 0=Not sure, 1=Rarely, if ever, 2= Sometimes, 3=Most of the time, 4=Always or 5=We are really good at this. Comments that are relevant to qualify or explain the scores entered for each practice should also be entered, especially if there is any doubt or ambiguity.

    Step Six: Perform a Gap Assessment

    The gap assessment process requires all the practices scores collected from the surveys to be collected and assembled to calculate an aggregate set of scores. Gap scores are calculated using a formula that weights each practice level score with the corresponding importance level score. A visual representation helps to identify practices areas and individual practices that have the highest, most important gaps.

    No matter how professionally written the practices and score definitions are, it is not unusual for some people to misunderstand and to enter scores that they would not have otherwise. For this reason, it is important to ensure that the data collected is validated before conducting the gap analysis. The preferred approach is to conduct a gap assessment workshop with key representatives from each role that participated in the survey. During the workshop, the scores for each practice that have a high deviation between survey responses are discussed and, if necessary, adjusted.

    Step Seven: Perform a Gap Analysis

    The gap analysis process involves extracting the high gap practices and tagging and ranking each of them against solution categories determined by a consultant or topic expert. The results of the gap analysis indicate where solution strategies and implementation roadmaps need to be focused to reduce the most important gaps.

    What This Means

    Those who are serious about DevOps will do well to learn from others and be serious about using recommended continuous testing practices. This blog explained a gap assessment approach, which uses my 15 categories of continuous testing practices. My free Continuous Testing Gap Assessment Tool facilitates performing a gap analysis for continuous testing and outlines priorities to improve DevOps practices for an organization. For those that want to dig deeper on this topic, additional definitions and contextual information related to the way I do gap assessments are clarified in my book Engineering DevOps.

  • Applitools Extends Visual AI Testing to Native Mobile Apps

    Applitools Extends Visual AI Testing to Native Mobile Apps

    At its online Future of Testing: Mobile event, Applitools today previewed the ability to apply visual artificial intelligence (AI) to applications that run natively on a mobile computing platform.

    Accessed via a cloud service that is currently available for free on a limited basis, the Ultrafast Test Cloud for Native Mobile offering will extend the company’s visual AI tool for browser-based mobile applications to applications that run natively on Apple iOS or Google Android devices.

    Applitools CEO Gil Sever said that while most mobile enterprise applications today rely on browsers to make it simpler to build and deploy applications, in time, many more of these applications will be running natively on iOS or Google Android platforms, mainly because they provide a better user experience.

    That approach also eliminates the need to rely on cumbersome platform emulators to test applications because the Applitools AI tool can learn the nuances of each platform in a way that streamlines the application testing process.

    In fact, visual AI tools will not only make developers and testing teams more efficient, but they also will make it easier for end users to participate in the application testing process, Sever noted.

    The Applitools approach to AI testing also provides developers with more control based on the application’s use cases. Different settings can be applied to test, for example, a medical application that requires perfect alignment of pixels versus an e-commerce application that may not require as much fidelity. Developers can then resolve regression issues by employing root cause analytics.

    It’s not clear how broadly visual AI tools that use computer vision algorithms to automate testing will be employed. There is no doubt the current pace of application development is faster than most dedicated teams of testers can handle. As a result, more responsibility for application testing is being shifted left toward developers that, in the interest of time, might be more inclined to rely on AI and other forms of automation that can be accessed via a cloud service.

    The rise of AI testing tools might also go a long way toward ensuring more applications are thoroughly tested before being deployed in a production environment. One of the first steps in any application development schedule that gets compressed when a project falls behind schedule is testing. As more rote tests are automated, there should be more opportunities for dedicated testing teams to focus on improving the end user experience.

    It may be some time before most organizations are routinely employing AI to test applications. However, none of those organizations should expect AI to replace the need for humans to test applications alongside AI testing platforms that can see the actual application. After all, there’s a world of difference between seeing the changes that were made to an application and understanding why they actually needed to be made.

  • Continuous Testing Practices – Part 3

    Continuous Testing Practices – Part 3

    In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing to help with understanding what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

    This article is part three, which covers the following categories of continuous testing practices.

    • Security tests
    • Pre-flight testing
    • Integration testing
    • Pre-production testing
    • Test in production

    You can read part one here and part two here.

    Security Tests

    • Credentials and secrets needed to operate any test are vaulted.
    • Customer sensitive data in application databases used for testing is obfuscated from misuse by people involved in testing.
    • New security tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
    • Security tests for each DevOps pipeline stage are automated and may be selected automatically according to predefined criteria.
    • Test results that indicate possible security concerns are tagged for security analysis.
    • Dynamic or interactive application security tests exercise the application to check for security vulnerabilities. Results of these tests are delivered to developers through tools and feedback loops native to their organization.
    • Attack patterns, abuse cases and tests are built according to application module profiles.
    • Unit, functional and integration tests around—and especially outside—boundary conditions are tested. Tests include error handling, exception handling logic and negative tests.

    Pre-Flight Testing

    • Products are architected to support modular independent packaging, testing and releases. In other words, the product itself is partitioned into modules with minimal dependencies between modules. In this way the modules can be built, tested and released without requiring the entire product to be built, tested and released all at once.
    • Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance, to ensure the changes will not break the integration branch.
    • Requirements for testing are included in agile backlog planning procedures.
    • Pre-flight tests include unit testing, static code analysis scanning, functional, regression and performance tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested, for use in later stages of the pipeline and subsequent regression tests.
    • Pre-flight tests, conducted prior to integration, include unit testing, static code analysis scanning, functional, regression, performance and security tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested for use in later stages of the pipeline and subsequent regression tests.
    • Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance to ensure the changes will not break the integration branch.
    • Test-driven development (TDD) is used to ensure tests are created ahead of the code to be tested. TDD is most often applied for unit level tests but may also be applied for functional testing. A key advantage of TDD is that it ensures working tests are available for use in all pipeline stages. Without using TDD, there is a tendency for test creation to fall behind the code. Another advantage of TDD is that tests are created without the bias of knowing how the code was written, because it hasn’t been written yet.
    • From a testing point of view, microservices introduce the requirement to validate the contract of each service with any other services that use it. The independence of the microservice or any interservice dependencies need to be well tested. Performance and reliability considerations of operating services over a network need to be verified. If the application changes, affected microservices, or any dependent group(s) of microservices also need to be regression tested.
    • Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
    • A strategy ensures that changes to a database, or changes to an application that uses a database, perform as expected, during each stage in the continuous delivery pipeline.

    Integration Testing

    • Software changes are tested incrementally by a cross-functional team over all stages of the continuous delivery pipeline.
    • During the code commit stage, assessments are conducted on the committed code to validate that the pre-flight test results satisfy acceptance criteria for integration into the integration branch.
    • During the build stage, a select group of tests are run on each build to ensure the integrated build satisfies build integration acceptance testing criterion.
    • Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
    • API contract testing is used to test microservices interfaces.
    • Test code and data for each application are stored and maintained in test containers so they can be managed using the same container creation, orchestration and monitoring capabilities as the application.

    Pre-Production Testing

    • After the integration and build stage, functional, performance, regression, customer acceptance and security tests are conducted to ensure the images resulting from the build or set of builds satisfy functional, performance, security and delivery assessment criteria.
    • Testing during deployment include dark launch strategies such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
    • Release assessment criteria are defined and use test results to decide whether a release meets release acceptance criterion.

    Test In Production

    • Testing during deployment uses strategies that minimize the impact of failures such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
    • Automated tests verify each deployment.
    • Chaos engineering is used for testing the resiliency of applications, deployment infrastructure and pipelines.
    • Tests for disaster recovery and restoration of services, infrastructure and pipelines are conducted periodically.
    • Fire drills on applications, pipelines and infrastructures are used to test procedures for detection and recovery of applications, pipelines and infrastructures.

    What This Means

    Continuous testing (CT) practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer Continuous Testing. This article is part of a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s Continuous Testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.