devopsy.kinsta.cloud

Tag: Automated testing

  • Survey Sees App Testing More Widely Distributed

    Survey Sees App Testing More Widely Distributed

    A survey of 900 application testing professionals conducted by PractiTest, a provider of a platform for managing testing tools, suggested the number of organizations that have dedicated testing teams is declining at a steady rate.

    The survey found a 15% decrease in organizations with dedicated testing teams that have six or more members. At the same time, however, the survey found only 18% of respondents reported that all testing is done by individuals that are not part of a dedicated testing team.

    Joel Montvelisky, test specialist and chief solution architect at PractiTest, said that shift indicates that more organizations are steadily incorporating testing within DevOps workflows. In fact, more than 60% of respondents said DevOps and Agile processes led to teams being more organized and stable, with two-thirds (66%) reporting that development teams were releasing features more frequently with more functionality.

    However, only 38% of respondents said they were active participants in a continuous integration/continuous delivery (CI/CD) workflow. A full 90% said they needed to coach or train other members within the team and just over half (54%) reported that estimations of the time required for development and testing were accurate.

    Two-thirds also said collaboration with developers is also improved when applying DevOps processes, with more than half reporting fewer serious bugs made it into production. More than half also said they monitor software in production and deployment environments. The number of testers who defined themselves as engineers has gone up by 14% year-over-year. However, only 44% report that automation has replaced 50% or more of their manual testing efforts.

    A full 95% or more of survey respondents identified application programming interface (API) testing, Agile methodologies, functional testing and scripting as high on their list of priorities. In contrast, big data testing, internet of things (IoT) testing and machine learning/artificial intelligence (AI) testing were cited as important or very important by only 60% of respondents.
    Finally, about a third of survey respondents noted that non-testers are now involved in testing activities.

    Montvelisky said the PractiTest survey makes it clear that while developers are a little more involved in testing than they were a few years ago, organizations still relied on testing specialists that are becoming more integrated into DevOps workflows. This is especially important as more organizations launch digital business transformation initiatives that require them to build software using the same types of DevOps processes that independent software vendors (ISVs) employ.

    It’s still too early to say just how far testing might shift left toward developers, but as testing becomes more automated—thanks, in part, to increased reliance on AI—the types of tests that will be conducted by a dedicated testing team will evolve. In fact, it’s unlikely automation will eliminate the need for those teams any time soon. Rather, testers will be augmented by AI technologies that will help them to keep pace with the accelerated rate of application development enabled by wider adoption of DevOps best practices within organizations both large and small.

  • EP 16: How DevOps has Changed Testing Forever

    EP 16: How DevOps has Changed Testing Forever

    DevOps has changed the game in almost every aspect of testing: who, what, when, where and even why. What’s working, what needs to change immediately, and what challenges—and opportunities—lie ahead?

    In this DevOps Unbound episode, hosts Alan Shimel and Mitch Ashley are joined by Abel Wang from Microsoft, Tracy Miranda from CDF, Caroline Wong from Cobalt, Kurt Chase and Ben Simo from Tricentis to share a variety of perspectives on how testing fits into the ever-evolving world of DevOps. The video is below, followed by a transcript.

    DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.

    Alan Shimel: Hey, everyone. Thanks for joining us on this special edition of DevOps Unbound. You’re watching this, probably, at the Tricentis Virtual Event and we are happy to have you join us, happy that you’re attending the event, and we’ve got what we think is gonna be a great discussion on testing and all of its various facets and what does the future – or near future – hold. We’ve got a great lineup of speakers that I’d like to introduce you to who are on our panel today. Let me first start with – she’s on the top of my screen – is Caroline Wong. Caroline, why don’t you say, “Hello” and introduce yourself?

    Caroline Wong: Hello. My name is Caroline Wong. I’m the Chief Strategy Officer at Cobalt. So, at Cobalt, what we do is pen testing as a service. We build a security software platform – like a lot of DevOps companies.

    We have data-driven product-based teams. We have – we really value automation and failing fast. I’ve spent 15 years in the information security industry. I wrote a book called Security Metrics. I host a podcast called Humans of InfoSec, and I teach the OWASP Top 10 Learning Path on LinkedIn Learning, and super excited to be here with this group today. Thanks for having me.

    Shimel: Thank you, Caroline. Next up, we have Kurt Chase.

    Kurt Chase: Hey, everyone. I’m Kurt Chase. I’m currently Head of Release Management and Engineering Services at Tricentis. Engineering Services is a group that supports the engineering function so, we do our own type of testing – testing all the processes and everything we use – to deliver our software at Tricentis. So, prior to Tricentis, I spent some years at Splunk and at AutoDesk. It’s great to be here with all of you today.

    Shimel: Fantastic.

    Chase: Be here today with all of you.

    Shimel: We got what you meant, Kurt. It’s okay. Next up, we have Ben Simo. Ben, why don’t you introduce yourself?

    Ben Simo: Hi. I’m Ben Simo. I’m a software tester of 30 years and past president of the Association for Software Testing who has recently stepped into a product management and product research role at Tricentis trying to discover and develop our testing platform of the future and bring new things to Tricentis customers and hopefully introduce new people to Tricentis with new products that enable new ways of testing.

    Shimel: Okay. Happy to have you here. Next up, Tracy Miranda. Tracy – well, I’m gonna let you introduce yourself. Go ahead.

    Tracy Miranda: Hi, everybody. I’m Tracy Miranda. I’m the executive director at the Continuous Delivery Foundation. It’s an open source organization that’s part of the Linux Foundation, and also home to projects to like Jenkins and Spinnaker, and our members include companies like Netflix and Google. And I have over 20 years of experience in open source so, I love open source and I’ve worked in it as a developer – also in QA and all different facets. So, yeah. Big fan.

    Shimel: Thank you, Tracy. Last, but not least, is my co-host on DevOps Unbound – my partner in crime here, Mitchell Ashley. Mitchell, why don’t you introduce yourself?

    Mitch Ashley: Thanks, Alan. Yes. I’m CEO of Accelerated Strategies Group, which is an analyst firm that focuses on, in addition to cyber security, digitial transformation, DevOps, and cloud-native kinds of applications, which really matches my background very well. I’ve been both a product creator CEO/CTO as well as running IT, CIO, CSO – that kind of thing. So, currently, I’m also CTO with MediaOps – I guess CSO at MediaOps, too, so, I’m trying to collect as many three or four-letter titles as I can. So far so good.

    Shimel: Alrighty. So, first of all, thank you all for coming on today. Secondly, testing – what a wild, mixed-up world we have in testing, right? Just from looking at the folks on our panel today, we have security testing – pen testing, in particular. We have the world of open source and what a factor that is in testing.

    We have product testing, right? We have sort of new product testing – taking testing to new heights. And you know, that variety is sort of a double-edged sword, right? Because it’s like jack of all trades, master of none type of thing. There’s so many aspects of testing, right?

    Even in our audience here at this virtual event, we have testers who come from, you know, what’s called old line QA testing, right, to DevOps enabled testing to visual testing – testers who, if it’s not an open source testing platform, they won’t use it, to proprietary systems. Through it all, though, a lot of times, all of this testing becomes sort of a red-headed stepchild or a whipping boy, if you will, right, to the whole application development system operations teams. It’s like, at the same time, their best friend and their worst enemy, right? They want everything tested, but they don’t want to give you the time, the budget, or the resources to do it, and then, when something doesn’t work, they said, “Well, we should have tested more, right?” So, how – first of all, do we agree that testing has been sort of the whipping boy of a lot of what we’ve seen go down in app dev and operations? Anybody want to jump on that?

    Chase: Well, I think it’s certainly gotten a bad name. I think some of that comes from prior to the digital transformation. A lot of the engineering was waterfall, and testing was always that last stop before the software went out live, you know? Nowadays, I think everyone’s come to realize the speed and scale at which we develop things, you have to do the testing up front at the very beginning. You can’t wait till the end, you know?

    You’re gonna be too slow. So, I think, from that perspective, absolutely. I think that’s why testing has really, I think, gotten a bad name, because they’re kind of the blockers to getting anything out into the public domain, whereas nowadays, you have to enable that up front because you want to get it into the public domain yesterday.

    Shimel: Yeah. And so, in many ways, to me, that sounds like, well, DevOps has been the best thing that ever happened to testing, right? In many ways, DevOps has been the best thing that’s happened to security testing, right? We’ve certainly seen that shift left, if you will. But has that shift left helped – has that rising tide lifted all boats?

    Miranda: I still see –

    Shimel: Oh, okay, Tracy. Go ahead.

    Miranda: I’m gonna jump in and say one thing that strikes me as funny. Like, we have, in our industry, everybody has embraced the DevOps terms, and then, you start to see all these spinoffs to put emphasis on things like, for security, we have DevSecOps, and you’ve got every variant of something with ops. But when it comes to testing, you know, there’s no TestOps. There’s no new buzzwords and it kind of speaks to how much it’s being neglected. Like, we had a conference on continuous delivery, and we asked for talks of everything in the life cycle, and we got lots and lots of submissions except for testing.

    So, again, I think it still is really under-appreciated and just not given the due credit for what it stands for.

    Shimel: Very –

    Simo: Yeah. And I think some of this comes from, as Alan mentioned, that testing is a very broad and diverse subject, and I’ve often watched people argue about testing that each interpret it to their piece of what they do with testing. And things like CI and some of the DevOps practices have helped us improve with the checking that our software can work – you know, demonstrating that it can work – and there’s a lot of people – if you primarily wear a developer hat, you’d look at that’s what testing is. And we’ve automated that.

    There’s been huge improvements in tooling and technology to do that, but sometimes, along with that is we throw out that we need someone involved that’s looking at things from a critical perspective that’s concerned about the safety and efficacy of what we’re delivering that is the, I guess, the one questioning and searching for information that they raise issues that we need to deal with that are reasons to not push things out to production now – so that we can make informed decisions. So, often, I think just that critical side gets thrown out as we improve the other side of testing, which is being able to demonstrate that things can work.

    Shimel: Anyone else?

    Ashley: You know, Alan, one of the ways you can tell who drew a diagram is to look at what’s in the center. And so, if we think about CICD, we kind of surround testing, but we don’t talk about testing. So, in some ways, it’s elevated, and some ways, it still sort of abstracts it out of DevOps. But my experience has been not really, because most teams know that testing has to get automated as part of that process, ’cause you can’t have this automated checking and automated deployment, then let’s kind of let a bunch of manual things happen. And not that testing hasn’t been automated – it has been – but I think it has elevated, at least in part, I think a good deal.

    My experience has been. I haven’t worked in every environment, maybe, where test still isn’t thought of very highly. By the way, the thing that the – it isn’t the last thing in the chain. Documentation is the last thing.

    Chase: That’s true. That’s true.

    Ashley: Let’s give documentation –

    Chase: And release notes. It’s true. I have tech doc’s team, too. You’re right. [Laughs]

    Shimel: And that hasn’t really changed much either, though testing has. No. No. But you know what? Caroline, I’d like you to focus in specifically.

    I mean, I think about – you know, Mitchell and I co-founded a security company, and we came out with a product in 2003 of just a vulnerability assessment and manager – “VAM”, we called it – and at that point, look, we used to beg people to scan their infrastructure at least once a year, right, if not quarterly. We were really pushing for quarterly. And you know, the idea of scanning and testing applications before you deployed them – unheard of. It was unheard of, right? We didn’t have the security tools or people.

    We did that after the fact. Now, that’s not normal anymore, right? Today, at least 85 percent of applications are tested prior to deployment. I don’t know who those other 15 percent are, but that’s a whole nother story. But talk to us.

    You’ve lived through this, Caroline. What do you think?

    Wong: Sure. So, you know, I think there needs to be an acknowledgment. There’s sort of these like, first order priorities and these second order priorities. When I think about security – and I think some of these concepts apply to testing more broadly – there’s a saying that, “You don’t take a $200.00 fence to protect a $5.00 asset.” I think security is about you have something of value and so, you want to test it – rather, so, you want to secure it.

    Testing, I think, is like you made something new and so, you want to test it. You’re not gonna test something unless you have something new. You’re not gonna secure something unless you have something of value. So, it’s okay that security and testing are not sort of the number one thing, you know? Inherently, by their definition, it’s sort of a second order attribute, if you will.

    Now, I think, you know, each of us has been through a version of annual organizational budgeting. You know, each organization says, “Okay. Here are the goals for the years. We’ve got limited resources and budget to accomplish those goals. How are we gonna allocated those resources?”

    And there are some things that, for a lot of organizations, are very clear. You know, even if I take a step way back and I say, “Okay. Most organizations might think about budget allocations in three buckets. Number one – general and accounting; number two – sales and marketing; number three – research and development.” You know, nowhere in those words is either security or testing.

    And that’s okay, you know? I think, Alan, when you share your story about VAM, you know, Vulnerability Assessment and Management, I agree that over the last, I would say, decade-two decades – so, when I started my security career 15 years ago, we were almost trying to convince folks that breaches really happened – that this was really a thing to be worried about. You know, today, everyone knows because it’s on the media headlines every day. This is real. And so, that has been a shift in the perception, I think, in our industry.

    There’s also been a big compliance driver. So, the – in the area that I’m very familiar with, security pen testing, a lot of that pen testing is actually driven by external compliance requirements – things that say, “Because of compliance framework A, B, C, you must actually conduct manual pentest on your application, for example, once a year.” You know, doing it more often than that is often because of perhaps what a lot of this group might think of as a really good thing – trying to actually secure your software, trying to actually test your software to ensure it does what you think and what you intend for it to do, but that’s not necessarily the first thing that comes to an executive’s mind when they’re thinking about budgeting, and there’s a variety of reasons for that, right? There’s an external perception thing. I think there’s also a lot of learning, certainly, that the security industry can do to be more approachable and to really understand better what is the software development methodology that we’re engaging with.

    What is the specific methodology and behaviors of these individuals? And how can we actually collaborate with them most effectively?

    Shimel: Fair enough. I mean, look, I don’t want to be the pessimist in the group or the bluebird of happiness here, but we have had success over the last 5-10 years, Caroline. You’re right. Especially in security, but not just security. We’ve had success in testing across the board.

    Tracy, to your point, the fact that we don’t have DevTestOps, I think, is, you know, testament to the fact that we’ve recognized that testing is so critical to the CICD process that it’s part of it. I mean, you can’t have DevOps, you can’t have CICD without a level of continuous automated testing in there – in my mind, anyway. I mean, prove me wrong. But, you know, I think that has been probably one of the biggest successes that the DevOps movement has brought – has been responsible for.

    Miranda: Yeah. And I really like Caroline’s analogy with security because, you know, that’s something else we think of as fundamental to DevOps, but it’s sort of elevated itself and, in a relatively short space of time, went from being ignored to being, you know, headline news. And I think there’s a lot of lessons there. And I – like, I will say, I do think testing is going through that journey and we’re probably on the cusp of it kind of really exploding and watching a few players in the industry and companies starting to do really interesting work in testing. I think that’s gonna turn around very shortly.

    So, yeah, there’s an exciting space that I think its moment is coming – just like securities has.

    Chase: Yeah. I think it’s all about the testing now, quite honestly. I mean, when you look at the future with low-code/no-code, it all becomes question-question mark ops, you know? Any function in a company could actually put something together and run it, and they’re gonna have to test it, you know, and it becomes less about all the engineering and more about the test patterns and things like that that are needed as you get to these future states. So – and speed and scale is gonna continue, I think.

    That’s driven everything with the digital transformation. It’s speed and scale. Everything needs to come faster, and we want more of it, you know? And so, I think that really pushes the envelope, and that’s what DevOps has responded to and, I think, addressed very, very well, but it’s not over. We’re gonna continue to scale up and go even faster.

    So, we have to be prepared for that, and I think it becomes more about testing in the future, and less about, you know, we’re gonna have these pre-built components – APIs you can piece together however you want – then, testing becomes, I think, almost first order, to Caroline’s point. And security, especially. I think security, we’re seeing that change now. If you’re a SAS company especially, security has to be top in line, ’cause you just can’t put something out there and not have it tested from a security perspective. And, furthermore, with compliance, it’s not just ISO and GovCloud and those programs, but large corporations now are saying, “Hey, if you want software in my ecosystem, these are my rules and I want to see your pentest reports and I want to see this.”

    So, I think more and more people are realizing that if the software’s not tested, it’s not worth it. And so…

    Simo: And, in my experience, part of adopting a DevOps culture and not just tool set is putting testing at the forefront. Some of the most productive teams that I’ve worked with that deliver some of the highest quality software I’ve seen – testing was center to absolutely everything that everyone on those development teams did.

    Ashley: Hm-hmm.

    Chase: Yep.

    Shimel: Yeah.

    Miranda: Yeah. We think – go ahead.

    Ashley: I’m just gonna jump in and say – there’s another factor that actually works to our benefit and that is not only to speed through automation and DevOps process, but contemporary software architectures, containerized micro-services, service mesh – we’re releasing smaller bits of code in encapsulated forms, which means, you know, we’re not testing monolithic applications all the time anymore so, we’re not having to run two weeks’ worth of tests just to get one pass through.

    Chase: Yeah.

    Simo: Right. _____ container solved the “It works on my machine” problem because we can ship the developer’s machine.

    Chase: Yep.

    Shimel: Yeah. That was always a big problem, right? Tracy, you were gonna say something though.

    Miranda: Yeah. No. I think the direction we see it heading is we’ll start breaking the testing monolith to talk about what is unit test, what is system test, and appreciate that is a specialization in itself. And the other things I’m starting to see from some of the end user teams we talk to is this real focus on culture – you know, a continuous quality culture – and seeing that hand in hand as part of the testing movement and again, _____ back to security – you know, the security first culture went hand in hand with that whole emphasis on security so, similarly, the continuous quality culture is starting to be something that – like, the really high performing teas we see pay a lot of attention to.

    Shimel: Absolutely. You know, Kurt, you said something there about low-code/no-code and how that could be a game changer. You know, some people refer to that as the rise of the citizen developer, and I’m wondering if maybe what we need is the rise of the citizen tester, right? Is testing too hard?

    Chase: Yes. I waste gonna say – the one thing we haven’t touched upon – I think Caroline did – is testing is hard. Security testing is super hard. And when you look at today’s modern environment and a lot of the ISVs – at least the ones I’ve seen and been associated with – you have a legacy of monolithic applications that you have to support that are bringing in a ton of revenue, and, at the same time, you’re transitioning – you know, going through your own digital transformation. So, that adds complexity after complexity in testing, and I do think, in the future, testing – you know, a lot of us, like Ben said, to his point, we have to push testing up – you know, shift left, move it far left as we can.

    And that’s gonna continue. I think that has to be the norm. To Caroline’s point, it has to become first order. And I think it’s starting to right now. I really do. Especially the – all the SAS solutions that are out there – you know, shipping code daily, hourly. You need excellent testing to do that and do it with competence.

    Shimel: Yeah. I think you do. But we do have to – and it’s funny that we’re talking about making testing easier because – not security testing, right? Security testing, pentesters – they were always sort of the hired guns of the security space, right? There were SAS or – there weren’t companies like Cobalt, right?

    The pentesters that I knew growing up – growing up – you know, that I knew coming up in the security space were gun slingers. These were, you know, white hat/gray hat folks who would come in and they’d do a little social engineering and pen testing using some open source tools, and you know, they give you a report. But you know, these were folks who traveled around the world doing that kind of thing. You know, we didn’t have a company like Cobalt that did this as a service, so to speak. So, it’s very different.

    But here’s the underlying thing with like, QA kind of testing – and, you know, I don’t mean any disrespect to anyone in our audience today, but for too long, a lot of folks in testing were viewed as failed developers or people who couldn’t be really good developers did testing or maybe testing was a weigh station on the way to becoming a real developer – like a Pinocchio stop or something, right, where you’re not a real boy yet. But that’s not true. There’s also – there’s a whole cadre of professional testers. And you know, in their capable hands, you could do great things. But I guess the question is – does testing have to be so hard that we need professional testing – professional testers to do testing?

    Or can we make testing automated enough that you – I’m not saying it has to be the dunces who don’t become developers. That’s ridiculous. But can we broaden the base of people who do testing? And that includes security testing? I’ll throw it out. Ben, you were a big-time guy in the testing industry. Tell us – what do you think?

    Simo: So, I think we can expand who’s involved in testing and that more people than those currently doing testing can learn to do testing. I don’t know _____ developers learning to become better testers and test their stuff. But I will say that ultimately, the part of the work that is the testing cannot be automated. Tasks that we do in testing can be automated. So, whether – not matter what hat someone primarily wears, if it’s a developer, you know, writing tests to demonstrate their code works or someone’s using TDD and they’re building some tests and then, building some code that matches it, the testing is the critical evaluation that’s going on in their mind of solving the problem and demonstrating that it’s solved – or evaluating how something might go wrong and coming up with some approach to demonstrate that it may go wrong and check for that.

    So, those things, whether you’re doing it automated or the execution is automated or done manually, the actually testing work, I’d argue, is the cognitive work that people are doing. And there is some – that we can never automate away. There are some ways that I think AI can help us. There’s also some ways that may lead us astray. But ultimately, I think we need people to learn to be critical thinkers to understand the technology, to understand the domain of the users and the problems that are being solved, and applying that critical thinking is the testing.

    Of course, then, we want to use tools to help us with some of our tests so, when we come into DevOps and CI, executing those demonstrations of the software working is a key part of that. Now, whether we do it as part of our, you know, building to those tests or we create them afterwards, then they can, at the very minimum, act as changed detectors going forward. But there was some testing in people’s minds that was going on that created those.

    Shimel: Caroline, what about – okay.

    Wong: I agree with quite a lot that Ben is saying. You know, I can’t help but think to myself that – I actually have this sort of visual in my mind, and it’s actually at a circus of a trapeze artists, you know? I think about these people, and they’re sort of like, whirling in the air and they’re doing these like, really incredible feats, you know, and they just do whatever they do, you know? But there’s like, a safety net to catch them if something goes wrong. It also reminds me of a metaphor that is often used in security that says, “We put brakes on a car so that you can go fast.”

    You know, when Mitch said earlier, you know, “These days, we’re releasing smaller pieces of code, modules of code” and there’s a way in which, yes, each small piece of code – each module – can be tested and some of that can be automated. We should automate performance testing. We should automate unit testing. But, to Ben’s point, there’s a lot that can’t be automated. There’s a lot of testing that actually depends on things like human judgment and opinion and it actually requires a person’s decision, you know?

    So, I think there’s – I think there’s a couple of frameworks for us to consider, right? If the first order is about creating value and making something new and the second order is about testing to make sure it does what you wanted it to do and protecting the value created, there’s also this spectrum that says there’s smaller bite-size stuff that may be a good candidate for automated testing, but, I think it’s super important to recognize that these small pieces of code – these modules that Mitch was talking about – it’s also super important to test what happens when they come together. So, to the extent that security folks need to be able to test chained exploits and business logic flaws, you know, those are exactly the types of things that will never be automated – is what I think, anyway.

    Shimel: I don’t disagree. I think security in particular is just there’s some things you’re just not gonna automate. I don’t care what you say. But that’s okay. Anyone else?

    Chase: I do want to make a comment on something you said. This was a long time ago now, when I was a dev manager, that I used to sense that when my developers – you know, if you were on my team, if you wanted to really get me mad, treat your tester badly. Treat them, you know, a bit, like you said, like they’re not quite at the level you are. I would be incensed as a manager. You know, that was a long time ago.

    Fortunately, I don’t see that as much nowadays. I think that role has been elevated where I don’t feel that at all on my teams, and if I do, that is, again, to this day, I’ll get super mad about that because I think it takes all kinds to develop software today and ship software. Testing role is just as critical as an engineering role, just as critical as the ops’ role, the security role. So, yeah, that was one thing that would drive me nuts ’cause I did sense that and go through that, and this back in, you know, the late ’80s-’90s, if you will, before the year 2000 – I’m dating myself – but it definitely was palpable in some of the large ISVs, you know? And as a manager, that was something that would drive me nuts.

    Shimel: Agreed. Agreed. Tracy, open source – look, testing has been inexplicably wound up with open source for as long as I’ve been around, right? Whether it’s security – open source security testing and you know, things like Metasploit and Nessus and Nmap or you know, in QA testing kind of stuff, it’s a rich history of open source testing tools. How do – and part of your mission at CICD is – at the CICD Foundation – is to nurture that, I’m going to assume, correct?

    So, how do you guys nurture that? What are you – what do we do to maintain that rich, open source tradition here?

    Miranda: Yeah. No. Absolutely. And, as I say, we at the Continuous Delivery Foundation, we want to look across the entire software delivery life cycle, and I think it is safe to say, you know, that testing hasn’t risen to the top. We’re actively – like the board wants it to be a more focused-on topic, the technical committee wants to encourage folks to bring us more projects in open source and with testing, but we’re still having to kind of fight through all the noise around other projects.

    And one of the ways we’re approaching it is, for example, our technical oversight committee wants to set out a landscape of the types of projects in each kind of bucket and say, “Okay, look. We need to have something in security. We need to have something in testing that covers these different aspects.” Almost like a call to action to say, “Bring us your testing projects and –” like my favorite saying about testing is, “There’s two key problems in testing – not enough tests and too many tests.” And yeah, I think we’re starting to see some really good frameworks around test generation.

    I’d love to see more open source options for that. And Kohsuke Kawaguchi, the famous creator of Jenkins, move on to focus on testing so, that says a lot about the importance of this area getting more open source focused. So, yeah, I’m excited to see what could come out, but we do have to actively encourage folks to come join us and have those conversations and not to, you know, have it assumed it’s all focused on delivery of pipelines for delivering software.

    Chase: I think another thing you’re seeing today is a lot of people, when they start developing an app, when they start creating something new, that’s the question they ask up front. “How are we gonna test this? How are we gonna automate it? How are we gonna pentest it?” I think those things, at least, are starting to bubble up a lot stronger in the cycle – you know, early on. ‘Cause I know that’s something we recommend highly – is that if you’re developing something new or need to head down a digital transformation, think about the testing. How are you gonna test that up front, you know? And products are actually being developed with that mindset now.

    Simo: And with short iterations, it’s how are we going to test this tomorrow, next week? It can’t be left off for months later to be someone else’s problem. So, people, I think, are learning to deal with those questions about testing from the start of their discussions about what the requirements are.

    Chase: The other thing I love – and in Caroline’s world, has been very successful – I know it is specialized skill set, but trying to teach that throughout your troops – you know, throughout the engineering team. Show them what, you know, across _____ scripting errors. What does it look like in code? That’s super successful. That really, really helps make – ’cause security is difficult, you know, no matter how you slice it.

    Even when you’re looking at product development – if you want to invest in fixing security defects or add a cool new thing for your users, you know, a lot of times, the cool new thing is gonna win out. But that’s starting to change as well, too. I think people are starting to value security – especially in today’s modern world, you know? It has to be up front, really.

    Ashley: Yeah. It’s all about AppSec at the moment. That’s one of the hot topics as well. You know, someone mentioned AI earlier. I just want to offer a perspective about that because we always seem to focus on the shiny new object.

    I know I have. I often do. But I think what we’ll see with AI is select applications when it comes to testing. The sort of vertical place is where it’s really well suited. When we say, “Yeah, AI”, we really mean machine learning in most cases, which is machine learning algorithms, and there’s two types – there’s supervised/unsupervised.

    Supervised is, “That’s a car. That’s a car. That’s not a car, right? That’s a bug. That’s a bug.”

    And unsupervised takes massive amounts of data and intellectual patterns and learnings from that. So, I think that as we’re taking our test automation and our test tools, there may be some of that, and it might be really useful for AI. I wouldn’t get worried about people being replaced as testers. Not in my lifetime, that’s for sure. It is still a – there’s very much a craft to it, just like there is software design.

    There is a craft to testing and a skill and a mentality about it that’s unique, just like with security.

    Shimel: No doubt. Caroline – oh, go ahead, Ben. I’m sorry. No, no, Ben. Go.

    Simo: Yeah. I was gonna say – and with any other sorts of automation – even you know, not learning automation that’s come before AI and machine learning based things, is that it not only has potential, in some cases, to replace some of the mundane work we do, but to enable to the skilled people to focus their work and apply their brain power to the things that really matter. So, I think there’s great potential for AI to help direct people’s attention to the things that maybe should be of concern and help deal with that big question of testing out of infinite possibilities of PaaS and data variations. Which one’s gonna show me the problem? And I think there’s huge potential for AI to help us find that if we’re also aware it may guide our eyes away from something that we do want to see.

    Shimel: Caroline, how does that relate to security, though? I mean, is AI helping Cobalt with pen testing? Be honest.

    Caroline Wong: So, the type of software that we’re building today is actually workflow and collaboration software. So, what we’re trying to do, is we’re trying to bring security folks and developers and penetration testers all together sort of in one place where they can collaborate. The technology piece, for us, is not about automating defect discovery. It’s not – I mean, if we’re gonna us ML or AI in the future, honestly, what we’re gonna do is we’re gonna use it to say, “What are the characteristics of this organization that wants us to do a test on this particular type of asset? What are the attributes of that asset and how might we match that to our talent pool – something that we do totally manually today?”

    I think that’s the opportunity. I don’t think that we’re gonna get to a point where we’re using the AI and the ML for automating the defect discovery itself. Frankly, I think that even if I take a step away from pen testing and I look at the security scans that are in existence today, I think, you know, whether or not they’re referred to as using AI or ML techniques, they’re just simply imperfect, you know? And there’s a super important place for them, but there’s a lot of false positives. There’s a lot of interpretation of data.

    And I think that’s always gonna be the case. I think that no matter what, when it comes to humans and machines, the machines can add a lot of value, but I don’t know that when it comes to software development testing and security in particular that there’s every gonna be a complete substitution. I don’t think that – I don’t think that makes sense.

    Shimel: Absolutely. So, guys, we’re coming up on the end of our time. I need to wrap up. Tracy, let me say this for you, if you don’t mind – and if I’m wrong, correct me. For you testers out there, the CICD Foundation – or CD Foundation, actually; I keep – we always say “CI” like it’s attached at the hip – but the CD Foundation is looking to hear from the testing community.

    Probably some companies – including Tricentis – may want to look into becoming involved with CD Foundation because if you don’t have a seat at the table, you don’t have a right to complain about not being heard. And so, you know, when we look at foundations like Linux Foundation and the CD Foundation, CNCF, we need the testing community – vendors, professionals, folks in it, you know – to be involved and get involved. Be heard and be part of it. So, I’m gonna extend that out there to everyone. But, short of that, though, Caroline, Tracy, Kurt, Ben, and, of course, Mitch, thank you so much for joining us on this special edition of DevOps Unbound.

    We hope everyone enjoys the rest of their Tricentis Virtual Summit. This is Alan Shimel for MediaOps, devopsy.kinsta.cloud, Security Boulevard, Container Journal, Tech Strong TV. Have a great day, everyone, and thanks for joining us.

    [Musical outro]

    [End of Audio]

  • EP 15: Why Testing is so Frustrating

    EP 15: Why Testing is so Frustrating

    In this episode of DevOps Unbound, hosts Alan Shimel and Mitch Ashley are joined by Adam Kalsey of Tricentis, Brian Dawson of the Linux Foundation and Caroline Wong of Cobalt to discuss why testing is considered the most frustrating bottleneck for developers and engineers. Panelists will explain the benefits of testing early on in the SDLC and how to accelerate the feedback cycle with automated and continuous testing, as well the pros and cons of testing, the value of shift-left testing and how to effectively balance quality and speed. The video is below, followed by a transcript.

    DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.

     

    Alan Shimel: Hey, everyone. I’m Alan Shimel, and you’re watching DevOps Unbound. DevOps Unbound is sponsored by our good friends at Tricentis, and it consists of a biweekly, 40, 45 minute video panel on relevant topics within the DevOps sphere, and featuring a panel of experts that rotates every other show, and really great people, and something I personally really enjoy doing.

    And then, about once a month or once every six weeks, we actually have a live studio audience—or there is no studio audiences in COVID, actually, but we have a live virtual audience that participates in our show. Today’s show is just our panel, though, without a live audience, and I’m really looking forward to our discussion.

    Our discussion today is around testing. You know, with the pressure on developers to deliver code faster, sometimes testing is considered a road bump in the way. Testing is considered slowing down the whole process. There are a lot of developers who’d like to push testing off that plate. Of course, no one wants to go back to testing after we’ve deployed software, primarily, but there is that tension there.

    Let me introduce you to our panel today; we’re gonna discuss this in depth. First of all, making his first appearance as a member of The Linux Foundation on our Tech Strong TV network is our old friend, Brian Dawson. Brian, welcome to DevOps Unbound.

    Brian Dawson: Thank you, Alan. Thank you, Alan. And it’s fitting that the first panel, first webinar I do since joining The Linux Foundation is with you.

    Shimel: It’s our pleasure to have you on. Brian, what’s your position at Linux Foundation now?

    Dawson: So, at The Linux Foundation, I am functioning as the VP of Developer Relations and Ecosystem Development. So, just to help, we’re gonna help The Linux Foundation to the development community while driving more of a DevOps opinion into a growing ecosystem of solutions that support open source development.

    Shimel: Fantastic. Next up is my good friend, Caroline Wong. Caroline is—oh, she’s an anchor on many of our different shows here at MediaOps and Tech Strong. Caroline, it’s great to see you again. A little background, maybe, for people who aren’t familiar with you?

    Caroline Wong: Great to be here. Thank you so much. My name’s Caroline Wong, I’m the Chief Strategy Officer at Cobalt. We’re a pen test as a service company.

    I actually started my DevSecOps career 16 years ago leading information security teams at eBay and Zynga. So, in both cases, we were running online operations 24/7 with millions of simultaneous users daily. And now, at Cobalt, we build security software. And, like many DevOps companies, we’ve got data driven, product based teams. We value automation and failing fast, and super excited or today’s conversation.

    Shimel: Fantastic. Welcome. Great to have you, Caroline. Our third panel member is another returning DevOps Unbound veteran, and that’s Adam Kalsey of Tricentis. Adam, welcome.

    Adam Kalsey: Thanks. Thanks for having me. I lead Developer Relations for Tricentis. I’ve been building web applications since 1995, and have always had a soft spot in my heart for bad applications and broken things. And so, I love to help developers build higher quality software and do it in a more efficient, faster way.

    Shimel: Absolutely, and thanks for being on with us today, Adam. Last but not least, in his Florida outfit here, visiting me in Florida, is my Co-host of DevOps Unbound, Mitch Ashley. Mitchell, welcome.

    Mitch Ashley: Coming to you from the MediaOps Studio B.

    Shimel: Yes.

    Ashley: Great to be here. I’m a former product, both security and software SaaS, CTO as well as a CIO, and now I run Accelerated Strategies Group, an analyst firm that focuses on those things.

    Shimel: Absolutely. Hey, Mitch, thank you.

    So, let’s get started. Adam, you’re with Tricentis, the sponsor of DevOps Unbound, and they’re a testing company, probably the leaders, one of the leaders worldwide in automated testing, continuous testing. Have we put too many straws on the developers’ camel’s back by making them, you know, be more responsible for testing to say we need to test more of our code earlier.

    Is it a bridge too far for many developers? Is it making them resent testing? You know, what’s the current state?

    Kalsey: I think where we’re failing developers is by handing them lots of more things to do without giving them any—without giving them what tools, and I don’t mean just necessarily software tools, but without giving them the things that they need to do it effectively. If we just pile things on developers and say, “Oh, this is your job now” but we don’t give them the support, we don’t give them the training, we don’t give them the expertise, we don’t give them anything to make that better—sure, it’s just gonna be harder on ‘em.

    Shimel: To play devil’s advocate—oh, those poor little testers making a quarter of a million dollars a year each. Do we want ‘em to do testing, too? Right? Is there some of that here, Brian? Or—go ahead, Adam?

    Kalsey: No, no, Brian—please, please, jump in.

    Shimel: Well, he’s from The Linux Foundation, he now represents these folks. [Laughter]

    Dawson: Well, first, not all of them make a quarter of a million dollars a year, right? But I do think what you raise is important when you bring up the investment, right? We are hiring talent with a particular expertise in a particular domain, a particular ability to solve problems and innovate.

    And there is an argument to be said, Alan, that, “Do I want to spend this money, this quarter of a million dollars plus 200,000 in RSUs on having somebody write test harnesses?” Now, I think—you know, that can be balanced by what Adam said. Well, first of all, yes, because we want everybody thinking about quality. But you know what, if we want a return on that investment, we wanna free the cognitive load, then you know what? We need to invest in automation, we need to invest in better collaboration and reducing friction from the process.

    Shimel: Agreed, agreed. Caroline, you know, it’s been a big—this whole DevSecOps movement and shift left has been about shifting security testing left and getting developers more involved in security testing, right? Yet another camel on the—yet another straw on the camel’s back. Should we be doing that?

    Wong: So, I—oh, my Internet is awful. Can you hear me?

    Shimel: Yes, we hear you fine.

    Wong: Okay, fantastic, awesome. I’ll just—I know that there’s post editing, so it’s okay for me to say that.

    Shimel: Mm-hmm.

    Wong: So, Alan, I bring a particularly unique perspective to this group in the sense that I actually oversee people at our company, I oversee Human Resources. And I’ve been with the company since we were 10 people, and today we’re 200. And one of the super interesting things that happens is, naturally, as a company grows, what people do, they specialize and then we have to figure out how to communicate more effectively.

    And I think that if you’ve got an engineering team and you say, “Okay, now you’re responsible for Development and you’re responsible for Ops, and your responsible for security, then there’s got to be some level, depending on the maturity and the size of the organization and what the organization’s trying to accomplish of specialized roles and protocols for engagement.”

    And I think that when, either number one, those roles are not clear to, I think Adam was making a point earlier about security folks need to be providing guidance to engineers where—because engineers are not expected to be security experts, frankly. Security experts are not expected to be engineers. And then, at the point where information is being handed off where the responsibility for taking an action, those, all of those sort of procedures, they have to be clear, otherwise they’re simply not going to happen.

    Shimel: Yeah, agreed. I think they have to be clear. But even if we make them as clear as we possibly can, is it just—are we putting too much on developers? I mean, I think fundamentally, that’s the question here, right?

    Wong: There’s—

    Ashley: Alan, it’s not only—sorry, I’m just gonna jump in. It’s not only quantity of work—is it the right work?

    Shimel: Right.

    Ashley: Because I know from writing software and being part of software teams for so many years, there’s this thing called unintended consequences or unintended use. Well, you’re not supposed to use it that way, but the person who designed it intended it to be used a certain way, but one of the important functions of tests is not just to uncover bugs and elicit those things or security issues, it’s also to push the boundaries about how you might use the software and essentially stretch it to try to find where it’s gonna break where they can. And oftentimes, that’s a different mentality than or perspective, certainly, than the person that wrote the code.

    Dawson: And if I may jump in and say maybe part of the discussion can get to generalist versus specialist, right? There is this idea, Alan, that when you talk about putting a lot in the developer’s hands, this idea that there is this prolific person called the full stack developer that can separate themselves from what they develop to look for security vulnerabilities, to test it, then ultimately deploy and run it, right? And I think we need to give thought to, are we trying to build a bunch of generalists, which doesn’t potentially enable them to do any job the best, or are we enabling specialists to better work together, share information, and crowdsource quality and crowdsource security?

    And I’m just curious if any of the other panelists have an opinion on this concept of full stack developers versus, say, needing traditional, lifelong quality experts.

    Kalsey: Yeah, I like to say that we decided to create cross-functional teams and we did it by removing all of the functions, and we centralized them all in a single person—and that’s not a cross-functional team. That’s—

    Shimel: That’s a cross-functional person.

    Kalsey: Yes. Well, and you end up with the jack of all trades, master of none problem. You have to have the specialists on even a cross-functional team. You have to have, there’s a lack of testers, there’s a lack of UX people, there’s a lack of security people. They have to be part of that team. You have to have people that, dedicated folks to do the hard work. A developer may be able, and a generalist may be able to do a lot of the easy things and a lot of the, “Oh, yeah, we’ve done this before or we’ve seen this, and this is a known know, let’s go fix that.”

    It’s the hard work of figuring out—what don’t we know, and then having the expertise to be able to know what that hard work is, even, to go after it. And without that, of course the developers are going to be overloaded, they’re being asked to do everything and they don’t know what everything is.

    Shimel: Agreed, agreed. I mean, I think, again, security is the poster child for this, right? We’re asking developers to be security people or to make security a priority. Now, you ask most security people four or five years ago, they’d tell you you’re smoking your socks. Developers don’t care about security. They never did. Only we care about security.

    It’s changed. It’s changed a little bit. We do think other—no one wants to develop insecure code. But we can’t ask them to be security people—or can we, Caroline?

    Wong: I sort of think we have to. I think that, you know, the big, bad thing that no one wants to happen is a security breach, and a security breach cripples an engineering team’s ability to execute on a roadmap. You know, if something big and bad happens, then entire teams have to rush and commit to fixing this problem sometimes for months, you know, time which, of course, could be invested more profitably elsewhere.

    I have a quote that I’d love to share from a colleague. Denali Lumma is the VP of Engineering over at Project Ronin, and she also makes a really good point about tech debt. She says, at a certain point, it’s too late, there’s too much tech debt, too much complexity, the code base is too big, and it becomes impossible to do anything about it. What you see there is a business will slowly die because shipping features becomes harder and harder, critical vulnerabilities and bugs are introduced more frequently, and eventually it kills the business.

    You know, I often think of software as a living organism, right? Whether you think of it as a tree or as an animal, there is this balance between making sure you’re doing sort of your fun things, your running, but you’re also resting when you need to heal. You’re also—you know, if we’re talking about a human being, you’re also sort of going to the doctor once a year.

    Shimel: Agreed.

    Wong: Yeah.

    Shimel: Agreed. Thoughts?

    Kalsey: Quality has to be a mindset, security has to be a mindset, user experience has to be a mindset on the developer time, among everybody there. The user experience person needs to understand and have a mindset of developing and of creating a secure experience and not, “Well, we’re gonna get rid of all the security because that’s hard for people to use.” Everybody has to have that mindset, but you still need the experts and you still need the expertise to be able to do things well. Just having a mindset doesn’t make you fantastic at it. You can say, “Oh, I love to get out and I’ve got an exercise mindset, so I’m gonna go run a triathlon tomorrow.” It’s probably not going to work.

    Shimel: Probably not.

    Ashley: You know, it’s interesting—we could’ve had this conversation 20 years ago, 10 years ago.

    Shimel: Absolutely.

    Ashley: You know, why are we still frustrated with tests? What is it with tests that is, you know, it used to be sort of the orphan of the development process, you know, the only thing that happens later than test is documentation. Well, now, [Laughter] it’s still an issue. Why is that? Why are we so frustrated with tests today? I’m curious what folks think.

    Dawson: I’m gonna start with one not very complex thought—it’s actually hard.

    Shimel: It damn right is.

    Ashley: A little more respect—we need a little more respect for tests. [Laughter]

    Shimel: Well, but this was like, I remember when we first launched devopsy.kinsta.cloud, a popular theme was that, you know, DevOps would kill the testing star, right? That there would be less QA jobs because everything was gonna be automated continuously and shift left, we wouldn’t need testers—poppycock, right? We have as many testers as we ever had, if not more. It didn’t kill off testing jobs. I think it was Adam that said we’ve made testing part of the DevOps team. So, we have people on the DevOps team who are responsible for the various, you know, styles and types of testing that we do, including security.

    I think what’s happened, though, is, these DevOps teams—when I first started devopsy.kinsta.cloud, if you spoke to developers, they’d say, “Oh, this DevOps is all about the Ops.” And then you’d talk to the Ops people and they’d say, “This is DevOps, it’s all about Dev, it’s Dev-centric,” right? And that’s the sign of a good compromise when they’re both pointing the finger at that.

    But it has come—at least as far as testing goes, from what I see, it has sort of tilted where, you know, as we’ve shifted testing left, we’ve shifted responsibility left as well, which kinda winds up closer to the Dev, or certainly to the DevOps teams.

    However, though, let’s look at the alternative, and I’m gonna ask all three of you—all four of you have been around the block once or twice. My personal feeling is, is that we do a much better job of test coverage today than we did 7 or 10 years ago, and I think DevOps and continuous automated testing is a big reason for that.

    Thoughts? Anybody disagree? Show me wrong.

    Wong: I will challenge you, Alan.

    Shimel: Okay. I was hoping someone did.

    Wong: I’ll challenge you, and the reason is because—so, we’re all familiar with the OWASP Top Ten.

    Shimel: Very much.

    Wong: There have been a handful of versions of the OWASP Top Ten since the first was produced in 2003. I believe we’re gonna see another version later this year. The current version is from 2017, and if you take the 2017 version of the OWASP Top Ten and you put it right next to the 2003 version, it is sort of mind blowing how similar they are.

    Shimel: Yeah.

    Wong: And what that says to me is, we actually know exactly what the problems are. We know how to find them, we know how to fix them, and we know how to prevent them, but they’re still around, and so I think that’s—

    Shimel: They are, but are there as many of them, right?

    Wong: Absolutely. [Laughter]

    Shimel: You think there’s as many cross-site scripting or buffer overflows than there was 10 or 15 years ago?

    Wong: So, there are some categories that have been eliminated. You know, there are some categories that, for which we can define a secure pattern, and we can encourage folks to use that secure pattern, and for which we can actually eliminate some of these things.

    But there’s others or which we cannot. You know, I think about—I recently visited my sister’s home, and they have two young children, a 2-year-old and a 4-year-old. And at night, what they do is, they run a Roomba, which is brilliant, you know? If you could spend your weekend playing at the park with your kids instead of vacuuming, of course you’d wanna do that. And the Roomba works because they sort of take the time to get all the toys off the floor and make sure everything’s clear and once they use the Roomba, they clean it. There is this different work involved in effectively using a Roomba that’s different from sort of manually using a vacuum.

    And I do think there are these two distinct categories of testing, one category which you could say is known, repeatable, and automatable, and another for which manual testing, I believe, will always need to exist. There are types of security flaws like business logic flaws, race conditions, chained exploits that I don’t believe that in my lifetime, we will be able to figure out how to teach a computer to find these particular types of issues.

    Kalsey: And it’s beyond just security testing. I mean, exploratory testing, from a tester, your automated tests are not going to find, “This error message makes no sense in this context. I did this, I expected—I don’t know what this error message is trying to tell me to do.”

    Your automated tools aren’t going to find that. All they’re going to be able to find is, “It didn’t work, we sent an error message. We had the things we wanted to put in the error message there.” You can test that your code is doing the right thing, but if you didn’t design it to do it right, you’re not gonna find that with an automated test.

    Shimel: Fair.

    Dawson: I would tend to say, for the surveys that we’ve issued and I’ve read, your original statement, “Has test coverage improved significantly?”—yes, I think all indications are, we are getting better at automated testing, we are getting better at that pursuit of that thing we call test coverage.

    But I think what’s more important is, has the quality of test improved? Are we testing the right things, right, from functional testing to security testing, various shades of security testing? And I’d say (1) yes, we have improved there by sort of the qualitative and quantitative measures that I’ve seen. But I also think, as we truly start to focus more on quality, we start to achieve more coverage, we start to implement better tests, it becomes more clear, as I think Adam and Caroline are speaking to how much more work there is to do, how much other things there are to improve testing on. Pair that with a constantly changing technological landscape, right? We’ve gotta learn new stacks, we’ve gotta figure out how to test new stacks. We’ve got half of our put in in a SaaS platform, but then we deploy on prem, et cetera, et cetera, et cetera. And then you add to that that, at the same time, we’re trying to transform organizations.

    So, Adam, you talked about exploratory testing; Caroline, the human element. I think one of the things that pursuit of the full stack Dev, that pursuit of the fully automated processes failed to do in a lot of orgs I speak to is, we talk quality first, but oftentimes, the DevOps transformation doesn’t involve quality engineering first, right? And I think one of the things we need to get better at is engaging with and partnering with Quality and establishing our automated and modern processes.

    Kalsey: Yeah. We talk good in the testing industry a lot about shift left and that testing is moving earlier in in the process, and too many organizations have taken that as, “Well, the developers are the ones that are early in the process, so that means the developers do it.” No! Shift your damn testers left. Get them involved in the design phase. Get them involved in talking about the architecture. Get them involved in the entire thing, not shift the task to the people who are earlier in the process. Shift the whole process, please.

    Dawson: Or to a script, yeah.

    Ashley: I’m gonna take a whole ‘nother tack, and I thought you were gonna go here, Brian, is, I think we need to shift tests down. We need full stack testing. It’s not application testing anymore. Most of the security errors that we have are due to configuration errors. And what Caroline said was absolutely true. We still have SQL injection, cross-site scripting, and overflows may be less so. But why? It’s because it’s not just in the app layer that those things happen, it’s in that software stack. And I know not every tester can be an expert at the stack itself, but maybe that’s, instead of kinda dividing the DAST and SAST and the different kinds of testing that we do and just deciding, can we shift those in one direction is, let’s think about it from a depth perspective, because that’s where a lot of those errors are coming from. It’s not just the app.

    Kalsey: Oh, yeah, and the tooling on this is also, can be super fragmented. You need one thing to test your infrastructure code, you need another thing to test your UI, you need another thing to do the security—and they’re not, consolidating those reports and combining them and figuring out what worked and everything is hard work and there’s nothing that does that. This is a place that Tricentis could definitely improve and improve our own tooling here.

    And then, once you’ve done that testing, the results are often in a silo. Okay, I found out I had a problem—how do I fix it? How do I go see the data behind what was going on in the system to understand this and solve this? And now I’ve gotta dive into a monitoring tool or my logs or an observability tool instead of just having everything all in one place.

    Shimel: So, I think there’s a—I’m sorry, Caroline, you go.

    Wong: No worries, and thank you, Alan. So, we’ve talked about shift left, shift right. Mitch is kinda talking about shift up and down. I think there’s another axis, if you will, which is sort of what I’ll call shift in and shift out. So, you can test things in and out of band, and that’s gonna make sort of a wildly different difference in terms of how those processes run.

    You know, I think 10 years ago, security people appreciated the power that came with being a gate in the process, being an end band process. You know, today, it actually makes sense, I think, for a lot of security testing to be out of band, but that removes the sort of power and authority of security people to say something like, “You must fix this, otherwise you’re not gonna go live if it’s happening out of band.”

    Shimel: Fair. So, I’m getting a little dizzy, we’re going left, right, up, down, in, out. But—

    Ashley: It’s a circle. [Laughter]

    Shimel: Yeah, you know, shake it all about. But here, fundamentally, though, are we just kinda dancing—no pun intended—are we just dancing around the real issue, which is, we wanna do more pre-deploy, we wanna do more, we want better quality software, right? I think we could all wrap our heads around that idea and vote for that. We all want better quality software. Security people are at a premium. We can’t shift security people left or en masse. Testing people are also at a premium, though we’ve integrated them into the DevOps teams. Developers are at a huge premium, they’re probably our most expensive resource in terms of humans.

    What do we need to do to make everyone’s job easier here, to make this function better? Is it a question we’ve gotta automate more? I mean, maybe that’s what it is, if we automate more, or is it a cultural thing, right, where—

    Ashley: There’s also test-driven design, right, which is, as you’re designing it, you talk about, you put together your test strategy of, “Okay, we do it this way. Let’s talk about how we are going to test it, start to lay out the framework for how we think this can be validated.” I mean, it’s not the only thing to do, but I think—so, if you start with that in mind, now you’ve got a path to pursue as opposed to, “Here’s a chunk of software, you folks figure out how to test it, right? We built it, you test it.”

    Dawson: Yeah, and I’d add, I think there’s no—again, not only is the practice of testing harder than we often like to admit, but the, you know, the path to kind of getting to where we fully wanna be, where we don’t dread testing, complex, hard path, and I’d say it’s the DevOps trinity, you know, Alan, and we all can probably hit a few points on each.

    But you have to address people and culture, as Adam brought in earlier. There’s no getting away from, security needs to be a thought for everybody, right? I can install all the locks I want on the front of my door. Actually, over 24 years—today is my anniversary—my wife has programmed me to lock the door when I come in the house without thinking, right? She’s kind of impressed a security culture on me.

    You do have to address people and culture, process, practice, tools and technology. Automation alone will not solve it. A mindset alone will not solve it. All of that needs to come to fruition.

    Shimel: Mitch, make a note—invite Brian’s wife to our next security panel.

    Dawson: [Laughter]

    Ashley: I’ve already looked up her e-mail address—thank you, Brian, appreciate it.

    Shimel: Happy anniversary, by the way, too, Brian.

    Dawson: Thank you, thank you.

    Kalsey: This is one of the places where I think what Caroline said is so important, that this in bound, out of bound—if you’re trying to do everything in line, of course, it’s gonna be harder. In shifting that mindset to doing things continually in the background, we already do so much stuff and process automation in the background where we don’t have people check, “Here’s a step in the process.” No, this is going along in the background, dependency watching and that sort of thing and updates.

    We can be doing testing in the background as well for a lot of it, and identifying what needs to be in the foreground and what needs to be in the background. Look, I’ve seen CI/CD pipelines that take hours and hours and hours to run because the tests take six hours to execute and to build. That can’t work in any sort of a reasonable DevOps system.

    Wong: I think there’s two parts to a solution, and I’m really, frankly, just parroting what Brian said. I think there’s a collaboration piece between the people. You know, to an earlier part of our conversation, you’ve gotta have the folks with those specialized skill sets. Ideally, you’re able to get access to those folks on demand, which historically has not been something that’s been possible, but that’s beginning to happen.

    The second thing is—so, you’ve got your people collaborating, you’ve also gotta have your technology integrated. So, for whatever security tool or processors, how do those outputs actually get to the engineers? What we need is bidirectional integrations for things like Dev ticketing systems. We need things like APIs for test results. I think collaboration at the people place and integration at the technology place—both of which we could call processes—then it’s just, it’s really a repeat of what [Audio skips].

    Shimel: Mm-hmm. I mean, but wasn’t this one of the promises of DevOps? I mean, I got into DevOps because I thought it was a better way to do security, truth be told. But one of the promises of DevOps is, we’re gonna automate more, so we could go faster with more quality—two out of three ain’t bad, right? And, you know, did we lose that somewhere? Is that no longer a goal, or have we taken that as far as it goes?

    Kalsey: We keep trying to solve it with technology instead of by looking at the culture and changing how people think. And you can’t think the same way and apply new technology to it and expect the technology just to magically solve all your problems.

     Shimel: Brian, you agree, or Caroline?

    Dawson: I absolutely 100 percent agree, and I’ll agree and then I’ll just add a point. And I know I’m being a bit nebulous about it, but there’s a human element to this. At the end of the day, it’s humans developing software for humans, right? Whether it’s the code on your electronic control module in your car or Waze on your phone or the OS, at the end of the day, the final endpoint is a human. And you know what? At least now, and I know we’re improving, there’s no way that we can ensure—and I don’t mean to accept security here, but to test the quality of the human experience, right?

    And think about this—at the end of the day, knowing that the language on a screen is self-descriptive and informs me what to do. Testing the fact that, or ensuring that when I submit my credit card number it’s not vulnerable, right? Whether we’re talking API testing, whether we’re talking security testing, all of this at the end of the day boils up to customer or user experience testing.

    And I guess what I’m gonna posit is that we can never entirely automate that. And where we’ve hurt ourselves and possibly made a false promise is, we’re gonna create a bunch of full stack engineers, we’re gonna get rid of, away from operations functions, testing functions, and we’re gonna use TDD and automate everything from the start, Gherkin, yadda yadda yadda. And I believe—look, that has enabled us to go faster, but our impedance mismatch, the limit we’ve hit right now is where we’ve failed to better connect people and enable them to collaborate and share knowledge. So, not just security, not just back end, not just data, but the entire customer experience is of quality. Dev can’t do that alone.

    Shimel: No, sir.

    Dawson: I’m sorry, Alan, did I get, did I—

    Alan Shimel: No, no, that’s all—I’m saying no, sir, you’re 100 percent correct, but I know Caroline wanted to say something.

    Wong: So, I do have a comment about automation. I think that DevOps has figured out how to do automation when it comes to defect discovery. I think we’re sort of good at that. I think what we’re still trying to figure out is automation when it comes to workflow and process so that, actually, again, to Brian’s point, the collaboration can happen automatically, rather than due to a human who’s involved and making it happen. How do we get the right information to the right people in the easiest, most efficient way at the right time, and how do we make it super easy for them to do their jobs?

    I do think that there remains—I do think that the next evolution of automation and DevOps is gonna go from defect discovery automation, which I think we’ve gotten quite good at, to workflow and process automation.

    Shimel: Fair enough. Adam, thoughts?

    Kalsey: I like it. I mean, identifying what you’re automating and what’s working in that automation is important, but also identifying what’s not being automated or what in your automation is broken. And not broken from the standpoint of, it doesn’t execute or it doesn’t run to completion, but broken in, it’s not solving the problem you wanted it to solve.

    Shimel: Fair enough. Guys, we have time for maybe one more question. I’m gonna pose it and throw it out there to you. Is it possible we’ve shifted too far left or that you can shift too far left and we need a little rubber band, here, going the other way, a little pendulum swing? Or you think we just scratched the surface? Caroline, you look like you wanna say something.

    Wong: I think that’s a great question. I do think it’s possible to shift too far left, and the reason is because, while we, I think, are all familiar with the statistics and the studies that say if you fix something earlier in the cycle, it’s cheaper—there’s stuff that exists to the right that doesn’t exist yet on the left. Software development is, you know, to sort of a little analogy that we talked about earlier, I think it’s a dance. I think it’s a dance between people. I think software comes about as a result of the behaviors and the actions and the decisions that various folks take together. And security and quality are emergent properties. They’re not Band-Aids, they’re not vitamins, they’re not painkillers. They have been, as a result of this dance. So, there are some things for which, if you try to test it too early, it doesn’t actually exist in its final form yet. If I take my 6-year-old to the doctor and then I don’t when she’s 21, you know, might we have caught something super important as a 6-year-old that we were able to step in and address right away that would’ve been great? Yes, but as a 21-year-old—guess what? She’s gonna have different problems that were not apparent at age six.

    Shimel: Spoken like a true mom and parent, right? Parents, we worry about that. Anyone else? Thoughts on that? Can we go too far left, or have we gone too far left?

    Dawson: I, Alan, think it just depends. I think we’ve interpreted left different ways. I think it’s a good, catchy term. It’s an important and powerful catchy term, but as I’ve heard others discuss, we sort of overemphasized this idea of left. One, to Caroline’s point, right, we just do everything here and we’re done, good to go, right? But then this idea that we can put everything in Dev? Well, no, I think—I think, for what it’s meant to communicate, shifting left is an apt term.

    I think in terms of ensuring that we’re doing everything we can as far left or as early in the process to mitigate issues later—great. Right? But we don’t just shift security left. Right? Like Caroline said, inbound, out of bound. Actually, we need to shift security everywhere. We don’t just shift quality left, we need to shift quality everywhere.

    So, I think there are—we need to be ensuring that it’s almost zero trust software delivery, right? At every stage of the process, from concept where you should be sitting down with QA and the security team to capture the requirements to test, to deploy, to run, we need to be thinking about it, also.

    Alan, I’d say shift left is good—shift everywhere is really what we meant to be doing, right?

    Shimel: It’s better. Great.

    Kalsey: Any time we engage in sloganeering, there’s gonna be people that pick the wrong things from the slogan. I mean, “Hey, we have DevOps. That means we don’t need Ops anymore!” And that’s not what it meant, and there’s people that have taken the wrong cues from shift left, and instead of shifting the processes left, they shifted the tasks onto the people who were left already, and that’s not gonna work.

    Shimel: Absolutely. Hey, Mitchell, you wanna tie a bow on this?

    Ashley: I’ll put a bow on this. You know, there are admittedly many, many things that I don’t know, but one of the things that I do know is that, if a developer says it’s tested, it’s ready and a QA person says it’s tested and it’s ready, five out of six dentists will choose the QA person. I will, too.

    So, there’s something there. There’s a reason why we don’t have DevQA or DevTest. We don’t talk about it that way, we talk about DevOps or DevSecOps. QA is extremely valid and necessary, and we can’t live without it, and I think we need to invest in helping, elevating how we test and how we think about our software applications to test in. So, those dentists will be happy.

    Shimel: And Crest, the Crest test for QA. Thanks very much, Mitchell. Caroline, Brian, Adam—thanks for joining us on this episode of DevOps Unbound. Great conversation. For our audience out there, do check it out. We have another episode coming up in—well, not next week, the week after, and then I think we have a roundtable the two weeks after that.

    But for now, though, we’re gonna wrap. Many thanks again to Tricentis for sponsoring DevOps Unbound. Thank you for watching, thanks to our panel. This is Alan Shimel, we’re out.

    [End of Audio]

  • Harness Employs ML to Streamline DevOps Workflows

    Harness Employs ML to Streamline DevOps Workflows

    At the online {Unscripted} 2021 conference, Harness today announced an update to its namesake DevOps platform that includes support for feature flags that expose new capabilities to a select number of users for testing purposes along with the ability to prioritize the running of those tests based on the likelihood an application is likely to fail them.

    In addition, Harness is adding a unified pipeline capability that enables DevOps teams to streamline the management of multiple tools that make up a workflow and has updated its cloud cost management module to enable DevOps teams to automatically stop idle workloads. That capability is based on technology the company gained with its recent acquisition of Lightwing.

    Steve Burton, chief marketing officer for Harness, said Harness is now employing machine learning algorithms to identify which tests an application is most likely to fail. That test intelligence module enables DevOps teams to run those tests first versus testing an application only to discover close to the end of that effort that an application will need to be updated and then retested. The amount of time spent on testing is then sharply reduced, noted Burton.

    Overall, Harness is reducing the amount of time and effort spent maintaining DevOps pipeline and workflows, said Burton.

    The cost of maintaining a DevOps platform will naturally vary as the overall environment becomes more complex. It may not be clear how much time a DevOps team is spending on testing, but the faster that process becomes the more likely an application will be thoroughly tested before it is deployed in a production environment. Fixing applications after they are deployed in a production environment is, of course, considerably more expensive.

    At the same time, as the number of applications being built simultaneously increases, the amount of cloud resources being consumed also increases. Developers, however, don’t typically pay close attention to what workloads are consuming which cloud resources, which can result in an unpleasant surprise when the monthly bill from the cloud service provider comes due.

    In general, Burton says machine learning algorithms and other forms of artificial intelligence (AI) will soon become table stakes for any provider of a DevOps platform. The expectation is the platform will routinely employ AI to streamline processes and reduce total costs, noted Burton.

    Many of the manual tasks that DevOps teams regularly perform to maintain a DevOps platform will increasingly become automated as usage of machine learning algorithms becomes more pervasive. It’s not clear to what degree organizations may abandon a legacy DevOps platform to take advantage of those capabilities. However, over time it will become apparent that some organizations are spending a lot more time building and deploying applications than others as the next generation of DevOps platforms, infused with machine learning algorithms, are installed.

    In the meantime, DevOps teams might want to make a list of the manual tasks they perform today as part of an effort to determine how much time they are wasting that might otherwise be spent on tasks that are more valuable to the organization.

  • How Shifting Testing Left Can Boost DevOps Velocity

    How Shifting Testing Left Can Boost DevOps Velocity

    DevOps is all about balance: How can speed and quality be balanced to best meet the demands of customers who want exceptional digital experiences as fast as possible? DevOps looks to meet these expectations head-on by helping to deliver a faster cadence of new builds to customers while keeping quality high.

    What’s more, quality should continuously improve, build-over-build, so that fewer fixes are needed with each iteration. While inserting more testing earlier in the development life cycle may at first seem like a hindrance to the speed DevOps promises, it will soon empower teams to release higher-quality software, faster, as shift-left testing becomes a routine part of the team’s processes.

    Many organizations wait until the very end of their software development life cycle (SDLC) before executing testing. With this structure, testing can and does find unexpected bugs late in the SDLC, where issues are the most difficult and expensive to resolve. With this approach, testing becomes a bottleneck, impacting that ever-important goal of efficient and Agile software development and deployment. An unexpected outcome: it also negatively affects team morale, as developers who spent plenty of time and effort to build and ship code quickly now have to delay their deployment process in order to correct issues.

    There is a more efficient approach that organizations can put into place, however.

    Shifting Testing Left

    Testing can be implemented during and throughout the critical stages of design and development of software, instead of after.

    This is true regardless of how a team approaches development methodologies: Agile, waterfall, continuous or a combination of these methods. When testing is shifted left, it becomes the responsibility of the entire organization, not just dedicated QA testers. That means SDETs, developers and others should all be active in testing along with QA. When these groups are involved in testing early on in the SDLC, there are plenty of benefits, including:

    • A list of user workflows and test validation conditions can be created to streamline testing during development processes.
    • Testers can catch potential issues much earlier in the process so they can be fixed during earlier stages, instead of after code is done being developed.
    • Developers and testers can work with code they are familiar with to edit existing code and fix issues themselves.

    Building a Continuous Feedback Loop

    Another major advantage of shifting testing left in the SDLC is that it creates a continuous feedback loop. Software can be updated and improved continuously as it is being built, instead of when it is “complete” and ready to ship. This model of a feedback loop saves time and ultimately ensures no show-stopping issues make their way into the hands of customers.

    Test Automation in the SDLC

    One more key element to shifting testing left is test automation. Unfortunately, automation is often not leveraged completely or to its fullest potential. A recent report found that 74% of QA testers see test automation taking place outside the regular test cycle. Moreover, only 45% of organizations use test automation, and only 50% use it for continuous integration/continuous delivery (CI/CD).

    For test automation to succeed, it must involve developers creating the automation code and testers, then editing, maintaining and executing that code. When test development, including automated testing, is moved into the SDLC properly, it leads to automated and integrated unit testing, which helps achieve that goal of increased efficiency for DevOps teams.

    Automation is obviously crucial to the success of DevOps, but manual processes must also be considered. Some types of testing – like exploratory, payments, usability and accessibility – cannot be automated (at least not fully), but they should also be accounted for as part of a shift-left strategy.

    The origin of DevOps methodology and the focus of developers involves the goal of building and deploying code as quickly as possible. But testing software to make sure it works as intended is an important step that should be included throughout the SDLC. When testing only happens at the end of the process, it can be an inhibitor to the overall goal of increased development velocity. But when testing is shifted left and involved earlier on in the SDLC, it can do the opposite – by creating a continuous feedback loop and implementing automation to drive improved DevOps velocity.

    Ultimately, shifting testing left in the SDLC leads to features getting to users and customers faster, an overall improvement of software quality and a reduction of QA testing time and bug fixes.

  • Low-Code Automated Software Testing Drives DevOps

    Low-Code Automated Software Testing Drives DevOps

    The DevOps world has quietly turned its focus to solving an essential challenge: enabling software teams to innovate quickly while meeting high customer expectations for quality. In other words, to build useful things faster with fewer mistakes. This tension has been growing for years, but 2020 added the challenges of remote work and accelerated digital transformation to the continued disruption brought about by cloud, SaaS and DevOps. In 2021, we need a road map for taking test automation to the next level with quality engineering, a discipline that incorporates best practices and data-driven testing into the end-to-end customer experience and drives organizational growth.

    Rethinking the Quality Process for the DevOps Era

    Quality engineering (QE) has traditionally lacked the tools needed to keep pace with digital transformation. Many QE teams struggle with complex, brittle automation frameworks and slow manual testing which create serious bottlenecks to DevOps adoption. While development accelerates, QE is left with disparate or outdated workflows and toolsets.

    Rising to the challenges and opportunities presented by DevOps and digital transformation will require innovation in our approach to quality. More than ever, we need to shift our focus away from quality assurance as a task that you perform late in the software development life cycle (SDLC) in favor of quality engineering practices throughout the entire life cycle. It starts with broad, effective test coverage – beginning when your developer is working on a local branch and continuing until a high-quality customer experience is confirmed in production across the entire surface area of the application – web, API, mobile, email, PDF or otherwise. It also goes deep; you evaluate all aspects of quality, ensuring that you satisfy requirements related to functional correctness, accessibility, performance, user experience and more.

    Low-Code Test Automation Unlocks a Culture of Quality

    With low-code, genuinely effective testing is at the fingertips of every role, from manual testers to developers, automation engineers, product owners and customer support. Requiring all of these roles to develop specialized programming skills is impractical, so low-code tools serve to democratize quality engineering by enabling everyone to participate in authoring tests, updating tests, and gaining insight from test results.

    Low-code testing brings several advantages to quality engineering, particularly in the transition from manual testing to automated testing. While traditionally this process would create major disruptions, low-code platforms are smoothing the way. In their IT Automation predictions for 2021, Gartner expects that by 2024, improvements in automation capabilities will refocus 30% of IT operations efforts from support to continuous engineering. For manual testing teams, that shift would likely trigger talent changes and a long process of pilot projects, retraining, and gradual rollouts. Low-code is key to managing this shift, upskilling workers and ensuring quality throughout.

    Modern Automation Solutions Embrace Rapid Change

    A key challenge with traditional test automation tools is that they were designed for applications that didn’t change frequently and were intended only to validate code. They relied on static selectors to identify page elements. If the selector (say, an ID) changed or objects (the XPath) moved on your webpage, then your tests failed. As more enterprises accelerated development via DevOps, Agile and CI/CD, the burden of test maintenance resulted in the same bottlenecks as manual testing.

    2020 was a breakout year for low-code test automation. New, easy-to-use solutions surpassed the capabilities of legacy frameworks that have dominated the sector. These solutions incorporate machine learning, data analytics and other advanced technologies to greatly reduce the burden of test maintenance and overcome the test rigidity challenges of legacy frameworks.

    With AI and machine learning, test automation solutions are trained by a tester simply navigating their product. As the tester moves through common customer journeys, the solution builds an understanding of each element and its context so that if an element moves or a property changes, tests containing that element are updated without human intervention. This drastically reduces the amount of maintenance required to ensure your tests are current. When tests do need to be updated, low-code models make it easy for anyone to jump in and make the updates. Looking ahead, auto-healing and low-code editing are crucial to supporting quality engineering in high-velocity teams.

    Testing the Entire Customer Experience

    Traditional pass/fail validation stops short of offering real insight into the user journey, a critical step effective quality engineering. The next generation of test automation platforms expands focus from validating code to validating the entire user experience. This starts with broadening testing to cover non-functional requirements, such as application performance, accessibility, and user experience. For example, identifying trends in load time for key pages and monitoring visual changes are done autonomously and essential to managing the customer experience. From there, you can prioritize quality efforts toward the components of your application that are most heavily used or highest risk. This enables you to make data-driven decisions about test coverage — focusing your testing efforts on high-priority pages like the checkout page for an e-commerce site, or high-risk pages such as those with multiple interactive elements that are linked from other pages, such as signup forms.

    But even a fully data-driven testing strategy that only focuses on the app or website UI is unlikely to cover the full end-to-end user journey, resulting in test coverage gaps and potential pain points for customers. For example, a common user journey may include opening an email, clicking on a link, checking a web site, and downloading a PDF. Whereas the limitations of legacy tools relegates testing to the web portion of that journey, modern solutions allow you to validate the entire user or customer experience (CX): receiving and opening an email, clicking on the link, logging into the web site, downloading the PDF and verifying its content. It’s even possible to confirm values in the backend via API – all without writing a line of code. Developing the tools to seamlessly monitor, repair, and manage the expanded CX is central to our innovation and to our customers’ success.

    Quality Engineering as User-Centric as You

    Quality is no longer an afterthought or just a box to check before deployment; it’s an essential part of the user experience, and therefore an essential part of maintaining a competitive advantage in every industry that uses software to serve customers. Test automation is critical for high-velocity teams racing to innovate faster, shift to DevOps, and build quality products while working remotely. Thankfully, a growing number of quality assurance teams are getting a seat at the table as DevOps is implemented, elevating the profile of quality as a strategy and growing demand for test automation solutions that can keep pace with the demands of DevOps.

    In response to this demand, a new generation of intelligent test automation solutions is enabling teams to rise to the challenge by embracing low-code and machine learning to maintain quality. To build a culture of quality engineering in this environment, your team needs test automation solutions that are as user-centric as you are: solutions that provide coverage across the entire user journey. Solutions that not only verify functional correctness, but provide rich insight into all aspects of quality and enable continuous improvement. Perhaps most importantly, you need solutions that enable everyone on the team to participate in quality. This ensures that the entire SDLC actively participates in building user-friendly products that continue to set the standard for quality at the speed of DevOps.

  • Pilot Testing DevOps in a Pandemic

    Pilot Testing DevOps in a Pandemic

    Savvy organizations are using the pandemic to drive innovation by pilot testing DevOps. As health experts brace for an exponential rise in COVID-19 cases across the nation and vaccine distribution ramps up, many states and cities are still implementing significant measures to reduce the virus’s spread. As shutdowns impact cities coast to coast, organizations are updating their business continuity processes and disaster recovery plans for COVID’s next wave.

    Software developers and testers are typically perceived as being able to adapt quickly to changes and the fast pace of IT. Since the beginning of the pandemic, SaaS adoption has become a go-to platform for businesses of all sizes looking to ensure business continuity and enable innovation during these uncertain times.

    To meet this demand, developers have worked at ever-increasing speed to configure, test and deploy advanced features and services to their clientele. This quickened pace, for some, has led to the abandonment of proper security guardrails.

    There are a few key ways to mitigate the pressure and risks involved in this accelerated environment, balancing continual innovation with ensuring developers and testers are applying robust security strategies.

    Fail Fast, Succeed Early

    At its core, DevOps is the process of using development technology to solve operational IT problems. Having a well-defined strategy helps; however, it’s nothing without proper implementation and calculated risks. Both of these factors help your organization achieve its goals in a shorter amount of time.

    Here’s an example. One of my colleagues worked on a software development project where, instead of testing one process at a time, they tested 30 different scenarios simultaneously to see which would be the better strategic fit for their organization. They could have planned and tested, one scenario at a time, for six months; however, in three weeks, they could determine their next step in a much shorter amount of time. Methods like this begin with a strong innovation culture where everyone aligns on the same goals.

    Additionally, access to funding also becomes essential, along with the ability to hyperscale. Cost does become a factor; however, this simultaneous testing and fail-fast methodology saves teams an extraordinary amount of time. The goal of failing fast in DevOps is not to increase the chances of failure, but to encourage experimentation, making it a surefire way to bring products to market quicker.

    Put the Cultural Shift into High Gear

    Whether or not a company finds itself in crisis due to the pandemic, leaders need to remember that tools are not the magic bullet of DevOps; culture is. DevOps is not only a philosophy, it’s a mindset that promotes collaboration, culture alignment and lends itself to innovation. Innovation is often a term that scares companies or teams, who may feel it is out of their reach. However, consider this: the very act of innovation involves trying. If you’re not putting yourself in the race, how can you win?

    From ideation to pilot testing, organizations need to understand that they can try something without being “all in.” Innovation is everywhere and within every process. What keeps the culture of innovation alive and well is building an ethos of collaboration; that leads to a team that not only thinks but believes that innovation can happen. Culture becomes everything when your organization starts to look for new and innovative ways to push your business and your clients further, successfully.

    There is an important caveat, though. Often, leaders introduce policies and procedures to their team without providing a long-term vision of the outcome(s). This often leads to long-term adverse effects, simply because the underlying organizational culture is not healthy. Even during a global pandemic, where budgets may be tighter, teams are leaner but innovation is still the goal, there must be vision, transparency and effective communication to allow for proper implementation and execution. All of these components start with culture.

    Scale DevOps Through Automation

    Working on a test automation strategy is the best way to set your organization up for success. Along with culture and collaboration, automation is a key pillar of DevOps. Research shows 60% of occupations have at least 30% of constituent work activities that will be automated by 2030, globally. Historically, teams have been skeptical about automating their work, and push forward with an “I can do it myself” way of thinking and working. COVID-19 created a massive uptick in demand for IT services, and teams’ pressure to perform has also increased.

    For DevOps teams to successfully “scale up” right now, the implementation of continuous automation into the testing process is critical. As with any type of new technology implementation, there are warning signs to consider.

    As wonderful as automation can be, the process often creates a skills gap; however, this does not mean leaders should shy away from the automation process. However, leaders must be aware of this and come up with a strategy to ensure training opportunities are available to close these gaps.

    The pandemic has reinforced how swiftly change happens, and that companies must be flexible and respond quickly just the same. While IT systems are built to extend and adapt, accelerating delivery cannot happen at the expense of business continuity. Pilot testing amid a global pandemic encourages a culture of transparency, collaboration and experimentation that will empower teams to track inefficiencies at an increased rate.

  • Enterprise Software Development Will Break the Speed Limit in 2021

    Enterprise Software Development Will Break the Speed Limit in 2021

    For some, 2020 was the year for software development speed … but not for everyone. Startups and SMEs climbed on board the CI/CD, shift-left and agile development train en masse – and reaped the benefits of faster iterations and tighter release schedules.

    But the “big kids” got left behind. Core enterprise applications – the legacy systems with 1+ million lines of code that do the heavy lifting in many corporations – have not yet made the switch to the world of rapid release cycles. And this is natural; enterprises are rightfully more cautious by nature, and because the technology driving this evolution was not mature by enterprise standards.

    But in 2021, that’s going to change.

    Here are four speedy predictions for pushing enterprises over the speed limit in 2021, and some of the companies making it happen.

    Four Enterprise Software Development Speed Predictions

    #1: AI-based test generation will make enterprise development faster and better

    Testing automation is crucial for any software development organization trying to transition to CI/CD. But enterprise stakes are arguably high, and enterprise code is by nature inflated, often carrying significant technical debt. Preparing suitable test coverage manually is resource-heavy, to say the least. But a new generation of AI-powered testing automation tools solves this problem for you. Solutions from companies like Ponicode and Diffblue allow enterprises with large code base apps to embrace CI/CD by bridging the technical debt gap in testing while still meeting release cycles.

    #2: Test avoidance tech will make a dent in unnecessary testing

    Running tests on an entire build for one small code change is overkill – no one’s arguing that. But until recently, there wasn’t much of an alternative to ensure the ironclad quality that enterprises require. In 2021, solutions from companies like Sealights and Launchable (which counts among its investors Jenkins creator Kohsuke Kawaguchi) will help enterprise software development teams deliver quality at speed. The key is the effective use of machine learning to reduce test cycle times and run only relevant tests, not the entire test suite, to ensure faster iteration cycles.

    #3: High Performance Computing will move toward distributed computing

    Compute power will remain a bottleneck to development in 2021 – but the processing arms race will likely change. As enterprises get used to virtualizing their infrastructure, they will think twice about investing heavily in dedicated multicore build/render machines powered by next-gen processors. Instead, they will look to virtualized processing power like distributed computing, cloud bursting, and spot instances – which provide the massive scalability enterprises require at build time, while still maintaining a more reasonable price-performance ratio.

    #4: Managed CI/CD in the cloud will automate enterprise release pipelines

    Managed CI/CD, like Amazon’s CodePipeline and Azure Pipelines, help enterprises save time setting up and maintaining CI/CD infrastructure, scaling up during peak times and maintaining security. These services, while not for everyone, will continue to gain popularity in 2021. A key reason for increased adoption is that they allow companies to scale much faster: without hardware or software, development teams can push a button and suddenly gain 20 more build servers. Moreover, managed CI/CD enables faster onboarding of new teams, simpler creation of repositories and more streamlined work with multiple geolocations. Finally, pay-as-you-go models, no upfront fees and no commitments are increasingly attractive to enterprises in tough times.

    Put the Pedal to the Metal

    Newly released AI-powered products and mature hosted services from big-name cloud providers are changing the face of enterprise-level development. In 2021, we’re going to see teams supporting backbone enterprise applications increasingly moving to CI/CD, shift left and agile development methodologies. Because everyone – even enterprises – deserves to break the speed limit.

  • How to Overcome Web Test Automation Challenges

    How to Overcome Web Test Automation Challenges

    Web test automation is transforming software development for the better. Automation empowers test teams to easily prepare test data, execute complex test cases across multiple platforms rapidly and accurately, and catch defects early on in the development cycle. Modern-day test automation solutions benefit organizations in multiple ways;  when used correctly, such solutions can boost a test team’s productivity, enhance software quality and accelerate time to market.

    Automation Is Transforming Web Testing

    Today, mobile and web-based applications are important revenue streams for organizations. Various business-critical activities revolve around web apps, so flawless functionality and superior user experience are critical. However, testing often takes a backseat to speed and quality suffers. The limited timeframes in agile development coupled with cost factors make it challenging for organizations to sufficiently identify all the underlying issues and ensure optimum release quality.

    Such concerns can be adequately addressed by implementing web test automation. Automating web testing ensures better test coverage, improves detection of all major underlying discrepancies and allows faster test execution. Test automation is critical to achieving high-quality web applications; however, testers face many challenges performing automated web testing.

    Challenges of Web Test Automation

    Web test automation enables testers to test the functionality of web applications thoroughly and quickly, ensuring there are no deviations in the results from the expected user experience and interactions. However, even this beneficial process involves certain challenges for testers, such as:

    • Handling frequent UI changes: Many components of a web application’s UI are prone to frequent changes to logic, or due to new integrations or the addition of new features or functionalities. As a result, maintenance of UI testing scripts becomes a challenge for testing teams.
    • Multiple error handling: Error handling can be a cumbersome process with web test automation, especially when test teams must tackle large projects with complex scenarios and large data volume.
    • Image comparisons: Accurate detection of pixel variations in shape, size, color and formats are an important part of the web testing process. Test automation can help make handling image comparisons more efficient.
    • Long test script execution time: Typically, web test automation is a time-intensive affair. It requires significant test script execution time, depending upon the number of test cases involved. However, in the fast-paced technology landscape and agile development environment, testers get limited time to ascertain the quality of releases.
    • Complexities in testing various UI elements: As technology evolves, so do the possibilities for enhancing web apps with new capabilities. Modern web apps have complex features such as embedded frames, flowcharts, diagrams, maps and more. As a result, functional and non-functional web testing becomes more challenging.
    • ROI of web testing automation: As mentioned earlier, any application’s UI undergoes constant changes, and the test scripts have to be adjusted accordingly. This primarily increases the time spent on the creation and maintenance of test automation scripts. As a result, it becomes difficult to calculate ROI while continuously performing web tests.

    How to Overcome Challenges

    Mentioned below are some measures that can be implemented to make web test automation efficient, impactful and cost effective.

    • Choose a codeless solution: To maximize benefits, it is essential to choose the right web test automation solution. It is necessary to consider codeless (scriptless) test automation solutions, as these tools ease the process of script maintenance, allowing the testers to save both time and money. Codeless solutions simplify web test automation and improve ROI.
    • Choose common UI elements to uncomplicate UI testing: Organizations should strive to uncomplicate web testing, as much as possible. Developers can help by using common UI elements, so testers more easily understand them and can strategize efficiently and test vigorously.
    • Leverage an object repository: The creation of a shared repository enables organizations to reduce the costs and efforts involved with test maintenance. To ensure the success of UI test cases, it is preferable to reduce the UI test cases during the early testing stages and then slowly increase the count as testing moves forward.
    • Adopt organizational-level best practices for automated testing: For web test automation to be truly successful and deliver sustainable results, it must become a fundamental part of the testing process. Organizations that successfully integrate web test automation have well-established best practices and governance mechanisms. They follow organizational-level best code review processes to ensure the best test automation outcomes.

    Web test automation is a critical testing process that allows testers to verify and validate the functionality of a web app’s UI and backend features. Successful adoption of web test automation enables QA teams to significantly reduce regression testing efforts and time. However, it’s critical to mitigate certain challenges associated with web test automation to truly benefit from such an implementation. Effectively handling these challenges helps teams to deliver a robust and user-friendly web app that delivers superior UI and UX.

  • Keysight Technologies Jumps into App Testing by Acquiring Eggplant

    Keysight Technologies Jumps into App Testing by Acquiring Eggplant

    Keysight Technologies, a provider of tools to test and measure embedded systems and mobile computing devices, has acquired Eggplant, a provider of a platform that automates application testing, for $330 million.

    Dr. John Bates, CEO of Eggplant, said the acquisition enables Keysight to expand the scope of its testing portfolio beyond hardware. Keysight traces its lineage back to Agilent Technologies, which it spun out of in 2014. Agilent was spun out of Hewlett-Packard in 1999. Keysight retains the test and measurement portfolio that began with HP back in 1939.

    Bates said Eggplant, which has annual revenues of $38 million, will operate for now as an independent arm of Keysight. Over time, however, there should be increased opportunities to converge testing of applications software running on embedded systems that increasingly are being deployed at the network edge, he noted.

    Eggplant’s Digital Automation Intelligence (DAI) platform is designed to automate the entire testing lifecycle process as part of a larger best DevOps practice. The rate at which application tests can be built and run winds up being a bottleneck as applications are being built. The DAI platform is designed to automate the creation of UI tests and then launch them against application code to eliminate as much testing friction as possible. IT teams can also track user behavior to create user journeys based on their interactions with applications and store tests created using modeling tools provided by Eggplant in its cloud service.

    Eggplant has been investing in deep learning algorithms that, for example, automatically identify and classify user interface (UI) elements such as buttons and text fields across any platform and then generate all the automation assets needed to instantly run an automated test.

    With the rise of DevOps, there’s now more application code to be tested than ever. As such, testing backlogs within many IT organizations are growing. The challenge is finding a way to thoroughly test applications without resulting in missed deployment deadlines. When those deadlines are in danger of being blown, testing typically is the first thing to be scaled back. Of course, when an application has to be rolled back, the primary reason often is because a key component was not tested properly.

    Bates noted testing becomes even more complicated as organizations embrace microservices. The level of dependency between microservices spanning multiple platforms makes testing even more challenging, he said, adding IT organizations will need to rely on automated testing platforms to test applications at levels of unprecedented scale.

    It’s not clear to what degree the COVID-19 pandemic may have accelerated demand for applications based on emerging 5G wireless services. Some organizations are forging ahead while others may be delaying initiatives as they cope with the economic impact of the pandemic. One way or another, a raft of applications running on a wide variety of edge computing platforms will need to be tested. Chances are good a much larger percentage of those applications soon will be tested by machines rather than humans that can’t scale to meet the challenge.