devopsy.kinsta.cloud

Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

  • DevOps Chat: Continuous Testing with Sandeep Johri, CEO, Tricentis

    DevOps Chat: Continuous Testing with Sandeep Johri, CEO, Tricentis

    In this DevOps Chat we take a dive into the continuous testing market, which is one of the most dynamic in the DevOps space. Our guest is Sandeep Johri, CEO of Tricentis, a leader in the space.

    Sandeep gives us his own personal background, as well as the history of Tricentis. The company is not an overnight sensation in the continuous testing space. In fact, it has been perfecting its offerings for many years. Tricentis literally helped usher in the continuous testing age, making continuous testing part of the CI/CD pipeline.

    This chat is a great way to learn about what some of the important points for continuous testing are. Have a listen. As usual, the streaming audio is immediately below, followed by the transcript of our conversation to follow along. Enjoy.

    Audio

    Transcript

    [su_pullquote align=”right”][su_custom_gallery source=”media: 120253″ link=”post” target=”blank” width=”230″ height=”230″ title=”never”]Sandeep Johri
    As the CEO of Tricentis, Sandeep brings over 25 years of enterprise software experience. He manages all business-related functions including business strategy, products, services, sales, marketing, and operations to achieve business profit and sales objectives. Sandeep most recently served as the COO of Appcelerator, the leading open source mobile application development platform. He was Vice President of Strategy and Industry Solutions at HP, where he also served as Vice President of Corporate Development. There, he was instrumental in developing and implementing the growth strategy that resulted in the software business growing 2X market over the last four years. Sandeep was instrumental in the formation of two startups, Determina (VMware) and Bluelane (VMware). He was also the CEO and founder of Oblix (Oracle), and co-founded eBoodle (EW Scripps), one of the first ecommerce comparison shopping services. Sandeep has an MBA from Stanford University, a Master’s degree in Industrial Engineering from Wayne State University, Detroit, and a Bachelor’s degree in Mechanical Engineering from Pune University, India. Sandeep is involved as an advisor and investor in various startups in Silicon Valley and India.[/su_pullquote]    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Got a really great chat lined up for you all today. I’d like to introduce you to Sandeep Johri. He is CEO of Tricentis. Sandeep, welcome.

    Sandeep Johri: Thank you, Alan.

    Shimel: First of all, Sandeep, I just wanted to make sure I’ve got your – we pronounced your name correctly.

    Johri: Yes, you did, actually. You got it absolutely right.

    Shimel: Good. All right. It’s always a good start. So, Sandeep, let’s start foundational here. Tricentis is a company that, really, had a very, very strong presence over in EMEA, I think, and has most recently really begun to exert themselves in the North American market. But some of our listeners may not be familiar with Tricentis at all. So why don’t we start with that, if you can give us a sort of a – not an elevator pitch, but a little background on Tricentis?

    Johri: Sure. So Tricentis was founded about eight, nine years ago. It was founded by Wolfgang Platz, who was actually a tester and working on complex enterprise applications for testing and was frustrated with all the tools out there. So, like any good entrepreneur, he decided to write his own tool. And the tool initially was focused primarily on automation. And it sold – they were – Wolfgang was based out of Vienna, Austria, and they sold it locally into the German-speaking market, so Austria, Germany, and Switzerland.

    Shimel: Mm-hmm.

    Johri: And they had a nice, little business going and they had a great product. I ran into them about 4 years ago, in 2013. I found them and felt that this could be a really interesting offering because what the market needed was automation, especially as people were going to Agile. Given my background with Mercury at HP, I knew that there was not much automation, but, if you go Agile, you have to have automation. It becomes a critical component – automation on the testing side becomes critical – and that’s why I joined the company. You’re right; they were initially focused primarily on Europe. And, remember, this is pre-coining of the term “DevOps,” but the need for automation was always there, especially as you kind of shift towards Agile on the dev side.

    So now, you know, cycle ahead four years and we are now a global company. We did as much business in US as in Europe, so we are now really very well balanced. And we also have now created a presence, over the last few years, in Asia-Pac, so Australia, New Zealand, and Singapore, so we’re truly kinda global, with a typical balance of about equal between US and EMEA, and then another additional APAC component to it. And, obviously, once DevOps became the kinda mantra in the industry, we had a really nice fit and people started talking about continuous testing. Well, with automation, what you really get is continuous testing, so that’s why we are all about continuous testing. But the core engine is an automation – is a test-automation engine, so for the enterprise.

    Shimel: Absolutely. So I wanna get into the whole automation and testing market, but you mentioned a few things that I, if you don’t mind, I’d like to dive into. I don’t mean to embarrass you, Sandeep, but, when you get to my age – and you’re a contemporary – you know, we like to reminisce. So you mentioned Mercury and your experience there. I mean, for those who aren’t looking at your CV or the management team details up on the Tricentis site, why don’t you give a little background on yourself to our audience so they get a flavor for this?

    Johri: Sure. Sure, Alan. I’ll try to keep it short. I’ve been in Silicon Valley for almost 30 years and I’ve been entrepreneur. I started a few companies. One was Oblix, which was a leader in the identity space. And then a couple of other companies, Blue Lane and Determina, which were then acquired by VMware. I joined HP after that because they were looking to build out their IT-management portfolio, and, over 3 or 4 years, from 2005 to 2008, we acquired about 13, 14 companies, including Mercury Interactive, which was the largest, but we also acquired Opsware for data-center automation; we acquired Peregrine for asset and service management. And that whole portfolio became HP ALM, so it was – that business went from about $600 million to about $3.5 billion.

    Shimel: Wow.

    Johri: And then everyone knows HP was having some challenges and were increasingly focused less on the software business, which they eventually sold last year. But I felt the need – I felt the opportunity was there for a next-generation tool or next-generation testing platform that better suits the Agile automation environment, which we now call more “DevOps” because it takes the whole cycles and kinda creates a continuous-delivery chain, if you may.

    Shimel: Absolutely. So, Sandeep, that’s sort of the quintessential Silicon Valley entrepreneur story, where you’ve done everything from start-ups that had some very successful exits. As I mentioned offline, I remember Oblix and I remember Blue Lane almost like it was yesterday. And then, of course, you had this big company, HP kind of experience on the resume, and I would imagine, knowing what I know about the Insight Venture team, who I believe is an investor in Tricentis, I would imagine you would have been a very, very attractive candidate, when they were looking for a – the team was looking for a CEO to take Tricentis to the next level, so right person at the right time there. Fantastic story.

    Johri: Yeah, I joined before Insight came in. Yeah, I joined four years ago – Insight came in last year – because I thought that they were really promising, and we spent the first couple of years kind of building out the product to be more – you know, filling out the product. We added service virtualization. We added test data in it. You know, we’ve added a whole bunch of capabilities and we were beginning to get recognized by Gartner and the like as a leader and we also started having some really large enterprise customers in the US. That’s when Insight also had a similar thesis, that, “Hey, HP seems to be walking away from this market, but there’s a need – there’s an opportunity there,” and they found us. And that’s why they invested $165 million late last year, so.

    Shimel: Wow. That’s quite an investment. And so sort of I had the horse before the cart, but congratulations. Right? This is not an easy thing with that kind of investment, but also comes when there’s that kind of opportunity. Let’s turn now, if we can, Sandeep, and talk a little bit about the testing market. You know, testing – before DevOps and Agile, the whole QA testing thing was really, really super geeky and I don’t know how interesting to people. Back in my day, at StillSecure, the hardest thing for us in testing was just building up a test lab that could handle the scale we wanted to test at. Right?

    Johri: Right. Right. Right.

    Shimel: And testing software and so forth was relatively a bunch of open-source tools that the development team did as part of the release cycle in the old Waterfall. But, you know, with Agile and automation and, look, there’s Agile with automation and then there’s automation in and of itself, throughout the software life cycle.

    Johri: Uh-huh.

    Shimel: It’s become imperative, right? And, all of a sudden, now –

    Johri: Right.

    Shimel: – yeah – testing has taken on a super role, right? I mean, it’s really become kind of the – you know, when we look at the continuous integration, continuous delivery pipeline, right, that testing, man, it sticks out. And it’s like swamping everything else, if you will, right? It’s swallowing it.

    Johri: Right.

    Shimel: And so tremendous opportunity. We’ve seen a number of companies try to enter this fray, many of them based on open source, whether they’re doing Selenium or something like that, but there’s different aspects of testing. Can you share with our audience, Sandeep, what type of testing is it that Tricentis is doing?

    Johri: Right. And so, Alan, I think you cover a lot of the history and perhaps I can comment on that because that’s really interesting. So, you know, testing, for an enterprise, has always been critical, but you’re right. It was kind of a “Oh, God, I have to do testing and it’s a lot of manual process,” so people mostly outsourced testing. And, when you were in Waterfall mode, that was okay because you did a bunch of development and then you threw it over the wall. In most cases, at an enterprise, it went offshore and people did it manually. But it has always been a huge spend. Even today, there’s a $30-billion spend on testing services.

    Now the challenge is that the tools that were out there, both open-source and commercial tools, were primarily script-based tools, so it was really difficult to automate. And, because you had outsourced, you kinda got the labor arbitrage and people continued to do it manually. And, even today, 80 percent of testing, enterprise testing, is done manually, which is kinda crazy, right? ‘Cause here we’re getting ready with autonomous cars on the one hand, but, on the other hand, we’re doing software testing which is – software’s so fundamental to every enterprise – we’re doing all the testing manually. Well, now – and that was okay in the Waterfall environment.

    As you go to Agile on the dev side, you start doing frequent builds. And, as you start doing frequent builds, if your testing cycle is still six to 12 weeks, you’ll fall right back into a Waterfall mode, even though your dev might be Agile. So testing is becoming – the need for automated testing is becoming critical because, otherwise, it becomes the bottleneck in your delivery chain. And what we do or our view of testing is that testing should be at the same cadence as development, so, if you’re doing daily builds, you should be testing on a daily basis.

    Now you mentioned developers. Developers do unit testing – they do subsystem-level testing as they build – but, for a large enterprise, which has a complex environment that has a mix of technologies, just doing subsystem-level testing doesn’t work. It is necessary but not sufficient. And, therefore, what you need to be able to do to really deliver faster or to really be getting – to get into a continuous-delivery mode, you need to be able to build continuously but also test end-to-end on a continuous basis. And that is fundamentally what we enable through automation. We can get to 90-percent automation; we leave manual testing only for usability and exploratory. And then, you know, you truly have your DevOps tool chain in a continuous mode, where you’re doing continuous builds and then –

    Shimel: Mm-hmm.

    Johri: You’re doing continuous builds on the one hand, you’re doing continuous testing on the same cadence, and then you can push it out into ops, so you’re right. It becomes absolutely critical, and the manual testing is rapidly decreasing, and people are shifting more to automation now.

    Shimel: Absolutely. And not only shifting. You know, it’s a global, massive shift to automation and especially with the testing. And, frankly, look, that’s why you get the kind of valuations we’re seeing. That’s why you’re seeing the kind of acquisitions by large companies that we’re seeing. You know, I don’t see how you can do testing, going forward, without automation, so this is a fundamental change in the way this is being done. You know, you look at some of the companies from the old way of doing it, who are maybe selling testing equipment and so forth. Right? And they’re kind of dead men walking, to a certain extent. Right? And I –

    Johri: Yeah.

    Shimel: Geez, I don’t know what to tell you with it.

    Johri: But, you know, Alan, one more point on that. I think when you think about a large enterprise – a bank, an insurance company, or the like – their business processes span technologies, meaning they have modern front ends, but they also have a lot of legacy systems. And most of the early DevOps work has been done only on the front-end systems, where you can go dev and, as long as you’ve done some unit testing, you can move to production. When you’re thinking about enterprises with large end-to-end systems that span every technology that’s been invented, you have to get into a continuous-testing mode; otherwise, you cannot do DevOps for a large enterprise. So the enterprise DevOps is a slightly different animal than, let’s say, a smaller startup in Silicon Valley doing DevOps, where they have a relatively modern tech stack and a relatively monolithic application, if you may.

    Shimel: And, frankly, I always like to say, Sandeep, those companies – first of all, it’s DevOps, right? I don’t care whether you’re big or little – you’re all like do DevOps – but they do DevOps because they just don’t have the resources to do it any other way, frankly, right? Everyone wearing multiple hats and the way they – they have to – it is just no other way. It’s in their DNA, if you will. Where, with large enterprises, it’s DevOps or die, to a certain extent, in today’s competitive marketplace. And when I say “DevOps,” I’m talking about continuous – continuous testing, continuous delivery, continuous integration. You know, that automation is wherever – I mean, you know, all of the things we’ve come now to recognize as DevOps. So –

    Johri: Mm-hmm.

    Shimel: Sandeep, I guess we see these mega-trends. We see the way the market’s heading. We’ve heard about your background and the company’s history. I mean, it’s an eight-year-old or nine-year-old startup, right?

    Johri: Mm-hmm. That’s right.

    Shimel: What does the future hold? Specifically for Tricentis.

    Johri: Well, you know, we see ourselves as – and also we have about 600 large enterprises across those regions that I talked about – US, most of EMEA, and then Australia, New Zealand, and Singapore – and we focus primarily on large enterprises, the Global 2000 or the Global 5000. Competitively, we’ll end up replacing the old HP, which is now called “Micro Focus.” Most enterprises are looking for a modern tool set, just like they use AppDynamics or New Relic to replace the old app-monitoring solutions and they’re replacing old defect-management systems with newer things like Jira and the like. So we see a lot of replacement, but the really exciting part is that the $30-billion spend that I was talking about, on testing services, that is, like I said, mostly manual, and, as you start automating that, what we are really doing is disrupting that $30-billion manual-testing market. So the global opportunity for us is incredible. There’s a multibillion-dollar opportunity, in terms of what a platform like us can be selling out there.

    Shimel: Mm-hmm.

    Johri: Mercury was a billion-dollar company – almost a billion-dollar company 10 years ago, 2007, and so we see that not only as a replacement as a replacement opportunity, but also, because we’re disrupting the services play, as you disrupt manual labor, you know, a big chunk of it goes towards the automation tools, and that’s what we see. So, in terms of the future, we see a very, very bright future. And we wanna be the marketshare leader in testing tools, so we have a very comprehensive solution already for end-to-end testing, both performance and load testing, and we will continue to enhance it so that we’re the innovative leader, as we are right now. We will maintain that and continue the leadership, so.

    Shimel: Sandeep, I have little doubt of that, as long as you’re at the helm, driving this. It sounds like, based on your track record – and I have to agree with almost everything you said about the market. That’s kind of where we are right now and it should be an interesting next couple of years. The opportunity is certainly there, right? Someone’s gonna dominate this.

    Johri: Wonderful. Yes.

    Shimel: Yep.

    Johri: Mm-hmm. Mm-hmm.

    Shimel: Sandeep, as I told you when we started, the 15 minutes goes really quickly, so we’re at 20 minutes already. So we’re gonna need to kinda wrap up on this, but, before we do, for people who may be – we’re doing a bunch of activities with Tricentis, where I believe we’ll have some webinars and some content up on devopsy.kinsta.cloud, but, beyond that, where can people maybe either see you guys at conferences or speaking? Or, obviously, they can go to tricentis.com – T-R-I-C-E-N-T-I-S.com – for information, but just if you happen to know off top of your head where people might get more info even or a little hands-on –

    Johri: Yeah, well, we have – yeah, absolutely. Well, we, like you said, Alan, we do participate in most of the major conferences, either on DevOps or testing. You know, there’s testing-focused conferences we participate in. We also participate in larger vendor conferences, like the Appalachian Summit and SAP’s SAPPHIRE. But what is most exciting for me is that, this year, we’re organizing our first US user conference, which will really be a conference focused on continuous testing. And we have scheduled that for April, yeah, here in – actually, no, no, wait a minute. It is scheduled for –

    Shimel: I don’t know.

    Johri: Oh, God.

    Shimel: You know what?

    Johri: Yeah, I should –

    Shimel: I’m looking it up myself for you. Hold on here. Tricentis Accelerate? Would that be it?

    Johri: Yeah, it’s – yeah, it’s on the 8th or 9th.

    Shimel: San Francisco, May 7th to May 8th.

    Johri: Yes. Yes. Yes. May 7th and 8th. Sorry about that.

    Shimel: It’s okay.

    Johri: But that conference is gonna be completely focused on continuous testing. We’re gonna have – obviously, show all our product capabilities and the like, but what’s more interesting and exciting is that we’ll have a whole bunch of large enterprise customers that are gonna talk about how they’re tackling continuous testing, as they are on their journey of DevOps, so it should be really exciting. I would really welcome everybody to come attend that conference in San Francisco on May 7th and 8th.

    Shimel: Yep. And also you’re doing, it looks like, one in Vienna, Austria, October 9th and 10th. And ’cause – you know what? – our audience, like DevOps itself, about 36, 37 percent are US-based and the rest of the world makes up the majority. For anyone interested, though, you can go to accelerate.tricentis.com – that’s accelerate, A-C-C-E-L-E-R-A-T-E, at tricentis – or accelerate.tricentis.com, to get information on both of those. And we’ll probably be covering them from devopsy.kinsta.cloud as well. Well, this has been a great introduction to Tricentis and a great discussion of the testing, Agile, DevOps market. Sandeep, thank you so much for joining us.

    Johri: It was wonderful, Alan. Thanks a lot for the time. And I have a special offer for you. For your listeners, we will make some tickets available to the conference and you can distribute it to your favorite listeners, so we will follow up and get you a whole bunch of tickets to the conference that you can distribute to your readership. Your readership is exactly who we target and who we – that’s in our domain. That’s exactly our domain, so we’d love to have as many of your readers – or listeners and readers –

    Shimel: As both, right.

    Johri: attend the conference as possible. Yeah. Wonderful.

    Shimel: Really kind for you. I will put it in the show notes as well as on the article on devopsy.kinsta.cloud, for people who are interested. We’ll raffle ’em off, depending how many we have here. Sandeep Johri, CEO, Tricentis, powerhouse in automated testing. Thanks for being this episode’s guest on DevOps Chat and continued success.

    Johri: Thank you.

    Shimel: Thank you. This is Alan Shimel for devopsy.kinsta.cloud, DevOps Chat. Thanks for listening, everyone, and we’ll see you soon on another DevOps Chat.

    — Alan Shimel

  • Sauce Labs Adds Analytics to Continuous Testing Platform

    Sauce Labs Adds Analytics to Continuous Testing Platform

    Sauce Labs has added additional debugging and analytics capabilities to its testing platform to make it easier to manage continuous testing across multiple application development projects.

    Lubos Parobek, vice president of product at Sauce Labs, said DevOps teams need to prioritize which bugs to address first across multiple development projects occurring simultaneously. In fact, according to the latest annual survey of 1,091 technology professionals conducted by Dimensional Research on behalf of Sauce Labs, DevOps teams on average are now taking longer to address bugs. These days there are many more issues that require debugging across multiple development projects, which is overwhelming DevOps teams, who must then prioritize some debugging requirements over others.

    Specifically, the survey found only 21 percent of respondents report they fix bugs immediately, down from 23 percent in 2017.

    Analytics capabilities included in Sauce Labs’ rechristened Continuous Testing Cloud include the ability to view test results by browser and operating system, visibility into pass/fail and error trends and the ability to see how a single test behaves over time so DevOps teams can more easily identify tests inconsistent with overall test run time and pass/fail rates.

    Scheduled to be available in the second quarter, the Continuous Testing Cloud makes it possible to more easily reconcile advances in the rate at which applications are being developed and constrained testing and debugging tools, Parobek said.

    The Sauce Labs survey results also show progress being made in the realm of mobile application testing. A full 77 percent of respondents said they now use a mix of both real devices and simulators or emulators for mobile testing, up from 34 percent in 2015. That approach results in higher fidelity to the actual experience of an end user on each device, said Parobek.

    That shift, however, may also explain why there is more manual testing. More than half (56 percent) said they do as much or more automated as manual testing, a minor decrease from 59 percent (2017) and 60 percent (2016), the report found. Just under half (45 percent) said they do, however, expect to increase spending on test automation in 2018.

    Parobek said organizations are moving into what he describes as the third era of testing. The first two eras were manual testing followed by a shift to automated testing. The third era is marked by continuous testing that leverages automation to make it possible to test code as it’s being developed rather than after, when the application is complete.

    Naturally, each organization is not as far down the path toward continuous testing as every other. Parobek said for the most part, early adopters of DevOps practices employ continuous testing more widely than traditional enterprise IT organizations that are still in the early stages of embracing DevOps. The good news is the latter have an opportunity to leverage a much larger body of DevOps best practices than the organizations that originally pioneered the concepts.

    — Mike Vizard

  • The Era of Intelligent Testing

    The Era of Intelligent Testing

    The state of the art in software testing has not kept pace with advances in software development over the past 10 years. Modern quality assurance (QA) groups struggle to play their role in a world of continuous delivery, leading to more defects in production, low QA morale and excessive testing overhead. Thankfully, innovations in machine intelligence and test automation have laid the groundwork for a better model—one that will allow QA to keep pace with development, increase productivity, improve product quality, and raise customer satisfaction across the software industry.

    Why We Need a New QA Toolset

    Existing QA solutions were built for a world where software changed infrequently and functionality was highly specified and documented.  This is no longer the case.

    Products Are Evolving Faster Than Ever

    Several forces are converging to accelerate development. Software as a service (SaaS) has removed many of the issues around operating system compatibility, packaging, deployment and versioning that consumed significant development time during the installed software era. Consider that in 2016 Amazon Web Services delivered more than 1,000 new features and services—a pace that would be impossible with packaged software.

    “Development cycles are getting shorter and new features are coming faster than ever…There simply isn’t enough time to test.”

                                                 – QA Leader at Fortune 250 company

    The proliferation of open source and cloud-based features as a service has also reduced the time that developers spend building generic functionality. At mabl, for example, the thought of building our own authentication engine never crossed our minds; we use Auth0. Instead of building our own data processing pipeline, we use Apache Beam and Google Cloud Dataflow. Rather than building our own ML framework, we use Tensorflow. And the list continues; reusable components and cloud-based services save us years of engineering effort, allowing our developers to deliver product features rapidly.

    Continued adoption of agile methodologies has further increased the efficiency of development teams. A recent McKinsey study of 1500 software projects demonstrated that agile teams are 27 percent more productive than those that follow methodologies such as waterfall.

    Finally, continuous integration and continuous delivery are accelerating the pace of change in software. A recent survey by Atlassian suggests that more than 50 percent of software teams are already using continuous integration and/or delivery, and more than 32 percent have plans to move to CI/CD. The same survey indicates that more than 50 percent of organizations are able to push changes to production daily.

    In the end, improved developer productivity means that software products are evolving faster than ever. Unfortunately, this places increased pressure on the QA function. As one QA leader from a Fortune 250 company said in a recent interview with mabl, “Development cycles are getting shorter and new features are coming faster than ever. And we have five productive developers for every QA engineer. There simply isn’t enough time to test.”

    Existing Automation Frameworks Impose Too Much Overhead

    QA rightfully relies on automation frameworks to improve productivity; tools such as Selenium, Appium and JUnit now enjoy widespread adoption. More than 80 percent of the respondents to a recent mabl survey use Selenium today, and QA managers are under constant pressure to increase automation. Unfortunately, while Selenium and other frameworks have helped many companies increase their QA velocity as compared to manual testing, these frameworks have significant flaws:

    • They all require specialized scripting expertise, and given the lack of available talent, teams have very limited capacity for QA automation.
    • They provide limited context in test results, which leaves teams with very little information to help them triage and address issues.
    • At scale, they require their own infrastructure, which takes time to provision, operate and scale.
    • They are tightly coupled to attributes of the product under test that change frequently (xPath, CSS, etc.), resulting in constant maintenance and false positives.

    These flaws only become more painful as the pace of development and change accelerates, compelling many teams to limit their use or discard them altogether, resulting in lower product quality.

    mabl survey – satisfaction with existing testing tools and processes (n = 104)

    “Please indicate the extent to which you agree with the following statements”

    Time Pressures Lead to Unhealthy Team Dynamics

    People in testing roles are increasingly at odds with their peers in development. Unable to keep up with the continuous pace of development, QA becomes a bottleneck to product velocity, leading to tension with their peers in development. This constant pressure also starves QA of its ability to make the very investments in automation and process improvements that are needed to improve throughput. The feeling of always being behind with little hope for improvement often leads to low morale within the team.

    Gaining consensus on the standard of quality has also become difficult for teams, since agile teams often steer clear of detailed functional specifications for each release, preferring instead to focus on user goals and requirements (see: Agile Manifesto). Because user goals and requirements are open to more interpretation, QA and development can disagree on not only the severity of defects but also the validity of the tests themselves.

    Modern software teams are also more open to evolving requirements during a given release cycle (see: “Responding to change” in the Agile Manifesto). Evolving requirements should drive updated test cases, but this requires deep engagement and coordination for already stretched personnel. Rather than testing for quality versus specified behavior, we rely much more on ad hoc and manually intensive exploratory testing to look for defects.

    There is Hope for QA

    While faster feature development, shorter release cycles, evolving requirements and unclear accountability for end-to-end quality all present real challenges for QA, a new generation of QA tools are emerging to confront these challenges head on.

    It Will Be Easy to Create and Maintain Tests

    Unlike existing test automation frameworks, next-generation QA tools will not require specialized scripting expertise. Instead, they will offer intuitive interfaces that allow anyone to quickly create and manage tests. This will help teams expand their test coverage and evolve their tests quickly as their products evolve.

    Tests Will Adapt Seamlessly to Change

    Because they use much more sophisticated, durable methods of simulating user behavior in automated tests, next-generation QA tools will not suffer from the brittleness associated with existing automation frameworks. Tests will not be tightly bound to individual elements within the front-end codebase that change frequently. Rather, they will use machine intelligence to create and maintain sophisticated models of tests, enabling tests to adapt the tests rather than fail when xPaths and other locators change.

    QA Will Run in the Cloud

    Today, teams struggle with the performance, cost and operational overhead of on-premises testing systems. Next-generation QA tools will take advantage of on-demand cloud computing resources to execute tests faster and more efficiently. They will tap into powerful data processing and analytics services to analyze test results and deliver insights. They will be delivered completely as a service, offloading the management and operational burden to tool vendors.

    QA Will Be Part of the Delivery Pipeline

    Next-generation QA tools will be tightly integrated into automated CI/CD pipelines orchestrated by Jenkins, Spinnaker, CircleCI and CodeShip. Tests will trigger automatically whenever teams make changes to the product under test and on-demand. The tools will notify team members when there are potential issues so that they can address them before they affect users. It will be simple to run tests and compare results across builds and environments. Decisions around releases, deployments, promotions and rollbacks will be largely automated.

    The Definition of Quality Will Evolve

    Beyond validating specific assertions in tests, the next generation of QA tools will use machine intelligence to automatically detect and highlight potential regressions in applications. As a result, we’ll think less about tests “Passing” and “Failing” and more about the extent to which changes improve or degrade the user experience. We will focus less on code coverage and more on product coverage. We’ll look beyond basic counts of passing and failing tests in favor of a more on holistic approach to assess the risk associated with a given build or deployment.

    Tools Will Be Deeply Embedded Within Our Teams

    Next-generation QA tools will effectively serve as extensions to development teams. They will share test results and insights through Slack, Jira email and other common communications channels. They will take feedback and training from the entire team—including product, development, QA and customer success—and incorporate what they learn into their tests. Most importantly, next-generation tools will describe tests and results in plain language, and they will provide useful context to help developers reproduce and fix issues, just as an effective QA teammate would.

    The Era of Intelligent Testing

    Software quality assurance has never been easy, and the accelerating pace of software development in recent years has made it even more challenging. Thankfully, a new generation of tools are emerging to address this challenge. Incorporating cutting-edge machine intelligence, delivered from the cloud and embedded into the modern development workflow, these tools will dramatically improve the effectiveness of QA, ushering in a new era of efficiency and innovation across the software industry.

     

    mabl survey – user sentiment (n = 104.  For simplicity, combined “Agree” and “Strongly Agree” into “Agree” category,  combined “Disagree”, “Strongly Disagree” into “Disagree” category.)

    “How satisfied are you with the following aspects of the testing process at your company?”

    — Dan Belcher

  • We’re All Load Testers Now (Maybe)­

    We’re All Load Testers Now (Maybe)­

    More than 94 percent of organizations already have Agile or DevOps initiatives or plan to adopt them within the immediate future, according to a survey by VersionOne. As part of the process transformation required to adopt these more iterative and rapid methods of delivering software, organizations are reassessing all practices associated with software development and test, including load testing.

    Traditionally, load testing has been deferred until the late phases of waterfall release cycles. Specialized performance testers applied legacy load testing tools to validate application performance prior to release. Why is this no longer sufficient?

    1. With teams moving to more componentized applications (including cloud-native apps, microservices, etc.), an application involves many highly distributed components—and a performance issue in any of them could have a ripple effect across the entire application.
    2. Now that new functionality is being released weekly, daily or hourly, each team needs instant insight into whether their incremental changes negatively impact performance.
    3. The later you start load testing, the more difficult, time-consuming and costly it is to expose, debug and resolve performance problems.

    In-depth load testing by performance-testing specialists remains critical, but it doesn’t provide the level of fast, on-demand load testing that’s critical for Agile and DevOps. Developers and testers need a way to expose critical performance issues before new functionality progresses through the delivery pipeline. To achieve this, DevTest team members must be able to easily:

    1. Create load tests that provide fast feedback on the functionality they’re evolving; and
    2. Execute and scale those load tests as needed—without the exorbitant costs and efforts traditionally required to establish, configure and maintain a performance test lab.

    Traditional Load Testing Isn’t for Everyone

    However, load testing has long been the domain of performance testing specialists for a reason: it’s difficult. Today’s developers and testers don’t have the time (or desire) to wrestle with all the technical details required to get load tests working correctly and to keep brittle load tests in sync with the rapidly evolving application.

    The traditional way of approaching load test scripting is at the protocol level (e.g., HTTP). This includes load testing with open source tools such as JMeter and Gatling, as well as legacy tools including LoadRunner. Although simulating load at the protocol level has the advantage of being able to generate large concurrent load from a single resource, that power comes at a cost. The learning curve is steep and the complexity is easily underestimated.

    The main culprit for this complexity is JavaScript. In 2011, there was usually less than 100 KB of JavaScript per page, which spurred around 50 or fewer HTTP requests. Now, that’s doubled: We see on average  200 KB of JavaScript per page, and this gives us more than 100 requests per page.

    Just running a search on a simple search page involves things such as XML HTTP requests processed asynchronously after page load. You also find things such as dynamic parsing and execution of JavaScript, the browser cache being seeded with static assets and calls to content delivery networks.

    For a more business-focused example, consider the SAP Fiori demo app. Assume we want to load test two simple actions: navigating to a page and then clicking on the “My Inbox” icon. This actually generates more than 120 HTTP requests at the protocol level.

    When you start building your load test simulation model, this will quickly translate into thousands of protocol-level requests that you need to faithfully record and then manipulate into a working script. You must review the request and response data, perform some cleanup and extract relevant information to realistically simulate user interactions at a business level. You can’t just think like a user; you also must think like the browser.

    You need to consider all the other functions that the browser is automatically handling for you, and figure out how you’re going to compensate for that in your load test script. Session handling, cookie header management, authentication, caching, dynamic script parsing and execution, taking information from a response and using it in future requests … all of this needs to be handled by your workload model and script if you want to successfully generate realistic load. Basically, you become responsible for doing whatever is needed to fill the gap between the technical and business level. This requires both time and technical specialization.

    You might be thinking, “Okay, we’ll use ‘record and playback’ tools, then.” Theoretically, you could just place a proxy between your browser and the server, record all the traffic going through and be set. Unfortunately, it’s not quite that simple. Even though you’re interacting at the UI level, the testing is still based on the protocol level. Assume we were looking at the traffic associated with one user performing the simple “click the inbox” action described above. When we record the same action for the same user two different times, there are tens if not hundreds of differences  in the request payload that we’d need to account for.

    Of course, you can resolve those differences with some effort. Unfortunately, when the application changes again, you’re back to square one. The more frequently your application changes, the more painful and frustrating this becomes.

    Taking a Browser-Based Approach

    To sum up the challenge here: modern web applications are increasingly difficult to simulate at the protocol level. This raises the question: Why not shift from the protocol level to the browser level—especially if the user’s experience via the browser is what you ultimately want to measure and improve?

    When you’re working at the browser level, one business action translates to maybe two automation commands in a browser as compared to tens, if not hundreds, of requests at the protocol level. Browser-level functions such as cache, cookie and authentication/session management work without intervention. There are a number of ways to simulate traffic at the browser-level: Selenium is clearly the most popular, but there are a number of cross-browser tools available—some of which let you test without getting into scripting.

    However, historically, it just wasn’t feasible to run these tools at the scale needed for load testing. In 2011, if you wanted to launch 50,000 browsers with Selenium, you would have needed something on the order of 25,000 servers to provide the infrastructure. Moreover, it would have been prohibitively expensive and time-consuming to provision the necessary infrastructure.

    Today, with the prominent availability of cloud-based technology, the concept of browser-based load testing is feasible. At the same time, projects Google Chrome and other projects are offering fast automation and better memory profiles with headless or UI-less variants of the browser. In fact, tests with headless Chrome show that it’s possible to go far beyond the industry standard benchmark of two to five browsers per machine and achieve around 50 browsers per machine.

    Suddenly, generating a load of 50,0000 browsers is a lot more achievable—especially when the cloud can now give you access to thousands of load generators that can be launched from any web browser and can be up and running in minutes. Instead of having to wait for expensive performance test labs to get approved and set up, you can get going instantly at an infrastructure cost of just cents per hour. Fast feedback on performance is no longer just a pipe dream.

    At Flood IO, we’ve captured our browser-level user (BLU) research and development in Flood Chrome: a browser-based load generation approach that builds upon APIs from Google for headless Chrome automation. This approach reduces the complexity of writing and maintaining tests while maintaining a viable level of concurrency and performance per machine. It creates scripts at a higher level of abstraction—the user level—then distributes those tests via the cloud for better economies of scale.

    Why consider this new approach to load testing?

    1. Simple scripting—or no scripting at all
    2. Reduced test complexity
    3. Test entire stack in one shot from the user perspective
    4. Capable of testing any user behavior
    5. Record network and user interaction times for front-end optimization
    6. Easier to test earlier and often
    7. Easier to maintain
    8. 10X faster than other Selenium load testing

    Or, in just one line: because it’s purpose-built for DevOps.

    There’s Still a Time and a Place for Protocol-Level Load Testing

    Of course, no single load testing approach is not going to solve everyone’s load testing needs all the time. For example, if you’re trying to test an application that’s not accessible from a browser, the BLU approach isn’t going to work for you.

    Moreover, there are still some situations where you’d be remiss to overlook protocol-level testing with tools like JMeter, Gatling, or API-based test cases. For example, if you want to simulate load against APIs, I’d still recommend running tests that exercise them directly at the protocol level.

    Ultimately, though, it’s important to remember that protocol-level tests can be higher maintenance and use them accordingly. If you have a single click that makes 20 background requests, would you rather wrestle with all the technicalities of scripting that at the protocol level, or have one line of a BLU script that achieves the same business functionality?

    Final Thoughts

    By reducing the complexity traditionally associated with load testing, BLU load testing gives developers and testers a fast, feasible way to get immediate feedback on how code changes impact performance. It’s geared to help people who are not professional performance testers quickly get started with load testing and create load tests that can be run continuously within a CI/CD process—with minimal maintenance. With this new “lean” approach to load testing, any developer or tester can get started with load testing.

    — Tim Koopmans

  • Continuous Testing and the Broken Promise of Test Automation

    Continuous Testing and the Broken Promise of Test Automation

    The Broken Promise of Test Automation

    For more than two decades now, software testing tool vendors have been tempting enterprises with the promise of test automation. However, most companies have never been able to achieve the desired business results from their automation initiatives. Recent studies report that test automation rates average around 20 percent overall, and from 26 percent to 30 percent for agile adopters.

    We believe that several factors contribute to these dismal automation results:

    Legacy software testing platforms were designed for a different age

    The most commonly used software testing tools today are predicated on old technology, but enterprise architectures have continued to evolve over the years. Development no longer focuses on building client/server desktop applications on quarterly release cycles, with the luxury of month-long testing windows before each release.

    Almost everything has changed since test automation tools such as those from Mercury, HP, Micro Focus, Segue, Borland and IBM were developed. Retrofitting new functionality into fundamentally old platforms is not the same as engineering a solution that addresses these needs natively.

    Legacy script-based tests are cumbersome to maintain

    Scripts are cumbersome to maintain when developers are actively working on the application. The more frequently the application evolves, the more difficult it becomes to keep scripts in sync. Teams often reach the point where it’s faster to create new tests than update the existing ones. This leads to an even more unwieldy test suite that still (eventually) produces a frustrating number of false positives as the application inevitably changes. Exacerbating the maintenance challenge is the fact that scripts are as vulnerable to defects as code—and a defect in the script can cause false positives and/or interrupt test execution.

    The combination of false positives, script errors and bloated test suites creates a burden that few QA teams can overcome. It’s a Sisyphean effort—only the boulder keeps growing larger and heavier.

    Software architectures have changed

    Software architectures have changed dramatically, and the technology mix associated with modern enterprise applications has grown immensely. We’re trying to migrate away from mainframes and client/server as we shift toward cloud-native applications and microservices. This creates two distinct challenges:

    • Testing these technologies requires either a high degree of technical expertise/specialization or a high level of business abstraction that allows the tester to test without diving into the low-level technical details.
    • Different parts of the application are evolving at different speeds, creating a process cadence mismatch.

    The software development process has changed

    Although most enterprises today still have some waterfall processes, there’s an undeniable trend toward rapid iterations with smaller release scopes. We’ve shifted from quarterly releases to bi weekly or daily ones, with extraordinary outliers such as Amazon releasing new code to production every 11.6 seconds. This extreme compression of release cycles wreaks havoc on testing—especially when most testers must wait days or weeks to access suitable test environment and test data.

    The responsibility for quality has changed

    In response to the desire for faster release cycles, there’s been a push to “shift left” testing. The people creating the code are assuming more responsibility for quality because it’s become imperative for getting to “done” on time. However, for large enterprises working on complex applications, developer-led testing focuses primarily on a narrow subset of code and components. Developers typically lack the time and the access required to test realistic end-to-end business transactions. Although the onus for quality has shifted left, the legacy platforms, rooted in waterfall processes, have a distinct bias toward the right. This makes it difficult to blend both approaches.

    Open source testing tools have changed the industry

    The rise of open source software testing tools such as Selenium and SoapUI have had both positive and negative effects. Traditionally, open source testing tools are laser-focused on solving a very specific problem for a single user. For example, Selenium has become an extremely popular script-based testing tool for testing web interfaces. Yet, although Selenium offers speed and agility, it does not support end-to-end tests across packaged apps, APIs, databases, mobile interfaces, mainframes and other components. There’s no doubt that most of today’s enterprise applications feature a web UI that must be tested. However, in large enterprises, that web interface is just one of many elements of an end-to-end business process. The same limitation applies to SoapUI and API testing.

    So, Now What?

    Software testing must change. Today’s software testing challenges cannot be solved by yesterday’s ALM tools. With disruptive initiatives such as DevOps, continuous delivery and Agile expanding across all industry segments, software testing becomes the centerpiece for data-driven software release decisions. This next wave of software delivery lifecycle (SDLC) maturity requires organizations to revamp antiquated testing processes and tools. This means that organizations must have technologies that enable continuous testing, or innovative ideas will remain hostage to yesterday’s heavyweight testing tools.

    — Wayne Ariola

  • Adopting Continuous Testing: 4 Success Stories

    Adopting Continuous Testing: 4 Success Stories

    Much has been written about the benefits and best practices of continuous testing. However, it’s not always clear how to make these concepts a reality. Here are a few stellar examples of how several leading organizations were able to apply continuous testing to remove roadblocks and achieve their organization’s distinct quality goals.

    The Middle Child Rises – Testing Evolves to Quality Engineering

    Ann Lewis, ExxonMobil

    ExxonMobil’s philosophy is centered on safety, security and controls as the foundation of our 135-year-old company. Based upon our fundamental policy, quality has become a key component in delivering software. We are currently undergoing a shift from traditional project management organizations and support organizations to product teams. This shift has highlighted the need for testing early in the development cycle and to combine new development with ongoing maintenance.

    Testing, the team “in the middle,” has become an integral component of the ExxonMobil development life cycle as “Quality Engineering.”

    To do this, we started with the basics—quality control manages risk—and then explored what is tracked today and whether quality is being measured. We also removed the roadblock to continuous testing. In the end, the Quality Engineering team has created the expectation that testing is a part of everyone’s mindset; that continuous testing and deployment can only be accomplished through model-based testing; and that managing quality reduces overall cost of ownership.

    Continuous Testing: How Vantiv is Maturing

    Sandra Baker, Vantiv

    Previously, Vantiv shared our automation journey and how we got started. Now, will share how we are maturing as a QA organization and in the use of Tricentis Tosca.

    Our focus has shifted to how we manage and execute our 15,000+ tests. We have focused on unattended execution to achieve maximum benefits. The next step was to further advance in CI/CT/CD and our maturity. To do this, we have introduced new metrics and processes.

    Finally, we are expanding our capabilities to end-to-end testing, performance testing, user acceptance, service virtualization and big data testing.

    How A1 Telekom Austria Speeds its Delivery with Continuous Testing

    Remove featured image

    Christian Fischl, A1 Telekom Austria & Walter Pindhofer, Qualysoft

    The automated tests enable A1 Telekom Austria AG to deploy new features faster and with less risk. This leads to shortened release cycles and more agility. While it is still an ongoing transformation, it has already become a huge success story. This session is all about A1 Telekom Austria AG’s journey to continuous testing, starting with the first automation to the sometimes-tough steps toward automated system regression tests, and a look at the future plans.

    How Varian Achieved Quality @ Speed in Regulated Environments

    Manish Mathuria, Infostretch

    In the new world of digital, agile and DevOps, traditional approaches to software delivery no longer apply. Development and testing teams are under immense pressure to deliver quality products at speed. Varian Medical Systems, a leading manufacturer of medical devices and software for treating cancer, implemented sales and service management solutions using Salesforce, ServiceMax and SAP. To keep up with continuous quality demands in regulated environment, the company partnered with Tricentis and Infostretch to automate testing of key processes. This session highlights Infostretch‘s automation solution using Tricentis Tosca, nuances of testing SaaS models, and key test case, test data and environment management strategies.

    — Jessica Wanivenhaus

  • A Crash Course in Continuous Testing

    A Crash Course in Continuous Testing

    What better way to start a blog on continuous testing than with this “I Love Lucy” clip, and then an analogy:

    https://youtu.be/8NPzLBSBzPI

    Like Lucy and Ethel, many software testers have been scrambling to keep pace with accelerated processes—then along comes the supervisor saying, “You’re doing splendidly … Speed it up!”

    This ever-increasing pace is just one of four aspects of the “perfect storm” driving the need for continuous testing.

    What is Continuous Testing?

    Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain feedback on the business risks associated with a software release candidate as rapidly as possible.

    Why do we need to rethink our approach to testing?

    The expectations associated with test are changing–but legacy testing platforms aren’t keeping up. Legacy testing platforms take a “heavy” approach to testing. They rely on brittle scripts, deliver slow end-to-end regression test execution and produce an overwhelming level of false positives. As a result, they’ve achieved limited success with test automation. The overall test automation rate is 18 percent, on average–8 percent for enterprises. In a polling question, the audience confirmed that the results of test automation have been so-so, at best.

    Now, software testers are facing increasingly complex applications, delivered at dramatically accelerated paces—and they’re expected to deliver trustworthy go/no-go decisions at the new speed of modern business. More of the same won’t get us there. We need to transform the testing process as deliberately and markedly as we’ve transformed the development process.

    The Differences Between Test Automation and Continuous Testing

    There are three main differences between test automation and continuous testing:

    • Risk
    • Breadth
    • Time

    Risk: Exposing more and more innovative functionality to the user is now a competitive differentiator—but it also increases the number, variety and complexity of potential failure points. Most tests are designed to provide low-level details on whether user stories are correctly implementing the requirements—not high-level assessments of whether a release candidate is too risky to release. Would you instantly stop a release based on test results? If not, your tests aren’t properly aligned with business risks.

    Breadth: Just knowing that a unit test failed or a UI test passed doesn’t tell you whether the overall user experience is impacted by recent application changes. To protect the end user experience, you need tests that are broad enough to detect when an application change inadvertently impacts functionality which users have come to rely on.

    Time: If your organization has adopted DevOps and is performing continuous delivery, software may be released hourly—or even more frequently. In this case, feedback at each stage of the process can’t just be “fast”; it must be nearly instantaneous. If the business wants to minimize the risk of faulty software reaching an end user, you need some way to achieve the necessary level of risk coverage and testing breadth—fast.

    What Are the Key Points of Continuous Testing?

    • Continuous testing’s primary goal is assessing business risk coverage.
    • Continuous testing establishes a safety net that helps the team protect the user.
    • Continuous testing requires a stable test environment to be available on demand.
    • Continuous testing is seamlessly integrated into the software delivery pipeline and DevOps toolchain.
    • Continuous testing delivers actionable feedback appropriate for each stage of the delivery pipeline.

    You can find a more extensive list (14 points) at the What is Continuous Testing page.

    Where Can I Learn More About Continuous Testing?

    Watch the complete continuous testing webinar to get more details on the above topics and also learn:

    • How continuous testing impacts the organization
    • What practices testers need to know to be successful with continuous testing
    • How non-automated practices such as exploratory testing fit into continuous testing

    If you want to learn more, download the white paper, “Continuous Testing vs Test Automation: What Testers Need to Know” or visit the What is Continuous Testing page.

    — Cynthia Dunlop

  • Continuous Testing Defined

    Continuous Testing Defined

    Across the software testing industry today, everyone has started to agree that continuous testing is necessary for keeping pace with Agile and DevOps. What’s not as clear, however, is exactly what continuous testing involves.

    Wikpedia defines continuous testing as the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

    At Tricentis, we believe that continuous testing …

    • Assesses business risk coverage as its primary goal
    • Establishes a safety net that helps the team protect the user experience
    • Requires a stable test environment to be available on demand
    • Seamlessly integrates into the software delivery pipeline and DevOps toolchain
    • Delivers actionable feedback appropriate for each stage of the delivery pipeline

    But what about the software testing community at large? How do you define continuous testing and what does it really mean to you? At Accelerate 2017, the continuous testing conference, we turned the question over to our attendees. We created a What is Continuous Testing board and invited attendees to share their thoughts.

    Some of the bravest attendees got the ball rolling with entries stressing the need for enterprises to embrace change and its importance for Agile and DevOps. 

    These entries (and perhaps the opening reception as well) opened the flood gates to lots of great attendee ideas. Here’s a glimpse at what people were talking about:

    As the board started to fill up, a few key words popped out:

    Many of the obvious buzzwords stuck out, such as Agile and DevOps—not to mention both Continuous and Testing, which were repeated often—albeit separately. Numerous cards indicate that enterprises see continuous testing as a journey or a goal to strive toward rather than a technical process component like continuous integration.

    Here at Tricentis, we are passionate about the future of continuous testing and are constantly innovating to help our customers stay ahead of the curve. To learn about what we believe continuous testing requires and our vision for where it’s headed, check out our Continuous Testing Resource Center and be sure to watch Tricentis Founder Wolfgang Platz’s keynote from Accelerate 2017.

    — Jessica Wanivenhaus

  • Plutora Brings Order to Application Testing Chaos

    Plutora Brings Order to Application Testing Chaos

    One of the reasons application testing doesn’t occur as often or extensively as most IT organizations desire is because managing multiple simultaneous testing projects is complex. To help simplify the management of applications testing at scale, Plutora has updated Plutora Environments, its continuous delivery management platform, which now includes an ability to manage the availability, usage and configuration management involved in testing applications at scale.

    Jeff Keyes, director of marketing, says Plutora Environments now provides a single repository through which all application testing can be managed, including identifying and provisioning available IT infrastructure resources.

    Made available as a software-as-a-service (SaaS) application, Plutora Environments provides needed governance and control capabilities. Without them, Keyes says, application testing tends to get de-prioritized when developers find it too challenging to quickly access the appropriate resources.

    Capabilities added to the latest release of Plutora Environments include the ability to:

    • Visually associate new code commits with the features or fixes via a dashboard,
    • Ensure accuracy of test coverage across high-velocity tests,
    • Trace updated code to specific defects or original requirements, and
    • Automatically link change requests to associated test cases.

    Keyes notes a large amount of IT infrastructure allocated to testing is being wasted due to a lack of oversight. In many ways, what’s been missing in many DevOps environments is some of the management functionality found it traditional IT service management (ITSM) software, he contends, and the latest release of Plutora bridges that gap within the context of an application testing environment.

    Now that developers are being held more accountable for the quality of their applications, demand for access to testing resources will steadily increase, Keyes says.  IT organizations then must balance a limited amount of resources against those demands—they may want to shift responsibility for application testing to the left as part of an integrated DevOps process, but to accomplish that goal, they will need to be able to provide developers with access to infrastructure resources for testing purposes on demand. Otherwise, developers will simply opt to continue writing code, even though many of the dependencies on which that code depends has not been tested.

    Application testing, regardless of era, regularly gets the short shrift. Developers are almost always falling behind schedule. One of the easiest ways to make up for lost ground is to reduce the amount of time allocated to testing. Some developers even believe agile development methodologies enable them to respond fast enough to any flaw by delivering additional updates—many of which don’t get tested either, because of a lack of time. End users, however, are becoming less forgiving when it comes to application experiences. If an application doesn’t perform as expected the first time, customers are less apt to give that application a second chance, much less a third or a fourth.

    Because of that reality, in many organizations there’s more focus than ever on application testing. The challenge is not necessarily creating or running the test, but rather, how best to manage the overall process end to end.

    — Mike Vizard

  • Dissecting the Role of QA Engineers and Developers in Functional Testing

    Dissecting the Role of QA Engineers and Developers in Functional Testing

    Until a few years ago, quality assurance was generally considered low-level R&D. As a result, quality assurance (QA) teams were characterized by a low budget, lack of attention and, more often than not, poor attitudes. Moreover, QA engineers were considered the enemies of developers and the product.

    However, today, this is no longer the case, and QA is part of the entire software life cycle. Today, organizations understand how a product’s quality affects the brand’s reputation and customer retention. In the Agile ecosystem, QA engineers are no longer solely focused on testing; they also fill an important role in the planning and delivery domains. The shift left approach—where testing is performed earlier in the life cycle—requires QA engineers to have higher technical abilities and involvement in the whole CI/CD process. Nevertheless, QA managers, who usually encourage assimilation in the scrum team, as well as rapid feedback for user stories and automation, must not forget the main objective of quality engineers: to ensure a high-quality product and a seamless user experience for customers.

    High quality demands testing and verifications. There are different types of testing that can be conducted to meet quality goals. In one of its blog posts, Testing Excellence describes almost 60 types of testing that can be performed on a product. We’re going to look at one that is quite clearly one of the most important types of testing: functional testing.

    Different than other forms of testing, functional testing is considered one of the most sophisticated test type, and the most comprehensive one.

    Same Team, Different Responsibilities

    One important fact every QA engineer must remember is that developers develop and QA engineers test. Even though QA and developers are part of the same scrum team, their work, methodologies and state of mind are completely different. In cases in which the significant contribution of QA isn’t visible to the whole team, testing might be considered an easy and clear task. Quite the contrary; testing an application is much more complicated and entangled than developing it. When planning tests and executing them, QA engineers must take into account several aspects developers likely never consider.

    Understanding How the User Thinks

    The primary distinction between dev and QA is state of mind. While developers write pieces of code that later become features in the application, QA is expected to understand how real users think, engage customers to learn business scenarios, and confirm that the application satisfies customer needs.

    Let’s look at eBay, for example. Every veteran R&D member will agree that eBay isn’t just a simple shopping application, but a comprehensive system that includes a whole set of capabilities on both web and mobile that requires attention and deeper understanding to determine whether they really function correctly. In such a rich and mature online platform, we can identify several types of users and many cases in which the application may be utilized. A seller performs different steps than a buyer and requires different functionality, while a support agent needs an entirely different version of the application. When planning tests, all these personas must be taken into consideration with a particular test plan for each. In other words, testing functionality is not just simulating user operations.

    In addition, eBay is not a standalone application—its web application lives inside the browser and subsequently depends on that browser’s rules and behaviors. This is true for eBay’s mobile apps, as well, which must meet the OS constraints to function correctly, whether it’s navigating to the seller’s store or making a purchase. This makes compatibility testing an integral part of functional testing. Most developers don’t think about the project availability matrix when they code, so QA is required to contemplate all the ways the customer will engage with the application and verify functionality for each.

    Viewing the Application as a Whole

    Dev and QA differ in their point of view—an obvious but important distinction. While developers focus on developing user stories, QA engineers test the user story, the feature it is part of and the application behavior with the new code. Integration testing is a crucial part of the application verification effort and requires a broader view of the application and how the new functionality is part of it. End-to-end testing is the fulfillment of the above two aspects, validating that real user scenarios in the context of multiple regions in the application behave as expected. End-to-end tests increase test coverage and reduce the risk of making the new code part of the application.

    Paying Attention to User Experience

    As noted, testing the application functionality is merely one step in functional testing. Testing the operations available in the application encompasses another type of testing—the user experience. If a new feature is developed, but users do not have easy access to it or cannot understand how to use it, it is as if the feature doesn’t exist. QA engineers must incorporate user experience as a main metric of features so users will not have room for error, which can lead to misuse and frustration.

    For instance, eBay’s “all categories” page previously displayed all the categories by default, even though some of the categories had no items. While this was the correct behavior from a functionality standpoint, it caused user confusion and mistrust. Consequently, the “all categories” view was changed to show only categories with items.

    Looking at the narrow current task, developers might implement features as they are told without questioning or contemplating the specifications. In the same way, testers who test features without realizing how customers perceive them are missing an important benefit of the test.

    The above three aspects—thinking like customers, testing the application as a whole and paying attention to user experience—revolve around the way developers and QA engineers relate to new features. While developers tend to follow the specification guidelines, QA engineers must “think outside the box” when it comes to user consumption of the feature.

    There are two other ways in which QA work is more complicated and yet invaluable—test maintenance and test automation.

    Maintaining High Quality Throughout an Efficient Release

    In general, developers split their time between developing new features and fixing defects. QA work is much more diverse. A substantial portion of the release is spent on testing functionality that already has been released and used. While the main effort of designing and writing test plans for this functionality was invested in past releases, every new feature developed requires a test plan and a maintenance effort on existing tests. One of the main goals for QA managers is reducing this effort to a minimum. Teams that succeed in creating clear and reusable tests simplify the maintenance struggle and optimize the test registration time.

    Eliminating Risk Through Automation

    However, for QA, the most cost-effective solution is test automation. At release end—the time when QA is at its most stressed and every commit might introduce a new risk—automation testing can eliminate risk and supplies early feedback on every change. Agile teams strive for the highest automation coverage percentage so the development team can have confidence in committing changes late in the release. That said, developers often aren’t interested in investing their time in automation. They support the QA automation effort, but they don’t want to “dirty their hands” by doing testing. Whether it’s UI automation—during which achieving element uniqueness is essential yet tricky and requires developers’ intervention—or API testing for APIs without proper documentation, QA faces intricate challenges that require sophisticated solutions, which developers are, in most teams, not part of. While the dev team focuses on the status of the new feature, QA attends to the application as a whole with extra attention to identify regressions created by a change.

    Join Forces for Higher Rate of Success

    Testing an application requires a completely different mindset than developing it. One of the main challenges in the Agile methodology is to create developer awareness of quality, involving other stakeholders in the quality assurance activities and making the QA process visible to everyone. QA managers must understand how day-to-day QA work is different than the work of other team members so they may truly represent the complexity, efforts and eventual success. R&D groups that join forces and understand that QA extends far beyond the actual tasks of the QA team have a much higher chance of improvement and set high standards for their product’s quality. Implementing the appropriate methodologies and utilizing the right tools lead to an increase in customer satisfaction and retention rate.

    About the Author / Ronit Eliav

    Ronit Eliav is the Director of Product Marketing at Panaya. With a deep interest in digital transformation and the latest IT trends, she enjoys creating thought leadership around such topics as agile and continuous delivery, DevOps, quality assurance and IT modernization projects. Connect with her on LinkedIn.