devopsy.kinsta.cloud

Author: Wolfgang Platz

Shift Left … With In-Sprint UI Test Automation (?!?)
UI testing is really the most intuitive testing approach imaginable: Just find someone who can represent the target user and let them poke around in the interface they use for their daily work—the UI. It’s natural. It’s intuitive. And it’s still the predominant method used for testing most software releases today.

This human-driven testing can never stop. Having smart testers creatively explore the UI using their creative, analytical minds to uncover issues that disrupt the user experience is absolutely essential. But it’s also essential to perform a level of regimented, routine checking with each change to ensure the modifications don’t introduce strange side effects. Performing such tedious checking at the required frequency and speed would be torture for human testers. Hence, the push toward test automation.

You would think, then, that the natural place to start with test automation would be to automate the most natural of all testing: humans interacting with the UI. Not so: UI test automation is actually one of the most dreadfully difficult test automation approaches.

Tests automated at the UI level historically have a fair share of issues. They’re slow to run. They’re fast to break. They’re somewhat technical and tedious to create even for common modern technologies such as web and mobile. They’re much more difficult to create when you have to deal with custom desktop applications, packaged applications, extremely old technologies, extremely new technologies or Citrix/remote applications. And creating automated UI tests typically requires a fully implemented—and stable—UI.

If you’re aiming for in-sprint test automation, this means that tests need to be implemented in that tiny sliver of time between when developers complete the final UI code (with a successful build) and when the sprint is supposed to be completed. Guess what? That rarely happens. Within the sprint, teams can certainly tackle human UI testing (a.k.a. exploratory testing). But automated UI testing? That’s often delayed until a later sprint … and/or scrapped altogether.

Clearly, there are a lot of issues with UI test automation. But if you really think about these issues, it all boils down to one thing: The test automation doesn’t think like a human:
- A human can interact with an application’s UI quite rapidly (the human eye can process 24 images per second).
- A human is not derailed by interface changes. Something such as a reimplemented drop-down menu or a new way to display results won’t phase a human tester at all.
- A human brain can interact with extremely old technologies (such as mainframes and COBOL-based apps) along with whatever new technology happens to be released tomorrow. And if your application is migrated from a monolith to a microservices architecture, the same tester’s brain can still test its UI in exactly the same manner.
- A human tester might not recognize whether the application is installed on their machine or is accessed via desktop-virtualization. They can test its UI just the same.
- A human doesn’t really need to wait for a fully implemented (and stable) UI implementation. They can start working on their tests the moment a rough prototype is available. And they can start performing tests as soon as the UI is at least kinda—If the underlying technical implementation is still shifting a bit, it’s not necessarily a showstopper.
So, if the key problems with UI test automation stem from the fact that human intelligence is greater than test automation, why not try to address those problems with the next best thing: artificial intelligence. By tapping the power of AI, we can achieve UI test automation that’s as smart and resilient as a human, but as efficient and scalable as a machine.

For this to work, the technology must learn how to understand user interfaces in the same way that a human can. For example, it needs to know that a drop-down is a drop-down just by “looking” at it. With that accomplished, you—well, any human, really—can define test automation by simply providing a natural language explanation of what business process should be tested (e.g., “Enter 10,000 in the list price input, click the confirm button, then … “). AI-driven test automation can take it from there. There’s no need to deal with shadow DOMs, web assemblies and the like. Emulated tables on SAP can be automated without any special customization. It’s not going to fail the next time there’s a minor UI tweak, and you won’t have to rebuild your tests if the application is reimplemented in a new technology.

In terms of shift left, you can start implementing your UI tests as soon as the team sketches out a rough prototype or mockup. Advanced UX designs like this are a great starting point:

You really could start anywhere though. Even more rudimentary whiteboard sketches would enable you to start on UI test automation without waiting on the actual implementation. This means less crunch time each release and ultimately eliminates the ripple effect of testing delays, unfinished stories and “hardening sprints.” And it enables an interesting flavor of TDD: Start running these automated UI tests before implementation begins, then have developers write code that makes them pass.

The catch? It’s not so easy to make UI test automation so easy. Two things to watch out for: accuracy and speed. For this to work, the technology must be able to accurately recognize and interact with UI controls that it’s never seen before. This requires visual object detection as well as optical character recognition (OCR). And that’s the easy part; the difficult part is doing this at speed. For a while now, AI-driven automation solutions have been struggling to come close to the speed of human processing. On average, you’re looking at tools processing 1.8 images per second, while humans can handle 25. You don’t need automation that mimics human speed, though—you need to beat it. If you want this UI testing to run as part of your continuous testing as a core part of your CI/CD process it needs to vastly surpass the speed of the human eye and the human brain.

Once you overcome that little challenge, AI-driven UI testing will remove a lot of not-so-artificial conflict on Agile dev teams:
- Testers waiting on developers to finish the UI so they can start implementing tests—and avoid the dreaded crunch time at the end of sprints.
- Developers waiting on testers to finish tests so that the user story is deemed “done done” within the current sprint.
- All the retro finger-pointing you get when delayed or unfinished testing results in unfinished user stories.
I admit: when you first tell the team that UI test automation will help you complete testing sooner, it could be a hard sell. But after a few rounds of having UI test automation ready before developers write a line of code, I’m confident that you’ll find some real support for this artificial intelligence-driven testing approach.
October 2, 2020
Quality at Speed During the COVID-19 Pandemic

Think back to 2015. The U.S. economy was in the early phases of a growth spurt that would continue for years. Ninety-five percent of organizations were practicing agile, to some extent (State of Agile, VersionOne). And the proportion of the IT budget allocated to QA and testing had just increased from 18% to 26% (World Quality Report, Capgemini).

“Quality at speed” became the catchphrase of the year in the testing world. With all the buzz around DevOps and agile, it was clear that speed mattered. In reality, most teams weren’t quite there yet. Agile adoption was still low (less than 50% of teams within each organization actually practiced it) and maturity was even lower (only 17% of those practicing it reported mature adoptions). Still, the writing on the wall was clear. Leaders wanted more application development, faster — so testers should at least prepare for the pending storm.

From that point forward, organizations across almost every industry all progressed to a sweet spot where time-to-market pressure was relatively high and cost pressure was moderate. By the end of 2019, quality at speed had graduated from buzzword to business as usual at most organizations.

But then came COVID-19. Now, we’re in the midst of a global pandemic. Every business, every non-profit and every governmental agency is impacted — but in different ways.

Speed expectations suddenly shifted into overdrive for an unlikely assortment of organizations, including government agencies and non-profits, healthcare and pharmaceutical companies, online retailers, online conferencing providers and telecoms. Yes, everyone had already been working to accelerate their software delivery processes. And many enterprise organizations had made great strides toward scaling the adoption of agile and DevOps. But, very few of these organizations were previously delivering applications at the extreme speeds required right now.

For another group, cost reduction suddenly became the top priority. Airlines, hotels, as well as other travel and leisure organizations experienced such dramatic changes in fortune that speed became a non-issue. The focus shifted to increasing cost efficiency in an effort to offset some of these staggering losses.

Assuming that high quality cannot be compromised, this leaves both of these groups with distinctly different — yet equally daunting — challenges from a quality perspective.

The organizations in the first group suddenly needed to build, test and release applications at the speed of a startup — but with the added burden of legacy systems, complex application landscapes and intense regulatory scrutiny. Quality at speed remains the goal, but the “speed” part is much, much faster.

The organizations in the second group need to determine how to continue meeting strict quality expectations with a significantly slimmer testing budget. Quality at speed isn’t so important right now. Speed gets replaced by the other corner of the classic cost-quality-speed triangle, making the new mantra “quality at cost.”

It’s hard to find good news in the current situation. But, from the software testing perspective at least, there’s some consolation. Whether you’re facing unprecedented time-to-market pressure or extreme cost pressure, the same core “test efficiency” strategies will help you either way.

Here’s a quick overview of ways you can test more efficiently at different steps of the software testing process.

Selecting and Designing Test Cases

I’m addressing these two steps together because they both face a common challenge: intuition. Most modern business systems are insanely complex — so complex that it’s just not feasible for most testers to intuitively identify the best set of tests to cover a specific test need. Having a more systematic way to find the smallest — yet most powerful — set of tests will save you both time and money.

For designing test cases for progression testing, methodologies like linear expansion and pairwise testing can help, and augmented intelligence can dramatically simplify them for you.

For selecting what test cases should be run for regression testing, consider change impact analysis. Some approaches identify which objects are impacted by a given change, then help you narrow the test scope to the impacted objects. This often reduces the testing scope by 15%. Not bad, but you can do much better.

AI-powered change impact analysis pinpoints exactly what needs to be tested in a given update, based on whether the changes put an object at risk. With this approach, you can reduce testing efforts by 85% or more. Not only is this approach faster and cheaper, it’s much more efficient at uncovering defects. The number of defects that hit production are reduced by 75% to 95%, sometimes even 100% (no defects in production).

Automation and Execution

Here, technical scripting is the top time and cost drain. Some level of maintenance is required to keep scripts running, and this maintenance will cut into your test efficiency. One way to combat this challenge is to take a model-based approach to test automation. Such approaches have been field-tested for years, and proven to solve the test maintenance nightmare that’s doomed all-too-many test automation initiatives.

The new kid on the block is AI-driven UI test automation. This is a big topic that warrants a much deeper discussion — but here’s a quick overview. Specify tests in natural language. Build the tests even before the actual UI is ready. Use the same tests as the technical implementation changes. Understand dynamic behavior like a human tester would. Test absolutely any technology — faster than a human eye. Curious? Stay tuned.

Analysis of Results

When it comes to results analysis, false positives are the vampire sucking your resources. False positives are tests that fail even though the application is behaving correctly. They will really kill you in terms of cost and speed. First, someone needs to figure out why the test failed. This might involve manually checking the related application functionality, inspecting the test automation, reviewing the test data, checking whether every element of the test environment is up and working as expected — maybe even all of the above. Once the exact problem is found, it needs to be fixed. Then, the test needs to be rerun. If it still fails, rinse and repeat.

How do you stop false positives? To start, stabilize the automation with model-based test automation or the AI-driven testing I mentioned in the previous section. As you move into more advanced automation scenarios, stateful test data management and service virtualization can help you stabilize tests to the point that they’re fit for CI/CD. Then, your final challenge is to deal with the flaky tests resulting from things like asynchronous application behavior, concurrency issues and test order dependencies. Again, AI and ML can offer tremendous assistance here—in flagging potential false positives, and even in helping you identify the cause and optimal resolution.

Final Thoughts

COVID-19 split up the original quality at speed sweet spot into the extreme speed camp and the extreme cost-savings camp. In both cases, the same core strategies to increase your test efficiency will help you increase speed as well as reduce testing costs. If you’re under extreme time pressure, showing that you also reduced costs while meeting speed expectations certainly won’t hurt. If you’re currently reducing costs because your industry is essentially paused, taking these steps now will prepare you to speed it back up when the world becomes more normal again.

June 4, 2020
Software Testing: The Comeback Kid of the 2020s

The past 15 years have been rough on software testers. The rise and fall of outsourcing, Agile growing from a manifesto to a movement and ever-shrinking release cycles have all taken a toll. By 2011, Albert Savoia walked on stage at the Google Test Automation Conference (GTAC) dressed as the Grim Reaper and declared testing dead. Since then, the emphasis on the “automation” part of “test automation” has fueled demand for Software Development Engineers in Test (SDETs) and left many testers wondering if they must learn programming to save their jobs.

If you look at how frequently people are searching for “software testing” on Google, the trends seem to suggest that the profession is, if not dead, at least flatlining.

However, I think the times are a-changin’. The pendulum will be swinging back—to the point where software testing could be the comeback kid of the 2020s. Sounds crazy, I know. But consider some developments that could turn things around.

Developers Don’t Want to Bear the Testing Burden

On Agile development teams, the party line is that unit testing is essential. It’s the most efficient way to expose implementation errors as they’re introduced. You can feasibly cover each function/method when you test at this level, plus when a test fails, you know exactly what function is broken. Moreover, having a suite of effective unit tests establishes a safety net that alerts the team whenever changes break previously-verified functionality. Those are just a few of the reasons why unit testing forms the foundation of the Agile test pyramid.

But, if you talk to developers in a casual setting, the truth starts to come out. Unit testing requires a lot of work, and they’re skeptical about whether it pays off. Just skim through a few of the 2.5 billion results returned by a Google search for “Is unit testing really worth it?” to get an idea of some of the issues.

Unfortunately, a seemingly endless amount of work is required to ensure that unit testing delivers long-term benefits as a regression testing safety net. Developers are committed to progression testing, not regression testing. Although developers on Agile teams usually write unit tests to check new functionality, few keep them up to date as the code base evolves. As the team moves on to another sprint, these tests become regression tests. Then, little by little, they start failing—but developers have moved on. The number of working unit tests at the base of the Agile test pyramid erodes, as does the level of confidence the test suite once provided.

In fact, data from our internal studies on financial instructions across EMEA shows that as an iteration ends, unit tests cover about 70% of the new code. After a few iterations of code getting extended, refactored and repaired, the original unit tests provide only about 50% coverage of that original functionality. This drops to 35% after several more iterations, and it typically degrades to 25% after six months.

Is Unit Testing Doomed?

Ultimately, developers don’t have the time or desire to keep these tests current over the long term. Unit testing has been a best practice for more than 20 years, yet despite waves of unit test automation tools (including one created by Albert Savoia not long before he declared testing dead), unit testing remains a thorn in developers’ sides. Does that mean we give up the benefits of unit testing altogether? Not necessarily.

In order to take on unit testing per se, testers would need to understand the developers’ code as well as write their own code. That’s not going to happen. But, you could have testers compensate for lost unit test coverage through resilient tests they can create and control. Professional testers recognize that designing and maintaining tests is their primary job and that they are ultimately evaluated by the success and effectiveness of the test suite. Let’s be honest, who’s more likely to keep tests current, the developers who are pressured to deliver more code faster, or the testers who are rewarded for finding major issues (or blamed for overlooking them)?

Another option is to consider new testing approaches that achieve unit testing benefits for today’s most popular development architectures—but without requiring knowledge of the code. Enter microservice contract testing.

Microservice Contract Testing: The New Unit Testing

Interest in microservices (the blue line on the graph below) has been rising rapidly since 2015. In fact, it seems to be on the exact opposite path from software testing (red line).

This rise is hardly surprising. From the architecture perspective, there are significant benefits to microservices. Initial forays into service-oriented architectures (SOA) were a huge step forward in enabling reuse and flexibility. Business logic formerly implemented in monolithic applications was freed and released as isolated independent services, which were loosely-coupled and highly-distributed.

Microservices take the decoupling and granularity to new levels. Business logic is now implemented in the smallest, most discrete units possible. Functionality that would have once been a single service is now distributed across tens, or potentially hundreds, of microservices, each of which can be independently developed and deployed.

In addition to enabling unprecedented agility and scalability, this extreme decoupling opens up promising new options for testing. With the initial waves of services and APIs, API tests had little choice but to pass through a fairly broad set of business logic, which was likely implemented through multiple units of code (e.g., functions).

If an API test failed, there was no clear pointer to what specific code function needed to be fixed. That required unit testing and/or additional debugging.

Now, with each microservice implementing an extremely focused unit of business logic, you know exactly what code is broken when an API test fails. Why spend the time and effort on unit testing if you’re already getting the benefits from something that’s considerably less painful?

You might be thinking, “Testing hundreds of independently-evolving microservices is just going to present a different pain.” Correct. Each integration point is a potential point of failure, and testing all the possible integrations is a logistical nightmare—so don’t. Instead of testing how each microservice interacts with all the other microservices it touches, perform contract testing instead.

Very basically, contract testing helps both microservice consumers and providers ensure that they’re meeting expectations in terms of message format and data. The microservice consumer adds a contract that specifies expected request and response pairs for its various interactions with other microservices. Each contract is used to create a virtual service that the consumer can then test against. If the consumer’s changes result in incorrect/unacceptable request messages, they’ll be alerted immediately by a contract failure message triggered from the virtual services.

On the flip side, the microservice provider can use the same consumer-specified contracts to ensure that their own changes don’t break the functionality which consumers depend on. If the actual provider response suddenly fails to meet consumer expectations, they, too, get a rather clear and telling contract failure message.

The beauty of this approach—even beyond the fact that neither consumer nor provider ever has to stand up the whole spider web of interconnected and independently-evolving microservices—is that you get very specific unit-level failure information without burdening developers. As soon as testing moves away from the code level to the API level, it doesn’t need developers doing it. You certainly don’t need developers to deal with the maintenance, and you really don’t even need them to set up the test. Instead, the task can be delegated to people who view testing—not development—as their profession. We call them testers.

Looking Beyond Unit Testing Is Essential for Ensuring a Positive User Experience

Nevertheless, hyperfocusing solely on unit testing of any flavor can leave you blind to many critical issues. For example: Does every step of the end-to-end transaction work as expected—from whatever mobile or browser interface the user happens to have, through the backend (which might interact with custom/legacy applications, packaged applications, mainframes, databases, third-party services, etc.) and back to the user? Will these same transactions perform acceptably under a sudden increase in demand? If the user exercises the application in ways that the developers didn’t anticipate, will the application respond in a reasonable manner, or at least fail gracefully?

The answers to such questions lie beyond the horizon of the testing performed by the typical Agile team, which rarely extends beyond unit testing and a little integration testing.

However, as organizations invest millions in digital transformation initiatives, this doesn’t cut it. Maximizing that investment requires a flawless user experience from start to finish, 24/7. Professional testers thrive on exposing the glitches that developers tend to overlook but that would cause a CIO to cringe.

I’m confident that software testing will surge in importance in the 2020s if we commit to expanding its scope and effectiveness, which will ultimately elevate its business value. As the pendulum swings back, testers can’t expect to just hang on for the ride. Willingness to jump on the momentum of this decade’s trends—for example, stepping in to solidify the foundation of the Agile test pyramid and embracing new practices such as microservice contract testing—will be essential for navigating the turnaround.

April 24, 2020
DevOps Places Software Testing in the Crosshairs
To keep up with the pace of innovation, testing must play a larger role in the software development life cycle

The speed of digital transformation is already staggering, and it’s only going to increase. To put this into some very concrete terms, consider that:
- There are 7.7 billion people in the world.
- 5 billion have regular access to a toilet.
- 5 billion own a mobile phone.
All of a sudden, a huge number of people jumped from a very provincial lifestyle straight into digital times—creating a tremendous demand for more, and more innovative, software.

The traditional ways of developing and delivering software aren’t adequate for meeting this new demand. Not long ago, most companies were releasing software annually or bi-annually. Now, iterations commonly last two weeks or less. While delivery cycle time is decreasing, the technical complexity required to deliver a positive user experience and maintain a competitive edge is increasing.

In terms of software testing, this brings us to an inflection point. In most organizations, testers were already racing to keep pace when delivery cycles were longer and application complexity was lower. Every quarter or so, the development team would pass a release candidate over the wall to QA, who would then scramble to validate it as thoroughly as possible in the allotted time—largely with manual testing.

Now, digital transformation initiatives such as Agile and DevOps are pushing traditional methods to their breaking point. Organizations are releasing much more frequently—from monthly on the low end to multiple times per hour on the high end. And as organizations increasingly edge toward continuous delivery with automated delivery pipelines, intermediary quality gates and the ultimate go/no-go decisions will all hinge upon test results.

Is your organization’s testing process up to the task?

We Have a Testing Problem (Two, Actually)

In most organizations, testing delays application delivery while providing limited insight into whether the applications are meeting stakeholder expectations. It’s not fast enough to help teams find and fix defects when it’s optimal to do so. And it’s reporting on low-level failures (e.g., 78% of our tests passed) rather than providing the business-focused perspective needed to make fast release decisions (e.g., Only 38% of our business risk was tested … and 25% of that didn’t work properly).

Let’s take a quick look at each of these problems in turn.

The Speed Problem

DevOps is all about removing the barriers to delivering innovative software faster. Yet, while other aspects of the delivery process are streamlined and accelerated, testing consistently emerges as the greatest limiting factor.

A recent GitLab survey that targeted developers and engineers found that testing is responsible for more delays than any other part of the development process.

Where in the development process do you encounter the most delays?

The same conclusion was reached by a DevOps Review survey that polled a broader set of IT leaders across organizations practicing DevOps. Again, testing was cited as the No. 1 source of holdups in the software delivery process. In fact, it “won” by a rather wide margin here—63% reported that testing was a major source of delays, while the second-highest source of delays (planning) was cited by only 32% of the respondents.

Where are the main holdups in the software production process?

Why is testing such a formidable bottleneck? That could be the topic of an entire book. For now, let’s summarize some key points:
- The vast majority of testing (over 80%) is still performed manually—even more at large enterprise organizations.
- Approximately 67% of the test cases being built, maintained and executed are redundant and add no value to the effort.
- At organizations that have significant test automation, testers spend 17% of their time dealing with false positives and another 14% on additional maintenance tasks.
- Over half of testers spend five to 15 hours per week dealing with test data (average wait time for data = 2 weeks).
- 84% are routinely delayed by limited test environment access (average wait time for test environments = 32 days).
- The average regression test suite takes 16.5 days to execute, but the average Agile sprint is two weeks, from start to finish—including planning, implementation and testing.
- The average application under test now interacts with 52 dependent systems—which means that a single end-to-end transaction could cross everything from microservices and APIs to a variety of mobile and browser interfaces, packaged apps (SAP, Salesforce, Oracle, ServiceNow…), custom/legacy applications and mainframes.
The software testing process wasn’t working perfectly even before the advent of Agile and DevOps. Now we’re asking teams to “just speed it up” at the same time that modern application architectures are making the process even more complex. Given that, it’s hardly surprising that speed expectations aren’t being met.

The Insight Problem

Only 9% of companies perform formal risk assessments on their requirements/user stories. Most attempt to cover their top risks intuitively, and this results in an average business risk coverage of 40%. Would you feel comfortable driving a race car with blinders on? That’s essentially what you’re doing if you’re rapidly delivering software with insight into less than half of your total business risk.

Moreover, most organizations can’t immediately differentiate between a test failure for a trivial issue and a business-critical failure that must be addressed immediately. Most results look something like this:

What insight does that really provide? It’s clear that …
- There’s a total of 53,274 test cases.
- Almost 80% (42,278) passed.
- Over 19% failed.
- About 1% did not execute.
Maybe the test failures are related to some trivial functionality. Maybe they stem from the most critical functionality: the “engine” of your system. Or, maybe the most critical functionality was not even tested at all. Trying to track down this information would require tons of manual investigative work that yields delayed, often-inaccurate answers.

Today’s go/no-go decisions need to be made rapidly—even automatically and instantaneously. Results that focus on the number of test cases leave you with a huge blind spot that becomes absolutely critical—and incredibly dangerous—when you’re moving at the speed of digital business.

Closing the Gap

How do you evolve from the slow, burdensome testing that delivers questionable results to the lean, streamlined testing that provides the fast feedback required to accelerate innovation and delivery? That’s what I aim to explain in my new book, “Enterprise Continuous Testing: Transforming Testing for Agile and DevOps.” This book is targeted to senior quality managers and business executives who need to achieve the optimal balance between speed and quality when delivering the software that drives the modern business. It provides a road map for how to accelerate delivery with high confidence and low business risk.

— Wolfgang Platz
December 18, 2019