devopsy.kinsta.cloud

Tag: UI test automation

  • Make a Plan for Test Automation

    Make a Plan for Test Automation

    The amount of testing that we could be doing is massive. Most of us don’t look at testing across the spectrum and all-inclusively, but let’s do that for a second.

    We have functional testing at the code level, which is reasonably well automated already, if your shop is using such automation. Then we have integration testing, which is a superset collection of API testing, code integration testing, usability and regression testing. Integration testing is automated, and sometimes that’s done well, but this massive space is still growing. Next up would be security testing, which covers much of the same code that other testing is interested in from a strictly security focused perspective. Then, there is actual usability testing, which is more important for some areas than others—embedded needs far less usability testing than web apps, for example—and has different levels of automation at this point in time.

    Really, when you think about it, the complexity of this environment is massive. We not only need to test an array of different functionalities from an array of different perspectives, but we have to do it on an array of platforms. Testing on Windows is different than testing on Linux; testing in the cloud can use different tools than testing in-house; clients can be on any given platform, depending on the application and target market … we really do have a beast, here.

    And testers have always been behind, or have not covered things well. This is not a ding on testers, this is the result of the complexity at hand and the fact that, in many instances, one line of code can go wrong in a bunch of different ways.

    Enter automation, which is growing rapidly in the world of continuous integration (CI). Automated testing tools range from broad to very focused, and all help facilitate testing. The DevOps increase in development means more testing, and testing was already behind. This has been discussed pretty thoroughly in the past, but it finally feels like we’re getting to a point where we have viable solutions to the problem. If testers become test developers that review results of automated tests, and through CI those automated tests are increasingly worked into the process of development, we’ll see testers getting better coverage and quality seeing a continued increase.

    But that does mean you should keep your eyes on the market as it continues to grow, and pick tools that offer the most functionality that is most useful to your test team. And get them up and running. More testing means more work setting up tests. Even no-code test solutions for UI require someone to record the actions that are necessary to complete the test, and that takes someone’s time. But the return is long-term testing that doesn’t require a person until a red flag is raised by automation.

    I’m all for it. Both testing and security have the most to gain from automation, because they’ve been the most behind historically, and corner-cutting has cost organizations dearly. So, let’s get with it; check out some new test tools (there are plenty listware articles giving you a place to start), and let’s make great software better!

  • How to Automate Testing for Optimal Customer Experience

    How to Automate Testing for Optimal Customer Experience

    During a sudden surge in online traffic like the one many companies have seen over the past few months, automated testing can help organizations optimize the customer experience

    Imagine you manage an e-commerce office supply company and are dealing with a sudden surge in traffic—with more people working from home, demand for your products has grown significantly. But when a customer attempts to complete a purchase, they encounter a debilitating error such as a broken button or a page that won’t load. Now imagine this happening to hundreds of customers simultaneously, during the most critical time of day for purchases on your site. Imagine the revenue lost, the customers never to return, the complaints on social media—it’s a situation everyone wants to avoid.

    But this is a situation many will have to handle at some point. Web pages are large, complex beasts made up of different images, fonts, style sheets, local JavaScript code, asynchronous requests to the backend and more. It’s easy to forget how many scripts hold a website together. And any small change can have a butterfly effect on the overall page performance.

    What does the user actually see and experience on the website? And what happens if changes are made in the backend? Does the “Buy” button suddenly disappear? Is everything still working as it should? To understand how an application behaves with users, companies need to understand those users’ experiences when they interact with the application over a browser. All too often this means manually checking the availability and functionality of individual online applications, and then hoping that users do not stumble upon additional problems that may have fallen through the cracks.

    Accounting for and identifying these potential problems and regressions is not only an uncomfortable process, but also a lengthy one. Especially in times of increasing traffic and new digital business, as we’ve experienced in these times of WFH and e-commerce, the continuous review of all digital experiences is essential. A customer’s first impression is crucial, and even a long-term customer may go elsewhere if a page doesn’t load or if a button doesn’t work. Options abound on the internet—you can’t risk someone giving up on your business and trying a competitor’s.

    Why Automating Browser Tests Is Important

    In browser-based applications and websites, browser tests can monitor the user experience and ensure that user actions, such as registering for a new account or adding items to a shopping cart, work smoothly—even after changes in the backend.

    Popular browser test solutions have long since reduced the manual effort required to confirm that new features didn’t break already existing functions, but the implementation and maintenance of these tests is very time-consuming. In addition to learning a new test framework for these solutions, teams need to set up a test infrastructure and update faulty tests with every change to the user interface, costing valuable time that could be used for new product development.

    The latest generation of fully automated browser tests, on the other hand, allows everyone on the team to record and automate rich browser-based test scenarios in minutes—without any code. The “Tester” opens the corresponding website or application via the test tool and starts the recording. As she interacts with the website or application, the solution automatically records all actions as a series of steps, which can then be edited or changed when reviewing the recording. Assertions can also be added, such as whether a user sees a welcome message when they log in.

    If a test is recorded, it is possible to define how often and on which devices it is to be carried out and from which places around the globe the tests are to be played to mimic users’ behavior. The recordings can be played over a variety of websites, including internal, non-publicly accessible applications or pre-production and CI environments. For each test run performed, the end-to-end duration of the test and a visualization of the duration of each step are provided. A screenshot is automatically created for each step and each query so testers can follow exactly what the user sees in each respective step. All the frontend errors and resources happening along the test are also collected for a simplified troubleshooting of failing test runs.

    Because websites are constantly modified by developers, once a test is created, machine learning algorithms detect changes to the application and automatically update the tests accordingly. If a browser test fails due to frontend or backend problems, an automated tool provides the context for quick troubleshooting. In addition to screenshots, frontend errors and frontend resources, the behavior of the backend services and infrastructure for this specific step is also provided.

    Using Time More Effectively

    Automated browser tests have traditionally been notoriously fragile, often breaking with minor changes to the user interface. To eliminate false positives, today’s ML-supported browser tests mimic human decision-making processes and adapt intelligently to changes in the application.

    For example, if a button is moved in a web application or the identifier of an element changes, the browser test derives from this how the existing test should be carried out in the updated user interface. If all steps and assertions in the test are successfully completed, the saved identification mechanism and test definition will be updated to reflect the changes on the page.

    If you’re always in a reactive mode, putting out fires that could have been avoided, the impact to revenue and customer satisfaction is only one half of the story. The revenue not generated, and customers not acquired, from improvements and new features, is the real problem for a business’ long-term health. Ultimately, browser tests give teams more freedom to focus on developing new features rather than manually reviewing functionality or redefining test frameworks over and over again. And in a distributed, work-from-home, always-online world, new product development is what wins the day.

  • Quality at Speed During the COVID-19 Pandemic

    Quality at Speed During the COVID-19 Pandemic

    Think back to 2015. The U.S. economy was in the early phases of a growth spurt that would continue for years. Ninety-five percent of organizations were practicing agile, to some extent (State of Agile, VersionOne). And the proportion of the IT budget allocated to QA and testing had just increased from 18% to 26% (World Quality Report, Capgemini). 

    “Quality at speed” became the catchphrase of the year in the testing world. With all the buzz around DevOps and agile, it was clear that speed mattered. In reality, most teams weren’t quite there yet. Agile adoption was still low (less than 50% of teams within each organization actually practiced it) and maturity was even lower (only 17% of those practicing it reported mature adoptions). Still, the writing on the wall was clear. Leaders wanted more application development, faster — so testers should at least prepare for the pending storm.  

    From that point forward, organizations across almost every industry all progressed to a sweet spot where time-to-market pressure was relatively high and cost pressure was moderate. By the end of 2019, quality at speed had graduated from buzzword to business as usual at most organizations. 

    But then came COVID-19. Now, we’re in the midst of a global pandemic. Every business, every non-profit and every governmental agency is impacted — but in different ways. 

    Speed expectations suddenly shifted into overdrive for an unlikely assortment of organizations, including government agencies and non-profits, healthcare and pharmaceutical companies, online retailers, online conferencing providers and telecoms. Yes, everyone had already been working to accelerate their software delivery processes. And many enterprise organizations had made great strides toward scaling the adoption of agile and DevOps. But, very few of these organizations were previously delivering applications at the extreme speeds required right now.

    For another group, cost reduction suddenly became the top priority. Airlines, hotels, as well as other travel and leisure organizations experienced such dramatic changes in fortune that speed became a non-issue. The focus shifted to increasing cost efficiency in an effort to offset some of these staggering losses. 

    Assuming that high quality cannot be compromised, this leaves both of these groups with distinctly different — yet equally daunting — challenges from a quality perspective. 

    The organizations in the first group suddenly needed to build, test and release applications at the speed of a startup — but with the added burden of legacy systems, complex application landscapes and intense regulatory scrutiny. Quality at speed remains the goal, but the “speed” part is much, much faster. 

    The organizations in the second group need to determine how to continue meeting strict quality expectations with a significantly slimmer testing budget. Quality at speed isn’t so important right now. Speed gets replaced by the other corner of the classic cost-quality-speed triangle, making the new mantra “quality at cost.” 

    It’s hard to find good news in the current situation. But, from the software testing perspective at least, there’s some consolation. Whether you’re facing unprecedented time-to-market pressure or extreme cost pressure, the same core “test efficiency” strategies will help you either way. 

    Here’s a quick overview of ways you can test more efficiently at different steps of the software testing process.

    Selecting and Designing Test Cases

    I’m addressing these two steps together because they both face a common challenge: intuition. Most modern business systems are insanely complex — so complex that it’s just not feasible for most testers to intuitively identify the best set of tests to cover a specific test need. Having a more systematic way to find the smallest — yet most powerful — set of tests will save you both time and money. 

    For designing test cases for progression testing, methodologies like linear expansion and pairwise testing can help, and augmented intelligence can dramatically simplify them for you. 

    For selecting what test cases should be run for regression testing, consider change impact analysis. Some approaches identify which objects are impacted by a given change, then help you narrow the test scope to the impacted objects. This often reduces the testing scope by 15%. Not bad, but you can do much better.

    AI-powered change impact analysis pinpoints exactly what needs to be tested in a given update, based on whether the changes put an object at risk. With this approach, you can reduce testing efforts by 85% or more. Not only is this approach faster and cheaper, it’s much more efficient at uncovering defects. The number of defects that hit production are reduced by 75% to 95%, sometimes even 100% (no defects in production).

    Automation and Execution

    Here, technical scripting is the top time and cost drain. Some level of maintenance is required to keep scripts running, and this maintenance will cut into your test efficiency. One way to combat this challenge is to take a model-based approach to test automation. Such approaches have been field-tested for years, and proven to solve the test maintenance nightmare that’s doomed all-too-many test automation initiatives. 

    The new kid on the block is AI-driven UI test automation. This is a big topic that warrants a much deeper discussion — but here’s a quick overview. Specify tests in natural language. Build the tests even before the actual UI is ready. Use the same tests as the technical implementation changes. Understand dynamic behavior like a human tester would. Test absolutely any technology — faster than a human eye. Curious? Stay tuned. 

    Analysis of Results

    When it comes to results analysis, false positives are the vampire sucking your resources. False positives are tests that fail even though the application is behaving correctly.  They will really kill you in terms of cost and speed. First, someone needs to figure out why the test failed. This might involve manually checking the related application functionality, inspecting the test automation, reviewing the test data, checking whether every element of the test environment is up and working as expected — maybe even all of the above. Once the exact problem is found, it needs to be fixed. Then, the test needs to be rerun. If it still fails, rinse and repeat.

    How do you stop false positives? To start, stabilize the automation with model-based test automation or the AI-driven testing I mentioned in the previous section. As you move into more advanced automation scenarios, stateful test data management and service virtualization can help you stabilize tests to the point that they’re fit for CI/CD. Then, your final challenge is to deal with the flaky tests resulting from things like asynchronous application behavior, concurrency issues and test order dependencies. Again, AI and ML can offer tremendous assistance here—in flagging potential false positives, and even in helping you identify the cause and optimal resolution. 

    Final Thoughts

    COVID-19 split up the original quality at speed sweet spot into the extreme speed camp and the extreme cost-savings camp. In both cases, the same core strategies to increase your test efficiency will help you increase speed as well as reduce testing costs. If you’re under extreme time pressure, showing that you also reduced costs while meeting speed expectations certainly won’t hurt. If you’re currently reducing costs because your industry is essentially paused, taking these steps now will prepare you to speed it back up when the world becomes more normal again.