devopsy.kinsta.cloud

Tag: continuous testing

  • 5 Unique Challenges of Mobile App Testing

    5 Unique Challenges of Mobile App Testing

    At first glance, testing a mobile app may not seem to be very different from testing a conventional desktop app. Mobile and desktop apps are often written in the same languages and hosted on the same servers. They must also meet the same basic user expectations in areas like loading speed and accessibility.

    But when you dive into the details, you realize that mobile apps are fundamentally different beasts from desktop apps–and that, by extension, mobile testing requires a unique approach. You can’t simply take a software testing strategy that works for desktop apps and graft it onto your mobile apps.

    This article breaks down five of the main reasons why mobile testing necessitates a different strategy from desktop application testing and which unique needs QA engineers should consider when testing apps.

    1. Mobile Configuration Variability

    Perhaps the biggest difference between mobile and desktop testing is that with mobile apps, engineers have a much wider set of possible configurations to contend with.

    In the desktop world, there are just two main operating systems – Windows and macOS – and a relatively small number of OS versions for each family. And although there are many types of PCs and laptops, they all conform to the same basic hardware standards.

    In contrast, there are over 24,000 distinct types of Android mobile devices out there – to say nothing of iOS devices, which add to the diversity of hardware. There is also a wider selection of mobile operating systems and versions.

    What this means for QA teams is that there are many more variables to test for with mobile devices. It also means testing must be more efficient so that engineers can test for as many potential configurations as possible without delaying software release cycles.

    2. Lack of Mobile Testing Standards

    With a traditional, web-based desktop app, there is a consistent set of standards that the app is supposed to comply with when rendering content – specifically, the standards set by the W3C, a consortium that advocates for a standards-based, interoperable World Wide Web.

    In the mobile realm, however, there is nothing equivalent to the W3C standards. Apps can render content in widely varying ways, many of which are device-specific.

    Here again, this increases the need for specific testing teams to account for more variations and edge cases. With desktop apps, ensuring that the app complies with the W3C standards is often sufficient, but mobile testing is not so simple.

    3. Unique Mobile Accessibility Requirements

    Accessibility testing, which ensures that accessibility features like the ability to increase text size function properly, is important for delivering a great experience to all users, whether they access apps using desktops or mobile devices.

    But with mobile devices, accessibility testing is harder, because there is more room for error when implementing accessibility features. For instance, the smaller screen sizes of devices – and the greater variation in average screen size – could mean that an increase in text size will cause the app to render some text off the screen. Or, a “nighttime mode” feature on a screen could result in lower-than-expected contrast between text and backgrounds, causing accessibility challenges for some users.

    A mobile testing strategy needs to be able to accommodate risks like these, which aren’t as pronounced for desktop apps.

    4. Mobile Environment Differences

    By definition, mobile devices may be used in a wide range of physical settings. Depending on the location from which a user accesses a mobile app, application performance could be impacted by a variety of environmental factors that don’t typically apply for desktop apps.

    For instance, limited network connectivity might undercut application performance when users who travel too far from cell towers. Or, energy-saving features on mobile devices that are running on low battery could decrease the speed at which apps render content.

    Once again, these factors create additional risks that QA teams need to address when planning testing routines.

    5. Higher Stakes for Mobile Testing

    It’s a best practice to strive to deliver the best possible experience to all users, whether they are on desktop or not. But the reality is that poor user experience in mobile apps tends to have a more negative impact on a company’s brand.

    The reason why is that it’s easy for users to highlight poor app performance by giving apps low ratings or leaving negative comments in marketplaces. They can’t do these things for most desktop apps, because, unlike mobile apps, most desktop apps are not downloaded through centralized marketplaces with user-rating features.

    This difference doesn’t make mobile testing more technically challenging, and to be clear, I’m not saying you can skimp on desktop testing because users will abandon your desktop apps if they don’t work as they should, but it does increase the stakes of perfecting a testing strategy. Businesses simply have more to lose, reputationally speaking, from delivering buggy apps than they do from low-quality desktop apps.

    Conclusion: The Need for Purpose-Built Testing Solutions

    All of the above is why we need more testing strategies and solutions that are designed specifically for mobile.

    Historically, QA teams have frequently tried to extend desktop testing strategies to address mobile as they’ve added apps to their catalogs, but that approach simply doesn’t work. Mobile apps and devices are too different in fundamental respects to be shoehorned into a desktop-centric testing routine. The sooner QA engineers realize this, the sooner they’ll optimize their user experiences.

  • Let the Machines Do It: AI-Directed Mobile App Testing

    Let the Machines Do It: AI-Directed Mobile App Testing

    AI has been in the news a lot lately. We’ve all heard a lot about ChatGPT and you’ve no doubt encountered the end result of a variety of other advances in AI technology of late, such as Lensa. But there’s also a much quieter and less visible shift happening behind the scenes during the development of your favorite apps—AI-directed mobile app testing. Let’s take a look.

    Behind the Scenes

    It’s no secret that every product, no matter the industry, goes through a gauntlet of tests before it’s unleashed on the general public. This goes for everything from a simple object like a cardboard box, for example, to a far more complex product, such as the latest electric vehicle.

    The same goes for the mobile apps on your phone. And if you haven’t given that much thought, then good! It means that the QA team behind the app’s functionality did its job, producing a seamless experience that functioned without a hitch, exactly as you wanted and expected.

    But that’s not always the way it goes, and mobile app testing is only growing more complex. That’s a growing problem that AI will eventually resolve.

    If your app didn’t perform as expected, you—and many other users like you—may delete it and move on to the next app that promises to do the same thing. While this may not seem like a big deal to you as the user, it’s a total disaster for the tireless teams working behind the scenes to ensure a successful app launch or update.

    The adage ‘You only get one chance to make a first impression’ is very true in the mobile app market. It’s a big, crowded and cutthroat mobile app world out there, and users simply have no incentive to put up with an app that doesn’t do exactly what it’s supposed to do. After all, for every single app, there are often several others standing by to scoop up your unhappy (former) users.

    This issue is by no means restricted to small development teams strapped for resources. I’m sure you can think of some high-profile software releases from major companies that have launched in a disastrous state over the past few years (particularly in the tremendously profitable arena of video games).

    So why is it that some teams cut corners when it’s time for testing, only to dearly wish they hadn’t?

    Well, the answer is pretty simple: Up until very recently, mobile app testing has traditionally been time-consuming, complex and bewildering, all too frequently delaying releases and flabbergasting QA teams.

    To understand this ongoing problem, let’s take a look at what this kind of mobile app testing entails and how this landscape has recently started to shift, as well as what’s on the horizon to make it all more efficient.

    Cutting Corners = Consequences

    Once upon a time—in an era that seems like it was really just yesterday—software testing was performed manually, entirely by hand. This was possible because quality assurance personnel only needed to consider a very limited set of variables: There were only so many use cases and form factors to consider, and programs were only expected to do so much.

    It’s hard to imagine testing a modern app manually and having the result be effective. The approach is only reasonable under very limited circumstances. Nowadays, teams use automated tests, which save them a tremendous amount of time, effort and resources.

    Yet these automated tests come with their own problems. Most of these automated tests are hand-coded, meaning that they must be set up and adjusted by specialists with a background in computer programming. Such specialists are not only costly and in demand, but they’re also needed for later aspects of the software development cycle, such as developing new product features and overseeing other aspects of production.

    And this is where companies start to cut corners and miss problems and issues, both large and small. Instead of what is known as shifting left (that is, testing as early in the cycle as possible), many teams, unfortunately, view testing as a burden or afterthought. Only when things go haywire and they have a full-blown crisis on their hands does the need for rigorous app testing become apparent.

    Testing, Machine-Made

    This is where machine learning and artificial intelligence enter the picture. For several years, we’ve known that testing performed by machines will be more and more necessary to handle increasingly complex apps, and it won’t be long before the cost associated with the complex testing scenarios will simply become too much for many developers.

    Fortunately, advances continue to appear on the horizon. These include the much-discussed no-code revolution, which abstracts away much of the noise associated with software development and opens the door to a much wider testing base.

    AI- and machine-learning-based testing is also on the horizon, yet there’s a significant challenge that must be overcome before this can happen at speed and scale. To develop mobile app tests with true intelligence and maximum efficiency, machines must be capable of autonomously completing the following steps:

    1. Comprehension: Draw from the ability to understand what’s changing in an app. For example, the machine must be able to detect differences and issues in a signup sequence or a checkout screen change.
    2. Impact analysis: The machine must know what test cases can cover that change. This sort of identification can be quite tricky.
    3. Test case generation: Next, the machine must possess the ability to generate test cases related to steps one and two. To do this, the machine must draw from several data sets. These include:
      1. Extent test collateral/test cases with product code
      2. Data from actual customer usage so that numbers one and two above may be identified
      3. What user environments look like, including information like how many users are active in the United States and in Europe
      4. Automatic execution: Perhaps the most crucial need is that the machine can perform testing entirely by itself. With the ability to self-execute tests, we’ll have reached a new era in testing.
      5. Reports: Before the testing cycle can begin again, the machine must be able to produce detailed reports that make sense to its human handlers. When it comes to test results, the more detail the machine can provide in as approachable a manner as possible, the better.

    Problems and Solutions

    Some testing platforms are now capable of many of the tasks I highlight above, and quality assurance teams that move away from self-coding tests and toward the future of testing stand to benefit from these solutions (particularly when their competitors do not).

    Partially enabled by AI and ML, no-code solutions provide a stopgap solution until machines can really get a handle on the above five steps. At this point, it’s really just a matter of time: We can already see glimpses of a wholly automated testing cycle where human input is limited and machines do the heavy lifting.

    It’s easier than ever to imagine a world where there’s no need for testing to be an afterthought and software disasters have become a thing of the past.

  • Shift Left Testing in Microservices Environments

    Shift Left Testing in Microservices Environments

    By now, it’s common knowledge that the later a bug is detected in the software development life cycle (SDLC), the longer it takes and the more expensive it is to fix that bug.

    In 2017, the Ponemon Institute found that it cost around $80 on average to fix a defect detected early in the SDLC and around $7,600 to fix the same defect if it was detected in the production stage.

    Relative Cost of Fixing a Defect Based on Development Stage

    Design

    Implementation

    Testing

    Maintenance

    Cost of Fixing Defects

    1x

    6.5x

    15x

    100x

    While these numbers can be debated, it’s clear that early detection of defects can save organizations significant time and money while enabling those organizations to deliver higher quality software more rapidly. For these reasons, many organizations are implementing shift left testing practices, including in microservices environments.

    Shift left is a set of practices intended to find and prevent defects early in the software delivery process instead of during the testing and maintenance stages. The idea behind shift left testing is to improve software quality by making actionable test signals and feedback available as early in the SDLC as possible.

    Here, we’ll highlight how shift left testing, in combination with test automation, can better enable both innovation and fast delivery of high-quality software. We’ll also address some of the challenges involved in implementing shift left testing in microservices environments, as well as introduce a valuable strategy for successful microservices implementation.

    Shift Left Testing Benefits

    The waterfall model of development involved the explicit passing of responsibilities between highly specialized design, development, QA, and release teams. It also involved lengthy feedback loops.

    Scrum and agile methodologies made the entire SDLC more flexible and nimble by introducing sprints and allowing more frequent iterative development and delivery. Further, DevOps and DevSecOps focus on removing the silos between development, operations, and security through tooling and automation. As a result, the time to market and quality have improved dramatically. Adding shift left testing into the mix better positions teams to handle the broad range of responsibilities from the design stage through the maintenance stage as effectively as possible.

    Shift left testing focuses on prevention rather than detection. Shift left benefits include the following:

    • Increase efficiency by eliminating bugs earlier in the SDLC.
    • Reduce human errors and associated costs.
    • Increase delivery speed and reduce the time between releases.
    • Improve the quality of software.
    • Gain a competitive advantage.

    It may be helpful to think of shift left testing like this: Test early and in parallel with development, instead of the traditional waterfall model of development followed by QA testing. 

    Shift Left Testing Challenges in a Microservices Environment

    By structuring an application as a collection of loosely coupled, independently deployable services, a microservices architecture enables frequent delivery of large, complex applications.

    While shift left testing practices bring many benefits, shift left testing also comes with three challenges that organizations must address when implementing it across a decentralized, microservices-based stack:

    1. Building and managing each microservice separately reduces development time. However, implementing shift left testing at the microservices level means shift left practices must be applied to each individual microservice.
    2. Microservices developers can choose the programming language and service dependencies best suited to their development needs. However, as a result, shift left testing practices need to be implemented and managed across different programming languages and dependencies.
    3. To ensure that microservices function as expected, teams usually run integration tests, end-to-end tests, and performance tests that assess functionality in the context of the larger stack. Shift left testing is especially challenging in the microservices context because of the complexity of testing against the stack.

    Microservices, each with their own SDLC pipeline

    Pipeline Scaling Considerations

    Several classes of automated shift left tests can be integrated into the DevOps pipeline, including unit tests, integration tests, performance tests, and security tests. Different microservices will have different testing needs.

    The key is to apply shift left testing practices to each microservice while maintaining the flexibility and decentralization of those microservices. The ideal workflow system and DevOps pipeline implementation must simplify pipeline creation while enabling development teams to manage the pipeline and requiring minimal input from DevOps and platform teams. In other words, shift left practices must be tailored to the specific needs of the microservice in question to make the testing process scalable and easy for development teams to manage on a continuous basis.

    As just one example, GitHub Actions and similar CI/CD pipeline tools enable development teams to use GitOps-based practices to configure, manage, and administer the entire pipeline for their microservice while leaving infrastructure testing and testing that relies on or affects other parts of the system to DevOps teams.

    Test Environment Considerations

    Test environments are environments where unit tests, integration tests, performance tests and security tests can be run on an isolated collection of microservices and configurations. The nature and quality of these test environments directly correlate with the effectiveness of shift left testing.

    A cloned production or staging environment can be used for testing when the number of microservices is low and overall complexity is relatively low. However, once an organization reaches a threshold number of microservices or a threshold level of complexity, runaway infrastructure costs and operational burdens make such environments untenable.

    In response, some organizations rely upon smoke testing (minimal viable stability testing) and mock-based integration testing early in the SDLC. While such tests can prove useful in the near term, deferring essential testing to a later stage in the SDLC often introduces test signal and validity problems in the staging environment and can lead to delivery delays and serious issues with software quality.

    So how can shift left testing be used successfully in a microservices environment? The solution involves the use of scalable, ephemeral test environments.

    Ephemeral Shift Left Testing Environments

    The availability of high-quality test environments is a common bottleneck for organizations looking to scale shift left testing practices. Using a few high-fidelity staging environments for signal testing can slow the release process. More to the point, it tends to slow down the entire SDLC and increase stack instability over time.

    Ephemeral test environments offer a fundamentally different approach. These sandbox environments use isolated environments that enable users to test independently without impacting other users. In particular, they enable isolation while sharing resources (e.g., external cloud dependencies and hosted databases) using application-level multitenancy instead of the resource duplication approach in traditional environments.

    The idea is that microservice test environments can be used to test changes against a much larger stack of services. Because multiple tests can be conducted at the same time, test coverage and resource efficiency vastly increase.

    High-quality sandbox environments can spin up in seconds and scale to tens of thousands of microservices. In fact, each pull request, branch, or commit can have its own test environment without significantly increasing infrastructure and operational costs. Any change, no matter how small, can be tested in isolation to ensure performance and stability against the entire stack.

    Shift Left Testing Solutions for Microservices

    A microservices-based organization looking to scale shift left testing across the organization must first address several challenges. Considerations include the need to apply testing practices to separate microservices, the need to accommodate different programming languages and service dependencies and the need to test against the stack.

    The right tooling and automation are needed before shift left testing practices are scaled across the organization. In addition, a cultural shift may be required in order to help developers understand the ways in which shift left testing can expedite and improve the core functionality that they are developing.

    To apply valuable shift left testing practices in a microservices environment, organizations should consider using ephemeral sandbox environments. Using the right set of tools in a sandbox environment can enable development teams to test microservices throughout the SDLC. In doing so, organizations can ensure those services follow security best practices and perform exactly as intended, both independently and against the stack–all while reducing costs, speeding time to market and improving software quality and user satisfaction.

  • Developer’s Guide to Fuzz Testing

    Developer’s Guide to Fuzz Testing

    Fuzz testing continues to grow in popularity among developers within the open source community. Google’s open source security (OSS) team recently reported finding more than 40,000 bugs in 650 open source projects through the use of fuzz testing. But if you have no clue what fuzz testing is (yet), don’t worry. You’ve come to the right place.

    What is Fuzz Testing?

    Fuzz testing is a dynamic testing method for finding bugs and security issues in software. During a fuzz test, a program or a function under test gets executed with invalid, unexpected or random inputs to uncover unlikely or unexpected edge cases. 

    Value of Fuzzing for DevSecOps 

    As we know, DevSecOps extends the approaches of DevOps and Agile by adding suitable security testing methods alongside every phase of the software development life cycle (SDLC), creating a dynamic and continuous testing process. 

    Part of that testing process can and should involve fuzzing to detect security and stability issues at all phases of the SDLC and empower developers to ship software quickly and securely. For example, to prevent regressions from going live by running automated fuzz tests on every code change.

    Especially in the automotive industry and complex embedded projects, where a lot of untrusted and complicated data gets processed, fuzzing brings excellent benefits to DevSecOps. Indeed, developers can apply fuzzing similar to unit testing to automatically test their security-critical modules for potential vulnerabilities. But with fuzzing on an integration level, they can also cover all the complicated and dangerous edge cases that can occur in the interaction between different modules. 

    Fuzz Testing Benefits

    Fuzzing code generates thousands of automated test cases in a second, with the fuzzer receiving in-depth feedback on code coverage and the program states reached to simulate the interaction of different modules during the execution of the source code. The fuzzer can adapt and mute its inputs based on runtime feedback in the following iterations to maximize code coverage. Some of the benefits are:

    1. (Almost) no false positives: If you find a crash, fuzzing provides you with the error source and the malicious input that caused the issue. So, you can be sure that each finding is a real vulnerability.
    2. Debug reproducible findings: Modern fuzz testing tools will provide you with the input causing the issue, including the source of the error. With this, you already have a lot of helpful information to reproduce the bug and load it directly into the debugger. 
    3. Find issues in forgotten edge cases: Because fuzzing is executing a program with invalid or random input, it also uncovers unlikely and unexpected edge case errors that would not otherwise have been revealed.

    Who Uses Fuzz Testing?

    As cybersecurity regulations and standards continue to expand globally, more and more organizations are extending cybersecurity programs and running automated security testing before they ship software. Whole industries, especially those that involve advanced quality and security regulations, are setting standards for security testing that recommend fuzz testing, including automotive, aviation, finance, healthcare, telco and energy.

    Digging deeper into the automotive industry, for example, a variety of International Organization for Standardization (ISO) and other organizational standards now recommend fuzzing:

    Best Practices

    The most effective way to do security testing, fuzzing included, is to do so continuously. This means integrating fuzzing right into your CI/CD pipeline. Doing so creates quick feedback cycles so developers can fix security vulnerabilities before the code is shipped. Integrating fuzzing tools into code hosting systems like GitHub is also valuable. Doing so allows communication and alerts to be shared across your DevOps team as soon as a problem is uncovered, which helps to facilitate fixing vulnerabilities. 

    What Bugs Can I Find?

    While the practice remains relatively new, fuzzing has already discovered thousands of bugs across different use cases. Bug and vulnerability types discovered include:

    • Remote code executions
    • Injections, which can be triggered by untrusted inputs 
    • Memory leaks: Incorrect memory allocation 
    • Exposure of sensitive data: Accidental exposure of personal data
    • Functional bugs: Inputs that are not being responded to 

    and many more bugs, such as undefined behavior, uncaught exceptions and buffer overflow. Learn more.

    Getting Started

    You can try out an open source fuzzer, like Jazzer (for Java testing) or the CI Fuzz CLI (for C/C++ testing). When you’re comfortable with fuzzing and want to use this approach in more complex environments, enterprise solutions offer added features like API fuzzing, reporting and CI/CD integration. 

    So give it a shot and join the community of open source developers using fuzzing to quickly and securely ship their software. 

  • Applying TestOps to Cultivate a Quality Culture

    Applying TestOps to Cultivate a Quality Culture

    Quality assurance is a universal concern, especially when it comes to application development. Enterprise applications are faster and more reliable than ever before. However, in addition to performance, application developers also must consider data security, regulatory compliance and cost-effectiveness, among other things. Organizations are embracing quality engineering (QE) as a foundation to create a software quality culture to address these challenges. Creating a quality engineering or TestOps environment is not simple, especially when QE departments must test for hundreds of applications across multiple enterprise environments. Whether it is a solution provider or a multinational corporation, applications often need to be tested across platforms, industries and geographies.

    TestOps encompasses a broader concept than just QE. With TestOps, there are testers and engineers using automated and manual testing to validate software for flawless application performance. In enterprise DevOps, the TestOps team serves as the bridge between development and deployment since they have a good understanding of the applications and a better perspective on the context and variables that can affect performance.

    Continuous testing is an integral part of any software development cycle. Quality engineers must be brought in early as part of Agile development to keep development on track and head off any problems that could escalate or become harder to resolve later. In Agile, testing becomes more crucial and frequent to ensure flawless performance.

    It pays to think horizontally when setting up a TestOps approach. For example, testing should be performed across different vertical environments such as media, manufacturing and finance. The goal is to create a test environment that can accommodate various industries.

    TestOps Across Environments

    In the age of SaaS and universal data access, it is essential to test across multiple platforms. In addition to ensuring enterprise performance, mobile web and remote data access must be tested. For streaming services, it is vital to test across mobile platforms, over-the-top (OTT) platforms such as Roku, Apple TV, Android TV and Fire TV, and even gaming consoles like PlayStation and Xbox.

    When shipping software updates, geography matters (especially when shipping across time zones), as one may never know when the software will be used or have control over the environment where they plan to use it.

    For instance, consider the challenges a leading mortgage lender faced with mobile application testing and how geography was a factor. First, Google Analytics was used to assess specific device and user requirements and ascertain performance in various parts of the United States since different regions use different carriers. Testing was then done across mobile platforms such as iOS, Android and combinations and across carriers since most banking and mortgage applications required one-time passwords (OTPs) and SMS-related capabilities. Further tests were implemented for new identity authentication strategies such as biometrics and facial recognition. Finally, performance testing by geography was conducted to determine whether a new feature or version would create issues.

    Testing for Geographies and Regional Compliance

    Localization testing is important since every region and every country has unique compliance issues and laws. To ensure proper localization, it is essential to perform complete testing, including lower environments and production systems.

    In any TestOps environment, the philosophy of continuous testing should already be part of the design environment, including understanding the nuances of production. Release readiness does not mean there are only a few known bugs in certain regions. It means the application has been tested for compliance in every geography where it could possibly be utilized.

    Testing Across Industries

    Testing application performance in specific industry settings brings along its own set of unique challenges. In manufacturing, for example, each sensor in an IoT device must be modeled against a conformance specification to determine if it can be automated using simulations. In media, one needs to test for different platforms, content management systems and business workflows.

    To accelerate time to market, one would want to support a dual-shift development lifecycle to gauge the consumer experience and get continuous feedback and also need a test environment with multiple automation tools such as Selenium, Appium, Cypress, Tricentis, UiPath, Test Complete and Winium.

    Artificial intelligence (AI) plays a larger role in testing, shortening time to market, and ensuring superior quality software. AI-driven test tools are optimized for domain-specific applications, such as testing for industries like media, manufacturing and fintech. AI can fill in the gaps to achieve optimization and offer a quicker turnaround with limited manual intervention.

    Any organization cultivating a TestOps culture needs to have specific technologies and tools in place. When assessing your testing technology stack, it is important to create an end-to-end testing ecosystem. This means the environments must be inclusive and capable of incorporating testing for unique applications, environments and scenarios. To support the needs of today’s Agile application development environment, one must plan for an agile testing infrastructure to keep development on track.

  • Survey Warns of Looming Software Testing Crisis

    Survey Warns of Looming Software Testing Crisis

    A survey of CEOs and IT professionals involved in application testing finds a significant gap in terms of how acceptable it is to release software that has not been properly tested.

    The survey, conducted by the market research firm Censuswide on behalf of Leapwork, a test automation platform provider, polled 480 CEOs in the U.S. and the United Kingdom (UK) from organizations with more than 500 employees and 480 heads of quality assurance, testers, release and delivery managers and site reliability engineers (SREs).

    A full 85% of the CEOs in the U.S. and 69% in the UK said it’s acceptable to release software that hasn’t been properly tested, as long as patches are tested and applied later. Less than half of the IT professionals surveyed (49%) agreed. A full 94% of CEOs also said they’re confident the software released is tested regularly.

    However, 79% of the IT professionals admitted that as much as 40% of software goes to market without sufficient testing. As a result, 68% of UK and 52% of U.S. IT professionals said their teams spend five to 10 days per year patching software.

    The survey also made it clear that everyone involved is well aware of the risks that lack of testing creates. Nearly every CEO (95%) is concerned about losing their job in the wake of a software failure. Slightly more than three-quarters of the IT professionals (76%) surveyed have the same fear.

    More than three-quarters of CEOs (77%) also noted that software failures have harmed their company’s reputation in the last five years.

    Many organizations today assume that Agile development and DevOps best practices will enable them to apply patches quickly in the event an issue arises. The trouble is, in the absence of sufficient testing many of those organizations are essentially experimenting on end users.

    Leapwork CEO Christian Brink Frederiksen said the primary reason so much software is not being properly tested is that most organizations continue to rely on manual testing processes that can’t keep pace with the rate at which applications are now being developed; one major factor is the use of low-code and no-code tools.

    Of the IT professionals at organizations that release insufficiently-tested software, a full 40% cited underinvestment in test automation as a factor, with only 43% of respondents using some type of automation. Other reasons cited for the lack of appropriate testing included lack of time (34%) and increased rates of deployment (29%).

    Just over a third of CEOs (34%) and about four in 10 testers (42%) whose companies use or develop in-house software that goes to market without being tested also cited a lack of available skilled developers. More than a third of CEOs (34%) also cited underinvestment in testing personnel.

    At a time when more organizations than ever are dependent on software to drive digital business transformation initiatives, it’s apparent that an application testing crisis is imminent. The only thing left to be determined is how catastrophic the fallout will be because existing processes don’t scale to meet the challenge.

  • How Automated Testing Closes the Development and Delivery Gap

    How Automated Testing Closes the Development and Delivery Gap

    For the past two years, companies have leaned into digital transformation projects to keep their digital offerings at pace with their in-person operations. Rather than become a stopgap, this digital approach became the new standard for the modern employee and customer experience. It also creates friction between a company’s development team and its user base. How long should customers wait for new or updated applications in our increasingly fast-paced business environment? Companies need time for development and testing to deliver quality products, but will the demands of their users force them to cut corners?

    Consider each stage of a customer’s journey. Visiting a website or app and putting something into the shopping cart is only the beginning. Behind the scenes, systems that monitor availability, sourcing, loyalty and promotions come into play. Following checkout and payment, there are systems for picking the product, packing it and delivering it with timely notifications along the way.

    Businesses find themselves in a bind when they have to decide which of those areas to cut back on. There’s really no good answer—enter the continuous delivery mindset. We like to say there is no continuous delivery without continuous testing, but those two areas are often at odds with each other.

    The good news is that there is a way to close the gap between customer demand and necessary development time: Test automation. Here are three ways that test automation can help companies satisfy both the needs of their business and their customers.

    Relieving the Testing Bottleneck

    Moving fast is becoming a competitive advantage for many companies. That speed-to-market opportunity has been heightened as cloud-based, low-code/no-code apps like Salesforce, MS Dynamics, Workday and Oracle have become an industry standard. Today, while plenty of development activities may be handled by low-code/no-code platforms, the test automation of these apps remains a high-code, high-maintenance activity.

    To keep from becoming a bottleneck, teams should plan automation and testing strategies early in the process. This is where continuous development includes continuous testing. Suppose companies can make test automation accessible to more people by removing the need for specialized knowledge and daunting technical constraints. In that case, they will be able to streamline ongoing development cycles.

    Ensuring Higher-Quality Output; Aligning Development and Testing

    One of the keys to putting out a higher-quality product is by adopting in-sprint testing, which aligns development and testing efforts. Each feature is tested and automated as soon as it is implemented so that teams can more easily catch errors early rather than at the end of a development phase.

    The automation also elevates the area of regression testing, as teams make sure new changes haven’t broken existing functionality. As automation moves into the mainstream, a wider range of collaborators become more involved in testing each additional feature.

    Creating a Better User Experience

    Historically, testing has been extremely focused on function and is very technical while user experience efforts remain focused on the stylistic and usability end of the spectrum. But creating satisfying end-to-end user experiences is a critical part of any application adoption, so the two sides should work toward a common goal.

    That common link lies in a company’s business processes. Each process is the starting point for building and designing a test scenario. From start to finish, everyone understands the user’s journey and works together toward making that path as effortless as possible.

    Conclusion

    Today’s modern customers don’t expect a measurable difference in experience between online and in-person. Expectations are at an all-time high, leading companies to push updates to their websites and apps more quickly and more frequently. As companies increase their dependence on software, the way to test that software is evolving to keep pace with development speeds.

    Automated testing can simplify and speed up the testing phase, catching many of the basic errors and leaving testers to focus on more complex functionality.

     

  • What Is User Acceptance Testing and Why Is it so Important?

    What Is User Acceptance Testing and Why Is it so Important?

    Anyone involved in software development knows how important it is to ensure a positive end-user experience. One way to do that is through the use of user acceptance testing (UAT). 

    UAT is one of the types of testing that you can execute as part of your overall quality assurance process. 

    Source

    It’s essential in helping you determine if your product is market-ready and if it can perform effectively when used by real end-users. 

    UAT helps you avoid implementing software, iterations, upgrades, or customizations that are filled with pesky bugs that can destroy the user experience and render your software unusable (not to mention the damage done to your business’s reputation). 

    When the stakes are so high, it’s clear to see why businesses need a process such as this. In this article, we take a look at user acceptance testing, why it’s so important, and six ways brands can use it to their benefit.

    What is User Acceptance Testing (UAT)?

    User acceptance testing (sometimes known as end-user testing, user acceptability testing or beta testing) is the last step in the entire testing process of any software.

    In product management terminology, acceptance testing takes place as beta testing whereby a small group of early adopters test out the platform and provide user feedback to let you know if the solution meets user expectations and business requirements. 

    It’s done after the system, functionality and all other tests are completed and before the product is launched. 

    The image below shows some of the quality assurance (QA) tests that precede user acceptance testing. 

    Source 

    As you can see from the image, as development work moves forward, UAT progresses from unit testing to integration testing, then system testing and, finally, acceptance testing.

    This final stage is arguably the most crucial element of software development because it ensures a positive end-user experience. 

    As its name implies, it helps you determine if your platform will be accepted by end-users. 

    Testers (end users) work from a checklist of expected outcomes from the test platform and then record what happens when they attempt to execute the different tasks, such as:

    • Logging into the user’s account
    • Placing an item into a shopping cart
    • Checking out 
    • Buying multiple items that qualify for free shipping

    … and so on. 

    If the expected outcomes are achieved when each task is executed, then the software is approved for release. If not, it’s sent back to the developers. 

    This testing process requires collaboration between different stakeholders including project sponsors, business owners, analysts, QA teams and developers. 

    Why is UAT Important to Software Development?

    Although there are many ways projects can benefit from UAT (some of which we’ll look at below), this process is often overlooked by developers. 

    Many see it as a time-consuming and unnecessary process, which is unfortunate because acceptance testing can actually help improve software quality as well as the overall acceptance rate of any newly implemented software. 

    Think about it: You would never drive an untested car, would you? Similarly, businesses should not take risks by implementing software that has not been tested. 

    Considering the time and money you stand to save by ensuring that your software works in accordance with a predetermined test plan, it’s clear that this is something developers need to take more seriously. 

    In fact, far from being the time drain that many developers think it is, UAT can actually gain you a lot of time. How? By preventing unnecessary waste. 

    And far from being a one-off when you come to the end of the project, user acceptance testing extends beyond business application projects because of factors like additional phases, hotfixes, releases/further modifications or customizations and upgrades. 

    Take a look: 

    • Additional Phases: If you choose to roll out at this stage or upgrade an existing application in a phased approach, each phase can have different types of users, different companies (for parent companies, acquisitions, mergers, etc.), additional locations, etc., and you’ll need to plan and conduct separate additional acceptance tests for each individual phase.
    • Hot Fixes: Bug reporting or developer testing during regular day-to-day operations can lead to hotfixes. However, these bug fixes will have to be tested by end-users to make sure that they actually solve the problems originally reported.
    • Further Modifications: Software creators regularly release updates, including limited distribution releases (LTR) and general distribution releases (GDR). LTR releases usually fix issues in a specific problem area, whereas GDR releases are typically heavily tested before release. Both need to be tested to ensure that they fit users’ real-life scenarios so you can achieve your business objectives.
    • Upgrades: Everyone wants to offer their customers the best and most advanced experience. That’s why, for the most part, brands update software regularly to follow industry trends and keep up with the latest innovative technology. Regardless of the upgrade approach you take, whether it’s a little or a lot of customization, it’s crucial that your plan is completed according to business requirements. This will help ensure that the project’s end result is in line with the original business requirements.
    • Privacy Policy Updates: Many end-users may enjoy the overall functionality of the software, but they may desire increased integrations with other platforms or the ability to use a different payment method than that which is currently accepted. While these upgrades and modifications may be easily completed, you may need to alter the information you input into privacy policy generators to ensure that your users legally consent to any new use of their data or access to it by third parties.

    Now that you understand what UAT is and why it’s so important, let’s look at some of the benefits of user acceptance testing to give you a better idea of all the different ways you can leverage it in your own business. 

    Benefits of User Acceptance Testing

    At its core, user acceptance testing will be pivotal in the successful launch and/or maintenance of any type of online platform, whether it’s a simple mobile app or something more complex, such as an eCommerce website. 

    If the test object doesn’t work as desired, the faulty business functions are documented and the item is sent back to developers for repair. 

    Going through this final check ensures that your platform is well-built and performs as expected and that users accept the finished product. 

    Listed below is a summary of the top benefits of UAT: 

    • Benefit: The UAT process allows you to demonstrate that the software’s required functions are operating in a way that’s suited to real-world use and circumstances. As such, acceptance testing focuses on software performance and user behavior to ensure that interactions between the test software’s functionality and the end user’s thought process are aligned. 
    • Benefit: Furthermore, you’ll be getting a much better piece of software because it will be tested and verified by the same people who will be working with it every day. You’ll get optimal opportunities to quickly identify and repair any broken features. So, it’s easy to see how the time you invest in these beta testing cycles will lead to significant improvements in the quality of your software.
    • Benefit: Not only that, but the quality review provided by UAT is also a great way for stakeholders to secure a healthy return on investment (ROI) out of projects. Also, it keeps maintenance costs low and it’s cheaper and faster to fix usability/functionality issues early on.
    • Benefit: You’ll also be able to ensure that your software integrates with third-party platforms including various website builders, email marketing software, project management tools, communication apps and everything else you or your end-users might need to effectively use the software. If your software is supposed to allow for social media sharing, for example, you want to ensure that end-users are able to easily share without going through multiple steps that decrease an otherwise positive experience with the software.

    As you can see, user acceptance testing is vital to the successful deployment of new business applications and software enhancements, as well as to ensure organizational efficiency and success.

    The process helps you to mitigate risk and boost the ROI of your project. But most importantly, UAT ensures that your customers don’t end up being acceptance testers by default, which would have a negative impact effect on your business.

    6 Ways to Use UAT

    Here are six ways you can use this process to achieve better outcomes.

    1. Use user acceptance testing for internal development.
    2. Use the testing process to review new features.
    3. Invite users to take part in the tests to help them become more invested in the brand and become brand ambassadors.
    4. Use beta tests as a premium feature to convert subscribers to higher tier packages, increasing sales for the company.
    5. Share information about UAT and its results on social media platforms, email and other channels as part of your digital marketing campaigns.

    These are just a few of the different ways your organization can benefit from UAT far beyond the initial goal of the testing process. 

    And remember, it’s all about the users. 

    UAT isn’t just another testing technique. This test puts the user at the center of the discussion to answer questions such as:

    • Are users able to use the software without error(s)?
    • Does the solution meet promised specs? Does it behave in an expected way?
    • Does the software continue to work when users take unexpected actions?

    …and so on. 

    Whether the platform is constructed in-house or by a third-party company, all solutions must go through user acceptance testing because your users are the ultimate audience who will pay for your solutions. For this reason, it’s worth taking the time to ensure that your software solution meets both user expectations and business requirements.

    Planning and Managing the UAT Process

    The key to a successful UAT testing process is to understand the needs of your target users and their expected behavior. This way, you’ll be able to develop defined acceptance criteria so you can come up with efficient tests to complete the testing process fast. 

    Also, since the goal is to assess the market readiness of the product, it’s important to record all evidence and track progress with the dev team as well as report to stakeholders so everyone is on the same page. 

    With that said, here are five steps to help you plan and manage the user testing process: 

    Step One: Planning. The user acceptance testing strategy mainly includes a time frame outlined during the planning step.

    Step Two: Designing Test Cases. Test cases cover the core functional frameworks of the platform in real-world usage. When designing test cases, you need to do the following: 

    • Determine the needs of target users and stakeholders 
    • Design test cases according to the user journey and product specifications 
    • Summarize test runbook, acceptance criteria, etc. 
    • Prepare test data, e.g. registration, three days in use, one week in use, etc. 

    Step Three: Setting Up Your UAT Team. The acceptance testing team is comprised of real-world end-users or users who are not part of the development or testing phases to ensure objectivity. You should never use developers to test their own software as they may bring bias (albeit unintentionally) into the testing process. 

    Step Four: Executing and Documenting Test Cases. The UAT team executes the test cases, together with relevant random exploratory tests, after which all bugs are logged into the bug tracker along with steps to reproduce.

    Step Five: Bug Fixing. Once you’ve executed the test cases, you need to keep track of the tickets and user feedback. The development team can account for all the bugs found by the testers and start fixing bugs depending on severity. 

    Step Six: Confirm Fixes and Get Stakeholders to Sign Off. Once all UAT tests have been run and the results are in hand, the acceptance decision can be made. If all users are satisfied, then it’s a go. If not, it’s a no-go. 

    Last But Not Least

    UAT is the last stage in the software development life cycle. But, it’s also one of the most important things you can do to ensure the success of your software. Just keep in mind that for this process to be effective, it should be viewed as validation and not verification. Use your test procedures to validate functionality in both smaller scope (as a way to test functionality) and in a larger context (to simulate real-world processes) so you can get an end-user vision of your product which will ultimately help you increase the robustness and usability of your software.

    Are you ready to use UAT to get feedback on your software so you can increase end-user happiness? Share your thoughts below!

  • What Donuts Teach Us About DevOps and Delivery Risk

    What Donuts Teach Us About DevOps and Delivery Risk

    Today, most DevOps teams do not think too much about their approach to risk. Usually, the typical attitude focuses on what to add to reduce risk further. In other words, “Can we add more tests? Can we deploy more carefully?” More tests, more processes and less risk are all obviously good things. Or are they?

    As a programmer, I’ve seen a lot of projects where there are so many tests and so many processes that they actually become counterproductive. Those ‘overheads’ presumably do reduce risk, but nobody knows by how much. I call this attitude ‘blind risk avoidance.’ This line of avoidant thinking usually builds up over time as we react to failures. It’s a very natural reaction.

    But when risk avoidance becomes your standard modus operandi, your delivery cycle gets slower and slower over time. Your team starts to feel they are trapped in molasses but, as time goes on, they not only gradually train themselves to tolerate this but to actively justify that it’s a good thing. In the meantime, nobody pays attention to the fact that your software is losing a competitive edge and that you’re being out-innovated and out-executed by others. Nobody pays enough attention to engineering team attrition; losing people as they drop off, one-by-one to move to more modern software development teams with less baggage. The ultimate conundrum lies here: If you move too fast, you break things. But if you move too slowly, your business and your team are doomed.

    So what is the optimal approach for DevOps teams to take when it comes to risk? The truth is, neither extreme is desirable. Blind risk avoidance leads to a failure of value delivery. On the other hand, blindly ignoring risk means crashing and burning.

    The ideal balance lies in not avoiding risk entirely or ignoring the reality of risk. Instead, a sensible middle ground can be found in taming risk.

    Moving forward, DevOps teams and developers must focus on taming risk to deliver software confidently with data-driven actions, speed and quality. 

    Automation Can Reduce Operational Overhead

    Taming risk should become a larger part of DevOps culture. One of the most famous ways this approach has been adopted is by Google’s site reliability engineering (SRE) team. 

    Google’s SRE team embraces two key principles when it comes to taming risk: Automation for reduced operational overhead and structured and rational decision-making for better, data-driven choices.

    To explore this mindset of risk-taming at Google, let’s dive into how the SREs use automation to slash waste, save time and reduce repetitious, low-value tasks and tests. 

    At Google, SREs are software engineers with an extremely low tolerance for repetitive, reactive work. (Aren’t we all?) But here’s the difference at Google: Thanks to a workplace culture of taming risk, operations that do not add value to a service are avoided or automated away.

    The vast majority of DevOps teams hold the same beliefs as Google: That automation cuts down on waste and saves everyone time.

    We all know automation allows for lower operational overhead, frees up software engineers to proactively assess and improve the services they support and boosts team morale. No one wants to spend their time or energy on repetitive, low-value tasks. Automation frees up DevOps teams to expend their energy on what matters most: Continuous quality.

    Bottom line? Automation is integral to the idea of taming risk. Automation of repetitive, low-value tasks is the perfect way to embrace a mindset of taming, not avoiding or ignoring risk entirely.

    Data-Driven Decisions

    Another way that Google’s SRE team tames risk is through structured and rational decision-making based on data. 

    Data-driven decisions are crucial to the success of any software engineering team, project or mission. Without proper data and feedback, there’s no way for software engineers to make the best, most calculated decisions and mitigate risks. 

    A data-driven decision methodology can—and should—be applied to DevOps testing cycles. 

    Another key aspect of data-driven decisions that speaks directly to the concept of taming risk? Service level objectives (SLOs).

    A service level objective is an element of a service level agreement that measures the performance of the service provider. SLOs are created (and agreed upon) between the service provider and the customer and help to mitigate disputes between providers and customers.

    Now, let’s take the concept of SLOs and apply them to engineering and QA teams.

    In traditional organizations, engineering and QA are forced to interact intentionally with a healthy amount of tension. One team produces software and the other team tears this software down. While this kind of necessary and healthy conflict between teams can lead to better results, the inherent push and pull between engineering and QA can create an excess of interorganizational infighting.

    Engineering teams will always advocate for more velocity or blind ignorance of risk, above all else. On the flip side, QA will always advocate for quality, or blind avoidance of risk, above all else.

    Of course, neither side is entirely correct (nor incorrect).

    Instead, the solution to this tension lies somewhere in the middle: The taming of risk, not ignorance or avoidance of risk. This is where an SLO, or the numeric target that represents ‘somewhere in the middle,’ comes into play. For engineering teams focused on velocity and QA teams focused on quality, creating an SLO—a happy medium to work toward with regard to risk—will create a more balanced workplace culture. In short, this approach can be considered a taming of risk.

    ‘Donut’ Slow Velocity

    Now, we’re finally going to get to the donuts. To demonstrate how embracing a mindset of taming, not avoiding, risk can create better, happier and faster DevOps teams, let’s take a look at Krispy Kreme Donuts.

    Krispy Kreme churns out millions of delicious, hot, glazed donuts every single day. However, during production, it’s inevitable that one or more of those donuts will be flawed. (I know, you’re asking, “Is there really such a thing as a flawed donut?!” For the purposes of this analogy, let’s say yes, there is.)

    One common donut defect occurs when the donuts bunch up on the production line and clog it.  Now, let’s assume this common error is caused by the speed of the line. If the Krispy Kreme team attempts to solve this problem by pushing the line even faster, tolerance gets smaller and the risk of even more smashed, defective donuts only rises. That’s certainly not a solution to this issue.

    On the other hand, if the Krispy Kreme team slows down the donut production line speed, there will be a smoother—albeit slower—operation.

    Is Krispy Kreme better served by slowing down its entire production line and sacrificing its velocity to eliminate the risk of a potential defective donut? Of course not! 

    If that’s not true, then is Krispy Kreme better served by cranking up their production line speed to go as fast as possible, regardless of the risk of defective donuts? Again, of course not!

    For sweet success, Krispy Kreme must embrace a mindset of taming risk. By baking careful measurement and an iterative feedback loop into their donut production process, Krispy Kreme is better able to create delicious, high-quality donuts at an optimal speed, producing some flawed donuts. This balance of speed with quality creates the best economic outcome and the mindset and measurement allow for a constant evolution to risk-taming in the future. As the production line changes, the optimal speed also needs to evolve.

    DevOps teams should take this same approach. Tame the risk without sacrificing speed and quality. ‘Donut’ slow velocity to fix a few minor issues!

    If you’re still not convinced, here’s another way to think about taming risk that is decidedly less delicious than warm, glazed donuts: Taxes. 

    The IRS balances velocity and accuracy during tax season as they process millions of tax reviews. While suspect returns are flagged for audits, the IRS simply cannot catch every single flawed tax return. It’s not economical or even possible to run detailed audits on every single problematic tax return, every single year.

    So, instead, the IRS tames its risk with automation and data-driven decisions. The IRS focuses on creating and using methods to detect patterns that correlate with flawed tax returns. Then, once these data-driven problem returns are detected by automation, experts can devote their precious time and mental energy to tackle the biggest of the problematic returns. 

    Again, this approach to taming risk creates an inherent balance between quality and speed and allows for an evolution of the process as economic activities and laws evolve.

    Whether you love eating donuts or doing your taxes, the bottom line is clear. Most organizations need to balance potential risks with maintaining speed, quality and team morale. DevOps teams are no different. 

    How to Tame Risk in DevOps Teams

    So how can we, as DevOps teams, tame the risk?

    The basic idea here is to do smarter testing, not just more testing. The good news is that if you have a reasonable level of automation, the data coming out of your existing automation can make your testing smarter. Approaches like predictive test selection can let you spend your computer hours more meaningfully, allowing you to improve speed and quality at the same time. Approaches like flaky test insights can let your precious engineering resources focus on the most impactful effort to improve the quality of tests.

    The Power of Taming Risk

    What is the right balance between speed and quality? This is the constant question DevOps teams face. But, here’s how I answer this riddle: Taking calculated risk is the most logical and rational approach in the workplace.

    It’s easy to avoid risk. Anyone can skirt around potential pitfalls by avoiding new solutions and sticking to the same old methodologies. But it’s so much more satisfying to take a calculated risk rather than ignore or avoid the risk outright. 

    If you want to improve the velocity and the happiness of your DevOps team, it’s time to embrace a new mindset: Tame the risk, gain control.

  • Survey Sees App Testing More Widely Distributed

    Survey Sees App Testing More Widely Distributed

    A survey of 900 application testing professionals conducted by PractiTest, a provider of a platform for managing testing tools, suggested the number of organizations that have dedicated testing teams is declining at a steady rate.

    The survey found a 15% decrease in organizations with dedicated testing teams that have six or more members. At the same time, however, the survey found only 18% of respondents reported that all testing is done by individuals that are not part of a dedicated testing team.

    Joel Montvelisky, test specialist and chief solution architect at PractiTest, said that shift indicates that more organizations are steadily incorporating testing within DevOps workflows. In fact, more than 60% of respondents said DevOps and Agile processes led to teams being more organized and stable, with two-thirds (66%) reporting that development teams were releasing features more frequently with more functionality.

    However, only 38% of respondents said they were active participants in a continuous integration/continuous delivery (CI/CD) workflow. A full 90% said they needed to coach or train other members within the team and just over half (54%) reported that estimations of the time required for development and testing were accurate.

    Two-thirds also said collaboration with developers is also improved when applying DevOps processes, with more than half reporting fewer serious bugs made it into production. More than half also said they monitor software in production and deployment environments. The number of testers who defined themselves as engineers has gone up by 14% year-over-year. However, only 44% report that automation has replaced 50% or more of their manual testing efforts.

    A full 95% or more of survey respondents identified application programming interface (API) testing, Agile methodologies, functional testing and scripting as high on their list of priorities. In contrast, big data testing, internet of things (IoT) testing and machine learning/artificial intelligence (AI) testing were cited as important or very important by only 60% of respondents.
    Finally, about a third of survey respondents noted that non-testers are now involved in testing activities.

    Montvelisky said the PractiTest survey makes it clear that while developers are a little more involved in testing than they were a few years ago, organizations still relied on testing specialists that are becoming more integrated into DevOps workflows. This is especially important as more organizations launch digital business transformation initiatives that require them to build software using the same types of DevOps processes that independent software vendors (ISVs) employ.

    It’s still too early to say just how far testing might shift left toward developers, but as testing becomes more automated—thanks, in part, to increased reliance on AI—the types of tests that will be conducted by a dedicated testing team will evolve. In fact, it’s unlikely automation will eliminate the need for those teams any time soon. Rather, testers will be augmented by AI technologies that will help them to keep pace with the accelerated rate of application development enabled by wider adoption of DevOps best practices within organizations both large and small.