devopsy.kinsta.cloud

Author: Rob Mason

  • FAQs for Software Testing: All the Background You Need

    FAQs for Software Testing: All the Background You Need

    No matter what type of application you and your team are building, testing is a critical step in the software development life cycle (SDLC) that can not be overlooked. In this article, I’ll answer frequently asked questions about software testing, why software testing matters and suggest best practices to help ensure your application is ready to delight customers by the time it is deployed.

    Why Do We Need Testing?

    Software testing involves evaluating an application to see if it meets the requirements and functions it was built for. The process includes identifying defects, errors and bugs, along with flaws in user experience. Software testing is so critical because finding and fixing these bugs and issues in the development process means the end product reaching customers works properly and reliably and delivers a high-quality user experience. Additionally, testing during the development process makes bugs easier and less expensive to fix as opposed to making fixes after an application has been deployed.

    Overview of Testing Types

    Let’s quickly define the common types of software testing:

    ● Unit Testing: This process evaluates individual components of software to ensure they are working properly.
    ● System Testing: This involves ensuring an entire application or system meets requirements and functions as it should, end to end.
    ● Integration Testing: This looks at whether or not different modules of an application or system work properly together.
    ● Regression Testing: This process revisits previously tested components of software to make sure they work correctly after a change has been made.
    ● Acceptance Testing: This process determines whether an application meets the needs of end users and is ready to be deployed.
    ● Usability Testing: This process assesses how user-friendly a customer will find the product.

    Creating the Plan

    Any solid software testing plan is going to incorporate multiple testing approaches, including manual, automated, user experience, exploratory and non-functional testing. You’ll start by defining the scope for what parts of your application, integrations and functionality you need to test for. Then, you can decide what type of testing methods are needed for each and document these to ensure consistency.

    You’ll want to combine code coverage and test coverage to ensure the code for your application is tested thoroughly. The type of tests you prioritize will vary based on different products, and your testing strategy will certainly evolve and grow over time.

    Measuring Your Strategy for Thoroughness and Effectiveness

    To ensure your testing strategy is thorough and effective, you will need to make sure to test for code, device coverage and all aspects of user experience. The bottom line for the program should be delivering value to the business by making sure the application works as intended for users. To be effective, testing should take place before an application reaches end users, so issues and bugs can be identified and fixed before deployment. To be thorough, testing strategies must be balanced, end-to-end, and look at functionality, usability, accessibility and more.

    Effective software testing can be measured in a few ways, including:

    ● Code Coverage: The percentage of code that is tested.
    ● Test Case Effectiveness: The percentage of test cases that found defects or bugs.
    ● Time to Detect and Fix Defects: The time it takes to find and fix issues in an application.
    ● Defect Density: The number of defects found per unit of code.

    Improving Your Testing Approach

    Your software testing program should be evolving and improving over time and can improve by:
    ● Including test-driven development
    ● Leveraging a shift-left approach
    ● Using automation where possible
    ● Benchmarking QA for current projects versus previous ones
    ● Conducting a cost-benefit analysis

    Testing Best Practices

    While the below list is not exhaustive, here are some top best practices to keep in mind as you build out a software testing practice at your organization:
    ● Incorporate the voice and feedback of your customer into product design
    ● Make accessibility a priority
    ● Test throughout the entire SDLC, including in pre-production, after changes are made to code, and in-sprint testing
    ● Maintain a matrix for the different devices using your application
    ● Prioritize high-quality user experience
    ● Automate repetitive tests where possible
    ● Review and adjust testing processes regularly, as needed
    ● Track and report results to analyze trends and identify areas for improvement
    ● Balance between leveraging different testing types

    Your organization’s software testing strategy directly impacts the quality of the end application you deliver to your customers. Poor usability, accessibility, and reliability of an application can frustrate users and send them to competitors, while high-quality user experiences and application performance can boost adoption and keep customers happy and loyal to your brand. With so much competition, taking the time to build out, adjust and prioritize software testing is critical for modern businesses.

  • Bug Bounty Vs. Crowdtesting Programs

    Bug Bounty Vs. Crowdtesting Programs

    Digital quality and excellent user experiences are more important today than ever. When a digital experience doesn’t work as expected or is rife with friction points or glitches, customers are quick to give up and move on to a better, easier solution. Bugs and defects often come with high stakes and can result in loss of revenue, dissatisfied customers and potential damage to a brand’s reputation.

    With this in mind, software testing is tasked with the important role of catching and fixing bugs and defects before they hit production. There are several approaches to finding bugs in software, two of which are bug bounty programs and crowdtesting. In this article, we’ll examine the values of each.

    Bug Bounty Programs

    A bug bounty program typically involves people outside an organization with software testing experience, leveraging their expertise to hunt down product defects in exchange for money. Bug bounty programs can be a cost-effective way to find defects that otherwise go unnoticed. They provide an outside perspective and a sanity check for digital products to find vulnerabilities and defects.

    From a security perspective, bug bounty programs are also valuable. Today’s security teams have trouble managing the sheer volume of software vulnerabilities found in software. That’s part of the reason behind the growing number of cyberattacks and breaches that continue to be a topic of discussion. Bug bounty programs leverage white-hat hackers to find and report vulnerabilities in an effort to improve overall security posture and support the organization’s own cybersecurity team.

    Bug Bounty Benefits

    Bug bounty programs offer a variety of benefits to organizations, including:

    ● Helping with blind spots: It doesn’t matter how well your software is designed and coded. Defects and bugs, whether minor or major, will find their way into applications. The developers themselves and internal testing teams may only test certain paths based on organizational norms. Bug bounty testers offer a fresh perspective that can help reduce overall blind spots in application testing.
    ● Resource flexibility: Most organizations don’t have the flexibility to scale internal resources on short notice. Bug bounty programs offer outsourced help and are flexible. They can be scaled up if needed to help with finding defects before a major launch. Or, if a business needs to save money, it can be scaled down by either reducing payouts or limiting testers to high-value defects.
    ● Continuous evaluation: Whether an application is in use by customers or is just a prototype, a bug bounty program can go to work finding and revealing bugs and vulnerabilities. Bug bounty programs can continuously search for bugs that help improve the quality of an application.

    Bug Bounty Challenges

    While there are some great benefits to using bug bounty programs, there are challenges, as well, including:

    ● Money focus: As the name suggests, this is a bounty program. High-severity defects come with a higher payout. If a large percentage of bug bounty hunters are focused on high-severity bugs and vulnerabilities because of the higher earning potential, they may miss the smaller defects. This can add up and result in a less-than-ideal user experience, with smaller issues going unfixed.
    ● The perspective of the tester: No matter how experienced and skilled a tester is, everyone is limited by their own blind spots and perspective. A diverse group of testers can help offset this, but there is still no guarantee that they will test the product in the same way that customers will use them. This challenge is multiplied when we add in different devices, operating systems, languages, and locations. Bug bounty hunters may be more focused on finding high-value issues to earn more, while an internal tester may be more invested in a specific product or feature and making sure it works as it should.
    ● Communication: Many have gotten used to working from home or from anywhere and collaborating remotely. Bug hunters work like this too, but they are outsourced assistance from an organization, which requires a bit of extra thought for communication, project scope and qualifications to make sure misunderstandings or lags in progress are kept to a minimum.

    Crowdtesting

    A crowdtesting model leverages skilled testers and testing teams that can be sourced from a globally dispersed community of independent digital experts. Depending on the size and diversity of the community, testers can apply a nearly limitless variety of demographic characteristics, locations, skill levels, spoken languages, abilities, devices, browsers and operating systems into a testing scenario. This breadth of feedback provides valuable insights into how real users will interact with an app, device or digital experience—which may not necessarily be the “typical” way it is intended to be used. This real-world interaction can unearth bugs, friction points and localization issues that may otherwise go unnoticed before the experience is released to actual customers.

    The remote and on-demand nature of crowdtesting provides scalability and timely results in comparison to more traditional testing methods, like offshoring or lab-based testing, which are typically more limited in scope and flexibility. Plus, the crowdtesting model inherently provides the ability to scale testing services up or down as needed to meet the project-based requirements of a company’s internal teams.

    Crowdtesting Vs. Bug Bounty Programs

    Both bug bounty and crowdtesting programs involve digital experts looking for app defects and vulnerabilities with the goal of improving digital quality. These programs can be aligned in this goal and even work in tandem, but there are some notable differences between them worth mentioning.

    ● Scope: Bug bounty program scopes will always be somewhat limited. Defects can be found outside of the payment scope and go unflagged. Crowdtesting allows for either a broader or more specific scope, depending on needs. Organizations can give specific instructions, like making sure UX considerations are taken into account.
    ● Volume: Both crowdtesting and bug bounties rely on skilled professionals who know how to root out bugs and defects. Crowdtesting, however, allows for as many or as few testers as needed. Instead of gathering a moderate amount of defects from a more technical audience, crowdtesting enables testing for any amount of defects from a diverse and vast pool of testers. As a result, crowdtesting finds a larger variety of defects, and across more devices, locations, and OS versions. This can also be a challenge, as internal staff can become overwhelmed by so many bug reports to look through and address. Therefore, prioritization is key.
    ● Flexibility: Digital quality is a multi-faceted process. It’s not just about functional and security bugs but also accessibility assessment, making sure to test using specific devices and use cases, and more. Bug bounties fall short here, while crowdtesting delivers the flexibility to test and address situations your potential customers may face when an application is live.
    ● Specificity: Most bug bounty programs are not tailored. They can be application- or region-specific, but that doesn’t always reflect a target customer use case. Crowdtesting enables an organization to focus on specific demographics in terms of user experience. An organization can specify testers in certain locations with a certain device or devices and who speak specific languages. Crowdtesting isn’t always just about technical ability; it’s about finding the correct group of people needed for the task at hand.

    Both bug bounty and crowdtesting programs provide unique benefits that help organizations improve overall digital quality. Bug bounty programs offer help with organizational blind spots while leveraging testing professionals in a flexible way. Crowdtesting enables businesses to harness the power of real people using actual devices out in the world to test for use cases. It delivers the power to augment internal testing initiatives, helping to improve overall quality, usability, accessibility and more. When improving digital quality and user experience are the goal and end result, bug bounty and crowdtesting programs can each be impactful and helpful for today’s businesses and for improving the user experience.

  • Why Software Usability Testing Is Essential

    Why Software Usability Testing Is Essential

    Keeping an eye on the bigger picture is important when doing just about anything. This adage holds true when testing software, too. There’s no doubt that the QA testing process for software is essential. It makes sure everything is running smoothly and as it should. But generally speaking, it doesn’t always fully take into account the user perspective. Even though developers, designers and testers all try to keep the end user in mind when building and testing, there is no true way to anticipate and test for the variability of actual user experience in the wild.

    This is why it is essential to consult actual users. That’s where the process of usability testing comes in. Every organization should consider implementing the practice before a software product launch.

    Usability Testing Defined

    Unlike functional testing, usability testing hones in on the user’s experience of a product. It focuses on how an application works in the real world with real users. In-house, it’s easy to test a feature and check a box to say it works. But that’s not necessarily the way an application will be used by every user in every situation.

    Usability testing involves having participants complete a few tasks with an application. Then, depending on how easy or difficult those tasks were, testers and developers can decide which, if any, improvements and changes need to be made. Usability testing helps organizations accomplish three key goals:

    1. Uncover opportunities to improve
    2. Identify issues in design
    3. Learn about intended user behaviors

    Differences Between Testing Types

    Software usability testing is sometimes confused with other similar approaches to quality assurance and control. Let’s look at and clarify similarities and differences between these methods.

    • User research versus usability testing: User research comes earlier in the software development life cycle and actually involves usability testing, but it also includes other components like interviews, surveys and focus groups to identify the behaviors of users.
    • User testing versus usability testing: User testing is put into place to validate a product idea. It’s used to understand how a user approaches a given situation and how a product might help address that situation. Usability comes later in the process of testing that specific product.
    • User acceptance testing versus usability testing: User acceptance testing confirms that a product works as intended; it’s about functionality. But usability testing will get into the specifics of user opinions, expectations and actual experience.

    Types of Usability Testing

    There isn’t just one way to conduct usability testing. There are a few options organizations can choose from based on differing objectives.

    • Moderated versus unmoderated: Moderated usability testing involves a moderator facilitating the progress of a tester through a series of tasks. This gives an organization the opportunity to ask questions during specific parts of the process. An unmoderated approach allows a user to complete tasks entirely at their own pace. This method can be conducted at any time, anywhere.
    • Remote versus in-person: In-person testing gives UX researchers the ability to provide specific devices to avoid technical issues, although it might mean missing issues with a user’s actual device that occur in the wild. It also allows for observing non-verbal reactions. Remote testing, on the other hand, allows users to complete tasks as they might in the real world from anywhere and with their actual device.

    Some specific usability testing methods and techniques that can be used include:

    • Screen/video recording: Gives insight into how long it takes to navigate and click through a website, application, system, etc. as well as tracking non-verbal cues.
    • Usability lab: A dedicated testing location. All variables can be controlled so the focus remains on the testing.
    • Guerilla testing: This is done in the wild, recruiting a few people to do a quick software test. While this can be quick and cheap, you can’t ensure you’re testing your specific target audience.
    • Contextual inquiry: Testing in a participant’s own environment with an interview and/or third-party observation of their use of the product. This allows researchers to know things like what device is used and where and how the application is used.
    • Usability testing partners: These are third-party organizations with testing expertise and access to a broad pool of participants that can be recruited and cultivated based on client needs.

    Benefits

    Usability testing can deliver a variety of benefits to an organization, including:

    • Direct feedback from the target audience
    • Cost savings through addressing concerns earlier in software development
    • Detection of usability issues before launch
    • Real-world testing
    • Unbiased assessment

    Costs and Challenges

    The most obvious concerns at the beginning of the software development life cycle are are development costs and time-to-market. While usability testing can take time and involve costs of its own, ultimately, the practice aims to address those concerns; in the long run, it is more expensive to launch a flawed product with poor user experience. A few factors that must be considered when it comes to the time and cost involved in performing usability testing include:

    • Sample size
    • Gathering the right participant profiles
    • Breadth and depth of testing practices
    • Amount of time allocated to the test

    All things considered, when done right usability testing delivers ROI that more than makes up for the cost and time involved.

    Consider the implications when an application is launched and certain users are frustrated with the experience and are left with a negative feeling toward the brand. Usability testing addresses this concern and allows for issues to be discovered and fixed earlier in the development life cycle. Investing in developing and testing for a positive user experience is always worthwhile and effective.

  • Best Practices for Codeless Test Automation

    Best Practices for Codeless Test Automation

    Low-code and no-code development platforms have revolutionized how companies think about application development. The same can be said for software for quality assurance (QA), where codeless test automation solutions handle the burden of coding for organizations that can’t allocate extra programming resources or keep up with extensive writing and maintenance of automated tests.

    Codeless test automation can be a big win for companies that are resource-challenged and trying to get applications to market quickly. A recent Applause survey of more than 2,000 digital experts revealed that nearly 54% of organizations have strategic plans to purchase a codeless test automation tool in the future.

    Despite the heat in the market, though, it is not a fit for every type of application or use case. It is not, in most cases, a replacement for scripted automation, but rather should be used as a complement to traditional automated testing and manual functional testing.

    An example of this is with dynamic content in applications, which have unpredictable or even subjective outcomes. Consider a streaming media platform. An automated solution might validate that the “Play” button triggers the code to play a movie, but it cannot validate that the movie plays without audio or video glitches.

    In complex cases that aren’t a fit for either codeless or traditional automated methods, manual testing is a better alternative.

    When to Use Codeless Test Automation 

    Test automation is not going anywhere, even as codeless tools become more prominent. That’s because automation enables brands to move faster and meet the fast-moving target of building and testing software that is both high quality and able to be released to end users quickly. The downside of traditional test automation, however, is that building and maturing the process can be time-, cost- and resource-intensive.

    This is because traditional automation requires experts in coding, not only to write tests at the outset but to also maintain them over time. The complexity and need for different tools—like Appium, Selenium, Apple simulators, Android emulators, element locators, locator strategies and so on—all contribute to the complexity of traditional automation and the need for specific expertise. Plus, software development engineers in test (SDETs) are constantly being taken from QA teams and moved into development roles, which often means that these projects get stuck halfway through and never fully completed, which can then result in a tremendous waste of both time and money for companies.

    The overarching value of codeless test automation, on the other hand, is that anyone can do it. With a codeless solution, a user simply moves through a test case and the codeless tool can then transcribe that experience into an automated test. And, while codeless test automation tools originally addressed only web applications, more tools now offer the ability to run sessions and create automated tests on mobile apps—on both Android and iOS—as well as web applications.

    What it all comes down to, though, is that organizations shouldn’t be thinking of codeless tools and traditional automation as an either/or scenario. Codeless test automation tools are great for less-involved scenarios—like smoke tests or portions of a regression testing suite. Using codeless tools in this way allows SDETs and dedicated automation resources to focus on higher-priority and more complex automation. Ideally, manual testing should be used alongside both traditional automation and codeless test automation solutions as a way to maximize the speed, scale and quality at which software can be delivered to end users.

    As a rule of thumb, every organization should balance test automation and codeless test automation and consider automating when the following factors are in play: 

    • Repeatable task 
    • Time-consuming testing activity 
    • A stable testing environment with little change anticipated 
    • Repetitive test(s) 
  • How Shifting Testing Left Can Boost DevOps Velocity

    How Shifting Testing Left Can Boost DevOps Velocity

    DevOps is all about balance: How can speed and quality be balanced to best meet the demands of customers who want exceptional digital experiences as fast as possible? DevOps looks to meet these expectations head-on by helping to deliver a faster cadence of new builds to customers while keeping quality high.

    What’s more, quality should continuously improve, build-over-build, so that fewer fixes are needed with each iteration. While inserting more testing earlier in the development life cycle may at first seem like a hindrance to the speed DevOps promises, it will soon empower teams to release higher-quality software, faster, as shift-left testing becomes a routine part of the team’s processes.

    Many organizations wait until the very end of their software development life cycle (SDLC) before executing testing. With this structure, testing can and does find unexpected bugs late in the SDLC, where issues are the most difficult and expensive to resolve. With this approach, testing becomes a bottleneck, impacting that ever-important goal of efficient and Agile software development and deployment. An unexpected outcome: it also negatively affects team morale, as developers who spent plenty of time and effort to build and ship code quickly now have to delay their deployment process in order to correct issues.

    There is a more efficient approach that organizations can put into place, however.

    Shifting Testing Left

    Testing can be implemented during and throughout the critical stages of design and development of software, instead of after.

    This is true regardless of how a team approaches development methodologies: Agile, waterfall, continuous or a combination of these methods. When testing is shifted left, it becomes the responsibility of the entire organization, not just dedicated QA testers. That means SDETs, developers and others should all be active in testing along with QA. When these groups are involved in testing early on in the SDLC, there are plenty of benefits, including:

    • A list of user workflows and test validation conditions can be created to streamline testing during development processes.
    • Testers can catch potential issues much earlier in the process so they can be fixed during earlier stages, instead of after code is done being developed.
    • Developers and testers can work with code they are familiar with to edit existing code and fix issues themselves.

    Building a Continuous Feedback Loop

    Another major advantage of shifting testing left in the SDLC is that it creates a continuous feedback loop. Software can be updated and improved continuously as it is being built, instead of when it is “complete” and ready to ship. This model of a feedback loop saves time and ultimately ensures no show-stopping issues make their way into the hands of customers.

    Test Automation in the SDLC

    One more key element to shifting testing left is test automation. Unfortunately, automation is often not leveraged completely or to its fullest potential. A recent report found that 74% of QA testers see test automation taking place outside the regular test cycle. Moreover, only 45% of organizations use test automation, and only 50% use it for continuous integration/continuous delivery (CI/CD).

    For test automation to succeed, it must involve developers creating the automation code and testers, then editing, maintaining and executing that code. When test development, including automated testing, is moved into the SDLC properly, it leads to automated and integrated unit testing, which helps achieve that goal of increased efficiency for DevOps teams.

    Automation is obviously crucial to the success of DevOps, but manual processes must also be considered. Some types of testing – like exploratory, payments, usability and accessibility – cannot be automated (at least not fully), but they should also be accounted for as part of a shift-left strategy.

    The origin of DevOps methodology and the focus of developers involves the goal of building and deploying code as quickly as possible. But testing software to make sure it works as intended is an important step that should be included throughout the SDLC. When testing only happens at the end of the process, it can be an inhibitor to the overall goal of increased development velocity. But when testing is shifted left and involved earlier on in the SDLC, it can do the opposite – by creating a continuous feedback loop and implementing automation to drive improved DevOps velocity.

    Ultimately, shifting testing left in the SDLC leads to features getting to users and customers faster, an overall improvement of software quality and a reduction of QA testing time and bug fixes.

  • Crowdtesting: How to Ensure Quality in Disruptive Times

    Crowdtesting: How to Ensure Quality in Disruptive Times

    The practice of crowdtesting has taken on greater importance during the COVID-19 pandemic, for good reason

    Entering 2020, digital transformation projects and initiatives dominated tech spending for most companies. But, between a global pandemic and fiscal uncertainty around the world, spending must be shrinking, right?

    Think again.

    If anything, the pandemic has created a sense of urgency for many companies, accelerating digital transformation in organizations. According to June data from TEKsystems, 56% of digital experience leaders—companies with mature digital transformation strategies—have increased spending on digital transformation. Remarkably, more than one-third of digital experience laggards—those that have historically had limited digital transformation initiatives in place—also increased spending on digital transformation projects.

    The rush can be attributed to COVID-19. For businesses, it has changed the future of work. Distributed workforces are nothing new, but very few companies were prepared to deal with a fully distributed model in which all employees are working remotely. The pandemic has changed buyer behaviors, too. In lieu of malls and brick-and-mortar stores, people have turned to e-commerce sites for the bulk of their shopping. Legacy companies, such as Pepsi, for example, are building direct-to-consumer e-commerce sites in response to the evolving market dynamics.

    Digital is now the primary means of communication for businesses. Digital quality is no longer a niche issue for the application and DevOps teams; it is a strategic business imperative in the C-Suite.

    Apps Everywhere, At All Times

    Apps need to be tested by real users, on every device, network and operating system, across the globe and at all times. The typical workday used to be 9 to 5, but that’s no longer the case. With most workforces now remote, the days are longer and the diversions greater (yes, I’m looking at you, juggling Zoom calls and your daughter’s remote learning plan at school). Customers, too, have higher demands for the technology they use. There can be no downtime; downtime means customers will look for alternatives, becoming ex-customers and resulting in lost revenues.

    In today’s extremely fragmented device landscape, QA teams face increasing difficulties in keeping up with the myriad device/OS combinations they need to test on. For obvious reasons, purchasing and maintaining a nearly endless pool of devices is not an option—not only is it too expensive but it’s also time-consuming and harms the team’s productivity.

    Beyond logistics, QA teams must collect data from real-world testing to make informed decisions. No matter how many test cycles are run, QA teams can’t uncover all unexpected bugs and edge cases unless they include in-the-wild testing in their strategy.

    To test anything related to hardware in an efficient manner, leveraging a community of remote testers is the best way to obtain reliable results without burdening internal QA resources, generating disproportionate costs or missing out on important use cases.

    Crowdtesting Replaces Offshoring as the De Facto Way to Test Software

    With a market greater than $50 billion in India alone, there is no denying the offshoring model worked. But, the pandemic exposed pre-existing cracks in the model, causing many businesses and entire industries to regroup, re-evaluate and find new ways to accomplish tasks that just a few months prior would have been routine.

    When the pandemic hit, the supply chain broke. Testing and product development facilities built on product teams working collaboratively in close proximity struggled to keep up with the most basic needs of their customers. Outsourcing companies and their employees lacked the infrastructure to work remotely, negatively impacting the business continuity of their clients.

    This new reality leaves companies that are used to the offshoring model for testing and QA with one big question: what is the best way to test software and digital experiences in a post-COVID-19 world?

    While crowdtesting has been a factor in application and digital testing for more than a decade, it has taken on increased importance as internal QA teams can no longer maintain business continuity while working remotely and the offshoring model has faltered amid the pandemic.

    The reason the crowdtesting model remains a reliable answer even during COVID-19 is that it is always done remotely, via a community of vetted testers who already work in remote environments. This setup allows testing teams to maintain business continuity, while often improving test coverage across devices, operating systems, geographies, demographics and more.

    As an example, let’s look at how the Israeli government used crowdtesting to bring its tracing app to the country. The Israeli Ministry of Health developed the app using cellular GPS data to provide alerts on areas with high concentrations of COVID-19 infections. The app is designed to prevent COVID-19 spread by notifying people who were in the vicinity of verified coronavirus carriers. Similarly, healthcare provider K Health has used crowdtesting to help it launch a new COVID-19 diagnostic feature. K Health was already using crowdtesting services prior to the onset of the pandemic to ensure its mobile app functioned properly across all relevant device/OS combinations and timezones but found the scalability and reliability of the model meshed perfectly with the demands on healthcare providers in the new normal. When K Health offered free services to all Louisiana residents, crowdtesters were there to scale testing at a moment’s notice and provide 24/7 test coverage for the K Health app.

    It was urgent to get the app out quickly, but it also needed to work properly. Did push notifications go through promptly and to everyone? Would the app work across devices, and in different configurations? Since launching the app in March, the Ministry of Health has been able to leverage continuous crowdtesting to identify dozens of bugs that could have prevented Israelis from using the app or trusting its results.

    For example, the testers discovered that the app crashed when it was used in dark mode, and certain situations led to the app showing that it detected data even when data and location services were off. Since this second error was directly related to the main functionality of the app, it represented a very critical bug for the Ministry of Health to fix.

    Crowdtesting uses real devices and real people in their daily environments, which accounts for all these variables that are not seen in traditional labs and offshoring facilities. That makes crowdtesting a valuable testing approach even after the COVID-19 pandemic and resulting lockdown ends.