devopsy.kinsta.cloud

Tag: regression testing

  • Quality at Speed During the COVID-19 Pandemic

    Quality at Speed During the COVID-19 Pandemic

    Think back to 2015. The U.S. economy was in the early phases of a growth spurt that would continue for years. Ninety-five percent of organizations were practicing agile, to some extent (State of Agile, VersionOne). And the proportion of the IT budget allocated to QA and testing had just increased from 18% to 26% (World Quality Report, Capgemini). 

    “Quality at speed” became the catchphrase of the year in the testing world. With all the buzz around DevOps and agile, it was clear that speed mattered. In reality, most teams weren’t quite there yet. Agile adoption was still low (less than 50% of teams within each organization actually practiced it) and maturity was even lower (only 17% of those practicing it reported mature adoptions). Still, the writing on the wall was clear. Leaders wanted more application development, faster — so testers should at least prepare for the pending storm.  

    From that point forward, organizations across almost every industry all progressed to a sweet spot where time-to-market pressure was relatively high and cost pressure was moderate. By the end of 2019, quality at speed had graduated from buzzword to business as usual at most organizations. 

    But then came COVID-19. Now, we’re in the midst of a global pandemic. Every business, every non-profit and every governmental agency is impacted — but in different ways. 

    Speed expectations suddenly shifted into overdrive for an unlikely assortment of organizations, including government agencies and non-profits, healthcare and pharmaceutical companies, online retailers, online conferencing providers and telecoms. Yes, everyone had already been working to accelerate their software delivery processes. And many enterprise organizations had made great strides toward scaling the adoption of agile and DevOps. But, very few of these organizations were previously delivering applications at the extreme speeds required right now.

    For another group, cost reduction suddenly became the top priority. Airlines, hotels, as well as other travel and leisure organizations experienced such dramatic changes in fortune that speed became a non-issue. The focus shifted to increasing cost efficiency in an effort to offset some of these staggering losses. 

    Assuming that high quality cannot be compromised, this leaves both of these groups with distinctly different — yet equally daunting — challenges from a quality perspective. 

    The organizations in the first group suddenly needed to build, test and release applications at the speed of a startup — but with the added burden of legacy systems, complex application landscapes and intense regulatory scrutiny. Quality at speed remains the goal, but the “speed” part is much, much faster. 

    The organizations in the second group need to determine how to continue meeting strict quality expectations with a significantly slimmer testing budget. Quality at speed isn’t so important right now. Speed gets replaced by the other corner of the classic cost-quality-speed triangle, making the new mantra “quality at cost.” 

    It’s hard to find good news in the current situation. But, from the software testing perspective at least, there’s some consolation. Whether you’re facing unprecedented time-to-market pressure or extreme cost pressure, the same core “test efficiency” strategies will help you either way. 

    Here’s a quick overview of ways you can test more efficiently at different steps of the software testing process.

    Selecting and Designing Test Cases

    I’m addressing these two steps together because they both face a common challenge: intuition. Most modern business systems are insanely complex — so complex that it’s just not feasible for most testers to intuitively identify the best set of tests to cover a specific test need. Having a more systematic way to find the smallest — yet most powerful — set of tests will save you both time and money. 

    For designing test cases for progression testing, methodologies like linear expansion and pairwise testing can help, and augmented intelligence can dramatically simplify them for you. 

    For selecting what test cases should be run for regression testing, consider change impact analysis. Some approaches identify which objects are impacted by a given change, then help you narrow the test scope to the impacted objects. This often reduces the testing scope by 15%. Not bad, but you can do much better.

    AI-powered change impact analysis pinpoints exactly what needs to be tested in a given update, based on whether the changes put an object at risk. With this approach, you can reduce testing efforts by 85% or more. Not only is this approach faster and cheaper, it’s much more efficient at uncovering defects. The number of defects that hit production are reduced by 75% to 95%, sometimes even 100% (no defects in production).

    Automation and Execution

    Here, technical scripting is the top time and cost drain. Some level of maintenance is required to keep scripts running, and this maintenance will cut into your test efficiency. One way to combat this challenge is to take a model-based approach to test automation. Such approaches have been field-tested for years, and proven to solve the test maintenance nightmare that’s doomed all-too-many test automation initiatives. 

    The new kid on the block is AI-driven UI test automation. This is a big topic that warrants a much deeper discussion — but here’s a quick overview. Specify tests in natural language. Build the tests even before the actual UI is ready. Use the same tests as the technical implementation changes. Understand dynamic behavior like a human tester would. Test absolutely any technology — faster than a human eye. Curious? Stay tuned. 

    Analysis of Results

    When it comes to results analysis, false positives are the vampire sucking your resources. False positives are tests that fail even though the application is behaving correctly.  They will really kill you in terms of cost and speed. First, someone needs to figure out why the test failed. This might involve manually checking the related application functionality, inspecting the test automation, reviewing the test data, checking whether every element of the test environment is up and working as expected — maybe even all of the above. Once the exact problem is found, it needs to be fixed. Then, the test needs to be rerun. If it still fails, rinse and repeat.

    How do you stop false positives? To start, stabilize the automation with model-based test automation or the AI-driven testing I mentioned in the previous section. As you move into more advanced automation scenarios, stateful test data management and service virtualization can help you stabilize tests to the point that they’re fit for CI/CD. Then, your final challenge is to deal with the flaky tests resulting from things like asynchronous application behavior, concurrency issues and test order dependencies. Again, AI and ML can offer tremendous assistance here—in flagging potential false positives, and even in helping you identify the cause and optimal resolution. 

    Final Thoughts

    COVID-19 split up the original quality at speed sweet spot into the extreme speed camp and the extreme cost-savings camp. In both cases, the same core strategies to increase your test efficiency will help you increase speed as well as reduce testing costs. If you’re under extreme time pressure, showing that you also reduced costs while meeting speed expectations certainly won’t hurt. If you’re currently reducing costs because your industry is essentially paused, taking these steps now will prepare you to speed it back up when the world becomes more normal again.

  • Codeless Automation: Shattering the Testing Bottleneck

    Codeless Automation: Shattering the Testing Bottleneck

    In his article, “Moving from ‘Tester’ to ‘Quality Enabler’,” Scott Edwards clearly lays out the predicament in which QA testing is currently at: with one foot in the quicksand of manual testing, while the other tries to jump over the hurdle of automation. Ever tried hurdle racing with one foot? You probably won’t reach the finish line.

    Edwards mentions the infamous label that is attached to testing—bottleneck—and cites the “DevOps Review 2017” research paper of Computing.co.uk that found two-third of DevOps practitioners report QA testing as the bottleneck of the development process.

    Why is that? Why is testing bottlenecking development, and how can it be untangled?

    What Creates the Development Bottleneck?

    The testing bottleneck is created primarily due to two reasons:

    1. The reliance on manual testing, and
    2. “All the code that is getting thrown” QA teams’ way.

    Manual testing is extremely time-consuming, sometimes to the point that it’s close to impossible for test teams to keep up with the fast pace of DevOps. Manual testing is also error-prone, simply because the testing is performed by humans.

    The code issue is related to automation. Due to the pressure imposed upon test teams, automation is being hailed as the great savior. While the word “automation” has the flair of hands-free simplicity, the reality of automated testing is quite the opposite.

    Automation relies on code and testers aren’t coders. So instead of simplifying and speeding up testers’ work, the opposite is being achieved. Testers are thrown out of their comfort zone, being bogged down by tasks that aren’t in their “job description.” The frustration of all involved is easy to understand. The development team feels that testing is slowing them down, while the testing team feels they are required to jump through hoops.

    Add to that the complex setup and heavy maintenance required of automation, and you start to doubt whether it is indeed the right path for testing.

    Well, it is. With the ever-faster pace of DevOps, the pressure to shorten release cycles and the cutthroat competition software and web application companies are facing, test automation is the right path, despite all the challenges.

    What must be done to clear that path is remove coding from automation.

    Codeless Test Automation to the Rescue

    Codeless is a growing force in mass technology. It has liberated the web. Codeless platforms such as Wix and Shopify allowed millions of businesses to create an online presence quickly and effortlessly and led to the incredible expansion of the web. On the enterprise side, codeless platforms are making strides with the likes of Salesforce, FileMaker and Nintex, which offer drag-and-drop solutions to speed up and optimize efficiency of large-scale administrative tasks.

    The idea behind codeless is the modern-day “power to the people.” Just a decade ago, a small family business from Minnesota would have never been able to afford and maintain an online presence, but now their 22-year-old can set up an online shop, promote it on Facebook and in no time start selling their goods to a consumer across the ocean.

    Applying the same logic to testing instead of coding from scratch test after test—which is time-consuming and costly, and requires acquired skill—codeless test automation allows for quick and simple creation of tests, no coding skills required. It allows QA testers to focus on what they do best: testing.

    Automation shouldn’t be a hurdle that testers need to jump above; it should be a tool that pushes them forward. As things stand now, automation in testing is at a counterproductive standstill. Don’t get me wrong: Automation means well, but at the moment it’s like a misguided teenager that brings havoc even when he aims for harmony.

    How Does Codeless Test Automation Actually Work?

    Codeless test automation functions very similarly to enterprise codeless platforms. Both take complex tasks traditionally achieved by coding and simplify them through automated code generation.

    With test automation a tester creates a flow by visually binding elements that represent “clicks” in the software or web application. The tester simply goes through motions of the scenario he wishes to test, and the code for the test is generated under the hood.

    We can get more technical than this, but there’s no need. That’s exactly the idea with codeless, isn’t it? Testers could focus their efforts on testing functionality and evaluating the user experience rather than struggling with test creation.

    This ties back to Edwards’ main point: Testers get so entangled with test creation that the whole point of testing—quality assurance—gets lost.

    Side Note: The proposed job title change suggested by Edwards—Tester to Quality Enabler—might sound merely cosmetic, but it can deliver real value. As illustrated in this collection of QA testers’ venting from a reddit forum, QA testers often harness feelings of underappreciation and of being looked down on by their developing colleagues. Recognizing testers’ valuable contribution to the overall quality of the product will surely elevate their feelings of self-worth and enable them a stronger sense of pride in their position.

    Using Codeless for Deeper Integration of Testing Into DevOps

    Codeless automation is best utilized in sanity and regression testing, the two most basic and frequently applied tests. Sanity testing ensures that all incorrect inputs a user could enter are accounted for. Regression testing is like inventory counting, in a way: you just need to go over your stockpiles and see that everything is in order.

    Thinking on inventory: Let’s say I have a huge warehouse full of thousands of items. Checking inventory by hand is a very time-consuming and error-prone process. How many boxes on a piece of paper (representing real boxes on shelves) can you tick before you’ll make a mistake? Now think of Amazon’s warehouse. Inventory counting is done using automation with the press of a button. It is instantaneous and error-free. It’s an action that’s been done so many times before, there’s no need to plan it—or to code it—all over again. Simply run the same command—“inventory”—and you’re done.

    If you’re using codeless automation, all you need for regression testing is to click a button and the scripts run. You are done and able to divert your attention to sanity testing, making sure the new iteration is solid.

    For the next iteration, you simply pile the sanity testing of the previous round on top of the regression scripts and, again, you’re done.

    Continuous testing is getting a whole new meaning with codeless automation as part of the equation. If regression testing is done with the click of a button, and sanity taken care of simultaneously, the reporting back to development is swift and focused and the bottleneck is shattered.

    Codeless enables deeper integration of testing into the development process simply by arming testers with automation tools that simplify tasks, not complicate them.

    About the Author / Assaf Dudai

    Assaf Dudai is the content marketing manager at TestCraft, a pure SaaS continuous test automation solution for QA professionals. Assaf is mostly looking into the future of testing, how automation, codeless and machine learning can and will help testing evolve. You can connect with Assaf on LinkedIn.

  • Move the Needle in DevOps with Continuous Testing

    Move the Needle in DevOps with Continuous Testing

    Shift-left testing, risk based testing, early testing and technical debt reduction are all predicted to be some of the most sought-after trends in product development life cycle. All of these can be achieved with continuous testing through DevOps, as it emphasizes quicker transitioning of quality products to production environment at a reduced business risk.

    Software Engineer in Test (SEiT) or Software Developer in Test (SDiT) was long-spoken earlier and now implemented as part of DevOps culture. As the software side of the organization moves to continuous testing practices, so, too must the product side of the organization. External market forces including competition, innovation quotient, faster time to market, scalability and lean process initiatives definitely play a role in driving the continuous testing trend.

    From a technical standpoint, test environment creation, full test coverage, regression testing and integration testing of multiple endpoints are key parameters to ensure a faultless production environment when a new feature is added.

    Adopting Testing Tools for DevOps Implementation

    Continuous testing implementation not only includes adoption of right tools and test automation, but also a cultural change to incorporate testing at each stage of the agile development life cycle. It will involve changes right from test planning, test scenarios creation, automation feasibility analysis to test execution.

    Teams need to be restructured in an agile setup to align testers closer to developers and operations teams, and have common goals for product/software delivery. This involves changing the culture and organization structure, which generally takes a longer gestation time. Test management tools such as JIRA and TestNG can help expedite this structuring, as both parties would be aware of the product features that need to be delivered and can view the progress at all times. There also are various open-source tools, including Selenium, Appium, Python robot framework/pytest and JMeter, that aptly meet the needs of agile testing.

    Regression testing generally takes up the bulk of the time, as new features need to be tested every time they are added to avoid critical failures. Hence, automating regression tests have become an imperative as part of continuous testing. JUnit, which is a unit testing framework, promotes test-driven development and early code quality improvement. Bamboo can run tests in parallel batches and support hundreds of build agents providing an early feedback to developers. Automation scripts for test data cleanup, load and execution saves the testing effort of the operations team. Selenium test automation framework for web applications is one of the most famous tools for continuous testing, as it works with a host of programming languages including Java, C#, Groovy, Perl, PHP, Python and Ruby. Appium is another tool used frequently for cross platform mobile application testing.

    Ensuring an Effective Product Release

    Testing UI components and functionality of a product through API-based test automation is another important phase in continuous testing. API test scenario creation also generally needs involvement from the ops team. Considering virtualization of devices, along with applications or services, will accelerate test setup environments. Device virtualization using Raspberry pi helps in testing end–to-end use cases in IoT scenarios. Network virtualization for performance testing ultimately helps improve uptime and reduce outages.

    The orientation of continuous testing with continuous integration and continuous delivery pipeline within an enterprise will determine the long-term success and standing of the final product in the market, from a quality perspective.

    About the Author / Urvashi Babaria

    Urvashi Babaria is a Product manager at eInfochips working in new age areas like IoT, DevOps and CloudOps. She is a techno commercial marketer with nine years of experience in product and project management, business analysis and transformation. She has keen interest in areas of data science including visualization, analytics and modeling. Connect with her on LinkedIn.

     

  • xMatters Sees Value in Continuous Testing

    xMatters Sees Value in Continuous Testing

    xMatters builds an intelligent communications platform that alerts and connects company stakeholders for IT incident response and in manufacturing line workflow support and DevOps systems scenarios. Customers who count on xMatters also count on the effectiveness of the vendor’s continuous testing mechanisms: A failure of continuous testing early on at xMatters could lead to a magnified impact on the customers’ crises response, resulting in costly repercussions.

    XMatters uses continuous testing to promote continuous delivery of its intelligent communications platform, delivering changes to a limited number of features in an accelerated fashion. “This reduces our risks of failures and makes it easier to resolve defects,” said Deepa Guna, QA Architect at xMatters.

    For xMatters, the proper use and placement of continuous testing in the DevOps pipeline is critical to ensure the intelligent communications solution does not introduce errors during customer crises.

    xMatters Software, Languages and Tools

    The xMatters platform consists of a service stack architecture with front-end technologies accessible via the web and back-end technologies that support it. According to Guna, the platform takes advantage of Java, RESTful JSON, Node.js and Postgres, which run on data centers that are geographically dispersed in North America, Europe and the Asia Pacific regions.

    “Our data centers are built on infrastructure as a service, which includes technologies such as OpenStack, Puppet, Fabric and Consul,” Guna said. XMatters also has a mobile application for iOS and Android platforms as part of its communication service.

    xMatters Testing Approaches, Before and After DevOps

    The company’s testing before the advent of the continuous delivery pipeline included manual test execution of regression tests in the midst of a slow process of releasing one large set of features per month and deploying customer-reported defects, Guna said.

    Today, xMatters uses automated and manual regression tests and unit test code coverage monitoring, releasing small sets of features and resolved defects weekly while enabling customers to provide feedback early in the feature development process.

    XMatters uses several tests during the development, test and release phases of the DevOps pipeline and when delivering the platform to the customer. The development phase includes the following testing tools and their associated types of testing:

    • SonarQube analyzes the quality of the code.
    • Jacoco measures the code coverage of the developer unit/integrations tests.
    • Git + Jenkins run the unit and integration tests before xMatters commits the changes.

    The testing phase, Guna said, runs tests as follows:

    • Jenkins deploys artifacts in test environments and runs automated tests.
    • An automation framework that includes Jmeter, Gradle, Java, TestNG and Selenium runs its tests.

    The release phase tests software after this manner:

    • Jenkins deploys artifacts in test environments and runs automated tests.
    • An automation framework that includes Jmeter, Gradle, Java, TestNG and Selenium runs its tests.
    • Splunk and Takipi do exception monitoring.

    When xMatters delivers the platform to its customers, Guna said, it performs the following tests:

    • Runscope monitors the health of the system to ensure critical services are running.
    • Pingdom monitors the status of the customer websites.

    Drilling Down into Release Testing

    “During the release testing phase, we put the application through 1,000 fully automated and manually initiated tests that we execute with every release cycle. We also have tools that are monitoring for exception during the release testing,” Guna said. Passing this phase successfully, the new software platform release is ready to be pushed into the production environment.

    XMatters uses the continuous delivery pipeline to deploy the new platform release first to the customer staging environment and next to its production environment, Guna said. “All production environments have ongoing smoke tests that we execute every five, 10 or 15 minutes, based on customer tier, to monitor the health of the services.”

    Results Metrics

    xMatters’ method of continuous testing seems to be working for the vendor. “We have reduced the number of major incidents in production due to application defects by 40 percent this year compared to last year,” Guna noted.

  • Mobile Testing Delivers Big for Pizza

    Mobile Testing Delivers Big for Pizza

    Mobile applications have been disrupting industries worldwide. For some industries, the disruption focuses on business internals, as enterprise apps enable manufacturing processes to be restructured and streamlined. In other industries, the disruption focuses on the customer and how mobility can make life more convenient.

    When it comes to pizza, however, mobile development changes everything.

    I remember how, as a kid, we would hope the teacher wasn’t looking when we used the public payphone to place an order for a pizza to be delivered a few minutes after we raced back home for lunch.

    Now you can do it with the click of a button. Using a mobile application, you can order exactly the type of pizza you like, and it will be delivered to your location whenever you want.

    There are few businesses better-suited for a perfectly working mobile app.

    When we were asked by a multibillion-dollar pizza giant to deliver on its mobile testing, we knew the pressure would be on. The company needed a new quality assurance strategy to handle a mobile app that would direct traffic to its more than 3,500 pizza parlors located within the United States alone.

    The Big Step: From Manual to Test Automation

    At the onset of mobile testing, manual testing was the primary method. As the mobile universe expanded to thousands of devices, a more efficient way had to be developed to support business-critical apps being used by people with different devices and mobile operating systems running over different local networks.

    Enter the dedicated mobile testing lab.

    DevOpsPizza2

    Utilizing our cloud testing solution, a lab of 60 devices was connected to the two mobile testing teams located in the United States and India. Mobile testers had access to the devices as they needed them. With a test automation solution in place, more than 200 scripts were written and deployed to handle the regression testing.

    Every device that wasn’t actively being used by a tester would be busy conducting regression tests. The 60 devices running at the same time enabled parallel execution, wherein an app could be tested over a matrix of devices and mobile operating systems. With our performance testing solution, different network conditions were simulated, expanding the overall mobile test coverage exponentially.

    A dedicated mobile device lab meant that the devices were made available exclusively to the business. This prevented any unnecessary breaches in security. At the same time, all remote mobile testing was conducted within the customer’s virtual private network (VPN), bolstering security even more.

    One of the biggest time-savers was the ease in which test cases could be re-created. As a DevOps solution, both the developers and the testers had access to the same devices. So when a specific bug was reported on a function that wasn’t executing properly over this device running that operating system, the developer didn’t have to worry about finding that device with the right operating system. He simply read the report, clicked the link and accessed the same device in the dedicated lab. The test cases did not have to be re-created because they were available in real time to the developer.

    Mobile testing became a 24/7 operation. Right as the American team was checking out for the day, the Indian team was running the reports on their mobile testing activities, and continuing their work. At the end of the day in India, the Americans came back to work to read the Indian team’s testing results.

    Mobile Testing Produces Sizzling ROI

    DevOpsPizza1

    By upgrading their regression testing from a manual process to an automated one with parallel execution through the mobile lab, the regression testing cycle went from 10 days to 17 hours. The testing coverage also increased to 90 percent for the native mobile app.

    As a result, the company was able to get the latest version of its mobile application to market sooner. The impact was felt beyond the DevOps team as the entire IT department began to plan their strategy based on the dramatically shorter release cycle.

    “What makes mobile testing a DevOps success is test re-creation and CI integration.” says Tal Barmeir, Experitest CEO.

    The customer uses Jenkins for its CI platform, uniting developers and testers toward automating as much of the DevOps process as possible. Granting the developers and testers access to the same devices established a powerful tool for test re-creation.

    Their entire DevOps process was retooled to generate returns that truly deliver!