devopsy.kinsta.cloud

Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

  • Continuous Testing – Accelerated !

    Continuous Testing – Accelerated !

    Except for cases where test duration is the point of the test, as indicated in my prior blog “Isn’t Continuous Testing a Longevity Test Anti-pattern?”, faster continuous testing cycles are preferred because failing early and often is a DevOps goal. There are multiple reasons for this, but it should be evident that faster tests reduce the wait time to get results and since there are typically many development team members looking for results the developer time savings multiplier effect is huge for every minute of test time saved on each CI/CT cycle. When aggregated over multiple code branches a few minutes saved per test can be man-years of time saved over a year!

    So what can be done to accelerate continuous testing? Frankly a thorough answer to this topic could fill a book. Below are some suggestions in a checklist format intended to be useful for DevOps .

    1. Organize the team for speed: When team members and the organization are explicitly selected for CT speed the entire DevOps process will be accelerated.
      • A test architect familiar with the principles of best practices test environments and continuous test design is critical to design and tune continuous testing for speed. The architect establishes the CT strategy and speed goal relative to the coverage goal.
      • A test analysis team reviews test results so no time is wasted on results analysis.
      • Developers and test members are trained on best practices test design and their test scripts are peer-reviewed to ensure they comply with the strategy.

    2. Select powerful test resources: Powerful, dedicated, scalable resources will accelerate all test processes and shorten the time to results.
      • Select the fastest processors, low latency storage and I/O to initialize and execute both the test systems and system-under-test entities quickly.
      • Dedicate resources that have longer initialization times so they can be pre-initialized.
      • Virtualize resources that can be scaled horizontally without sacrificing speed.
      • If affordable, arrange enough resources to run ALL tests in parallel.

    3. Choose test tools that are designed for fast test execution: Look for the following features when you are evaluating test tools. Not only will this save you a lot of headaches in the long term, it will give you the flexibility to design tests that integrate with other DevOps tools.
      • Restful APIs that do not require wait states between commands and responses
      • Priority queuing of test cases
      • Creation of test topologies that can be pre-loaded
      • Ability to spawn out tests in parallel to run on any number of machines
      • The ability to park a test that needs to wait for an event and return to it after proceeding with subsequent tests
      • Caching features that enable initialization and results data to be processed for one test while the tool proceeds to execute other tests
      • Multiple devices, session types and results data processing utilities implemented as an embedded part of the tool and not requiring slow speed scripting
      • The ability to render commonly used test utility functions in a fast execution language.

    4. Organize test schedules and test suites to fail early: The sequence and approach for scheduling of test cases greatly affects the speed of testing. If possible the scheduler should launch all tests in parallel as soon as the build is ready. Otherwise priority schemes are needed.
      • Organize the test scheduler and test suites to avoid any unnecessary tests.
      • Cache initialization and topologies data and resources ahead of the next test execution.
      • Check the test environment before running tests, abort if invalid conditions are detected.
      • Organize tests so the most important ones are run first.
      • Set a failure escape threshold and abort the test if exceeded.
      • Eliminate dependencies between tests so that each one can run standalone or in parallel. Queue the next test while a test is running.
      • Use smart gray box testing methods to limit test selection to the most important tests according to the software change and skip tests that are low risk.
      • Run longer tests in parallel with shorter ones.

    5.Design test cases to fail early: The test cases themselves need to be designed with the strategy to get test results as fast as possible.
    • Design each test case with a single unique test purpose and make it self-contained so it may be scheduled to run standalone or in parallel with any other test case.
    • If a test case has multiple verdicts arrange the highest priority verdicts first and close the test at the first failure.
    • Prune or refactors tests that have a low failure history.
    • Shorten dead-script timers, and make any wait timers conditional so they can expire as soon as the condition is met.
    • Reduce the number of test iterations to fit a time budget.
    • Reduce communication wait times by using the fastest protocols. Optimize the most frequently used utilities.

    1. Arrange test results reduce to minimize analysis wait times: Arrange the test results to be consumable as soon as the tests are run so the system does not bottleneck.
      • Report test results as the tests are run and accumulate the results across parallel test targets.
      • Establish thresholds for aggregated test results and abort the tests and release test resources to be used by other tests if the thresholds are exceeded for any run.
      • Keep track of test timings and adjust the test schedules according the average and peak run times for each test case.
      • Separate test result report processes to persistent resources so no test process is waiting to finish a report and the test process can proceed to the next test in parallel with test results.
      • Correlate the test results to the changes and use trends analysis to help speed up diagnostics and as an input to test selection such that the most failing test cases are given higher weight in future test runs.

    The above is a partial list of suggestions for accelerating continuous testing that have been proven to yield good results for accelerating continuous testing and DevOps.

    At Spirent we think testing has a bright future in DevOps. You can read more about our views at Spirent.com/solutions/devops

    What do you think of these suggestions and do you have others that should be mentioned?

  • Isn’t Continuous Testing a Longevity Test Anti-pattern?

    Isn’t Continuous Testing a Longevity Test Anti-pattern?

    Accordingly to Jim Coplien: “an anti-pattern is something that looks like a good idea, but which backfires badly when applied. “ In DevOps context, continuous testing is an attractive solution for the narrow context of testing specific software components quickly (in cycles measured in minutes or hours) but may be “bad” if the system that the software is a component of has failure modes (E.g., effects of memory leaks) that will only be observed during long term testing over several days and if the time between releases is greater than that. In other words, the pattern of quick test cycles prescribed by continuous testing appears to be a contrary Anti-pattern to the fundamental concept of long duration “longevity” testing. So what are the well formulated positive patterns that are applicable in its stead?

    In my prior blog entitled “From QA to Continuous Testing” I indicated “QA test experts that are most skilled in the best practices of test design are valued because they will produce the best quality products.” QA test experts assert that it is unwise to release a software product that has not been tested for durations comparable to the lifecycle of a release. Only tests that are run for a significantly long time compared to the release cadence are deemed sufficient. Just how long is long enough? This is a subject for further discussion beyond the scope of this short article, but my own observation is that best practices test environments run accelerated level load and impulse load traffic patterns on a build candidate for at least 48 hours or more before a release is determined sufficiently tested for at least limited deployment. Even 48 hours is much longer than typical “continuous testing” cycles. So “what positive continuous longevity testing patterns are applicable in its stead?”

    Continuous Longevity Testing Patterns may sound like an oxymoron but in fact longevity testing is compatible within the context of continuous testing principles. Below I describe three Continuous Longevity Testing Patterns that are sometimes used.

    Pattern 1: Longevity testing time windows are reserved such as weekends. In this pattern the regular Continuous Integration/Test cycles are interrupted during the weekend and longevity tests are only run on the build that was last created prior to the weekend time window. This has the advantage of not requiring any special test resources other than the resources used for regular CT tests and does not require any sophisticated scheduling to orchestrate the test environment. The disadvantage is that the longevity test results are only available weekly, it is hard to diagnose which build caused a longevity failure and problems with release candidates will delay the release for a week until the next longevity test run validates the fixes.

    Pattern 2: Longevity parallel testing is scheduled frequently, perhaps as frequently as every few builds, or perhaps daily. This pattern has the advantages that the regular Continuous Integration/Test cycles are not interrupted any time (they run 24/7), the longevity test results are available for more builds than Pattern 1, which reduces diagnostic complexity and may reduce release delays, but has a disadvantage that separate test resources must be reserved for each longevity test in parallel with the regular CT tests. For complex systems under test with many test targets and lots of builds this can be very expensive so there is a tradeoff between test resource costs, and the cost of delays if not done frequently enough.

    Pattern 3: Intelligent longevity testing is scheduled only when certain conditions trigger the longevity tests to be started according to smart rules that correlate software changes to problems with longevity performance. This pattern has the same advantages as Pattern 2 and has the further advantage that the cost of test resources is usually much less than Pattern 2 if the rules for the triggers don’t fire too frequently. The disadvantage is the entire scheme depends on the creation of a rules engine and rules that understand the risk of changes for long duration testing. If the rules are too lenient then this pattern might allow too many longevity failures to escape but if the rules are too tight then this pattern is not any better than pattern 2 and yet more complex to implement.

    So which Continuous Longevity Testing Pattern is best? The best answer is the one that best matches organizations’ goals for release cadence, cost of testing, and risk.

    So indeed, Continuous Testing is a Longevity Test Anti-pattern but fortunately DevOps infrastructures implemented according to best practices have a variety of complementary Continuous Longevity Testing Patterns to choose from.

    At Spirent we think testing has a bright future in DevOps. You can read more about our views at
    Spirent.com/solutions/devops

    What do you think of these patterns or do you have others that should be mentioned?

     

  • From QA to Continuous Testing

    From QA to Continuous Testing

    My prior blog “DevOps – It’s All About Continuous Testing” posits that QA testers don’t need to fear DevOps because continuous testing QA is a central part of DevOps infrastructures that are implemented according to best practices. That is not to say however that continuous testing is precisely the same as traditional QA testing or that QA test engineers can rest on their laurels in the new world of DevOps and continuous testing.

    Traditional QA testing is a serial process that constitutes a distinct phase that follows the development phase and is conducted by an SQA test team “independent” of the development team. The idea was clearly described by Glenford Myers in his seminal books “Software Reliability – Principles and Practices” and “The Art of Software Testing” published in 1976 and 1979 respectively. According to Myers “4th Axiom [of testing] “It is impossible to test your own program”. Myers felt programmers shouldn’t do any testing, even unit testing. While the application of this axiom to unit testing was not followed by most practitioners in the software industry, the idea of independent software functional and system level testing has been a widely accepted practice for many years. The underlying rationale is that it is “impossible” for the same person (the developer) to objectively test his/her own code due to the inherent bias to want the code to work, while a good QA tester must have the opposite attitude to want to break it in order to uncover bugs in the code. If you follow this logic only “unusual” developers would be able to perform both roles successfully and therefore building a team based on this rare personal quality is essentially not practical.

    Despite the obvious logic of “independent QA test” concept, many very complex software projects without an independent QA test team resulted in extremely high quality products. There are many high quality open source software products that had no independent test organization behind each release version. Meanwhile there are countless examples of software projects that had massive independent test resources that resulted in products that failed miserably. How can this be? Perhaps it is not “who” is doing the test that matters but rather it is the quality and frequency of the testing itself that is important.

    Continuous testing, with its fundamental concept of “fail early and fail often”, is the new QA test method of choice. Sure, testing is still essential and QA test experts that are most skilled in the best practices of test design are valued because they will produce the best quality products. Years of software engineering experiences have demonstrated that developers can learn these skills if they are taught and even “normal” developers can do a great job of testing. Instead of QA testing “after development”, testing is conducted continuously as the software is developed, build-by-build, feature-by-feature. Obviously it is better to find bugs earlier rather than later. So why didn’t we always have continuous testing as part of development processes?

    Several industry developments converged to make continuous testing the winning method for software engineering quality assurance. Numerous “modern” test methods such as agile testing, test driven development, behavior test methods, model based testing, and automated regression testing, to name a few, describe how quality software can be accomplished without the need for an independent QA test team. The advent of Agile development processes and DevOps infrastructures, in which development build-test cycles are accelerated to minutes or hours instead of days or weeks, has accelerated the development and adoption of automation and orchestration tools and methods. There is simply no other way to get a reasonable amount of testing done on each build-test cycle unless the tests themselves and the environment for testing are supported by orchestration tools and reliable regression tests are systematically automated as the product is developed.

    Meanwhile virtualization technologies have enabled a wide variety of product test topologies to be stood up on demand at a speed compatible with the rapid build test cycles and yet not require expensive dedicated equipment configurations. It is this combination of improved test methods, Agile processes, continuous integration speed afforded by automation and orchestration tools, and low-cost test dynamically configurable infrastructures afforded by virtualization that has resulted in DevOps.

    Continuous testing is “the new QA.”

    At Spirent we think testing has a bright future in DevOps. You can read more about our views at spirent.com/solutions/devops

  • DevOps – It’s all about continuous testing !

    DevOps – It’s all about continuous testing !

    I loved reading Steve Wilson’s blog post “Don’t Fear Testing Team. DevOps is Here!” https://devopsy.kinsta.cloud/blogs/dont-fear-testing-team-devops-is-here/  The idea that DevOps  somehow reduces the need for testing  certainly puts fear in the hearts of QA and test teams who may envision the demise of product quality,  reduced test resources and even their very livelihood.  I understand the sentiment of the idea about testing going away, but this is misleading.

    Testing is important to DevOps. Prior to code integration to trunk, developers pre-flight test their code to make sure it doesn’t blow up the trunk. During the CI cycle and code commits from multiple developers integration testing is used to verify merges.  During the continuous testing phase where nightly and weekend regression runs are automated for trunk and release branches it is testing that does the job of finding problems before the software release.  During the development process the verdicts trends from testing are a primary measurement of progress. During the release deployment phase more testing verifies the release packages are ready for release.  Rest assured test team, you are needed.  What DevOps really does is bring testing into the mainstream of development processes, and avoids the problems created by having “big-bang” testing left to the end of the cycle, such as release delays, and quality issues.

    Testing maturity is a key differentiator of best practices DevOps infrastructures. My experiences with DevOps infrastructures over the years indicate that testing maturity is a key differentiator of overall DevOps infrastructure maturity. Many organizations can automate their integrations, builds and delivery processes but still have trouble with the subtleness of test orchestration and automation.  There is a vital role for testing architects and testing teams to offer their expertise for test design, test automation and test case development with DevOps.   Whether the organization is using a test driven development methodology, behavior based test creation, model-based testing or whatever they use, testing is a vital part of the overall DevOps process — not only to ensure verify code changes work and integrate well — but to ensure the changes do not blow up the product. So clearly testing is still an integral part of product development and delivery.

    At my company Spirent we think testing has a bright future in DevOps. You can read more about our views at http://www.spirent.com/solutions/devops