Tag: automation best practices

  • Continuous Testing Practices – Part 3

    Continuous Testing Practices – Part 3

    In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing to help with understanding what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

    This article is part three, which covers the following categories of continuous testing practices.

    • Security tests
    • Pre-flight testing
    • Integration testing
    • Pre-production testing
    • Test in production

    You can read part one here and part two here.

    Security Tests

    • Credentials and secrets needed to operate any test are vaulted.
    • Customer sensitive data in application databases used for testing is obfuscated from misuse by people involved in testing.
    • New security tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
    • Security tests for each DevOps pipeline stage are automated and may be selected automatically according to predefined criteria.
    • Test results that indicate possible security concerns are tagged for security analysis.
    • Dynamic or interactive application security tests exercise the application to check for security vulnerabilities. Results of these tests are delivered to developers through tools and feedback loops native to their organization.
    • Attack patterns, abuse cases and tests are built according to application module profiles.
    • Unit, functional and integration tests around—and especially outside—boundary conditions are tested. Tests include error handling, exception handling logic and negative tests.

    Pre-Flight Testing

    • Products are architected to support modular independent packaging, testing and releases. In other words, the product itself is partitioned into modules with minimal dependencies between modules. In this way the modules can be built, tested and released without requiring the entire product to be built, tested and released all at once.
    • Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance, to ensure the changes will not break the integration branch.
    • Requirements for testing are included in agile backlog planning procedures.
    • Pre-flight tests include unit testing, static code analysis scanning, functional, regression and performance tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested, for use in later stages of the pipeline and subsequent regression tests.
    • Pre-flight tests, conducted prior to integration, include unit testing, static code analysis scanning, functional, regression, performance and security tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested for use in later stages of the pipeline and subsequent regression tests.
    • Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance to ensure the changes will not break the integration branch.
    • Test-driven development (TDD) is used to ensure tests are created ahead of the code to be tested. TDD is most often applied for unit level tests but may also be applied for functional testing. A key advantage of TDD is that it ensures working tests are available for use in all pipeline stages. Without using TDD, there is a tendency for test creation to fall behind the code. Another advantage of TDD is that tests are created without the bias of knowing how the code was written, because it hasn’t been written yet.
    • From a testing point of view, microservices introduce the requirement to validate the contract of each service with any other services that use it. The independence of the microservice or any interservice dependencies need to be well tested. Performance and reliability considerations of operating services over a network need to be verified. If the application changes, affected microservices, or any dependent group(s) of microservices also need to be regression tested.
    • Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
    • A strategy ensures that changes to a database, or changes to an application that uses a database, perform as expected, during each stage in the continuous delivery pipeline.

    Integration Testing

    • Software changes are tested incrementally by a cross-functional team over all stages of the continuous delivery pipeline.
    • During the code commit stage, assessments are conducted on the committed code to validate that the pre-flight test results satisfy acceptance criteria for integration into the integration branch.
    • During the build stage, a select group of tests are run on each build to ensure the integrated build satisfies build integration acceptance testing criterion.
    • Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
    • API contract testing is used to test microservices interfaces.
    • Test code and data for each application are stored and maintained in test containers so they can be managed using the same container creation, orchestration and monitoring capabilities as the application.

    Pre-Production Testing

    • After the integration and build stage, functional, performance, regression, customer acceptance and security tests are conducted to ensure the images resulting from the build or set of builds satisfy functional, performance, security and delivery assessment criteria.
    • Testing during deployment include dark launch strategies such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
    • Release assessment criteria are defined and use test results to decide whether a release meets release acceptance criterion.

    Test In Production

    • Testing during deployment uses strategies that minimize the impact of failures such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
    • Automated tests verify each deployment.
    • Chaos engineering is used for testing the resiliency of applications, deployment infrastructure and pipelines.
    • Tests for disaster recovery and restoration of services, infrastructure and pipelines are conducted periodically.
    • Fire drills on applications, pipelines and infrastructures are used to test procedures for detection and recovery of applications, pipelines and infrastructures.

    What This Means

    Continuous testing (CT) practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer Continuous Testing. This article is part of a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s Continuous Testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.

  • Continuous Testing Practices – Part 2

    Continuous Testing Practices – Part 2

    In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing to help explain what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

    This article is part two, which covers the following categories of continuous testing practices:

    • Test automation
    • Test tools
    • Test infrastructure management
    • Test scripts
    • Test results

    You can read part one here.

    Test Automation

    • Test automation requirements are defined.
    • New automated tests are created for each new feature in compliance with the test strategies for the product or service.
    • Automated test cases are reviewed and tested to ensure they satisfy the purpose of the test.
    • Automated test cases are compliant with test case standards for the product or service.
    • Automated test cases are designed to run independent of each other.

    Test Tools

    • All testing tools in a pipeline are integrated into a common framework which enables orchestration of test resources and automation of test tasks.
    • A test framework supports automation of everything needed for testing including test cases, test tools, applications, test infrastructures, and test data.
    • A test framework can be invoked, controlled and released through an application programming interface (API) so that it can be integrated into a DevOps continuous delivery toolchain.
    • Features provided by test frameworks include capabilities to orchestrate the test infrastructure, select tests, control the execution of tests, monitor tests in progress and report test results, logs and verdict data.
    • A selection criterion is defined for test tools. The criterion includes requirements for coverage, automation features, support, compatibility with the automation framework, compatibility with test progress reporting systems.
    • Test tool software versions are maintained in a source version management system.
    • A common test automation framework is defined for the entire organization to ensure all test tools can integrate and be orchestrated with common test automation practices.
    • Test automation tools are available for testing all features of an application that can be automated.
    • Manual testing tools are available for testing features that must be tested manually. These tools can report results to a common test management reporting system.
    • Test authoring tools support input error checking and features which help encourage scripts development according to best practices. Examples of such features include the ability to create and use a repository of known good test utilities for control and analysis of the system under test.
    • DevOps-ready test tools are employed that easily integrate into a test framework and toolchain and elastically scale on demand vertically or horizontally, to match the varying capacity and workload demands of tests of software changes going through the continuous delivery pipeline.
    • DevOps-ready tools can be orchestrated, scaled, invoked, controlled and monitored from an API.

    Test Infrastructure Management

    • Standards for describing resources and topologies for test configurations are documented.
    • Techniques used to accelerate the time for executing tests are used such as vertical and horizontal scaling of test campaigns.
    • Test resources and topology files are maintained in a software version management system.
    • Orchestration and automation tests are conducted to verify the software that supports orchestration and automation layers satisfy their assessment criterion
    • The test infrastructure used for each stage in the pipeline is a replica of production configuration variations, as much as is practical.
    • Infrastructures for testing are elastic, with the ability to stand up and release (i.e. orchestrate) infrastructure configurations as needed for use with specific tests. Typical techniques that are employed to orchestrate test environments include dynamic infrastructure configuration tools, infrastructure-as-code (IaC) strategies, cloud services, testing-as-a-service (TaaS) providers and use of tools to stand up and release applications and test resources packaged in immutable containers.
    • Database migration, replication and restore tools also need to be tested to ensure they can function quickly enough to keep up with the speed of the DevOps continuous deployment pipeline cadence.
    • Health checks on the test environment are used to reduce the chance of false negative test verdicts that could be caused by test environment failures.

    Test Scripts

    • Test scripts are maintained in a software version management system.
    • Guidelines for test scripts and test suites practices are documented.
    • Test utilities, tests and test suites follow guidelines for documentation, test results reporting, reuse and maintenance practices.
    • Test script peer reviews are conducted to encourage collaboration and to ensure test script best practices are used.
    • Each test script has a unique identification.
    • The test type of the script is identified (for example, sanity test, build test, smoke test, security, acceptance test, conformance test, functional test, unit test, patch test, bug fix test, integration test, regression test, performance test, system test, other).
    • Evidence, including an execution log, demonstrates that each verdict of tests work.
    • Tests determine at least one verdict, which may be ‘Pass,’ ‘Fail’ or ‘Inconclusive.’
    • Verdict and assertions priorities are recorded or reported.
    • Tests are data-driven. Data inputs control execution of the test.
    • Each test script is is tagged with a codified risk level.
    • Failure verdicts reported by the script include a tag for risk priority.

    Test Results

    • Techniques are used to ensure test results analysis keeps up with accelerated tests. For example, test design techniques, analysis tools built into test frameworks, test results dashboards and/or test results analyzers that run in parallel with the tests.
    • Quality assurance goals include specific thresholds for test coverage prior to release such as run-to-plan (RTP) and pass-to-plan (PTP) minimum thresholds. RTP is a measure of how many tests of the total planned tests have completed. PTP is a measure of how many of the completed tests pass. For example, 95% RTP and 90% PTP.
    • Quality goals include a measurement and threshold for reliability. For example, S-curves for pass rates can indicate reliability when the pass rate stabilizes for a period. Example: 95% pass rate for three consecutive test cycles.
    • Time to recover a failed service is routinely tested and used to calculate mean-time-to-recover (MTTR).
    • Monitoring and management tests are conducted to verify monitoring and management capabilities of the pipeline satisfy their assessment criterion.
    • Technical and business metrics are designed to monitor the success of DevOps testing for technical continuous improvement and to justify business cases needed to implement those improvements.
    • Input and output metrics and acceptance criterion exist for each stage in the value stream.

    What This Means

    Continuous testing (CT) practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer continuous testing. This article is the second in a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s continuous testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.

  • Continuous Testing Practices (Part 1)

    Continuous Testing Practices (Part 1)

    In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing (CT) to help those understand what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

    This article is part one, which covers the following categories of continuous testing practices:

    • Continuous testing tenets
    • Leadership
    • Culture
    • Test strategy and plans
    • Test management

    Continuous Testing Tenets

    • The continuous testing principle “shift left” is practiced, in which tests are conducted as early in the pipeline as possible.
    • The continuous testing principle “fail early” is practiced, in which tests are arranged so that the most likely problems are found in the early stages of the pipeline.
    • The continuous testing principle “fail often” is practiced, in which tests are run frequently and with many different conditions such as different variations of operating systems, production topologies, browsers and production versions of services.
    • The continuous testing principle “test fast” is practiced, in which tests are run in quick cycles to get to a verdict as quickly as possible.
    • The continuous testing principle “be relevant” is practiced in which tests are focused on the most important tests and results for each purpose and stakeholder.

    Leadership

    • Leaders establish a definition and vision for software quality.
    • Leaders establish a definition for test coverage.
    • Leaders clarify how everyone’s roles fit with the CT strategy.
    • Leaders establish incentives for CT performance.
    • Leaders proactively sponsor quality assurance testing as a strategic element of their budgets and operations rather than viewing testing as a cost to avoid. This includes budgeting time and money for DevOps test training, test resources, test frameworks, test tools, test management and establishing policies for assessments.
    • Operational and capital expense budgets are planned together with all stakeholders to ensure all stakeholders agree there is enough opex and capex budget for the organization to accomplish quality goals for the budget period being planned.
    • Cost accounting practices ensure the capex and opex expenditures are tracked and made visible to stakeholders.
    • Schedules for CT improvement projects are identified and tracked.
    • Skills development programs provide individuals and teams opportunities to learn CT practices.
    • The performance of continuous quality assurance programs are monitored and made visible to stakeholders.
    • Leaders participate in creating and maintaining guidelines for codifying and tagging business risk priorities.

    Culture

    • Quality is everyone’s concern. Responsibility for attaining high quality software products is shared by everyone on a cross-functional team.
    • Cross-functional team members collaborate for testing. This reduces confrontation and friction between people who find failures and those who fix the code.
    • Dev teams embrace the creation of good tests and analysis of test results.
    • Ops team members contribute to test planning, test creation, test execution and test results analysis.
    • Policies for test creation and test coverage are agreed upon by the cross-functional team including Dev, QA, security and Ops team members.
    • A test architect is a role within the organization. The role has expertise in CT practices and helps cross-functional teams understand and apply them.

    Test Strategy and Plans

    • Test strategies and plans are created for each product or service and reviewed by cross-functional teams including development, QA, Ops and security teams.
    • Test strategies and plans identify the type of tests, and the extent of testing activities, for each stage in the value stream.
    • Test requirements and tasks are included in agile planning procedures for each theme, epic and user story.
    • Test strategies and plans define a test coverage standard, test case standards, and input/output criterion for each pipeline gate.
    • Test plans identify the test cases and all the supporting requirements to implement and execute the test cases as required satisfying the test strategy and test management goals.
    • Test strategies and plans identify the approach for system and acceptance testing.
    • Test strategies and plans identify any tests that are to be conducted in production.
    • Test strategies and plans identify any tests that are to be conducted for security testing.
    • Test strategies and plans identify a target for test automation and manual testing for each type of testing. For example, 100% of all tests in the integration stage and 85% of test in the pre-production stage is a typical goal.
    • The test strategy requires that changes to a database or changes to an application that uses a database have tests defined for the database and the application use of the database.

    Test Management

    • All aspects of test management, such as test planning, test guidelines, test code version management and version management of test results are handled by tools that are well-integrated into the continuous delivery pipeline and any test framework being used for test automation.
    • Schedules for all test tasks are identified and tracked.
    • New features have a scheduled task to ensure test goals are met for each new feature.
    • Each test case has associated attributes that can be used to find tests according to the attributes. For example, an attribute could be type of test and the feature purpose of the test.
    • Test environments and resources needed to satisfy requirements to run each test are identified and available programmatically for scheduling purposes.

    What This Means

    Continuous testing practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer continuous testing. This article is the first in a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s continuous testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.

  • SauceCon 2020 Speaker Profile: Nikolay Advolodkin

    SauceCon 2020 Speaker Profile: Nikolay Advolodkin

    This is the fifth in a series of articles highlighting speakers at SauceCon 2020. SauceCon brings together the global community of Sauce Labs users and continuous testing experts to learn from each other and level up their automated testing and continuous delivery skills. This year’s event will be held April 27-29 in Austin, Texas, and devopsy.kinsta.cloud will be there, broadcasting live.

    SauceCon speaker and automation expert Nikolay Advolodkin says that test automation is insanely hard, and that there are thousands of things we can do to make it really good—or really bad.

    “Learning best practices is a great way to create a solid foundation for test automation success,” said Nikolay. “I want to share all of the things that I’ve discovered through my career to make everyone successful at test automation.” 

    You’d be hard-pressed to find someone more qualified than Nikolay to do that. As a senior solution architect at Sauce Labs and founder of UltimateQA.com, Nikolay educates customers and testing professionals worldwide on testing. Most recently, he was a contributing author to Continuous Testing for DevOps Professionals.

    He was named one of 33 Test Automation Leaders to follow in 2020 by TechBeacon.com. And according to Udemy.com, he is one of the top Selenium WebDriver instructors across the globe, educating more than 50,000 students on the ins and outs of test automation from more than 120 different countries.

    At SauceCon, Nikolay will host a full-day “Best Practices in Automated and Parallel Testing” workshop that takes attendees on a journey through a really poor test that follows multiple anti-patterns. Along the way, Nikolay will help attendees transform this test into a production-ready suite of tests that follows automation best practices: making them extremely robust, fast, atomic and ideal for parallelization. He plans to make this workshop session fun with games that will push the limits of test automation beliefs. 

    When he isn’t teaching people how to be automated software testing masters or revolutionizing the test automation world as we know it, Nikolay is into physical fitness. He said it helps him stay calm and rock in test automation. He also loves animals and the environment and supports both every day with his plant-based lifestyle.

    You can hear from Nikolay and more than 30 other expert speakers at SauceCon 2020, the fourth annual Sauce Labs user conference. To see a full list of presentations or to register, visit the SauceCon website.

    — Rebecca Cramer