devopsy.kinsta.cloud

Tag: continuous testing

  • Continuous Testing Practices – Part 2

    Continuous Testing Practices – Part 2

    In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing to help explain what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

    This article is part two, which covers the following categories of continuous testing practices:

    • Test automation
    • Test tools
    • Test infrastructure management
    • Test scripts
    • Test results

    You can read part one here.

    Test Automation

    • Test automation requirements are defined.
    • New automated tests are created for each new feature in compliance with the test strategies for the product or service.
    • Automated test cases are reviewed and tested to ensure they satisfy the purpose of the test.
    • Automated test cases are compliant with test case standards for the product or service.
    • Automated test cases are designed to run independent of each other.

    Test Tools

    • All testing tools in a pipeline are integrated into a common framework which enables orchestration of test resources and automation of test tasks.
    • A test framework supports automation of everything needed for testing including test cases, test tools, applications, test infrastructures, and test data.
    • A test framework can be invoked, controlled and released through an application programming interface (API) so that it can be integrated into a DevOps continuous delivery toolchain.
    • Features provided by test frameworks include capabilities to orchestrate the test infrastructure, select tests, control the execution of tests, monitor tests in progress and report test results, logs and verdict data.
    • A selection criterion is defined for test tools. The criterion includes requirements for coverage, automation features, support, compatibility with the automation framework, compatibility with test progress reporting systems.
    • Test tool software versions are maintained in a source version management system.
    • A common test automation framework is defined for the entire organization to ensure all test tools can integrate and be orchestrated with common test automation practices.
    • Test automation tools are available for testing all features of an application that can be automated.
    • Manual testing tools are available for testing features that must be tested manually. These tools can report results to a common test management reporting system.
    • Test authoring tools support input error checking and features which help encourage scripts development according to best practices. Examples of such features include the ability to create and use a repository of known good test utilities for control and analysis of the system under test.
    • DevOps-ready test tools are employed that easily integrate into a test framework and toolchain and elastically scale on demand vertically or horizontally, to match the varying capacity and workload demands of tests of software changes going through the continuous delivery pipeline.
    • DevOps-ready tools can be orchestrated, scaled, invoked, controlled and monitored from an API.

    Test Infrastructure Management

    • Standards for describing resources and topologies for test configurations are documented.
    • Techniques used to accelerate the time for executing tests are used such as vertical and horizontal scaling of test campaigns.
    • Test resources and topology files are maintained in a software version management system.
    • Orchestration and automation tests are conducted to verify the software that supports orchestration and automation layers satisfy their assessment criterion
    • The test infrastructure used for each stage in the pipeline is a replica of production configuration variations, as much as is practical.
    • Infrastructures for testing are elastic, with the ability to stand up and release (i.e. orchestrate) infrastructure configurations as needed for use with specific tests. Typical techniques that are employed to orchestrate test environments include dynamic infrastructure configuration tools, infrastructure-as-code (IaC) strategies, cloud services, testing-as-a-service (TaaS) providers and use of tools to stand up and release applications and test resources packaged in immutable containers.
    • Database migration, replication and restore tools also need to be tested to ensure they can function quickly enough to keep up with the speed of the DevOps continuous deployment pipeline cadence.
    • Health checks on the test environment are used to reduce the chance of false negative test verdicts that could be caused by test environment failures.

    Test Scripts

    • Test scripts are maintained in a software version management system.
    • Guidelines for test scripts and test suites practices are documented.
    • Test utilities, tests and test suites follow guidelines for documentation, test results reporting, reuse and maintenance practices.
    • Test script peer reviews are conducted to encourage collaboration and to ensure test script best practices are used.
    • Each test script has a unique identification.
    • The test type of the script is identified (for example, sanity test, build test, smoke test, security, acceptance test, conformance test, functional test, unit test, patch test, bug fix test, integration test, regression test, performance test, system test, other).
    • Evidence, including an execution log, demonstrates that each verdict of tests work.
    • Tests determine at least one verdict, which may be ‘Pass,’ ‘Fail’ or ‘Inconclusive.’
    • Verdict and assertions priorities are recorded or reported.
    • Tests are data-driven. Data inputs control execution of the test.
    • Each test script is is tagged with a codified risk level.
    • Failure verdicts reported by the script include a tag for risk priority.

    Test Results

    • Techniques are used to ensure test results analysis keeps up with accelerated tests. For example, test design techniques, analysis tools built into test frameworks, test results dashboards and/or test results analyzers that run in parallel with the tests.
    • Quality assurance goals include specific thresholds for test coverage prior to release such as run-to-plan (RTP) and pass-to-plan (PTP) minimum thresholds. RTP is a measure of how many tests of the total planned tests have completed. PTP is a measure of how many of the completed tests pass. For example, 95% RTP and 90% PTP.
    • Quality goals include a measurement and threshold for reliability. For example, S-curves for pass rates can indicate reliability when the pass rate stabilizes for a period. Example: 95% pass rate for three consecutive test cycles.
    • Time to recover a failed service is routinely tested and used to calculate mean-time-to-recover (MTTR).
    • Monitoring and management tests are conducted to verify monitoring and management capabilities of the pipeline satisfy their assessment criterion.
    • Technical and business metrics are designed to monitor the success of DevOps testing for technical continuous improvement and to justify business cases needed to implement those improvements.
    • Input and output metrics and acceptance criterion exist for each stage in the value stream.

    What This Means

    Continuous testing (CT) practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer continuous testing. This article is the second in a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s continuous testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.

  • Continuous Testing Practices (Part 1)

    Continuous Testing Practices (Part 1)

    In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing (CT) to help those understand what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

    This article is part one, which covers the following categories of continuous testing practices:

    • Continuous testing tenets
    • Leadership
    • Culture
    • Test strategy and plans
    • Test management

    Continuous Testing Tenets

    • The continuous testing principle “shift left” is practiced, in which tests are conducted as early in the pipeline as possible.
    • The continuous testing principle “fail early” is practiced, in which tests are arranged so that the most likely problems are found in the early stages of the pipeline.
    • The continuous testing principle “fail often” is practiced, in which tests are run frequently and with many different conditions such as different variations of operating systems, production topologies, browsers and production versions of services.
    • The continuous testing principle “test fast” is practiced, in which tests are run in quick cycles to get to a verdict as quickly as possible.
    • The continuous testing principle “be relevant” is practiced in which tests are focused on the most important tests and results for each purpose and stakeholder.

    Leadership

    • Leaders establish a definition and vision for software quality.
    • Leaders establish a definition for test coverage.
    • Leaders clarify how everyone’s roles fit with the CT strategy.
    • Leaders establish incentives for CT performance.
    • Leaders proactively sponsor quality assurance testing as a strategic element of their budgets and operations rather than viewing testing as a cost to avoid. This includes budgeting time and money for DevOps test training, test resources, test frameworks, test tools, test management and establishing policies for assessments.
    • Operational and capital expense budgets are planned together with all stakeholders to ensure all stakeholders agree there is enough opex and capex budget for the organization to accomplish quality goals for the budget period being planned.
    • Cost accounting practices ensure the capex and opex expenditures are tracked and made visible to stakeholders.
    • Schedules for CT improvement projects are identified and tracked.
    • Skills development programs provide individuals and teams opportunities to learn CT practices.
    • The performance of continuous quality assurance programs are monitored and made visible to stakeholders.
    • Leaders participate in creating and maintaining guidelines for codifying and tagging business risk priorities.

    Culture

    • Quality is everyone’s concern. Responsibility for attaining high quality software products is shared by everyone on a cross-functional team.
    • Cross-functional team members collaborate for testing. This reduces confrontation and friction between people who find failures and those who fix the code.
    • Dev teams embrace the creation of good tests and analysis of test results.
    • Ops team members contribute to test planning, test creation, test execution and test results analysis.
    • Policies for test creation and test coverage are agreed upon by the cross-functional team including Dev, QA, security and Ops team members.
    • A test architect is a role within the organization. The role has expertise in CT practices and helps cross-functional teams understand and apply them.

    Test Strategy and Plans

    • Test strategies and plans are created for each product or service and reviewed by cross-functional teams including development, QA, Ops and security teams.
    • Test strategies and plans identify the type of tests, and the extent of testing activities, for each stage in the value stream.
    • Test requirements and tasks are included in agile planning procedures for each theme, epic and user story.
    • Test strategies and plans define a test coverage standard, test case standards, and input/output criterion for each pipeline gate.
    • Test plans identify the test cases and all the supporting requirements to implement and execute the test cases as required satisfying the test strategy and test management goals.
    • Test strategies and plans identify the approach for system and acceptance testing.
    • Test strategies and plans identify any tests that are to be conducted in production.
    • Test strategies and plans identify any tests that are to be conducted for security testing.
    • Test strategies and plans identify a target for test automation and manual testing for each type of testing. For example, 100% of all tests in the integration stage and 85% of test in the pre-production stage is a typical goal.
    • The test strategy requires that changes to a database or changes to an application that uses a database have tests defined for the database and the application use of the database.

    Test Management

    • All aspects of test management, such as test planning, test guidelines, test code version management and version management of test results are handled by tools that are well-integrated into the continuous delivery pipeline and any test framework being used for test automation.
    • Schedules for all test tasks are identified and tracked.
    • New features have a scheduled task to ensure test goals are met for each new feature.
    • Each test case has associated attributes that can be used to find tests according to the attributes. For example, an attribute could be type of test and the feature purpose of the test.
    • Test environments and resources needed to satisfy requirements to run each test are identified and available programmatically for scheduling purposes.

    What This Means

    Continuous testing practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer continuous testing. This article is the first in a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s continuous testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.

  • Using Chaos Engineering to Build Resilient Systems

    Using Chaos Engineering to Build Resilient Systems

    Over the past decade, chaos engineering has become one of the most popular approaches in DevOps. It’s uniquely adapted to complex cloud-based systems and has the potential to succeed where more conventional approaches may not.

    Chaos Engineering Explained

    Traditionally, DevOps teams worked with systems hosted on single servers. They would analyze logs with a fine-toothed comb, looking for potential anomalies. They implemented automated tests to ensure the system worked to spec.  They employed root cause analysis to debug applications.

    Modern systems use cloud-based distributed architectures and are much too complex for this approach to be successful; they contain hundreds, and sometimes thousands of instances.  In many cases, their behavior can be chaotic, making it nearly impossible to predict.

    Because of this, modern systems need to be designed for resiliency from the outset. Chaos engineering was developed to foster resilient system design by deliberately anticipating failure modes.

    Two Steps to Chaos

    Here’s how it works. After characterizing a state of “health” for their system, DevOps teams perform a chaos experiment. Just like physics or chemistry experiments, chaos experiments involve setting up two versions of a system: the experimental version and the control.

    The experimental system is then subjected to a barrage of virtual kicks and punches which could range from injecting faults to disrupting a data center. DevOps teams then pore through the resulting logging data, attempting to understand how the chaos experiment affected the state of the system.

    Chaos engineering has two major differences from conventional DevOps practice. First, while conventional forms of testing supply a controlled input and test for a predefined output, chaos experiments are far more open-ended. They’re designed from the outset to expect the unexpected, requiring analysis of the whole system to give novel insights.

    Second, they are performed on live systems. This makes chaos engineering controversial, as its experiments have the potential to affect real users. Still, it makes it much more powerful than alternative approaches. Chaos engineering encourages resilient design by taking genuine risks with systems in the wild.

    Why Companies Embrace Chaos Engineering

    Chaos engineering has been around for over a decade. In that time, it’s been adopted by several of the biggest names in tech. Netflix, the company that pioneered the approach, is still the most well-known for their use of fault injection.

    In 2008, Netflix was moving from an in-house data center to the cloud. They knew that, as they expanded, their application would undergo massive horizontal scaling.  Because any one of its thousands of instances could fail at any moment, it needed to be resilient.

    They created the first chaos experiment, Chaos Monkey. This worked by randomly turning off instances in Netflix’s live application to check it was fault-tolerant. This practice sounds dangerous, but it encouraged Netflix engineers to design resilient system architectures.

    Chaos Monkey sounds bananas, but it worked. In its first five years of operation, there was only one failure which was promptly fixed by an on-call engineer. That failure was caused by Chaos Monkey itself.

    Facebook is another big company to adopt chaos engineering. They created Project Storm in the wake of 2012’s Hurricane Sandy.  Although the hurricane itself didn’t affect Facebook, it served as a wake-up call. A storm hitting one of Facebook’s data centers could wreak havoc.

    In 2014, Facebook took the step of turning off traffic to one of their data centers. Although no end users were affected, traffic loading went haywire on all sorts of different subsystems.

    Once Facebook had successfully identified this weakness in their system’s resilience, they could preemptively design traffic management software that would be resilient in the face of a live data center shutdown.

    As more and more companies use cloud computing and hyperscale data centers, chaos engineering is likely to become an essential feature of DevOps strategy.

    Applying Chaos Engineering In Your Company

    Chaos engineering is the ideal approach for complex, distributed systems. Chances are, your DevOps team either runs their application in the cloud or is planning to migrate to it.

    Serverless computing, in the guise of systems like Kubernetes and Heroku, is popular with an increasing number of organizations. Now it’s common for systems to contain hundreds or even thousands of instances, hosted in massive data centers.

    So far, we’ve only seen chaos engineering from 10,000 feet. It’s time to look at how you can make it work on the ground.

    First, we need to understand the roles observability and monitoring play in chaos engineering.  To successfully implement chaos experiments, there needs to be a high level of system visibility.

    Dealing with Data

    Marius Moscovici outlines four strategies for extracting valuable insights from data. He discusses the importance of detecting anomalies in your data stream, something a chaos engineer needs to be good at.

    Whether it’s the haywire traffic in Facebook’s Project Storm or the deactivation of Netflix instances, chaos experiments rely on the ability to diagnose unusual patterns in data.

    1. Profiling Normal System Behavior

    In order to detect anomalies, you need to have a good idea of what your system normally does. In their book on chaos engineering, Netflix used the analogy of human vital signs. Just as doctors know that 98.6°F is a normal human body temperature, their IT counterparts need to characterize a “normal” state of “health” for their DevOps system.

    Many log analysis and observability tools in the market offer machine learning capabilities to help achieve this.

    2. Detecting Anomalies Across The Board

    Anomaly detection should be integrated into every step of your data analysis. You need to be able to detect anomalies across the board, not just in places you happen to be looking at. Because chaos engineering, unlike more conventional forms of testing, is designed to reveal new and unexpected information it’s vitally important to have a completely open data strategy.

    3. Data Scientists On the Lookout

    Always be on the lookout for potentially anomalous behavior. Chaos engineering will require your company to take its DevOps strategy to a new level, both in terms of your team’s mindset and the sophistication of their logging tools.

    Because chaos engineering requires you to analyze all available data without making assumptions, it’s necessary to leverage machine learning to extract patterns from your logs.  Chaos engineering requires a scientific mindset and an appreciation of the power of data.

    4. Create Alerts

    Finally, make sure that anomalous behavior is being flagged and that the team is aware when it occurs. If possible, dynamic alerts should be used to avoid false positives and alert fatigue which is common with static thresholds.

    Engineering with Chaos

    More and more organizations are migrating to the cloud and building highly distributed application infrastructures. DevOps teams are being forced to confront chaos and complexity on a daily basis.

    While the complexity of modern IT systems has necessitated chaos engineering, it has also enabled it. In a chaotic world, chaos engineering brings the order of an engineering approach.

  • Grafana Labs Acquires k6 to Add Open Source Load Testing Tool

    Grafana Labs Acquires k6 to Add Open Source Load Testing Tool

    During its online GrafanaCONline 2021 conference today, Grafana Labs revealed it has acquired k6, a provider of an open source load testing tool.

    Raj Dutt, Grafana Labs CEO, said load testing will complement the observability platform Grafana Labs currently provides based on open source Prometheus software. Most of the output of any load testing processes are typically evaluated using a platform that analyzes metrics, he noted.

    The k6 platform has been gaining traction among developers that require access to a tool that makes it simpler to dynamically scale load testing up and down as required versus having to guess how much infrastructure might be required to test an application workload, said Dutt. That capability not only makes it simpler to integrate testing within a DevOps workflow, Dutt added, but also serves to reduce testing costs.

    In general, Dutt noted testing is shifting further left as developers assume more responsibility for quality assurance. The challenge is finding a way to enable developers to automatically incorporate testing within a continuous integration (CI) process that allows them to easily observe how synthetic tests are executing, added Dutt.

    There’s no doubt developers are running more tests before promoting workloads into a production environment as part of an effort to reduce the number of issues that might be encountered when an application is deployed in a production environment. In many cases, the support calls that arise now go directly to developers who now have a vested interest in reducing the volume of those calls.

    The issue that developers are contending with is as they embrace microservices, the dependencies that exist across multiple applications makes it more challenging to successfully deploy those applications. Grafana Labs has been making a case for a cloud platform that makes it simpler to centrally view data collected from instances of Prometheus distributed across multiple application development environments. Prometheus, now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) that oversees the development of Kubernetes, is now being deployed more widely by IT teams to observe legacy monolithic applications alongside emerging microservices-based applications.

    It’s not clear the degree to which organizations are also employing dedicated application testers. In many cases, those testers will surface issues that developers overlooked (or simply chose to ignore) in the face of tight application delivery deadlines.

    In the longer term, however, it’s also apparent artificial intelligence (AI) will be playing a larger role in application testing. Someone will still need to review those tests, but many more rote tasks will increasingly be automated.

    One way or another, the total cost of testing is declining, thanks in part to increased reliance on open source testing tools that, in many cases, developers are simply downloading themselves. Those tools are reducing much of the licensing friction and cost restrictions that developers might otherwise encounter.

    In fact, the easier it becomes to test applications, the better overall application quality will become in the months and years ahead. At at time when organizations have never been more dependent on software, that potential improvement to the overall state of application quality can’t arrive soon enough.

  • Harness Employs ML to Streamline DevOps Workflows

    Harness Employs ML to Streamline DevOps Workflows

    At the online {Unscripted} 2021 conference, Harness today announced an update to its namesake DevOps platform that includes support for feature flags that expose new capabilities to a select number of users for testing purposes along with the ability to prioritize the running of those tests based on the likelihood an application is likely to fail them.

    In addition, Harness is adding a unified pipeline capability that enables DevOps teams to streamline the management of multiple tools that make up a workflow and has updated its cloud cost management module to enable DevOps teams to automatically stop idle workloads. That capability is based on technology the company gained with its recent acquisition of Lightwing.

    Steve Burton, chief marketing officer for Harness, said Harness is now employing machine learning algorithms to identify which tests an application is most likely to fail. That test intelligence module enables DevOps teams to run those tests first versus testing an application only to discover close to the end of that effort that an application will need to be updated and then retested. The amount of time spent on testing is then sharply reduced, noted Burton.

    Overall, Harness is reducing the amount of time and effort spent maintaining DevOps pipeline and workflows, said Burton.

    The cost of maintaining a DevOps platform will naturally vary as the overall environment becomes more complex. It may not be clear how much time a DevOps team is spending on testing, but the faster that process becomes the more likely an application will be thoroughly tested before it is deployed in a production environment. Fixing applications after they are deployed in a production environment is, of course, considerably more expensive.

    At the same time, as the number of applications being built simultaneously increases, the amount of cloud resources being consumed also increases. Developers, however, don’t typically pay close attention to what workloads are consuming which cloud resources, which can result in an unpleasant surprise when the monthly bill from the cloud service provider comes due.

    In general, Burton says machine learning algorithms and other forms of artificial intelligence (AI) will soon become table stakes for any provider of a DevOps platform. The expectation is the platform will routinely employ AI to streamline processes and reduce total costs, noted Burton.

    Many of the manual tasks that DevOps teams regularly perform to maintain a DevOps platform will increasingly become automated as usage of machine learning algorithms becomes more pervasive. It’s not clear to what degree organizations may abandon a legacy DevOps platform to take advantage of those capabilities. However, over time it will become apparent that some organizations are spending a lot more time building and deploying applications than others as the next generation of DevOps platforms, infused with machine learning algorithms, are installed.

    In the meantime, DevOps teams might want to make a list of the manual tasks they perform today as part of an effort to determine how much time they are wasting that might otherwise be spent on tasks that are more valuable to the organization.

  • SAST, DAST, SCA: What’s Best For AppSec Testing?

    SAST, DAST, SCA: What’s Best For AppSec Testing?

    According to the most recent Verizon Data Breach Investigations Report, almost 90% of data breaches are driven by financial gain, up from 71% in last year’s report. Most noteworthy, however, is that cloud platforms are particularly at risk, with web application attacks increasing by 43%. As more information is stored within cloud infrastructures, and as more enterprises rely on web applications to maintain business continuity, security teams should consider how security controls can be integrated to DevOps without impacting productivity. However, the security market is often shrouded by acronyms and buzzwords. With the increase in automated security testing tools, such as SAST, DAST and SCA, it’s important to understand the difference between each of these practices, and when to use them in the development cycle.

    DevOps is the convergence of software development and IT operations that aims to continuously build, test and release software. Understandably, DevOps is becoming an integral part of any business as it facilitates delivering new products to end users faster and driving increased revenue. However, failure to secure these digital assets can open organizations to cyberattacks resulting in operational downtime, loss of customer trust and even hefty fines from a regulatory body. The sharp rise of application layer attacks proves that enterprises must proactively address security issues earlier in the software development stage.

    The evolution of DevSecOps takes the concept of DevOps one step further, putting security where it should be: at the heart of the development life cycle, using automation. DevSecOps has helped security professionals understand how they can successfully integrate automated security processes without slowing down development while allowing DevOps to focus on what matters most – building and creating software.

    There are a host of automated application security testing tools out there, each with unique benefits, that are commonly used in the agile software development cycle. This raises the question: what tools should we use, and when, to ensure a secure software development life cycle (SDLC)?

    The Difference Between SAST, SCA and DAST

    The most popular application security testing tools businesses implement in their development cycles are static application security testing (SAST), software composition analysis (SCA) and dynamic application security testing (DAST). Knowing the differences between them and when to use them is crucial to enhance your DevSecOps strategy and secure your web applications.

    Static Application Security Testing (SAST)

    As the pressure to deliver high quality and secure code increases, and DevOps embraces new agile methodologies and technologies, integrated static analysis within the continuous software development pipeline can help detect vulnerabilities early in the software development process from the inside out (white box testing). The benefit of a SAST solution is it doesn’t require a running system to perform code security checks and won’t slow you down like a manual code review or penetration test.

    One aspect that makes SAST unique is that it automatically checks for vulnerabilities within the application source code, byte code or binaries, line by line. This ensures security weaknesses – such as those listed in the OWASP top 10 and the 2019 CWE top 25 – in the proprietary code are detected from the start and during development. As a result, developers can identify the risks and fix any vulnerabilities before they make it into production. They are empowered to shift left and recognize early code defects in the development cycle, a security best practice, which also reduces the cost of fixing security issues.

    However, SAST does have some disadvantages in that it requires access to source code and the underlying framework, making it unsuitable for testing commercial off-the-shelf software (COTS) or working with outsourced development. It is also unable to identify runtime and environment-related issues, which makes it difficult to determine the actual and contextual risks of security flaws and suggest meaningful remediation.

    Dynamic Application Security Testing (DAST)

    Unlike SAST, dynamic application security testing (DAST) is done from the outside in (black box testing), and identifies vulnerabilities when the application is running. DAST tools crawl web pages, locate web services endpoints, inputs and outputs; it requires a working version of a web application for the testing to work. Without looking into the source code, dynamic analysis works to simulate penetration testing to uncover exploitable vulnerabilities and business logic issues from a hacker’s point of view, with reliable results.

    Yet, because this scanning occurs towards the end of the SDLC (running applications), the findings can be substantial and often put additional pressure on DevOps teams to fix these runtime vulnerabilities on short notice, creating friction between the security and development teams. Further drawbacks of DAST include the risk that vulnerabilities can be found too late in the SDLC, resulting in rushed or delayed remediation.

    Software Composition Analysis (SCA)

    Releasing new features on a deadline means that developers often don’t write all their own code. The use of prepackaged open source code is commonplace in modern DevOps practices, and so is the need for security governance. Some SAST solutions now include software composition analysis (SCA) functionality to locate weaknesses in proprietary code and vulnerabilities in open source code.

    SCA tools are designed to analyze the open source components by detecting software licenses, deprecated dependencies and known vulnerabilities and potential exploits in a codebase. SCA enables DevOps teams to manage their security exposure and license compliance while still allowing the use of prepackaged code to speed up development processes. In addition, composition analysis can be run and expanded to newer architectures, including containerized environments, to automate detection for publicly disclosed vulnerabilities within your containers and prevent those disguised in public registries, such as Docker Hub, from being brought into the project.

    With open source components now present in 96% of commercial applications, the ability to spot software supply chain vulnerabilities in development, in and out of the cloud, are vital to reduce web application risks – all of which can be a potential entry point for malicious actors, as we’ve learned from the Equifax and SolarWinds breaches.

    Choosing the Best Application Security Testing Tool

    There is no shortage of application security testing tools. The real conundrum arises when decision makers are presented with a host of various options and pressured to choose. However, these tools are not mutually exclusive. Instead, they form a holistic, almost linear, approach to the entire DevOps pipeline from plan to implementation. What does vary, however, is the application development model. Which tool(s) should you choose for your specific use case? Here are some general guidelines:

    Outsourced Development and Commercial off-the-shelf (COTS) Applications: When it comes to critical business applications for prepackaged code, black box testing is advised for enhanced security peace of mind. Without access to the source code, DAST scanning and/or penetration testing are ideal to test the running application for any critical software vulnerabilities before deployment and to monitor security compliance in the long run.

    Internal Development and Software Customization: For businesses with the resources and infrastructure to create applications in-house, it’s important to build security testing into the SDLC from the beginning. Doing so can help protect against application vulnerabilities including cross-site scripting. For true DevSecOps, arm your development teams with SAST/SCA tools to ensure code security in the ‘code and build’ stage, and apply black box testing (DAST) for security assurance before deployment. Once deployed, regular penetration testing and continuous vulnerability assessment of the underlying infrastructure are also recommended to keep security issues in check with each new release.

    As cyberattacks targeting web applications increase, it is essential for organizations to have complete application security across the SDLC, providing maximum security coverage for development and ensuring they can keep up with business operations.

  • Applause Delivers Testing Integration Tool

    Applause Delivers Testing Integration Tool

    Applause today made a bi-directional integration capability that simplifies requesting application code tests of, dubbed Applause In-Sprint Testing, generally available.

    Kristin Simonini, vice president of product for Applause, said application owners will be able to, for example, use a Jira project management application to create a ticket that requests a specific piece of code be tested either by a member of dedicated testing team or by a developer that has assumed greater responsibility for application testing within the context of a DevOps workflow.
    Alternatively, that request could be sent to a tester that is a member of uTest, an online community of professional testers that Applause oversees.

    The company also published the results of a global survey of 1,800 quality assurance (QA), product, engineering and DevOps professionals that found 44% of development teams that have shifted application testing left are spending a full workday every week on application testing. More than half the respondents (52%) noted that amount of time spent on testing is impacting developer productivity.

    On the plus side, a full 88% of respondents that release code multiple times per month said their organizations now test features as they are developed versus waiting until the end of application development process, the survey also found.

    The major benefits of shifting testing left cited by respondents are reducing the number of bugs released into production environments (57%), reducing the cost of late-stage bug fixes (48%), and reducing the need for hot-fixes (39%).

    The top two barriers to shifting responsibility for testing left toward developers is they are too busy coding (45%) and that development and quality assurance (QA) efforts remain siloed (37%). Nearly half of developers (44%) said they have issues with context switching as they shift between coding and testing tasks.

    Simonini said Applause In-Sprint Testing will make it easier for application owners to decide when they want to make use of a professional tester versus having a developer run a test. The Applause Product Excellence Platform then enables IT organizations to leverage a software-as-a-service (SaaS) platform to record a series of test case scenarios that can be automatically converted into an automation script without anyone having to write code.

    The overall goal is to simplify testing at a time when the number of applications being built has accelerated in part because of the increased focus on digital business transformation that occurred in the wake of the economic downturn brought on by the COVID-19 pandemic. As organizations realize how dependent they have become on those applications to differentiate themselves, the focus on quality assurance naturally intensifies, noted Simonini.

    There’s no doubt more organizations are testing with greater frequency. The efforts of developers that previously tended to address large numbers of bugs as part of a series of application updates after deployment in production are now being frowned upon. Organizations realize their applications only get one chance to make a first impression with end customers. As a result, the level of attention to detail being applied to application testing has, for better or worse, never been greater.

  • Continuous Testing: The Quest for Quality at Speed

    Continuous Testing: The Quest for Quality at Speed

    The DevOps goal to reduce lead times by accelerating software deliveries presents a serious challenge for assuring the quality of the software that is delivered. As I indicated In my SlideShare post, “The Quest for Quality at Speed,” testing activities span end-to-end software value streams and often account for more than 50% of bottlenecks.

    The answer to these DevOps testing challenges is referred to as continuous testing (CT), which is more than simply automating tests. My blueprint for continuous testing identifies eight pillars of practices, each with strategies to reduce bottlenecks. In this article, I enumerate the strategies for each of the eight pillars of continuous testing and explain how to approach implementation for greater adoption of CT across an organization.

    Continuous Testing Pillars

    Continuous Testing Tenets

    There are five tenets which are guiding principles that enable continuous testing:
    • Shift Left: Conduct each test as early in the pipeline as possible.
    • Fail Early: Arrange the tests so the most likely problems are found during the earliest possible stage in the DevOps pipeline.
    • Fail Often: Run tests frequently and with many different conditions.
    • Test Fast: Arrange tests to run in quick cycles.
    • Be Relevant: Focus on the most important tests and results.

    Leadership and Culture

    Leadership and culture practices have a major impact on influencing behaviors and habits of everyone involved in testing across the DevOps value stream. Bottlenecks to CT include inadequate funding of testing due to lack of quality vision and goals at the executive level, and bottlenecks caused by distrust and lack of cooperation between Dev, QA and Ops. Executive sponsors need to clarify the definition of quality for the organization, and emphasize quality’s importance. Executives can communicate the vision and goals for quality and establish incentives that foster a culture of collaborative behavior.

    Test Strategies and Plans

    Test strategies and plans are needed to guide testing approaches for each product/service pipeline. Bottlenecks to CT include confusion caused by lack of product/service test strategies, and unplanned testing activities and resources are not included sufficiently in planning process. It is important to establish test coverage standards, test case standards, gate criterion and planning process that include testing requirements. CT strategies include making sure test coverage requirements are defined, all types of tests required are identified, allocate tests that are required at input gates, indicate test results are required at output gates and enforce that test tasks are included in planning backlogs and sprints.

    Test Management

    Test management is needed to coordinate testing resources and test schedules. Bottlenecks to CT include poor utilization of test resources due to lack of visibility and schedule bottlenecks due to lack of coordination of test schedules. It is important to have a database of tests and test resources, and test scheduling tools. CT strategies include version management of tests and resources, an inventory of test resources and dynamic test scheduling.

    Test Automation

    Designing, creating, selecting and executing tests in an automated fashion to get to a verdict as fast as possible is the primary strategy to accelerate testing. Bottlenecks to CT include slow test creation, slow test execution and slow time to realize verdicts. It is important to have effective automation strategies for test design, test creation, test selection, and test execution. CT strategies include test design templates, advanced test creation tools (E.g., TDD, keyword, behavior, model-based), dynamic test selection, tests designed for fast verdicts and horizontal and vertical scaling tests.

    Test Tools

    Test tools are designed to automate specific types of tests and integrate with DevOps platforms through APIs for orchestration, execution and reporting. Bottlenecks to CT include test tools that are not designed to scale, limited test configurations and lack of complete APIs, which make integration with DevOps platforms inefficient and error-prone. It is important to have scalable test tools with good APIs. CT strategies include limiting the selection of test tools to ones that are DevOps-ready or DevOps-native and that have well-defined, stable APIs.

    Test Environment Management

    Scheduling and setting up resources into the needed topologies for running different types of tests for developers and testers can be a major bottleneck if not managed efficiently. Bottlenecks to CT include lack of test resource scheduling, which causes conflicts for resources, manual test setups, which are time consuming, causing long wait times for testing and delayed retesting. It is important to have test resources scheduling, test environment orchestration and resource utilization management. CT strategies include test environment inventory tools, test environment sharing tools, test environment orchestration tools and monitoring test environment utilization.

    Test Results Analysis

    Reporting and analyzing test results can be a major bottleneck after test execution is completed. Bottlenecks to CT include poorly written tests which can report confusing results and that take a lot of time to analyze. It is important to have design standards for test results and test results reporting standards. CT strategies include test verdict design in test cases, common test reporting tools and test analysis tools.

    Steps To Implement Continuous Testing

    Start with one or two services and a small set of critical CT practices, then expand to more services and more advanced CT practices.

    A gap assessment is a great way to determine the current state of CT practices, and will help focus the implementation plan. Given a list of practices from each of the pillars in the continuous test blueprint, a cross-functional team of developers, testers and ops people can assign scores to measure importance and current level of practice to generate a gap score. The combined set of gap scores will give guide and direct current priorities for the CT plan.

    For example, beginning CT priorities could be to establish a strategy sponsor, define quality goals, obtain an inventory of test resources and requirements, put in place version management for tests and test standards, tool selection and test refactoring to match the standards.

    An intermediate CT goal could be to complete tool integrations with DevOps platforms, automate test scheduling, orchestrate test environments and optimize test results reporting.

    With maturity, advanced CT goals could be to put in place advanced test creation strategies, accelerated test execution, automated test environment management and optimize test results analysis.

    What This Means

    Testing activities are typically a major source of bottlenecks that slow the quest for quality at speed. Continuous test strategies that accelerate all aspects of testing can have a major impact on speeding up lead time for continuous delivery of quality software. Implementation strategies that start by applying specific continuous test strategies to a targeted small number of applications, and then expand to more advanced strategies applied to more applications is an effective approach to CT adoption across the organization.

  • Survey Sees Application Testing Shifting Further Left

    Survey Sees Application Testing Shifting Further Left

    PractiTest, a test management platform provider, published a report today that suggests professional application testers have become more adept at working within DevOps workflows.

    The annual survey of more than 1,000 application testing professionals finds that 80% of all application testing within organizations is being done by test engineers, test leads, testers and automation testers. A full 92% of survey respondents also noted they are working within the context of an agile methodology, while 42% cited DevOps specifically. Half of respondents said they work within the framework of a continuous integration/continuous delivery (CI/CD) platform.

    Significantly, more than half of respondents (59%) said testing is shifting left within their organization, but only a third (33%) said programmers participate in the testing process.

    PractiTest CEO Yaniv Iny said that level of DevOps inclusion suggests that, rather than shifting testing completely left on to the shoulders of application developers, a more nuanced approach to application testing is taking hold as professional testers learn to work within the sprints that make up a DevOps workflow. A full 90% of respondents said testing has become more automated within their quality assurance (QA) processes.

    By now, most application testers have come to terms with automation and, increasingly, AI, said Iny. The assumption now is that large swaths of the testing process will be automated, Iny added. A full 97% of respondents said that functional testing automation was either important or very important for them to succeed, while 96% said test automation patterns, principles and practices are also critical.

    Usage of behavior-driven development (BDD) as a testing model has also increased, with 27% of organizations now employing it to go beyond simply testing functions.

    However, the survey also found that, instead of automation decreasing the size of testing teams, those teams are getting larger. The number of organizations with 16 or more testers increased by 10% to 34% compared to a year ago. That suggests organizations are launching either more complex application development projects or launching multiple projects at the same time as they embrace digital business transformation initiatives, Iny said.

    The survey found nearly 70% of respondents have at least five years of experience. Nearly three-quarters (71%) of respondents reported that their income was not affected by the economic downturn brought on by the COVID-19 pandemic. However, nearly half of respondents (54%) reported that they were concerned about job stability, an increase of more than 30% from a year ago.

    The pandemic did change the way testers acquire new skills. The number of testers reporting that they attended online conferences, meetups and seminars increased to just under half (49.5%), up from 40.5% in 2019. Participation in online communities and forums grew to 44%, while only a third of respondents receiving any kind of formal training.

    Testing professionals with five to 10 years of experience earned an average salary of $89,000 in North America, which compares to $76,000 in Australia/New Zealand, the survey also found.

    Application testing as a distinct discipline doesn’t appear to be fading away anytime soon. Rather, organizations appear to be investing even more in testing to deliver higher-quality applications.

  • The Surprising Secret to DevOps ROI

    The Surprising Secret to DevOps ROI

    As a result of my many years of DevOps consulting, I have discovered a simple, but not well-understood, secret ingredient that drives ROI for all the Three Ways of DevOps.

    Measuring and optimizing return on investment (ROI) for DevOps is not well-understood. It is a proven fact that DevOps, when engineered well, delivers tremendous value to organizations by way of continuous delivery practices. Reduced lead time, more frequent releases, lower failure rates and faster time to restore services are well-accepted indicators of DevOps value. It is not a secret that the combination of engineering DevOps practices for people, processes, and technologies around the Three Ways of DevOps – continuous flow, continuous feedback and continuous improvement – gets results. As valuable as these things are, they are not a measure of ROI, and do not tell you what to do to improve the return on your DevOps investments.

    How Do You Calculate and Optimize DevOps ROI?

    First, it is important to understand that ROI for DevOps is best measured using a simple ratio of return (R) divided by investment (I). (OK – for the accounting purists out there; I know the real ROI formula is actually (((R-I)/I)*100%) but that is a technicality that we don’t need for this argument.)

    The “return” of DevOps value is the important factor in the numerator of the ROI ratio. While there are many values that can be attributed to DevOps, the most direct value that can be measured and justified for legitimate comparison purposes is the cost savings associated with accelerating software delivery value streams. Application developers, and others, can get more applications released faster because of DevOps, which has a dramatic effect on valuable cost reductions. The “investment” in DevOps projects is the denominator of the ROI ratio. To optimize ROI, simply increase return by getting more application releases out faster, while reducing the investment required for a DevOps project. Sounds easy enough. But wait! How do you do that?

    The primary factor in the return is derived from reducing bottlenecks. I have found, in more than half of the thousands of DevOps value streams I’ve studied, that the biggest bottleneck is testing! It is a myth that continuous testing is just the execution of automated tests. I am talking about the whole range of ‘testing things,’ including test strategy, test planning, test resource scheduling, test design, test implementation, test environment management, test execution, test results reporting, test result analysis, troubleshooting, fix, retest and test management. That is a lot! In fact, testing things are a source of bottlenecks for every stage of a DevOps pipeline, for all three ways of DevOps. The point is that ROI can be greatly improved by improving testing.

    As it turns out, the primary factor in reducing investment costs also has to do with testing! Putting ‘testing things’ in place is expensive in terms of labor and capital costs for both people and systems. The primary factor to reduce total investment cost is to standardize, share and use ‘testing things’ with as many application pipelines and stages as possible.

    What is the secret ingredient that I have discovered? After all, the importance of testing itself is not much of a secret. I have isolated the key ingredient that is the heart of all ‘testing things.’ The purpose of a test is to determine a verdict. A test verdict is the codified representation of a pass, fail or inconclusive result state of a test, assessed relative to the purpose of the test. Test verdicts are essential, no matter what kind of test it is; no matter the stage in the pipeline a test is used. Without a verdict, all the ‘testing things’ in the world are not valuable.

    So – the secret ingredient for optimizing DevOps ROI is to optimize test verdicts. Those that know how to optimize test verdicts hold the key to optimizing DevOps ROI.

    As I indicated in this eBook I authored for the DevOps Institute, verdicts from properly engineered testing at each stage in the pipeline are essential for assessing the artifacts produced in each stage.

    How Can You Optimize Test Verdicts?

    As I have pointed out in prior articles, well-engineered DevOps requires balancing the nine pillars of DevOps.
    Surprisingly, test verdicts play a crucial role in optimizing all nine pillars of DevOps, and are vital to accomplishing the best return for the least cost investment in DevOps. Here are some examples of how this works for each pillar:

    Leadership – Leaders tend to underestimate the importance that ‘testing things’ have on DevOps ROI, and that the central ingredient of test effectiveness is the optimization of test verdicts. By sponsoring DevOps projects and training programs that drive improved test verdicts, leaders will have a direct and lasting impact on ROI.

    Collaborative Culture – How often have you seen people wasting time arguing over the meaning and importance of a test? Collaboration between application developers and QA and Ops teams is more effective when test verdicts are clear and based on a common understanding of the purpose and importance of each test.

    Design – The creation of excellent tests, designed to efficiently resolve to clear verdicts during the design phase, makes for reusable test assets in all subsequent stages and releases, and can be reused with other applications.

    Continuous Integration – The validation of builds is faster when all integration tests are arranged to get clear verdicts quickly during the build process.

    Continuous Testing – Test verdicts associated with each test are the key information ingredient that test management systems can key off to optimize test selection and test execution for each stage in the continuous delivery pipeline.

    Elastic Infrastructure – Test environment management (TEM) systems can optimize the orchestration of test resources better when test verdicts are clearly understood. Knowing a history of verdicts for a group of tests indicate which resources and how much of each resource are needed to run each test. This knowledge is essential to drive test environment management systems that have scaling algorithms for optimizing the use of ephemeral test environments, and to ensure test resources are used optimally.

    Continuous Monitoring – Reporting and analysis of test results can be a major bottleneck in the pipeline, but can be dramatically accelerated when the test verdicts are clearly defined.

    Continuous Security – Security tests are not much use without a clear verdict. While there is no such thing as 100% security assurance, the aggregate set of security test verdicts helps to profile security for applications and releases.

    Continuous Delivery – Algorithms that govern the ‘doneness’ of an application release candidate, and decisions to promote a release to production, depend on analysis of service level objectives (SLOs) derived from service level indicators (SLIs), including test verdicts. The fidelity and quality of those verdicts is essential for operations and site reliability engineering (SRE) teams to be confident enough to not bottleneck an application release.

    What This Means

    Measuring and optimizing DevOps ROI requires strategies that deliver value by improving returns through reducing bottlenecks and reducing investment costs. ‘Testing things’ are a dominant factor contributing dramatically to both the return and investment components of the ROI equation. A comprehensive strategy for optimizing test verdicts is the secret ingredient for optimizing and leveraging investment in ‘testing things,’ which, in turn, has a dramatic effect on improving DevOps ROI.