devopsy.kinsta.cloud

Author: Cynthia Dunlop

  • What Quality Metrics Matter Most for DevOps?

    What Quality Metrics Matter Most for DevOps?

    The way we develop and deliver software has changed dramatically in the past five years—but the metrics we use to measure quality remain largely the same. Despite seismic shifts in business expectations, development methodologies, system architectures and team structures, most organizations still rely on quality metrics that were designed for a much different era.

    Every other aspect of application delivery has been scrutinized and optimized as we transform our processes for DevOps. Why not put quality metrics under the microscope as well?

    Are metrics such as number of automated tests, test case coverage and pass/fail rate important in the context of DevOps, where the goal is immediate insight into whether a given release candidate has an acceptable level of risk? What other metrics can help us ensure that the steady stream of updates don’t undermine the very user experience we’re working so hard to enhance?

    To provide the DevOps community an objective perspective on what quality metrics are most critical for DevOps success, Tricentis commissioned Forrester to research the topic. The results are published in a new eBook, “Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics—Ranked by Industry Experts.”

    The goal was to analyze how DevOps leaders measured and valued 75 quality metrics (selected by Forrester), then identify which metrics matter most for DevOps success. Here’s a look at the process:

    1. Survey 603 global enterprise leaders responsible for their firms’ DevOps strategies.
    2. From that sample, identify the firms with mature and successful DevOps adoptions (157 met Forrester’s criteria for this distinction).
    3. Learn what quality metrics those experts actually measure and how valuable they rate each metric they regularly measure.
    4. Use those findings to rate and rank each metric’s usage (how often experts use the metric) and value (how highly experts value the metric).
    5. Compare the DevOps experts’ quality metric usage versus that of DevOps laggards. If there was a significant discrepancy, the metric is considered a DevOps differentiator.

    The 75 DevOps quality metrics were divided into four categories:

    • Build
    • Functional validation
    • Integration testing
    • End-to-end regression testing

    For each category of quality metrics, we came up with a heat map showing usage versus value rankings. For example, here is the heat map for the Build category metrics.

    We also plotted the data for each metric into a quadrant with four sections:

    • Value Added: Metrics that are used frequently by DevOps experts and consistently rated as valuable by the organizations that measure them.
    • Hidden Gem: Metrics that are not used frequently by DevOps experts but are consistently rated as valuable by the organizations that measure them.
    • Overrated: Metrics that are used frequently by DevOps experts but not rated as valuable by the organizations that measure them.
    • Distraction: Metrics that are not used frequently by DevOps experts and not rated as valuable by the organizations that measure them.

    For example, here is the quadrant for Build category metrics:

    The eBook provides both heat maps and quadrants for all four categories, a quick look at each of the 75 metrics per-category and overall analyses, as well as a few fun lists.

    Here’s a preview:

    Hidden Gems

    The following metrics are not commonly used (even among DevOps experts), but are ranked as extremely valuable by the select teams who actually measure them:

    1. New defects (IT)
    2. Critical defects (FV)
    3. Automated tests prioritized by risk (Build)
    4. Code coverage (Build)
    5. Test cases executed (Build)
    6. Static analysis results (Build)
    7. Variance from baselines of percent of test cases passed (E2E)
    8. Release readiness (E2E)

    Top DevOps Differentiators

    DevOps experts/leaders measure the following metrics significantly more than DevOps laggards measure them:

    1. Automated tests prioritized by risk (Build)
    2. Percent of automated E2E test cases (E2E)
    3. Risk coverage (IT)
    4. Release readiness (FV, IT, E2E)
    5. Test efficiency (FV and IT)
    6. Requirements covered by tests (Build, FV, IT, E2E)
    7. Test case coverage (Build, E2E)
    8. Static analysis results (Build)
    9. Variance from baseline of percent of test cases passed (E2E)
    10. Test effectiveness (FV, IT, E2E)

    Most Used by DevOps Experts

    The following metrics are the most frequently used (overall) by DevOps experts/leaders:

    1. Test case coverage (E2E)
    2. Pass/fail rate (FV)
    3. API pass/fail rate (IT)
    4. Number of tests executed (E2E)
    5. API bug density (IT)
    6. Requirements covered by tests (FV)
    7. Requirements covered by tests (E2E)
    8. Blocked test cases (FV)
    9. Percent of automated E2E test cases (E2E)
    10. Successful code builds (build)

    Most Valued by DevOps Experts

    The following metrics are the most highly valued (overall) by DevOps experts/leaders:

    1. Requirements covered by API tests (IT)
    2. Percent of automated E2E tests (E2E)
    3. Requirements covered by tests (E2E)
    4. Requirements covered by tests (FV)
    5. Count of critical functional defects (FV)
    6. Total number of defects discovered in test (E2E)
    7. Number of test cases executed (E2E)
    8. Pass fail rate (FV)
    9. New API defects found (IT)
    10. Automated tests prioritized by risk (build)

    — Cynthia Dunlop

  • CXOs Overestimating DevOps and Continuous Testing Maturity

    CXOs Overestimating DevOps and Continuous Testing Maturity

    More work is needed to achieve continuous testing, Forrester report shows

    Forrester recently completed an interesting new research project exploring what separates DevOps leaders from DevOps laggards. After analyzing data collected from 603 global enterprise respondents responsible for their firms’ DevOps strategies, Forrester learned that firms with the most mature DevOps practices do several key things differently:

    1. Allocate proper testing budgets and focus on upgrading their testing skills.
    2. Implement continuous testing to meet the demands of release frequency and support continuous delivery.
    3. Include testers as part of their integrated delivery teams.
    4. Automate end-to-end functional testing.
    5. Shift left testing to earlier in the development life cycle.

    Firms who follow these five best practices experience testing as a way to gain speed and efficiency, rather than a bottleneck to delivering faster.

    This was not terribly surprising. On the one hand, study after study shows that testing is the No. 1 bottleneck to accelerated application delivery. On the other, we’ve seen firsthand how organizations that commit to transforming testing successfully shift from waterfall-style manual testing processes to automated continuous testing integrated into CI/CD pipelines.

    What caught us off guard, however, was CXOs’ highly inflated estimation of their continuous testing, DevOps and Agile maturity.

    According to the report, only 26 percent of non-CXOs reported that their organization was following all five of these best practices—but 57 percent of CXOs believed that they were. According to the report:

    While most Agile and/or DevOps firms have implemented some of these five core testing practices, few have actually implemented all of them — only about a quarter of respondents say they somewhat or completely agree with all five best practice statements.

    This is an important distinction. While many CXOs and high-level decision makers believe they are ahead of the pack, our research shows that only the small fraction of firms that follow these best practices have more advanced Agile and DevOps practices compared to their peers — and their behaviors and attitudes support that assertion.

    Moreover, CXOs also reported much higher levels of Agile and DevOps maturity across the board than non-CXOs did.

    The good news for DevOps professionals is that CXOs are your champions: They believe you’re doing a great job! The bad news is that, at some point, you need to tell them about the gaps that currently exist in the process. You also need to collect and share some of your own data to make the case that actually adopting these five proven best practices will help you increase delivery speed while reducing business risk.

    Detailed Report Findings

    This was just one of the many interesting findings from research project, which was led by Forrester Vice President and Principal Analyst Diego Lo Giudice.

    As a high-level summary, the research uncovered the following:

    • Understanding of business risk is the most important determining factor of DevOps and Agile maturity.
    • Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).
    • Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

    Detailed results and analysis are available in the complete report: “Forrester Research: What Separates DevOps + Agile Leaders from Laggards?”

    Report Methodology

    In this study, Forrester conducted an online survey of 603 enterprise organizations in North America, Europe and Asia Pacific to evaluate current software testing practices and metrics tracked during survey development. Survey participants included decision-makers and individual contributors responsible for their organizations’ Agile and/or DevOps development strategies. Questions provided to the participants asked about their firms’ attitudes toward software development automation, risk management and testing practices, as well as the metrics they track and value in the software development life cycle.

     

    — Cynthia Dunlop

  • Continuous Testing in the ‘2018 Accelerate State of DevOps’ Report

    Continuous Testing in the ‘2018 Accelerate State of DevOps’ Report

    Here’s a peek into what continuous testing means to DevOps today

    The “2018 Accelerate State of DevOps” report is the brainchild of Dr. Nicole Forsgren, Gene Kim and Jez Humble at DORA (DevOps Research and Assessment). Based on five years of research, with more than 30,000 data points from thousands of companies, the project aims to understand precisely what practices enable teams to deliver better software faster.

    The 2018 report places an increased focus on testing and quality, including a new report section dedicated to continuous testing.  Here are three key report takeaways that are important for continuous testing:

    1. Testing must change—significantly—to keep pace with DevOps

    Consider this:

    • Elite performers deploy code on-demand, multiple times per day.
    • At elite performers, less than one hour elapses between code commit and production.

    The rate of change is accelerating—fast. Testing must somehow keep pace. Evolving testing for DevOps is critical for realizing DevOps’ full potential in terms of accelerated delivery, agility and all the derivative business outcomes outlined in the previous point.

    2. High rates of test automation (~90%) are essential for DevOps

    At elite performers, manual testing represents only 10 percent of the total effort. Even the 30 percent manual testing rate for low performers is significantly better than the industry average of 80 percent+ manual testing (including teams that have not yet adopted DevOps).

    The importance of test automation is underscored by the findings of a recent Forrester report that’s focused specifically on what quality practices separate DevOps leaders from DevOps laggards. That research found that DevOps leaders are:

    • Transforming software testing into continuous testing by adopting five core practices, such as automating end-to-end functional testing and integrating testers into cross-functional teams.
    • Almost twice as likely to consider automating the software quality processto be a “critical business differentiator” (53 percent vs. 27 percent).
    • Significantly more likely (by 23 percentage points) to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).

    3. Quality is correlated to DevOps success

    Elite performers have a 7X lower change failure rate than low performers. They also spend 2X less time fixing field-reported defects.  Exactly what DevOps practices are contributing to the impressive quality outcomes being achieved in the top-performing organizations? And how are different quality practices contributing to DevOps success?  This is certainly an area that’s ripe for further research and analysis.

    Want to learn more about this report’s findings—and what it all means for software testing?

    Join Nicole Forsgren, PhD, CEO and Chief Scientist at DORA and Wolfgang Platz, Founder and Chief Strategy Officer at Tricentis, for a webinar to explore “Continuous Testing’s Role in DevOps success.” The focus will be on helping IT leaders understand:

    • Why and how high-performing teams make Continuous Testing a linchpin of their DevOps process.
    • What specific strategies and practices enable high-performing teams to deliver faster and dedicate more resources to innovation.
    • Which quality-related technology and process investments have the greatest impact on DevOps success.

    — Cynthia Dunlop

  • Continuous Testing Insights from 2018 DevTest Research

    Continuous Testing Insights from 2018 DevTest Research

    The year is far from over, but there already several interesting DevTest surveys worth your attention. These studies don’t just quantify the obvious; they actually report some unexpected findings regarding how far and how fast we’re advancing, and offer some very specific advice on what’s needed to improve.

    We strongly recommend that you spend some time reading all three of these surveys in their entirety. However, in case you’re short on time (or impatient … or both), we wanted to highlight the findings that are most pertinent for readers practicing or researching Continuous Testing.

    Sauce Labs – Testing Trends for 2018: A Survey of Development and Testing Professionals

    [Read the complete report]

    2018 marks the fourth annual “Testing Trends” report, which is based on a global survey of more than 1,000 technology professionals responsible for developing and testing web and mobile applications.

    Key findings in terms of testing include:

    • 87 percent report that management supports test automation initiatives.
    • 45 percent expect to increase spending on test automation in 2018 (55 percent at large companies).
    • The number of respondents with high levels of test automation dropped to 28 percent in 2018 from 32 percent in 2017 .
    • The release cadence is actually slowing, with hourly deployments dropping to 5 percent from 14 percent and daily deployments dropping to 27 percent from 34 percent.

    In other words, everyone recognizes the value of test automation and most companies are willing to invest in it. However, test automation rates are actually decreasing, while Agile and DevOps adoption are steadily increasing. In the 2017 report, test automation rates increased slightly, and delivery speed also increased slightly. The 2018 reported a similar correlation: Test automation rates decreased, and the release cadence slowed down.

    GitLab – 2018 Global Developer Report

    [Read the complete report]

    This expansive survey polled 5,296 software professionals from around the world. The majority of respondents were software developers or engineers who worked for small- to medium-sized businesses (SMB) in the hardware, services and SaaS industries.

    Testing wasn’t a common topic in this development-focused research, but it did earn a prominent spot in the report. Testing was the No. 1 response to the question, “Where in the development process do you encounter the most delays?” A dubious honor—but not a surprising one. Last year’s DevOps Review polled an entirely different audience and came up with the exact same finding.

    VersionOne – 12th Annual State of Agile Report

    [Read the complete report]

    The 12th edition of the world’s longest-running Agile study found that while 97 percent of the 1,492 respondents’ organizations are practicing Agile, 84 percent report that their Agile adoption is not yet mature.

    Respondents feel strongly that two testing-related items would help them increase process maturity across both Agile and DevOps:

    • 83 percent want end-to-end traceability from business initiative through development, test and deployment.
    • 82 percent want better identification and measurement of risk prior to deployment.

    Respondents also reported a relatively high level of adoption of development testing and “shift left” testing techniques. Adoption levels were reported at:

    • Unit testing – 75 percent.
    • Coding standards – 64 percent.
    • Pair programming – 36 percent.
    • TDD – 35 percent.
    • BDD – 17 percent.

    Testers might also be interested in the survey’s feedback on Agile management tools. Usage rates were reported at:

    • Atlassian Jira – 58 percent.
    • VersionOne – 20 percent.
    • Microsoft TFS – 21 percent.
    • HP (now Micro Focus) Quality Center / ALM – 14 percent.

    The most highly recommended tools were VersionOne, Jira and CA Agile Central. HP Agile Manager, Hansoft and HP Quality Center /ALM were the least likely to be recommended.

    — Cynthia Dunlop

  • A Crash Course in Continuous Testing

    A Crash Course in Continuous Testing

    What better way to start a blog on continuous testing than with this “I Love Lucy” clip, and then an analogy:

    https://youtu.be/8NPzLBSBzPI

    Like Lucy and Ethel, many software testers have been scrambling to keep pace with accelerated processes—then along comes the supervisor saying, “You’re doing splendidly … Speed it up!”

    This ever-increasing pace is just one of four aspects of the “perfect storm” driving the need for continuous testing.

    What is Continuous Testing?

    Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain feedback on the business risks associated with a software release candidate as rapidly as possible.

    Why do we need to rethink our approach to testing?

    The expectations associated with test are changing–but legacy testing platforms aren’t keeping up. Legacy testing platforms take a “heavy” approach to testing. They rely on brittle scripts, deliver slow end-to-end regression test execution and produce an overwhelming level of false positives. As a result, they’ve achieved limited success with test automation. The overall test automation rate is 18 percent, on average–8 percent for enterprises. In a polling question, the audience confirmed that the results of test automation have been so-so, at best.

    Now, software testers are facing increasingly complex applications, delivered at dramatically accelerated paces—and they’re expected to deliver trustworthy go/no-go decisions at the new speed of modern business. More of the same won’t get us there. We need to transform the testing process as deliberately and markedly as we’ve transformed the development process.

    The Differences Between Test Automation and Continuous Testing

    There are three main differences between test automation and continuous testing:

    • Risk
    • Breadth
    • Time

    Risk: Exposing more and more innovative functionality to the user is now a competitive differentiator—but it also increases the number, variety and complexity of potential failure points. Most tests are designed to provide low-level details on whether user stories are correctly implementing the requirements—not high-level assessments of whether a release candidate is too risky to release. Would you instantly stop a release based on test results? If not, your tests aren’t properly aligned with business risks.

    Breadth: Just knowing that a unit test failed or a UI test passed doesn’t tell you whether the overall user experience is impacted by recent application changes. To protect the end user experience, you need tests that are broad enough to detect when an application change inadvertently impacts functionality which users have come to rely on.

    Time: If your organization has adopted DevOps and is performing continuous delivery, software may be released hourly—or even more frequently. In this case, feedback at each stage of the process can’t just be “fast”; it must be nearly instantaneous. If the business wants to minimize the risk of faulty software reaching an end user, you need some way to achieve the necessary level of risk coverage and testing breadth—fast.

    What Are the Key Points of Continuous Testing?

    • Continuous testing’s primary goal is assessing business risk coverage.
    • Continuous testing establishes a safety net that helps the team protect the user.
    • Continuous testing requires a stable test environment to be available on demand.
    • Continuous testing is seamlessly integrated into the software delivery pipeline and DevOps toolchain.
    • Continuous testing delivers actionable feedback appropriate for each stage of the delivery pipeline.

    You can find a more extensive list (14 points) at the What is Continuous Testing page.

    Where Can I Learn More About Continuous Testing?

    Watch the complete continuous testing webinar to get more details on the above topics and also learn:

    • How continuous testing impacts the organization
    • What practices testers need to know to be successful with continuous testing
    • How non-automated practices such as exploratory testing fit into continuous testing

    If you want to learn more, download the white paper, “Continuous Testing vs Test Automation: What Testers Need to Know” or visit the What is Continuous Testing page.

    — Cynthia Dunlop