devopsy.kinsta.cloud

Tag: automated tests

  • What Quality Metrics Matter Most for DevOps?

    What Quality Metrics Matter Most for DevOps?

    The way we develop and deliver software has changed dramatically in the past five years—but the metrics we use to measure quality remain largely the same. Despite seismic shifts in business expectations, development methodologies, system architectures and team structures, most organizations still rely on quality metrics that were designed for a much different era.

    Every other aspect of application delivery has been scrutinized and optimized as we transform our processes for DevOps. Why not put quality metrics under the microscope as well?

    Are metrics such as number of automated tests, test case coverage and pass/fail rate important in the context of DevOps, where the goal is immediate insight into whether a given release candidate has an acceptable level of risk? What other metrics can help us ensure that the steady stream of updates don’t undermine the very user experience we’re working so hard to enhance?

    To provide the DevOps community an objective perspective on what quality metrics are most critical for DevOps success, Tricentis commissioned Forrester to research the topic. The results are published in a new eBook, “Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics—Ranked by Industry Experts.”

    The goal was to analyze how DevOps leaders measured and valued 75 quality metrics (selected by Forrester), then identify which metrics matter most for DevOps success. Here’s a look at the process:

    1. Survey 603 global enterprise leaders responsible for their firms’ DevOps strategies.
    2. From that sample, identify the firms with mature and successful DevOps adoptions (157 met Forrester’s criteria for this distinction).
    3. Learn what quality metrics those experts actually measure and how valuable they rate each metric they regularly measure.
    4. Use those findings to rate and rank each metric’s usage (how often experts use the metric) and value (how highly experts value the metric).
    5. Compare the DevOps experts’ quality metric usage versus that of DevOps laggards. If there was a significant discrepancy, the metric is considered a DevOps differentiator.

    The 75 DevOps quality metrics were divided into four categories:

    • Build
    • Functional validation
    • Integration testing
    • End-to-end regression testing

    For each category of quality metrics, we came up with a heat map showing usage versus value rankings. For example, here is the heat map for the Build category metrics.

    We also plotted the data for each metric into a quadrant with four sections:

    • Value Added: Metrics that are used frequently by DevOps experts and consistently rated as valuable by the organizations that measure them.
    • Hidden Gem: Metrics that are not used frequently by DevOps experts but are consistently rated as valuable by the organizations that measure them.
    • Overrated: Metrics that are used frequently by DevOps experts but not rated as valuable by the organizations that measure them.
    • Distraction: Metrics that are not used frequently by DevOps experts and not rated as valuable by the organizations that measure them.

    For example, here is the quadrant for Build category metrics:

    The eBook provides both heat maps and quadrants for all four categories, a quick look at each of the 75 metrics per-category and overall analyses, as well as a few fun lists.

    Here’s a preview:

    Hidden Gems

    The following metrics are not commonly used (even among DevOps experts), but are ranked as extremely valuable by the select teams who actually measure them:

    1. New defects (IT)
    2. Critical defects (FV)
    3. Automated tests prioritized by risk (Build)
    4. Code coverage (Build)
    5. Test cases executed (Build)
    6. Static analysis results (Build)
    7. Variance from baselines of percent of test cases passed (E2E)
    8. Release readiness (E2E)

    Top DevOps Differentiators

    DevOps experts/leaders measure the following metrics significantly more than DevOps laggards measure them:

    1. Automated tests prioritized by risk (Build)
    2. Percent of automated E2E test cases (E2E)
    3. Risk coverage (IT)
    4. Release readiness (FV, IT, E2E)
    5. Test efficiency (FV and IT)
    6. Requirements covered by tests (Build, FV, IT, E2E)
    7. Test case coverage (Build, E2E)
    8. Static analysis results (Build)
    9. Variance from baseline of percent of test cases passed (E2E)
    10. Test effectiveness (FV, IT, E2E)

    Most Used by DevOps Experts

    The following metrics are the most frequently used (overall) by DevOps experts/leaders:

    1. Test case coverage (E2E)
    2. Pass/fail rate (FV)
    3. API pass/fail rate (IT)
    4. Number of tests executed (E2E)
    5. API bug density (IT)
    6. Requirements covered by tests (FV)
    7. Requirements covered by tests (E2E)
    8. Blocked test cases (FV)
    9. Percent of automated E2E test cases (E2E)
    10. Successful code builds (build)

    Most Valued by DevOps Experts

    The following metrics are the most highly valued (overall) by DevOps experts/leaders:

    1. Requirements covered by API tests (IT)
    2. Percent of automated E2E tests (E2E)
    3. Requirements covered by tests (E2E)
    4. Requirements covered by tests (FV)
    5. Count of critical functional defects (FV)
    6. Total number of defects discovered in test (E2E)
    7. Number of test cases executed (E2E)
    8. Pass fail rate (FV)
    9. New API defects found (IT)
    10. Automated tests prioritized by risk (build)

    — Cynthia Dunlop

  • It’s Time to End the Automated Tests Arms Race

    It’s Time to End the Automated Tests Arms Race

    I grew up during the end of the Cold War. During that time, I remember hearing that there were enough nuclear weapons to destroy the world 20 times over. That always seemed silly to me. Wasn’t once or twice plenty? Wasn’t it more important to have the fastest and most effective weapons rather than to simply have the most weapons? Before I could devise a satisfactory explanation, “Rocky IV” was released, successfully ending the Cold War.

    Number of Automated Tests:  The New Iron Curtain

    Now, 30 years later, I’m beginning to notice another arms race taking place. At a recent DevOps conference, I saw speaker after speaker boasting about the number of automated tests their respective shops were running daily. Again, I was confused. Those numbers provide no real information. What’s more important than the quantity of tests you are executing is the efficacy of those tests and the speed of execution. Why did no presenter have those numbers? It felt like Cold War braggadocio all over again.

    issues_arms-race_nuclear Why do I care? Shouldn’t I view the embracing of test automation as a good thing, regardless of the metric used to measure it? When creating a culture of automation, test counts do serve a purpose. First of all, It’s a simple metric. Just count the number of tests and dance in the self-exulting joy of your automation prowess. It also shows the power of automation compared to manual execution. So what’s the problem? The reason I care is this: Test-case count, while simple to gather and easy to understand, puts the focus on irrelevant data and ignores more critical information, similar to judging a military on number of weapons alone (yes, I’m forcing the analogy). Number of test cases executed daily isn’t just the wrong metric; it’s the wrong unit of measurement! Trying to determine a development shop’s test automation maturity by counting number of tests executed daily is like trying to determine how many gallons a good developer weighs. It’s just nonsensical data that doesn’t answer the important questions:

    • What percentage of your code base is covered by your automated tests? If you knew that your 5,000 tests only covered 15 percent of your code, would you still stand on a stage at a conference and boast?
    • How long do your tests take to run? If it takes all night to execute the suite, then you can kiss multiple releases a day goodbye.

    Using Proper Metrics, or the Strategic Defense Initiative

    So what are the automated metrics we need to worry about? I have a few favorites, in no particular order:

    • Test execution time: Speed matters. A decrease in testing time allows for an increase in number of test iterations, an increase in features per release, an increase in release frequency, etc.
    • Percentage of code covered: The tool you use to get this number doesn’t matter. The key thing is that you have a standard way to track this metric and whether you are trending upward. This trend gives you the next important metric …
    • Automation progress: How fast is the discipline growing? When are you projected to hit the point where all tests that can be automated have been automated?
    • Defect density: While this isn’t specific to automation, the trend in this metric helps show the benefit of automated testing. Bonus points for having a pre-automation baseline.
    • Number of automated test executions: WHAT? Hey, sometimes you have to parade the military through Red Square to put on a show of strength. This isn’t a bad metric when used in conjunction with the others. But it is not enough by itself!

    Keep on Rockin’ in the Free World

    By focusing on the above metrics and not solely on test execution counts, you will ensure you are moving in the right direction in terms of test automation. Similar to how Gorbachev moved the Soviet Union in the right direction with his policies of perestroika and glasnost. OK, OK. The analogy is officially dead.