devopsy.kinsta.cloud

Author: Wayne Ariola

  • How the World Measures DevOps Quality

    How the World Measures DevOps Quality

    With continuous everything, knowing whether each new release will ultimately enhance or undermine the overall user experience is essential. Yet, most of today’s go/no-go decision still hinge upon quality metrics designed for a different era.

    Every other aspect of application delivery has been scrutinized and optimized for DevOps. Why not re-examine quality metrics as well?

    Are classic metrics like number of automated tests, test case coverage and pass/fail rate important in the context of DevOps, where the goal is immediate insight into whether a given release candidate has an acceptable level of risk? What other metrics can help us ensure the steady stream of updates are truly fit for production?

    To provide the DevOps community an objective perspective on what quality metrics are most important for DevOps success, Tricentis commissioned Forrester to research the topic. The results are published in the 55-page report, Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics—Ranked by Industry Experts. The report takes a deep dive into the global findings, complete with heat maps, quadrant mappings and some fun lists such as “Most Overrated,” “Hidden Gems” and “Top DevOps Differentiators.”

    One of the most common questions we received after publishing the report was: “How do the results vary across regions?” In response, we performed some additional regional analysis—and I’d like to share those results here.

    To start, let’s take a look at the global top 20. The following metrics were ranked as the most valuable by the DevOps experts who measure them (across all regions).

    Europe DevOps Quality Metrics Trends

    Looking specifically at Europe, the top 20 changes as follows:

    Interesting trends in this region:

    • There is a greater commitment to measuring quality metrics. European respondents reported a higher level of DevOps quality metrics measurement across the board. For almost all metrics, the usage rate was at least 6% higher than the global average. For metrics related to time, coverage, risk, effectiveness and efficiency, the usage rate was over 14% higher. This speaks to European organizations’ commitment to scrutinizing and continuously optimizing their quality processes—especially in terms of time and resource utilization.
    • Risk and coverage metrics are valued more than in the global average. European respondents also ranked risk and coverage quality metrics a surprising 21% higher than the global average. This could be related to the fact that the respondents from this region came primarily from the financial services and insurance sector, with healthcare and government close behind. In such highly-regulated industries, measuring and mitigating risk is certainly a core concern. This finding could also indicate European organizations place a greater emphasis on protecting the corporate brand.
    • Test data preparation time seems to be a greater concern. European respondents were more likely to measure (+18%) and highly-value (+23) time spent preparing test data than their global peers. Given the restrictions GDPR placed on test data as of May 2018, it seems likely that European organizations have significantly changed their test data management processes (e.g., to masking and more synthetic test data generation), and are cautiously monitoring how the changes are impacting their overall efficiency.

    Asia Pacific DevOps Quality Metrics Trends

    Now, let’s shift focus to Asia Pacific. The following 20 metrics were ranked as the most valuable by the Asia Pacific DevOps experts who measure them:

    Notable trends in this region:

    • End-to-end testing metrics are valued—and measured—more than in the global average. Although Asia Pacific respondents measured fewer build and functional validation metrics than the global average, they measured (and valued) end-to-end testing metrics much more than their peers around the world. For example, percent of automated end-to-end tests was measured by 47% of the organizations (versus 36% globally) and highly-valued by 84% (versus 70% globally). Risk coverage measurement was significantly higher; it was measured by 49% (versus 34%) and highly-valued by 71% (versus 59%). This speaks to the region’s focus on digital transformation and commitment to delivering exceptional user experiences.
    • API testing metrics were also valued—and measured—more than in the global average. Asia Pacific respondents also measured and valued API testing quality metrics more than the global average. Overall, API quality metrics were measured by 16% more organizations in this region than globally. The highest valued API quality metrics were API test coverage (63% versus 39% globally) and API risk coverage (79% versus 62% globally). This prioritization of API testing is likely a side effect of the regional trend towards API-driven open banking (the majority of respondents indicated they were in the financial services and insurance sector).
    • There is a greater leader/laggard quality metrics measurement gap. Part of the study involved classifying the respondents as either DevOps leaders or DevOps laggards, based on their responses to various questions about the maturity of their processes. Although the percentage of DevOps leaders in the region was lower than the global average (18% versus 26%), the DevOps leaders from Asia Pacific generally measured quality metrics at a comparable rate to their global peers. However, the DevOps laggards in Asia Pacific generally measured quality metrics at a much lower rate than their global peers. This suggests the select set of firms that have truly prioritized DevOps initiatives have made great strides—and the laggards have a lot of catching up to do in order to remain competitive.

    Research Methodology

    Here’s a look at the process behind the research discussed above:

    1. Survey 603 global enterprise leaders responsible for their firms’ DevOps strategies.
    2. From that sample, identify the firms with mature and successful DevOps adoptions (157 met Forrester’s criteria for this distinction).
    3. Learn what quality metrics those experts actually measure, and how valuable they rate each metric that they regularly measure.
    4. Use those findings to rate and rank each metric’s usage (how often experts use the metric) and value (how highly experts value the metric).
    5. Compare the DevOps experts’ quality metric usage versus that of DevOps laggards. If there was a significant discrepancy, the metric is considered a DevOps differentiator.

    — Wayne Ariola

  • Reinvent Testing to Galvanize Digital Transformation

    Reinvent Testing to Galvanize Digital Transformation

    We’re well past software “eating the world.” According to Tricentis CEO Sandeep Johri, software has become irreversibly intertwined with the world. You can’t find a modern enterprise that doesn’t depend on software, or a business transaction that doesn’t require software at some point in the end-to-end process. In the new world of digital business, companies that deliver differentiating customer/user experiences through software have the clear competitive advantage. Those that cannot are living on borrowed time.

    In response, every CIO is now focused on digital transformation initiatives that ensure the company is disrupting, not disrupted. From an IT perspective, this requires faster delivery of innovative software and greater agility—the ability to pivot as soon as you identify a new opportunity or challenge.

    Everyone agrees that this is a priority. Developers are producing software faster than ever before, and Ops can now deliver applications in a matter of seconds. So, what’s holding us back? Testing.

    When Johri talks to CIOs about how their digital transformation is progressing, the same basic story always emerges: Organizations transform Dev and Ops, but then testing cannot keep pace … and they get stuck. The tools and processes architected for traditional months-long release schedules simply don’t fit modern delivery cadences, which require immediate quality feedback with each new build. It’s a sad but simple fact: If you have a slow testing process standing between highly accelerated development and operations processes, there’s just no way that you can achieve the desired delivery speed.

    Many companies have already recognized this—and they’ve galvanized their digital transformation initiatives by modernizing their testing. By re-examining and re-inventing software testing across their organization, they went beyond removing the “testing bottleneck”—they’re also driving the positive user experiences that can now make or break a business.

    Many of these distinguished quality leaders joined thousands of software testing professionals from around the world at Accelerate 2018 in Vienna, Austria. This year, the conference theme was “Digital Transformation Requires Continuous Testing.” To ensure that the conference thoroughly covered both the digital transformation and continuous testing sides of that topic—as well as the core intersection between them—we invited CIOs as well as quality leaders to share their perspectives. Here are some of the key insights that are most pertinent for executives leading digital transformation initiatives.

    Archaic Testing Approaches Can Cause a ‘Brain Drain’

    Merck sought out top development talent to advance their digital transformation initiatives. When those developers started quitting, (now retired) Merck CIO Clark Golestani wanted to know why. He learned that the No. 1 reason for these developers leaving the company was the company’s testing practices. Developers were “fed up” because the company was still using traditional testing practices within Agile development processes—and developers believed it was undermining their hard work and dedication.

    That’s the precise moment that Merck launched its testing transformation. Ultimately, the testing organization changed the way testing was done, dramatically reducing attrition and enabling faster delivery with superior levels of quality. According to Golestani: “If it weren’t for the testing organization driving that change, they would have been an inhibitor. We would have never been able to release the types of products we do, and the technology would never work the way that it needs to. It’s absolutely essential.”

    Agile, DevOps, Cloud and Testing are Inextricably Linked

    Agile, DevOps, cloud and testing transformation are all interlinked as part of a broader digital transformation. Andy Sturrock, CIO of one of the world’s largest energy companies, stated: “All of these things are self-reinforcing. You certainly don’t get the full benefit of one without doing the rest.”

    He explained the Ops team used to be the primary bottleneck to moving applications to production (just getting servers provisioned could take weeks or months). Now, the cloud relieves that bottleneck—and that places a lot more pressure on the “apps” team, which tends to emerge as the new bottleneck. “You can write code as fast as you want, but if you can’t test it, you’re not going to get into production quickly,” he said. “Testing has to become automated, and the role of the tester must evolve.”

    When Software Runs Your Business, Software Testing Becomes a Business Issue

    According to Golestani, nearly $1 billion US in revenue passes through Merck’s SAP instance each week. Consider that this instance is updated at least four times a year, and more than a thousand changes are made each time. Manual testing is simply not an option. No amount of manual testing can thoroughly de-risk the core business processes, given the frequency of updates, scope of changes and breadth of functionality that’s impacted.  Extreme test automation is the only way to achieve the required business risk coverage with the necessary speed.

    Due to the nature of Merck’s business, lives also are at stake. Merck ships billions of medicines and hundreds of millions of vaccines each day. The company’s continued focus on digitizing the business means that software runs everything from the shop floor to the way that they interface with customers and educate 4 million healthcare practitioners. As Golestani explained, “When the software driving all this fails, nothing ships, manufacturing shuts down, research shuts down … and people die.”

    It’s Not Easy, but Transforming Enterprise Testing Pays Off—Significantly

    Global 2000 organizations that want to implement continuous testing—the extreme test automation required for DevOps—need to overcome challenges such as:

    • Highly complex application architectures (e.g., processes that cross browsers + APIs + mobile + SAP + other packaged apps + custom Java/.NET apps + mainframes).
    • Different application types, planning cycles, development methodologies and tools existing in parallel.
    • Deeply ingrained quality processes across different groups and projects.
    • Scarce test automation resources.
    • “Bloated” test suites that delay the process while providing limited business value.

    There is no silver bullet. Transformations take time and they must involve all of the people, processes and technologies associated with your application delivery pipeline.

    Nevertheless, you can rest assured that your efforts will pay off. Consider just a few examples of how organizations that committed to transforming testing are now growing the business and enhancing the customer/user experience:

    • Duke Energy achieved a $9 million cost savings during the first year of its testing transformation—and it also reduced testing time by ~ 85 percent. This enables the company to more rapidly deliver software that transforms the customer experience, modernizes its energy grid and creates a smarter energy future.
    • SPAR accelerated time to market 3X, enabling the company to expand its online presence across Europe, deliver more customized shopping experiences and provide seamless “omnichannel” experiences across the customer journey.
    • EdgeVerve now releases 30 percent more new functionality in cycles that are 60 percent shorter with 50 percent fewer escaped defects. This enables the company to help its global banking clients onboard, sell, service and engage customers with flawless personalized experiences.
    • ExxonMobil saved $140 million in cost avoidance and increased productivity 3X. This enables the company’s IT department to take on more projects and complete them faster—which in turn helps the company boost sales and increase customer loyalty.
    • Vantiv (now Worldpay) transitioned from eight-week testing cycles to daily testing, which has been critical for enabling rapid growth and agility during the recent Vantiv-Worldpay merger.

    Continuous testing was once just an aspiration. Now, the drive toward digital transformation has made continuous testing a business imperative. Organizations such as these are paving the path forward, demonstrating that continuous testing can—and must—become a reality in enterprise organizations.

    — Wayne Ariola

  • DevOps Leaders vs. Laggards: Continuous Testing is a Key Differentiator

    DevOps Leaders vs. Laggards: Continuous Testing is a Key Differentiator

    New Forrester research, “The Definitive Software Quality Metrics for Agile + DevOps,” reveals what software quality metrics and practices differentiate DevOps/Agile leaders from DevOps/Agile laggards. The study was commissioned by Tricentis and led by Forrester VP and Principal Analyst Diego Lo Giudice.

    The study found that companies with the most successful Agile and DevOps initiatives do a number of things differently than their peers:

    • They are transforming software testing into continuous testing by adopting five core practices, such as automating end-to-end functional testing and integrating testers into cross-functional teams.
    • They are almost twice as likely to consider automating the software quality process to be a “critical business differentiator.”
    • They are significantly more likely to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).

    However, only 26 percent of firms with active DevOps adoptions are currently practicing the five core continuous testing practices shown to separate leaders from laggards. And, as the infographic below highlights, many CXOs have a severely inflated sense of their firms’ DevOps and continuous testing maturity.

    Different DevOps Quality Metrics

    The study also identified the 20 most important Agile and DevOps quality metrics that separate DevOps/Agile experts from their less advanced peers. Forrester examined 94 quality metrics that were classified into four categories. Key findings include:

    • Understanding of business risk is the most important determining factor of DevOps and Agile maturity.
    • Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).
    • Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

    To determine the “most important” metrics, the research surveyed 603 global enterprise participants responsible for their firms’ Agile and/or DevOps strategies. Participants were first asked a variety of questions to assess the maturity of their Agile/DevOps practices. Next, they were asked what quality metrics they actually used, then asked to rank the value of each metric they regularly measured. Results from the Agile/DevOps experts were later separated from, and compared to, those of the “other” respondents. The most important metrics were determined by analyzing the experts’ value rankings for the metrics that they actually used.

    The Risk ‘Blind Spot’

    Risk was a prevalent thread throughout the various parts of the report. Although risk-related metrics did not rank high in overall popularity, Agile and DevOps experts measure them significantly more frequently than the non-experts did. In fact, Agile and DevOps experts consistently ranked risk-related metrics among their top three most valuable metrics in different phases and categories.

    Interestingly, most firms (80 percent) believe they deliver within acceptable business risk, but fewer than a quarter state that their QA and testing processes completely cover business risk. Only 15 percent of respondents say that their test suites reliably provide a good indication of business risk.

    From Tricentis’ analysis of Global 2000 companies’ test case portfolios, we’ve found that most test suites actually cover about 40 percent of the organization’s business risks. However, these test suites have a 67 percent level of redundancy on average—meaning that over two-thirds of the tests don’t actually increase their business risk coverage. This might be one cause of the disconnect: people assume that more tests means more business risk coverage, but that’s often not the case.

    Another likely cause is that most test results focus on pass/fail status, which doesn’t provide the required insight into whether the release has an acceptable level of risk. As companies move towards continuous delivery and automated release processes, real-time insight into business risk becomes critical for promoting acceptable releases as rapidly as possible while stopping potentially-damaging ones from proceeding down the release pipeline.

    Details and Additional Findings

    If you want to learn more about how Forrester arrived at these findings—and see how the 94 quality metrics ranked—you can read the complete “Definitive Software Quality Metrics for Agile+DevOps” on the Tricentis website.

     

    — Wayne Ariola

  • Continuous Testing and the Broken Promise of Test Automation

    Continuous Testing and the Broken Promise of Test Automation

    The Broken Promise of Test Automation

    For more than two decades now, software testing tool vendors have been tempting enterprises with the promise of test automation. However, most companies have never been able to achieve the desired business results from their automation initiatives. Recent studies report that test automation rates average around 20 percent overall, and from 26 percent to 30 percent for agile adopters.

    We believe that several factors contribute to these dismal automation results:

    Legacy software testing platforms were designed for a different age

    The most commonly used software testing tools today are predicated on old technology, but enterprise architectures have continued to evolve over the years. Development no longer focuses on building client/server desktop applications on quarterly release cycles, with the luxury of month-long testing windows before each release.

    Almost everything has changed since test automation tools such as those from Mercury, HP, Micro Focus, Segue, Borland and IBM were developed. Retrofitting new functionality into fundamentally old platforms is not the same as engineering a solution that addresses these needs natively.

    Legacy script-based tests are cumbersome to maintain

    Scripts are cumbersome to maintain when developers are actively working on the application. The more frequently the application evolves, the more difficult it becomes to keep scripts in sync. Teams often reach the point where it’s faster to create new tests than update the existing ones. This leads to an even more unwieldy test suite that still (eventually) produces a frustrating number of false positives as the application inevitably changes. Exacerbating the maintenance challenge is the fact that scripts are as vulnerable to defects as code—and a defect in the script can cause false positives and/or interrupt test execution.

    The combination of false positives, script errors and bloated test suites creates a burden that few QA teams can overcome. It’s a Sisyphean effort—only the boulder keeps growing larger and heavier.

    Software architectures have changed

    Software architectures have changed dramatically, and the technology mix associated with modern enterprise applications has grown immensely. We’re trying to migrate away from mainframes and client/server as we shift toward cloud-native applications and microservices. This creates two distinct challenges:

    • Testing these technologies requires either a high degree of technical expertise/specialization or a high level of business abstraction that allows the tester to test without diving into the low-level technical details.
    • Different parts of the application are evolving at different speeds, creating a process cadence mismatch.

    The software development process has changed

    Although most enterprises today still have some waterfall processes, there’s an undeniable trend toward rapid iterations with smaller release scopes. We’ve shifted from quarterly releases to bi weekly or daily ones, with extraordinary outliers such as Amazon releasing new code to production every 11.6 seconds. This extreme compression of release cycles wreaks havoc on testing—especially when most testers must wait days or weeks to access suitable test environment and test data.

    The responsibility for quality has changed

    In response to the desire for faster release cycles, there’s been a push to “shift left” testing. The people creating the code are assuming more responsibility for quality because it’s become imperative for getting to “done” on time. However, for large enterprises working on complex applications, developer-led testing focuses primarily on a narrow subset of code and components. Developers typically lack the time and the access required to test realistic end-to-end business transactions. Although the onus for quality has shifted left, the legacy platforms, rooted in waterfall processes, have a distinct bias toward the right. This makes it difficult to blend both approaches.

    Open source testing tools have changed the industry

    The rise of open source software testing tools such as Selenium and SoapUI have had both positive and negative effects. Traditionally, open source testing tools are laser-focused on solving a very specific problem for a single user. For example, Selenium has become an extremely popular script-based testing tool for testing web interfaces. Yet, although Selenium offers speed and agility, it does not support end-to-end tests across packaged apps, APIs, databases, mobile interfaces, mainframes and other components. There’s no doubt that most of today’s enterprise applications feature a web UI that must be tested. However, in large enterprises, that web interface is just one of many elements of an end-to-end business process. The same limitation applies to SoapUI and API testing.

    So, Now What?

    Software testing must change. Today’s software testing challenges cannot be solved by yesterday’s ALM tools. With disruptive initiatives such as DevOps, continuous delivery and Agile expanding across all industry segments, software testing becomes the centerpiece for data-driven software release decisions. This next wave of software delivery lifecycle (SDLC) maturity requires organizations to revamp antiquated testing processes and tools. This means that organizations must have technologies that enable continuous testing, or innovative ideas will remain hostage to yesterday’s heavyweight testing tools.

    — Wayne Ariola