devopsy.kinsta.cloud

Tag: API testing

  • How the World Measures DevOps Quality

    How the World Measures DevOps Quality

    With continuous everything, knowing whether each new release will ultimately enhance or undermine the overall user experience is essential. Yet, most of today’s go/no-go decision still hinge upon quality metrics designed for a different era.

    Every other aspect of application delivery has been scrutinized and optimized for DevOps. Why not re-examine quality metrics as well?

    Are classic metrics like number of automated tests, test case coverage and pass/fail rate important in the context of DevOps, where the goal is immediate insight into whether a given release candidate has an acceptable level of risk? What other metrics can help us ensure the steady stream of updates are truly fit for production?

    To provide the DevOps community an objective perspective on what quality metrics are most important for DevOps success, Tricentis commissioned Forrester to research the topic. The results are published in the 55-page report, Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics—Ranked by Industry Experts. The report takes a deep dive into the global findings, complete with heat maps, quadrant mappings and some fun lists such as “Most Overrated,” “Hidden Gems” and “Top DevOps Differentiators.”

    One of the most common questions we received after publishing the report was: “How do the results vary across regions?” In response, we performed some additional regional analysis—and I’d like to share those results here.

    To start, let’s take a look at the global top 20. The following metrics were ranked as the most valuable by the DevOps experts who measure them (across all regions).

    Europe DevOps Quality Metrics Trends

    Looking specifically at Europe, the top 20 changes as follows:

    Interesting trends in this region:

    • There is a greater commitment to measuring quality metrics. European respondents reported a higher level of DevOps quality metrics measurement across the board. For almost all metrics, the usage rate was at least 6% higher than the global average. For metrics related to time, coverage, risk, effectiveness and efficiency, the usage rate was over 14% higher. This speaks to European organizations’ commitment to scrutinizing and continuously optimizing their quality processes—especially in terms of time and resource utilization.
    • Risk and coverage metrics are valued more than in the global average. European respondents also ranked risk and coverage quality metrics a surprising 21% higher than the global average. This could be related to the fact that the respondents from this region came primarily from the financial services and insurance sector, with healthcare and government close behind. In such highly-regulated industries, measuring and mitigating risk is certainly a core concern. This finding could also indicate European organizations place a greater emphasis on protecting the corporate brand.
    • Test data preparation time seems to be a greater concern. European respondents were more likely to measure (+18%) and highly-value (+23) time spent preparing test data than their global peers. Given the restrictions GDPR placed on test data as of May 2018, it seems likely that European organizations have significantly changed their test data management processes (e.g., to masking and more synthetic test data generation), and are cautiously monitoring how the changes are impacting their overall efficiency.

    Asia Pacific DevOps Quality Metrics Trends

    Now, let’s shift focus to Asia Pacific. The following 20 metrics were ranked as the most valuable by the Asia Pacific DevOps experts who measure them:

    Notable trends in this region:

    • End-to-end testing metrics are valued—and measured—more than in the global average. Although Asia Pacific respondents measured fewer build and functional validation metrics than the global average, they measured (and valued) end-to-end testing metrics much more than their peers around the world. For example, percent of automated end-to-end tests was measured by 47% of the organizations (versus 36% globally) and highly-valued by 84% (versus 70% globally). Risk coverage measurement was significantly higher; it was measured by 49% (versus 34%) and highly-valued by 71% (versus 59%). This speaks to the region’s focus on digital transformation and commitment to delivering exceptional user experiences.
    • API testing metrics were also valued—and measured—more than in the global average. Asia Pacific respondents also measured and valued API testing quality metrics more than the global average. Overall, API quality metrics were measured by 16% more organizations in this region than globally. The highest valued API quality metrics were API test coverage (63% versus 39% globally) and API risk coverage (79% versus 62% globally). This prioritization of API testing is likely a side effect of the regional trend towards API-driven open banking (the majority of respondents indicated they were in the financial services and insurance sector).
    • There is a greater leader/laggard quality metrics measurement gap. Part of the study involved classifying the respondents as either DevOps leaders or DevOps laggards, based on their responses to various questions about the maturity of their processes. Although the percentage of DevOps leaders in the region was lower than the global average (18% versus 26%), the DevOps leaders from Asia Pacific generally measured quality metrics at a comparable rate to their global peers. However, the DevOps laggards in Asia Pacific generally measured quality metrics at a much lower rate than their global peers. This suggests the select set of firms that have truly prioritized DevOps initiatives have made great strides—and the laggards have a lot of catching up to do in order to remain competitive.

    Research Methodology

    Here’s a look at the process behind the research discussed above:

    1. Survey 603 global enterprise leaders responsible for their firms’ DevOps strategies.
    2. From that sample, identify the firms with mature and successful DevOps adoptions (157 met Forrester’s criteria for this distinction).
    3. Learn what quality metrics those experts actually measure, and how valuable they rate each metric that they regularly measure.
    4. Use those findings to rate and rank each metric’s usage (how often experts use the metric) and value (how highly experts value the metric).
    5. Compare the DevOps experts’ quality metric usage versus that of DevOps laggards. If there was a significant discrepancy, the metric is considered a DevOps differentiator.

    — Wayne Ariola

  • Top 6 Challenges of API Testing

    Top 6 Challenges of API Testing

    Testing teams face several unique challenges when testing application programming interface (API) implementations. API testing is part of the integration testing process and concerns the communication between two systems—one of which your staff likely has no control over.

    Below you’ll find the following six common challenges when testing APIs and our recommended solutions.

    Initial Setup of API Testing

    Manual testing helps confirm whether something works. Automated testing is necessary with APIs to gauge how well they perform under pressure. Getting the testing infrastructure up and running is often one of the most challenging parts of the process, not because it is particularly difficult, but because it can be a substantial motivation-killer. Once you manage to motivate your team to get through the process, however, it pays off over the long-term.

    Solution: API testing should start in the design phase—how is the API going to be tested? Ensure interval checking of your APIs for 100 percent uptime.

    Updating the Schema of API Testing

    The schema—in other words, the data formatting that handles requests and responses for the API—needs to be maintained throughout the testing process. Any updates to the program that create additional parameters for the API calls need to be reflected in the schema configuration.

    Solution: In today’s API economy, having an ever-changing API helps facilitate the demand. However, there can be downtime caused by those changes. One way to avoid that is to thoroughly test the API in beta and alpha environments. The chances of issues decrease to 90 percent when testing in these environments.

    Testing Parameter Combinations

    APIs handle communication between systems by assigning data values to parameters and passing those parameters through data requests. It’s necessary to test all possible parameter request combinations in the API to test for problems pertaining to specific configurations. A larger project could end up assigning two different values to the same parameter, or create instances where numerical values appear when text values should be. Adding an additional parameter exponentially increases the number of possible combinations.

    Solution: We’ve learned from our customers that having release candidates are important. For example, picking one or two applications to utilize the API on and release it to just those applications. Pick applications that aren’t critical to daily operations. This way you’re able to see how the API is being utilized and if there are any configurational changes that need to be made to have a general availability (GA) release.

    Sequencing the API Calls

    In many cases, API calls need to appear in a specific order to work correctly. This creates a sequencing challenge for the testing team. For example, if a call to return a user’s profile information goes through before the profile is created, the request will return an error. Alternatively, a call to create a map needs to run prior to placing location pins on the map to work correctly. This process can get increasingly difficult when working with multiple-threaded applications.

    Solution: This relates to the point above of having a release candidate to help in managing and seeing what the sequencing of the API calls are. We recommend making a flowchart to visualize the API calls. This will help developers build API calls and integrate them faster without causing issues.

    Validating Parameters

    Testing teams may find validating the parameters sent through API requests challenging as well. The sheer number of parameters and use cases for those parameters can make it a daunting task. The team needs to make sure all parameter data uses the correct string or numerical data type, fits within length restrictions, fits within a designated value range and passes other validation criteria. For example, U.S. phone numbers should appear in a 10-digit format, and returning a 5-digit zip code should trigger an invalidation error.

    Solution: This can be avoided by having continuous synthetic API monitoring to catch the problem early on. Moreover, it should be combined with an APM solution to get an overall 360-degree view. Validating parameters is one of the most important things. It should show you how fast you’re serving those parameters. If making those calls has significant impacts on your system, implementing synthetic and APM solutions will ensure you catch the problem early on.

    Tracking System Integration

    The final challenge is making sure the API testing system is working correctly with the data tracking system. This is necessary to bring back correct responses on whether a call is working correctly. It’s also used to passively monitor API performance. Since this step can come late in the process, the team may be too fatigued to give it the attention it needs.

    Solution: Think about your application in the design phase and how it will integrate with other systems. Different applications have different requirements that need to be met. Make sure you aren’t causing applications to fail by testing in parallel with critical integration systems. This can be accomplished by implementing and including load testing in your continuous delivery.

    API testing is an essential part of application development in the modern business environment.

    About the Author / Sven Hammar

    Sven Hammar is co-founder and Chief Strategy Officer of Apica, a provider of technology for testing, monitoring, and optimizing the performance of cloud and mobile applications. Mr. Hammar has more than a decade of experience and expertise in web performance and web optimization, e-commerce, cloud services, IT entrepreneurship and the Internet. He is also a serial entrepreneur who has founded several successful IT companies over the years. Connect with him on LinkedIn.