devopsy.kinsta.cloud

Category: Features

Big Gains Are Made In The ‘Last Mile’ of DevOps

A new study by mabl found a correlation between complete, fully automated test coverage and a significant increase in customer happiness and DevOps maturity. In contrast, those with moderate DevOps maturity and nearly automated testing coverage tend to experience a sharp decline in end-user satisfaction.

Mabl recently released its 2021 State of Testing in DevOps Report, a study that highlights the state of DevOps adoption and quality testing throughout the software industry. According to Dan Belcher, co-founder, mabl, the study results clearly show that true business returns are made in the last mile of DevOps adoption.

I recently met with Belcher to hear his viewpoint on the major takeaways. Below, we’ll explore these and other insights from the 2021 State of Testing in DevOps Report.

State of DevOps Maturity

Nearly one in four teams have a mostly mature DevOps methodology and automated process. Still, a significant portion, 52%, are still aspiring or striving to fully embrace a mature DevOps model. A small percentage, 11%, have a fully automated DevOps software development pipeline in place. The bulk of teams are still using manual deployment processes and disparate frameworks, limiting the potential for full automation.

Interestingly, non-technical barriers remain the most common inhibitor to realizing a completely automated development pipeline, found the report. Out of these obstacles, reluctance to change was the most common barrier, cited by 29% of respondents. Other challenges to DevOps adoption include budgetary constraints, DevOps not being a company priority and internal politics.

When added up, non-technical roadblocks account for 82% of the problems that hinder complete DevOps adoption. On the one hand, this depicts the cultural challenges associated with introducing DevOps in an organization. On the other, it could signify the wealth of DevOps packages and open source tooling available to manage areas like automated deployment and version control. The report found GitHub, Jenkins, Gitlab, Azure Pipelines and BitBucket among the five most commonly used DevOps Tools.

Although it’s growing rapidly, continuous delivery is still an emerging trend; almost half of developers still deploy code less than once a month. Only 15% of teams are deploying code daily or multiple times per day. This echoes findings from CDF’s State of Continuous Delivery Report, which pegged this figure at 11%.

Fully Automated Testing

Testing is a critical component of the software life cycle. With the need to ensure functional applications and high-grade security, automated testing is helping teams discover more errors earlier in the development process. According to the report, the most popular test types include UI or functional tests, unit tests, regression tests, integration tests and end-to-end tests. Non-performance tests and visual testing ranked lower. Naturally, the teams that have implemented fully automated pipelines are automating more types of these tests.

In general, it appears test automation aligns with DevOps maturity. The report found that those with fully automated testing experienced significant net customer experience gains. For example, 72% of teams that were automating regression testing indicated their customer happiness is pretty good or amazing, whereas this figure sank to 6% at companies not automating regression testing. And 80% of teams with high test coverage report high customer happiness, whereas only 30% of teams with low test coverage report high customer happiness. Nonetheless, much has yet to be fully automated—similar to the state of continuous deployment, the bulk of teams rely on a combination of automatic and manual testing approaches.

In terms of tooling preferences, Selenium ranked as the most popular option for functional testing, with 41% of developers using it. The report also found migration toward Cypress, Smartbear and other functional testing tools. Interestingly, the report also uncovered a noticeable lack of adoption when it came to automated API testing—only about one out of three are automating their API tests. Postman, SoapUI, internal tools and ReadyAPI are among the most popular tools for running manual API tests.

As I’ve covered before, APIs are surging in use across digital transformation initiatives. Web APIs are an increasingly common mechanism for delivering data to partners and consuming third-party software-as-a-service (SaaS). Full API testing coverage will be essential to audit the entire end-to-end digital experience. Low-code API testing is one way to reduce barriers to auditing the highly technical area of integration.

Key Takeaways

DevOps success is a ‘last mile’ problem. There is much variance in DevOps maturity and testing coverage. It’s pretty flat until you reach that last point, when breakthrough results occur. “Success in DevOps is a ‘last mile’ problem,” said Belcher. “The difference between 90% and 100% of the way there is dramatic.”

Quality really matters. Though much tech discussion focuses on increasing delivery speed, code quality is just as important. “This isn’t about speed at all costs, it’s about innovation with velocity and quality,” said Belcher. “Quality engineers are just about as key to success as developers writing code.”

DevOps success is less about technology and more about people. As the report found, the majority of the roadblocks to DevOps maturity are non-technical. “The things holding us back, as an industry, are more about leadership and figuring out our processing than introducing more technology,” said Belcher.

Breaking down role silos is still fundamental to DevOps. Some see DevOps as instrumental to opening collaboration between roles. Simultaneously, there is the idea that new tooling dashboards will arm individual developers with complete ownership to develop, test and deploy software (known as full cycle development).

Belcher believes that the full cycle momentum has been misinterpreted to mean developers alone have full accountability. Instead, he frames developers as a pitcher on a baseball team; leading the game, but utterly incapable without the rest of the team. Similarly, disparate roles should be working together, said Belcher. Reaching that ‘last mile’ of DevOps is not only about CI/CD or full cycle development; it’s about working together as a group.

I’ve only scratched the surface of many findings from The State of Testing in DevOps. The report queried 600 quality engineers and software development professionals and offers loads of additional insights on the state of DevOps, testing, CI/CD and more. You can view the full results here.

July 7, 2021
Survey Finds Biggest DevOps Obstacles are Organizational

A survey of 600 IT professionals published this week by mabl, a provider of a test automation platform, finds the biggest barriers to adopting DevOps practices are more closely related to organizational issues rather than any technology.

The survey identified non-technological issues as the main inhibitor to DevOps by a wide margin (82%). Only 11% of respondents said their organizations has fully implemented DevOps practices, while another 24% said they were most of the way toward achieving that goal.

The top two barriers to adoption are slow processes and speed of adaptation (29%), followed by budget and funding (21%). Only 18% identified technology limitations as an issue.

Nearly half of respondents (46%) said they have been able to implement continuous integration, while more than one-third (36%) said they have also been able to achieve continuous delivery. Just over a quarter (27%) said they have also achieved continuous deployment. However, only a third of respondents said they deploy new code at least once per week. Almost half still deploy less than once per month. The number of teams that deploy new code daily or multiple times a day (15%) is roughly equivalent to those that deploy code on a quarterly basis (16%).

The most widely used DevOps tools are GitHub (34%) and Jenkins (33%), followed by GitLab (21%), Azure Pipelines (20%) and Bitbucket (19%).

In terms of testing, the most widely automated tests are user interface (UI)/functional and unit tests at 42% each, followed by regression at 41% and integration at 35%, respectively. The most widely used functional testing tool is Selenium (41%), followed by Cypress at a distant second place (14%). The most widely employed tool for testing application programming interfaces (APIs) is Postman, followed by SoapUI (21%) and homegrown tools (17%).

Dan Belcher, co-founder of mabl, said that while it’s clear most organizations still have a long way to go before they fully embrace DevOps best practices, it’s now more a question of degree rather than if. As organizations realize the extent to which they are dependent on software today, the focus on both quality and ensuring timely delivery will increase. The stakes involved in software delivery these days have never been higher, noted Belcher.

Nevertheless, the survey makes it clear how resistant to change organizations are. Some organizations have been building software the same way for decades. The only thing more difficult than convincing IT and business leaders there is a fundamental need for change is driving that change all the way down to individual developers and IT operations teams.

Competitive pressure, however, will inevitably force the issue. If rivals are delivering better application experiences that create a business advantage within the context of a digital business process, an organization will have to adapt. Otherwise, their fate will be, over time, to fade away or be acquired. Equating DevOps with that existential threat may be a challenge, but unless that connection is made it’s only a matter of time before organizations that aren’t embracing DevOps best practices will slide toward irrelevance.

July 2, 2021
The State of Continuous Delivery in 2021

One of the hallmarks of DevOps is to improve the delivery of software. Therefore, mature DevOps systems employ continuous delivery (CD), the process of releasing bite-sized software changes on a regular basis. Amid high demand for digital innovation, streamlining this release pipeline is becoming more and more crucial for enabling overall software fluidity.

So, what is the state of CD among today’s development teams? The Continuous Delivery Foundation (CDF) recently released its 2021 State of Continuous Delivery Report, a study that helps establish baseline continuous delivery metrics. Below, we’ll review some key takeaways from the report and discover mean rates for things like deployment frequencies, recovery times and average lead time for code changes.

Benefits of Continuous Delivery (CD)

Before we look at the broad metrics surrounding CD adoption, let’s consider: what are the benefits of CD? A big net benefit of CD, the report noted, is that it helps realize feedback earlier in the software development cycle. A lean, CD-driven development approach could help quickly discover features that won’t succeed, which avoids wasting precious development resources.

CD also increases overall delivery speed for small software chunks. In an era where microservices architecture is all the rage, this process is a necessity for continually initiating many disparate updates. For example, Netflix utilizes Spinnaker to deliver hundreds of microservices and thousands of daily deployments.

With these results, organizations can reap increased responsiveness and agility. CD can enable fast feedback to changes, which can help businesses adapt to unforeseen conditions, such as zero-day exploits. An investment in CD can also bring net positives to developer team dynamics, according to the report.

Continuous Delivery Statistics

Software performance can be indicative of overall organizational performance. So, how should an organization measure its software delivery success? Well, the book Accelerate outlines four crucial metrics to watch: lead time for changes, deployment frequency, time to restore service and change failure rate. The CDF report establishes current industry averages for three of them.

In terms of deployment frequency, the study found that the highest percentage of developers (31.3%) release once per week to once per month. A large percentage of developers, 27.3%, release every month to six months. Only 10.8% of developers are elite performers, releasing multiple times per day. This indicates that rapid release cycles are still maturing across organizations.

The average lead time for changes is more variable than you might think. It takes 28.3% of developers one week to one month to go from committed code to running code successfully in production. For 27.3% of developers, this lead time is increased, from one month to six months. Only 5.74% of developers say it takes less than an hour, indicating that a truly on-demand delivery mechanism remains out of reach at most organizations.

On the other hand, developers are much quicker to act when it comes to restoring services. The survey showed 34.4% of respondents said it takes them one hour to one day to respond to an unplanned outage. A significant number, 15%, respond to incidents in less than one hour. Still, about 50% of outliers are responding somewhere between one day and more than six months.

Decreasing the mean time to recovery (MTTR) is imperative to respond to broken applications, quickly address security vulnerabilities, and meet evolving customer expectations. As a result, MTTR has become a de facto metric for Site Reliability Engineers (SREs) to watch closely.

Factors That Affect Software Delivery

Other factors may influence software delivery agility, such as industry type and programming language preferences. Out of the many industries attempting to refine their delivery models, the report highlighted retail as an industry that ranks high in both speed and stability metrics. Other sectors, such as financial services, government and energy ranked high on stability yet received only average scores for speed. Industries like telecommunications received poorer overall scores.

Programming language choices may also dictate overall software delivery performance. The study found that shell scripting languages, Go, JavaScript, PHP and Scala were among the top five programming languages, based on the combined software delivery performance data for organizations using each language.

DevOps Aims to Improve CD

Continuous delivery is vital to responding quickly to changes and enabling iterative, progressive delivery of new features.

Another facet of modern CD that can’t be understated is the widespread prevalence and reliance on open source tools. CDF open source packages like Jenkins, Jenkins X, Spinnaker and Ortelius remain vital to enabling CD processes for numerous developer teams.

The 2021 State of Continuous Delivery Report provides a baseline of standard continuous delivery practices across companies and sectors. The findings are based on SlashData’s Developer Nation survey, which reached 19,000+ respondents from 155 countries from late 2020 to early 2021. One area where reporting is lacking is change failure rates, which we hope future studies will benchmark. For more insights, you can download the report here.

July 2, 2021
Continuous Testing Practices Gap Assessment

Continuous testing is an essential part of DevOps. To get the maximum value from continuous testing, it is important to use recommended best practices, inclusive of people, processes and technologies. A gap assessment is a great way to efficiently evaluate an organization’s practices for DevOps and determine a strategy for improvement. Gap analysis provides valuable input for formulating a strategy and a roadmap to improve continuous testing.

My SlideShare, Gap Survey, Assessment and Analysis For Continuous Testing, explains how my continuous testing gap assessment tool greatly speeds up the gap survey, gap assessment and gap analysis process for continuous testing.

It is important to clarify a few definitions for this blog because I have found that different people and organizations use these terms differently. In this blog, and in my consulting work, I use the following definitions.

• Gap survey is a discovery tool used to collect information about the current state of practices for a topic such as continuous testing.
• Gap assessment is a process to determine differences (gaps) between current practices and recommended practices for a topic. Gap assessment is also used to refer to the entire collection of activities in this list.
• Gap assessment workshop is a meeting in which the gap assessment is validated prior to gap analysis.
• Gap analysis is the process of determining priorities for reducing gaps found during a gap assessment.
• Gap assessment tool is software that helps perform gap surveys, gap assessments and gap analysis.

I define and use a seven-step process for gap assessments.

The steps are as follows:

1. Pick a topic to analyze. The same processes and tools can be applied equally well to almost any topic, such as continuous testing.
2. Determine recommended practices for the topic. Something that is worth doing a gap assessment for, such as continuous testing, will normally be fairly complex with tens if not hundreds of practices. Categorize the practices into groups (I call the groups “pillars”) to make the list of practices more understandable and meaningful. To do this requires some research and discovery work. For example, I have a database of practices and pillars for DevOps, continuous testing, DevSecOps and SRE as well as some other topics.
3. Enter the practices from step two into a gap assessment tool. The tool can be a simple survey tool you create, a commercial survey tool or something custom-made.
4. Determine who needs to be included in the gap survey.
5. Collect data using the gap survey.
6. Perform a gap assessment.
7. Perform a gap analysis.

For the remainder of this blog, I will explain in detail the seven steps for continuous testing.

Step One: Pick a Topic to Analyze

In this blog, the topic is continuous testing.

Step Two: Determine Recommended Practices for Continuous Testing

For an example continuous testing assessment, I used the 15 practices categories described in this three-part series:
Continuous Testing Practices Part 1
Continuous Testing Practices Part 2
Continuous Testing Practices Part 3

The 15 practice category pillars are:

• Continuous testing tenets
• Leadership
• Culture
• Test strategy and plans
• Test management
• Test automation
• Test tools
• Test infrastructure management
• Test scripts
• Test results
• Security tests
• Pre-flight testing
• Integration testing
• Pre-production testing
• Test in production

Step Three: Enter the Practices from Step Two Into a Gap Assessment Tool

A gap assessment tool, pre-loaded with sample continuous testing practices, can be found in a file called “Continuous Test Practices Assessment Tool v2. It can be downloaded for free from one of the resource pages found on my website Engineering DevOps.

You can edit the practices categories and add/delete practices from each category if you prefer to make changes.

Step Four: Determine Who to Include in the Gap Survey

For the gap analysis to be comprehensive, people in roles that are affected by continuous testing practices need to be surveyed, or at least represented, to ensure their perspectives are included. The following are example roles that typically are included.

• Business leaders – because they influence culture and budgets for continuous testing.
• Developers – because they need to design in accordance with continuous testing practices.
• Project owners – because they influence product work priorities.
• QA testers – because they need to test in accordance with continuous testing practices.
• Ops – because continuous testing practices affect operations.
• Security – because continuous testing affects security practices.

The survey can be conducted for an individual application, a group of applications, or all the applications in the enterprise. However, it is important that the gap assessment and gap analysis be performed on the organizational segment that is being targeted for improvement.

Step Five: Collect Data Using the Gap Survey

A gap survey should allow each surveyed person to enter an importance level score, a practice level score and comments for each practice. All this information is essential for the gap assessment and gap analysis. In the gap survey included within my gap assessment tool, survey respondents are asked to score the importance of each practice as one of 0=Not relevant, 1=not Important, 2=Nice to have, 3=Important, 4=Very important or 5=Critical. Practice level choices are 0=Not sure, 1=Rarely, if ever, 2= Sometimes, 3=Most of the time, 4=Always or 5=We are really good at this. Comments that are relevant to qualify or explain the scores entered for each practice should also be entered, especially if there is any doubt or ambiguity.

Step Six: Perform a Gap Assessment

The gap assessment process requires all the practices scores collected from the surveys to be collected and assembled to calculate an aggregate set of scores. Gap scores are calculated using a formula that weights each practice level score with the corresponding importance level score. A visual representation helps to identify practices areas and individual practices that have the highest, most important gaps.

No matter how professionally written the practices and score definitions are, it is not unusual for some people to misunderstand and to enter scores that they would not have otherwise. For this reason, it is important to ensure that the data collected is validated before conducting the gap analysis. The preferred approach is to conduct a gap assessment workshop with key representatives from each role that participated in the survey. During the workshop, the scores for each practice that have a high deviation between survey responses are discussed and, if necessary, adjusted.

Step Seven: Perform a Gap Analysis

The gap analysis process involves extracting the high gap practices and tagging and ranking each of them against solution categories determined by a consultant or topic expert. The results of the gap analysis indicate where solution strategies and implementation roadmaps need to be focused to reduce the most important gaps.

What This Means

Those who are serious about DevOps will do well to learn from others and be serious about using recommended continuous testing practices. This blog explained a gap assessment approach, which uses my 15 categories of continuous testing practices. My free Continuous Testing Gap Assessment Tool facilitates performing a gap analysis for continuous testing and outlines priorities to improve DevOps practices for an organization. For those that want to dig deeper on this topic, additional definitions and contextual information related to the way I do gap assessments are clarified in my book Engineering DevOps.

July 1, 2021
Applitools Extends Visual AI Testing to Native Mobile Apps

At its online Future of Testing: Mobile event, Applitools today previewed the ability to apply visual artificial intelligence (AI) to applications that run natively on a mobile computing platform.

Accessed via a cloud service that is currently available for free on a limited basis, the Ultrafast Test Cloud for Native Mobile offering will extend the company’s visual AI tool for browser-based mobile applications to applications that run natively on Apple iOS or Google Android devices.

Applitools CEO Gil Sever said that while most mobile enterprise applications today rely on browsers to make it simpler to build and deploy applications, in time, many more of these applications will be running natively on iOS or Google Android platforms, mainly because they provide a better user experience.

That approach also eliminates the need to rely on cumbersome platform emulators to test applications because the Applitools AI tool can learn the nuances of each platform in a way that streamlines the application testing process.

In fact, visual AI tools will not only make developers and testing teams more efficient, but they also will make it easier for end users to participate in the application testing process, Sever noted.

The Applitools approach to AI testing also provides developers with more control based on the application’s use cases. Different settings can be applied to test, for example, a medical application that requires perfect alignment of pixels versus an e-commerce application that may not require as much fidelity. Developers can then resolve regression issues by employing root cause analytics.

It’s not clear how broadly visual AI tools that use computer vision algorithms to automate testing will be employed. There is no doubt the current pace of application development is faster than most dedicated teams of testers can handle. As a result, more responsibility for application testing is being shifted left toward developers that, in the interest of time, might be more inclined to rely on AI and other forms of automation that can be accessed via a cloud service.

The rise of AI testing tools might also go a long way toward ensuring more applications are thoroughly tested before being deployed in a production environment. One of the first steps in any application development schedule that gets compressed when a project falls behind schedule is testing. As more rote tests are automated, there should be more opportunities for dedicated testing teams to focus on improving the end user experience.

It may be some time before most organizations are routinely employing AI to test applications. However, none of those organizations should expect AI to replace the need for humans to test applications alongside AI testing platforms that can see the actual application. After all, there’s a world of difference between seeing the changes that were made to an application and understanding why they actually needed to be made.

June 29, 2021
Fugue Aims to Simplify Securing Infrastructure-as-Code

Fugue today unveiled a 1.0 release for Regula, an open source policy engine for infrastructure-as-code (IaC) security that comes with prebuilt libraries for implementing hundreds of policies that validate configurations on Amazon Web Services (AWS), Microsoft Azure and Google Cloud services.

Regula is based on the Open Policy Agent (OPA) software being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) and is compatible with both Terraform and AWS CloudFormation tools for configuring cloud infrastructure.

Developers can also build custom rules on top of OPA using libraries that Fugue created for the Rego programing language that is part of the OPA specification. Regula supports output formats such as JUnit, Test Anything Protocol (TAP) and JSON to make it easier to integrate Regula with other tools and frameworks that make up a DevOps workflow. Input formats supported include Terraform HCL, Terraform plan JSON, AWS CloudFormation and Serverless Application Model templates.

Regula also provides out-of-the-box support for the CIS Foundations Benchmarks in addition to other policies defined by Fugue, such as dangerously permissive identity access management (IAM) policies, Lambda function policies allowing global access, volumes with encryption disabled and untagged cloud resources.

It is deployed as a pre-packaged binary that includes a command line interface (CLI) through which Regula can be incorporated within a DevOps workflow and can be installed using tools such as Homebrew or deployed as a Docker image.

Fugue CEO Josh Stella said Regula provides DevOps teams with an extensive library of rules that check for common security and compliance violations as well as advanced multi-resource misconfigurations. That latter issue is especially challenging because developers typically don’t have much visibility into dependencies that can inadvertently provide cybercriminals with access to, for example, an S3 cloud storage bucket that otherwise appears to be configured properly. Regula will also identify any required resources that might be missing, as well, added Stella.

Stella said Regula 1.0 provides a simpler approach to securing cloud infrastructure. Cloud infrastructure is frequently misconfigured simply because a developer made a mistake. Unfortunately, cybercriminals have become especially adept at scanning cloud platforms for these misconfigurations, noted Stella.

It’s not clear to what degree a backlash might be building against IaC in the wake of recent high-profile breaches of software supply chains. Organizations have embraced IaC tools to make developers more productive. However, each security incident involving a misconfiguration of cloud infrastructure resources makes it all too apparent that developers lack the expertise required to securely provision IT environments. Rather than creating a series of cybersecurity reviews that would reduce developer productivity, Stella said Fugue is making a case for a compliance-as-code approach that developers can readily understand and implement.

One way or another, cloud infrastructure has to become more secure. Business leaders are growing weary of security incidents that, from their perspective, are caused by careless mistakes. Their first instinct is to empower cybersecurity teams to resolve those issues once and for all. The only issue that needs to be resolved now is just how cumbersome those security reviews need to be if developers are provided with the tooling required to secure cloud platforms themselves.

June 29, 2021
JFrog Acquires Vdoo to Advance DevSecOps

JFrog today announced it has agreed to acquire Vdoo for $300 million in cash to gain a set of analytics tools that discover vulnerabilities in application binaries.

Vdoo’s scanning tools, infused with machine learning algorithms, will be fully integrated with the JFrog Xray vulnerability detection tools along with the rest of the JFrog continuous integration/continuous delivery (CI/CD) platform in 2022. In the meantime, the Vdoo scanning tool will continue to be made available via a software-as-a-service (SaaS) platform that Vdoo built.

The Vdoo platform is already integrated with the JFrog Artifactory repository and JFrog Pipelines. Vdoo’s platform also is integrated with DockerHub, Jenkins, GitHub, GitLab and Azure Pipelines via REST application programming interfaces (APIs) that make the Vdoo scanning tools accessible via a command line interface (CLI).

JFrog CTO Yoav Landman said acquiring Vdoo is critical because when it comes to implementing DevSecOps best practices, the only meaningful place to discover and remediate vulnerabilities is in the binaries deployed in production environments. As such, the Vdoo platform provides a superior alternative to both static application security testing (SAST) and dynamic application security testing (DAST) tools that scan for vulnerabilities in source code, added Landman.

Developers require accurate intelligence that can be quickly acted upon to secure applications running in production. They can quickly take action once they know how a specific vulnerability impacts a binary, versus being told to update source code that could take days or weeks to complete, noted Landman. The Vdoo platform also makes it possible to leverage machine learning algorithms to detect zero-day vulnerabilities, malware, exploits, backdoors, supply chain risks and other threats before they become commonly known. Those vulnerabilities can be detected in everything from application binaries to firmware running on embedded devices. Vdoo is also recognized by the Mitre Corp. that oversees the Common Vulnerabilities and Exposures (CVE) database as a CVE Numbering Authority (CNA) for discovering vulnerabilities.

As responsibility for application security continues to shift left toward developers, many organizations are finding the tools that cybersecurity teams employed to discover threats don’t lend themselves to the workflows that developers have created to build applications. As a result, many of these tools are being replaced by developer-friendly alternatives that developers can invoke via a CLI as part of DevOps workflow. The goal, now, needs to be reducing the level of noise created by security tools to enable developers to focus on the vulnerabilities that specifically impact their code, noted Landman. Otherwise, developers are simply overwhelmed by a massive number of alerts generated by security tools that often don’t appear to be especially relevant, Landman added.

It’s not clear how quickly responsibility for application security is shifting left. However, in the wake of a series of high-profile software supply chain breaches, the urgency surrounding adoption of DevSecOps best practices has increased considerably. The challenge, as always, is getting the right tools into the hands of developers at the right time.

June 29, 2021
Continuous Testing Practices – Part 3

In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing to help with understanding what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

This article is part three, which covers the following categories of continuous testing practices.

• Security tests
• Pre-flight testing
• Integration testing
• Pre-production testing
• Test in production

You can read part one here and part two here.

Security Tests

• Credentials and secrets needed to operate any test are vaulted.
• Customer sensitive data in application databases used for testing is obfuscated from misuse by people involved in testing.
• New security tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
• Security tests for each DevOps pipeline stage are automated and may be selected automatically according to predefined criteria.
• Test results that indicate possible security concerns are tagged for security analysis.
• Dynamic or interactive application security tests exercise the application to check for security vulnerabilities. Results of these tests are delivered to developers through tools and feedback loops native to their organization.
• Attack patterns, abuse cases and tests are built according to application module profiles.
• Unit, functional and integration tests around—and especially outside—boundary conditions are tested. Tests include error handling, exception handling logic and negative tests.

Pre-Flight Testing

• Products are architected to support modular independent packaging, testing and releases. In other words, the product itself is partitioned into modules with minimal dependencies between modules. In this way the modules can be built, tested and released without requiring the entire product to be built, tested and released all at once.
• Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance, to ensure the changes will not break the integration branch.
• Requirements for testing are included in agile backlog planning procedures.
• Pre-flight tests include unit testing, static code analysis scanning, functional, regression and performance tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested, for use in later stages of the pipeline and subsequent regression tests.
• Pre-flight tests, conducted prior to integration, include unit testing, static code analysis scanning, functional, regression, performance and security tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested for use in later stages of the pipeline and subsequent regression tests.
• Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance to ensure the changes will not break the integration branch.
• Test-driven development (TDD) is used to ensure tests are created ahead of the code to be tested. TDD is most often applied for unit level tests but may also be applied for functional testing. A key advantage of TDD is that it ensures working tests are available for use in all pipeline stages. Without using TDD, there is a tendency for test creation to fall behind the code. Another advantage of TDD is that tests are created without the bias of knowing how the code was written, because it hasn’t been written yet.
• From a testing point of view, microservices introduce the requirement to validate the contract of each service with any other services that use it. The independence of the microservice or any interservice dependencies need to be well tested. Performance and reliability considerations of operating services over a network need to be verified. If the application changes, affected microservices, or any dependent group(s) of microservices also need to be regression tested.
• Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
• A strategy ensures that changes to a database, or changes to an application that uses a database, perform as expected, during each stage in the continuous delivery pipeline.

Integration Testing

• Software changes are tested incrementally by a cross-functional team over all stages of the continuous delivery pipeline.
• During the code commit stage, assessments are conducted on the committed code to validate that the pre-flight test results satisfy acceptance criteria for integration into the integration branch.
• During the build stage, a select group of tests are run on each build to ensure the integrated build satisfies build integration acceptance testing criterion.
• Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
• API contract testing is used to test microservices interfaces.
• Test code and data for each application are stored and maintained in test containers so they can be managed using the same container creation, orchestration and monitoring capabilities as the application.

Pre-Production Testing

• After the integration and build stage, functional, performance, regression, customer acceptance and security tests are conducted to ensure the images resulting from the build or set of builds satisfy functional, performance, security and delivery assessment criteria.
• Testing during deployment include dark launch strategies such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
• Release assessment criteria are defined and use test results to decide whether a release meets release acceptance criterion.

Test In Production

• Testing during deployment uses strategies that minimize the impact of failures such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
• Automated tests verify each deployment.
• Chaos engineering is used for testing the resiliency of applications, deployment infrastructure and pipelines.
• Tests for disaster recovery and restoration of services, infrastructure and pipelines are conducted periodically.
• Fire drills on applications, pipelines and infrastructures are used to test procedures for detection and recovery of applications, pipelines and infrastructures.

What This Means

Continuous testing (CT) practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer Continuous Testing. This article is part of a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s Continuous Testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.

June 29, 2021
CircleCI CEO Sees Growing Need for Developer Experience Engineers

There is no shortage of titles to go with all the tasks that make up a DevOps workflow, but given the critical role developers now play within any organization, it might be time for organizations to start creating a developer experience engineer (DXE) function within DevOps teams that is specifically tasked with increasing developer productivity.

CircleCI CEO Jim Rose said it’s become apparent there is more friction than ever in the application development and deployment process, as more complex applications are constructed, thanks, in part, to the rise of microservices.

A DXE, as outlined in a report published by CircleCI, would be responsible for implementing a common set of DevOps principles, maintaining the right tools and putting in place a cohesive set of standard processes that ensure developer success. Today, there are simply too many languages, frameworks, engineering styles and processes being employed within the same organization, and that conspires to reduce overall developer productivity, said Rose.

While software engineers are, today, addressing subsets of these issues, Rose said the time has come to assign these tasks to an engineering team that views itself as an advocate for developers by making available more responsive infrastructure.

As organizations of all sizes increasingly realize how dependent they are on software to drive digital business transformation initiatives, Rose noted that developer productivity is becoming a major corporate issue. Revenue forecasts and profit margins are now being directly impacted by the rate at which software is being delivered and updated.

Software engineers today routinely build DevOps pipelines for developers, but not much time and effort is put into optimizing the developer experience, noted Rose. That’s become not only a major productivity issue, Rose said, it has a material impact on the ability of organizations to attract and retain developer talent. One of the reasons so many developers prefer to work for smaller startups than large enterprises is because their daily work experience is simply better. Large organizations need to reduce the amount of friction developers experience if they want to be able to compete for that same talent, said Rose.

A DXE would also be more inclined to solicit input from developers on tool selection to best optimize productivity, noted Rose. Many software engineers today construct pipelines with little to no regard for how simple or hard it might be for a developer to employ tooling.

Organization today are, of course, starting to become more obsessed with measuring application development velocity. However, in many cases, simply making someone responsible for achieving a goal is going to result in more gains at a lower cost. Analytics tools have a role to play, but obsessing over key performance indicators (KPIs) may not generate the same result as quickly as simply giving an engineer a specific mandate.

As responsibility for application management continues to shift left toward developers, it should not come as a surprise that there is a lot more interest in automating the underlying infrastructure. The issue, at this point, is determining to what degree software engineers are going to achieve that goal versus forcing developers to seek out yet another platform that promises to do it on their behalf.

June 28, 2021
Continuous Testing Practices – Part 2

In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing to help explain what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

This article is part two, which covers the following categories of continuous testing practices:

• Test automation
• Test tools
• Test infrastructure management
• Test scripts
• Test results

You can read part one here.

Test Automation

• Test automation requirements are defined.
• New automated tests are created for each new feature in compliance with the test strategies for the product or service.
• Automated test cases are reviewed and tested to ensure they satisfy the purpose of the test.
• Automated test cases are compliant with test case standards for the product or service.
• Automated test cases are designed to run independent of each other.

Test Tools

• All testing tools in a pipeline are integrated into a common framework which enables orchestration of test resources and automation of test tasks.
• A test framework supports automation of everything needed for testing including test cases, test tools, applications, test infrastructures, and test data.
• A test framework can be invoked, controlled and released through an application programming interface (API) so that it can be integrated into a DevOps continuous delivery toolchain.
• Features provided by test frameworks include capabilities to orchestrate the test infrastructure, select tests, control the execution of tests, monitor tests in progress and report test results, logs and verdict data.
• A selection criterion is defined for test tools. The criterion includes requirements for coverage, automation features, support, compatibility with the automation framework, compatibility with test progress reporting systems.
• Test tool software versions are maintained in a source version management system.
• A common test automation framework is defined for the entire organization to ensure all test tools can integrate and be orchestrated with common test automation practices.
• Test automation tools are available for testing all features of an application that can be automated.
• Manual testing tools are available for testing features that must be tested manually. These tools can report results to a common test management reporting system.
• Test authoring tools support input error checking and features which help encourage scripts development according to best practices. Examples of such features include the ability to create and use a repository of known good test utilities for control and analysis of the system under test.
• DevOps-ready test tools are employed that easily integrate into a test framework and toolchain and elastically scale on demand vertically or horizontally, to match the varying capacity and workload demands of tests of software changes going through the continuous delivery pipeline.
• DevOps-ready tools can be orchestrated, scaled, invoked, controlled and monitored from an API.

Test Infrastructure Management

• Standards for describing resources and topologies for test configurations are documented.
• Techniques used to accelerate the time for executing tests are used such as vertical and horizontal scaling of test campaigns.
• Test resources and topology files are maintained in a software version management system.
• Orchestration and automation tests are conducted to verify the software that supports orchestration and automation layers satisfy their assessment criterion
• The test infrastructure used for each stage in the pipeline is a replica of production configuration variations, as much as is practical.
• Infrastructures for testing are elastic, with the ability to stand up and release (i.e. orchestrate) infrastructure configurations as needed for use with specific tests. Typical techniques that are employed to orchestrate test environments include dynamic infrastructure configuration tools, infrastructure-as-code (IaC) strategies, cloud services, testing-as-a-service (TaaS) providers and use of tools to stand up and release applications and test resources packaged in immutable containers.
• Database migration, replication and restore tools also need to be tested to ensure they can function quickly enough to keep up with the speed of the DevOps continuous deployment pipeline cadence.
• Health checks on the test environment are used to reduce the chance of false negative test verdicts that could be caused by test environment failures.

Test Scripts

• Test scripts are maintained in a software version management system.
• Guidelines for test scripts and test suites practices are documented.
• Test utilities, tests and test suites follow guidelines for documentation, test results reporting, reuse and maintenance practices.
• Test script peer reviews are conducted to encourage collaboration and to ensure test script best practices are used.
• Each test script has a unique identification.
• The test type of the script is identified (for example, sanity test, build test, smoke test, security, acceptance test, conformance test, functional test, unit test, patch test, bug fix test, integration test, regression test, performance test, system test, other).
• Evidence, including an execution log, demonstrates that each verdict of tests work.
• Tests determine at least one verdict, which may be ‘Pass,’ ‘Fail’ or ‘Inconclusive.’
• Verdict and assertions priorities are recorded or reported.
• Tests are data-driven. Data inputs control execution of the test.
• Each test script is is tagged with a codified risk level.
• Failure verdicts reported by the script include a tag for risk priority.

Test Results

• Techniques are used to ensure test results analysis keeps up with accelerated tests. For example, test design techniques, analysis tools built into test frameworks, test results dashboards and/or test results analyzers that run in parallel with the tests.
• Quality assurance goals include specific thresholds for test coverage prior to release such as run-to-plan (RTP) and pass-to-plan (PTP) minimum thresholds. RTP is a measure of how many tests of the total planned tests have completed. PTP is a measure of how many of the completed tests pass. For example, 95% RTP and 90% PTP.
• Quality goals include a measurement and threshold for reliability. For example, S-curves for pass rates can indicate reliability when the pass rate stabilizes for a period. Example: 95% pass rate for three consecutive test cycles.
• Time to recover a failed service is routinely tested and used to calculate mean-time-to-recover (MTTR).
• Monitoring and management tests are conducted to verify monitoring and management capabilities of the pipeline satisfy their assessment criterion.
• Technical and business metrics are designed to monitor the success of DevOps testing for technical continuous improvement and to justify business cases needed to implement those improvements.
• Input and output metrics and acceptance criterion exist for each stage in the value stream.

What This Means

Continuous testing (CT) practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer continuous testing. This article is the second in a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s continuous testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.

June 28, 2021