By adopting a multilevel approach to site reliability engineering and arming your team with the right tools, you can unleash benefits that impact the entire service-delivery continuum
In today’s application-driven economy, the infrastructure supporting business-critical applications has never been more important. In response, many companies are recruiting site reliability engineering (SRE) specialists to help them improve reliability, availability, latency and a host of other metrics that impact customer satisfaction. These vital new people resources are focused on keeping sites and services running in peak form.
There are clear limitations to this operations focus, though. A service is made up of many application components that can impact business-critical services. How do you ensure reliability is baked into each app and each code change if your engineers focus only on backend performance?
Some trailblazers are beginning to tackle this issue head-on, tasking their SREs with taking on new roles that support key stages of the DevOps life cycle. The goal is to take early, proactive steps to ensure quality and reliability are built in from the beginning.
A Path Pioneered by Test Teams
To understand why this “shift left” is so important, consider recent revolutionary changes in the software testing function. Not that long ago, testing was considered a distinct phase in the software life cycle, timed to begin well after the work of application developers was done. Testing Centers of Excellence uncovered defects and passed code back to the development team for rework.
As developers moved to agile sprints, though, this approach to testing quickly became a bottleneck. By the time Centers of Excellence uncovered issues, developers had moved on to other projects. In response, new continuous testing programs emerged, staffed by a new breed of software development engineers who had honed their testing skills.
These individuals are now embedded with development teams to ensure applications are built better from the start. Open source test automation tools help them test early and often, without impeding agile delivery schedules.
It’s important to note that Testing Centers of Excellence haven’t gone away. Instead, they have taken on a broader leadership role and are now responsible for coordinating testing across the enterprise in support of continuous software integration and delivery.
A 3-Level Model for SRE
There is much that SRE teams can learn from the experiences of their testing colleagues. To produce optimal results, it is important to integrate reliability experts into your organization’s overarching agile framework. Consider adopting the following three-level model that shifts the function to the left and integrates skilled engineers at key points in the DevOps life cycle:
Application level. Embed SREs with development and testing teams to ensure application components and new releases won’t disrupt reliability. These individuals focus on application-level service objectives, error budgets and integration into the DevOps pipeline.
System level. Assign SREs to focus on the applications in your organization’s release train. These individuals support release and launch coordination, evaluate system architecture readiness and ensure your organization meets systemwide service-level objectives.
Enterprise level. Establish a Site Reliability Engineering Center of Excellence, staffed by engineers who oversee the governance of your enterprise architecture. These individuals establish best practices and select the tools and resources required to support your site reliability function companywide.
An Expanded Toolset
To support this broadened role, you will need tools that can support your engineers across the entire DevOps life cycle. You will need to be able to see and track issues engineers uncover and to evaluate system-level readiness and performance against your service-level objectives. You will also need a big-picture view that spans activity across the entire enterprise.
Fortunately, a new generation of solutions is now available to help, powered by artificial intelligence, machine learning and intelligent automation. These resources can help your team process massive amounts of data from disparate toolsets and turn that data into actional insights. As a result, you’ll have all the lifecycle analytics you need to be effective in your new and broadened role.
SRE and Shift Left: Broad New Benefits
By adopting a multilevel approach to sire reliability engineering and arming your team with the right tools, you can unleash benefits that impact the entire service-delivery continuum.
You can build a continuous cycle of feedback, collaboration and governance that spans design, development and the launch and operation of new services. You can better manage configuration changes, service levels and error budgets. You can get a clearer view of application reliability and production readiness to guide planning initiatives and to help you prioritize investments. And you can build better-informed business cases to move your organization forward. Shifting left is clearly a win-win move.
The concept of shift-left has been a popular trend in continuous testing practices for a while. We are now beginning to see shift-right practices as an emergent trend in testing.
Shift-right entails doing more testing in the immediate pre-release and post-release phases (i.e. testing in production) of the application lifecycle. These include practices such as: release validation, destructive/chaos testing, A/B and canary testing, CX-based testing (e.g. correlating user behavior with test requirements), crowd testing, production monitoring, extraction of test insights from production data, etc.
Shift-right not only introduces such new testing techniques, but also requires testers to acquire new skills, make aggressive use of production data to drive testing strategies and collaborate with new stakeholders, such as site reliability engineers (SRE) and operations engineers.
In shift-right of testing, and in the increased collaboration with operations disciplines, we see the evolution of a new discipline in DevOps that we call TestOps.
In this blog, we will discuss the various shift-right testing trends and practices, what’s driving them, the new skills and collaboration required so that TestOps benefits can be realized.
Emerging Trends in Shift-Right Testing
In a recent survey of continuous testing conducted jointly by Capgemini and Broadcom, testing in production was ranked as the number one practice (with 45% respondents) either currently implemented or planned (see Figure 1). In addition, 39% of the respondents mentioned the use of analytics from operations to determine or optimize test coverage.
Figure 1.
When asked how customers measure the effectiveness of continuous testing processes, production data and user feedback and adoption of new functionality were ranked number one and number two respectively (see Figure 2).
Figure 2.
Finally, in response to how customers test their digital applications to ensure that they meet real world requirements, almost all of the responses were related to production data (see Figure 3). Use of production data for designing test strategy and chaos testing emerged as new methods compared to similar survey done a year ago.
Figure 3.
The survey data clearly indicates that customers are actively practicing (or considering adoption of) shift-right testing techniques. This report has a chapter (with my contributions) dedicated to shift-right that discusses the techniques and recommendations for its adoption, as well as relevant customer case studies.
What’s Driving Shift-Right Testing Trends?
The goal of shift-right testing is to ensure correct behavior, performance and availability over the production of an application.
There are several drivers for adoption of shift-right testing trends.
Customer Experience (CX) Is a Key Quality Metric for Digital Applications
Unlike classic testing, this takes into account real-world users and their experiences. An application with perfect traditional quality scores (such as FURPS) may still suffer from poor CX if it fails to delight the customer. CX is measured using various metrics such as Customer Effort Score (CES), Net Promoter Score (NPS), Customer Satisfaction Score (CSAT), etc. While it is possible to shift-left some of these measurements to some degree, most CX measurements are deduced from systems in production (or close to production). Examples of CX-based testing include the following:
Requirements validation based on real user journeys, behavior and feedback.
Deriving functional and performance test scenarios based on the above.
A/B testing and canary testing, to experiment how well customers like (or dislike) changes.
Crowd testing, to better understand real-world user experience.
In a World of Agile Delivery, It may Not Be Possible to Test Everything Before Release to Production
Time to market (or lead time for changes) is a top business imperative for digital applications, which is why QA/testing is considered a major bottleneck in continuous delivery. While it is possible to optimize testing effort and cycle time using shift-left practices (such as model-based testing, change-impact testing, test process and execution automation etc.), those approaches may still take too long (or too much effort) during the shrinking release cycle time. In some cases, the exact usage patterns may not even be fully understood before release. The idea is to learn usage patterns from production and use it for better testing strategy within shift-left.
Complex and increasingly distributed systems using microservices (using thousands of components) and cloud-native techniques that allow releases at a very granular level make it more and more difficult to test fully in pre-production environments. This is complemented by the fact that since such releases are so granular, they may be backed out easily (rollback) in case they cause problems.
Hence, enterprises test to assure “good enough” quality (to ensure timely release) and rely on fast remediation (or rollback) to address defects or problems that arise.
Also, CX measurements (as described above) often provide real-user feedback in shades of gray, unlike the black-n-white pass/fail measurement of classic testing. Which means it is not possible to perfect pre-production test suites to ensure thorough CX coverage.
A Hundred Percent Reliability (or Quality) Is Often an Unrealistic Goal
Related to the above, one of the key principles we learn from the DevOps discipline of SRE is that 100% reliability is not only unrealistic, but often too expensive to accomplish. SRE establishes the concept of service level objectives (SLOs) and error budgets to quantify acceptable risk tolerance in production systems. The same principle applies to testing and overall quality. See my related blog for more on this subject
Some Validations Are Difficult to Run in Test Environments
These include large-scale performance tests when the test environments are not appropriately sized (or configured) relative to production environments.
Another example is destructive or chaos testing. While it is possible to run isolated chaos testing in test environments by using techniques such as service virtualization (to simulate failure in dependent components), it is difficult to execute for large-scale destructive tests. For example, Netflix does a significant amount of their testing in production.
Similarly, enterprise monitoring to gather insight about real-world usage patterns and failures is only possible in production. Though we advocate turning on monitoring in test environments (as part of shift-left), such monitoring only helps to do localized monitoring (e.g. of the specific app or system being tested), and that too under test conditions. However, performance test scenarios that have been developed by testers can be re-used in production (shift-right) as synthetic monitors to measure application performance and regression.
Lastly, as an extension of monitoring, testing of some AI-based applications may be difficult without access to production data. Some machine learning algorithms, for example, need to be constantly refined based on real world data. While these algorithms are developed using a limited set of training and testing data, they need to be monitored and tuned, based on its performance in production.
Developers and Testers Need Easier Closed-Loop Feedback from Production
Easier closed-loop feedback from operations allows developers and testers to better understand the behavior of the app, predict the success/failure of releases and respond to production incidents (e.g. reduce MTTR).
Interpretation and use of voluminous production data is often difficult and tedious for development and test teams. Use of shift-right techniques allows developers and testers to access production data more easily in a form that is easily consumable and actionable–by leveraging the right insights from AIOps disciplines. This data collaboration is a key tenet of the DevOps paradigm. For example, synthetic monitors (which are typically created by developers and testers in the dev/test stages) provide means for developers to get monitoring feedback in a form that they are familiar with.
In addition, automated root-cause analysis of defects/incidents in production can provide developers and testers insight on failure patterns and trends, their sources (when production data is correlated with development data) and what’s causing them, and how they may be better prevented.
Similarly, data usage scenarios (e.g. from database logs and RR pairs from service transaction logs) can be used for generation of relevant test data.
In essence, shift-right testing practices are necessary to create a continuous feedback loop from real user experience with a production application back to the development and testing process.
The Evolution of TestOps Discipline
It is clear from the customer survey data and drivers discussed above that there is an evolution toward shift-right practices.
Just like we did with shift-left testing, in order to make these practices sustainable and to realize the benefits, we need to also evolve tester use cases, skills, tools and collaboration patterns with people and disciplines to the right of the application lifecycle, i.e. operations. In other words, a new discipline needs to be established—we call this new discipline TestOps—a sub-discipline within the broader DevOps context.
Though the term TestOps (like all the X-Ops sub-disciplines within DevOps) implies the collaboration between testing and operations, it isn’t simply about shift-right. Why? Because with DevOps, operational disciplines themselves have shifted left (such as shift-left of monitoring, configuration management, SRE, etc.). Hence TestOps—true to the principle of continuous testing—refers to better collaboration with operations disciplines across the DevOps lifecycle.
Let’s consider some of the key practices in TestOps shift-left (see Figure 4).
Figure 4.
Early reliability testing:this includes static scalability analysis of the system architecture, as well as performance and stress testing.
Configuration quality and testing: this includes testing of “as-code” (anything “as-code” can and should be tested as such, just like application code) environment and deployment configurations, as well as deployment testing (to test the correctness of deployments) and rollback testing (to ensure that rollbacks can be done quickly and successfully).
Early monitoring: includes monitoring of systems early in the lifecycle (e.g. in development and test environments) as well as development and testing of synthetic monitors before they are deployed to production.
Early AIOps insights: this includes early use of AIOps techniques (in the dev/test environments) to glean testing and quality insights, for example for defect and release risk prediction.
Below are some of the key TestOps shift-right practices (see Figure 5).
Chaos engineering: this includes testing the reliability of the system through the injection of controlled failure scenarios.
A/B testing: consists of a randomized experiment with two (or more) variants to study which are more effective with users.
Canary releases: consists of techniques used to reduce the risk of introducing a new software version in production by gradually rolling out the change to a small subgroup of users, before rolling it out to the entire platform/infrastructure and making it available to everybody.
Synthetic monitoring: includes monitoring techniques using an emulation or scripted recordings of transactions (that reflect typical user behavior).
CX testing and analytics: includes extracting of CX data to understand customer experience as well as to derive insights from it (such as customer problems, new feature requirements, etc.).
Other testing in production: includes performance tests and testing of AI-based applications.
Testing insights from AIOps data: as described above, a rich variety of testing insights can be gleaned from operations data.
Figure 5.
Key TestOps Skills and Tools
Let’s consider the new skills required for the key TestOps practices. This includes the following:
Data analytics skills: Since most of the production data is both voluminous, diverse and often temporal, testers must now pick up data analytics so that they quickly glean insight from such data, as well as perform correlations with other datasets to identify actions. Advanced analytics skills (such as predictive analytics or machine learning) are also necessary to predict events (e.g. release quality). While these analytics skills are well understood within the operations community (with the increasing adoption of AIOps), these are relatively new for developers and testers.
CX skills: As we discussed, CX is now considered a key metric for measuring quality in production. New types of CX testing include A/B and canary tests as well as crowd testing. Like other operational data, CX data is also typically voluminous and often unstructured. While CX teams specialize in such principles, testers need to acquire sufficient skills to glean insight from the CX process and data, and collaborate effectively with such teams.
Monitoring and operations skills: Understanding of monitoring principles (e.g. creation, testing and deployment of monitors and use of data from monitors) is a key skill required for TestOps. Similarly, testers must also understand operations principles (incidents, failures, alarms, MTBF, MMTR, configuration definitions, etc.) so that such information can be leveraged for testing purposes as well as for collaboration with ops teams.
Reliability skills: Reliability is also another key operational quality attribute that testers need to be familiar with. This includes chaos/resiliency testing, deployment and rollback testing, and configuration testing. The discipline of SRE has evolved within DevOps to address reliability, and many of its principles overlap with TestOps.
New tools skills: In addition to using traditional testing tools, TestOps also requires the use of additional tools such as monitoring, data analytics (for production and CX data), CX testing (such as A/B and canary testing) and reliability testing. Technology innovations that support testers with these new capabilities include RunScope (a lightweight API monitoring tool that can be used for monitoring both in test and production environments), Blazemeter (for performance testing for CX and reliability), Optimizely for A/B testing. For data analytics, technologies such as automation.ai simplify the work of data aggregation and modeling so that testers can more easily extract actionable insights from data collected across the lifecycle.
Key TestOps Collaborations
In addition to new skills, testers must now learn to collaborate better with additional stakeholders as alluded to above. These include:
Collaboration with data science and AI/ML engineers: This is necessary for testers to tweak or build out new algorithms and models required to perform the analytics needed for TestOps.
Collaboration with CX engineers and teams: This is necessary for testers to monitor the appropriate CX metrics and data, and to study the impact of their testing/QA activities on CX.
Collaboration with Operations engineers and teams: This is fundamental for TestOps to not only support all the shift-right testing activities, but also to gain access to production data that is required for TestOps analytics. This is analogous to how testers collaborate with developers as part of shift-left.
Collaboration with SREs: As mentioned, there is a fair degree of overlap between SRE and TestOps principles, and hence collaboration with SREs is key for testers. See my blog on how testers and SREs can work better together to improve reliability.
Conclusion
Hopefully, this article has provided readers with some insight into the emerging discipline of TestOps. As we build increasingly distributed microservices and cloud-native systems, and as the discipline of SRE matures, we expect to see more of shift-right testing practices that enrich TestOps.
Transforming DevOps by leveraging the power of analytics and autonomics
In a previous blog, I have written about how Quality Assurance is getting transformed to Quality Intelligence using the power of big data analytics and autonomics.
In this blog, we will discuss how DevOps solutions of the future can become more intelligent by taking advantage of these same emerging digital technologies such as autonomics and analytics.
Technologies such as predictive analytics, autonomic computing (autonomics), artificial intelligence and robotic process automation (RPA) are already driving unprecedented levels of automation in many other businesses. Combined, these three technologies are driving the next generation of automation—what is commonly referred to as “intelligent automation.”
Intelligent automation goes beyond basic automation and orchestration of manual tasks to incorporate adaptive self-learning—leading to fully autonomous systems. The figure below (courtesy IRPA) illustrates the evolution of automation as it moves toward full autonomy. The Google car (and other similar solutions) is a great example an intelligent automation system, capable of learning from its environment and adapting behavior accordingly.
Automation in Traditional DevOps
Digital systems are driven by new age technologies such as social, mobile, big data, analytics, cloud, internet of things (IoT), artificial intelligence, augmented and virtual reality, genomics, etc. These technologies introduce significant change and complexity to how modern systems are built and deployed. Businesses that leverage these technologies to transform themselves into cognitive enterprises demand new and intelligent approaches to the underlying processes and techniques to support them. Such processes and techniques need to be more than agile; they need to adaptive and capable of reacting dynamically to frequently changing circumstances.
Let’s look at some examples of current DevOps techniques and how they will need to evolve to support dynamic digital systems. Automation within traditional DevOps solutions is generally limited to scripting and orchestration as shown above. Automation adoption levels vary, due to various reasons ranging from complexity to skills and organizational challenges. Maintenance of such scripts is itself sometimes a bottleneck, as applications and environments change rapidly, the scripts need to be updated manually and often become obsolete. New age agile digital businesses, by contrast, demand automation that is able to adapt dynamically and self-heal on demand.
Further, classic DevOps automation techniques are typically driven by static predefined rules. For example, the criteria for promotion of an application build through different stages of the pipeline is often statically defined. This is a limitation for new age systems where the criteria needs to be dynamic and may vary depending on multiple circumstances. The automation system needs to able to look at past data, keep learning from current data and make adaptive intelligent predictions about the right course of action.
DevOps for internet of things (IoT) systems is very different than that for traditional software systems. The target environment for production deployment in consumer IoT is geographically dispersed, typically not configuration controlled, may have unreliable network connectivity and may even be fragile. Further, IoT systems generate huge amounts of data that necessitate robust data mining and self-learning (adaptive) techniques not provided by traditional lifecycle automation tools.
Similarly, customer experience (CX) is a key new metric for digital systems that transcends traditional DevOps metrics such as release velocity and quality. CX data is unstructured, fuzzy, voluminous and volatile. CX-driven DevOps (or CX-Ops) is an emerging discipline that requires big data analytics and cognitive approaches (such as natural language processing or NLP) to decode meaningful insight from such data.
Hence, as digital enterprises evolve and businesses demand greater agility and flexibility, the DevOps function to support such systems will need to transform as well.
Next-Generation Intelligent Automation Landscape
Before we jump into what intelligent DevOps would look like, let’s first look at the different types of automation systems being adopted in the industry in general. The following types of automation are described as part of the “intelligent automation continuum”:
“Systems that Do”: These are basic automation systems that replicate human keystroke actions and static (pre-defined) rules-based activities. They also take advantage of descriptive analytics that reflect past trends and patterns. Examples of such systems include speech and image recognition.
“Systems that Think”: These use algorithms and intelligence to find meaning in data, handle judgment-oriented tasks using diagnostic analytics and make recommendations based on patterns. Examples of such systems include natural language processing and recommendation engines (such as email spam filters).
“Systems that Learn”: These understand context, interpret and dynamically adapt based on circumstances; they typically leverage predictive and prescriptive analytics to solve problems independently. Examples of such systems include self-driving cars and neural networks.
A Peek at Intelligent DevOps
So, what would transformed intelligent DevOps look like? Intelligent DevOps automation would take advantage of cognitive and autonomics systems to enable smarter adaptive lifecycle automation based on analytics. In my previous blog, I discussed how QA can take advantage of big data analytics. Intelligent DevOps, to a large extent, depends on such capability.
Based on the above model, let’s look at the different types of DevOps automation we can create:
DevOps Systems that Do: This includes traditional DevOps automation systems (e.g. for continuous integration and testing, continuous deployment), as well as pipeline lifecycle automation that are based on static rules (e.g. traditional release management automation)
DevOps Systems that Think: This includes advanced automation systems such as:
Automated automation systems, for example, generation of automated test cases from manual tests (or test models) using NLP, generation of virtual services based on request-response data logs
Self-healing automation, for example virtual services (or test scripts) that can auto-update based on change in application endpoints (or behavior)
Monitoring of IoT systems, for example “smart homes,” which require continuous use of diagnostic analytics to mine huge volumes of data to understand failure modes and prescribe recovery techniques
Automated verification of system requirements based on customer experience analytics
Automated generation of test scenarios based on analytics on production logs
DevOps Systems that Learn: This includes sophisticated test automation systems such as:
Adaptive continuous delivery pipeline—Learning systems that analyze past data to manage the pipeline based on dynamic rules. For example, correlate code quality and defect detection and slippage patterns to dynamically decide what tests to be run and what gates to be enforced (vs enforcement of statically defined gates) for different teams and products for promoting application builds
DevOps process optimization based on insight across the life cycle. For example, correlation of production log data with past code change data to determine the level of failure risk in different application modules. See more on this in the section below.
We embody the evolution to intelligent DevOps in CA’s maturity model (shown below), which also aligns with the RPA maturity model described above. Beyond the continuous maturity level, Intelligent DevOps is the driving force behind continuous optimization and improvement.
Cross-Life Cycle DevOps Intelligence
Intelligent DevOps enables us to perform process optimization based on analytics from data correlated across the system life cycle, from planning through operation. Each process area generates a great amount of data that is typically analyzed within the process (and sometimes organizational) silo.
While such analytics is useful in itself, correlation of data across these process areas (see figure below) may be used to provide a wide variety of intelligent lifecycle insight (and process improvement possibilities), such as:
Correlating configuration data analytics with code change and defect analytics helps us pro-actively identify failure modes related to code and infrastructure changes.
CX analytics (from the Operations process) can be used to validate requirements in the plan and define processes
Production log analytics may be correlated to test log data to identify missing test cases
In the spirit of the “Continuous Everything” philosophy in DevOps, we envision a new solution stream around “Continuous Insight” where analytical insights are generated and acted upon continuously (and autonomously) as processes execute. We will discuss more on the subject of Continuous Insight in a future blog.
In conclusion, we believe that intelligent approaches described above will be infused into every aspect of DevOps going forward and revolutionize the way DevOps is performed. Stay tuned my friends, and may all our DevOps be intelligent!
