As a product management professional, I am trained to understand customer pain, especially for customers who have challenges with downtime due to poor software quality. Last Friday I experienced the problem firsthand though when I arrived at Heathrow Airport early, only to discover that the Microsoft outage had grounded flights across the world. My flight from London was only delayed a couple of hours, but my connecting flight was canceled completely. Since it was the last flight of the day to my hometown, I had to find a hotel room for the night, but all of the rooms within 30 miles of the airport were sold out. All of this is because of a bug in one small file not much larger than this article, impacting millions of people across the world.
Most software releases contain defects. I have never seen production code that didn’t have issues, especially enterprise-level applications. Findings that are released to production are usually cosmetic, or minor bugs that impact a seldom-used feature. Defects like the one that took down the airlines, however, are another situation entirely. Now it’s time to consider what lessons we can learn from this event.
Lesson one: Shift Right is NOT optional. Shift Right is the concept that you should test in production, after the release. You have to wonder if this was done in the case of last week’s outage. Even if testing was performed in the pre-release environments, you can’t be sure that your pre-release environments are the same as production. You must smoke-test in production at a minimum. Better yet, run regression tests regularly. Even when testing in production, testing a single process or component may not be enough. Which leads us to the next lesson.
Lesson two: Integration and end-to-end system testing is a must. Most enterprise-class applications today are built from a complex collection of interconnected services and systems. Each service in isolation can function precisely as expected, but when you test it together with all the other moving pieces, you may find that small delays here and there will break the end-to-end process. These timing-related issues are exacerbated by unusually high volumes, which brings us to the next lesson.
Lesson three: Performance and load testing are important too. Performance and load testing may not seem important under normal circumstances, but when recovering from an outage, there is typically a bow wave of users trying to get on with the delayed work. This could be considered a cascading effect, but the situation often rears its ugly head at month and quarter end for some businesses.
In the case of last week’s incident, a huge number of canceled flights led to an increase in volume on the airline reservation system. A system that may not have been impacted by the actual bug is impacted by an unusual level of activity. Load testing is not an easy job, and it may not be necessary to do it regularly, but you shouldn’t wait until disaster strikes to learn that a 20% above-average volume will render a system unusable.
Lesson four: You don’t have to release to everyone all at once. Two concepts would have helped prevent a global meltdown. The first is “dogfooding” aka eating your own dog food. This is the practice of using your own software before releasing it to others. That way if there are major issues, you find them fast and you contain the suffering in your own organization.
The second concept is called canary releases. Just as miners used to take canaries into the coal mines to warn of a deadly condition, so too can you release code to a small subset of external users for a short period to ensure that a release works outside of your organization. Once again, you contain the negative impact.
Lesson five: The cost of system outages has cascading impacts. It’s not just the lost productivity during the downtime but it’s also the domino effect. For example, downtime in the order entry system can cause challenges with the inventory management system which can delay shipments and impact the top line as well as the bottom line. Lost revenue and profit may cause shareholders and customers to vote with their feet.
Lost productivity is compounded by the need for overtime to catch up. For hourly workers, this is a direct cost to the company. For salaried employees, like developers and testers, this means a negative impact on their job satisfaction and time away from their families. Increased turnover in IT has a cost as well.
The decline in customer satisfaction is another indirect cost of poor quality that is not always tied directly back to an outage. Even if your competitors were impacted by the same incident, your customers may not know that. They were traveling with you that day.
The next time you are negotiating a budget for testing tools and resources, remember that there are not only significant downtime costs, but many indirect costs as well. Sometimes I think the cost of testing tools and personnel should be classified as insurance because it is. The fact that bugs very seldom lead to downtime means the pain is most often not acute. We live with it. Our customers live with it. Until one day it knocks you off your feet.
Don’t wait until the pain is unbearable. Invest in people, processes and tools to ensure that you will never be put in the position of explaining why a bug in one small file impacted the entire world.
The operations team receives a page at 2:00 a.m. Something isn’t working correctly. Do they know how to fix it? Perhaps more importantly, do they know whether it matters to customers or how it impacts the business? There are different techniques that companies implement to maximize the quality of their software. We talk about shifting left or right, chaos engineering and others. While they each have their own goals to improve quality, there’s an underlying common goal as well: All three techniques increase your team’s knowledge of the system and their ability to resolve problems quickly.
Here, we’ll walk through the software life cycle and discuss the common challenges to software quality, running complex systems and managing dependencies between teams. The common thread is a proper observability strategy which can help software quality, hand-offs between different teams and resolve problems quickly by ensuring whole-system resiliency from design to maintenance.
Techniques for Improving Quality
Shift left is a concept that has attracted much attention recently. The practice revolves around the idea that detecting and correcting problems sooner minimizes their impact. As the thinking goes, they’re easier to fix, they wind up costing less and there are fewer ongoing issues with bad code.
The rise of DevOps and DevSecOps has propelled the popularity of shift left. Yet, there’s a fact that development teams often miss: no matter how early an organization spots errors, problems will inevitably remain. Early testing can’t catch everything. As a result, some organizations have begun to rebel and move to a shift-right framework. It normalizes errors and aims to better resolve issues across the development process, including post-production.
Shift right is the perhaps less-extreme cousin to its rebellious relative, chaos engineering. With chaos engineering, team members cause problems on purpose to keep everyone on their toes and test the limits of the system. They shut down servers, disconnect wires, delete files, etc. The goal is to confirm system resilience–from a user perspective, there should be no impact even with all of this chaos. A resilient system means that teams can fix failed components at their leisure during the day. With a resilient system, critical failures are less likely to happen and it becomes second nature to fix them when the dreaded 2:00 a.m. page does occur. Unlike chaos engineering, shift right doesn’t explicitly break things–it just makes the assumption that broken things will exist so teams need to be ready to handle them.
Both concepts (and yes, even chaos engineering) are valid—but each company needs to carefully analyze which is right for them. One variable to consider is the complexity of the system you’re building. Working on a rocket launch or a surgery robot? Probably should shift left as much as possible. Changing the color of a button? Maybe it’s not the end of the world to verify the fix in production. For most of us, projects fall somewhere between these two extremes–so sometimes shifting left is necessary, and other times shifting right is pragmatic. A careful analysis of the risks should be performed to determine what kind of testing is needed.
Although teams tend to be pretty good at analyzing the complexity of their software and making development process decisions, they often forget the complexity of other companies’ code which they depend on. Even a simple web application deployed today has tons of dependencies: The server it’s running on, the local ISPs, transit ISPs, load balancer, DNS provider, TLS certificate, CDN and browser, just to name a few! After analyzing the complexities of all of the dependencies across the internet, it becomes clear that finding every possible issue that might impact their application is a daunting task.
The goal, again, is to find any issues before they reach production. Or if they reach production, fix them before they impact customers. Or if they impact customers, fix them as soon as possible after that. To do any of these things quickly, the level of observability of the overall system is paramount. With simple systems—or even complex systems that are entirely under one company’s control—tools such as application performance monitoring (APM) are common. But when the scope of the system is as big as the internet, internet performance monitoring (IPM) tools are necessary instead.
With both APM and IPM tools, the goal is to know as much about the system as possible – enabling teams to identify problems quickly, pinpoint the root cause, and fix it. When an organization is equipped to monitor development and operational processes effectively from application to delivery, there’s a shift toward higher quality code, fewer errors, improved security and lower costs for the enterprise. There are also fewer disruptions and distractions for engineers, DevOps teams and others.
Dealing With Disconnects
Whether shift left or shift right is used, there will be problems that enter production. There is an invisible wall in many organizations–where engineering teams “throw code over the wall” to the team which is now responsible for running it. In some companies, this is the operations team; in others, it’s site reliability engineering or production engineering–the common trend is that in most companies, it’s not the same team who created the software.
These operations teams often lack the nuanced information they need to resolve production issues. They might spend hours or even days attempting to track down the source of the problem. There’s often finger-pointing between different teams who created or own different portions of the system. Things can quickly devolve into chaos—and tempers can fray. After all, these kinds of problems always happen at 2:00 a.m., right?
That’s where end-to-end observability enters the picture. Imagine a framework with proof that the development team has performed sufficient knowledge transfer to the operations team. That the operations team has asked all of the relevant questions about running and maintaining the application. Using the right tools, engineering and operations teams need to monitor every aspect and every component of the system. Creating this monitoring strategy means that everyone involved also builds an understanding of the system–everyone is on the same page. All of a sudden, there is a common language to discuss problems, and teams operating the software in production can take debugging a lot further before they need to ask for help.
Ultimately, shifting left or shifting right is a personal choice for every organization. Even if every possible test case is executed before deployment to production though, issues will happen–so take the opportunity to implement a robust system-level monitoring strategy that can be used to solve technical questions during that 2:00 a.m. call.
“Companies are spending a great deal on security, but we read of massive computer-related attacks. Clearly, something is wrong.
The root of the problem is two-fold: We’re protecting (and spending money on protecting) the wrong things, and we’re hurting productivity in the process.
InfoSec is hurting productivity in the organization it is supposed to be protecting.”
Bellovin clarified what he meant by “the wrong things.” He observed that security measures tend to stick around even when the threat they were intended to address has faded into the past. Those security costs have only increased since 2015 when the book was published, and today represent a trillion-dollar drag on the global economy despite the emergence of security-integrated models of software delivery such as DevSecOps, shift left and shift right.
The 2021 IBM Security Threat Intelligence Index reports the No. 1 attack vector was a surveillance technique known as scan-and-exploit. Defenders face an asymmetric engagement with such network threats. The successful attacker needs only to exploit a single vulnerability whereas defenders must secure 100% of the network’s ports of entry 100% of the time. The fundamental problem is the anonymous and permissionless nature of the internet. This makes it both useful and easily abused. Anyone anywhere may conduct reconnaissance and launch attacks at a massive scale with little cost or risk. How can we make it prohibitively difficult or costly to successfully surveil and attack our own applications? What if we could reliably defend against the entire category of active scanning tactics?
There are no secure networks, only isolated networks. Taking this back to first principles, a network’s purpose is to convey packets of data from sender address to recipient address. The network has no built-in ability to identify either party or to discern the legitimacy of any activity. Active scanning techniques identify potential targets that are then probed by scripts that may be intrusive or stealthy or both. We could simply reject all incoming network traffic with a firewall, but application servers must be continually reachable in order to listen for incoming requests. Any interruption of availability causes errors like “connection refused” and “service unavailable” for the user.
This necessary network exposure leaves the upper layers of the network and the application server itself vulnerable to intrusion and denial of service attacks. “Private” enterprise applications are similarly exposed to the famously-vulnerable group that are VPN users’ devices, and so are by no means immune to this problem. This presents the problem of automatically discerning which incoming connections are legitimate, continually, forever. Are we trying to solve the wrong problem?
Ideally, we would identify the sender before they connect to the application server. That way, we can know whether they have permission to connect at all. This is something we simply can not do when the sender is known only by a network address. We can’t ask for a credential because that introduces yet another application layer with all of the same problems. This is precisely why VPNs do not go far enough: A superficial barrier is erected for users that prevents the application server from checking for permission until after surveillance and subsequent attacks have occurred.
Embedded connectivity-as-code solves the right problem. This represents a shift in thinking about the relationship between the application and the network. Now, the application brings its own point-to-point overlay with a verifiable identity on both ends. This has significant implications for the security of the application that can be summarized in terms of shifting security left in the development timeline and shifted right in the life span after deployment.
Shift left: OpenZiti makes embedding connectivity-as-code convenient for developers and transparent for auditors and users. As open source software, it fits into DevSecOps processes that scan the source code. The management side of the OpenZiti network is cloud-native and API-driven and fits into the DevOps paradigm of continuous integration testing. For example, a network-as-code may be continually created and destroyed by a pipeline to exercise any imaginable configuration of endpoints, services and policies, or to facilitate dynamic scanning of the data edge and fabric APIs.
Shift right: OpenZiti allows us to shift right security to measure and monitor applications after development, in production, with a novel ease and without compromising security. Operators maintain perfect visibility of deployed applications and need not cope with the complexity of hopping through security gateways or bastions or worrying about external malicious actors and inbound connections. This best-of-both-worlds reality is a natural effect ofan access control scheme that is based upon verifiable cryptographic identity, not a weak identifier like an IP address. Importantly, this means it is always knowable which identity is accessing what services, when they were accessed and to what extent.
OpenZiti gives applications superpowers! To name a few: Identity-driven addressability, private domain name system, shelter from unauthorized connections, portable and programmable micro-segmentation, end-to-end encryption, solves the IPv4 shortage without IPv6, smart routing to choose optimal routes across the internet and is transparent to end users.
Learn more about OpenZiti by checking out our Quickstarts or engaging in conversation on Discourse.
The concept of shift-left has been a popular trend in continuous testing practices for a while. We are now beginning to see shift-right practices as an emergent trend in testing.
Shift-right entails doing more testing in the immediate pre-release and post-release phases (i.e. testing in production) of the application lifecycle. These include practices such as: release validation, destructive/chaos testing, A/B and canary testing, CX-based testing (e.g. correlating user behavior with test requirements), crowd testing, production monitoring, extraction of test insights from production data, etc.
Shift-right not only introduces such new testing techniques, but also requires testers to acquire new skills, make aggressive use of production data to drive testing strategies and collaborate with new stakeholders, such as site reliability engineers (SRE) and operations engineers.
In shift-right of testing, and in the increased collaboration with operations disciplines, we see the evolution of a new discipline in DevOps that we call TestOps.
In this blog, we will discuss the various shift-right testing trends and practices, what’s driving them, the new skills and collaboration required so that TestOps benefits can be realized.
Emerging Trends in Shift-Right Testing
In a recent survey of continuous testing conducted jointly by Capgemini and Broadcom, testing in production was ranked as the number one practice (with 45% respondents) either currently implemented or planned (see Figure 1). In addition, 39% of the respondents mentioned the use of analytics from operations to determine or optimize test coverage.
Figure 1.
When asked how customers measure the effectiveness of continuous testing processes, production data and user feedback and adoption of new functionality were ranked number one and number two respectively (see Figure 2).
Figure 2.
Finally, in response to how customers test their digital applications to ensure that they meet real world requirements, almost all of the responses were related to production data (see Figure 3). Use of production data for designing test strategy and chaos testing emerged as new methods compared to similar survey done a year ago.
Figure 3.
The survey data clearly indicates that customers are actively practicing (or considering adoption of) shift-right testing techniques. This report has a chapter (with my contributions) dedicated to shift-right that discusses the techniques and recommendations for its adoption, as well as relevant customer case studies.
What’s Driving Shift-Right Testing Trends?
The goal of shift-right testing is to ensure correct behavior, performance and availability over the production of an application.
There are several drivers for adoption of shift-right testing trends.
Customer Experience (CX) Is a Key Quality Metric for Digital Applications
Unlike classic testing, this takes into account real-world users and their experiences. An application with perfect traditional quality scores (such as FURPS) may still suffer from poor CX if it fails to delight the customer. CX is measured using various metrics such as Customer Effort Score (CES), Net Promoter Score (NPS), Customer Satisfaction Score (CSAT), etc. While it is possible to shift-left some of these measurements to some degree, most CX measurements are deduced from systems in production (or close to production). Examples of CX-based testing include the following:
Requirements validation based on real user journeys, behavior and feedback.
Deriving functional and performance test scenarios based on the above.
A/B testing and canary testing, to experiment how well customers like (or dislike) changes.
Crowd testing, to better understand real-world user experience.
In a World of Agile Delivery, It may Not Be Possible to Test Everything Before Release to Production
Time to market (or lead time for changes) is a top business imperative for digital applications, which is why QA/testing is considered a major bottleneck in continuous delivery. While it is possible to optimize testing effort and cycle time using shift-left practices (such as model-based testing, change-impact testing, test process and execution automation etc.), those approaches may still take too long (or too much effort) during the shrinking release cycle time. In some cases, the exact usage patterns may not even be fully understood before release. The idea is to learn usage patterns from production and use it for better testing strategy within shift-left.
Complex and increasingly distributed systems using microservices (using thousands of components) and cloud-native techniques that allow releases at a very granular level make it more and more difficult to test fully in pre-production environments. This is complemented by the fact that since such releases are so granular, they may be backed out easily (rollback) in case they cause problems.
Hence, enterprises test to assure “good enough” quality (to ensure timely release) and rely on fast remediation (or rollback) to address defects or problems that arise.
Also, CX measurements (as described above) often provide real-user feedback in shades of gray, unlike the black-n-white pass/fail measurement of classic testing. Which means it is not possible to perfect pre-production test suites to ensure thorough CX coverage.
A Hundred Percent Reliability (or Quality) Is Often an Unrealistic Goal
Related to the above, one of the key principles we learn from the DevOps discipline of SRE is that 100% reliability is not only unrealistic, but often too expensive to accomplish. SRE establishes the concept of service level objectives (SLOs) and error budgets to quantify acceptable risk tolerance in production systems. The same principle applies to testing and overall quality. See my related blog for more on this subject
Some Validations Are Difficult to Run in Test Environments
These include large-scale performance tests when the test environments are not appropriately sized (or configured) relative to production environments.
Another example is destructive or chaos testing. While it is possible to run isolated chaos testing in test environments by using techniques such as service virtualization (to simulate failure in dependent components), it is difficult to execute for large-scale destructive tests. For example, Netflix does a significant amount of their testing in production.
Similarly, enterprise monitoring to gather insight about real-world usage patterns and failures is only possible in production. Though we advocate turning on monitoring in test environments (as part of shift-left), such monitoring only helps to do localized monitoring (e.g. of the specific app or system being tested), and that too under test conditions. However, performance test scenarios that have been developed by testers can be re-used in production (shift-right) as synthetic monitors to measure application performance and regression.
Lastly, as an extension of monitoring, testing of some AI-based applications may be difficult without access to production data. Some machine learning algorithms, for example, need to be constantly refined based on real world data. While these algorithms are developed using a limited set of training and testing data, they need to be monitored and tuned, based on its performance in production.
Developers and Testers Need Easier Closed-Loop Feedback from Production
Easier closed-loop feedback from operations allows developers and testers to better understand the behavior of the app, predict the success/failure of releases and respond to production incidents (e.g. reduce MTTR).
Interpretation and use of voluminous production data is often difficult and tedious for development and test teams. Use of shift-right techniques allows developers and testers to access production data more easily in a form that is easily consumable and actionable–by leveraging the right insights from AIOps disciplines. This data collaboration is a key tenet of the DevOps paradigm. For example, synthetic monitors (which are typically created by developers and testers in the dev/test stages) provide means for developers to get monitoring feedback in a form that they are familiar with.
In addition, automated root-cause analysis of defects/incidents in production can provide developers and testers insight on failure patterns and trends, their sources (when production data is correlated with development data) and what’s causing them, and how they may be better prevented.
Similarly, data usage scenarios (e.g. from database logs and RR pairs from service transaction logs) can be used for generation of relevant test data.
In essence, shift-right testing practices are necessary to create a continuous feedback loop from real user experience with a production application back to the development and testing process.
The Evolution of TestOps Discipline
It is clear from the customer survey data and drivers discussed above that there is an evolution toward shift-right practices.
Just like we did with shift-left testing, in order to make these practices sustainable and to realize the benefits, we need to also evolve tester use cases, skills, tools and collaboration patterns with people and disciplines to the right of the application lifecycle, i.e. operations. In other words, a new discipline needs to be established—we call this new discipline TestOps—a sub-discipline within the broader DevOps context.
Though the term TestOps (like all the X-Ops sub-disciplines within DevOps) implies the collaboration between testing and operations, it isn’t simply about shift-right. Why? Because with DevOps, operational disciplines themselves have shifted left (such as shift-left of monitoring, configuration management, SRE, etc.). Hence TestOps—true to the principle of continuous testing—refers to better collaboration with operations disciplines across the DevOps lifecycle.
Let’s consider some of the key practices in TestOps shift-left (see Figure 4).
Figure 4.
Early reliability testing:this includes static scalability analysis of the system architecture, as well as performance and stress testing.
Configuration quality and testing: this includes testing of “as-code” (anything “as-code” can and should be tested as such, just like application code) environment and deployment configurations, as well as deployment testing (to test the correctness of deployments) and rollback testing (to ensure that rollbacks can be done quickly and successfully).
Early monitoring: includes monitoring of systems early in the lifecycle (e.g. in development and test environments) as well as development and testing of synthetic monitors before they are deployed to production.
Early AIOps insights: this includes early use of AIOps techniques (in the dev/test environments) to glean testing and quality insights, for example for defect and release risk prediction.
Below are some of the key TestOps shift-right practices (see Figure 5).
Chaos engineering: this includes testing the reliability of the system through the injection of controlled failure scenarios.
A/B testing: consists of a randomized experiment with two (or more) variants to study which are more effective with users.
Canary releases: consists of techniques used to reduce the risk of introducing a new software version in production by gradually rolling out the change to a small subgroup of users, before rolling it out to the entire platform/infrastructure and making it available to everybody.
Synthetic monitoring: includes monitoring techniques using an emulation or scripted recordings of transactions (that reflect typical user behavior).
CX testing and analytics: includes extracting of CX data to understand customer experience as well as to derive insights from it (such as customer problems, new feature requirements, etc.).
Other testing in production: includes performance tests and testing of AI-based applications.
Testing insights from AIOps data: as described above, a rich variety of testing insights can be gleaned from operations data.
Figure 5.
Key TestOps Skills and Tools
Let’s consider the new skills required for the key TestOps practices. This includes the following:
Data analytics skills: Since most of the production data is both voluminous, diverse and often temporal, testers must now pick up data analytics so that they quickly glean insight from such data, as well as perform correlations with other datasets to identify actions. Advanced analytics skills (such as predictive analytics or machine learning) are also necessary to predict events (e.g. release quality). While these analytics skills are well understood within the operations community (with the increasing adoption of AIOps), these are relatively new for developers and testers.
CX skills: As we discussed, CX is now considered a key metric for measuring quality in production. New types of CX testing include A/B and canary tests as well as crowd testing. Like other operational data, CX data is also typically voluminous and often unstructured. While CX teams specialize in such principles, testers need to acquire sufficient skills to glean insight from the CX process and data, and collaborate effectively with such teams.
Monitoring and operations skills: Understanding of monitoring principles (e.g. creation, testing and deployment of monitors and use of data from monitors) is a key skill required for TestOps. Similarly, testers must also understand operations principles (incidents, failures, alarms, MTBF, MMTR, configuration definitions, etc.) so that such information can be leveraged for testing purposes as well as for collaboration with ops teams.
Reliability skills: Reliability is also another key operational quality attribute that testers need to be familiar with. This includes chaos/resiliency testing, deployment and rollback testing, and configuration testing. The discipline of SRE has evolved within DevOps to address reliability, and many of its principles overlap with TestOps.
New tools skills: In addition to using traditional testing tools, TestOps also requires the use of additional tools such as monitoring, data analytics (for production and CX data), CX testing (such as A/B and canary testing) and reliability testing. Technology innovations that support testers with these new capabilities include RunScope (a lightweight API monitoring tool that can be used for monitoring both in test and production environments), Blazemeter (for performance testing for CX and reliability), Optimizely for A/B testing. For data analytics, technologies such as automation.ai simplify the work of data aggregation and modeling so that testers can more easily extract actionable insights from data collected across the lifecycle.
Key TestOps Collaborations
In addition to new skills, testers must now learn to collaborate better with additional stakeholders as alluded to above. These include:
Collaboration with data science and AI/ML engineers: This is necessary for testers to tweak or build out new algorithms and models required to perform the analytics needed for TestOps.
Collaboration with CX engineers and teams: This is necessary for testers to monitor the appropriate CX metrics and data, and to study the impact of their testing/QA activities on CX.
Collaboration with Operations engineers and teams: This is fundamental for TestOps to not only support all the shift-right testing activities, but also to gain access to production data that is required for TestOps analytics. This is analogous to how testers collaborate with developers as part of shift-left.
Collaboration with SREs: As mentioned, there is a fair degree of overlap between SRE and TestOps principles, and hence collaboration with SREs is key for testers. See my blog on how testers and SREs can work better together to improve reliability.
Conclusion
Hopefully, this article has provided readers with some insight into the emerging discipline of TestOps. As we build increasingly distributed microservices and cloud-native systems, and as the discipline of SRE matures, we expect to see more of shift-right testing practices that enrich TestOps.
The cloud-native approach is not just about compute power and cloud services. A “set it and forget it” attitude won’t work with cloud native; there are far too many moving pieces and there is just way too much complexity. Instead, this approach requires constant testing, optimization and improvement. This is epitomized in the two key phrases of the DevOps methodology: continuous integration and continuous delivery.
To make this a reality, however, it takes a strategic approach to defining application patterns that will govern the way in which cloud-native applications are built and shipped, rather than a one-off effort. Since AWS is the leading cloud provider, we should discuss which application patterns are relevant to the AWS platform. There are many aspects to consider when it comes to AWS application patterns, and we will cover them here.
Operational Excellence
The best starting point for application patterns is a commitment to operational excellence. Typically, the term “Ops” is used to identify the IT team, but in this case, operational excellence rests on the shoulders of Dev, QA and Ops teams. In cross-functional DevOps teams, every member should strive for operational excellence at every step–even in the smallest of tasks.
In their white paper on operational excellence, AWS encourages its users to manage operations as code. This helps avoid human error, enables automation of complex processes and allows for quick and easy rollback when errors occur. Failure, according to AWS, is an opportunity for learning. They even advise performing “pre-mortems” that try to anticipate potential points of failure before a deployment.
AWS Trusted Advisor is a useful tool that enables this type of accountability and shared responsibility within a team. It checks and reviews your new and existing AWS workloads looking for opportunities to save costs, improve performance, tighten security or follow best practices for scaling operations.
Security
Security is a pattern that is non-negotiable. The term “DevOps” has inspired another variation called DevSecOps that signifies this fact. Security operations take center stage when talking about DevOps.
The interesting thing about DevSecOps is it actually results in more freedom, flexibility and trust across teams when done right. This is unlike traditional approaches to security which created silos, stifled creativity and slowed down innovation. Security is now a key piece of the puzzle, not an afterthought. AWS has many machine learning-based security solutions such as Macie which automatically identifies and secures personally identifiable information (PII).
Along with this, configuring role-based access controls using AWS IAM is essential to have greater control over who or which applications access data and for how long. Access Control Lists govern access to data stored in AWS S3 buckets.
The news is full of instances of data breaches due to weak security policies and access controls. Capital One is a recent example of a data breach that affected 100 million of its users. While security in the cloud is more complex than before, AWS has the necessary tools and systems in place. It’s up to organizations to implement a security strategy that prevents misuse of their data.
Reliability
Today, there is a low tolerance for downtime because of the potential loss of revenue. With the stakes so high, reliability is a pattern any application custodian would have high on their list of priorities. AWS has multiple reporting and health monitoring tools to keep you constantly on the pulse of your applications.
Architectural patterns like microservices enable better resilience by decomposing monolithic applications into more manageable chunks called services. This “divide and conquer” approach leads to higher availability, which is critical for SaaS applications.
Patterns like chaos engineering seek to intentionally kill random instances and services and force the DevOps team to figure out workarounds and backups for every scenario.
Application Performance
With the explosion of data, applications are always dealing with lots of data to transfer, store and process. An application’s performance depends on three factors: the backend data storage layer, the middleware components and the frontend.
The backend data layer is the most critical component when it comes to performance. Choosing the right storage formats, database types, storage devices and organization patterns all contribute to the speed of data access. AWS has an amazing array of storage services, including S3, RedShift, Aurora and Glacier. AWS gives its users every type of storage solution, all of which should be considered to improve data management.
AWS is devising new ways to query large datasets without bearing the load on your existing systems. RedShift Spectrum is one example of this. It is a great way to “minimize the need to scale Redshift with a new node, which can be expensive.” If you’d still rather not have to manage any RedShift clusters on your own AWS Athena is a great serverless alternative
That gives you all the power you need to query large datasets in S3 without having to maintain any clusters of your own.
This pay-per-use model is making its ways across the entire AWS stack with services such as ECS and Fargate offering completely serverless experiences when operating containers at scale. What started out with AWS Lambda is catching on fast, and many AWS services like the ones mentioned above enable serverless experiences with a focus on application performance.
Cost Optimization
When moving to the cloud, pricing is completely different. Rather than paying for perpetual licenses, you now pay a small fee for every small resource that you use. Though small on its own, you’ll soon realize that cloud costs can shoot up easily if you’re not paying close attention to your spending.
Cost optimization is necessary as you scale operations in the cloud. Cost optimization includes looking for more efficient services, whether that’s within the same vendor platform or a different platform. At a minimum it may require a lift-and-shift approach to transition from expensive old infrastructure to inexpensive new alternatives. For example, shifting workloads from VMs to containers to serverless within the same platform can show great cost savings, since containers and serverless are generally more cost-effective than VMs.
It pays to leverage the expertise of experts in this area. They should ideally be people who have spent years using AWS, understand Well-Architected best practices and have found ways to cut costs and derive more value from the same budget.
Application Patterns Enable Extensibility
Microservice applications adopt a pluggable architecture. This enables them to be easily integrated with external applications. Typically, API-based communication patterns are most common. AWS API Gateway is a service that manages API design for AWS applications. In some cases, an SDK is the better choice–for example, when you need more control over the developer experience and integrations. It’s important to know when to choose either option.
Application Patterns Enable Portability
Portability is the ability to move workloads across instances without interrupting their functioning. As instances change, the workloads which they support shouldn’t be affected. This is only possible if there is a clear separation layer between the workloads and the infrastructure that runs them.
Application Patterns Enable Shift-Left
The DevOps methodology is known to encourage a shift-left approach to software delivery. This means that not only Dev, but also QA and Ops contribute to the design and development stage of an application. This fosters deeper collaboration across teams and breaks down silos.
A shift-right approach holds true as well. Shift-right means developers are responsible for the code they write. They are equal owners with operations teams for the resilience of the apps they build and also the bugs they introduce into the system. This practice of putting ownership and responsibility where it belongs is very healthy for a DevOps team.
AWS facilitates pipeline management using tools such as CodePipeline, CodeCommit and CodeDeploy. They each perform similar but complementary roles and help build a CI/CD pipeline. They make it easy to get started with CI/CD within the AWS platform.
For organizations further down the line in their cloud journey, AWS’ pipeline management tools may seem too simplistic for something like GitOps. Instead, they’d want to look outside AWS for tools such as Jenkins X or Weave Flux which are designed with GitOps automation in mind. Whether it’s the simple or advanced route, a pipeline brings great benefits over a manual development process.
Conclusion
Though tools enable the implementation of helpful application patterns, on their own, they are ineffective. Application patterns need a deliberate strategy and an approach for a start. Only after a strategy is in place, can you move to deciding which tools to use. With a strong strategy in place, any of the tools mentioned here can be swapped out for other better tools and the results will be the same. It’s important to not be distracted by the next shiny tool but keep a keen focus on the strategy for implementing AWS application patterns.
As we’ve seen here, application patterns are diverse, but they all build upon each other. Without even one of these pillars, organizations would not be able to realize the goal of cloud-native applications. On the other hand, if they give careful thought to each of these application patterns, organizations would deliver cutting-edge applications that can only get better with the passing of time.
This sponsored article was written on behalf of Eplexity.
In some areas of IT, we utilize the idea of shift right quite a bit. Testing and security are two areas where it comes up often. The idea is that tracking late in the process–usually while the application is in production–leads us to find more insidious issues that might have remained hidden, and gives us a kind of quick reaction force to mitigate issues.
All true, and it works relatively well. Some shops use this concept to push off necessary testing, but for the most part it allows us to keep moving while assuring the system is essentially correct. But the idea of catching issues late in this iteration might be misguided. It does do just that, but in the world of cyclical iterations, shift right is also shift left for a future iteration. The idea being that we’re catching something today that will be resolved for the next iteration. And presumably for every iteration after that.
We often put a temporary fix to resolve the in-production issue, then list it as something to be more permanently resolved in a future iteration. Adding to a future iteration’s to-do list is the ultimate shift left. And that matters. We’re catching it late for this iteration, but early for the next one. And as a result, overall quality increases.
Standardizing this process would be the next step. If it’s an easy fix–a minor change to code or config–it’s shifted right. If it’s a larger issue that will need to be considered in a future iteration, it is considered shifted left for that iteration. It’s an input, not just a fix. And an input that deserves a high priority if it reflects a negative impact upon the running system.
In the daily grind, this is pretty much irrelevant. We have a problem and say, “there, that’ll keep it going. Put in an issue to make a better fix and move on.” But in the next iteration’s prioritization, it is far more relevant, “we have this massive pile of things to do, what makes this iteration?” Should include a heavy weighting for things that are limping along on a manual fix. Often, the existence of a workaround reduces the weighting, and these things don’t make the cut. But permanent fixes mean you don’t have to hustle around fixing the same old problem each time the app is deployed, and that’s greatly what DevOps is all about.
There is an interesting dichotomy between shops/groups with a high iteration rate and those with a much slower cadence. The cost of not permanently fixing these issues is too high when you’re deploying frequently, so they tend to be handled this way already. But slower iteration shops/groups often will let these fixes slide in favor of new functionality. The burden this creates is not worth a shiny new bit on the UI. Start slating permanent fixes for production issues as highest priority, don’t add steps to deployment because you failed to resolve the more fundamental issue.
DevOps requires speedy resolution, but long-term, any type of development effort requires adequate resolution. With iteration after iteration, you are paying a price for putting off fixes, so don’t. Shift them left in the next iteration, and spend that time of each release on something more productive.
I am always surprised by the urban legends around DevOps. One of my favorites is, If you implement a DevOps methodology, you can stop doing test. People famously quote that Netflix only tests in production as evidence of this. This is perhaps more like of game of Telephone, where much of the story gets lost in translation. It is true that Netflix created a tool called “Chaos Monkey” which generates failures in production as a way to test the fault tolerance of its software in large-scale production use. However, this is not Netflix’s only test; it is an additional type of testing at scale that allows the company to test in new and different ways. Netflix still tests in software development.
Okay, now that we are past the notion that DevOps is about eliminating testing, let’s look at what really happens with test in the DevOps world. In fact, the goal is for testing to happen everywhere and all the time. Let’s look at the two directions test can shift: left or right.
Shift Left
DevOps methodology encourages automation of activities across the entire development process. This can be realized by automating the link between development and test. Often implemented with an automated build tool such as Jenkins, a set of automated tests are triggered whenever a developer completes a build of new code. Of course, this means that both development and testing processes must be automated. Once you reach this goal, then testing can happen all the time. Every time developers want to check in new code, they can trigger a small set of “smoke tests.” If those succeed, they can check in code and when the build happens, they can trigger a larger set of regression tests. The result is a left shift of testing into development.
Benefits of left-shifting test include:
Move testers into scrum teams to work on stories together with developers.
Design tests in parallel with development.
Give developers and testers the flexibility to help each other to reduce bottlenecks.
Early detection of bugs, shortening the critical metric for time between creating a bug and fixing it.
Increased transparency of code dependency issues because they are raised during the nightly (or more often) regression testing.
Virtual elimination of the test bottleneck that occurs after development is completed.
Shift Right
The urban legend about Netflix may be wrong, but that doesn’t mean that there isn’t merit to right-shifting test into production. Automated testing in production is effective for tests that would be impossible to perform outside of production, including scale testing and resiliency testing. Production is also a great place to introduce A/B testing by offering different interfaces to some customers and measuring the differences in business results.
Benefits of testing in production include:
Better understanding of the real production environment and workloads.
Early feedback from customers on new features and user experience.
Better resiliency.
Overall better user experience for customers.
No Shift
Wait, if we shift left and shift right, is there any testing left to be performed in test? There are still a range of tests that need to be performed by the test team. With full automation, these tests can be performed at a variety of times and don’t need to be a bottleneck to release. These tests include:
Performance testing
End-to-end use-case testing
Security testing (don’t think about doing this in production!)
Manual testing – often for user interface or mobile testing that is most difficult to automate
DevOps methodology is not promoting the end of testing, but in fact a widening of testing to encompass all of the steps in the DevOps cycle. You know what they say: Test early and often! And next time you are sitting around the campfire discussing urban legends, let’s leave Netflix out.
About the Author/Joan Wrabetz
Joan Wrabetz is the CTO for QualiSystems. Most recently, she was the VP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.
Joan Wrabetz is the CTO for QualiSystems. Most recently, she was the VP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.