Throughout the history of IT, we’ve had our share of purists. OS purists, language purists, etc. The stark one was the “That is not Object Oriented Programming” crowd. While the poster-children of IT purity, we don’t really hear from them anymore—and thank goodness for that.
But a new crowd has reared its head in DevOps specifically, development more generally. They don’t believe there is ever a reason to modify a running container. Call them deployment purists, though maybe one of you has a better term.
Like OO purists, this approach ignores the reality of the environment we work in. Yes, the best option most of the time is to fix an issue and roll out a replacement container. But, the fact is that claiming anything else is not using containers “correctly” is patently wrong. My go-to example for why I say this is the easy one, because explaining complex infrastructures while trying to make your point is a waste of time.
Rolling feature releases in most organizations that do them regularly are achieved with environment variables on feature tags. But changing an environment variable is changing an instance. So, are all these organizations wrong, or perhaps the idea of purity is the problem when it comes to IT? Certainly, the driver for DevOps in many organizations was that purity had created islands of toolsets with varying delivery timelines and quality controls.
I contend it is past time to let purity be a guideline instead of a mantra. Yes, in most situations rolling out a new instance is the correct response, but in many–live applications that will drop connections and lack enough redundancy to bleed users off, for example–the correct solution is to modify the instance. “Never change production” is a good mantra, but it is a terrible purity test. We should avoid it, while acknowledging that not all situations allow us to make that choice.
Keep on rocking it. We’re weeks into COVID, and I can hear the stress in my operations friends’ voices. Know that the rest of us are grateful to have you keeping things running for us and bending over backward to make certain we have the access we need. Thank you, from me, for all of the people relying upon you.
As we expand our digital world, there is an ever more urgent need to look after businesses that need to operate faster and with more agility. Because of this, DevOps has grown quickly and become central to a lot of organizations as they pursue a competitive advantage. However, some organizations struggle to implement DevOps as they are unsure how to approach them.
Here are our top six tips on how your team can integrate DevOps successfully.
Discipline and Process
It isn’t just a case of installing DevOps and then getting started with the tools. Your whole company needs to have real clarity about what DevOps is and what exact business needs it can solve. Most importantly, all your team needs to be prepared to change the way things have been done in the past.
Start this process by working out your current application value streams. This is the process needed to move your products from development to production. Being clear on what your constraints, bottlenecks and wait queues are will help you to identify activities to concentrate on.
By identifying areas of inefficiency in your current delivery process, you have an opportunity to make changes in your organization. Encourage your team to ask questions: “Why do we do this? What’s its business value? How can we make it more efficient?”
Release Rates
Any initiative within DevOps should focus on your business’ requirements and be specific to you, not just about getting improved tools. Try not to launch a DevOps initiative before getting clear on the business reason for doing so.
Instead of just focusing on release rates and how to do things faster, look at the business value that will enable. So, for example, an improved release rate could help your team to innovate more quickly.
Collaboration
Many people associate DevOps with automation processes, but it is primarily about collaboration and communication. If you do not have strong collaborative practices that are embraced by everyone in the team, and across all stages of development, simply automating the processes will not give you the benefits that your business is looking for.
By using collaboration, you can get the input of the whole team into which areas need the DevOps implementation first and what impact it can have to each stage of the development process.
Communication and Culture
Here we need to look at your company’s manual processes, which tend to be the most frequent cause of mistakes and delays. Look at your pipeline and keep an eye out for the word “manual.” This could be manual testing or deployment. Anywhere that involves your team communicating through a ticket process. These are classic problem areas for waste and may be where DevOps automation could lead to big rewards.
To achieve this, you need to get your teams onboard with the changes, especially operations, as they are often overlooked when achieving buy-in. The whole organization needs to be excited about the changes and the same process needs to be implemented across the pipeline.
Get Clear on Data
Select metrics that are closely linked to your business goal. How long does it currently take for your product to get from idea to market?
There are three pieces of data to measure to determine this. The lead time, deployment frequency and mean time to recover. All three of these are key to understanding which parts of the process can have the biggest impact when you implement DevOps.
Autonomy
The important thing is to use the same automated system whenever a product goes to any stage of the pipeline, whether that’s development, quality assurance or production. You do not want team members having to manually click configuration steps, or copying software. Each deployment should have the same automation so that the process can operate smoothly.
Starting first with the slowest part of your pipeline, introduce automation one stage at a time. Once the slowest has been automated, look at the next slowest. This way you are always improving processes but getting the biggest shifts each time.
Provided that you have all team members in alignment with the purpose for DevOps, every new step of progress is a win. Then win more support for the bigger steps that the company may need to make.
As more organizations use feature flags and experiments, it’s critical to understand that some of these changes should exist for only a short or specific period of time, and then be removed from your codebase. Removing flags when appropriate avoids technical debt and complexity that can make your code fragile, harder to test and more difficult to support.
The simplest ways to evaluate when a feature flag or experiment should be removed include looking at how long it’s been in the system and how many different states it’s in. If the feature flag has been in your system for a long time and all of your users have the same state of the feature flag, then it should likely be removed.
However, evaluating based on just those two factors isn’t enough. It’s smart to also evaluate the purpose of a flag or experiment before finalizing the decision to remove it. The appropriate lifetime of a feature flag or experiment, and when it should be removed from your codebase, depends heavily on its use case.
When to Remove Temporary Flags and Experiments
If a feature is designed to be rolled out to everyone and you don’t expect to experiment on the feature once it’s launched, then you’ll want to ensure you have a ticket tracking its removal as soon as the feature has been fully launched to your users. These temporary flags may last weeks, months or quarters. Examples include:
Painted-door experiments: These experiments are intended to be used only in the early phases of the software development lifecycle and aren’t intended to be kept in the product after they have validated or invalidated the experiment hypothesis.
Performance experiments: These experiments are intended to put two different implementations against each other in a live, real-world performance experiment. Once enough data has been gathered to determine the more performant solution, it’s usually best to move all traffic over to the higher-performing variation and remove the related flag.
Large-scale refactors: When moving between frameworks, languages or implementation details, it’s useful to deploy these rather risky changes behind a feature flag so that you have confidence they won’t negatively impact your users or business. However, once the re-factor is done, you hopefully won’t go back in the other direction.
Product re-brands: If your business decides to change the look and feel of your product for brand purposes, it’s useful to have a gradual rollout to gracefully move to the new branding. After the new branding is established, it’s a good idea to remove the feature flag powering the switch.
How to Review the Lifetime of Permanent Flags and Experiments
If a feature is designed to have different states for different customers, or if you want to control its configuration for operational processes, then it’s likely the flag will stay in your product for a longer period of time. Examples of these flags and experiments include:
Permission flags: These flags are useful if you have different permission levels in your product, such as “read-only” that doesn’t allow edit access to the feature. They are also useful if you have modular pricing like an inexpensive “starter plan” that doesn’t have the feature, but a more costly “enterprise plan” that does have the feature.
Operational flags: These flags control the operational knobs of your application. For example, these flags can control whether you batch events sent from your application to minimize the number of outbound requests. You could also use them to control the number of machines that are used for horizontally scaling your application. In addition, they can be used to disable a computational expensive, non-essential service or allow for a graceful switchover from one third-party service to another in an outage.
Configuration-based software: For any software or product that is powered by a configuration file, this is a great place to seamlessly insert experimentation that has a low cost to maintain and still allows endless product experimentation. For example, some companies may have their product layout powered by a config file that describes in abstract terms whether different modules are included and how they are positioned on the screen. With this architectural setup, even if you aren’t running an experiment right now, you can still enable future product experimentation.
Note that even if a flag is meant to be permanent, it’s still critical to regularly review these flags in case they become obsolete or should be deprecated, which would then add technical debt to your codebase.
Establishing a Feature Management Process
Creating an always-on feature management process goes hand-in-hand with using feature flags and experimentation effectively. Some organizations use a combination of a task tracking system and their feature flag and experiment platform to manage this cleanup process seamlessly and quickly. If the state of feature flags and experiments can be synced with a ticket-tracking tool, then an engineering manager can run queries for all feature flags and experiments whose state has not been changed in the past 30 days, and track down owners of the feature flags and experiments to evaluate their review. Other teams schedule recurring feature flag and experiment removal days to ensure engineers review the oldest items on a regular basis to minimize technical debt.
The benefits of feature flags and experimentation to help teams ship code and launch new features quickly and safely far outweigh the needed management to minimize codebase technical debt and complexity. When managed as a recurring process, looking at all key factors (lifetime, state and use case), teams can feel confident that their systems will remain efficient and reliable.
Zero-configuration hosting solutions such as Zeit Now and Netlify are taking the frontend community by storm due to their ease of use. If we add static site generators such as Gatsby/Next.js/Nuxt.js/Hugo/Jekyll to the mix and deploy our statically generated pages to a content delivery network (CDN), we get something highly desired with minimal effort: distributed auto-scaling applications with zero configuration.
The usage of static site generators and CDNs became quite popular, so much so that Netlify decided to give it a name: JAMstack. At first, this might not strike you as something very newsworthy. Haven’t we all deployed static sites such as blogs or documentation on CDNs? What is different about this approach?
To answer that, let’s first look at the definition that is typically used to explain JAMstack: The JAMstack is a static HTML site, rebuilt automatically every time you update the content, and deployed directly to a CDN.
Of course, that still does not explain why this approach is different. The first time I read this definition, I wondered how I could ever build something more dynamic than a blog with such a stack. However, note that nothing in the previous description says that every element of our site has to be statically generated. In fact, the JAM acronym stands for JavaScript, APIs and Markup, where the letter A is the answer to dynamic content.
The idea is to pregenerate markup where you can and then enrich these statically generated pages with dynamic content fetched from APIs. The APIs of choice in a JAMstack application are typically APIs from third-party services (SaaS platforms) or custom-built with Functions as a Service (FaaS). Both Netlify and Zeit have great zero-configuration FaaS offerings to write APIs for your JAMstack. In essence, developers in the JAMstack community are often looking for a specific set of requirements:
Zero configuration—easy setup, easy deploy, no server management.
Scaling—your application can handle sudden spikes of increased traffic without extra effort.
Speed—low latencies and fast page loads (thanks to the distributed nature of CDNs).
Pay as you go—only start paying when your page actually serves customers.
By combining easy-to-use hosting solutions, static site generators, CDNs and scalable APIs in one stack, we can relatively easily get these requirements.
What If My Application Requires Highly Dynamic Data?
Imagine that you are building a social network for professionals on which professionals can create profiles. Professionals create profiles to promote their skills, and companies search the network to hire professionals for specific projects. For such an application, you need to store, search, filter and paginate data.
To build such an application, you need a database, but not just any database. Ideally, you want to avoid losing the zero-configuration, pay-as-you-go, scaling and speed benefits that a JAMstack application typically delivers. There are several features a database can possess that make it a perfect fit for a JAMstack application.
In this article, we will describe these features and explain how each of them makes a database a better candidate for a JAMstack application. Since such applications often rely on serverless functions, many of these properties also apply for applications built with FaaS.
Database Properties that Work Well with JAMstack and FaaS Applications
Multi-Region, Globally Replicated
To understand why multi-region distribution is desirable, let’s revisit why static websites on CDNs are incredibly fast. A CDN is fast to deliver your content because it contains copies of your content at different locations. When content is requested from the CDN from a specific location, it will attempt to deliver that content from the closest location to the requestor. In order to get an idea of how much that matters, take a glance at the Zeit CDN status page which shows you the difference in latency between your current location and other locations. By deploying our applications to a CDN, our pages automatically load from the closest location to the user, which results in low loading latencies. And low latencies result in a great user experience.
In order to keep this user experience, the dynamic data that will be loaded from our APIs has to exhibit low latencies as well, and the best way to achieve this is to use a distributed database. Databases that feature multi-region replicas such as FaunaDB, Google Spanner or Amazon Aurora not only provide the dynamic querying that you need, but also deliver your data from the closest location, just like a CDN.
Connectionless
Many traditional databases are not connectionless, which means that they require persistent connections with the process that queries the database. There is often a limit on the number of connections that a database can handle, and setting up a connection takes time and puts more strain on the database. A database that supports a JAMstack application will either be called from the frontend (if it’s possible to do securely) or from serverless functions. An increase in the number of users increases the number of running functions or frontend calls.
Having a limit on connections and/or creating/destroying many connections then quickly becomes a problem. An ideal JAMstack database behaves like an API; it does not require connections. There is an excellent article on this topic that describes the connection problem from a FaaS perspective.
Flexible Security
Providing security out-of-the-box takes away tedious configuration from developers. If this security layer can also provide a means to securely access the database from the frontend, we can potentially write our application backendless, eliminating an extra hop and reducing the low latency we’ve gained from distribution even more. When a database is connectionless and can be accessed securely from the front-end, we consider it to be frontend-friendly. However, security requirements can be complex. If we completely rely on the database security layer, it has to be flexible enough to express custom rules. Firebase Cloud Firestore, FaunaDB and the upcoming Userbase offer direct access via clients, like browsers.
Scaling and Pay-as-You-Go
Before we add a database, our typical JAMstack application is completely auto-scaling and pay-as-you-g(r)o(w), which is considered an advantage for many startups. We only pay for the CDN bandwidth and function calls that our application actually uses. If our database is also pay-as-you-go and scales automatically, we do not lose this advantage.
Strong Consistency
Strong consistency is quite a technical matter, so we’ll explain it briefly, and for the technical details, this article is a great resource. Strong consistency is important in distributed databases when your application stores and updates data. In that case, there are two requirements to preserve a great developer experience:
You never lose data: When a write or update is acknowledged, it can no longer be dropped.
Data updates are predictable: When you save or update data, a consequent read will reflect these changes.
Strong consistency (or strict serializability) is the opposite of eventual consistency. By adopting a database that does what you expect it to do, you can focus entirely on your application. Only a few distributed databases provide strong consistency without limitations.
Conclusion
JAMstack applications are not limited to pages that are entirely static; they can be partly static and partly dynamic by retrieving data from APIs. A distributed database can be one of these APIs and is essential in case your application contains dynamic pages with filtered or paginated data. Ideally, such database fulfills the six described requirements so that we can build dynamic JAMstack applications without losing speed, scalability, pay as you go or the zero-configuration approach that we love.
In case you wonder how a partly static, partly dynamic JAMstack can be created in an elegant way with modern frontend frameworks, take a look at hydration, which might be the topic of a forthcoming article.
There is little argument that automated testing has emerged as an industry best practice. Yet, there remains a wide variety of adoption and maturity in the market. While automated testing is imperative for DevOps maturity and success, I see in my work several popular canards that often stand in the way.
Reject these five common myths within your organization to improve your testing maturity and propel your DevOps objectives.
Myth One: We Don’t Have the Time to Slow Down and Add Automation
You don’t have the time not to add automation. Your competitors are most likely already automating their test processes, and are reaping the benefits from it. For example, companies like Amazon and Netflix have instituted end-to-end automation that allows them to push builds to production in record time. Organizations like this have become juggernauts as they are able to quickly address customer expectations with new software and services at the speed of the market.
While Amazon and Netflix are obviously examples of extreme automation, moving down the path to automated tests will begin the process of adding distinct benefits to the organization. Benefits that will grow key business goals like reducing time to market, and growing quality that ensures customer satisfaction. For example, automated testing brings with it higher accuracy rates, greater code coverage, enhanced detection of bugs and greater reusability.
Myth Two: Automated Testing Is Too Difficult and Frankly, Just Not Worth It
Automated testing is worth the investment. Much like insurance, the added time and expense to begin integrating automated testing into your processes are necessary trade-offs to reduce business risk. Automated testing helps reduce failures, avoiding all-hands-on-deck type emergencies—and, honestly, at 5 p.m. on a Friday, wouldn’t you rather not fight a four-alarm blaze? That said, automated testing should not be a singularly focused QA effort. Good quality efforts really should be organization-wide initiatives.
Automated testing is less difficult than trying to engineer quality at the tail end of anything. If you do not have any test automation in place today, consider starting with an assessment. Identify areas where automated testing is needed and begin addressing those gaps in a prioritized manner. While your ultimate goal may be to automate every gap on the list, start small, take steady steps towards your goal and be sure to celebrate early wins. A parallel approach enterprises take when getting started is to build automation into greenfield projects, which gives them automated tests out of the gate.
Myth Three: We Have to Hire a Specialized Person
I think the Washington Post would give this claim four Pinocchios as it is blatantly untrue. Anyone on your team can learn the needed skills to start down the path to automated testing. And, honestly, learning and growing in your field is an important skill in-and-of-itself as new trends, technologies and tools emerge on a seemingly daily basis. Learning how to add automation to the testing process can help create a virtuous cycle, too, where you will see dividends paid back in time savings from the time you took to learn these key skills. And, the faster feedback loops will make you and your peers more efficient and effective.
Some may counter-argue that there are specialty testing types such as PCI testing, for example. And, that a specialist is needed for these. Yes, many teams have special circumstances that require special tests. However, don’t let this myth be an artificial roadblock here, either. For, there are many automated tools on the market that will help ensure success with these special cases. Moreover, many of these tools are platform-agnostic, making for easy adoption.
Myth Four: We Have to Automate Everything
This all or nothing viewpoint is just not true as it is not possible to automate everything. While I would encourage some tests, such as unit tests, to have a test for every line of code, other tests like front-end tests would be particularly challenging to automate. The greater the organization’s maturity, the more nuanced their view of when and what to automate. Conversely, an “automate everything” approach is a sign of a less mature enterprise.
On one hand, I see organizations that have completely automated unit testing and have developed a tailored approach for front-end and performance testing. On the other hand, I see organizations with little to no automation where software is rushed out the door and dev is not given any time to develop unit tests. In these cases, software sent to the market is not fully tested—and while rare, sometimes it’s not tested at all.
Bad code is risky to a company’s customer satisfaction, reputation and customer loyalty. Going back to test code once it has shipped is a multi-fold lift compared to doing it as an automated part of the development process. Automation also makes it easier to make improvements to existing features in your software or infrastructure. When you have automated tests in place (especially unit tests), you drastically reduce the amount of time refactoring and enhancing takes.
Myth Five: Continuous and Automated Testing Are Synonymous and We Don’t Have the Resources for Continuous Testing
While continuous testing and automated testing are related, they certainly are not the same thing. You can’t obtain continuous testing without the help of automation. However, you can have automated testing without continuous testing. On the spectrum of testing maturity, continuous testing is certainly at the far end as it requires DevOps processes, pipelines and automated reporting as prerequisites. If your organization isn’t at this level of maturity yet, that is OK. Start small and work your way toward continuous testing as a gold standard. For example, consider running unit tests with each new build, implementing a fast feedback loop to development several times a day.
According to a survey by QASymphony and TechWell of testers and QA leaders at mid-size and large enterprises, “widespread test automation adoption still has a way to go, with more than three-quarters of respondents automating less than 50% of their testing.” Yet, it’s important for organizations to speed their processes in order to stay competitive in a quickly evolving marketplace. Dispel these myths within your organization to begin competing more effectively as you progress your automated testing maturity.
In my last piece on cultural SRE, I covered the basics of defining what SRE means for your team. However, some teams may not need dedicated SRE support. This is where the central organization comes in handy. Dedicated resources are focused on building guidelines, standards, services and platforms for teams to consume that can operate autonomously in your environment. Having a broadly communicated central team also allows for these other teams to know where to go when they need assistance or would like some recommendations on their technology stack.
Application/Service SRE Versus Infrastructure SRE
Application SRE includes:
Embedded for application/service teams.
Architecture guidance for new services and infrastructure.
Design and implementation for modern technologies.
Support within the application/service team.
Automation for apps and services for the team.
Automation for operational work.
Monitoring/metrics for application/service team.
Benchmarking and performance assistance for code.
Infrastructure deployment and architecture for applications and services.
Write documentation and runbooks for alerts issued by application/service stack.
Infrastructure SRE includes:
Build and management of “plumbing” technical infrastructure (provisioning, OS, dns, dhcp, networking, central auth, etc.).
Build and management of consumable services and tools (message queues, databases, distributed compute farms, API services/integrations, containers, etc.).
Infrastructure as Code (IaC).
Implement Global IR&IM process.
Support for Application SRE teams.
Architectural guidelines and best practice documentation.
Organizational Layout
Organization structures, whether we like to believe it or not, really help in the flow of communication throughout the team. It allows for the team leaders to form a single mission and have that mission carried out by their team members. I mentioned in Part 1 of my SRE Narnia guide that I believe this organization must be centralized to be successful. Below is a high level overview that shows the separation between application SRE and infrastructure SRE.
The leads can handle multiple teams and disciplines. This can shrink and scale as you see fit within your organization. Application SRE is straight forward. The infrastructure SRE could be embedded across existing infrastructure teams or form a new team focused on a specific project.
Example SRE Interaction
Below is an example interaction between an application team (this one runs a ruby app) and an SRE hierarchy. You will see a team called “Frontline Support” as the initial interaction. This team is not required but is a nice to have. It helps in any environment big or small, and can really offload a significant amount of the operational workload along with having a global view of issues coming in.
The duties for frontline support are to follow runbooks for alarms flowing in from the monitoring systems. It is similar to a traditional NOC but with more expertise across the stack they are watching. It also allows for trend analysis and bringing data to conversations around reoccurring issues or bugs in the system.
I do recommend using this as a starting point for junior level team members. It allows them to learn the system quickly and buddy up with more seasoned members that are either on this team or have the pager for the week. It is a great way to train and onboard your technical employees. Nothing works better than showing them what is broken.
Summary
Overall, everyone’s journey will be different. There are quite a few books and examples that exist out there, but please only use them as ideas to form your own opinion on how these organizations should work. There is no prescription and nothing is ever perfect. I have left out quite a bit of information in this post but I am always interested in having conversations about this topic and others alike. There is one consistent theme you’ll see across the board and that’s shaping the culture. Focus on your culture and hire great talent along with great leaders. Good luck and happy hacking.
As developers and entrepreneurs create new products and services, it is critical for a powerful communication solution to sit at the heart of the solution and support all customer engagement. Reason being, when customers begin to engage with the application, they’ll expect easy access to troubleshooting tools like live support chat.
Neglecting to offer such support could make or break the overall customer experience. In fact, 78% of customers have reportedly backed out of a purchase due to a poor customer experience, and at least 54% of respondents said they would prefer to speak to a chatbot instead of a human customer service representative if it saved them 10 minutes.
However, doing so is no easy feat when it comes to automated communications services. Human-level support is effective, but often creates unwanted lag and is not easy or cost friendly for companies to scale. Automated customer service technologies and solutions are a viable choice but need to be able to quickly recognize the intent and context of what a human is saying in order to solve their problem effectively.
Not only that, bots need to be able to maintain a live dialogue with the customer and keep the conversation flowing in the same way a human customer service representative would. This ultimately creates an even bigger challenge for the developer, outside of the initial task of developing an effective and engaging application.
How Have Developers Traditionally Approached Customer Communications?
We often see companies looking to implement things like text message, video or voice calling into their customer communication strategy. While it may seem like a small task from afar, most in-house or self-built solutions require intensive upkeep and are often difficult to scale.
Not only that, once a company decides the type of communication it wants to deploy, the developer often struggles to find the right overall architecture and means for implementation. Once these are finally in place and working properly, expanding that functionality down the road makes for yet another grueling process. Because of this, many developers often decide to outsource all together.
Most enterprises and contact centers will have communications systems, such as Private Branch Exchange (PBX), that are based on software as opposed to being delivered as a service. This software may be completely run within an office or partly inside the enterprise’s cloud environment. This software was not designed for the modem cloud era, and the more advanced software option generally won’t offer any flexibility to alter or customize communications flows.
What Are Some Best Practices for Integrating Customer Communications into New Applications?
There are a few things for developers to keep in mind when building out their customer communications capabilities. First, you’ll need to decide what type of communications methods are required for optimal customer experience, whether it be text message, voice, video or a combination of the three.
From there, you’ll need to determine the communication logic that leads to the best customer experience. For example, there is no longer a need to follow the traditional model where a customer calls, navigates through an Interactive Voice Response (IVR), waits on hold and then finally talks to someone. Communications flows can be centered around what provides the best experience for the customer, not what the communications system can handle. If a customer service call results in scheduling an appointment, why not text details of the appointment for easy reference by the customer?
Proactive reminders as the appointment approaches are even better. Rather than subjecting customers to annoying IVRs with many simple prompts, they can leverage the technologies that drive Alexa and Google Assistant to help direct customers. Instead of the customer struggling to accurately describe what they are seeing, why not escalate to a video call so a technician can see what the agent is seeing? We also see this with options like automated call backs instead of having the customer waiting on hold to prolonged periods of time.
Next, you’ll need to find the appropriate technology stack that aligns with those standards. This often means using a CPaaS (communications platform as a service) platform that is compatible with your needs. Once you’ve found a solution that can cater to your needs appropriately, you can start building and integrating your communications system.
Choosing the Right Solution
There are a few things to consider when choosing the right solution.
The solution you’re choosing needs an API function that suits your needs. Most of the apps and services need to connect your account with a consistent user ID that offers context into a user’s information and history. The most common practice is using a phone number as the ID because SMS is the fastest and easiest way to reach the customer, but you’ll need robust API capabilities in order to do that in a way that’s quick and catered to the individual’s account.
If scaling your call-handling staff is a concern, you’ll need a solution that automates parts of the interaction to offload human callers. When implemented correctly, this will increase the efficiency of customer communications by solving customer problems faster. Done right, you can increase call volumes without adding to call hold times.
Unfortunately, scale is useless if you’re compromising quality. Automated customer service technologies and solutions need to be able to quickly recognize the intent and context of what a human is saying. Solid Artificial Intelligence (AI) capabilities can maintain a live dialogue with the customer and keep the conversation flowing in the same way a human customer service representative would. It doesn’t matter if the robot sounds and listens like a human if it doesn’t know how to respond to questions and statements in a human-like manner.
Customer communications should be a priority when it comes to building out any new application, but that doesn’t mean it has to be a hassle to implement. By keeping the best practices above in mind, you can deploy quality customer support in a way that’s approachable, scalable and simple to augment as needed.
It seems that every firm is trying to include DevOps in their software delivery cycle these days. This is understandable. According to the 2019 Accelerate State of DevOps Report, companies using DevOps are more likely to achieve elite performance in software development and delivery.
Companies such as Google and Amazon can be deploying thousands of updates a day. These rapid updates improve company performance and result in the provision of better software solutions for the end-user.
In this article, we will look at seven effective tips to get the most out of your DevOps so you can maximize success. Before we look at some of these tips, let’s briefly define DevOps.
What is DevOps?
DevOps is a term for a group of management practices that aim to quicken the deployment of changes to a system while ensuring high quality delivery. It incorporates management theories such as agile infrastructure and agile development. One of the most influential books on the topic, which resonated with a lot of influential business leaders in Silicon Valley, is “The Lean Startup” by Eric Ries.
At its heart, DevOps boils down to the idea that you should be able to implement changes quickly, so you can release a superior product to your customers faster. This ability to bring a superior product to market fast, gives companies that can achieve this edge an advantage over their competitors.
Review Your Organizational Structure
Collaboration is the bond that allows software companies to deploy and execute changes fast. Companies that are experts at DevOps classically reduce the obstacles to collaboration.
In traditional IT organizations, departments are siloed. They interact with each other only when necessary. Hence, they have limited knowledge of each other’s work. This model presents a linear process of developing, testing and deployment.
To reduce these silos, you need to first analyze your organizational structure, and identify obstacles to collaboration. This starts by ensuring management is on board with your vision. You need to:
Review the job description of managers and create KPI that support collaboration.
Create remuneration packages that foster closer collaboration between departments.
Addressing structural issues that impede collaboration between departments will help. You also need to put in place a system of communication between departments.
Fortunately, there are useful tools to smoothen continuous collaboration between different lines of business. Channels of communications should not be limited to sending emails or conference calls. Remember, DevOps requires continuous and fast collaborations to ensure fast deployment.
Usingunified communications (UC) is an effective way to foster collaboration in a company. Unified communications integrates phone calls, email, chat, document sharing, conferencing and other platforms necessary to exchange information within the organization. Importantly, you can also access much of this information in one place.
Reduce Operational Hurdles for Deployment
The way an update is deployed will differ by company. There is usually a system in place to discuss targets and a series of checks in place before deployment. For example, department heads will come up with an idea. They will be developed, then reviewed by quality assurance before operations deploy it.
This can be a cumbersome and time-consuming process, which slows down the speed of development. The problem for many companies is that changes, both large and small, need to go through the same process of quality assurance before deployment.
While it makes sense to have this process in place for major updates, it should be possible to push smaller updates faster. Two things need to happen to achieve this:
Empower staff to make decisions independently while giving them the opportunity to learn from mistakes without risk.
Reduce the administrative hurdles to deployment by creating a ranking system for reviewing updates. This will allow developers to push small updates independently, or with minimal oversight.
Empowering staff to push changes enables you to speed up DevOps.
Plan in Small Batches
Planning in small batches is important in promoting DevOps in an organization. With small batch testing, the cycles of testing hypotheses, acquiring feedback and going at it again can be done within a very minimal time frame. It is fast and efficient—simple.
This feedback might feel obvious. Perhaps the better question to consider when planning in small batches is: How small is small? Let the INVEST concept in agile development help you out.
Independent: Planned small batches should be independent of other parts.
Negotiable: Every batch should be iterable and negotiable upon receipt of feedback from deployment.
Valuable: Each one should add value to the project and the end-users.
Estimable: Batches should contain estimable information for scope determination.
Small: Small should be small enough to be processed within a few hours up to a few days.
Testable: A planned batch should be testable and can be monitored continuously.
Results of deployment, testing and monitoring of each batch should then be deployed at least once a day. Also, in planning small batches, different departments should be kept in the loop. This will greatly benefit DevOps efforts in the organization.
Create Space for Collaboration
Even without DevOps, fostering collaboration between departments is imperative. You can formalize collaboration through regular meetings. However, departments are usually situated in separate office areas, floors or even different buildings. Think about having to walk to a conference room on a different side of the facility just for a meeting.
This kind of setup is an example of a bottleneck for collaboration in an enterprise. And with collaboration being the heart of DevOps, creating a company culture and office space conducive to teamwork is of utmost importance.
SpaceX is an example of a company that promotes collaboration throughout the organization through company culture and physical space management. Offices are open, and each person can see what everyone else is working on. This promotes motivation for employees to work passionately. Also, everyone has access to camera feeds from different parts of the factory, which inspires curiosity about what others are doing. Because of this culture, employees become dedicated and excited to accomplish results.
Take Advantage of Microservices
In processing complex IT systems, it is more beneficial to break bulks to smaller, more digestible pieces. This is where microservices come in.
Microservices or microservice architecture is an architectural style wherein a complex system is broken down into smaller services. Rather than treating a system as a whole, it is perceived as a collection of manageable pieces.
Teams simultaneously working on parts of a system as a whole makes it more prone to breakdown. Utilizing microservices makes monitoring, maintenance and testing easier. They are also loosely connected to each other which means fixing wouldn’t much affect the other parts. Moreover, each microservice is releasable independently.
As an example, Wix started out with a monolith which was useful at first when they were only handling 1 million sites. However, interdependencies and poor quality coding resulted in downtime which is ultimately bad for business. Now that they shifted to using microservices architecture, they quickly scaled up reaching 70 million sites.
Maintain Continuous Monitoring
DevOps promotes fast and continuous integration and deployment. A fast-paced environment is more susceptible to problems. Something at any point can fail within the software development life cycle.Continuous monitoring helps you identify mistakes fast. It ensures the optimal performance, reliability and health of the software.
Setting monitoring targets, like development milestones, can improve this process. A development milestone is an overarching category that can give an insight into your team’s actual performance. Monitoring tells you the performance of your DevOps implementation.
A method to consider is synthetic monitoring wherein your application undergoes simulated customer interaction. This way, you can monitor the exact action and performance of your software before they get released to actual users.
Practice Change Management
DevOps employs continuous integration, continuous deployment and continuous monitoring. It is a cycle of fast-paced changes. Change management is essential to keep the system run like clockwork.
It might be easier said than done because changes have effects on various stakeholders. This includes your customers as the end-users, security, regulatory compliance and company policies. With this, changes within the development, testing, deployment and monitoring should be cohesively managed.
Change management is intrinsically reactive. It usually happens because of feedback received after deployment. But with DevOps, change management should be proactive. Instead of managing large-scale changes, processes are broken down into smaller parts with clearly defined scopes. Furthermore, anticipated changes are scheduled, scaled and communicated across the board. Possible risks and problems are efficiently managed because of this.
Summary
No system is perfect nor exactly the same. The key to a successful DevOps implementation is assessing what works best for your organization. DevOps is not a standardized process. It is not a system that will work for every company. Rather, DevOps is a theory of management.
Improving DevOps at your firm will involve a system of continuous review and improvement. You need to understand what is working, the problems and what you can improve. This can all be done through continuous processes with the help of automation and collaboration. No matter the size of your software development projects, these simple tips and tricks will help you make the most out of your DevOps.
You would be hard-pressed to find something more complex than a modern data center. A state-of-the-art building stocked with all the necessary software for a major company might run into billions in costs if you decide to build and operate your own from the ground up.
Even though many pundits had predicted data centers would become a thing of the past due to the ever increasing adoption of SaaS, as we previously predicted, data centers are still popular with companies and evolving for 2020. One phase of that evolution is for businesses to box up all of the gear in their data center and install it with what is called a colocation provider, or “Colo” if you’re looking for a snappy acronym.
“Going Colo” refers to renting space in a building, usually a rather large building, and moving all your physical servers and network hardware there. You’ll have neighbors who are doing the same thing because this is just a fancified way to say you’ll have a landlord.
The Colo market is getting big in a hurry. The global market was $31.5 billion for this type of data center arrangement in 2017, but is expected to expand to at least $62.3 billion by 2022. What a company, large or small, has to ask itself is if this type of arrangement is a good thing for DevOps. Let’s ponder that.
It’s Cheaper When You Don’t Own the Building
If you remove the possibility that you could be thrown out of the space at a moment’s notice (a worst case scenario that a good rental agreement should address), the idea is a no-brainer from a cost savings perspective.
For an enterprise, let someone else take on the risk of construction and infrastructure maintenance. For a small-to-medium-sized business, the cost of building and monitoring a modern data center makes it impossible.
The unfortunate reality is that, when you do own the building, someone has to take care of it. In a perfect world, every company would have a dedicated IT staff; if you’re Microsoft, Apple, Google or Facebook, that certainly is the case.
For much of the tech world, the choice to maintain a data center onsite often results in personnel being frequently dragged out of their role in DevOps and asked them to do something related to the data center, often a task they are not qualified for. This creates holes and inefficiency in the development process. That’s a bad thing.
Good Infrastructure Means Better Uptime
Guess what’s also bad for DevOps? The team’s least favorite scenario—besides an unexpected meteor strike—would be system crashes and downtime related to hardware issues. Downtime can be incredibly expensive for a multitude of reasons besides the obvious—employees sitting around twiddling their thumbs. There is also the scenario of actual lost income. As Amazon found out the hard way, the number can be astronomical.
With no internet connection, business payment systems, no matter how sophisticated, are likely to be non-functional, though some will queue payments and run them through when the system goes back online. This is why it’s a good idea for businesses to find ways to speed up transactions from clients and customers, such as automating invoices and customizable invoicing tools that allow multiple options for straightforward payments.
This is also another area where going Colo can be a huge help to development. Think about it. In return for scratching a monthly rent check, you get diversified and redundant power sources, the most reliable internet on the planet and enough bandwidth to strangle an elephant.
The bottom line is that moving equipment to a Colo space should be accompanied by a significant reduction in downtime. If it’s not, you need a different landlord. Downtime is the enemy of product/service development progress. A modern Colo arrangement means you don’t have to spend time running down to the local Dollar General in search of an extension cord when one flames out. Okay, you really shouldn’t be using extension cords from a dollar store in your data center, but you get the idea.
Secured Data and Premises
The only type of downtime worse than that caused by insufficient infrastructure is when data is breached either through hacker online trickery or physical penetration of the premises. The latter happens more than you might think. Bad guys decide to go old school and break into a Colo center in the middle of the night to physically drag servers out of the rack and spirit them away.
With the intense global focus on thwarting hackers, DevOps is already in the process of evolving into DevSecOps as we speak. The reason is simple. Development teams are realizing that a focus on security from the start reduces the chance that a project gets interrupted mid-development resulting in—you guessed it—downtime.
With a Colo provider, let them worry about security.
In a Word, Scalability
With the everyday lexicon expanding to include cloud storage possibilities, the idea of scalability is at least somewhat familiar to the masses and certainly to businesses.
Google actually started out in a colocation data center before their data collection grew to approximately the size of the Milky Way galaxy. For IT in general and DevOps in particular, scalability is the Holy Grail because no company expects to stay the same size forever.
A smart DevOps team, especially one that works in any of the growing number of SaaS business models, will put the bug in management’s ear about the benefits of Colo long before the need arises.
The worst time for an upwardly mobile small or medium-sized company to start thinking about making the switch is after you’ve already outgrown present resources. For example, when working with inadequate bandwidth, a peak time data spike can slow development as local computers struggle to find enough bandwidth to function.
Another dirty little secret is that Colo providers typically get better internet rates than a single company, so there’s that. With the cost of a spike spread across all tenants, the actual cost to you will be considerably less than trying to manage your own data.
Final Thoughts
The bottom line is that the service provided by a Colo center should reduce the strain on a company’s IT budget and resources. Common services provided to a tenant are support personnel, who monitor and troubleshoot systems and issues before they grow into a big, fat calamity. Money you don’t spend providing basic service to an onsite data center is money you can funnel toward doing your company’s main work.
There’s one more thing a Colo site provides that shouldn’t be undervalued, and that is peace of mind, because you have removed much of the stress, cost and personnel drain that accompanies running your own data center. Farm it out. Let someone run it who hires staff with the particular skills needed and has a vested interest in keeping you happy.
The concept of shift-left has been a popular trend in continuous testing practices for a while. We are now beginning to see shift-right practices as an emergent trend in testing.
Shift-right entails doing more testing in the immediate pre-release and post-release phases (i.e. testing in production) of the application lifecycle. These include practices such as: release validation, destructive/chaos testing, A/B and canary testing, CX-based testing (e.g. correlating user behavior with test requirements), crowd testing, production monitoring, extraction of test insights from production data, etc.
Shift-right not only introduces such new testing techniques, but also requires testers to acquire new skills, make aggressive use of production data to drive testing strategies and collaborate with new stakeholders, such as site reliability engineers (SRE) and operations engineers.
In shift-right of testing, and in the increased collaboration with operations disciplines, we see the evolution of a new discipline in DevOps that we call TestOps.
In this blog, we will discuss the various shift-right testing trends and practices, what’s driving them, the new skills and collaboration required so that TestOps benefits can be realized.
Emerging Trends in Shift-Right Testing
In a recent survey of continuous testing conducted jointly by Capgemini and Broadcom, testing in production was ranked as the number one practice (with 45% respondents) either currently implemented or planned (see Figure 1). In addition, 39% of the respondents mentioned the use of analytics from operations to determine or optimize test coverage.
Figure 1.
When asked how customers measure the effectiveness of continuous testing processes, production data and user feedback and adoption of new functionality were ranked number one and number two respectively (see Figure 2).
Figure 2.
Finally, in response to how customers test their digital applications to ensure that they meet real world requirements, almost all of the responses were related to production data (see Figure 3). Use of production data for designing test strategy and chaos testing emerged as new methods compared to similar survey done a year ago.
Figure 3.
The survey data clearly indicates that customers are actively practicing (or considering adoption of) shift-right testing techniques. This report has a chapter (with my contributions) dedicated to shift-right that discusses the techniques and recommendations for its adoption, as well as relevant customer case studies.
What’s Driving Shift-Right Testing Trends?
The goal of shift-right testing is to ensure correct behavior, performance and availability over the production of an application.
There are several drivers for adoption of shift-right testing trends.
Customer Experience (CX) Is a Key Quality Metric for Digital Applications
Unlike classic testing, this takes into account real-world users and their experiences. An application with perfect traditional quality scores (such as FURPS) may still suffer from poor CX if it fails to delight the customer. CX is measured using various metrics such as Customer Effort Score (CES), Net Promoter Score (NPS), Customer Satisfaction Score (CSAT), etc. While it is possible to shift-left some of these measurements to some degree, most CX measurements are deduced from systems in production (or close to production). Examples of CX-based testing include the following:
Requirements validation based on real user journeys, behavior and feedback.
Deriving functional and performance test scenarios based on the above.
A/B testing and canary testing, to experiment how well customers like (or dislike) changes.
Crowd testing, to better understand real-world user experience.
In a World of Agile Delivery, It may Not Be Possible to Test Everything Before Release to Production
Time to market (or lead time for changes) is a top business imperative for digital applications, which is why QA/testing is considered a major bottleneck in continuous delivery. While it is possible to optimize testing effort and cycle time using shift-left practices (such as model-based testing, change-impact testing, test process and execution automation etc.), those approaches may still take too long (or too much effort) during the shrinking release cycle time. In some cases, the exact usage patterns may not even be fully understood before release. The idea is to learn usage patterns from production and use it for better testing strategy within shift-left.
Complex and increasingly distributed systems using microservices (using thousands of components) and cloud-native techniques that allow releases at a very granular level make it more and more difficult to test fully in pre-production environments. This is complemented by the fact that since such releases are so granular, they may be backed out easily (rollback) in case they cause problems.
Hence, enterprises test to assure “good enough” quality (to ensure timely release) and rely on fast remediation (or rollback) to address defects or problems that arise.
Also, CX measurements (as described above) often provide real-user feedback in shades of gray, unlike the black-n-white pass/fail measurement of classic testing. Which means it is not possible to perfect pre-production test suites to ensure thorough CX coverage.
A Hundred Percent Reliability (or Quality) Is Often an Unrealistic Goal
Related to the above, one of the key principles we learn from the DevOps discipline of SRE is that 100% reliability is not only unrealistic, but often too expensive to accomplish. SRE establishes the concept of service level objectives (SLOs) and error budgets to quantify acceptable risk tolerance in production systems. The same principle applies to testing and overall quality. See my related blog for more on this subject
Some Validations Are Difficult to Run in Test Environments
These include large-scale performance tests when the test environments are not appropriately sized (or configured) relative to production environments.
Another example is destructive or chaos testing. While it is possible to run isolated chaos testing in test environments by using techniques such as service virtualization (to simulate failure in dependent components), it is difficult to execute for large-scale destructive tests. For example, Netflix does a significant amount of their testing in production.
Similarly, enterprise monitoring to gather insight about real-world usage patterns and failures is only possible in production. Though we advocate turning on monitoring in test environments (as part of shift-left), such monitoring only helps to do localized monitoring (e.g. of the specific app or system being tested), and that too under test conditions. However, performance test scenarios that have been developed by testers can be re-used in production (shift-right) as synthetic monitors to measure application performance and regression.
Lastly, as an extension of monitoring, testing of some AI-based applications may be difficult without access to production data. Some machine learning algorithms, for example, need to be constantly refined based on real world data. While these algorithms are developed using a limited set of training and testing data, they need to be monitored and tuned, based on its performance in production.
Developers and Testers Need Easier Closed-Loop Feedback from Production
Easier closed-loop feedback from operations allows developers and testers to better understand the behavior of the app, predict the success/failure of releases and respond to production incidents (e.g. reduce MTTR).
Interpretation and use of voluminous production data is often difficult and tedious for development and test teams. Use of shift-right techniques allows developers and testers to access production data more easily in a form that is easily consumable and actionable–by leveraging the right insights from AIOps disciplines. This data collaboration is a key tenet of the DevOps paradigm. For example, synthetic monitors (which are typically created by developers and testers in the dev/test stages) provide means for developers to get monitoring feedback in a form that they are familiar with.
In addition, automated root-cause analysis of defects/incidents in production can provide developers and testers insight on failure patterns and trends, their sources (when production data is correlated with development data) and what’s causing them, and how they may be better prevented.
Similarly, data usage scenarios (e.g. from database logs and RR pairs from service transaction logs) can be used for generation of relevant test data.
In essence, shift-right testing practices are necessary to create a continuous feedback loop from real user experience with a production application back to the development and testing process.
The Evolution of TestOps Discipline
It is clear from the customer survey data and drivers discussed above that there is an evolution toward shift-right practices.
Just like we did with shift-left testing, in order to make these practices sustainable and to realize the benefits, we need to also evolve tester use cases, skills, tools and collaboration patterns with people and disciplines to the right of the application lifecycle, i.e. operations. In other words, a new discipline needs to be established—we call this new discipline TestOps—a sub-discipline within the broader DevOps context.
Though the term TestOps (like all the X-Ops sub-disciplines within DevOps) implies the collaboration between testing and operations, it isn’t simply about shift-right. Why? Because with DevOps, operational disciplines themselves have shifted left (such as shift-left of monitoring, configuration management, SRE, etc.). Hence TestOps—true to the principle of continuous testing—refers to better collaboration with operations disciplines across the DevOps lifecycle.
Let’s consider some of the key practices in TestOps shift-left (see Figure 4).
Figure 4.
Early reliability testing:this includes static scalability analysis of the system architecture, as well as performance and stress testing.
Configuration quality and testing: this includes testing of “as-code” (anything “as-code” can and should be tested as such, just like application code) environment and deployment configurations, as well as deployment testing (to test the correctness of deployments) and rollback testing (to ensure that rollbacks can be done quickly and successfully).
Early monitoring: includes monitoring of systems early in the lifecycle (e.g. in development and test environments) as well as development and testing of synthetic monitors before they are deployed to production.
Early AIOps insights: this includes early use of AIOps techniques (in the dev/test environments) to glean testing and quality insights, for example for defect and release risk prediction.
Below are some of the key TestOps shift-right practices (see Figure 5).
Chaos engineering: this includes testing the reliability of the system through the injection of controlled failure scenarios.
A/B testing: consists of a randomized experiment with two (or more) variants to study which are more effective with users.
Canary releases: consists of techniques used to reduce the risk of introducing a new software version in production by gradually rolling out the change to a small subgroup of users, before rolling it out to the entire platform/infrastructure and making it available to everybody.
Synthetic monitoring: includes monitoring techniques using an emulation or scripted recordings of transactions (that reflect typical user behavior).
CX testing and analytics: includes extracting of CX data to understand customer experience as well as to derive insights from it (such as customer problems, new feature requirements, etc.).
Other testing in production: includes performance tests and testing of AI-based applications.
Testing insights from AIOps data: as described above, a rich variety of testing insights can be gleaned from operations data.
Figure 5.
Key TestOps Skills and Tools
Let’s consider the new skills required for the key TestOps practices. This includes the following:
Data analytics skills: Since most of the production data is both voluminous, diverse and often temporal, testers must now pick up data analytics so that they quickly glean insight from such data, as well as perform correlations with other datasets to identify actions. Advanced analytics skills (such as predictive analytics or machine learning) are also necessary to predict events (e.g. release quality). While these analytics skills are well understood within the operations community (with the increasing adoption of AIOps), these are relatively new for developers and testers.
CX skills: As we discussed, CX is now considered a key metric for measuring quality in production. New types of CX testing include A/B and canary tests as well as crowd testing. Like other operational data, CX data is also typically voluminous and often unstructured. While CX teams specialize in such principles, testers need to acquire sufficient skills to glean insight from the CX process and data, and collaborate effectively with such teams.
Monitoring and operations skills: Understanding of monitoring principles (e.g. creation, testing and deployment of monitors and use of data from monitors) is a key skill required for TestOps. Similarly, testers must also understand operations principles (incidents, failures, alarms, MTBF, MMTR, configuration definitions, etc.) so that such information can be leveraged for testing purposes as well as for collaboration with ops teams.
Reliability skills: Reliability is also another key operational quality attribute that testers need to be familiar with. This includes chaos/resiliency testing, deployment and rollback testing, and configuration testing. The discipline of SRE has evolved within DevOps to address reliability, and many of its principles overlap with TestOps.
New tools skills: In addition to using traditional testing tools, TestOps also requires the use of additional tools such as monitoring, data analytics (for production and CX data), CX testing (such as A/B and canary testing) and reliability testing. Technology innovations that support testers with these new capabilities include RunScope (a lightweight API monitoring tool that can be used for monitoring both in test and production environments), Blazemeter (for performance testing for CX and reliability), Optimizely for A/B testing. For data analytics, technologies such as automation.ai simplify the work of data aggregation and modeling so that testers can more easily extract actionable insights from data collected across the lifecycle.
Key TestOps Collaborations
In addition to new skills, testers must now learn to collaborate better with additional stakeholders as alluded to above. These include:
Collaboration with data science and AI/ML engineers: This is necessary for testers to tweak or build out new algorithms and models required to perform the analytics needed for TestOps.
Collaboration with CX engineers and teams: This is necessary for testers to monitor the appropriate CX metrics and data, and to study the impact of their testing/QA activities on CX.
Collaboration with Operations engineers and teams: This is fundamental for TestOps to not only support all the shift-right testing activities, but also to gain access to production data that is required for TestOps analytics. This is analogous to how testers collaborate with developers as part of shift-left.
Collaboration with SREs: As mentioned, there is a fair degree of overlap between SRE and TestOps principles, and hence collaboration with SREs is key for testers. See my blog on how testers and SREs can work better together to improve reliability.
Conclusion
Hopefully, this article has provided readers with some insight into the emerging discipline of TestOps. As we build increasingly distributed microservices and cloud-native systems, and as the discipline of SRE matures, we expect to see more of shift-right testing practices that enrich TestOps.
