Tag: blue-green deployment

  • Blue-Green Deployment: What Are the Options?

    Blue-Green Deployment: What Are the Options?

    Blue-green deployment is a change management strategy for software releases. Blue-green deployments require two identically configured hardware environments. One environment is active and serves end users while the other remains idle.

    Blue-green deployments are typically used for applications with strict uptime requirements. First, new code is released to staging environments and fully tested. Once the code is reviewed, the team adjusts the router configuration to redirect application traffic to the staging environment. This staging environment then becomes the production environment.

    If problems are found after the switch, the team switches traffic back to the old production environment and continues running the original version. Once the new code is validated in production, the team can update the code in the idle production environment, and use it to test the next release. For more background, see this detailed blog about the blue-green deployment pattern.

    Blue-Green Deployment Benefits

    Here are some of the main advantages of blue-green deployments.

    Standby System
    A blue-green deployment always ensures a server is on standby, providing a crucial backup when the system fails. In addition to facilitating software updates, this approach improves risk management, allowing you to switch to the backup server during a malware breach.

    Redirecting traffic to the other system version is easy because it has the same infrastructure. The system can stay live and functional while the teams fix issues.

    Smooth, Fast Releases
    A blue-green deployment allows product owners to push software to production quickly. It offers the flexibility to release an update at any time, with no downtime required. End-users don’t feel the impact of each release. DevOps teams can also benefit from smooth releases because they don’t have to race to meet deadlines for scheduled outages.

    Simple Rollback Process
    Rolling back an update is also fast in a blue-green deployment. Simple rollbacks reduce the risk of experimenting in a production environment. You can implement rollbacks by setting the application to read-only or using a load balancer to handle active transactions during the cutover. This can be significant for Kubernetes security because if security vulnerabilities are discovered in the cluster, you can easily roll back to a previous safe version.

    No Downtime
    A blue-green deployment allows you to avoid downtime, whether scheduled or unexpected. You can shift users to the other model during maintenance checks and fixes. Upgrades can occur anytime because there is no need to wait for times with lower traffic (i.e., midnight).

    Testing in Production
    Testing in production allows you to check a live product’s functionality, showing how it performs in the real world. The drawback of this approach is that it can expose end-users to bugs during testing. A blue-green deployment lets you test the product in an inactive production environment, minimizing the risk of bugs making it to the live environment.

    Blue-Green Deployment in Kubernetes

    Many development teams are migrating to Kubernetes as an environment for development, testing and production deployments. Kubernetes makes it easy to deploy applications and can also be used to automate blue-green deployments.

    However, Kubernetes does not provide out-of-the-box blue-green deployments. It has a Deployment object that allows “rolling updates”—this allows you to update your applications without downtime by incrementally replacing pods with the latest version of an application.

    This is similar to a blue-green deployment but doesn’t offer all the benefits. Rolling deployments can make it difficult to roll back if there are issues with the new version. You might require Kubernetes health checks to determine the status of pods within a deployment. Applications might take time to roll out, while the pure blue-green pattern allows you to switch over to a new version instantly.

    Blue-Green Deployment in Kubernetes: What are the Options?

    Argo Rollouts
    Argo Rollouts is a set of Kubernetes controllers and CRDs that provide advanced deployment features for Kubernetes such as blue-green and canary deployments, experiments, and other progressive delivery capabilities.

    Argo Rollouts optionally integrates with Kubernetes ingress controller implementations and service meshes, leveraging traffic shaping to incrementally switch traffic to new versions during updates. Rollouts can also query and interpret metrics from other systems to validate key KPIs, and automatically promote or roll back releases.

    Flagger
    Flagger is a progressive delivery tool that automates the release process of applications running on Kubernetes. It reduces the risk of introducing a new software version into production by gradually migrating traffic to the new version while measuring metrics and performing conformance testing.

    Flagger implements multiple deployment strategies (canary releases, A/B testing, and blue-green deployments) and integrates with various Kubernetes ingress controllers, service meshes, and monitoring solutions.

    GitLab CI/CD
    GitLab is a collaborative DevOps platform and open-source code repository for large software development projects.

    In addition to a code repository, GitLab also provides CI/CD automation and issue tracking. GitLab repositories allow the hosting of different software versions and entire development pipelines, allowing users to roll back software and environments in case of unexpected problems.

    GitLab supports both manually triggered deployments and scheduled deployments to Kubernetes production systems, using progressive deployment (including blue-green deployment).

    CircleCI
    CircleCI is a CI/CD platform that helps engineering teams build, test, and deploy software, while viewing code changes in real-time in a dashboard UI.

    CircleCI can be deployed to virtually any target and can be easily configured to integrate with other services, for deployment strategies such as feature flags, blue-green and canary deployments. It supports both fully automated deployment processes or a process with manual approvals, with the ability to easily customize deployment configuration.

    Conclusion

    In this article, I explained the blue-green deployment pattern, its benefits, and how it can be beneficial for Kubernetes deployments. I introduced four tools that can help you implement blue-green deployments in your Kubernetes clusters:

    ● Argo Rollouts – Part of the open source Argo project. Includes Kubernetes controllers and CRDs that provide advanced progressive deployment features.
    ● Flagger – An open source progressive deployment tool for K8s that supports canary releases, A/B testing, blue-green, and more.
    ● GitLab CI/CD – Supports both manually triggered deployments and scheduled deployments to Kubernetes production systems.
    ● CircleCI – Supports deployment to Kubernetes clusters, with deployment strategies such as feature flags, blue-green and canary deployments.

    I hope this will be useful as you learn to implement progressive deployment patterns in your Kubernetes environment.

  • 5 Testing Strategies For Deploying Microservices

    5 Testing Strategies For Deploying Microservices

    With rigorous development and pre-production testing, your microservices will perform as they should. However, microservices need to be continuously tested against actual end-user activity to adapt the application to changing preferences and requests. This article will cover five deployment strategies that will help developers and DevOps teams when releasing new features or making changes to microservices-based applications.

    Always Test in Production

    A staging environment is a necessary but insufficient step toward building a robust microservices infrastructure. Although indicative, it is not a substitute for evolving user traffic and behavior. When real users interact with the software during production, the services may not perform as efficiently.

    Users are unpredictable. They may make requests you have not accounted for when drawing up communication paths. Or your microservices architecture may not sustain the actual production traffic. When testing with real end-users, you are sure of your software’s performance and can improve it or add new functionality. 

    The disadvantage of working with real users is that they will be impacted by any errors. However, with specific deployment strategies, you can identify the issues and minimize their impact on those users. 

    Depending on your goals, you can implement the following testing strategies in the deployment stage. They can be performed individually or in various combinations, as each approach varies in its contribution to deployment.

    Blue-Green Deployment

    A blue-green deployment uses two distinct yet identical production environments. For example, you deploy an inactive version of your application in the blue environment. At the same time, the latest version available to end users runs in the green environment. Next, you push code to add new functionality to the blue environment. 

    At this stage, you test the services in the blue environment to ensure that they are running smoothly. You can automate this process by using basic smoke tests. When the microservices pass these tests, you can instruct the load balancer to gradually redirect user traffic to this new environment.

    If the new functionality poses no problems, the router moves all traffic from the green to blue environment, but can immediately revert traffic in case of any errors. 

    Due to the identical nature of both environments and their reliance on each other, there is little to no downtime. The end user’s experience is largely unaffected and the developers can move quickly to fix any bugs.

    The old environment serves as a backup, making it possible to execute a rollback without any downtime. After releasing the latest version, you can decide to kill the old instances, but only after you are confident that the new environment will continue running without errors. You can also clone both blue and green environments to repeat the process with other feature deployments in the future.

    Canary Deployment

    Canary deployment is similar to blue-green testing in that it protects against release-related risks.

    The phrase “canary deployment” originates from an old mining practice of placing caged birds in mine shafts to detect the presence of harmful gases. If present, the gases would kill the canary before affecting the miners, thus providing an early warning to vacate the mine immediately.

    A canary deployment works on a similar principle. Instead of creating two separate environments, a canary deployment operates within the same microservice or infrastructure. The developers roll out a new service or application version with changes only to a fraction of the end-users.

    You can quickly detect errors or vulnerabilities while using the new service. The impact is temporary and minimal as the experiment is only run on a subset of users with most remaining unaffected. Once they are working and have passed verification, you can scale up the changes. 

    The biggest challenge for those running canary deployments is the requirement to run multiple versions of the same microservices so that they can be deployed to a few users at a time. Implementing multiple versions also means you need to keep track of which users are on which versions to run accurate business metrics and analytics. Such monitoring can get increasingly complex with multiple canary deployments within the same application.

    Feature Flags

    A feature flag or feature toggle is a change or feature written inside conditional code. Developers can turn the feature on or off depending on their testing requirements while the application is running and in use.

    The code is already deployed and there are two code paths: One with the code that implements the feature and one without it. The developer only needs to choose one of the two paths. When the switch is toggled on, the code chunk is executed as part of the flow. When switched off, that code is skipped and the feature is not implemented. The rest of the source code continues to run as usual, independent of the feature flag’s condition, with no disturbances in the end user’s experience.

    Once integrated, a feature flag allows you to turn on a feature for a select group of users. Unlike a canary deployment where the selection is random, feature toggles are usually employed in specific cases. For example, developers do not create two separate applications to implement free and paid subscription tiers. Instead, operating a feature flag allows you to make a particular feature accessible only to subscription users.

    Feature flags are especially attractive during testing as developers can use them to run small experiments throughout the application without relinquishing control. Being able to flip the switch remotely also makes rollbacks easier. The risk is negligible as the application is self-sufficient in that it can continue to run without the feature. 

    Feature flags generate complex user-to-service paths by way of permutations and combinations. For a specific user, it is difficult to determine the exact route they took when using the services. Feature flags complicate this by allowing each user to have a different experience, ultimately making the application hard to debug. These toggles are easy to get started with minimal risks, but their maintenance can quickly become complex. It is best to use them only when needed and with other deployment strategies.

    Traffic Shadowing

    With traffic shadowing or mirroring, the router duplicates incoming traffic to an already-released service and gives the copy to another service. The request and response mechanism between the user and the existing service remains intact. On the other hand, the second service with a copy of the traffic contains new features that require testing. Consequently, it does not interfere with the existing process. Instead, the copy is used to test its functionality.

    Its most significant benefit is that it allows the new version to receive the same traffic that is currently being received by the service it seeks to replace. There is no need to create test data or worry about replicating scale. This accuracy comes with little risk, as there is no tangible impact on the existing services. Developers can run all relevant tests on this production environment, such as testing for errors and performance metrics. The new version’s responses, which are not sent to users, can also be compared to those of the production service. Both versions operate independently of each other with different end goals. This can happen in real-time or a copy can be saved and replayed for future testing.

    Traffic shadowing can be used with other deployment techniques like blue-green or canary deployments. After successfully shadowing a production environment, the changes can be rolled out gradually using a canary deployment to gain maximum confidence before a full release.

    Shadowing can have unintended consequences, though, so you should exercise caution when deploying this strategy with services that have third-party dependencies. 

    A/B Testing

    Unlike blue-green and canary deployments, A/B testing is focused on user perception and experience of new features. It measures if and how end users are interacting with these features, whether they are easy to notice and use and gauges the application’s overall functionality. It provides developer teams with business-level insight to improve their application as the features are ultimately implemented using code.

    A/B testing divides users into groups that access different features. Group A, for example, sees a different user interface than group B, although members in both groups are making the same requests to access the application. Traffic is routed to separate builds or different configurations on a common build. This is done by considering aspects like the users’ operating systems and user agents. It must be run on a sample that is representative of your end users. The test also needs to display statistically significant results to be valid.

    This test can be combined with blue-green or canary deployments as they handle the actual feature deployments that this strategy tests. After comparing the versions shown to the groups, the one that has performed better can be pushed to release for all users. 

    Building your Deployment Strategy

    Each strategy outlined can be used independently or together in a combination that best suits your goals, workflow and microservices requirements. They allow you to identify and reduce the impact of any vulnerabilities that may only surface at the final stage of your software’s release. Implementing them can be a somewhat complex process, especially with larger architectures and dependencies. But by implementing any of these strategies, your team will release the best possible version of your application. 

  • 5 Steps to Succeed With Blue-Green Deployment

    5 Steps to Succeed With Blue-Green Deployment

    By allowing teams to maintain two production-ready environments at the same time, the blue-green deployment technique can significantly boost reliability. But blue-green deployment can also be difficult to execute and manage. Let’s unpack how blue-green deployment works, why it’s important and which best practices to follow for blue-green deployment success.

    Blue-Green Deployment, Defined

    Blue-green deployment is a software deployment technique in which teams maintain two identical but distinct environments that are capable of hosting a production application. One environment is “live” and actively serving users, while the other is kept in reserve.

    When it comes time to deploy a new version of your application, you deploy it to the reserve environment. You then test and validate it to make sure it meets performance and reliability requirements.

    If it does, you reconfigure your DNS or load balancers so that user traffic is directed to this environment, which becomes the new production environment. Meanwhile, the previous production environment is placed in reserve.

    Benefits of Blue-Green Deployment

    The blue-green deployment technique can boost reliability in several ways:

    • Environment parity: Because the reserve environment in which you test your application becomes the live production environment, blue-green deployment ensures parity between testing and production. In turn, it reduces the risk that configuration differences could cause unforeseen problems in production.
    • Failed deployments: If a deployment fails, it doesn’t impact your users because you deploy into the reserve environment. You can fix the issue while a stable user-facing version of your app continues to run in the other environment. 
    • Redundant environments: Although blue-green deployment is not a data protection or disaster recovery technique, having two production-ready environments available is useful in the event that one of your environments is wiped out by an event like a data center failure. In that case, you can restore operations quickly by using the other environment (provided it’s hosted in a separate data center, of course).

    How to Use Blue-Green Deployment: Steps and Best Practices

    While blue-green deployment is a great way to increase reliability, it does present some challenges. The following best practices can help you to handle them.

    Use Scalable Infrastructure

    One obvious potential downside of blue-green deployment is that it can essentially double the amount of resources you need to host your application since you have to maintain two identical environments. That can double your hosting bill.

    One way to reduce the cost impact of blue-green deployment is to create environments that can scale rapidly without changing their core configuration. You could do this using cloud-based virtual machines (VMs) and autoscaling or via containers and an orchestration engine (like Kubernetes) that can scale them.

    Using this approach, you can set up two environments that are effectively identical except that the production environment includes more VM or container instances than the reserve. Although you could argue that this is not quite a blue-green deployment because the environments are not perfectly identical, that semantic downside is outweighed by the benefit of not wasting a bunch of money on infrastructure you don’t need.

    Chaos Engineering

    Typically, chaos engineering entails experimenting with a production system to test its reliability. Chaos engineering offers many benefits, but it presents the obvious risk of creating a problem in production that impacts your end users.

    Blue-green deployment allows you to work around this risk by practicing chaos engineering in the reserve environment. Of course, because you won’t have real users, performing chaos engineering under these settings may not allow you to test every aspect of your production application. But you can still test many of the configuration variables by issuing requests yourself within the reserve environment—all without worrying about breaking things for actual users.

    Managing Database State

    Keeping databases in sync when you switch between environments is one of the biggest challenges of blue-green deployment. Even small discrepancies in database state could cause big reliability issues.

    The simplest way to address this issue is to turn off the database before switching and perform tests to ensure that databases in both environments are identical before they are reenabled. Of course, this approach may impact application functionality, because the app may not work properly if the database is turned off.

    Another solution is to sync databases between your two environments. This should ensure that database state remains identical even as you switch from one production environment to the other. But it also means that problems with the database will affect both environments.

    The bottom line: There is no simple way of managing database state in blue-green deployment. You should evaluate the different options available and consider which one best meets your needs. If you can tolerate a little downtime, stop the database and restart it during the switch. If you need total continuity, use syncing, but be sure to manage the associated risks.

    Change Load Balancers, not DNS

    There are two main ways to manage where traffic is directed, and hence which is the production environment: You can change your DNS configuration when you want to switch. That’s easy to do—you would simply change the mapping for yourapp.com from one IP address to another, depending on which environment is the live one.

    You can also use load balancers to manage traffic. Load balancer configuration is more complex than simply changing DNS entries, but it also enables more sophisticated and granular control.

    In general, it’s a best practice to manage blue-green deployment switches via load balancers, due to the extra control they provide.

    Prepare for the Unexpected

    While blue-green deployment can reduce the risk of user-impacting disruptions, it’s no silver bullet. You must also be sure that you have the reliability management tools you’ll need to detect and remediate issues within both of your environments.

    Blue-green deployment is a powerful reliability engineering technique. But to get the most out of it, you must optimize the way you handle requirements like infrastructure management, traffic routing and database state.