devopsy.kinsta.cloud

Tag: immutable infrastructure

  • Immutable Infrastructure: The Next Step for DevOps

    Immutable Infrastructure: The Next Step for DevOps

    Immutable infrastructure, a concept that is rapidly gaining traction within the dynamic field of DevOps, offers a transformative approach to organizational security for companies of all sizes. 

    Instead of the traditional method where existing components are continuously updated and modified, immutable infrastructure advocates for an entirely different philosophy: Components are replaced wholesale with new versions and never altered after deployment. This type of paradigm shift promises a plethora of advantages, chief among them being enhanced consistency and reliability, which is a potential game-changer for DevOps practices.

    This article offers a thorough exploration of immutable infrastructure, taking a look at its core principles and examining why it’s becoming a focal point in the DevOps community. To help further bring these concepts to life, we’ll thoroughly examine the tools and techniques that are pivotal in its implementation.

    Understanding Immutable Infrastructure

    Immutable infrastructure is rapidly gaining traction in the world of cloud services and DevOps for its innovative approach to managing servers and components. Once deployed, these elements are not modified or updated; instead, new servers are created from a common image whenever changes are necessary. 

    This type of approach simplifies rollback procedures and minimizes troubleshooting, as the integrity of servers is maintained. Furthermore, this methodology, facilitated by OS-level virtualization, marks a significant departure from more traditional practices.

    And with the AI market already worth $313 billion, integrations of models and solutions will only become more popular. Integrating AI into immutable infrastructure can significantly enhance its efficiency and reliability. 

    In such systems, AI can be used to monitor and analyze system performance continuously. By applying machine learning algorithms, it can predict potential issues and automate the process of deploying new server instances before problems arise. 

    Comparing Immutable and Mutable Infrastructures

    The Immutable Approach

    Immutable infrastructure offers streamlined change tracking and rollback, with each new server or VM being easily identifiable. Its uniform configuration across servers simplifies testing and the unaltered state post-deployment reduces complexity. 

    This makes it ideal for environments with numerous interdependent elements, such as in DevOps and cloud computing, virtually eliminating configuration drift.

    Challenges With Mutable Infrastructure

    In contrast, mutable infrastructure necessitates more time for server provisioning due to manual configurations. It’s prone to various update failures and connectivity issues, with each server’s unique configuration complicating diagnostics. 

    This traditional approach struggles with unpredictability, making it less suitable for modern DevOps practices and resulting in frequent configuration drifts.

    Advantages of Immutable Infrastructure

    Opting for immutable infrastructure brings numerous benefits. Support calls are significantly reduced and security is enhanced, thanks in part to the use of machine learning to identify anomalies. 

    The implementation of safelisting rules strengthens security controls. Moreover, this approach mitigates configuration drift and negates the need for synchronization during updates, leaning on cloud platforms for swift VM management. It also ensures rapid recovery due to easy rollbacks and encourages experimentation, thus fostering innovation.

    Best Practices for Immutable Servers

    Adopting immutable servers involves several best practices all organizations would be wise to apply. Building images in a chroot environment helps isolate changes while keeping test suites lean and efficient while avoiding overburdening the change process as a whole. 

    Caching installation files near the build location optimizes update speeds, layering images on top of a base server image expedites updates, and minimizing the OS image reduces boot times and security risks. Additionally, using containers can dramatically speed up and simplify the process.

    Essential Tools for Immutable Infrastructure

    When it comes to immutable infrastructure, certain key tools have emerged as indispensable for efficiently managing and maintaining this modern approach. These tools help create, manage and deploy server images and infrastructure-as-code throughout organizations. 

    Here’s a closer look at some of these essential tools:

    Packer

    Packer is a crucial tool for creating identical machine images for various platforms from a single source configuration. This tool is instrumental in the immutable infrastructure because it allows for the creation of fixed server images that can be quickly deployed, ensuring consistency across environments.

    Terraform

    Terraform by HashiCorp is another key player, enabling users to define and provide data center infrastructure using a declarative configuration language. It helps maintain the infrastructure-as-code, which is an essential aspect of immutable infrastructure. 

    Terraform allows for safe and predictable changes to infrastructure, with efficient management of both cloud and on-premises resources.

    Docker

    Nowadays, Docker is synonymous with containerization, allowing applications to be packaged into containers—consistent, isolated environments that encapsulate all the necessary components of immutable infrastructure. 

    Docker helps to simplify the deployment process and ensures that applications run the same way everywhere, aligning perfectly with the core principles of immutable infrastructure.

    Kubernetes

    While not directly mentioned earlier, Kubernetes deserves recognition for being a powerful orchestration tool for containers, including those that are managed by Docker. Kubernetes excels in automating deployment, scaling and management of containerized applications, which aligns well with immutable infrastructure principles.

    SaltStack

    SaltStack offers configuration management and orchestration tools, standing out for its ability to manage and automate more complex and scalable infrastructures, making it suitable for environments where immutable infrastructure is implemented. SaltStack ensures that server configurations are enforced and maintained as defined, preventing drift.

    Jenkins

    An open source automation server, Jenkins plays a significant role in the continuous integration and continuous deployment, or CI/CD, pipeline. Jenkins helps to automate the various stages of the delivery pipeline, which is crucial in testing and deploying immutable infrastructures.

    Ansible

    While Ansible is generally associated with configuration management, it also excels in automated deployment. Because it doesn’t require agents on the servers it’s managing and uses a simple, human-readable YAML syntax, it’s useful for setting up and maintaining immutable servers.

    Git

    Git, a version control system, is essential for tracking changes in the configuration code, scripts and templates used to create and manage immutable infrastructure. It allows teams to manage their infrastructure-as-code, ensuring a detailed audit trail of what was changed, when and by whom.

    Ultimately, a slew of tools and platforms are available to make the transition to immutable infrastructure as seamless as possible. From container orchestration with Kubernetes to configuration management through Ansible, you have a robust array of options at your disposal. 

    Yet, when it comes to handling specification files or any form of documentation, you want an added layer of security. A robust custom document viewer with extra security features, for instance, could be an invaluable asset in this regard, offering the ideal blend of accessibility and protection while ensuring that your infrastructure is immutable and secure from prying eyes.

    Choosing the Right Tools For Your Organization

    When integrating immutable infrastructure into your operations, selecting the right mix of tools that align with your specific requirements and existing workflows is crucial. Essential factors such as compatibility, scalability and ease of use should be considered to create a robust, efficient and effective immutable environment. 

    Each of the tools highlighted above has its own strengths and is a part of a broader ecosystem designed to support the reliable, consistent and secure deployment of applications and services across organizations of all sizes. Companies can fully embrace the immutable infrastructure model by learning how to properly harness these tools, improving stability, scalability and efficiency across all of their IT operations.

  • Speaking the Language of Applications

    Speaking the Language of Applications

    I arrived early to a meeting, and the pleasant security guard at AT&T suggested I take a look at their innovation museum while I wait. Among motion picture sound technology and life-size photos of switchboard operators on roller-skates stood a transistor prototype. The transistor was invented in Bell labs in 1947, and it changed the world. Steve Wozniak considered it the greatest innovation of all time in his 2014 article.

    But almost nobody needs to think about transistors anymore. The transistor had to become transparent and abstracted, so the next generations can build great things in higher level. The transistor made a huge influence on our lives as an enabler, but if people had to keep talking about transistors it wouldn’t be possible.

    Abstracting technologies is key to advancing to a higher level, making it possible to solve real business problems.

    The language of a client-focused business is applications. If we want to make a business execute faster these days, we need to make the development and release of its applications execute faster. But one thing stands in our way: the infrastructure. Applications reside on infrastructure–they are installed on servers, bundled in containers, on cloud instances, in networks, using storage.

    If we want to speak the language of applications, we need to abstract them from the infrastructure, and make the infrastructure transparent.

    The concept of abstracting applications is what stands at the basis of every new approach to application development and release around us.

    • Containers are all about abstracting services from specific infrastructure and making them mobile.
    • Serverless takes it to the extreme.
    • Public cloud makes it easier not to worry about owning physical infrastructure.

    Technology screams, “Think applications, soon you will not have to worry about infrastructure!”

    And yet, most of us mix applications and infrastructure in an unhealthy way when we try to automate them. It is unhealthy because mixing applications and infrastructure prevents reuse of automation assets and makes it very difficult to connect infrastructure usage to a business context.

    Understanding how mixing applications and infrastructure is manifested requires looking back into the evolution of infrastructure and application automation.

    The Evolution of Infrastructure: Manual Deployment and the Inception of Its Automation

    In the beginning there was manual. We got a server; it was sitting on a shelf. We got the application artifacts, maybe on a CD or a USB stick. We installed the application on the server. Our application was ready for use. When we needed to upgrade, we went back to the shelf and installed the update on our server. It was very hard to make this operation faster.

    This is where automation of applications and infrastructure started.

     

    Behind any automation tool or standard stands a model. This model reflects how its creators originally perceived the problem they were trying to solve. It consists of the entities they defined as first-class citizens, the operations they wanted to apply on these entities, their attributes.

    “All models are wrong, some are useful,” said George E. P. Box in 1976. This is true for automation models too. Automation models are usually nice and clean when we start off. But as the problems we try to solve transform, our models need to grow and adapt. Until at some point they are not useful anymore and we need to replace them with next generation modelling approaches.

    The Configuration Management Model

    Configuration management tools such as Puppet and Chef were originally built to manage application deployment and configuration on static infrastructure. This new approach to Ops optimized manually updating applications on servers. Better network connectivity made it possible for this model to thrive, with the ability to configure applications on servers remotely.

    But within a decade we witnessed the fast rise of immutable infrastructure. Immutable infrastructure is an approach to deploying applications where the infrastructure components are replaced instead of updated. This approach somewhat pulled the rug from under the configuration management model. With each update or change, the application must be re-deployed. This forced adding dynamic infrastructure provisioning to the configuration management model. Combining infrastructure automation and immutable infrastructure concepts into the configuration management model is a difficult task, since this model was never designed to accommodate something like that.

    The Infrastructure-as-Code Model

    Pure Infrastructure-as-Code tools were built to provision infrastructure dynamically. Terraform is an example for an open source tool that does this well. This model is created with infrastructure in mind.

    Whether we look into Terraform, AWS, CloudFormation or ARM templates, we will see the infrastructure is the starting point (defining network connectivity, VM instances and services) and the applications are an implicit addition. They are artifacts that happen to be deployed on and are tightly coupled to the infrastructure. We may look at a component that says “App A” but it actually stands for “specific infrastructure with app A installed on it.”

    This seems natural, because it’s the same way it worked in the good old physical-infrastructure world: We fetch a box, we install something on it. We focus on the infrastructure, because this is our starting point. The application is the artifacts we put on the infrastructure and scripts we run after we provision it. Presumably, the application cannot live without something to be on.

    Infrastructure-as-Code tools evolved the way open tools do–their base model was extended by the community because there was a need to automate applications deployment and not just infrastructure. But it is becoming difficult to extend this model to reflect a world where applications are built to move between clustered servers, and the starting point is no longer the infrastructure, but rather the application’s requirements.

    Mixing Infrastructure and Applications

    Trying to fit infrastructure provisioning and application deployment in a single model that was originally designed for just one of them creates issues. Infrastructure is lower level than applications. Applications reside on the infrastructure. The same app can reside on different infrastructure configurations. This critical relationship is completely flattened if you mix them.

    Moreover, from a business perspective it is important to make applications explicit. It’s our product. Our website. Our mobile app. The applications need to be the center of the attention and discussion. This kind of modelling puts applications in the shade of infrastructure.

    Four Great Reasons to Model Applications as a First-Class Citizen Right Now

    The models we use today are broken, and it is time to evolve. No matter if your applications reside in containers, or on cloud instances, or include native cloud services or a mix of them. Applications should be a first-class citizen, and not some artifact installed as an afterthought on the infrastructure we started from.

    The notion that applications should be abstracted from infrastructure doesn’t eliminate all the challenges of infrastructure provisioning and application deployment, but it creates a foundation for better and more scalable automation.

    1. Achieve Application Re-Use

    Modelling applications as a first-class citizen is the only way to make applications a re-usable element throughout the development and release pipeline.

    If there’s one thing I learned in my automation journey, it is that if automation doesn’t get frequently and consistently reused, you don’t get value for your money.

     2. The Path to Multi-Cloud

    Abstracting application is also the best approach for moving application workloads between cloud providers. If everything starts from infrastructure, we need to start from scratch when we move. If our applications are abstracted and defined separately—this definition stays the same when we move, only the infrastructure changes.

    3. The Path to Breaking Silos

    The vision of Dev, Sec, and Ops all dealing with the same language (code) is wonderful. But bringing everyone to the same code pool is not always easy. Applications and infrastructure are not the same thing, even though we can code both. Mixing them makes them unnecessarily hard to build and troubleshoot. It forces us to deal with fuzzy boundaries when it comes to responsibility for this code. Basing automation on a model that organizes the world in a way the different stakeholders can understand it, can help overcome this barrier.

    4. Connect Infrastructure to Business Needs

    Infrastructure-as-Code is a huge breakthrough, but the bigger story is about making application development and release faster. This doesn’t undermine the importance of infrastructure; infrastructure is the foundation of everything. But if we want to solve business problems, we need to talk the language of the business, and that’s the language of applications. Our automation should be built in a way that reflects this.

    — Maya Ber Lerner

  • Securing Immutable Servers in a Serverless World

    Securing Immutable Servers in a Serverless World

    Snowflakes are beautiful, unique creations. But, let’s keep them in nature. They don’t belong in our server infrastructure. Snowflake servers, where every configuration is just a little different, can introduce unnecessary security vulnerabilities and complications. While common in IT infrastructure, in the DevOps realm, they are gradually becoming ancient history.

    At the recent All Day DevOps conference, Erlend Oftedal (@webtonull), with Blank and head of the OWASP Norway chapter, discussed the benefits of immutable infrastructure practices within serverless architectures. Erland walked us through the positive effects on security as well as offered insight on potential problems.

    Erlend outlined the progress from physical servers to immutable servers, discussing the benefits of each step and the concerns that lead to the next phase, ending with immutable servers—that is, they can’t be changed. Once set up, they are set. Automated processes coupled with immutability principles can eliminate snowflake servers.

    Immutability isn’t a new concept. Some classic examples include: live Linux CDs, internet cafes, schools, libraries, hotel business centers, etc. To set up immutable servers, Erlend outlines these steps:

    1. Find a suitable base image
    2. Build a specific image for the application (dependencies, application, secrets)
    3. Deploy image to server(s)
    4. Repeat steps 2-3 for new versions.

    Of course, this immediately raises questions.

    What about data? He notes that, because data inherently changes, data has to be stored externally.

    How do you deploy changes? Well, you don’t change the server. You take one out and put a new one in. If that one fails, roll back to the previous one.

    How do you manage secrets? You can build secrets in, leverage cloud key management or use third-party services.

    How do we apply security patches? If truly immutable, you deploy a new server with the patch. However, you can have a semi-immutable server that allows for automatically installing patches but still doesn’t allow for logins or manual changes.

    Speaking of security, what are the security advantages? For starters, if an attacker is in your server, it will be thrown out when you change servers, although it may come back if the vulnerability isn’t patched. It also allows for very specialized images by removing all unneeded packages. With these specialized images, you can more easily audit and monitor the system for unexpected file changes, logins and connections.

    Erlend discussed containers because they are a step closer to a serverless world. Containers isolate processes and all run on the same OS. Of course, you can make them immutable, but, whether they are or not, there are some key security best practices you’ll want to consider:

    • Don’t run as root inside container
    • Implement user namespaces so the root account within the container is separated from the root account outside the container
    • Utilize the least privileges necessary for capabilities and resources
    • Keep them up to date
    • Automate scanning using tools such as Sonatype Nexus, Twistlock, Clair or Aqua.
    • Separate untrusted and trusted containers

    Erlend also dove into serverless architecture. While there is debate over what exactly qualifies as serverless, there are five principles that Erlend highlighted:

    • Use a compute service to execute code on demand
    • Write single-purpose, stateless functions
    • Design push-based, event-driven pipelines
    • Create thicker, more powerful front ends
    • Embrace third-party services

    Migrating to immutable, serverless infrastructure is a topic to be discussed well beyond this blog post and even Erlend’s session. Whatever you do in the future, though, take his parting words to heart: “The cloud can solve many of your problems, but in the end you cannot transfer your responsibility for security.”

    Erlend’s session was packed with information. If this piqued your interest, you should watch his complete session in his full All Day DevOps conference session (just 30 minutes). The other 56 presentations from the All Day DevOps Conference are also available online, free of charge here.

    — Derek E. Weeks

  • DOES 2015 – Josh Corman, John Willis on Immutable Awesomeness

    DOES 2015 – Josh Corman, John Willis on Immutable Awesomeness

    Continuing recapping my favorite sessions of DevOps Enterprise Summit 2015, here is Josh Corman and John Willis talking about Immutable Awesomeness. What is Immutable Awesomeness you ask? Have a listen to my interview with John and Josh and then listen into the actual session where they go over it.

    Just a couple of hints: 1. Software supply chains are a great way to look at your development/deployment process 2.In a world where software controls everything, it really is a case of life and death 3. If you can’t get excited about what this all means, listen to the session again.

    Of course what makes this interview and session great is the unbridled enthusiasm that both John and Josh bring to the subject. In fact there was a palpable buzz in the theater when they walked off the stage and over to my video studio. The same enthusiasm carried over to our interview too. It was a lot of fun to interview these two.

    So have a listen and enjoy. Also if you enjoy John Willis presenting, he will be headlining the DevOps Connect: cdSummit SoCal on December 3rd, joining some other great speakers.

    Here is John and Josh’s actual presentation:

  • Are we really back to “just reboot it”?

    Are we really back to “just reboot it”?

    The server infrastructure management style of pets vs cattle was originally coined by Bill Baker of Microsoft though Randy Bias of CloudScaling further popularized it in 2012 and revised and clarified in this presentation. Essentially the original “pets” style of server management has mainly hand-built, named, and configured servers that are heavily cared for as much time has gone into their setup while the newer cloud style of “cattle” management has servers that are often numbered and created and scaled via automation and simply removed from service in the case of any issue. Colloquially the removal is termed “shoot the offending node in the head (STONIH)”.

    In the past due to poor practices or OS service instability (NT4 comes to mind), sysadmins and developers often did not bother to figure out the root cause of issues and simply made reboots part of standard operation procedure. In the times before 24/7 global customer bases and always on broadband and smartphones it may not have even been a big deal.

    On the surface cavalierly destroying and refreshing machines misbehaving in some way seems eerily similar to reboots. It’s different this time. Really.

    Virtualization

    Virtualization itself can cause a special class of issues that are out of your control. Noisy neighbor problems, underlying VM infrastructure/network or other issues can often only be resolved by starting a fresh node (possibly in a different logical area).

    High Availability (HA)

    HA has become more ingrained in systems architecture. For most instances, without inordinate effort you should be able to smoothly take out a bad server out of a cluster or load balancer with no or minimal downtime so you’re not making your users suffer and in fact will be giving them a better experience than potentially landing on a server with intermittent or worse issues.

    Following a few proper procedures provides some important differentiation and further opportunities to investigate compared to simple reboot techniques.

    Automation and Configuration Management

    Solid automated provisioning allows you to bring up the EXACT same machine configuration, put it in a different network subnet, or tweak it to have newer OS patches or software versions for testing if needed. Recreation is not a binary choice. You have options on how you do it and opportunities to learn from it.

    Centralized Monitoring

    Monitoring has come a long way from simple CPU/Disk/Memory and service uptime. Many vendors offer deep views into your application through application performance management (APM). It’s also easier to historically dig into not just that a CPU spike occurred but which processes were hogging cycles at the time to get at a root cause. You can more readily identify if an issue lies with your application or some supporting/OS services and with good automation simply spinning up a new server to test a theory is easier than excessive pondering.

    Centralized Logging

    Centralized logging can allow easier further log inspection after the node is removed and with much better search, analysis, and correlation capabilities than opening each file in a text editor on the machine itself.

    See you later, not farewell…

    The node often doesn’t have to be destroyed. It merely can be pulled out of service or stopped. It or its virtual disks can be examined further at a later time if needed. Exercise caution with spinning up a node again if it has any behavior to auto-join a cluster/farm or will start processing jobs, sending e-mails, etc on its own. Having a practiced isolated network environment can be handy for further analysis and can double as a test bed in the case of potential security breach analysis.

    Herd away

    By following the guidelines above, STONIH will minimize the time any misbehaving nodes affect your user experience. Just don’t forget to follow-up on a root cause especially if patterns or recurrences start!

  • Is Docker leaving Immutable Infrastructure?

    Is Docker leaving Immutable Infrastructure?

    According to Chad Fowler’s definition, Immutable Infrastructure can be defined as:

    “[S]ervers (or whatever) are deployed once and not changed. If they are changed for some reason, they are marked for garbage collection. Software is never upgraded on an existing server. Instead, the server is replaced with a new functionally equivalent server.”

    Though the idea is powerful implementing it is actually very difficult, especially with hypervisors. Thus, the DevOps community has advocated Docker from its launch to market because of its capability of building the Immutable Infrastructure, due to its layered image approach and super fast “booting” performance.

    However, last week Docker introduced us all to a new feature to allow “Writable /etc/hosts, /etc/hostname and /etc/resolv.conf”:

    “You can now edit /etc/hosts, /etc/hostname and /etc/resolve.conf in a running container. This is useful if you need to install bind or other services that might override one of those files. Note, however, that changes to these files are not saved during a docker build and so will not be preserved in the resulting image. The changes will only “stick” in a running container.”

    It‘s an interesting move as the Writable feature opens another door. Things are no longer static, at least not 100%. For example you could change the state of a running container. The question now being asked is: Where is Docker heading to next? Are they moving away from ‘Immutable Infrastructure’ and if so, is ‘Immutable Infrastructure’ wrong?

    The answer is Yes and No.

    Immutability itself is a hugely valuable way to build, deploy and operate applications. But like all patterns, there is some trade-off.

    One of the most discussed benefits of Docker is the consistency across different deployments: dev, qa, staging, prod, etc. However, there is always some context-specific configuration, e.g. IP, dns name. As the result, those parameters have to be set at the runtime, instead of being hard-coded in the “read-only” images. There are various ideas on how to solve this service discovery problem [1][2][3]. Yet the issue remains that all of these solutions are more or less intrusive to the code and application, which means quite some effort in testing the theories. Then, the whole thing becomes a decision based on personal preference. Even though we of course all want better ways to manage the infrastructure, the engineering cost with the 100% immutability is a concrete no go for a lot of people.

    —

    The new Writable feature gives us the option to non-intrusively determine the configuration at the runtime, so that it will not cost you too much effort, while at the same time retaining the immutability. Assuming you are deploying a WordPress application, which involves web-tier containers and DB containers. You certainly don’t want to hard code the DB instance’s IP, nor bring in an etcd cluster, which adds more moving parts to manage and increases the odds of faliure . Instead, an orchestration tool could do the job by leveraging the Writable feature of Docker:

    Immutable Infrastructure

    As in the above diagram, the orchestration tool will take care all heavy lifting, from instance provisioning, container deployment, to a properly setup /etc/hosts file in the web containers. More importantly, if your DB instance’s IP changed, the orchestration engine will automatically rewrite /etc/hosts to fix the DB connection. And all you need to do is just to specify the following in /etc/hosts:

    Immputable infrastructure - hosts

    Pretty easy, right?

    In conclusion

    While Docker lays a great foundation of code deployment and it is especially useful in keeping the unit as immutable as possible, the deployment context plays an important part in the service discovery. By leveraging the new Writable feature and the Orchestration tool, one could really benefit from immutability, whilst avoiding having to change too much.

    [1] smartstack [2] etcd [3] consul