devopsy.kinsta.cloud

Tag: infrastructure as code

  • Infrastructure-Testing Dominoes

    Infrastructure-Testing Dominoes

    Understanding the artifacts of a given technology component is critical to an enterprise-class DevOps service offering. Second-most critical is understanding the dependencies between given components. Testing helps illuminate those dependencies.

    For example, let’s assume I have a server running the previous version of the Windows operating system (OS) that I now want to upgrade to the latest version. The previous deployment package of this version of the OS would be an artifact. It should also include all of the test automation I run to validate that version of the OS is working after I install it. The artifact also might include configuration files I deploy when I install that version of the OS on this type of server in this class of environment. Having these bits of automation bundled together allows me to do a successful install of my OS (assuming no hardware errors) on perhaps many servers.

    As I create a new artifact for the latest Windows OS, I keep in mind all the things I had in my last version. I might include a newer config file and perhaps a set of tests (albeit new ones) to make sure it is working when I am done. Bundling all these bits gives me a new Windows OS artifact or package that I can use over and over on this kind of server in this class of environment. The idea is that I am creating a single kind of artifact or package to cover this kind of technology—in this case the latest version of the Windows OS operating system. All my artifacts should be version-controlled. So when I’m done I have an artifact for the previous Windows OS and one for the latest version. Each artifact version has embedded or included testing (pointers to, or discreet automation) to verify the validity of the install, and perhaps a little more to prove the OS works.

    Within my DevOps tooling, I likely have some sort of artifact management system. It may not be as formal as a full-on configuration management database (CMDB) system, but whichever tool I use will have some sort of database to manage my artifacts or packages by version. How long I retain these artifacts will be defined in my retention policies, but likely I will retain every version in use at my company today. This is step one, and it forms the foundation for my next effort in cataloging the dependencies that exist between artifacts.

    Enter the Spider’s Web

    Under the scenario I have described, when I deploy either the previous OS or the latest one, I would run a series of tests specific to that version to ensure the OS is working properly. As an infrastructure engineer, I tend to think my job is done at this point. But it isn’t. To truly validate that my OS is working, I really need to know what is running on the server I just deployed to. If my server was bare or new, then I am really done. But if my server has middleware running on it, then the job is not finished until I re-deploy the right version of the middleware on top of my new OS and verify it is still working properly.

    To pull this off, I will need a catalog of the technologies that depend on each other. The OS example is kind of simple, because it lives at the bottom of the stack where a server is concerned. But on top of the OS might reside a middleware platform (perhaps messaging or a database). On top of the middleware might reside an application or perhaps several applications, depending on the horsepower of the server in question. Upgrading the OS component implies I am redeploying the middleware layer and then the application layer on top of that.

    Enter full-stack engineering. It takes an application-centric view of the world, but this implies a single application-centric view. It looks at one app from top to bottom and treats the entire stack as a single version-controlled artifact. But if you have legacy environments that run multiple applications on the same physical box, perhaps on the same virtual instance, you have the makings of a spider web of dependencies. There are no full-stack solutions (to my knowledge) that have figured out how to incorporate multiple applications at the top of the same stack, varying middleware in the middle and “n” number of virtual instances that ride on one OS on one physical box. Instead, each piece of the technology tends to be treated separately. Thus the need for some sort of catalog that tracks the interdependencies between them all.

    Starting the Domino Testing

    So once I have discreet testing that is version-controlled (or associated with) each type of technology component, I will have to re-execute those tests if anything “under” me undergoes a change. For example, if the version of my WebSphere is upgraded, beyond just the core WebSphere testing that occurs, I really should execute the tests associated with each application to ensure “that app” is still working after the upgrade. You could apply this premise to a database layer (say, upgrading the current version of Oracle, as an example). To really know if the database upgrade was successful, you should run the testing of each dependent app and ensure “that app” still works after the upgrade. The results are likely to be surprising.

    To execute this kind of domino testing, even if you have the web of dependencies defined and the artifacts or packages managed, you can take one of two approaches: redeploy everything, although this method may not actually kick off tests if it detects the versions did not change from the last deployment (smart overwrite), or set up discreet testing jobs within your release orchestration software to execute the entire web of dependencies and track the results. In either case, you are going to need human oversight on whether it is better to keep the new version or roll back to the old one. Only a human can prioritize the impact of what might seem small or catastrophic (based on a human perspective) of a given app or series of apps.

    Impact to the Bottom Line

    True quality can be assessed only through testing. Part of breaking down silo-thinking is taking a holistic view of the technology your organization supports. Just because the OS support engineer’s testing passes does not mean the upgrade was a success. For an upgrade to be a success, the tests of everyone else who relies on that technology underpinning must pass to prove it worked. The entire team needs validation when the underlying infrastructure undergoes change. The dominoes of tests must pass from end to end. Only then can true quality or true success be measured.

    Of course, most engineers are reluctant to have “app software” guys influence whether they did their own jobs right. After all, most software quality problems are in the code, not due to infrastructure errors. However, if the applications upon which your business relies cannot work, no matter what the reason, after an infrastructure upgrade, it is a problem for the entire business, not just the software guys. To tear down the silos, we need to look at impacting events for change across boundaries and drive toward overall results, not just departmental ones. When this occurs, the cost of quality is significantly reduced.

    To continue the conversation, feel free to contact me.

  • Infrastructure as Code is SDN for Ops

    Infrastructure as Code is SDN for Ops

    As we continue to recognize the importance of DevOps—particularly its automation and orchestration aspects—to achieve the agility necessary for continuous deployment of apps, the question of how the term “Infrastructure as Code” (IaC) applies to traditionally network-bound systems and services is bound to crop up more frequently. After all, in addition to core IP-based networking characteristics required for an app to communicate and serve up content, an entire ecosystem of app services that support, secure and scale it must be considered.

    IaC is driven by the application of development principles to network infrastructure. Configuration, scripts, templates, profiles and the like are treated as code, in that they should be reviewed, versioned and provisioned from a centralized repository (one hopes on-premises) rather than managed as discrete network elements.

    In a recent article in TechBeacon, Boyd Hemphill, director of evangelism at StackEngine, noted, “The basic principle is that operators (admins, system engineers, etc.) should not log in to a new machine and configure it from documentation.”

    Christopher Null, CEO of Null Media and author of the article, then wrote, ” … But IaC is a concept that extends beyond simple infrastructure automation. IaC requires applying DevOps practices to automation scripts to ensure they’re free of errors, are able to be redeployed on multiple servers, can be rolled back in case of problems, and can be engaged by both operations and development teams.”

    The idea is that you can describe any given service as a set of artifacts and deploy (and redeploy) them at will. So rather than set up a system and configure it directly, its configuration any associated artifacts (such as profiles or templates) are treated as code and stored in a common repository. A controller—the orchestrator—is responsible for distributing those artifacts when necessary. The controller uses an API (or protocol such as OpenFlow) to manage the configuration of services across a distributed infrastructure. It’s decoupling the service definition from the service and its platform.infra as code

    Read that again. It decouples the service definition from the service and its platform and assigns responsibility for provisioning and configuration to a controller. Kind of like decoupling the control plane from the actual forwarding and switching actions (the data plane) and assigning operational responsibility to an external controller.

    At its simplest, treating infrastructure as code is separating configuration from the platform and enabling a software-driven or software-defined approach to provisioning, configuring and managing infrastructure. It’s applying the principles of software-defined networking (SDN) to operations, rather than to devices and systems.

    But as Null makes clear in his article, IaC goes beyond just automation and orchestration, adopting the best practices of development with respect to code reviews, versioning and management of that “code” to ensure consistent, predictable and repeatable deployments.

    It’s all driven by software, ultimately, and deployed by software, and defined by software but always under the auspices of the operational experts: the engineers and architects.

    Treating infrastructure as code is basically applying the architectural principles of SDN to operations itself. It’s SDN for ops.

  • Agile Infrastructure: Because your Infrastructure Deserves Tests, Too

    Agile Infrastructure: Because your Infrastructure Deserves Tests, Too

    Editor’s Note: This post was updated Aug. 4, 2018, to reflect an updated slide presentation at the end of the article.

    Infrastructure as Code is a truly powerful concept whose significance is best described with the words of Adam Jacob, CTO at Chef, in Web Operations: Keeping the Data On Time: “Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources.” And so it’s not for nothing that treating infrastructure as code is regarded a key principle for reducing cycle time and maximizing flow in Continuous Delivery and DevOps, respectively. Even if your plans are less bold as you read this article, wouldn’t you agree that consequent validation of what you automate on the infrastructure level — long before it goes into production — would be extremely powerful?

    So, bear with me for a while and please don’t rush head over heels into automation by simply picking one of the established infrastructure automation tools such as Chef or Puppet, getting everyone trained in Ruby, and blindly start to “automate all the things”. If that’s how you do it, then you are most probably doing it wrong. In You’re Doing DevOps Wrong. Automation in the Enterprise., author Alan Sharp-Paul suggests that, before doing any automation, you first have to understand precisely the requirements and how to validate them to gauge success and maintain quality of your automations over time:

    • Are the configuration files in place? Do they contain the right settings?
    • Is this port open? Is that port closed?

    Why Agile Infrastructure?

    Honestly, you and I wouldn’t expect anyone to come up with reliable software merely by choosing an arbitrary programming language and a nifty IDE without having some sort of engineering process in place, right? Likewise, the (automated) provisioning of a solid production environment demands a no less methodical approach. Luckily, Infrastructure as Code enables us to treat the provisioning of infrastructure as well as automating deployments and configuration as an Agile engineering discipline. Although the concepts of Agile Infrastructure are not entirely new, testing against live infrastructure has only recently been facilitated with the consolidation of tools such as Ansible, Chef, Puppet, Docker, Vagrant and Serverspec under the hood of Test Kitchen (see more on that below). Here are some of the key implications of applying Agile software development practices to the infrastructure level:

    Alignment of Devs and Ops

    Agile software development teams are driven by customer demand and aim to produce outcome in short, regular intervals. Enabling IT Operations and Infrastructure departments to work in small iterations and react to changes quickly allows them to align with and effectively become driven by software development. The outcome of these joint efforts will be both working and deployable software at the end of each iteration.

    Fail Fast: Feedback Loops

    Executing infrastructure tests as part of your Continuous Delivery build pipeline (before an application is deployed and long-running tests are executed) establishes another important feedback loop that enables you to “fail fast” and get the right people involved. The underlying principle is simple: by expressing your intentions twice — once in the code and once in a test — you’ll know you’ve caught a bug whenever these assumptions don’t match.

    Regression Safety

    Believe it or not: whether you are using Ansible, Chef, Puppet or any other — you’re running on code! Their respective issue lists are full of unresolved issues of various severities with new issues coming in regularly. Having a sufficient safety net of tests for your infrastructure in place allows you to quickly identify bugs as they creep in (from either side) and protects you against unwanted regressions. After all, a bug in your infrastructure can be very hard to track down and can be much more severe than a bug in your application.

    Refactoring

    Code refactoring becomes a breeze when you have tests in place that immediately alert you when you broke the desired state of your infrastructure (and all you wanted was to make your code simpler and more maintainable). This is invaluable when you want to recompose your existing infrastructure from 3rd-party components offered on public hubs such as Ansible Galaxy, Chef Supermarket or Puppet Forge, and especially when you are migrating away from your existing automation stack.

    The Red, Green, Refactor Cycle of Test-Driven Development

    Test-Driven Development (TDD) is a popular software development process that involves writing code in short, repeating development cycles with the goal to enforce good design and enhance confidence. While it is certainly no silver bullet and has been controversially discussed in the past, the Red, Green, Refactor Cycle, which is at the heart of TDD and whose principles I want to quickly summarize here, hopefully gives you some inspiration on how to incrementally build up your infrastructure:

    1. Red: Understand the requirements and the state you want to implement. Then think about how to capture this state in a test. Implement the test, then run your entire test suite and see the added test fail — and if it doesn’t, something is really wrong.
    2. Green: Implement just enough functionality to make the test pass. Don’t care much about readability, simplicity and design for now — just make it work. Run your tests again and watch all tests pass. If they don’t, either fix your test or your implementation, or take a step back (revert to a point in your code that’s known to work) and start over with a smaller increment.
    3. Refactor: Now that your tests pass, improve your implementation and make sure, by running your tests again, that you didn’t break anything. Once you are done refactoring, start the cycle over again.

    Keep in mind that each repetition of the cycle refers to a small increment. In terms of infrastructure, such an increment could be the installation of a user, a package or the placement of a configuration setting. When you arrive in a spot where you broke something, you can easily revert to a point in your code that’s known to work. Leveraging such practices really makes the difference between Infrastructure Automation and Infrastructure as Code, as described in Infrastructure as Code – Automation Is Not Enough by Keith Morris.

    Test Kitchen, Terraform, AWS and awspec

    Now, how to get there? Test Kitchen is an easily extensible test harness that allows you to test your code written in Ansible, Chef, Puppet (and others) on various cloud providers, virtualization providers and operating systems with tests written in a variety of test frameworks.

    In the following presentation, I give hands-on examples for you to quickly dive into the topic of using Test Kitchen with awspec as a framework to test Terraform deployments on AWS:

  • Version your infrastructure

    Version your infrastructure

    Infrastructure is code, and like any code, you need to version it. Notice how that works:  If it’s infrastructure, then it’s a fundamental part of the system; nothing’s going to work without it, which makes it valuable. So valuable that in full-stack deployments, it is part of the application. And if it’s code, then it should be under version control.

    (more…)

  • Network Engineers as Programmers? Not so fast.

    Network Engineers as Programmers? Not so fast.

    It is interesting, as the triplet new waves of hype evolve – cloud, SDN, and DevOps – that we are repeatedly hearing The Voice of Reason say things like “All network engineers need to be programmers”. While this is certainly not the only voice that demonstrates what I’m talking about, an example of it is Kirk Byers’ commentary on Network Computing’s site. (http://www.networkcomputing.com/data-centers/programming-an-essential-skill-for-network-engineers/a/d-id/1297898) I do not know Mr. Byers, and want to be clear, he’s not the only one portraying this wisdom, so the reference is merely that – a reference to the syndrome I discuss here.

    In DevOps, certainly having staff members that understand both application development and network engineering is a bonus. Being one of those fabled unicorns, I also know that they’re few and far between. I’ve worked with some great ones over the years that “just get it”, and they certainly add value to an automation project that involves networking. But there is a reason that there are few of us that understand both.

    I’m a programmer. I’m a very good programmer. In the course of my career, I’ve also become very good at strategy and even technical marketing. I learned networking because of my career path. When I chose to gear my career at new challenges, there were many. From datacenter administration to IT management, from storage and servers to networking to security, from cloud to embedded. I’ve enjoyed every aspect of my career, but I’m still ‘a software guy’ at heart… Just one that knows about networking, storage, and servers at a deep level.

    The thing is, my knowledge is not some magical formula. There are people in your enterprise that know all of this, and not all of them are software people.

    SquarePegI’m just suggesting we stop and think about it. When making the move to DevOps, there is a need for a certain set of skills. Those skills absolutely do not have to be vested in a single individual. There’s more overhead if you’re coordinating efforts, but think about what is going into place – automation scripts (of one form or another, no matter what tools you use) that have to have a big initial push to get developed, and then, unless your environment is in perpetual flux, just get used and occasionally modified.

    This does not require retraining of the entire networking staff. In fact, in maintenance mode, those scripts are not generally going to require regular work. Indeed, the largest changes to these scripts will occur as part of a pre-planned project that can take them into account.

    Looked at another way, developers sometimes need to deal with networking issues… Does that mean all application developers need to be retrained as network engineers?

    And there’s a risk. If you decide all of your network staff needs to learn how to code effectively in whatever language and/or tool you settle upon, some of them are likely to leave. Because they’re network engineers and not application developers for a reason. Just as I am at heart a developer, they consciously chose network engineering. For the amount of maintenance required after the initial push to get the automated portion of DevOps into place, one network engineer who is also interested in development is plenty. It doesn’t have to be the whole team. If it did, then all network engineers would need to be security engineers also, no?

    It’s a wave. It will wash over your IT shop in stages, cresting with the workload required to get the automation part of DevOps running, then receding, leaving ripples, but not a ton of water behind. Treat it as such. Get the help necessary to fill the gaps, put a team together that can address all the issues, then let them all go back to their roles, with added (and reduced) responsibilities that DevOps brings. Let developers do the development, working in tandem with network engineers. Find the network engineer that is most interested in development, and have them sit with the developers, working it out, getting everything right and learning what the scripts do. Then let that person handle maintenance, and let the developers roll back until the next major project needs teamwork.

    In the context of network engineering, system engineering, and application development, it’s a project, not a lifestyle change, to automate things. We know how to do projects, they don’t generally involve upheaval and retraining of entire teams.

    But if you’re still convinced you need the unicorn that can do it all, give me a call. That is, after all, what my firm does – bring knowledge both broad and deep to the table to help you solve problems – but honestly, most organizations don’t need someone like me nearly as much as they need a process, a list of attainable goals, buy-in from IT management for the manhour investment, and a plan.

  • Infrastructure-as-code, new rules for the old game

    Infrastructure-as-code, new rules for the old game

    There’s been a lot of discussion regarding DevOps and its role in fostering quality and continuous improvement through increased levels of automation. In turn, this automation is fostered through emerging infrastructure and software architectures that are exposing control plane and management programming interfaces, which has been dubbed in many circles as Infrastructure-as-Code.

    This movement allows development and operations personnel to have real-time interactions with the underlying environment—the network, servers, operating systems, and storage—that are used to deliver applications and data. It also means that as the environment becomes more programmatic the business must treat this this effort as a first-class application development process and ensure that the corresponding assets follow common coding practices and meet with a consistent and sustainable architecture.

    Those that are unfamiliar with the activities of system administrators and data center operations may have an ill-conceived notion after reading emerging literature on DevOps that operators have been running the business on the backbone of purely manual processes. Even as a life-long IT person, until I started working with infrastructure and operations as part of my role, I was unaware of what happens behind the great curtain that keeps the digital bits flowing through the organization.

    The truth is that operations and administrators have been developing scripts for years to automate routine tasks. These scripts have typically been used to dynamically generate or update configuration files that were then applied after a restart. This change to Infrastructure-as-Code is significant because now these configuration changes can be made on the fly and are often immediately acted upon without having to take the system offline. If you’re now putting your hand over your mouth upon the realization of the potential disastrous effects this can have, you understand the impact of this power.

    Thanks to Software Defined Networking, hypervisors, and even Software Defined Storage, an operator can programmatically change the entire environment underlying the application without having to notify the application of this change. Architecturally, this is a wonderful thing. Pragmatically? Well, there’s a lot of bad code out there that makes assumptions about the underlying infrastructure and is about to be exposed when migrated to these modern infrastructures.

    Hence, building, testing and deploying infrastructure control applications must become part of the overall Software Development Life Cycle (SDLC) and begin to be built, tested, deployed and managed using the same controlled techniques that are hopefully being used for any applications that are being developed by application engineering.

    You may be asking, “Wait, doesn’t that mean my QA environment must have the same infrastructure components as my production environment?” To wit I respond, “Maybe!” In some cases this may be advantageous for ensuring that production deployment can support continuous deployment processes, which will ensure limited human intervention from the point of user acceptance through deployment, which will, hopefully, lead to limited opportunity for production failures. However, because we’re moving to a more programmatic infrastructure many vendors can also supply virtual instances of their infrastructure code for purposes of testing and this may be satisfactory for your needs.

    Of note, there are many businesses out there that do not develop their own software. They primarily use Commercial-Off-The-Shelf (COTS) applications. Thus, these businesses may not have an established SDLC practice. Likewise, these businesses may not believe DevOps is relative for them since they do not do “development”. If your business falls into this category, the move to Infrastructure-as-Code and Software Define Data Centers should be perceived as a driver for DevOps adoption as well as formally adopting one of the well-defined SDLC approaches. Even if these components won’t impact you for a few years until your next refresh cycle, the advantages DevOps brings around improving quality and consistency in delivery is of definite value to your business now.

    Infrastructure-as-Code is a significant change in capability that operations has for delivering compute and storage resources to the business. It will allow them to have greater insight into the hardware allowing them to do predictive maintenance, migrate applications to various resources based on demand, and tune the overall environment to meet the business’ needs at a given moment. In short, it will allow IT to deliver compute services with a high degree of availability and security. It will also expose shortcomings in existing applications that have made assumptions about the underlying environment on which it is operating. Thus, if operations and application engineering are not working together and the entire application environment inclusive of the infrastructure control apps is not tested and deployed as a unit, the likelihood for failure will be high.

  • Meet Infrastructure as Code

    Meet Infrastructure as Code

    Maybe you have already heard the terms “Programmable Infrastructure” and “Infrastructure as code”.  But even if you have not, if you are interested in DevOps it’s something you need to know.   Most new lingo causes me to roll my eyes.  But in this case, it’s not just new lingo, it’s an entirely new concept. Besides making the nerdy feel cool when they say it, these terms describe a new idea of what an application consists of. (more…)