devopsy.kinsta.cloud

Author: Rob Hirschfeld

  • Are LLMs Leading DevOps Into a Tech Debt Trap?

    Are LLMs Leading DevOps Into a Tech Debt Trap?

    Everyone thought AI would free us from toil. Unfortunately, that’s not true. What it will do, however, is lead to an explosion of technical debt. Some of the most alluring uses of LLMs have the potential to lead to some of the worst outcomes for enterprises and their DevOps and platform engineering teams.

    SK Ventures and Adrian Cockcroft raised the alarm bells in their warning posts about the tipping points LLM introduced into work and society. And in my own Gluecon 2023 presentation, I challenged just how powerful and disruptive the new generation of AI/ML will be for ITOps and DevOps efforts.

    Working Faster is not Working Smarter

    There is no doubt that LLMs are powerful and will change how we think about expertise and knowledge work.  While they are not truly innovating, their ability to draw on such a vast pool of human expertise makes it difficult to distinguish from true creative work. Since they are unbounded, they will cheerfully use the knowledge to churn out terabytes of functionally correct but bespoke code.

    It’s easy to imagine a future where LLMs crank out DevOps scripts 10x faster. We will be supercharging our ability to produce complex, untested automation at a pace never seen before! On the surface, this seems like a huge productivity boost because we (mistakenly) see our job as focused on producing scripts instead of working systems.

    But we already have an overabundance of duplicated and difficult-to-support automation. This ever-expanding surface of technical debt is one of the major reasons that ITOps teams are mired in complexity and are forever underwater. Generating even more scripts, even if they were better written, does not reverse the trend.

    Can AI/ML Avoid This Technical Debt Trap?

    It depends on how we use the expertise in the models. Instead of asking it to generate new code, we could ask it to interpret and modify existing code. For the first time, we have tools to take down the “not invented here” barriers we’ve created because of the high cognitive load of understanding code. If we can help people work more effectively with existing code, then we can actually converge and reuse our systems. By helping us expand and operate within our working systems base, LLMs could actually help us maintain less code.

    Imagine if the teams in your organization were invested in collaborating around shared systems! We haven’t done this well today because it takes significantly more time and effort. Today, LLMs have thrown out those calculations.

    Taking this just one more step, we can see how improved reuse paves the way for reduction of the number of architectural patterns. If we improve our collaboration and investment in sharing code, then there is increased ROI in making shared patterns and platforms work. I see that as a tremendous opportunity for LLMs to improve operations in a meaningful way.

    It comes down to whether we ignore complexity by replacing it or embrace complexity by understanding it. I see this as a choice between independence or collaboration, as illustrated by the table below.

    LLM Driving Independence

    LLM Driving Collaboration

    Create new code

    Interpret existing code

    Replace existing code

    Maintain existing code

    Design new architectures

    Be a guide and mentor

    Build new stacks

    Evaluate alternatives

    Bypass documentation

    Improve documentation

    Can AI Help us Collaborate?

    We keep taking the same hardware and software components and then building unique systems. That variation makes it impossible to leverage each other’s work. At its heart, the reusability problem lies in the difficulty of understanding systems, and that makes it nearly impossible for us to work together, collaborate, reuse and accelerate.

    Breaking this cycle has been our mission at RackN for nearly a decade. We’ve seen tremendous ROI from availability, reliability and speed when customers are able to leverage standardized infrastructure pipelines. We’ve seen firsthand that it is possible to guide teams into collaboration with sufficient guidance, mentoring and training. Collaboration creates 10x outcomes even without AI, but it takes building expertise, deliberate effort and leadership discipline to implement.

    Why does collaboration create 10x ROI?  It works because true innovation actually comes from understanding and building on top of systems rather than rebuilding them from scratch over and over again. The key is being able to leverage LLMs to better understand existing automation and platforms instead of creating yet another bespoke stack.

    Enterprises Should use LLMs to Drive Collaboration

    I believe that the most effective enterprises will use these tools to elevate their IT leaders and architects to focus on collaboration and reuse. They must actually direct them to produce less bespoke automation; instead, they can use expertise in these tools to find and maintain existing systems. LLM presents options, but collaboration requires new thinking and strong leadership.

  • Infrastructure as Code and Six Key Automation Concepts

    Infrastructure as Code and Six Key Automation Concepts

    This summer, I walked into an Infrastructure as Code (IaC) flame war because I was blissfully unaware of just how broadly IaC had been reinterpreted. Reviewing the comments, it’s clear that thinking has evolved since Kief Morris’ 2016 Infrastructure as Code tomb. So, after strapping on my asbestos suit, I decided to dig into what is creating such a wide range of opinions and see if I can share some 2019 insights.

    Why does IaC provoke such passion? The reason is simple: As an industry, we aspire to something that delivers on our collective vision of fully-automated, self-governing infrastructure. In the 00s we called it autonomic computing, then software defined data center (SDDC) and, now, IaC. Sadly, IaC is not it—or, more specifically, it’s not it alone.

    The actual definition of IaC is way smaller than what people want it to be.

    My take-away from IaC discussions is they are built around six key automation themes. Let’s take a few minutes to examine each of the themes. Then we’ll have a basis for answering “If that’s not IaC, then what is it?”

    Programmatic Configuration (IaC)

    Programmatic configuration captures the idea that we can define our infrastructure in a machine readable format. For example, you describe a planned cluster build in YAML to execution by a tool such as Terraform that requests the resources from your dynamic infrastructure and, hopefully, tracks that resource descriptors you get back from that request. Basically, it’s just your IaaS input format.

    Dynamic Infrastructure (or IaaS)

    It describes a system that allows operators to change the resource allocations via an API. It’s a critical part of the discussion because there must be a dependable way to pragmatically change the infrastructure. Some would consider this cloud; however, we should not assumes either managed services or VMs because dynamic infrastructure is much broader.

    Desired State (or State Reconciler)

    Desired state actively compares dynamic infrastructure to programmatic manifest and makes automatic adjustments to keep them synchronized. Typically, the idea of maintaining state is separated from configuring systems. Kubernetes and other Autoscaling cluster managers perform this role.

    Deployment from Source Control (or Immutability)

    It implies that all of the configuration components of the system can be stored as versioned artifacts and deployed from source. A Dockerfile that builds a container is a good example of an immutable process and the generated artifact. The repeatability offered by deploying from source control is transformative compared to past generations of operations that relied on either manual or scripted changes to production systems. Immutability is not IaC: We are focused on an generated deployment artifact rather than managing the infrastructure.

    Composable (or Modular) Automation

    It allows us to manage complex structures through layers of abstraction. In concept, composability is as simple as using platform plugin to script against an API.

    CI/CD Pipeline

    CI/CD pipeline is the glue that pulls all these pieces together to create a resilient environment by continuously adjusting the system components based on the operators’ IaC descriptions. The pipeline relies on having dynamic infrastructure managed to desired state as a target; however, it adds the critical concepts of staged changes over time to the system. Since we need to create systems that can adapt and improve over time, the CI/CD pipeline provides mechanism for controlled implementation change over time.

    Together, these six concepts reflect an integrated new way to think about the infrastructure that is beyond IaC. They allow us to maintain and control our systems over time in a way that is resilient and automatic. More importantly, they bring complex infrastructure problems back into terms that people can comprehend by creating more deterministic and predictable chains of change. That allows us to keep our hands out of the infrastructure and our minds focused on what we are delivering: a continuously integrated data center.

    Since all six concepts are interdependent, continuously integrated data center (CI DC) completely captures our aspirations for IaC. The term intuitively builds from source code into a fully operational infrastructure. More importantly, CI DC demands that we acknowledge the ongoing maintenance required. For that reason, I’ve embraced CI DC to replace IaC.

    Does the term work for you? How do you see IaC or CI DC being delivered in 2020? I’d like to hear your thoughts.

    — Rob Hirschfeld

  • SRE vs. DevOps vs. Cloud Native: The Server Cage Match

    SRE vs. DevOps vs. Cloud Native: The Server Cage Match

    This post supports my DevOpsDays Austin Presentation on May 4

    I don’t believe in DevOps shaming. Our community seems compelled to correct use of DevOps as an adjective for tools, teams and teapots. The frustration is reasonable: DevOps clearly taps into head space for both devs and operators who see a brighter automated future together. For example, check out this excellent DevOps discourse by Cindy Sridharan.

    As an industry, we crave artificial conflict so it’s natural to try and distill site reliability engineering (SRE), DevOps and cloud native into warring factions when they are not. They all share a focus on Lean process.

    Very simply: SRE is a job function, DevOps is a process approach and Cloud Native is an architecture.

    These three concepts have tremendous alignment because they all have the same fundamental goal: increase product delivery through improved flow. It’s really that simple. We are looking to enhance productivity using system thinking concepts from Lean manufacturing.

    SREs improve productivity by eliminating manual operations, reducing errors and improving infrastructure quality. They do this starting from the production infrastructure and working backwards. Highly automated delivery, integrated logs, systemwide monitoring and blameless root cause analysis are key tools. Before we get wrapped up in tools, the distinguishing factor for SRE teams is that they take ownership of the production infrastructure.

    There is tremendous power in assigning ownership for delivery. That’s why I see industry benefit in defining an SRE job function. A DevOps strategy is a system approach and it becomes an anti-pattern to making an individual or team responsible to own “the DevOps.”

    In DevOps, we look for ways to streamline the entire product creation flow. That often drives toward tools and techniques that SREs require; however, our goals are not department-specific. We want to work with all stakeholders including developers, product managers, testers, SREs, executives and customers. This system focus for developers becomes CI/CD pipelines, platforms, work tracking tools and other workflow elements. These tools are not specifically DevOps tools, but confusion is natural since they are often pulled into organizations due to DevOps thinking.

    This SREs should expect to collaborate deeply with other functions under a DevOps process.

    Cloud-native architecture also falls under the DevOps mandate because it strives to decompose large application stacks into highly automated collections of services. There is no magic “DevOps platform” in cloud-native design; however, teams using the architectural approach get modularity, strong APIs, service orientation, simplified configuration and build automation practices that fit perfectly into DevOps thinking.

    For example, using containers does not mean a dev or SRE team is following DevOps process or using cloud-native architecture; however, it is an indicator that they are leaning toward smaller deliverables and improved dev-production fidelity.

    Yet, SRE, Cloud Native and DevOps do have a shared enemy: Complexity.

    We’re doing a great job creating best practices and abstractions that accelerate developers and creating innovative infrastructure options in both cloud and physical. That means we are seeing dual explosions in developer productivity and infrastructure complexity. Unfortunately, this wealth has not made it into operations automation, which has been delivering incremental improvements at best. Even worse, the added workload and complexity adds pressure to SRE functions.

    Ultimately, all three of these concepts are aspects of Lean business practices; however, acceleration from DevOps and cloud native is outpacing progress by SREs. This imbalance creates risk for everyone involved because Lean is about system performance. Delays and failures cannot be isolated to a single silo.

    The mistake of seeing Dev and Ops as silos means creating false contention between SRE, DevOps and cloud native.

    If you are hearing that in your organization, then take it as a warning about how your teams are integrating Lean systems thinking into your processes.

     

    — Rob Hirschfeld

  • How the Lure of an ‘Easy Button’ Installer Traps Projects

    How the Lure of an ‘Easy Button’ Installer Traps Projects

    As preamble for this post, I wrote about the Unicorn Installer challenge. The challenge is that user needs are too deeply heterogeneous for good reasons to create a vertically integrated way (aka an “easy button”) that works for all users. The problem is not just current diversity, it’s also about how we cope with needed changes in the future that would force users to straddle configurations.

    Don’t despair! There is a practical solution to shared ops on project installers: composability.

    If projects focused on maintaining install modules for their specific components, then the community could collaborate on well-bounded problems that should be infrastructure-agnostic. Even building in multiple scripting DSLs (Ansible, Chef, Puppet, Salt or Bash) is a much smaller-scope problem. Further, all of these DSLs have modularity concepts so it’s feasible to pull in community roles/modules/gains/plays into larger automation script. This approach allows us to decouple snowflaked infrastructure and prerequisite configuration from the project configuration.

    At that point, the proverbial “easy button” becomes an opinionated set of project, prerequisite setup and infrastructure-specific steps. These then become vendor-specific buttons: “Press here for the fast AWS install with Dejour SDN and L8istSh9y O/S” as was showcased to great effect at Dockercon.

    With that said, let’s look at how trying to create an “easy button” installer inside the project backfires.

    Cloud Foundry chose to have an in-project installer, BOSH, that has an opinionated installation approach based on generated images. To work, the installer has to maintain an increasing number of BOSH-specific integrations and image generators. These efforts are redundant with other tools non-core to installing Cloud Foundry. Further complicating the story is Pivotal, a primary Cloud Foundry maintainer, which considered BOSH a product differentiator. This creates ecosystem conflict with other Cloud Foundry vendors. While BOSH helps create robust installations, it also drags along a lot of baggage that keeps those improvements from being more generally accessible. For all of the expected ease of installation for BOSH, Cloud Foundry is still considered difficult to install.

    OpenStack installers are a very confusing story, with multiple competing in-project installers. Starting from the single-node DevStack, which provides all of the continuous integration/continuous delivery (CI/CD) testing to the confusing over-cloud/under-cloud OpenStack-On-OpenStack (aka Triple-O) concept. Neither of these initial in-project attempts created practical, universal installers. In fact, they’ve spawned waves of other installers and configuration scripts. Recently, OpenStack decided to allow many of these attempts into the project (aka “big tent”) without coordinated governance making it impossible for operators to determine which efforts they should be tracking. Having some efforts stamped with “community approved” only makes it more difficult to solve cross-platform issues. The effect is further balkanization, as vendors and advanced operators attempt to build working automation silos against competing interests.

    As a positive example, Linux pre-seed and kickstart options are distribution-specific and support a healthy ecosystem. It would be easy to think of consolidating these installs into a single system; however, the likely result would be to lock together rivals and slow down innovation.

    What we want is a thriving ecosystem around a project without endorsement as one being THE install process.

    Why is lack of endorsement critical? It allows the project to advance without being slowed down by installation entanglements created when a single way is anointed. Worse, it discourages ecosystem activity around the project.

    The first successful installers were targeted at a single vertical stack, such as Kubernetes on CoreOS or a version of Salt Kube-up on Google. These monolithic install approaches are helpful jump-starts but limit community diversity. Yet, if they are presented as “the right way,” then other participants will be discouraged from innovating. It only takes one variation in a vertically integrated process for the whole process to become exception hell.

    Our early experience with Crowbar was that our initial installation ideas were not going to remain useful as the project and targets expanded. It was important that we could maintain Crowbar AND replace it with something better. The fundamental lesson is that operations around projects is an evolving practice.

    Hindering the evolution by selecting winners early actually slows the development process.

    This may be counter intuitive; most projects want to embrace “easy button” installers to reduce start-up costs. These critical builders of initial traction may actually undermine projects as they enter a operational maturity phase by creating a fidelity gap between initial and sustaining use cases.

    — Rob Hirschfeld

  • It’s Time to Slay the Universal Installer Unicorn

    It’s Time to Slay the Universal Installer Unicorn

    While many people want a universal “easy button installer,” they also want it to work on their unique snowflake of infrastructures, tools, networks and operating systems. I call them snowflakes because their small differences expand combinatorially. That divergence turns the idea of a prescriptive reference architecture into a destructive myth that limits community and ecosystem growth.

    Because there is so much needful variation and change, it is better to give up on open source projects trying to own an installer and instead focus on making their required components more resilient and portable.

    Universal Installer Fallacy

    So what does an easy button install look like? Let me tell you about one that I built.

    Crowbar Mascot

    It was 2011, the dawn of the OpenStack Era, and our team at Dell was already battle-weary from trying to make repeatable installs around Azure, Eucalyptus, Joyent and Hadoop. The simple reality was there was no success pattern we could follow deploying any one of those platforms—and certainly none cross-platform. Even though all used basically the same compute and network infrastructure, each had custom deployment code and configuration approaches. We jumped on becoming an OpenStack founding partner as a way to try to break the failure pattern.

    What was failing? Each platform had deeply embedded environmental assumptions that were not repeatable or portable.

    For example, one of those platforms required thirty (30!) full racks of gear and tethering back to the master deployer because its management and topology was hardwired into its infrastructure. Others were not much better: Installation often was “ssh apt-get install platform,” then “scp pre-built configuration files.” Even worse, those steps were always proceeded with 30 pages of prescriptive manual system prep steps that were updated daily.

    Incorrectly, we believed that the solution was to create a better installer for OpenStack.

    The Crowbar project, which is still used by SUSE, created a fully integrated install sequence from PXE boot and discovery to testing the final OpenStack installation. We could build a cloud on a complete rack of gear from boxes in minutes. The open source project even used open tools such as Chef and hardware-agnostic options to increase its community appeal. As the first installer (it predated even DevStack), the project was widely used in early OpenStack evaluations.

    Despite that success, as a v1 technology, Crowbar’s major shortcoming was that it tightly coupled components and services together. That made the installation and management process very rigid and non-conforming to disparate operational models and environments.

    Our install scars showed us the real battle lines that prevented portability. Even minor infrastructure variations resulted in broken or site-specific configurations. A major part of the challenge is that variation is not neatly contained; instead, it is spread throughout the environment. We needed to create automation that could both cope with heterogeneity across the community and be change-tolerant after installation to enable upgrades.

    We came to accept that there is no single installer. Instead, scalable installers must assume that they will operate with other parts of the system in a coordinated way, because the diversity of options makes it impossible for them to control everything. In fact, we’ve seen that smaller, composable install steps are much easier to automate consistently. Even more importantly, smaller steps make it possible to manage systemwide upgrades in a controllable way.

    Asking for an “easy button” installer is the wrong question. Instead, let’s ask, “Can we create sharable components that can be automated in multiple ways?”

    There’s no universal installation script to create cloud rainbows; however, there are strategies we can use to make individual projects installable over a wide range of options. These approaches include composability, consistent configuration and use of standard tools and patterns for operation. These items help projects focus on their own concerns and allow operators to apply best practice techniques.

    Basically, the first step is for project developers to limit their scope and delegate operational concerns to operators.

    Even in a platform-as-a-service (PaaS) project that attempts to hide operational complexity from developers, such as Kubernetes, there remains someone who must operate that platform using foundational infrastructure and tooling. We make that job more complex when we get distracted trying to apply the new concepts to the old infrastructure. The addressing of an impedance mismatch pulls effort away from the central abstraction.

    Since we do need consistent installs from the community, projects do have a responsibility to help those efforts.

    How can they help? By ensuring that their component services are well-bounded with clear input configurations, operational requirements, upgrade patterns and output expectations.

    But really, that’s a future post. Please let me know what you think and where we should go from here.

  • 5 Key Aspects of High Fidelity DevOps

    5 Key Aspects of High Fidelity DevOps

    As much as we talk about how we should have shared goals spanning Dev and Ops, it’s not nearly as easy as it sounds. To fuel a DevOps culture, we have to build robust tooling, also. That means investing up front in five key areas: abstraction, composability, automation, orchestration, and idempotency. Together, these concepts allow sharing work at every level of the pipeline. Unfortunately, it’s tempting to optimize work at one level and miss the true system bottlenecks.

    Creating production-like fidelity for developers is essential: We need it for scale, security and upgrades. It’s not just about sharing effort; it’s about empathy and collaboration.

    But even with growing acceptance of DevOps as a cultural movement, I believe deployment disparities are a big unsolved problem. When developers have vastly different working environments from operators, it creates a “fidelity gap” that makes it difficult for the teams to collaborate.

    Before we talk about the costs and solutions, let me first share a story from back when I was a bright-eyed OpenStack enthusiast:

    I was at the fourth OpenStack Summit in Boston. My team at Dell had created open deployment automation, Crowbar, that could turn raw servers and install a fully working OpenStack cluster in about 30 minutes. I don’t want to oversell it; there were plenty of rough edges and preconditions. However, the automation worked and OpenStack came up like a kudzu seedling. In fact, we’d actually brought a half-rack of servers to the summit for live production demos.

    On my way back up to the demo rack, I was riding an elevator with one of the founding developers of the project. He was positively glowing over a new project that he’d created called “Devstack” and wanted to show off the cool logo they’d designed. He gushed, “It’s awesome! It runs all of OpenStack in a single python environment on my laptop! You don’t even need Vagrant anymore—it’s a super fast and easy for developers to get OpenStack running because all the configuration is in Python.” I watched as he typed a command, text scrolled and welcomed me to the OpenStack API. “See, now I’m running OpenStack!”

    At the time, it seemed like a harmless toy, since a single-node cloud deployment is useless to users, testers and operators. I pointed out that we’d gotten our multinode automation working in virtual machines (VMs) now so it would be easy to use even without dedicated hardware. He asked what we’d written it in. “Mainly Chef with some Rails UI,” I responded, thinking we’d made some solid 2011 DevOps choices. He was less enthusiastic, because we were not using the developer-preferred Python. Even so, I encouraged him to come by the rack for a demo, but without any anticipation.

    We left the elevator on a friendly wave of OpenStack synergy. He sprinted into the developer lounge and I trudged back to the user and vendor showcase. Even without words to describe it, I saw that the project had started diverging between developers and operators. Developer collaboration drove a need for something fast to set up that ran on a single laptop. More importantly, Devstack stripped away all that messy multinode networking and DevOps tooling that made it hard to configure and run OpenStack. In what turned out to be a one-way journey, there was now a universal developer environment.

    Unfortunately, that developer environment configuration did not translate into production.

    In fact, OpenStack deployments are notoriously challenging and there is an overabundance of divergent approaches. Even with so many efforts, few have managed basic operational milestones such as upgrades and high-availability configurations. I believe a wide fidelity gap is a key, if sad, part of this story.

    We don’t have to live with fidelity gaps!

    In fact, DevOps exists to close those gaps by bringing developers and operators to a shared space. You are already well-versed in the need to treat development as a pipeline with automation to keep it repeatable.

    If we know the challenge, why do we allow these gaps? It’s because fidelity can have too high a cost in terms of complexity, system overhead and time. These costs multiply the faster we try to iterate. That creates a conflict between the organizational benefits of agility and fidelity; sadly, few teams have been willing to burden developers with fidelity or operators with change frequency.

    How Can We Make Fidelity Cost-Effective in Initial and Ongoing Costs?

    There are five components to consider: abstraction, composability, automation, orchestration and idempotency/repeatability.

    Platform abstraction

    Platform abstraction hides the differences between environments and tools that make developer and production environments different. For true fidelity, we use “platform” in a very broad way. A cloud API or server is a platform, but so is an operating system or container system. In practice, there are many possible and acceptable abstraction layers.

    Teams need to invest in ways that allow the same automation to work on multiple platforms in the same way. That requires modifying the automation to be resilient in different environments and accept configurations instead of making assumptions. These changes are good hygiene, anyway, and make your automation more robust and portable.

    The simplest way to add abstraction is externalize environmental configuration. Since it’s difficult for operational systems to detect the broader system configuration, we need to inject that information into your deployments. Unfortunately, that leads to the “massive pre-deploy inventory file” antipattern and our next component.

    Composability

    Composability goes beyond making automation modular. It means that we can flow information, à la functional programming, through the discrete parts of the install.

    In a good composable design, each unit has defined inputs, outputs, preconditions and behaviors. That allows us to chain them together to accomplish work. Our systems should be able to connect outputs from earlier modules to inputs from later ones. This eliminates the “pre-configuration file of doom” because we can build up system knowledge as the work is done. Since the functions are well-known, we can determine if we’ve completed all the needed setup.

    Substitution is even more important than having a chained configuration flow. Since modules are isolated actions and known parameters, we can create alternate modules based on user- or infrastructure-driven requirements. For example, choosing CentOS vs. Ubuntu changes package management; however, downstream applications only care that their packages are installed. We need to make it acceptable to detect and adjust modules based on situational awareness without breaking up our downstream components.

    This type of change tolerance is essential to fidelity.

    Automation

    You may assume this is obvious; however, we all have a tendency to overlook manual steps or setup in our daily work. Common statements such as, “It takes a day to build my dev environment,” or, “We always run this pre-config script the systems are up,” indicate that we are not as automated as we want to admit.

    Fidelity gaps open when teams have different processes.

    We don’t expect developers to use hundreds of physical servers; however, we can expect them to isolate components in ways that are similar to operations. That adds overhead to their work that requires automation. We don’t expect operators to reset and redeploy systems 10 times a day; however, we can expect them to use repeatable configurations that do not require manual intervention.

    Remember: It does not have to be the same automation in all cases! Composability allows us to have equivalent but different paths to the same state.

    Sequence/Orchestration

    There is a downside to composability. If you’ve replaced your fragile monolithic automation with robust composable modules, then you’re going to have to add some complexity to chain them together. It an unavoidable tradeoff because the composite system needs glue to hold together. That glue is orchestration.

    Truthfully, adding orchestration is not a downside. It is a necessity for scale. Any multisystem configuration process requires orchestration to be sustained. The nature of scale applications is that we have to coordinate activities horizontally across the system during deployment, scaling and upgrade. That allows us to create trust relationships between nodes, set up networks before trying to use them and coordinate servers and clients while we upgrade APIs. Wait loops and iterative convergence are difficult to troubleshoot and more difficult to predict.

    There is a concrete benefit when you combine composition and orchestration: Small work units create a system that is very transparent and parallelization in operation. Those are key values in scale ops.

    Idempotency / Repeatability

    The last component is one of the most elusive: idempotency. The objective is to allow multiple runs of modules in our automation system to leave the system in a consistent state. That allows us to repeat execution of a component without harming or undoing work that we’ve already done.

    The concept is not simply skipping work we think we completed! If we rerun an action, then the action should exit in the requested state every time. If the state is already good, then do nothing. If the state is not configured, then create it. If the state is not good, then fix it. Basically, we want to be confident that we can achieve our end goal every time but never make unneeded changes.

    This is not a simple task! The nature of configuration is to change systems, and we usually assume that we are going from unconfigured or misconfigured to correctly configured. For idempotent scripts, we have to guard against going from correctly configured to broken.

    This may seem like a lot of work for developers who usually start from clean systems, but it’s life-saving for operators when a timing error causes a unit to fail.

    Hey, You’re Adding a Lot of Work to my Deployments!

    Following these five principles does add some overhead; however, all of the changes are good hygiene anyway. They make your automation more robust and portable. More importantly, they reduce friction between your development, test and ops processes.

    Deployment fidelity is a real benefit that helps drive success. Hopefully, these five principles have made it a more concrete and obtainable goal.

  • What If We Had Git for DevOps?

    What If We Had Git for DevOps?

    I used to hate Git. Compared to my previous source control systems (SVN, SourceForge and floppies), it was cryptic, opaque and hard to learn. It was comfortable to create a set of files and folder, then snapshot them as a unit. It was simple to understand and improved my individual productivity.

    But that snapshot created a huge tangled mess when I tried to collaborate with other developers.

    Once I got over the learning curve, Git revolutionized how I wrote code. Small changes made a lot more sense with frequent integrations. I could easily review the impacts of collaborators’ changes. Even more critically, I could keep my work around while I tooled their incremental fixes and improvements. Development—especially open source development—is a team sport and requires tools that facilitate collaboration at the most fundamental level.

    But it was not just learning Git; I also had to adapt how I structured projects and collaborated.

    In that context, DevOps automation has been missing the equivalent of Git. We need composability!

    Can a Git Model Work in DevOps?

    We’ve been building great configuration automation tools (Chef, Puppet, Ansible and Saltstack) over the last few years and now there’s a new wave of cloud templates (AWS CloudFormation, OpenStack Heat, Terraform) entering the fray. These tools are like my original source control systems: they create a vertically integrated stack of configuration that works for a specific case and platform. There are ways to merge them into new stacks, but that ends up feeling like creating a whole new stacks.

    Why does merging scripts create new stacks? Changes in other components often break each other.

    When you reuse someone’s script, their changes can quickly break your carefully built automation. Instead of collaborating, you’re just forking their work. It feels like we’re running around data centers with automation on floppies. When we fix something in one module, it’s very hard to push that back into the community—we have not built clear, functional boundaries between modules that isolate changes in a way that reduces the risk of using other authors’ scripts and makes it easier to contribute improvements.

    Isolating changes does not mean we don’t break integration! It just makes it easier to collaborate upstream.

    The answer in our experience is to think of ops scripts like Git: Even inside of a larger set, we need to make the parts independent and decomposable. How do we translate Git into ops terms? Repos are like full systems—directories map to components, files to roles. Like Git, when we make incremental changes to any item, the whole system is impacted. However, we can still isolate that change down to a single line so we can manage collaborative work. This works when the impacts of changes are isolated (like a text file) and fails when changes are part of a tightly integrated unit (like an image file).

    We already have git with ops scripts. How is this different?

    We are suggesting an even more composable approach. Each ops role should be designed as a functional role with clear inputs and minimal side effects. To achieve that isolation, we’ve been building an orchestration platform, Digital Rebar, to feed the right inputs into existing opts scripts. That allows collaboration within a module without unexpected external side effects. It also allows reuse of modules because changes are well isolated.

    When changes are contained, incremental and managed, the whole system is more tolerant to multiple authors. In that way, we may build system automation that’s more Git-like.

  • DevOps and the Challenge with Services

    DevOps and the Challenge with Services

    At RackN, we’ve been working for years to make DevOps portable across platforms. For us, that means the ability to move between clouds and from cloud to metal. Of the several challenges to portability—composability, API heterogeneity, sequence, cross-system dependencies—making configuration management tools work with services has been particularly important, as significant parts of building systems rely on services that are not directly configurable.

    Let’s look at DNS as an example. Whenever we configure services on a node, there are DNS implications. If we bring up a web server, we want to use a logical name for the node. If it’s a secure web server, that name should match our certificate. If we want to make that service discoverable, we likely want to add a logical name for that node, such as “clustermanager6.example.com” or similar. If you include names for BMC and rack location—not to mention IPv6 naming—it’s easy to image nodes with 10 names. Even cloud nodes have similar issues. That’s a lot of entries to manage!

    The problem with all those DNS names, and similar services, is that they are NOT configurable from the node. They are services with APIs that require restricted authorization.

    Services and Automation

    DNS is just one simple example.  System-level DevOps has many places where the configuration workflow needs to include external service actions. In cloud environments, generally that means either writing pre- or post-configuration steps that talk to the specific cloud API or using one of the cloud-specific template files. In physical environments, generally that means a lot of added manual steps or tightly constraining automation to inventory map files. No matter what, nodes don’t own DNS configuration—it’s a node external service with the API elements that are site specific.

    All of these approaches make DevOps brittle and site-locked. Five years ago that was okay, when choices were more limited and scripts were hand-crafted by DevOps artisans. Today, we need our automation to be robust, portable and transparent.

    We realized system automation needed to treat services interactions as equal to configuration. That means when we’re working on provisioning, we make sure service steps can be woven into the sequence of configuration steps. They must not be assumed correct or delegated to pre- or post-conditions because that makes it impossible to test and re-create working environments.

    What can you do? We’ve found service registration tools (we like Consul.io) are essential and relatively easy to implement to help track services that we manage or proxy. More broadly, we’ve been working in the open on Digital Rebar to blend configuration and services into a single workflow.

    The first step is to re-examine your configuration management scripts (and their prerequisites) for hidden service API interactions. Exposing those external dependencies will help you improve your automation repeatability and portability.

  • Two paths to metal devops: cloud-like API driven & cluster building

    Two paths to metal devops: cloud-like API driven & cluster building

    I’ve been seeing a rising interest in metal DevOps fueled by containers and scale-out data center platforms (like Hadoop, Ceph & OpenStack) that run at the metal level. While I see this is a growing general trend (Packet, Internap, RackSpace, OpenStack Ironic, MaaS), I’m going to stay firmly within my wheelhouse and use OpenCrowbar as my reference here.

    Building on the API-driven metal features of OpenCrowbar, this has translated into two paths for workloads to run on metal:

    1) “Cloudify” the metal using APIs from tools like Chef Provision, SaltStack Libcloud, Docker Machine, Cloud Foundry BOSH. These tools have clients that target cloud APIs like OpenStack and Amazon. These same clients work against cloud are easily ported to Crowbar’s APIs.  Five years ago, conventional wisdom was that we’d need a universal cloud API; however, practice has shown it’s not very difficult to wrap APIs in a way that does not reduce every cloud to a least common denominator.

    2) DevOps deploy the workload using hand-offs to tools like Chef, Saltstack, Puppet or Ansible. This approach leverages the community scripts (Cookbooks, Modules, Playbooks) for the workload with the critical ability to create a tuned environment and inject the needed parameters directly into the scripts.  A critical lesson we learned going from Crowbar v1 to v2 was for our scripts to have crisp attribute input/output boundary to avoid embedding environmental knowledge into the code.

    While I’m casting this in Crowbar terms, I see this approach to metal as coming into the market by force fuels by a desire for containers-on-metal and devops-on-metal.

    Let’s look at some of the unique and shared use-cases for each approach:

    Metal API

    		 Both

    Metal Cluster
    • Easy Cloud to Metal Migration
    • Minimal Tool Customization
    • Portability of DevOps Scripts
    • Take advantage of power cycling
    • Enables constant refresh cycles
    • Leverage Hardware features
    • Advanced Network topologies

    In either case, you have to handle bespoke (hipster word for custom) steps in the provisioning flow that are unique to the your operational needs. Our experience is that each site (even each server!) is unique in some incremental way.  For example, one site may require teamed networks with VLANs while another requires flat networks with an SDN layer.

    These differences are not mistakes or errors: the reality of physical ops and individual operational choices mean that there are a lot of valid configurations. Rather than attempt the Sisyphean task of enforced conformity, we work to abstract differences so that they can be ignored when they are not material.

    In the end, the choices are not mutually exclusive. Metal APIs are often faster but harder to optimize. You can use them to get started quickly and then invest time to optimize a cluster for long term operations. The underlying physical orchestration can support both.

    Are you looking at getting closer to metal?  Which of the options above makes the most sense to you?  I’d love to hear about your use-cases, architecture and configuration requirements.

  • Are VMs becoming El Caminos?

    Are VMs becoming El Caminos?

    In 2010, it looked as if virtualization had won. We expected all servers to virtualize workloads and the primary question was which cloud infrastructure manager would dominate. Now in 2015, the picture is not as clear. I’m seeing a trend that threatens the “virtualize all things” battle cry.

    IMG_20150301_170558985Really, it’s two intersecting trends: metal is getting cheaper and easier while container orchestration is advancing on rockets. If metal can truck around the heavy stable workloads while containers zip around like sports cars, that leaves VMs as a strange hybrid in the middle.

    What’s the middle? It’s the El Camino, that notorious discontinued half car, half pick-up truck.

    The explosion of interest in containerized workloads (I know, they’ve been around for a long time but Docker made them sexy somehow) has been creating secondary wave of container orchestration. Five years ago, I called that Platform as a Service (PaaS) but this new generation looks more like a CI/CD pipeline plus DevOps platform than our original PaaS concepts. These emerging pipelines obfuscate the operational environment differently than virtualized infratructure (let’s call it IaaS). The platforms do not care about servers or application tiers, their symantec is about connecting services together. It’s a different deployment paradigm that’s more about SOA than resource reservation.

    On the other side, we’ve been working hard to make physical ops more automated using the same DevOps tool chains. To complicate matters, the physics of silicon has meant that we’ve gone from scale up to scale out. Modern applications are so massive that they are going to exceed any single system so economics drives us to lots and lots of small, inexpensive servers. If you factor in the operational complexity and cost of hypervisors/clouds, an small actual dedicated server is a cost effective substitute for a comprable virtual machine.

    I’ll repeat that: a small dedicated server is a cost effective substitute for a comparable virtual machine.

    I am not speaking against virtualize servers or clouds. They have a critical role in data center operations; however, I hear from operators who are rethinking the idea that all servers will be virtualized and moving towards a more heterogeneous view of their data center. Once where they have a fleet of trucks, sports cars and El Caminos.

    Of course, I’d be disingenuous if I neglected to point out that trucks are used to transport cars too. At some point, everything is metal.