devopsy.kinsta.cloud

Tag: transient environments

  • The Disposable Development Environment

    The Disposable Development Environment

    Wait! Just wait a dadgummed minute! It takes days to build a development environment, and now I am supposed to read an article on a disposable development environment?

    Yes.

    It seems like it took an entire sprint to upgrade PHP! Now I am supposed to let them build something so they can trash it?

    Yes … yes, you are.

    Disposable DevOps Thinking

    DevOps is based on five pillars:

    • Culture
    • Automation
    • Lean Thinking
    • Measurement
    • Sharing

    Because we automate environments such that they are easy to share, we should be able to measure meaningful business impact. This model is a DevOps practice of modeling developer workflow for operational tasks.

    Why a Disposable Development Environment

    Log in to your workstation and delete the runtime for Java, PHP, etc. Remove, or change, a few config files and empty the trash. Now what? What if your entire team did this because they had a great idea? Panicking yet? What if getting started in any team was as easy as:

    • Cloning the project from source control
    • Starting the runtime environment
    • Obtaining caffeine
    • Getting to work

    The majority of developers today have a hand-built, artisanal, bespoke workstation setups with the following characteristics:

    • It takes days to onboard a new team member.
    • Innovation around runtime environments is stifled.
    • Moving from one team to another is the work of days.
    • Workflow is more like Ops instead of Dev.

    Let’s set some goals:

    • No more than 30 minutes for a new team member to make their first code commit.
    • No more than five minutes to throw away an environment and start over.
    • No more than five minutes for a developer to move from one team to another.
    • Runtime changes happen the same way as application changes.

    None of this is possible unless your teams can throw away a runtime environment at will, thus disposability is an “[-ility]” that we should expect to be delivered with our software.

    Disposable is Indispensable

    The most important concept is handling runtime of the application with a virtual machine (VM) on the workstation. The application code remains on the workstation. This means that developers still use all the tools and workflows they have today. If a mistake happens on the VM, kill it and start over. If it happens on the workstation, then its time to rebuild the laptop.

    At its core, the ideas borrows from common concepts in our production environment (and a specific tool example):

    • Hypervisor for supporting guest virtual machines on the workstation, such as Virtualbox
    • Provisioner to configure the guest machine, such as Chef
    • Source Control to track and manage changes to the application and the configuration, such as git
    • Glue so the provider and provisioner can be expressed as code and managed through source control, such as Vagrant

    Onboarding a New Team Member

    Have the new employee follow a set of directions. It should take no more than 30 minutes for the employee to install the tools and the application he or she is to work on. As a manager, this must be measured and tested relentlessly. New hires should be directed to keep notes on trouble spots and anything that takes a long time. They improve the documentation for the next new hire.

    Accelerate Innovation

    Developers say, “I wish I could upgrade to PHP X! I would be done in minutes, not days!”

    Developers are capable of such an upgrade, so the real problems are:

    • It takes a few hours to upgrade a language version.
    • If the attempt fails, it could be longer to unravel all the hand-rolled changes.
    • If the attempt succeeds, everyone on the team is committed to undertake a fraught journey to upgrade their bespoke workstation.

    With management asking about things such as feature velocity, time to value and internal hurdle rates, who is willing to take that sort of time only to succeed and cause everyone else on the team grief, too? This is why security upgrades are always done with such pain. They are delayed because they are “non-functional.”

    Now abstract the runtime to a VM and the configuration of the VM to code. Assume the developer wants to upgrade PHP.

    • Developer destroys the current VM with PHP 5.2.6 on it.
    • Developer goes into the Chef code (for example) and changes the PHP version to 5.3.29.
    • Developer launches a new VM with a single command.
    • Developer runs automated tests.

    The most likely outcome is some tests fail. With a bespoke environment, this is where the upgrade effort stops. Fixing all the tests is a commitment to getting the entire team to move in lock-step. On a team of just five people this is daunting. Developers are not project managers.

    With a disposable development environment, it takes five minutes to make the change. Developers know it is an identical effort for their colleagues, so the thinking changes.

    Given an arbitrary time window, the tests are fixed and the PHP upgrade saves the time or improves system performance or patches a security hole. To share this change across the team this same developer commits the change in the Chef code. They push it to the code repository and message the team, “Hey PHP 5.3.29 works, please adjust.” One person does all the work.

    Team members stop what they are doing. They stash the work they have inflight. They pull the changes from the code repository. They start the VM and Chef configures it. They unstash their work and get back to providing business value. In less than five minutes the entire team is done.

    Accelerate Security and Compliance

    What if a developer’s upgrade causes security issues? Changing something as important as the language version is meant to illustrate this power. But this is not the right question to ask. Instead, ask this one, “How do we ensure we are secure before we get to production?”

    Assume that any change in the development environment is reflected in production through the same process. For example, when code is deployed to quality assurance (QA), the Chef cookbooks run and they bring up the machine with the new PHP version. Why would there not be tests to show that the application is at least as secure as previously?

    Even more to the point: By making it easy to upgrade the base language, test the change and share it across the team, upgrades are much less risky.

    This thinking has some new assumptions:

    • You trust test automation at the application level. (common)
    • You trust the test automation at the infrastructure level. (uncommon)
    • You have automated security and compliance practices that check before a production release. (rare)

    As managers we must help our:

    • Ops team becomes better at the practices of code management since the infrastructure will be code.
    • Development team makes responsible changes in concert with operations, security and compliance groups.
    • Security teams adopt a security-as-code mindset to facilitate testing before promotion from development.
    • Compliance team joins the development effort so changes are already part of audit output. Compliance-as-code is already on the way.

    Business Outcomes of a Disposable Development Environment

    We hire new people when we need them. Taking two weeks just to get into a working state means we are delaying time to value. Delaying 30 minutes is less time than filling out the HR paperwork. We need this critical time to fill the need that caused the hire.

    Developers have more freedom in the infrastructure to try their ideas. The also have the safety net of putting it back like it was in less than five minutes when the idea does not pan out.

    Patches to the runtime environment for security can be applied at development time where they can be tested through the full deployment cycle. This takes minutes instead of days.

    If all teams are using a standard method and tool chain, then it is possible to adhere to the maxim, “Put your best people on your biggest opportunity.” Developers can move from Team A to Team B in less than five minutes and get started learning the code. They can move back in the same five minutes.

    Changes in the runtime happen at development time, and they are promoted and release with the application. So, perhaps most of all, if it works on the developers laptop then it will work in production.

    Without the ability to simply throw away the environment and easily start over, Dev and Ops cannot innovate toward these benefits. So go ahead and dispose of those bespoke workstations, and be sure to manage toward the benefits:

    • No more than 30 minutes for a new team member to make their first code commit.
    • No more than five minutes to throw away an environment and start over.
    • No more than five minutes for a developer to move from one team to another.
    • Runtime changes happen the same way as application changes.
  • The Transience or Permanence of Provisioning

    The Transience or Permanence of Provisioning

    I realize the title of this article may conjure up ideas about the impacts of entanglement from quantum computing, but no, we are not there yet. Instead, I am referring to the benefits an organization can achieve from determining which environments in its software development life cycle (SDLC) should remain permanent and which are better candidates to be transient to understand the discreet costs of innovation (ideally if virtualized).

    First, to keep an even playing field, make sure your DevOps services are using the “build once, deploy many” philosophy. Under this thinking there is a single build constructed that is deployed unilaterally into each class of SDLC environment (not a unique build for each class). Uniformity under test eliminates variability of results. Uniformity of the basic process of deployment, builds consistency and predictability also will reduce errors and the time it takes to resolve them.

    Second, the actual computing capacity you deliver any environment on may be virtualized, whether you provide your own bare metal assets (private cloud foundation) or if you use a public cloud entirely is less the issue. Understanding whether a class of environment is transient or permanent only affects how you plan for the costs and policies of operating it, and it leads back to the discreet cost of innovation as a result.

    The Cost of Creation

    When this topic is usually considered the first and immediate point of view is usually R&D, or the development class environment. And yes, the ability to provision a DEV class environment and then tear it down when the effort has completed is the normal use-case to justify virtual infrastructure at all (public, private or hybrid cloud usage). The obvious benefits include having no “throttle” on how many concurrent efforts might be spun up.

    Transient DEV environments also build in a concept of failure, or elongated “playing,” not intended to produce a specific result but intended to dabble with innovation to see what “might” be possible. All of this without the capital investment it would take to build and maintain an R&D lab to accommodate it, or set aside fixed computing capacity for it. DEV efforts with intent (i.e., part of a formal project) can be tracked for costs discreetly with that planned innovation. DEV efforts without intent (i.e., more like what used to occur in R&D labs) can be associated to the overhead costs of all innovation.

    The Cost of Evaluation

    But beyond DEV, the next major class of environment to consider is the integration (INT) class. If your company has more than one application, has apps that work together or has applications with upstream or downstream dependencies, you likely need an INT class environment. The major responsibility of this class environment is to provide “something to point to.” Your app (undergoing change) has an environment where it can point to all the other applications to test how the interconnections work. Logically, this class of environment is a better candidate for permanence. This is because you can rarely predict which apps may need this service at which time across the entire portfolio. Many apps undergoing change may be using this at one time to insure they all work with each other (i.e., assessing their interoperability with future versions, early, or prior to production).

    Another reason for maintaining a permanent INT class environment may involve your interaction as a vendor to other companies who “point” their apps at yours as a service provider. Even organizations who have little interaction between their own apps, may well be a significant provider to a wide swath of customer applications. If your customers need a place to point to, the INT class environment is typically the one you want. So in this instance permanence is the logical need.

    Next up, becomes the performance test (PT) class of environments. Emulating production, or being able to project differences between the platform you use, and what production provides is key to success. However, this class of environment is a perfect candidate for transience (especially if your development and production environments are already virtual). Why maintain expensive hardware on the floor, only to conduct periodic testing, which requires massive reconfiguration and preplanning. Dump it. Instead, use a virtualized platform where provisioning is near instant and can exactly mimic both development and production – remember: build once, deploy many.

    Another significant class of environment in the SDLC to examine is the User Acceptance Testing (UAT). Logically, this class environment may get a lot of usage, but ideally only after the app has passed testing in development, INT and PT. Because of this, the UAT should be transient as well. A good candidate to tear down when not in use. You may wish to plan for a setup and tear down of this class to mimic what happens in formal development efforts.

    Finally, the quality assurance (QA) or preproduction (PreProd) class environments, if in use, are designed to emulate the current state of production completely, allowing the app-under-change to run against a more sophisticated type of INT class environment (with possibly copied production data, etc.). This happens more often when the earlier INT class environment contains a great number of other applications under change. This one provide a “cleaner” place to point to, only production operating apps. The QA environment may also be used as an after action, when an error is discovered in production for debugging or more extensive monitoring. Obviously, if your organization finds a need for this class of environment, it will carry the permanence characteristic.

    The Cost of Operations

    Production is by nature permanent. Virtualizing production offers the ability to scale on demand. But a scalable production solution benefits from more mature DevOps services (whether you own the data center or not). Consider for a moment, the number of errors attributed to code post a DevOps embrace. The number of errors goes down, the time to resolve them goes down, the size of the effort gets smaller for less risk, and higher speed. This combination means production is going to suffer far less from aberrant code running amok (asking for tons of memory, CPU or disk due to error). Those kinds of problems get detected earlier (because of DevOps) in the SDLC process and hardly ever reach production after DevOps is engaged.

    With a mature use of DevOps, it means that any kind of scalable solution of capacity-on-demand will get engaged for only two reasons: a dramatic increase from customer demand (the ideal solution) or a denial of service attack attempting to drive your costs higher without real customer use. The monitoring solution you employ will help decipher the difference. If your application has logging features built into it (i.e., you produce a log detailing what customer is performing what key feature in your app and when) you should be able to avoid the latter scenario with an automated monitoring detection effort.

    New Abilities to Measure the Cost of Innovation

    Understanding the cost delta of transient and permanent environments is worth examining. It will help you understand better the cost of innovation on an annualized basis. And it may also reveal how much real effort goes into testing of each kind of class. If user adoption is suffering for example, and you determine that UAT class environments are largely unused or torn down, you know better where to address the problems earlier in the cycle. The same can be said for PT as well, how much test, for what kind of scaled performance in production.

    If your organization maintains INT or QA class environments you may want to allocate their costs to SDLC overhead as they tend to be permanent, and value shared. The development, PT, and UAT environments, if transient, represent the variable cost of a particular innovation effort and can be tracked with good discretion. When considering the timing, value, and benefits of virtualization, you may wish to factor in where you are on the DevOps continuum. If you have had DevOps running for a while, the timing for virtualization and scalable capacity on demand may be better, than if you are just starting out.

    Keep in mind that most organizations who have transitioned away from waterfall and into agile, will lose a good deal of the former financial controls that worked in waterfall, and do not work as well in agile. DevOps delivery is also capable of moving change at orders of magnitude higher speeds than former manual methods. This combination makes trying to cost innovation discreetly more difficult. Being able to associated transient environment costs with formal development efforts (ones with intent) may restore some of that financial discretion. It may also help identify the true overhead costs of innovation located in the operation of environments classed better as permanent.

    To continue the conversation, feel free to contact me.