devopsy.kinsta.cloud

Author: Boyd Hemphill

  • The Disposable Development Environment

    The Disposable Development Environment

    Wait! Just wait a dadgummed minute! It takes days to build a development environment, and now I am supposed to read an article on a disposable development environment?

    Yes.

    It seems like it took an entire sprint to upgrade PHP! Now I am supposed to let them build something so they can trash it?

    Yes … yes, you are.

    Disposable DevOps Thinking

    DevOps is based on five pillars:

    • Culture
    • Automation
    • Lean Thinking
    • Measurement
    • Sharing

    Because we automate environments such that they are easy to share, we should be able to measure meaningful business impact. This model is a DevOps practice of modeling developer workflow for operational tasks.

    Why a Disposable Development Environment

    Log in to your workstation and delete the runtime for Java, PHP, etc. Remove, or change, a few config files and empty the trash. Now what? What if your entire team did this because they had a great idea? Panicking yet? What if getting started in any team was as easy as:

    • Cloning the project from source control
    • Starting the runtime environment
    • Obtaining caffeine
    • Getting to work

    The majority of developers today have a hand-built, artisanal, bespoke workstation setups with the following characteristics:

    • It takes days to onboard a new team member.
    • Innovation around runtime environments is stifled.
    • Moving from one team to another is the work of days.
    • Workflow is more like Ops instead of Dev.

    Let’s set some goals:

    • No more than 30 minutes for a new team member to make their first code commit.
    • No more than five minutes to throw away an environment and start over.
    • No more than five minutes for a developer to move from one team to another.
    • Runtime changes happen the same way as application changes.

    None of this is possible unless your teams can throw away a runtime environment at will, thus disposability is an “[-ility]” that we should expect to be delivered with our software.

    Disposable is Indispensable

    The most important concept is handling runtime of the application with a virtual machine (VM) on the workstation. The application code remains on the workstation. This means that developers still use all the tools and workflows they have today. If a mistake happens on the VM, kill it and start over. If it happens on the workstation, then its time to rebuild the laptop.

    At its core, the ideas borrows from common concepts in our production environment (and a specific tool example):

    • Hypervisor for supporting guest virtual machines on the workstation, such as Virtualbox
    • Provisioner to configure the guest machine, such as Chef
    • Source Control to track and manage changes to the application and the configuration, such as git
    • Glue so the provider and provisioner can be expressed as code and managed through source control, such as Vagrant

    Onboarding a New Team Member

    Have the new employee follow a set of directions. It should take no more than 30 minutes for the employee to install the tools and the application he or she is to work on. As a manager, this must be measured and tested relentlessly. New hires should be directed to keep notes on trouble spots and anything that takes a long time. They improve the documentation for the next new hire.

    Accelerate Innovation

    Developers say, “I wish I could upgrade to PHP X! I would be done in minutes, not days!”

    Developers are capable of such an upgrade, so the real problems are:

    • It takes a few hours to upgrade a language version.
    • If the attempt fails, it could be longer to unravel all the hand-rolled changes.
    • If the attempt succeeds, everyone on the team is committed to undertake a fraught journey to upgrade their bespoke workstation.

    With management asking about things such as feature velocity, time to value and internal hurdle rates, who is willing to take that sort of time only to succeed and cause everyone else on the team grief, too? This is why security upgrades are always done with such pain. They are delayed because they are “non-functional.”

    Now abstract the runtime to a VM and the configuration of the VM to code. Assume the developer wants to upgrade PHP.

    • Developer destroys the current VM with PHP 5.2.6 on it.
    • Developer goes into the Chef code (for example) and changes the PHP version to 5.3.29.
    • Developer launches a new VM with a single command.
    • Developer runs automated tests.

    The most likely outcome is some tests fail. With a bespoke environment, this is where the upgrade effort stops. Fixing all the tests is a commitment to getting the entire team to move in lock-step. On a team of just five people this is daunting. Developers are not project managers.

    With a disposable development environment, it takes five minutes to make the change. Developers know it is an identical effort for their colleagues, so the thinking changes.

    Given an arbitrary time window, the tests are fixed and the PHP upgrade saves the time or improves system performance or patches a security hole. To share this change across the team this same developer commits the change in the Chef code. They push it to the code repository and message the team, “Hey PHP 5.3.29 works, please adjust.” One person does all the work.

    Team members stop what they are doing. They stash the work they have inflight. They pull the changes from the code repository. They start the VM and Chef configures it. They unstash their work and get back to providing business value. In less than five minutes the entire team is done.

    Accelerate Security and Compliance

    What if a developer’s upgrade causes security issues? Changing something as important as the language version is meant to illustrate this power. But this is not the right question to ask. Instead, ask this one, “How do we ensure we are secure before we get to production?”

    Assume that any change in the development environment is reflected in production through the same process. For example, when code is deployed to quality assurance (QA), the Chef cookbooks run and they bring up the machine with the new PHP version. Why would there not be tests to show that the application is at least as secure as previously?

    Even more to the point: By making it easy to upgrade the base language, test the change and share it across the team, upgrades are much less risky.

    This thinking has some new assumptions:

    • You trust test automation at the application level. (common)
    • You trust the test automation at the infrastructure level. (uncommon)
    • You have automated security and compliance practices that check before a production release. (rare)

    As managers we must help our:

    • Ops team becomes better at the practices of code management since the infrastructure will be code.
    • Development team makes responsible changes in concert with operations, security and compliance groups.
    • Security teams adopt a security-as-code mindset to facilitate testing before promotion from development.
    • Compliance team joins the development effort so changes are already part of audit output. Compliance-as-code is already on the way.

    Business Outcomes of a Disposable Development Environment

    We hire new people when we need them. Taking two weeks just to get into a working state means we are delaying time to value. Delaying 30 minutes is less time than filling out the HR paperwork. We need this critical time to fill the need that caused the hire.

    Developers have more freedom in the infrastructure to try their ideas. The also have the safety net of putting it back like it was in less than five minutes when the idea does not pan out.

    Patches to the runtime environment for security can be applied at development time where they can be tested through the full deployment cycle. This takes minutes instead of days.

    If all teams are using a standard method and tool chain, then it is possible to adhere to the maxim, “Put your best people on your biggest opportunity.” Developers can move from Team A to Team B in less than five minutes and get started learning the code. They can move back in the same five minutes.

    Changes in the runtime happen at development time, and they are promoted and release with the application. So, perhaps most of all, if it works on the developers laptop then it will work in production.

    Without the ability to simply throw away the environment and easily start over, Dev and Ops cannot innovate toward these benefits. So go ahead and dispose of those bespoke workstations, and be sure to manage toward the benefits:

    • No more than 30 minutes for a new team member to make their first code commit.
    • No more than five minutes to throw away an environment and start over.
    • No more than five minutes for a developer to move from one team to another.
    • Runtime changes happen the same way as application changes.
  • HomeOps – Learning DevOps At Home

    HomeOps – Learning DevOps At Home

    When I took a job as Technology Evangelist for StackEngine, my wife’s first question was, “Why would someone pay to fly you around the country and talk about how you work?” My youngest son put a finer point on it, “Yeah Dad, we love you and we don’t listen when you talk about work!”

    I tried to explain Lean principles. That went over like a lead balloon. I tried a more human approach in explaining John Willis’ Karojisatsu post and the rampant stress and depression in the operations industry. They felt sorry for those people, but still didn’t see 100’s of folks crowding in a room to hear about it.

    How am I going to foster organizational change when my own family doesn’t care? I needed a way to communicate my ideas so anyone could approach them.

    So, I started doing a single load of laundry each day.

    LaundryOps

    Really it was pretty obvious looking back. My wife typically did our laundry each Sunday. We could either stay at home most of the day, or we could put it off and stress about running out of clean underwear. It’s the perfect metaphor for the problems with large batches.

    The following Sunday my wife noticed there was about a third of a load in the hamper. She said, “Hey, there’s no laundry to do today, let’s go to the water park this morning before it gets crowded.”

    My wife was pleased. The kids were thrilled. I was thinking, “Hmmm, this continuous delivery model is interesting.” I was also thinking that single piece flow provides supporting precedent.

    LaundryOps – Continuous Delivery

    In their seminal book Continuous Delivery (CD), Humble and Farley stress that CD makes the working environment more humane.

    Lean principles say that I should identify what variable I am trying to impact. This tenant is expressed as the measurement pillar from the CALMS acronym coined by Willis and Damon Edwards.

    My goal was to maximize quality family time. It was not to conserve water or energy. We have done a great deal more hiking, swimming and other fun things because our weekend is full two days.

    Laundry Ops – Reasoning About Change

    So, if I practice continuous delivery, and do a load a day, what happens when I go on a trip and come back with 5 days worth of clothes?

    Well, the first time I panicked! It was apparent that CD did not apply to special situations. I needed to break process. But, after considering the pile of clothes, I realized that doing a large load today and tomorrow would bring the system back to equilibrium the following day.

    Adopting CD thinking for our laundry exposed a stunning amount of spare capacity in our “softener, detergent, laundry cycle” or SDLC for short. I did, however, recognize the inclination to regress to my comfort zone at the first sign of trouble in the new process.

    I need to evolve my thinking about the batch size of work.

    Often during the software delivery life cycle (huh, SDLC too) a developer/project manager/stakeholder thinks: “I will do this one more thing before deploying.” This is often because the current deploy process is highly manual, risky or time-consuming. Certainly none of those things were true of laundry.

    I need to find ways to make deploys easier, safer and repeatable.

     

     

    ClosetOps – Unexpected Consequences

    Doing one load per day rather than many on Sunday led to a number of other interesting results that all apply to software delivery.

    ClosetOps – Cash Flow

    If our clothes are not in the hamper, washer or dryer, then where are they? Turns out our closets were burgeoning. At this point my wife said, “I wonder how much money is in there?” So we went through her closet with It’s Deductible and learned the answer was about $3500. Retail we estimate $8500. We figured our house contained about $30,000 in clothing. Today that estimate is about $13,000.

    We released $17,000 of cash in roughly one year.

    Today, in your source control management (SCM) system, you have weeks or months of features, bug fixes and innovation building up. You have paid the salary of the developers, testers, project managers et al who contributed to the effort. Because you are waiting for some date long in the future to release those features, how much cash is in your closet … er … SCM?

    ClosetOps – Tools and Costs

    Too often DevOps is boiled down to tools. For laundry, our use of a low water low energy washing machine and gas dryer means that even though we run them every day, there is no significant impact on our energy and water bills.

    Choosing a tool should be done only in the context of the measures being optimized.

    Back in the Real World

    In my real job as Director of DevOps, customer satisfaction was the bottleneck to sales. We would work for months behind the curtain to produce a beautiful custom web site. Except it wasn’t what the customer wanted. And it was weeks late. And it was way over budget. And making any changes took far too long.

    We changed three things:

    • We instituted a weekly show-and-tell meeting and delivered the site to the project manager for their approval in the production environment.
    • We automated the developer environment and matched it exactly to the production environment.
    • We ensured from the first day of the project that any committed work by the developer was available to be seen by the project manager

    We achieved wild success:

    • Project manager wanted the account manager involved. Account managers brought customers with them. The value chain was fully represented.
    • Customers were thrilled to be part of the process
    • Site launches were a non-event. We had been launching daily or more for months while under construction.
    • Making updates and changes to the site became trivial

    Customer satisfaction measurably improved:

    • Account managers more new business with the confidence we could deliver.
    • Account managers won repeat business from existing customers who appreciated our transparency and delivery.

    SuckOps – DevOps Sucks

    My wife has a number of allergies. If I vacuum the house each day, her quality of life increase dramatically. Since wife maintenance is cheaper than wife repair, I was doing this duty and I was tired of the 30 minutes a day it took. In reality, vacuuming was happening every 3 to 4 days.

    SuckOps – Automating to the Right Degree

    Santa Claus brought us a Roomba last Christmas. It is possible to set this particular model up so that it automatically vacuums sections of the house each day.

    I have never bothered to fully automate this process.

    It turns out that manually setting the towers is easy. These fences are so easy to move that it was not worth the effort to fully automate vacuuming. So each day I close a couple of doors and turn it on. I then set up towers and move it to the living room and leave for work.

    When our kids get home, they reset the towers and run it in their bedrooms, hallway and kitchen. I set it up to cover the office and dining room at night.

    SuckOps – Tools Enable Collaboration

    Notice that the job of vacuuming is now shared with my kids. My wife can run the vacuum too since she does not have to be in the house with the running vacuum. The tool has enabled us to share this chore.

    Choosing tools that foster collaboration between traditional silos is a key factor in tearing down the wall of confusion.

    SuckOps – Quality Through Repetition, Not Control

    The Roomba does not do the same high quality job as our Dyson. However, because it does it far more often, the overall effect is that our house is more allergen free.

    Again, adhering to the key metric: improving the livability of the house for my wife, we can clearly see Santa brought us DevOps with this automation tool.

    Santa gets DevOps.

    Back in the Real World

    The automated development environments allowed the Ops team to share the configuration with developers in a way developers were used to seeing. This is often referred to as Infrastructure as Code https://devopsy.kinsta.cloud/2014/05/05/meet-infrastructure-code/.

    By the third week of using this tool (Vagrant + Chef), some developers were making requests in the code to change the configuration. This fostered a closer working relationship between the developers and operators.

    Because development of the site was closely coupled to delivery of the site, quality of the deployment process was achieved through simple repetition.

    The key pay off, however, was the ability of any developer to work on any customers project with a single command. This one outcome saved a measured 8 hours per week for each engineer!

    PaintOps – My Family Understands Me

    Does my family understand DevOps? Yup.

    Traditionally to paint the interior of our home we would take a couple of days off, buy tons of supplies, move furniture, stress out, eat out … It underscores the problems of large batches again.

    My wife suggested that instead of trying to get it done over 4th of July holiday as we had originally discussed that we instead just painted a bit each weekend when time allowed.

    Measuring Time

    The project took roughly 30% longer total time than it would have. It spanned about 2.5 months. However, time was not the key measure of success. Stress was. Once we go through the first several batches and understood set up and tear down (drop clothes, moving furniture, washing brushes) I would often come home in the evening to find my wife painting a wall.

    My wife was able to deliver paint any time she felt like it.

    Software engineers working under a continuous delivery paradigm can deliver features whenever the feature is ready.

    PaintOps – Surprisingly Lower Costs

    The cost of the project was about 10% smaller than the traditional way. This was due to our ability to easily estimate the amount of paint to buy for a small batches very accurately.

    Market fluctuation in the price of paint (sales) kept this number from being closer to 15%. There was a certain discipline in not buying a very large quantity of paint when the price was temporarily lower.

    PaintOps – A Small Bit of Chaos

    Having our living room half old and half new for a couple of weeks is something we could tolerate. Its part of our family culture.

    Any paradigm shift in how work is done must match the culture of the organization.

    Culture must evolve with, or even before, new models for work can be attempted.

    Back in the Real World

    A DevOps journey is always a work in progress. We constantly change the way we use tools, the tool, the process etc. This means we occasionally miss a deadline, delete a production server or a root file system.

    As part of a cultural shift, the business understood that we were in a far better place than 11 months ago. This shift was enabled by many small wins that built trust over time. Our culture had to be that each mistake came with an effort to ensure it never happened again.

    A bit of chaos is necessary to constantly improve. Measuring the impact of each change justifies the cost of mistakes.

    HomeOps – Food for Thought

    When staring at a DevOps transformation, many executives and leaders are frustrated by the lack of a manual. There is no prescriptive way to do DevOps like there is for ITIL or ISO compliance.

    The reason for this is that the only prescription of DevOps is to continuously improve the system by which you deliver value to your customer.

    Find something small in your life that you want to improve. Measure it based on the metrics that define “improve.” Change behaviors or introduce a tool and note the difference in your metrics. React accordingly. Repeat.

    Choosing small problems leads to solving large ones over time. Soon enough you will feel ready to tackle a problem in your professional life.

    You are now free to DevOps.

    (See Boyd present on HomeOps at DevOps Connect: CD Summit the afternoon of Oct 8 at Austin Convention Center. Tickets are sold out but walk ups are welcome and there is room)

  • Technology is All About the Business

    Technology is All About the Business

    In 2011 I joined a company as their fifth employee. They had been in business a little over a year. They had a product. They had customers. Oh, and the other four people? They were rockstars.

    Customers, however, were unhappy. The system was frequently down. Rebooting a machine was often necessary and lead to poor performance for hours.

    I was hired to fix “the database problem” in a multi-tenant SaaS system. On my first day I was given a tour of the technology by the engineer. The code was immaculate, but the database had some core issues. We had the database in good working order in a couple of weeks. We lost our first customer about two weeks after that.

    Why did this customer leave? Ultimately because we were not working on the problem that mattered. We created a local optima in the database and another in code quality. It turned out that there were more fundamental problems.

    Achieving local optima is a false economy. (1) We needed to think outside our core skill sets. Systems thinking pointed the way.

    The Reaction that Mattered

    We had three months cash left. We had two months until our make-or-break event. The business closed a sale for two months operating capital! The customer was a very big brand whose needs would stress the existing software beyond its limits.

    At this point the problem that mattered was immediate and clear. How do we guarantee this customer’s success? As with many things, the answer was obvious once we thought of it: We stood up a duplicate of the production software just for them.

    The Patterns we Recognized

    Standing up a second production environment was no easy task. Over 200 bash commands lovingly executed across five hand-crafted cloud servers. It took days to get our bespoke second system stitched together.

    While it was ideal that we landed such a big contract, the truth of the matter was the business was not confident in our product’s technology. For each potential sale after landing the big one, I would get a call, “Are you sure we can handle this next customer?” The answer was already apparent. We, the tech team, were not confident either.

    Changing the Software

    We needed to improve the software so one customer’s activity had no possibility of impacting performance for another. This would localize problems to a single customer and increase the business’ confidence. We needed to do it quick. We needed to do it with a single engineer and DBA.

    Each customer would get their own full copy (share nothing) of the software. Deployment of the software would be reduced to the simplicity of stamping out parts in a factory. With this model we rewrote the system and closed three new customers on it for our big event.

    Moving to this “share nothing” paradigm is likely the single biggest contributor to the company staying in business, thriving and eventually being acquired. The confidence of the sales force increased dramatically. Sales increased dramatically. Our runway moved out past the six month mark.

    Why Share Nothing

    We recognized our system was a B2B offering, not B2C. This a type of pivot know as a business architecture pivot. (2)

    Trying to build all of the infrastructure for a self service system that scales to unknown demand, while really cool, was not the problem we needed to be solving. Yes, there are inefficiencies in such a share nothing system. No, none of them impacted our bottom line.

    For example, we didn’t care that we were wasting compute. The price of the compute was built into the product. The customer was not paying for compute. They were paying for the product and they were willing to pay so much that this cost didn’t matter.

    Powerful Consequences

    We effectively rewrote the original system in two months. This still seems miraculous, and to some extent it is a testament to the amazing quality of the business logic code. There were, however, other consequences to the “share nothing” model that gave us a significant, if temporary, market advantage.

    • We had no need to spend engineering cycles on plumbing such as user management.
    • We had no need for a billing system. Human-time invoicing in a B2B world was plenty effective.
    • We had no massive scale of data or load. We used familiar technologies.
    • When an instance went down, only one customer called. We learned most of these things were training issues.

    For each item above, the need for engineering was eliminated or dramatically decreased. This means the engineer used that time to create new features. More features means more revenue if you are building the right things. Our CEO and founder was a genius; we were building the right things.

    And it Broke

    Sales trusted the system. New customers were coming in. Existing customers were happy. Infrastructure costs decreased.

    It was short lived. The tools and technology used to implement the repeatable, “share nothing” software were not up to the task. The success of sales was creating a new problem, though a much nicer one to have! We needed more systems thinking.

    Did DevOps Fail

    We did not initially choose the right tool. Is this story a DevOps failure? If you subscribe to the idea that process informs tool choice, then the answer is a surprising, “No.” The DevOps thinking was sound, we simply need to choose better tools to support the model we had chosen to implement. Technology is all about the business at the end of the day.

    Part Two of this case study will highlight some of the evolution of the tools and processes we put in place that lead us to further success.

    References

    (1) The Goal – Eliyahu Goldratt, Jeff Cox
    (2) The Lean Start up Eric Ries