devopsy.kinsta.cloud

Author: Scott Rogers

  • DevOps and Change Management

    DevOps and Change Management

    In “The Phoenix Project”, the character Erik introduces the underpinning principles of DevOps as “The Three Ways”.  Let’s talk about the First and Third Ways and how change management works with DevOps, but first let’s spend a few minutes talking about baseball caps.

    Last weekend I was looking for my Atlanta Brave’s baseball cap and found three as I expected for a family of three with two adults and one child.  Since they all looked alike, I compared sizes to figure out which one was mine and was surprised to find they were all the same size!  How can one size fit three very different sized people?  The answer was simple: They have an adjustable strap.  Let’s use this idea to explore the relationship between DevOps and change management.

    One size fits all with an adjustable strap

    IT is being asked to iterate faster and keep our systems highly available while doing so.  Those two ideas seem at odds: How do we fail fast and keep our infrastructure highly available?  Change management can help!

    Many people I speak with view change management as an anathema, as being incompatible with the principles DevOps.  The word “onerous” comes up a lot.  Inflexible is also quite popular.  I understand the sentiment and would like to help the community see change management differently.

    I think of change management as a continuum spanning rigorous to mild, heavyweight to lightweight.  Just like the baseball cap with an adjustable strap, change management can be adjusted for as needed.

    The First Way

    When describing the First Way, Erik invokes Deming’s quote, “appreciation for the System”.  In “The Top 11 Things to Know about DevOps” Gene Kim refines the idea as, “seeking to achieve profound understanding of the system”.  In my article on devopsy.kinsta.cloud titled “Systems of Things”, I assert that understanding the system is required before improving the system.  It seems pretty clear the First Way is about understanding the system we want to improve.  It is also about delivering business value to the customer.

    The Third Way

    The Third Way is about encouraging experimentation and building a culture that rewards risk as a way to speed maturation.  Rajat Bhargava expands upon that idea in his article, “DevOps Demands we Fix Twice” where he asserts that we should release quickly to speed product maturation.  It seems pretty clear to me that the Third Way is about speeding maturation of the features and functionality that deliver value.  The Third Way suggests we iterate faster, accepting some amount of risk as a trade-off.  I assert this risk is solely focused on features and functionality and not infrastructure.

    Tighter for infrastructure and looser for features

    After reading “The Goal” I began to think of code, the features and functionality that is ultimately delivered to the customer, as the material that flows through the factory and IT infrastructure as the machinery in the factory.   The machinery factories only exist to process material.  IT infrastructure is analogous to machinery; it only exists to allow features flow through to customers.

    In a well-run manufacturing facility the material flowing through the factory and maintenance on the machinery are both governed by well-defined and well-understood processes.  These processes are the analog to IT change management.

    The machinery should only be down for maintenance at very specific times.  At all other times the machinery should be ready to produce widgets.  The same is true for the Ops part of DevOps; Ops exists to allow feature and functional to flow through to the customers.  Ops should endeavor to keep the systems up and available for Dev to push code.

    Practical Application

    In practical terms, this means that infrastructure changes and code releases should be treated very differently.  The frequency and timing of code releases should be decoupled from the frequency and timing of infrastructure changes.  Infrastructure changes should not impact code releases and vice versa.  Infrastructure changes should come with a high amount of rigor while feature functionality should come with much less.

    A single change management process accomplishes this by:

    • Decoupling code releases and infrastructure changes when possible
    • Allow the frequency and timing of code release to be independent of infrastructure changes
    • De-conflicting infrastructure changes and code releases

    I think my colleague Stephen Fishman summed it up nicely, “The mission of Change Management… is to simultaneously increase the frequency of intentional change executions while reducing the duration and impact of interruptions for services provided….”

    Rent to own

    It can be tough to see change management and DevOps as compatible.  Instead of agreeing or disagreeing, just try the concept on for a while.  Rent the idea (it’s free) that change management rigor is a continuum, an adjustable strap and see how if it works for you.

  • Systems of Things

    Systems of Things

    What do Chevy V8s and DevOps have in common?  They’re both complex systems with many interactions between their moving parts.

    The first time I did minor surgery on a Chevy engine I learned a lot about systems.  The engine was a tired 283 cubic inch V8 in a 1966 Chevy Malibu  which my grandfather purchased new in 1966 and passed it on to my (much older) brother around 1980.  During my brother’s ownership it was driven hard and received a less than flattering paint job.  By the time it was passed to me it ran about as well as it looked, which is it say it was rough.  A buddy of mine who thought he knew something about engines convinced me to replace the valve train with parts from a junkyard.  The results were not good; we only succeeded in making it run worse.

    Interchangeable Parts

     The problem was that valve train components are not interchangeable after they wear-in.  Sure, when they’re brand new the parts are interchangeable.  But after a while the parts wear together and become fitted to their neighbor.  Had I better understood this complex system I wouldn’t have expected old valve-train parts to improve the engines performance.

    In “The Phoenix Project”, we are encouraged to think of DevOps as an interconnected system, in the same way “The Goal” taught us to think about manufacturing processes.  In an interconnected system, the parts have to work together with precision to achieve the goal.  In DevOps, we talk about shortening cycle-time and increasing release frequency, requiring Dev and Ops processes to work together with precision timing.

    Unexpected Results

    How many times have you made a change to something (anything) and afterwards thought to yourself, “That didn’t go as planned”?  I know I have had the experience on more than one occasion.    Taking the time to understand the system is hard, but it pays off over time.  Just as Erik in “The Phoenix Project” reminds us, “…Step 1 is to identify the constraint”.

    Be mindful of the system and seek to understand before suggesting improvements.  Failing fast and fixing twice makes sense when talking about features, but not so much when talking about system availability.

    Getting Better All the Time

    Embrace the system of things… seek to understand the interaction between the moving parts.  Try to be aware of processes that have become fitted to their neighbor and what will happen if just one of those processes is changed.  Avoid the urge to suggest changes without first understanding the system.

    I got better at rebuilding engines as I learned the underlying systems.  Knowing the systems allowed for changes that improved performance.  Understanding the tradeoffs in the system was also important; an engine tuned to develop maximum horsepower at full throttle will run rough at idle.

    Take the time to understand the interconnected systems we are attempting to improve.

  • DevOps and The Goal

    DevOps and The Goal

    Reading, “The Goal” was a pivotal event in my career.  More than a decade ago a friend of mine who was an operations professor at Georgia Tech told me it was his favorite book.  We were discussing the challenges of getting work done in a corporate environment.  I was grumbling about leading a web dev team in a company that did not embrace the software development discipline.  Already a fan of Tom DeMarco and Timothy Lister’s, “Peopleware”, and being very vocal about the lack of ability to focus whether due to environmental distractions or the constantly changing priorities and schedules, I was receptive to the message in “The Goal”.  It gave me a vocabulary and a roadmap for improving our throughput issues.

    Our throughput problems were a textbook case for development disaster.  When I joined the project, we had three source code repositories, from none of which could we build production and when we did produce a build it would work in pre-production environments but not production (this just slightly predates the widespread acceptance of continuous integration).  We also had the classic inconsistent release cycles that would lengthen based upon insistence by the business owners to include more features.  Looking back, it is not surprising in the least that we were not successful.

    Not surprisingly, these issues led to a lack of trust with the business.  Overcoming the technical problems was straightforward using brute-force tactics.  We addressed the source code problem by recruiting a senior developer and empowering him to make improvements.  We overcame the build quality issue by using a primitive form of agile and enforcing knowledge transfer; the operations engineer sat between two developers during the build and release process and he was the only one allowed touch the keyboard.  Our tactics were basic, and they worked.

    The bigger challenge facing us was getting our business owners to accept a different SDLC model.  We were saddled with a classic waterfall process, each phase being dependent upon the last with a lot of idle time built-in at each phase gate.  The shortcomings of the waterfall process were amplified because the business lacked confidence about when, if ever, they would get a feature released to production. This lack of confidence led to a constant rearranging of the priorities to get the most important feature du jour in the next release.  The constantly changing priorities created even more idle-time and a lot of work-in-progress (WIP) which Goldratt’s Theory of Constraints (TOC) tells us is at the root of inefficiency.

    Using TOC as a guide, we redefined our SDLC from requirements through QA.  We focused on developing a process that de-coupled the phases, reduced idle-time and work-in-progress (WIP).  De-coupling the phases meant the work could queue between each phase allowing the teams, (requirements, development, QA, and operations) to vary independently; they didn’t have to wait on the team before or after to complete their work which reduced idle-time.  We then focused on improving the slowest areas, which seemed to be requirements and deployments.

    We convinced our business owners to trial a monthly release cycle (this was a long time ago, even before continuous integration was widely accepted!).  The monthly release cycle helped give the business confidence that they would see their features in production at an expected pace and it allowed requirements to cycle on schedule independent of development and QA.  Some features would get done in a month and some would span several months.  Over a period of three months we got into a rhythm that worked for about 14 months until organizational focus changed and we focused elsewhere.

    Reading “The Phoenix Project” reminded me about those days and the challenges Goldratt helped us overcome.  I really liked “The Phoenix Project” and many of the situations rang true.  I have since moved from development to operations (twice!) which gives me a unique perspective as I see both sides.  I see DevOps as the embodiment of Goldratt’s ideas in IT; reducing work-in-progress, reducing cycle-time and always focusing on the most constrained processes.  Add a consistent release schedule and it looks a lot like DevOps.  Everyone has their own ideas about what is DevOps and how best to implement, I suggest we follow Goldratt’s advice and focus our efforts where they will have the most benefit.