devopsy.kinsta.cloud

Tag: autotrader

  • Systems of Things

    Systems of Things

    What do Chevy V8s and DevOps have in common?  They’re both complex systems with many interactions between their moving parts.

    The first time I did minor surgery on a Chevy engine I learned a lot about systems.  The engine was a tired 283 cubic inch V8 in a 1966 Chevy Malibu  which my grandfather purchased new in 1966 and passed it on to my (much older) brother around 1980.  During my brother’s ownership it was driven hard and received a less than flattering paint job.  By the time it was passed to me it ran about as well as it looked, which is it say it was rough.  A buddy of mine who thought he knew something about engines convinced me to replace the valve train with parts from a junkyard.  The results were not good; we only succeeded in making it run worse.

    Interchangeable Parts

     The problem was that valve train components are not interchangeable after they wear-in.  Sure, when they’re brand new the parts are interchangeable.  But after a while the parts wear together and become fitted to their neighbor.  Had I better understood this complex system I wouldn’t have expected old valve-train parts to improve the engines performance.

    In “The Phoenix Project”, we are encouraged to think of DevOps as an interconnected system, in the same way “The Goal” taught us to think about manufacturing processes.  In an interconnected system, the parts have to work together with precision to achieve the goal.  In DevOps, we talk about shortening cycle-time and increasing release frequency, requiring Dev and Ops processes to work together with precision timing.

    Unexpected Results

    How many times have you made a change to something (anything) and afterwards thought to yourself, “That didn’t go as planned”?  I know I have had the experience on more than one occasion.    Taking the time to understand the system is hard, but it pays off over time.  Just as Erik in “The Phoenix Project” reminds us, “…Step 1 is to identify the constraint”.

    Be mindful of the system and seek to understand before suggesting improvements.  Failing fast and fixing twice makes sense when talking about features, but not so much when talking about system availability.

    Getting Better All the Time

    Embrace the system of things… seek to understand the interaction between the moving parts.  Try to be aware of processes that have become fitted to their neighbor and what will happen if just one of those processes is changed.  Avoid the urge to suggest changes without first understanding the system.

    I got better at rebuilding engines as I learned the underlying systems.  Knowing the systems allowed for changes that improved performance.  Understanding the tradeoffs in the system was also important; an engine tuned to develop maximum horsepower at full throttle will run rough at idle.

    Take the time to understand the interconnected systems we are attempting to improve.

  • DevOps and The Goal

    DevOps and The Goal

    Reading, “The Goal” was a pivotal event in my career.  More than a decade ago a friend of mine who was an operations professor at Georgia Tech told me it was his favorite book.  We were discussing the challenges of getting work done in a corporate environment.  I was grumbling about leading a web dev team in a company that did not embrace the software development discipline.  Already a fan of Tom DeMarco and Timothy Lister’s, “Peopleware”, and being very vocal about the lack of ability to focus whether due to environmental distractions or the constantly changing priorities and schedules, I was receptive to the message in “The Goal”.  It gave me a vocabulary and a roadmap for improving our throughput issues.

    Our throughput problems were a textbook case for development disaster.  When I joined the project, we had three source code repositories, from none of which could we build production and when we did produce a build it would work in pre-production environments but not production (this just slightly predates the widespread acceptance of continuous integration).  We also had the classic inconsistent release cycles that would lengthen based upon insistence by the business owners to include more features.  Looking back, it is not surprising in the least that we were not successful.

    Not surprisingly, these issues led to a lack of trust with the business.  Overcoming the technical problems was straightforward using brute-force tactics.  We addressed the source code problem by recruiting a senior developer and empowering him to make improvements.  We overcame the build quality issue by using a primitive form of agile and enforcing knowledge transfer; the operations engineer sat between two developers during the build and release process and he was the only one allowed touch the keyboard.  Our tactics were basic, and they worked.

    The bigger challenge facing us was getting our business owners to accept a different SDLC model.  We were saddled with a classic waterfall process, each phase being dependent upon the last with a lot of idle time built-in at each phase gate.  The shortcomings of the waterfall process were amplified because the business lacked confidence about when, if ever, they would get a feature released to production. This lack of confidence led to a constant rearranging of the priorities to get the most important feature du jour in the next release.  The constantly changing priorities created even more idle-time and a lot of work-in-progress (WIP) which Goldratt’s Theory of Constraints (TOC) tells us is at the root of inefficiency.

    Using TOC as a guide, we redefined our SDLC from requirements through QA.  We focused on developing a process that de-coupled the phases, reduced idle-time and work-in-progress (WIP).  De-coupling the phases meant the work could queue between each phase allowing the teams, (requirements, development, QA, and operations) to vary independently; they didn’t have to wait on the team before or after to complete their work which reduced idle-time.  We then focused on improving the slowest areas, which seemed to be requirements and deployments.

    We convinced our business owners to trial a monthly release cycle (this was a long time ago, even before continuous integration was widely accepted!).  The monthly release cycle helped give the business confidence that they would see their features in production at an expected pace and it allowed requirements to cycle on schedule independent of development and QA.  Some features would get done in a month and some would span several months.  Over a period of three months we got into a rhythm that worked for about 14 months until organizational focus changed and we focused elsewhere.

    Reading “The Phoenix Project” reminded me about those days and the challenges Goldratt helped us overcome.  I really liked “The Phoenix Project” and many of the situations rang true.  I have since moved from development to operations (twice!) which gives me a unique perspective as I see both sides.  I see DevOps as the embodiment of Goldratt’s ideas in IT; reducing work-in-progress, reducing cycle-time and always focusing on the most constrained processes.  Add a consistent release schedule and it looks a lot like DevOps.  Everyone has their own ideas about what is DevOps and how best to implement, I suggest we follow Goldratt’s advice and focus our efforts where they will have the most benefit.