devopsy.kinsta.cloud

Tag: the phoenix project

  • The Phoenicorn Project

    The Phoenicorn Project

    The Unicorn Project is the sequel to the ground-breaking book The Phoenix Project. Most people know the story about Bill the IT manager at Parts Unlimited, who is tasked with fixing a derailed IT project or having his IT department outsourced. The Unicorn Project takes it a step further, with developer and architect Maxine in the starring role, helping Parts Unlimited to develop new products in new markets–the riskiest combination of all. This requires gambles that are only justified when the organization can experiment rapidly, break the rules and deviate from the processes that govern their current–but diminishing–business.

    The content of the book revolves around five so-called Ideals:

    1. Locality and simplicity in our code and organization.
    2. Focus, flow and joy in our work.
    3. Enablement of improvement and achievement.
    4. A culture of psychological safety.
    5. A relentless focus on our customers.  

    These are a natural progression of the themes in The Phoenix Project, in particular the first and third of The Three Ways of DevOps: 

    1. Attending to the performance of the entire system, as opposed to the performance of a specific silo of work or department (related to Ideal 2).
    2. Creating short feedback loops, enabling necessary corrections to be continually made.
    3. Creating a culture that fosters continual experimentation, taking risks and learning from failure; and repetition and practice as the prerequisite to mastery (related to Ideal 3).

    Topics related to the first four Ideals have been explored in other books in the same family of publications, in particular The DevOps Handbook. The last Ideal–customer focus–is for me the most significant addition. It really draws attention to how IT and business have to converge and even merge in order to develop new business models. Meaning that those people-previously-known-as-IT have to up their game and develop a much deeper understanding of the jobs customers need to get done. The business value of DevOps has interested me since I got involved in the movement, and I’ve captured some thoughts in a white paper for the DevOps Agile Skills Association.

    It’s another easy-to-read, fast-moving and plausible story that will undoubtedly inspire the DevOps community. So far so good. But as a facilitator of GamingWorks’ The Phoenix Project DevOps game, it got me thinking how I can incorporate the Five Ideals (and more) into this highly-interactive game.

    My first analysis is encouraging. Starting with the last Ideal and working backwards, the game–that is based on the book–has strong focus on the customer while at the same time ensuring that technical debt (another key topic in the book) is kept under control. Psychological safety comes up a lot in the game–when it’s clear that somebody isn’t really participating, there’s a time out to discuss the importance of people feeling safe to share their concerns without fear for their reputation and position. Continual improvement is hard-wired into the game, with retrospectives after each sprint, leading to hypotheses and experiments for the next sprint. As for focus, flow and joy in our work, we encourage the participants to visualize the value stream and see how work flows from role to role. There’s also plenty of fun, particularly when they hit their revenue targets.

    Locality and simplicity in our code and organization is probably the least dominant Ideal in the game–it’s a non-technical game so there’s no coding, and it’s a small team so almost by definition it’s local. We do discuss, however, the challenge of scaling–which is when this Ideal comes into play. I’ll experiment with the Five Ideals in my next The Phoenix Project games so watch out for further blogs.

    The Five Ideals triggered another association. I was lead editor of the upcoming ITIL 4 High velocity IT module. This is about how digitally-enabled organizations place higher demands on IT, requiring different ways of thinking and working. One of my contributions is the concept of five Key Behavior Patterns:

    1. Help get customers’ jobs done–helping customers become who they seek to become.
    2. Keep raising the bar–taking things to a significantly higher level.
    3. Trust and be trusted–as professional knowledge workers in a healthy workplace.
    4. Accept ambiguity and uncertainty–not scared of not knowing an answer.
    5. Commit to continual learning–all as part of their daily work.

    I believe many practitioners want to work in organizations that foster these behaviors. As such, I see them as aspirations. As marketing legend Seth Godin said, “people like us do things like this.” You can see the similarities with the Ideals: 

    • “Help get customers’ jobs done” equates with “A relentless focus on our customer.”
    • “Keep raising the bar” and “Commit to continual learning” are close to “Enablement of improvement and achievement.”
    • “Trust and be trusted” is the same as “A culture of psychological safety.”

    It is so encouraging to see similar thoughts emerging within different communities. This will hopefully break down the silos that exist in the IT industry and bring us all closer together. All too often, conferences and communities are organized by specialization. They stick safely to themselves and reinforce their stereotypical stories about other disciplines. Praise and power to the rebels who practice what they preach (T-shaping and multi-functionality) and liaise with their industry colleagues, organize joint conferences etc. Evolution thrives on diversity and it would be a shame to see inbreeding in communities, leading to extinction. Let’s not only read The Unicorn Project but also do something useful with it.

    — Mark Smalley

  • The Power of a Game: What I Learned from Facilitating The Phoenix Project Simulation

    The Power of a Game: What I Learned from Facilitating The Phoenix Project Simulation

    I recently had the opportunity to become certified to lead a simulation based on the best-selling IT book, “The Phoenix Project.” The Phoenix Project simulation is not a product as much as it is an experience. The realistic scenarios, challenges and pain points embedded in the game built a creative tension in a safe environment and allowed everyone (including me) to discover new insights into applying DevOps to improve flow throughout the value stream—from the business request through IT and to customers.

    DevOps and Agile practices are built on the back of Lean principles that go back decades before the Toyota Production System. As someone who attempted to apply Lean to IT long before DevOps came into existence, I have always said that DevOps is about creating an environment where it is safe to observe, experiment and learn based on outcomes. I was fortunate to meet with Gene Kim many times while he was writing “The Phoenix Project.” We always talked about Lean thinking, Lean systems and Lean tools. Those discussions eventually morphed into The Three Ways—the core principles of DevOps.

    Alignment with the Business Changes the Feel of Work

    The simulation put people in a position where it was easy to see and feel the impact of not being aligned with business and not being coordinated with all the elements of it (AppDev, engineering, IT Ops, change management, etc.). In each round, the team was able to reflect and learn from the experience and experiment with new ways of working. What we all experienced was profound: The DevOps principles, systems and tools are effective only when the team directly experiences the frustration of a broken work system and they work together to solve the problem.

    It was amazing to see how quickly people took on the personas of the functional role they had been assigned. The woman who took on the role of retail operations—you may remember Sarah in “The Phoenix Project”—became rather aggressive as she demanded results. The CISO—John in the book—constantly nagged people about SOX-404 issues. And of course, the CEO was a royal pain to everyone. Although we all knew this was a simulation game, everyone reported the stress and tension they felt as we embarked on round one with no real plan of how we were going to transform into a DevOps team.

    You Can’t Improve What You Can’t See

    Among the chaos of trying to get the work done, it became instantly apparent that the team had no visibility of all the work or priorities. True to life, there was more work that our people could handle so we needed a way to see the work, compare it to our capacity to do the work, identify trade-offs and then work with the business and other stakeholders to set priorities. Once the team was able to accomplish this (through some trial and error) we all could feel the alignment of purpose, see the smiles on people’s faces as we created value and feel what collaboration between silos is really like.

    This discovery was just the tip of the iceberg. As a coach and facilitator, I was able to ask the team questions that positioned them to reflect on what had just happened and see within themselves additional changes that needed to take place to improve the quality, speed and volume of our deployments. Some of the other topics, which were experienced and later expanded on, include:

    • A recipe for coordination among silos.
    • Making time for improvement, learning and technical debt.
    • DevOps without Value Stream Mapping can be hazardous to your health.

    At the end of the day, DevOps is about creating an environment where it is safe to observe, experiment and learn based on outcomes. The Phoenix Project Simulation creates an environment to give everyone that experience. From there, they return to work and begin their journey.

    — Mike Orzen

  • Simulating DevOps to Discover the ‘Culture Thingy’

    Simulating DevOps to Discover the ‘Culture Thingy’

    DevOps … is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably. — Wikipedia

    We are all, I believe, familiar with the fact that culture is an important factor when adopting DevOps.

    Groan!

    Most engineers and technologists are probably tired of having the “C” word thrown at them: culture.

    Because let’s be brutally honest: DevOps value comes from the automation stuff. The latest “State of DevOps” report confirmed “that automation is a key differentiator for organizations.” All the shiny technology that supports the “continuous-es”: continuous integration, continuous deployment, continuous delivery, with infrastructure as code and spinning up servers in the cloud and then collapsing them when no longer needed, with other cool technology concepts such as containers, microservices and a whole array of sexy build, test and monitoring tools. Just when we start getting excited about it and drooling about the new tools we get to play with, somebody else pipes up with that culture. stuff again …

    Groan. Just let us get on and do IT!

    But equally, probably the most important part of the “continuous-es” is continual learning and improvement, because again, let’s be honest: The real value of DevOps takes years to realize. It is not a one-time, overnight migration with a simple press on the ‘Install DevOps” button. It is an iterative exercise involving people from the end-to-end value stream; therefore, it is all about continual experimentation, learning and improvement—and doing it with people from different siloes who don’t usually like talking to each other. So, it’s a “cultural thingy.”

    If it’s not somebody talking about culture and collaboration and communication, it’s one of those “frameworky-geeks” talking about agile, scrum, lean thinking and IT service management (ITSM) and how we need “flow,” “process engineering,” “voice of the customer,” “kaizan,”kanban,” and a whole host of other Japanese-zen-like mysterious terms which only consultants are qualified to talk about and do, as they are the enlightened-ones (probably with their own secret handshake or Doctor Spock type split fingered salute!).

    “DevOps is a philosophical movement,” we hear. Now, if there are two things that I wouldn’t normally associate with each other it is technical engineers (I was one!) and philosophy. Although I do remember trying to stare down a server once, hoping by the power of positive thought and sheer will power for it to start working again; the alternative was a sledgehammer …

    Where was I? Oh, yes. So what are some of these “culture things” we should care about, apart from the “Wall of Confusion” that everybody keeps talking about? “We’re not confused! It’s the other lot on the other side of the wall that don’t get it!”

    Below are some of the key discoveries relating to the culture, or soft skills, issues that teams experience and recognize during the DevOps business simulation, “The Phoenix Project.” Before you start getting all excited again, this is not a computer game in which you can win points by zapping Ops people who get in the way or setting phasers to stun when you see a developer about to release some more untested code or beam up an uppity product owner who tells you that what you coded may have been what they asked for but isn’t what they were thinking. This business simulation is a dynamic, interactive, classroom-based workshop in which people have to get together—face to face—and, well, communicate and collaborate! The simulation workshop has been performed with literally hundreds of DevOps teams across the globe and across different industries, and is used to support DevOps training and to help empower teams. (see a blog posted on this site).

    What are Some of These Key Learning Points and Takeaways?

    I have mapped these discoveries onto the DASA (DevOps and Agile Skills Association) competence model, particularly the “Skills & Behaviors” area.

    DASA has developed the competence model to help organizations assess and develop the right knowledge and skills within their DevOps teams. The competence model is based upon industry feedback from practicing organizations, and helps organizations assess their level in terms of Novice, Competent, Proficient, Expert and Master. It looks at four key Skills & Behavior areas and eight Knowledge areas.

    Skills & Behavior Areas: Courage, Team-building, DevOps Leadership, Continuous Improvement.

    Simulation exercise discoveries for Courage:

    • “Feedback” is often seen as negative. People don’t like or dare to give feedback. It is perceived as personal, as criticism and often as blame. A blame culture makes it even more difficult to create open, honest feedback which is a barrier to learning and improvement.
    • “Fail fast” often means “blame fast,” or “pass the blame.” A blame culture creates a lack of trust and willingness to experiment for “fear of failure.” Often in the simulations if things go wrong, people revert to asking,”Whose fault was that?” rather than asking, “What happened? How can we stop that happening again?”
    • Sharing knowledge. Still in many organizations, knowledge is power; people don’t like to share knowledge or don’t think others are capable of using knowledge. A critical aspect of DevOps is multi-functional teams with broad shared skills and knowledge.

    Simulation exercise discoveries for Team-building:

    • Teams are unaware of the need for active listening, confirmation of agreements and decisions and verifying “assumptions”—many mistakes are the results of “assumicide” and poor understanding, which causes disruptions and excessive hand-off moments to gain the right information resulting in waste, delays and rework.
    • Ownership, responsibility and being proactive. Many IT people wait to be told what to do. Many people don’t like to admit they don’t know or that they need help. others don’t actively seek out and ask, “How can I help,” asking upstream and downstream, “What do you need from me to be able to do your job?”
    • Many teams have an aversion to getting together as they have poor experiences of pointless meetings with lots of talking and then not a lot of decisions being made or action being taken.
    • Many teams don’t set for themselves goals for a meeting or learning objectives from a training session. They do not ask themselves, “What do I need to take away from this session?” They often don’t ask, “What do you expect from me?” “What should my contribution be?” They often cannot answer the question, “What problem do you want to be able to solve after attending this?” Many times in a session people say, “I was told to come.”
    • Silo thinking: Many teams still think from siloes, e.g the technology team they are in or the development team, and don’t think end-to-end. They are building and supporting a technology solution and not managing end-to-end business value.

    Simulation exercise discoveries for DevOps Leadership:

    • When there are managers in the simulation, people often sit back and wait to be told or managers feel they must come up with the solution and give instructions.
    • Managers have difficulty letting go, and sometimes don’t have the patience to let teams learn and improve themselves.
    • Managers fear experimentation and failure, as they may be blamed.
    • Managers often blame teams for not becoming self-steering, becoming self-empowered or for not taking ownership, yet they do not help coach and empower teams.
    • Often managers are unable to answer the “why” question: “Why are we doing DevOps?” If managers do not know then how can they expect team members to know?

    Simulation exercise discoveries for Continuous Improvement:

    • Teams are poor at reflecting, analyzing, thinking and coming up with solutions. They lack skills in “problem thinking,” also in terms of Kolb learning cycle. Often they recognize one symptom and then leap to conclusions and solutions.
    • Lack of skills in facilitating a stand-up meeting. See the five key team skills mentioned by Jan Schilt on devopsy.kinsta.cloud. These skills are: The stand-up; Facilitating a retrospective; Managing the Kanban (Visual Management System); Managing the Andon: (and swarming problems); and Collaboration.
    • Many teams struggle to record issues in their work, such as a CSI (continual service improvement) register. They agree it is a good idea, then nobody does it, or if they do then nobody looks at the list again. If I ask, “Who owns this?“ they usually look at each other the point to the business or IT manager. There is no sense of ownership for improvements. This confirmed our own mini-survey of 650 organizations: Less than 20 percent had formal CSI.
    • With improvements, teams wait for somebody to say, “You will do this,” rather than suggest, “I can pick up that.” Improvements mean more work and teams are already busy enough, allocating time for improvements and rewarding improvements are not valued.
    • Teams often say they do NOT reserve time to reflect and discuss improvements. They are often too busy and managers do not stimulate, or allow them to reserve time for thinking and acting on improvements to the way of working. Priority is given to completing the ever growing backlog of work.

    Knowledge Areas

    Business Value Optimization:

    • Many technology people do not understand the business they are supporting, the critical business activities and business times, how a system delivers value to the business, what happens at a given time if a critical IT feature, function or technology is not available. There is a technology mindset as opposed to a business mindset.
    • Many technology teams have never seen a user or seen how users or customers use and experience the technology. Many think they don’t need to know this. This is a top-scoring ABC (Attitude, Behavior, Culture) card worldwide in global workshops and has been for 15 years. “IT has too little understanding of business impact and priority,” and is also a top concern for business managers.
    • In every single simulation and workshop I conduct worldwide, the issue of prioritization is raised, again in relation to business impact and business value, yet all teams recognize a growing pressure on scarce resources, a dilemma between “innovation” and “maintenance work” and a need for clear prioritization.
    • Product managers in teams are often unaware of technical debt and its impact. ITSM teams have difficulty quantifying the impact of technical debt and how this impacts current and future business plans.

    And finally, the latest “State of DevOps” report also confirmed the importance of skills and behaviors, particularly what it calls “Transformational Leadership”: “By 2020, half of the CIOs who have not transformed their teams’ capabilities will be displaced from their organizations’ digital leadership teams.” The survey continues: “Leaders are the ones who set the tone of the organization, and reinforce the desired cultural norms.” Oh! There is that “C” word again. Are we ever to escape it?

    What are the characteristics of a transformational leader, according to the State of DevOps findings?
    Vision, inspirational communication, intellectual stimulation (challenges people to think about problems in new ways), supportive leadership, personal recognition. These characteristics are one of the reasons why leadership is one of the four key Skills & Behaviors in the DASA competence model.

    Conclusion

    A simulation experience helps make the “C” word become a reality. Delegates can translate it from a buzzword into meaningful behavior. The simulation allows delegates to see, feel and experience the benefits of communication and collaboration, while helping them to translate DevOps theory into practice.

    As a leader, you can use a simulation exercise as an instrument to communicate your vision in an inspirational way. End-to-end stakeholders can be brought together and stimulated to explore and develop DevOps skills, knowledge and competences together. At the end of the simulation experience teams have been empowered to capture takeaways they want to apply.  As a leader, you can support them in translating the captured takeaways into real behavior and helping shape together a DevOps culture for your organization.

    About the Author / Paul Wilkinson

    Paul has been actively involved in the IT industry for more than 35 years. Fulfilling a wide range of roles ranging from computer operator to IT infrastructure operations manager. He was co-author of the ITIL publication, “Planning to Implement IT Service Management,” and was a member of the ITIL advisory group for ITIL Version 3, and the architects team for ITIL Practitioner. Paul is also co-director and owner of GamingWorks, the company that developed the internationally renowned “Apollo 13 – an ITSM case experience” ITSM simulation game and most recently the “Phoenix Project – DevOps business simulation,” which is based upon the Phoenix Project book . He was also co-author and developer of the “ABC of ICT (The Attitude, Behavior and Culture of ICT)” publications, having conducted ABC workshops and simulation workshops with delegates representing more than 4,000 organizations worldwide. Follow him on Twitter or connect with him on LinkedIn.

  • Lean versus the bandwagon, lean wins

    Lean versus the bandwagon, lean wins

    I’ve just finished reading “The Phoenix Project,” by Kim, Behr and Spafford. To be honest, it was on my reading list but had I not picked up a copy at Camp DevOps I don’t know that I would have ever read the book. My main issue with books of this ilk, such as “Who Moved My Cheese” and “The Energy Bus,” is that the story is often contrived for purposes of delivering the learning points. So, I was surprised by how much I was able to relate to the issues in Phoenix giving it an air of reality I was not expecting.

    Phoenix follows the life of a new VP of IT Operations through his early weeks on the job. Plucked from obscurity as the Director of Mid-range Computing after his management chain was seemingly asked to depart the company, Bill’s eyes are opened to a world of heroics and chaos that surrounds him in IT and has had detrimental impact on the company’s financial and competitive performance. For anyone who has logically questioned, “how can that happen?” in their own companies, the authors do a great job of illustrating the paths that lead to the unimaginable outcomes we see every day in our own IT world.

    That said (spoiler coming), I ask the authors, did we really need to make Bill’s dad an alcoholic? Sorry, completely tangential point, but interestingly, one of the few times where I thought the authors had reached into their “drama box” for reasons that are still unclear. Nonetheless, after reading this book, I reflected upon the content that is being created around DevOps movements and realize that there is a polarization effect occurring.

    Camp 1 I categorize as the “lean camp”. This camp represents those that believe DevOps should be the application of lean manufacturing principles on IT. In this perspective IT is represented as a critical hub through which business is enabled and transacted. In accordance with this perspective, work enters the hub through multiple endpoints and must be prioritized and coordinated in order to ensure that a) critical IT resources are sourced appropriately and with maximum value to the business and b) constraints and bottlenecks are minimized, thus allowing work-in-progress to complete successfully. Anyone who has studied any aspect of logistics, manufacturing or supply-chain management understands the number of variables involved in this type of scheduling, but applying this to IT is significantly more challenging since it requires the cooperation at both the 30,000 foot view and the 10 foot view, with the latter required to understand the impact of scheduling conflict and help foster collision avoidance that could lead to outages.

    Camp 2 I categorize as the “bandwagon camp”. This camp represents individuals that are using DevOps as a lever for their careers. Certainly, DevOps is the hot topic of the day and having it on the resume is certainly good for a few extra brownie points and higher salaries. Ask these individuals what DevOps is about and you hear answers, such as, “it’s about changing the culture in IT,” “it’s about getting development and operations to cooperate more effectively,” or my personal favorite, “it’s about automation.” The truth is that DevOps needs aspects of all of these factors but driven under an umbrella of work described in the Camp 1 description above. Acted on independently under the guise of DevOps and we will see DevOps nose dive into Gartner’s trough of disillusionment very quickly.

    Will things get fixed in Camp 2? Probably. Certainly implementing controls around who can make changes and when they can occur may help many businesses limit the unplanned work that is crippling their ability to address other work efforts. However, since this is not the focus of the DevOps effort, unplanned work will continue to come from other areas, it’s just the nature of the beast and without correcting the actions that foster unplanned work, it will continually have a negative impact on the business.

    The authors of Phoenix have done a fantastic job of providing IT leaders with a goal for their DevOps programs. If they are wise, they will rapidly adopt the principles defined therein and make sure that all other so-called DevOps efforts in the company are aligned for purposes of reigning in the chaos and free-wheeling efforts of rogue IT employees. I recently met with a VP of IT for one of the major food distribution companies. I was awed by how effectively his IT organization operates, but now I realize, that his company is primarily a logistics and supply-chain company that happens to move food. It only makes sense that the principles they use to run their business would be adopted for IT as well; especially with their IT leaders coming from other parts of the business. In combination with Phoenix, the meeting locked it for me that the lean principles are key to positioning IT as the business enabler needed to be competitive in the current era.

  • Systems of Things

    Systems of Things

    What do Chevy V8s and DevOps have in common?  They’re both complex systems with many interactions between their moving parts.

    The first time I did minor surgery on a Chevy engine I learned a lot about systems.  The engine was a tired 283 cubic inch V8 in a 1966 Chevy Malibu  which my grandfather purchased new in 1966 and passed it on to my (much older) brother around 1980.  During my brother’s ownership it was driven hard and received a less than flattering paint job.  By the time it was passed to me it ran about as well as it looked, which is it say it was rough.  A buddy of mine who thought he knew something about engines convinced me to replace the valve train with parts from a junkyard.  The results were not good; we only succeeded in making it run worse.

    Interchangeable Parts

     The problem was that valve train components are not interchangeable after they wear-in.  Sure, when they’re brand new the parts are interchangeable.  But after a while the parts wear together and become fitted to their neighbor.  Had I better understood this complex system I wouldn’t have expected old valve-train parts to improve the engines performance.

    In “The Phoenix Project”, we are encouraged to think of DevOps as an interconnected system, in the same way “The Goal” taught us to think about manufacturing processes.  In an interconnected system, the parts have to work together with precision to achieve the goal.  In DevOps, we talk about shortening cycle-time and increasing release frequency, requiring Dev and Ops processes to work together with precision timing.

    Unexpected Results

    How many times have you made a change to something (anything) and afterwards thought to yourself, “That didn’t go as planned”?  I know I have had the experience on more than one occasion.    Taking the time to understand the system is hard, but it pays off over time.  Just as Erik in “The Phoenix Project” reminds us, “…Step 1 is to identify the constraint”.

    Be mindful of the system and seek to understand before suggesting improvements.  Failing fast and fixing twice makes sense when talking about features, but not so much when talking about system availability.

    Getting Better All the Time

    Embrace the system of things… seek to understand the interaction between the moving parts.  Try to be aware of processes that have become fitted to their neighbor and what will happen if just one of those processes is changed.  Avoid the urge to suggest changes without first understanding the system.

    I got better at rebuilding engines as I learned the underlying systems.  Knowing the systems allowed for changes that improved performance.  Understanding the tradeoffs in the system was also important; an engine tuned to develop maximum horsepower at full throttle will run rough at idle.

    Take the time to understand the interconnected systems we are attempting to improve.

  • My DevOps Bookshelf

    My DevOps Bookshelf

    Aside from the many technical manuals on our bookshelves, I bet everyone has a few that are more process or philosophically oriented which hold a special place in our hearts.   To me, these are the ones we continue to go back to and that have helped develop us as engineers, software, operations, quality assurance or otherwise.  In no particular order, the following are the top four, not strictly technical, books in my library.

    Web Operations by John Allspaw and Jesse Robbins

    Allspaw and Robbins cover many topics in this text ranging from monitoring and metrics to databases and operations development relations.  I appreciate this one so much it is required reading material for anyone who joins my operations teams.  In my opinion, it illustrates many best practices for an operations organization.

    Continuous Delivery by Dave Farley and Jez Humble

    This book bridges the gap between good software development practices and getting to continuous delivery.  It made me really think about testing and quality assurance, which in my opinion is one of the most difficult parts of a continuous delivery world to wrap your head around.  While there isn’t one right answer, you should really think about the role testing plays in the overall process.

    The Phoenix Project by Gene Kim, Kevin Behr and George Spafford

    “The Pheonix Project” is a relatively new addition to my library.  While it may sound strange, this book was an absolute page-turner and I really related to it.  It presents many Agile and DevOps concepts via a fictional story that doesn’t seem fictional because we have all been there.  Despite the fact that it is fictional, the concepts presented and techniques are applicable to each of our real-world situations.

    Etsy – Moving Fast at Scale

    While this isn’t actually a book, but a video, to me this is a pivotal piece of information has continued to guide my pursuit of DevOps methodologies and practices, so it holds a special place on my figurative bookshelf.  In the video, the guys at Esty discuss a number of topics including deployment processes, application toggles, metrics and more.  Most importantly, they not only discuss the why but also the how and I find myself frequently referencing this video.

    For me each one of these books or videos has played an important role in pursuit and real-world implementations of DevOps practices.  What are the texts digital or otherwise that you hold dear and/or have influenced you?

  • Don’t Fear, Testing Team! DevOps is here!

    Don’t Fear, Testing Team! DevOps is here!

    I love Gene Kim’s analogy about how all unicorns used to be horses; its  a great visual that I use all the time when encouraging IT organizations to change from thinking “why not” to “why not us” when it comes to starting their DevOps transformation.  If culture is really one of the greatest enemies to faster adoption, then you have to start with where the horses live.

    One of the biggest herd of horses in every organization is the team of people dedicated to testing and validating applications.  Even though these teams are not being included in the conversation, they are in fact the most pivotal part in the transformation.

    A Dangerous Love Triangle

    DevOps is touted as the marriage of development and operations; a match made in heaven.  One small problem though:  in most traditional organizations there is a love triangle between operations, development, and testing, as the flow of bits roll from one team to the other until the application is delivered to production.  With the conversation shifting to how can development and operations work better together the testing teams are being alienated.  The question is where does a testing team fit into this conversation, when the focus is all about bringing development and operations closer together?  How can the testing team assert themselves to help facilitate this transformation?

    Horn Growing Engagement

    The Phoenix Project is a great narrative that introduces robust DevOps concepts.  One cannot go far on devopsy.kinsta.cloud without seeing “The Three Ways” and what they mean.  The first “way” is about getting a really good understanding of the delivery chain and how it works as a whole to deliver applications to the end user.  The team that knows the most about this is the testing team as they are the glue that binds operations and development together. If you are going to start growing a horn, you need to start here.

    Engage with the testing team early and often.  This team knows more about the flow of information from development to production than any other team.  They understand the process of moving applications from development environments, testing them, and delivering them to production. They understand the infrastructure and how it should work and what operations team needs to deploy applications.  This knowledge is the perfect horn-growing fertilizer.

    Performance-Driven Feedback

    In my last post I talked about how performance and change management were two pillars to start building on.  As you start to move from simply understanding the system as a whole and increasing the rate of feedback, performance data is critical.  Remember, in science there is no right or wrong answer – only data that says the outcome is desired or not. Performance data can help take the emotion out of determining whether or not processes are going right or wrong after change is made.  Most of the performance engineering folks are embedded in the testing groups since this type of expertise is needed to drive a successful DevOps transition. However, be careful as you start to utilize these resources as they have the potential to become your biggest constraints.  Protect them and get the mechanics of performance testing and engineering out of their head and into the heads of your development and operations teams.

    Continuous Experimentation

    Once you have the testing team on board and helping in this process an interesting event should occur.  The testing teams will start to embed themselves into Development and Operations.  This is when an environment of experimentation becomes possible, just like in a factory, quality is being tested continuously. This leads to a more combined focus on the task at hand while ensuring the quality of the applications and the speed of delivery.

    So really the phrase should be: DevOps, don’t fear! The testing team is here!

     

  • DevOps and The Goal

    DevOps and The Goal

    Reading, “The Goal” was a pivotal event in my career.  More than a decade ago a friend of mine who was an operations professor at Georgia Tech told me it was his favorite book.  We were discussing the challenges of getting work done in a corporate environment.  I was grumbling about leading a web dev team in a company that did not embrace the software development discipline.  Already a fan of Tom DeMarco and Timothy Lister’s, “Peopleware”, and being very vocal about the lack of ability to focus whether due to environmental distractions or the constantly changing priorities and schedules, I was receptive to the message in “The Goal”.  It gave me a vocabulary and a roadmap for improving our throughput issues.

    Our throughput problems were a textbook case for development disaster.  When I joined the project, we had three source code repositories, from none of which could we build production and when we did produce a build it would work in pre-production environments but not production (this just slightly predates the widespread acceptance of continuous integration).  We also had the classic inconsistent release cycles that would lengthen based upon insistence by the business owners to include more features.  Looking back, it is not surprising in the least that we were not successful.

    Not surprisingly, these issues led to a lack of trust with the business.  Overcoming the technical problems was straightforward using brute-force tactics.  We addressed the source code problem by recruiting a senior developer and empowering him to make improvements.  We overcame the build quality issue by using a primitive form of agile and enforcing knowledge transfer; the operations engineer sat between two developers during the build and release process and he was the only one allowed touch the keyboard.  Our tactics were basic, and they worked.

    The bigger challenge facing us was getting our business owners to accept a different SDLC model.  We were saddled with a classic waterfall process, each phase being dependent upon the last with a lot of idle time built-in at each phase gate.  The shortcomings of the waterfall process were amplified because the business lacked confidence about when, if ever, they would get a feature released to production. This lack of confidence led to a constant rearranging of the priorities to get the most important feature du jour in the next release.  The constantly changing priorities created even more idle-time and a lot of work-in-progress (WIP) which Goldratt’s Theory of Constraints (TOC) tells us is at the root of inefficiency.

    Using TOC as a guide, we redefined our SDLC from requirements through QA.  We focused on developing a process that de-coupled the phases, reduced idle-time and work-in-progress (WIP).  De-coupling the phases meant the work could queue between each phase allowing the teams, (requirements, development, QA, and operations) to vary independently; they didn’t have to wait on the team before or after to complete their work which reduced idle-time.  We then focused on improving the slowest areas, which seemed to be requirements and deployments.

    We convinced our business owners to trial a monthly release cycle (this was a long time ago, even before continuous integration was widely accepted!).  The monthly release cycle helped give the business confidence that they would see their features in production at an expected pace and it allowed requirements to cycle on schedule independent of development and QA.  Some features would get done in a month and some would span several months.  Over a period of three months we got into a rhythm that worked for about 14 months until organizational focus changed and we focused elsewhere.

    Reading “The Phoenix Project” reminded me about those days and the challenges Goldratt helped us overcome.  I really liked “The Phoenix Project” and many of the situations rang true.  I have since moved from development to operations (twice!) which gives me a unique perspective as I see both sides.  I see DevOps as the embodiment of Goldratt’s ideas in IT; reducing work-in-progress, reducing cycle-time and always focusing on the most constrained processes.  Add a consistent release schedule and it looks a lot like DevOps.  Everyone has their own ideas about what is DevOps and how best to implement, I suggest we follow Goldratt’s advice and focus our efforts where they will have the most benefit.