devopsy.kinsta.cloud

Tag: phoenix project

  • DevOps: Assessing Behavior Flows Through an Organization

    DevOps: Assessing Behavior Flows Through an Organization

    I’ll admit it, this article is a little strange. It is an experiment. But DevOps is all about experimentation, right? This article explores how information and behaviors in fact “flow” through an organization. It is based upon my observations with literally hundreds of DevOps teams. I have used the Three Ways of DevOps to show how and why many organizations struggle with this flow of information and behaviors, and what you can do to identify and prevent behavioral “defects” in your organization.

    Simple, Really: Mindsets and Behaviors

    IT organizations embarking upon Agile or DevOps ways of working are basically attempting to realize behavior change—simple. Consistent, new “desirable behaviors” that deliver value and reduce risks, wasted effort and costs—simple. This sustainable behavior change demands a change in mindsets and beliefs; it also means unlearning old ways of behaving—not so simple. This requires feedback and correction.

    Start with the ‘Why’

    Changing mindsets relies on the right messaging. Understanding the “why” and the “purpose” is seen as a critical factor in changing mindsets and ultimately behaviors. The right messaging is not just in terms of words, but also in terms of behaviors, such as people “walking the talk” and “leading by example.”

    Desirable behaviors are about people doing the right things. Doing the right thing implies making the right decisions; for example, prioritization of work. To be able to make the right decisions requires having the right information, such as mentioned above—the why. Knowing and understanding the why helps underpin decision-making and prioritization. However, we see many teams that do not fully understand the why behind new ways of working. Somehow the message did not arrive. Somewhere in the flow of information the “why as a justification” ended up as the “why as a question.”

    This made me realize that information and behavior flows through an organization. I decided to look at many of the challenges organizations face when adopting new ways of working, by using the Three Ways of DevOps.

    The First Way: Flow

    The First Way of DevOps is all about the flow from left to right, understanding the flow of work, seeking to increase the flow and never passing a defect downstream.

    Understanding the Flow

    The messaging I mentioned above flows through the organization traditionally top-down and bottom-up; with the shift to Agile ways of working, this flow is through the end-to-end value stream. This flow is upstream and downstream: Downstream is the flow and hand-offs; upstream is feedback.

    I mentioned above that desirable behavior is related to making decisions. If we consider the traditional decision-making authorities in an organization, we have from left to right: strategic, tactical and operational decision-making. This can be seen, for example, as the CIO and business management team (strategic decisions), e.g., product portfolio and key goals; product owners and IT delivery managers (tactical decisions), e.g., product backlog, IT changes; teams and employees (operational decisions), e.g., sprint planning, incident prioritization.

    The importance of decision-making was also confirmed in a recent Forbes article, “The Netflix Decision Making Model Is Why They Are So Successful.” Netflix’s desirable behaviors are driven by a set of corporate values. The first value is judgment, which “outlines the company’s decision-making ways and accompanying expected behaviors.  Notably, Netflix encourages its employees to ‘make wise decisions despite ambiguity’.”

    Every so often, it is decided to embark upon an Agile transformation at a strategic level. This decision is communicated throughout the decision-making levels as an intent—usually the why is vague and left open to interpretation. A set of values or principles are also communicated, such as customer-centric, end-to-end responsibilities and continuous improvement. However, these usually are not translated into desired behaviors. An implied set of behaviors are linked to these values and principles, and can end up being inconsistent with the original intention. This miscommunication of the why and the desired behaviors downstream is a form of a defect—often causing assumptions, driving poor decisions and resulting in undesired behaviors.

    For example, as a frustrated senior level manager complains that teams won’t take ownership and frustrated teams protest about lack of empowerment, desired behaviors were not defined and agreed. As a result, they were not displayed, waste occurred, frustration, delays, lack of return-on-value from the effort and behaviors spent.

    Another concept of flow is constraints—constraints stopping or slowing down the flow. In terms of behaviors, a constraint could be attitudes and beliefs (not understanding or believing in the new behavior), lack of skills or ability to perform the behavior and allocation of time to learn and experiment or improve behaviors. So, not only must we ensure that we do not pass defects downstream, we also must ensure feedback can identify constraints, impediments and barriers that are stopping the right behaviors from flowing smoothly through the end-to-end stream.

    Other types of constraints could be ignorance, indifference, confusion, fear, uncertainty, doubt—or, as Mike Orzen said in a discussion, focus, attention and trust.

    So, on one side we have:

    • Intent, implied, assumptions and inconsistent behaviors flowing from left to right (cause), meeting….
    • Indifference, confusion, fear, uncertainty, doubt, lack of trust etc. (effect)—you get the picture.
    • A management solution to this is then to tell people to collaborate, or to tell people they are now empowered—again, without defining the associated behaviors.

    “… Application delivery is a set of very human-oriented tasks, a key factor that determines the speed to value is trust,” wrote Carmen DeArdo in his latest book, “A Leader’s guide to Digital Transformation,” co-authored by Jack Maher. “As teams lose trust in the validity of the work as it flows through the lifecycle, a large amount of rework and waste is introduced.” he wrote. Trust is probably one of the biggest enablers or barriers to the smooth flow of behaviors through the system.

    The Second Way: Feedback

    The Second Way of DevOps talks about shortening and amplifying feedback right to left so that corrections can be made. One form of this is giving feedback when poor information is passed downstream or undesirable behaviors are displayed, enabling us to correct the messaging and the behaviors. Inconsistent messaging, in the form of managers giving conflicting signals or failing to “walk the talk,” are forms of defects. A commonly chosen ABC card (Attitude, Behavior, Culture) in global workshops is, “Don’t do as I do … do as I say.” However, if there is a culture of blame or a lack of safety associated with giving feedback to correct defects in behavior, then feedback isn’t given. Undesirable behaviors and inconsistent messaging goes unchecked.

    Failing to stimulate and act on feedback is another form of defect. This means undesirable behaviors continue to occur. All these behavioral defects create cultural and behavioral debt. As this debt accumulates, the ability to deliver value diminishes. In response to this, managers often revert to old management and control styles of behavior (cause), creating more resistance and reducing the willingness to give feedback or experiment (effect). Again, examples of passing defects downstream and creating more barriers and impediments to the flow of desired behaviors.

    As suggested in DeArdo’s book, “Feedback should be treated as though it were the most precious of our resources, because it is.”

    The Third Way: Experimentation and Learning

    The Third Way of DevOps is about continuous experimentation and learning—recognizing that repetition and practice are the prerequisites to mastery.

    Agile ways of working are still relatively new. DevOps is still evolving. Old management styles and models are no longer fit for purpose in this new world. Leadership in this new world is also characterized by experimentation. As one CIO stated, “Leaders are expected to walk the talk, but we don’t know what that looks like in this new model.” Leaders often fall back to an S1 leadership style—which is in conflict with the culture associated with Agile ways of working. Poor leadership styles and behaviors, as mentioned above, are also a form of defect—more behavioral debt.

    Feedback becomes even more important to validate experimental approaches and to learn and improve. However, the Third Way also stresses the need to allocate time for the improvement of daily work. We sometimes see impediments on team backlogs, but we don’t see behavioral or cultural debt as a backlog item. We don’t see this on the backlog of management teams’ behavioral debt, either. Teams often are frustrated with a lack of time reserved for technical debt, impediments or improvement work—let alone behavioral debt. As Orzen stated, “Even more important than daily work is the improvement of daily work.” Continual learning and improvement should be core skills and behaviors.

    As stated in the Third Way, repetition and practice are prerequisites to mastery. There is nothing new about this. It isn’t that the DevOps movement came up with this “new” philosophy: Aristotle, around 300 B.C., stated, “We are what we repeatedly do. Excellence, then, is not an act but a habit.”

    We need to practice and repeat behaviors until they become habits, or culture—like “the way we do things here.” Practice also requires feedback to validate improvement gains as a result of practice.

    Top athletes practice with the guidance and feedback from coaches. Scrum and Agile coaches are often assigned to Agile and DevOps teams as they practice and grow. These coaches are skilled in training on Agile and DevOps tools, such as standups, retrospectives, value stream mapping, Kanban and Kaizan. But what we see globally is that not all coaches have behavioral management skills.

    Another form of feedback from practice is measurement. In terms of DevOps, teams measure metrics such as number of releases—successful releases. We don’t often measure the behavioral change. We can measure the frequency and decisions from standups, but can we measure giving feedback, confronting undesirable behavior? One small example I use is measuring the amount of “yehbuts”—yes, but!—in a dialogue, or worse still, “yehbuts” before somebody has even finished talking. This is an example of a communication defect. Another example is assuming that we understand what somebody has told us without confirming if what we believe we heard was the actual intention.

    Communication defects are a major barrier to successful flow of information. Another measurement needs to be related to the impact of behavior change, such as improved customer and employee satisfaction and less rework because of assumptions.  Although measuring the actual impact is difficult, having the dialogue itself is valuable: How is what we are doing adding or reducing value to the team and to the organization?

    Conclusion

    If we want desirable, consistent behaviors focused on creating high-performing teams, we need to look at how messaging and behaviors flow through the system and recognize defects we are passing downstream. Our vague intentions need to be quantified in terms of desirable behaviors. We need to build in feedback mechanisms to signal defects and undesirable behaviors. We need to identify the constraints and impediments that may be causing undesirable behaviors such as fear, uncertainty, doubt—behaviors that may be slowing down or stopping our transformation to new ways of working. We need to recognize that experimentation and practice is needed—knowing that we will struggle and fail, but ensuring we learn and improve as an end-to-end capability. Leadership skills, styles and behaviors need to be aligned to facilitating the three ways of behavior flow. Finally, remember there is really nothing new about this.

    Case

    In a Phoenix Project simulation, after recognizing the above defects, a team posted the strategic portfolio and goals on the visual management board. The CFO explained these goals and why each backlog item was prioritized against value creation (benefits and value to be realized)—an example of clarity of the why. At the same time, the CFO displayed the right behaviors—being at the standup to explain to the team and ask for direct feedback and understanding.

    At the same standup, the CISO and IT team explained the potential “value leakage” backlog items in terms of debts, risks and defects, so that together—business and IT—could make the right decisions on prioritization.

    “This made me realize,” said the product owner, “I was now able to make a better-informed decision, because of the team’s input on value leakage—giving feedback from the usage of the applications and the negative impact. It also gave me more trust in the team as they demonstrated an understanding of the ‘why’ and had a clear understanding of value. If, as a team, we did not fully know or understand the value, this created a great dialogue to explore it and ensure we didn’t pass a ‘defect’ (wrong decision on prioritization downstream).”

    The team in the simulation learned that as well as visualization of impediments on their backlog, they also needed to visualize the behavioral defects that needed work on and allocated time.

    — Paul Wilkinson

  • DevOps: Using Games to Make it Practical

    DevOps: Using Games to Make it Practical

    This blog stems from my experience of being one of the earliest DevOps Foundation trainers of DOI and successfully completing 19 batches of DOI Foundation course in the last year.

    In retrospect of the year starting when TaUB Solutions signed up as a registered education partner (REP) of DOI, my opinion is this has been the best service portfolio offering for our customers and partners.

    There are few things that appeal best when it comes to DOI Foundation course and I can summarize them with the collective feedback I’ve received so far:

    “Challenges the preconceived myths and practices around DevOps with practitioner stories.”

    “Gives a holistic overview of philosophy behind DevOps. Philosophy should precede science and science should precede engineering.”

    “This course maintains the discipline to be truly advisory in nature by illustrating tool chain without recommending vendor-specific tools.”

    “The group exercises during the class and the facilitator practical insights helped us to put the aspects in to our organizational context of delivering products and services.”

    “We have never seen a course such beautifully designed that could be the first basic building block to know ‘Do’s’ and ‘Don’t’s’ of DevOps across application developers, infrastructure operations, project managers, infosec professionals and business leaders.”

    To enhance the overall learning curve of the audience, we put them in to the third day of experiential learning using the Phoenix Project Business simulation. The simulation was divided into four rounds and played with maximum of 12 people per batch. The Team was entrusted to achieve a target of $300,000 within four rounds and take the share value from $21 to $45.

    While the target milestones were clearly laid down at the beginning of game it was difficult for the teams to get hold of the situation due to various unknown parameters. It took them quite sometime to understand the different moving parts of flow, dependencies, business impact of projects, issues and features that came up in every round and make  appropriate choices and decisions.

    The teams experienced significant learning, sharing and collaboration over the four rounds and had loads of fun. One common feedback from the teams was for the game to be mandatory after every two-day DevOps Foundation course.

    The consolidated feedback after playing the DevOps business simulation after the two-day course:

    “It demonstrated that it is not easy to build collaboration, communication and integration. It takes time and there is a lot of patience and team work needed to make this a reality. The DevOps culture had to be nurtured through self-organizing teams with mutual respect, sense of pride and collective ownership to work toward a common goal.”

    “It gave a working knowledge on how ITSM and DevOps are complementing each other [Ex. Change Management and Improved Flow].”

    “The Principles of DevOps, namely the First Way (Improve Flow from left to Right), Second Way (Shorten Feedback Loops) and Third Way (Repetition through Mastery) was visibly evident over the course of four rounds.”

    “The speed of execution improved over the rounds using short feedback loops, continual improvement and applying lessons learned from previous rounds.”

    “It gave a good understanding of how different roles and responsibilities in the organization contribute to the overall success of the project.”

    Finally, the teams made over and above the required target of revenue and stock price. The winning moment was enhanced by the significant learning opportunity and ability to bond with diversified cross functional teams.

    The experience of playing the business simulation reinforced practical understanding of DevOps and also transferred skills that they could implement immediately at their workplace.

    About the Author / Suresh GP

    sureshSuresh GP is the Managing Director of TaUB solutions and Regional leader of BRMI (APAC). He has more than 15 years of experience in IT Service Management, IT Governance, Program Management and Organizational Change Management. He is regular blogger and International speaker at itSMF UK, itSMF Australia, itSMF USA, itSMF Finland and itSMF Norway. He is a Certified DevOps, Agile Service Manager, Business Relationship Management Professional, PMP, CGEIT, ITIL V2 Service Manager, ITIL V3 Expert,  ISO 20000 practitioner. He was awarded the ITSM contributor of the year by itSMF Singapore in 2013.

  • DevOps is a Symptom, Not a Diagnosis, and Continuous Delivery is the Cure

    DevOps is a Symptom, Not a Diagnosis, and Continuous Delivery is the Cure

    You wake up and you have a sore throat and nasal congestion. Is it the common cold? Perhaps the flu? Is it bacterial? Viral? Any good physician will tell you that diagnosing disease purely from symptoms is a fool’s errand (or maybe malpractice). A diagnosis identifies disease which leads to prescribing a treatment plan for eliminating or placing in suspension the symptoms. In a similar vein, DevOps is not a diagnosis either. DevOps is merely the symptom(s).

    Initially, DevOps was the recognition that development and operations were very often not aligned to a common goal of satisfying the business customer. Over time DevOps became representative of other symptoms of IT dysfunction, such as configuration drift, poor release management controls, lack of testing, high numbers of defects during development, and a culture of finger pointing when systems failed. Again, these are just more symptoms of the same diagnosis, which is application delivery dysfunction.

    This realization that DevOps is a symptom has been a long journey for me dating back to July of 2011. At first, I viewed DevOps as the modern day version of the Haymarket Riots. It was a way for the overburdened and under-appreciated efforts of the IT workers responsible for delivering and maintaining today’s systems to take control and bring some stability to their jobs. From there it transformed into the movement focused on dealing with the technical and organizational issues that impact application delivery.

    Today, DevOps seems to be a catch-all for so many facets of application delivery that it’s quickly becoming a useless term. When a sales executive calls me and says a client is interested in DevOps, I respond by asking for more details regarding their interest areas, such as continuous integration, release management, continuous testing, etc. To help them understand the comprehensive nature of their request, I put together a checklist of items for them to review with the client.

    Perhaps, even more frustrating than the uselessness of the term DevOps in deciphering need, are the conversations with people that say, “we’re doing DevOps,” but refuse to drill down into what they mean. It’s very reminiscent of conversations where businesses state, “we’re doing cloud already.” However, if we look at DevOps as a symptom, and you say you’re doing DevOps, I’d look at you and say, “I bet you are.” After all, many businesses are struggling with application delivery. And as your IT doctor, the real fun of giving you a checkup and identifying the causes of your application delivery dysfunction diagnosis can then begin.

    Ultimately, I came to the realization, DevOps doesn’t solve anything (or for some it solves everything, either way its lacking focus) , which led me to the realization that DevOps is the canary in the coal mine highlighting the need for change in the way we build, release and operate applications.

    So, if DevOps represents the symptoms and application delivery dysfunction is the diagnosis, what is the treatment plan? Continuous Delivery. Continuous delivery is to the application delivery pipeline what enterprise architecture is to all systems in the business. It’s a holistic approach to enabling the business to leverage its IT assets to operate better and innovate faster. Achieving maturity in continuous delivery doesn’t mean you’ve fixed every issue limiting application delivery, but it does imply that business has taken an honest look at it’s bottlenecks and constraints and are removing them when possible thereby constantly improving.

    Interestingly, if you accept this premise, the connection to lean manufacturing often associated with DevOps, and illustrated by The Phoenix Project, makes perfect sense. That is, as we seek to improve the continuous delivery pipeline we can use tools and practices from lean manufacturing, such as value stream mapping and Kanban designed for this type of problem. Following on our analogy, lean tools and practices become the pharmaceuticals supporting our treatment plan (continuous delivery) eliminating our ailment (DevOps).

  • Five Great Books on DevOps

    Five Great Books on DevOps

    While the DevOps movement began in earnest in 2009, there are still a lot of questions about what DevOps means, how it’s done, and what the purported benefits of DevOps are. Big questions, for sure. And there are no quick answers. In fact, to thoroughly answer each would require numerous books.

    Which brings us to the question, what are the best books that define DevOps, describe how to operationalize it in the real world, and show what beneficial outcomes one could expect?

    Here are my choices of five great books on DevOps, in no particular order:

    Book: The Phoenix Project

    Authors: Gene Kim, Kevin Behr, George Spafford

    This one is a fictional story, written in engaging novel style, that tells the story of how to apply DevOps to unleash IT’s potential. The story is about Bill, the IT manager at the fictional company Parts Unlimited. Not an exciting business name, but the adventure starts in earnest when on a drive to work, Bill gets the call from the CEO that sets the story in motion.

    The book’s cover description tells it best:

    The company’s new IT initiative, code named Phoenix Project, is critical to the future of Parts Unlimited, but the project is massively over budget and very late. The CEO wants Bill to report directly to him and fix the mess in ninety days or else Bill’s entire department will be outsourced.

    With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

    In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again. 

    As one Amazon reviewer put it: “This “parable” of how applying DevOps to pull out of an IT death spiral of delays and defects is an enjoyable read backed with great references and support materials for understanding the principles underlying DevOps. It’s not enough to get you all the way there, but it will get you inspired to start down the path.”

    Book: Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

    Authors: Jez Humble and David Farley

    This seminal read is all about putting into practice rapidly and consistently delivering new highly-functional software to your users. From the back cover: “Through automation of the build, deployment, and testing process, and improved collaboration between developers, testers, and operations, delivery teams can get changes released in a matter of hours—sometimes even minutes-no matter what the size of a project or the complexity of its code base.”

    The book covers automated infrastructure management and data migration and the use of virtualization, and automating creating, integrating, testing, and deploying software, as well as improvement collaboration within the teams.

    As one Amazon reviewer put it, “This is an absolute must read for anyone in software or application development. It is amazing how well the authors capture the real pain points and provide all the information you need to getting to a more effective IT practice.”

    Book: The Visible Ops Handbook

    Authors: Kevin Behr, Gene Kim and George Spafford, IT Process Institute

    This is the first book I ever read on IT process improvement, and it was an eye-opener for me. This book came out about 4 years before the term DevOps was coined, but it nailed the essence of what high-performance IT is all about. And it’s an essential read today:

    Visible Ops codifies how these [High performance enterprise] organizations achieved their transformation from good to great, showing how interested organizations can replicate the key processes of these high-performing organizations in just four steps:

    1. Stabilize Patient, Modify First Response – Almost 80% of outages are self-inflicted. The first step is to control risky changes and reduce MTTR by addressing how changes are managed and how problems are resolved.
    1. Catch and Release, Find Fragile Artifacts – Often, infrastructure exists that cannot be repeatedly replicated. In this step, we inventory assets, configurations and services, to identify those with the lowest change success rates, highest MTTR and highest business downtime costs.
    1. Establish Repeatable Build Library – The highest return on investment is implementing effective release management processes. This step creates repeatable builds for the most critical assets and services, to make it “cheaper to rebuild than to repair.”
    1. Enable Continuous Improvement – The previous steps have progressively built a closed-loop between the Release, Control and Resolution processes. This step implements metrics to allow continuous improvement of all of these process areas, to best ensure that business objectives are met.

    As this Amazon reviewer this had to say, this fall, on the book: “This is the first and foremost required reading on my team! If you are an IT professional, you must read this. It is a concise, powerful expression of how to reinvigorate your operation and become a top performing IT organization.”

    Book: DevOps for Developers

    Author: Michael Hüttermann

    This book is for software developers and engineers that aim to help improve the delivery of software by using lean practices to get apps and software into production.

    From its description:

    DevOps for Developers is for motivated software engineers, particularly programmers, testers, QA, system admins, database admins, both beginners and experts, who want to improve their software delivery process. It’s the perfect choice for engineers who want to go the next step by integrating their approaches for development and delivery of software. This book is for engineers who want to shape their processes and decide on and integrate open source tools and seek for guidance how to integrate standard tools in advanced real world use cases.

    And as an Amazon reviewer put it: “Developers tend to produce updates in short cycles whilst admins prefer a stable system. And if something bad happens, they often start finger-pointing to blame the other side. One solution is to tear down the silos of development and operation. In other words, set up integrated teams.

    This is the core message which Michael Hüttermann would like to stress out. Agile methods and tools might be used for development as well as for deployment and operation. In his book, Michael explains how to do and which (free) tools to use.”

    To be fair, the book has taken dings about not being a programming book, and repeating some of its concepts. But the value in this book is its real-life examples, and explanations as to how different roles on teams in Dev and Ops can work together more collaboratively. It also dives into common pitfalls organizations hit in their journeys of operationalizing DevOps.

    Book: The Goal: A Process of Ongoing Improvement

    Authors: Eliyahu M. Goldratt, Jeff Cox Author, David Whitford (Other Contributor)

    The Goal is the second fictional book in this list. Written more than 30 years ago, it could have been written yesterday. When discussing concepts and systems, many times story is the way to go because the motivations and thought processes of different roles can be discussed in depth and elegantly so. The Goal succeeds in this, and is a must read for anyone who seeks to help be a catalyst of improvement in their organization.

    From the book description:

    One of Eli Goldratt’s convictions was that the goal of an individual or an organization should not be defined in absolute terms. A good definition of a goal is one that sets us on a path of ongoing improvement.

    Pursuing such a goal necessitates more than one breakthrough. In fact it requires many. To be in a position to identify these breakthroughs we should have a deep understanding of the underlying rules of our environment. Twenty-five years after writing The Goal, Dr. Goldratt wrote Standing on the Shoulders of Giants. In this article he provided the underlying rules of operations.

    This article appears at the end of this book.

    Here’s what an Amazon reviewer had to say: “I have often heard myself complaining that companies over complicate the simplest of activities. Measurements do not truly measure the end result and are often held in a high regard by executives. These measurements cause further loss and mask the true issues. This book was fascinating, very “simple” in a sense which lends to making it even more fascinating. I learned a lot about the production aspect of my “department” which would often not be looked as a production department in the literal sense of the word. Thought provoking and inspirational, every leader can benefit from the story told. A must read.”

    I would love to hear from you, what are the most important books on DevOps that you’ve read?

  • Why “Enterprise DevOps” Doesn’t Make Sense

    Why “Enterprise DevOps” Doesn’t Make Sense

    Recently, some folks have been suggesting that we need a new form of DevOps, something called “Enterprise DevOps” or “Big DevOps.”

    DevOps is great, the argument goes, but large enterprises don’t operate the same way as the agile web startups where DevOps was birthed, and thus they can’t implement the same DevOps methodologies as those startups. Large enterprises have bigger teams, more inherent operational complexity, and greater governance controls (both self-imposed and externally-imposed); therefore, they need a different type of DevOps that caters to their sensibilities and not those of agile web startups.

    While it’s true that large enterprises do operate in a different environment than agile web startups, none of that makes a difference from the point of view of adopting DevOps.

    In short, if you’re advocating for Enterprise DevOps as something different than Plain Ol’ DevOps, then you’re missing the fundamental essence of DevOps.

    Here’s why.

    Early Thinking: DevOps = Tools + Culture

    To best understand DevOps, we need to understand where it came from.

    DevOps started as a bottom-up collection of best practices with few formal underpinnings. System administrators and developers were looking at the successes of agile development practices and trying to apply some of the same ideas to the whole development-to-operations process. Practitioners in the trenches stepped forward and said, “This is what we’re doing and it seems to work for us.”

    One of the earliest and most visible examples of that was John Allspaw’s and Paul Hammond’s presentation at Velocity in 2009, titled “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr.”

    [youtube http://www.youtube.com/watch?v=LdOe18KhtT4]The video is well worth your time, as is the presentation on Slideshare:

    [slideshare id=1628368&doc=allspawhammondvelocity2009-090623161942-phpapp01]

    Note that this presentation predates the actual creation of the name “DevOps.” Allspaw and Hammond talk about how Dev and Ops teams can work productively together, but they never say “DevOps.” Patrick DeBois would coin the term later that year when he created the first DevOpsDays.

    In the presentation, Allspaw and Hammond discussed the success they were able to achieve at Flickr and provided a set of recommendations that they think are important, separated into two buckets:

    1. Tools
    2. Culture

    In the Tools category, they provided six recommendations:

    1. Automated infrastructure
    2. Shared version control
    3. One step build and deploy
    4. Feature flags
    5. Shared metrics
    6. IRC and IM robots

    Along the way, they mentioned that Flickr used:

    • Chef, Puppet, CFEngine, and other automation tools
    • SVN for version control
    • Hudson for builds
    • Ganglia for monitoring

    Finally, in the Culture category, they listed four additional recommendations:

    1. Respect
    2. Trust
    3. Healthy attitude about failure
    4. Avoiding blame

    Unfortunately, most people missed Hammond’s remark ahead of the Tools section of the presentation (timecode 7:40):

    These are some of the tools that work for us. They are not necessarily going to work for everyone, and throughout we’re going to give examples of the specific tools that we use, but the important thing that we’re trying to get across is the concepts that are important.

    In the years following, many self-proclaimed “DevOps engineers” (a bit of an nonsensical title if you understand that DevOps is about bringing Dev and Ops engineers together, not creating new types of engineers focused on nothing but DevOps) insisted that if you were automating your releases using Jenkins and Chef and treating the other members of your team with respect, you were “doing DevOps.”

    While Allspaw’s and Hammond’s presentation was excellent (seriously, spend the time watching it) and the advice was practical and valuable, many people walked away with only a vague notion of what would later be called DevOps.

    For instance, in the “One step build and deploy” recommendation, Allspaw and Hammond describe their automated system that moves code into production. Developers can click a single button to make a build live. The button is jokingly labeled “I’m feeling lucky” (in homage to Google, I’m sure).

    If you’re an agile web startup, that’s par for the course.

    But if you’re a large financial services company, you’re likely to think, “No! Our governance folks will never let developers push code straight to production with a single click, never mind the ‘I’m feeling lucky’ labeling. No way. Not gonna happen.”

    When people suggest that we need “Enterprise DevOps,” they’re typically reacting negatively to the choices that others have made. They recognize that those solutions won’t work for them for any number of reasons, and they say, “I can’t use that. I need something different.”

    But here’s the thing — DevOps isn’t about the specific implementation choices. Those are all optional. You can make a completely different set of choices than Allspaw and Hammond and still be “doing DevOps.”

    Unfortunately, subsequent conferences were filled with DevOps presentations that just reiterated the “DevOps = Tools + Culture” mantra and provided similar sets of recommendations that were often drawn from agile startups rather than from large enterprises.

    So, does that mean Tools and Culture aren’t important for DevOps?

    No, not at all. Both Tools and Culture are critical for DevOps in the long run.

    The problem that DevOps seeks to solve, however, is not a Tools problem or a Culture problem but rather something more fundamental.

    In order to understand DevOps, you need to focus on the ultimate goal.

    The Firm Foundation: The Toyota Production System and Lean Manufacturing

    The Phoenix ProjectIn 2013, Gene Kim, Kevin Behr, and George Spafford wrote The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.

    The trio’s contribution to the DevOps movement was in recognizing that enterprises had confronted a similar set of issues previously, but in a different domain — manufacturing — and that similar solutions could be applied to software engineering and application operations. (The group took another cue from the manufacturing world, copying the novelization technique from Goldratt’s The Goal to create a fictional case study where the characters were allowed to wrestle with the fundamental DevOps truths being discussed.)

    The key takeaway from The Phoenix Project is that DevOps seeks to create a smooth and increasingly speedy flow of change from development into production, with as little waste along the way as possible.

    The GoalThe models here were the Toyota Production System (TPS), Lean Manufacturing, and Goldratt’s Theory of Constraints.

    In the world of manufacturing, we have a physical product that moves through a factory, either on a formal assembly line, or from processing step to processing step, as part of a value stream. Each step along the path is supposed to “add value” to the product until it completes its journey and exits the factory to be shipped to customers.

    If you’re a manufacturing engineer, you need to do everything possible to create a smooth flow through the value stream by identifying constraints in the system and eliminating waste of time or resources.

    In the software world, application artifacts and changes move through the software development and operations value stream, with each step designed to “add value” and get the changes into production.

    If you’re a development or ops engineer, you need to do everything possible to smooth the flow through to production. The techniques you use are similar to a manufacturing engineer — you identify constraints in the system and go about eliminating waste of time and resources.

    The Kata is the Key

    Following Toyota’s success, the TPS and Lean Manufacturing were all the rage in the manufacturing world.

    Consequently, many western manufacturing engineers studied Toyota’s systems in great detail. They noticed a variety of tools and solutions that Toyota used to optimize manufacturing flow, and they recommended their use to their employers and clients. Toyota’s Kanban, for instance, was heavily promoted in the manufacturing world, and later made the jump to software engineering.

    Interestingly, the manufacturing world has suffered much of the same confusion that’s now happening in the DevOps world. When they tried to employ the tools and techniques they had learned from studying Toyota, things didn’t work as expected.

    Toyota KataMike Rother, author of Toyota Kata: Managing People for Improvement, Adaptiveness, and Superior Results, described the situation in the manufacturing world this way:

    Looking back, we naturally put Toyota’s visible tools in focus first. That is where we started— the “door” through which we entered the Toyota topic. It was a step in the learning process (which will also, of course, continue after this book). Since then I went back to the research lab–several factories–to experiment further, and present what I learned in this book. The visible elements, tools, techniques, and even the principles of Toyota’s production system have been benchmarked and described many times in great detail. But just copying these visible elements does not seem to work. Why? What is missing?

    Later, Rother continues:

    Toyota opens its factory doors to us again and again, but I imagine Toyota’s leaders may also be shaking their heads and thinking, “Sure, come have a look. But why are you so interested in the solutions we develop for our specific problems? Why do you never study how we go about developing those solutions?” Since the future lies beyond what we can see, the solutions we employ today may not continue to be effective. The competitive advantage of an organization lies not so much in the solutions themselves—whether lean techniques, today’s profitable product, or any other— but in the ability of the organization to understand conditions and create fitting, smart solutions.

    Focusing on solutions does not make an organization adaptive. For example, several years ago a friend of mine visited a Toyota factory in Japan and observed that parts were presented to production-line operators in “flow racks.” Wherever possible the different part configurations for different vehicle types were all in the flow racks. This way an operator could simply pick the appropriate part to fit the particular vehicle passing down the assembly line in front of him or her, which allows mixed-model assembly without the necessity of changing parts in the racks. Many of us have been copying this idea for several years now.

    When my friend recently returned to the same factory, he found that many of the flow racks along that Toyota assembly line were gone and had been replaced with a different approach. Many of the parts for a vehicle are now put into a “kit” that travels along with the vehicle as it moves down the assembly line. When the vehicle is in an operator’s workstation, the operator only sees those parts , and she always reaches to the same position to get the part.

    My friend was a little upset and asked his Toyota hosts, “So tell me, what is the right approach? Which is better , flow racks or kitting?” The Toyota hosts did not understand his question, and their response was, “When you were in our factory a few years ago we produced four different models on this assembly line. Today we produce eight different models on the same line, and keeping all those different part variations in the flow racks was no longer workable. Besides, we try to keep moving closer to a one-by-one flow. Whenever you visit us, you are simply looking at a solution we developed for a particular situation at a particular point in time.”

    As we conducted benchmarking studies in the 1980’s and 90’s and tried to explain the reasons for the manufacturing performance gap between Toyota and other automobile companies, we saw at Toyota the now familiar “lean” techniques such as kanban, cellular manufacturing, short changeovers, andon lights, and so on. Many concluded— and I initially did too— that these new production techniques and the fact that Western industry was still relying on old techniques were the primary reasons for Toyota’s superior performance.

    Rother concludes that the secret to the Toyota Production System and Toyota’s success was not any one particular tool or technique, but rather the Toyota Improvement Kata, which focused on continuous improvement of everything at Toyota, seeking to establish flow and eliminate waste.

    Toyota Kata is a great book and if you want to understand DevOps, I encourage you to get a copy and read it for yourself. Kevin Behr recommended it to me and I was pleased to see Jez Humble reference Rother’s story, above, in his recent QCon talk on DevOps.

    What is DevOps?

    So, after all that, what are we left with?

    What is DevOps?

    Cue the trumpet fanfare…

    Fundamentally, DevOps is the activity of optimizing the development-to-operations value stream by creating an increasingly smooth, fast flow of application changes from development into operations, with little waste. Optimization of the value stream takes place continuously using various continuous improvement techniques like the Toyota Kata.

    “Is that it?!” I hear you cry.

    Yes, that’s it.

    “No way!”

    Way.

    “But what about Tools and Culture?! What about continuous delivery and Jenkins?! What about automation? What about mutual respect?!”

    Yes, those are all in there, too. But those are the equivalents of Rother’s “kanban, cellular manufacturing, short changeovers, andon lights, and so on.” They are solutions to particular problems that you might be faced with.

    If, during the course of trying to optimize your value stream, you’re faced with the problems that those tools and techniques solve, then you might try them out as solutions. Or you might not, if your situation dictates differently.

    The key is to understand that simply deploying a tool or even a broad solution without thinking through your unique situation and the current state of your value stream and its current constraints is foolish. After all, the specific solutions that are applicable to an agile web startup have a high likelihood of being inapplicable to a large enterprise. And vice-versa.

    Why “Enterprise DevOps” Doesn’t Make Sense

    So, that brings us right back to where we started.

    Enterprise DevOps doesn’t make sense because it confuses the forest for the trees. Those advocating for Enterprise DevOps make the mistake of focusing on specific tools and solutions (Jenkins, Kanban, etc.), then find them wanting for various (extremely valid) reasons in an enterprise context, and then reject DevOps in favor of Enterprise DevOps, which they suggest is somehow different.

    But DevOps is about flow and continuous improvement, not about specific solutions. Flow and continuous improvement are equally applicable to a large enterprise as they are to an agile web startup. And if you miss that, you’re lost, regardless of the tools and solutions you choose.

    To implement DevOps, you need to learn how to map your value stream, identify the constraints in it, and then use a continuous improvement kata to develop specific solutions for your particular problems at this point in time in your particular organization.

    If you’re a large enterprise, the problems and constraints that you face will undoubtedly be different than those of an agile web startup. So, you’ll necessarily need to select different solutions.

    That doesn’t mean you “aren’t doing DevOps!”

    Rather, it means that you’re doing DevOps correctly.

    And remember that whatever solutions you come up with to your particular problems and bottlenecks, they are likely to be temporary. As Toyota said to Rother’s colleague, solutions are only applicable to a particular problem at a particular point in time.

    Epilogue: A Bitter Irony

    Some of the earliest work on what would become DevOps grew out of some ideas that Andrew Shafer was kicking around in late 2008.

    At the Agile 2008 Conference in Toronto, Shafer proposed a birds-of-a-feather session about “Agile Infrastructure.” Only a single person showed up — Patrick DeBois. He found an empty room. It turned out that Shafer skipped his own session because he received such poor feedback about it and thought nobody was interested. Shafer and DeBois later discussed Agile Infrastructure and many ideas that would later become DevOps in the hallways at the Agile 2008 Conference.

    Shafer also presented on Agile Infrastructure at Velocity 2009:

    [slideshare id=1637386&doc=velo09-agile-inf-090625020850-phpapp01]

    Note slides 92 through 97. The key points from this sequence are:

    • Dogma Sucks
    • “Do not seek to follow in the footsteps of the wise. Seek what they sought.” This is a quote from Japanese poet, Matsuo Bashō.
    • The Tao that can be spoken is not the eternal Tao.
    • You are a special snowflake, just like everybody else.

    Even from the start, Shafer was on to the point that a binding set of rules is self-defeating. DevOps cannot be codified in a set of rules. Because it’s bigger than that.

    Note the Matsuo Bashō quote:

    Do not seek to follow in the footsteps of the wise. Seek what they sought.

    The wise DevOps practitioners do not seek to implement any given tool or technique out of adherence to DevOps dogma. Rather, they seek to create a smooth, quick flow of changes through the value stream. Do not try to follow in their footsteps using the exact same set of tools or techniques, but rather seek the same thing that they seek and find your own way to the same destination.

    So, the bitter irony here is that we’ve been warned all along, by Shafer, Allspaw, Hammond, and others, to focus on the foundational problems, not the specific solutions they have employed. Yes, we can learn something from their experiences, but every situation is unique.

    (Unfortunately, I haven’t been able to find video of Shafer’s Velocity 2009 presentation. If you find it, let me know. But you can see a later version of Shafer’s ideas, mixed with those of Paul Nasrat, in a presentation from Agile 2009, made available on InfoQ.)

    Other Resources

    I’ve hopefully salted this post with enough links to key presentations and resources, but there’s one more that should be included. If you’re at all interested in the history of DevOps, and you should be if you really want to understand it better, then you need to view Damon Edwards’ “The (Short) History of DevOps”on YouTube:
    [youtube http://www.youtube.com/watch?v=o7-IuYS0iSE]

    Connect With Me

    Think I’ve left something out or put too much in, got it totally wrong or completely nailed it? Let me know! Continue the conversation by commenting on this post and following me on Twitter.

    About the Author

    dave-roberts-headshot_400x400Dave Roberts brings 25 years of technology experience to his role of Executive Advisor at BMC Software. A thought leader in agile IT and digital transformation, Dave routinely speaks at conferences around the world. Dave is a member of the Twitter #clouderati and blogs at BMC.com and Leverhawk.com. He holds nine patents and has written three books on computer programming and data communications technology.

  • Product Management’s Role in a DevOps World

    Product Management’s Role in a DevOps World

    When we started Datical, we knew that in order to be successful we needed to practice what we were preaching: DevOps. As the VP of Products and product owner of Datical DB, our flagship Database Lifecycle Management tool, I recognize that Product Management is well positioned to facilitate DevOps adoption across the entire organization. It’s our job to ensure that the solutions we deliver to the market are aligned with our customers needs, valuable, and appropriately represented. To achieve this end, my team has embraced the Three Ways and established practices that map them.

    The First Way: Systems Thinking

    “Systems Thinking” in the product management group goes beyond application development and delivery. Our PM team must also consider how we improve, market, sell, and support our solutions to fully realize our company’s potential. To gain the deep understanding of all aspects of our Product Lifecycle, we’ve agreed upon the following guiding principles:

    • Every Sales Call is a Product Call – My team and I spend a lot of time shadowing calls in all stages of the sales cycle. By attending as many calls as we can we learn what first motivates our market to seek a solution in our space. We get insight into what features resonate most and what areas of our solution are uninteresting to them. We hear how our Sales team is positioning the product and we understand the challenges they face when presenting our solutions to interested parties.
    • We Are a Required Member of our Scrum Teams – The PM & Engineering organizations are highly engaged with one another. Product & Engineering management meet 2-3 times a week to discuss sprint progress, new development stories, and long term product roadmap. Product Managers attend daily development stand ups to answer questions and eliminate roadblocks in the current sprint based on the needs of internal and external customers. We balance technical debt, infrastructure requirements and feature development during sprint planning to ensure high-quality product releases and a reliable delivery chain.
    • We are Marketing’s Lyricist – Our deep understanding of our customers, our market, and our products allow us to arm our marketing group with the content and messaging they need to be heard over the chatter. We influence marketing plans with what we know is working in sales and what we know is coming out of engineering.
    • Everyone is Customer Support – We take reactive and proactive participation in customer support seriously. PM responds directly to customer support requests and we constantly reach out to customers we haven’t heard from to make sure they are getting value from the product and that their needs are being met.

    The Second Way: Amplifying Feedback Loops

    The Second Way, Amplifying Feedback Loops, is where the product group provides the greatest value to the company. By adhering to our guiding principles of Systems Thinking detailed above, the product group has a good handle on all aspects of our business. We know how we sell, we know what we need to do to increase our value, we know the engineering effort involved in delivering new features, we know where our customers are thriving and where they are struggling, and we know what we’re telling the market.

    The physical manifestation of the Second Way in the product group is our Product Roadmap. Too often, roadmaps become little more than flashy marketing tools that promise a lot, but often fail to deliver on the hype. We’ve found that customers respond more favorably to a smaller roadmap that is accurate than they do to a road map that is full of wild, cleverly named features whose delivery is uncertain at best.

    At Datical, we take what we’ve learned from participation in sales and support activities to prioritize our development efforts on real customer needs. We apply what we’ve learned from engineering about the effort involved to deliver on our customer prioritized feature list, balance those estimations with what’s necessary to maintain a smooth operating environment for development, and provide realistic delivery goals for product features. Once we have dates for new feature delivery, marketing can plan their strategies ahead of time and sales can speak with confidence and certainty when discussing our future with prospects and customers. When every person in your organization can speak confidently about your customers, your product, and your market, the deck is stacked in your favor.

    The Third Way: Embracing Experimentation

    Product Managers that know their customer’s problems are crucial to reaping the most reward from Embracing Experimentation. A great example of this is a feature we are releasing in our 3.0 version of Datical DB this Friday. We call it the Rules Engine, and it is the direct result of Datical’s culture of encouraging experimentation.

    Datical DB has a feature called Forecast that allows you to simulate your proposed database updates in memory without actually affecting the database instance. This takes the uncertainty out of the deployment process by identifying errors prior to execution. Customers really liked the feature, but there were always questions about the extent of Forecast’s capability. Typically they would ask about standards and best practices for their specific organization and whether we could modify the Forecast feature to accommodate their specific use cases. If we were to commit to modifying Forecast to suit everyone’s needs, it’s all we would have time to do. We recommended that the customers manually review the changes in those cases before promotion to downstream environments, something they were already doing with piece meal SQL scripts.

    While discussing the frequency and consistency of these feature requests with our VP of Engineering, we had an idea. What if customers didn’t have to rely on us to modify Forecast for their specific needs? If we could deliver on this, we would drastically reduce the time and money companies were spending on reviewing database change prior to approval. We decided to turn our best minds on the idea with a very sparse set of loosely defined requirements.

    Our first attempt failed. We reviewed the results of our research and prototype attempts, refined the story, and tried again. Our second attempt showed promise. We refined it further, built on what we learned, and had a viable solution in sight.

    We began talking to our customers about it. I actually heard a grown DBA giggle with excitement! He had DBAs on his team spending 70% of their time manually reviewing SQL scripts from development to make sure they were in compliance with their standards.

    After collecting customer feedback, we combined it with the results of our technical experimentation, and are very thrilled to be bringing this feature to market at last. By encouraging experimentation and not bowing to initial failures, we’re delivering a great differentiating feature to the market.

     

    About the Author

    High-Res-Peter-Pickerill-6571-150x150

    Pete Pickerill is Vice President of Products and Co-founder of Datical. Pete is a software industry veteran who has built his career in Austin’s technology sector. Prior to co-founding Datical, he was employee number one at Phurnace Software and helped lead the company to a high profile acquisition by BMC Software, Inc. In addition to managing a development team and product schedule at BMC, Pete worked directly with large financial services institutions to provide the features they needed to achieve greater efficiency and cost savings in their application deployment processes.

  • No DevOps Roadmap? No Problem.

    No DevOps Roadmap? No Problem.

    I’m a huge fan of The Phoenix Project and I’m honored to count Gene Kim both as one of my professional inspirations and also as a personal friend. There is, however, a dangerous precipice inside of the Phoenix Project that, if misunderstood, could cause many DevOps practitioners and enterprise IT shops a daunting problem in adopting and progressing in the ways of DevOps.

    Roadmap or Roadblock?

    One of the main concepts discussed in both Kim’s The Phoenix Project and Goldratt’s The Goal is “the bottleneck” and what what to do about it. The basic premise explained is that all actions to increase throughput before a bottleneck, pile work up at the bottleneck and all actions to increase throughput after a bottleneck create a void of work to address after the bottleneck. The logical conclusion from this idea is that: in order to have a net positive effect on throughput, action must be taken at the, holistically viewed, tightest constraint on throughput within the operational system.

    This concept model is both accurate and valuable as the IT shop and the manufacturing floor have a lot in common. The power of this concept model, also happens to be where the aforementioned dangerous precipice is revealed – getting alignment on which bottlenecks to focus on can turn into a multi-month effort in many enterprises which can require consultants, workshops, deep-dives, process studies and other activities that are at once highly valuable and not always likely to be approved by a shop that has not fully bought in to the three ways of DevOps.

    One of the lesser understood secrets of corporate america is that even when you have formal authority, permission is required to do almost anything of substance. One “no” from the wrong party can kill even the most promising initiatives. This “veto effect” doesn’t require an absolute “no” either. Much like a parent saying “let me think about it”, the veto effect only requires the absence of a “yes”. This absence can take the form of “maybe”, “let me think about it”, “let’s wait and see”, or even a “run this by the other department heads and see what they think”.

    Given that any reasonably sophisticated roadmap takes more than a few weeks to develop, vett and roadshow around a medium to large enterprise, a mature DevOps roadmap suffers from a classic chicken/egg paradox for many organizations. I extend my deepest sympathies to those who have found that a clear and easily understood roadmap can somehow both provide and require the catalyst to consensus and permission to proceed in the ways of DevOps. You can take heart, however, in the fact that neither you, nor DevOps, are alone in this conundrum as this is what can be referred to as “the corporate condition” that we must all bear.

    While company may not always provide solace to the miserable, there is however, another way.

    Just Follow the Sun

    What is this one side road still available to you that has the power to relieve almost any non-alignment traffic jam? Start heading west! Just like the American pioneers, you can start following the sun to the wide open spaces of DevOps land where automation is plentiful and provable performance gains via instrumentation are gold. All you need is one or two comrades with some skills worthy of the road ahead and you can be off!

    Just because you can’t get the unix team to automate environment creation, doesn’t mean you can’t head west and automate your build processes. Just because you can’t automate your entire production deployment doesn’t mean you can’t head west and automate your performance testing and benchmarking processes. Just because you can’t get everyone to agree on the tightest constraint, doesn’t mean you can’t head west and alleviate the ones you already know about. To head west on any of these things, you don’t need full concensus, you only need a traveling partner or two.

    The Trail Has Already Been Marked

    For mid to large size enterprises, a true DevOps transformation happens over the course of three or more years. Allowing a lack of consensus to become a total standstill is not the right answer. The tool at your disposal to overcome the veto affect is public consciousness.

    Take UX design principles for example. It was less than a decade ago that design activities were considered a luxury item that produced pretty pictures; and then Apple happened and tipped public consciousness over. How many people are willing to casually veto design now?

    When public consciousness to the value of DevOps tips over, it will no longer be you going to the powers that be to get the “OK to proceed”. It will be them coming to you and asking “what is taking so long?”. In that one moment, you’ll be able to fire back at them: “Here’s my roadmapping proposal. I’d love to have you be my business owner!”

  • The Importance of the Bus Driver

    The Importance of the Bus Driver

    The topic of DevOps has been a topic for many a debate over recent months years. Without adding fuel to the fire I’d like to present a slightly different take on the DevOps culture.

    The phrase ‘what if I was hit by a bus?’ can often be heard within the vicinity of software development teams. It relates to a measurement known as the ‘bus factor‘. Individual talent should not be discouraged, however when it becomes silo’d that talent translates to risk for a business. Try to think back over your career – do the following sound familiar?

    “I don’t know how to do that, X usually does that”

    “X is away at the moment so we’ll need to put that on hold”

    “X left yesterday and didn’t cover this in their handover”

    If you have read ‘Phoenix Project‘ – think of Brent…

    Using the fated bus driver I’d like to take us on a journey (excuse the pun) through a DevOps culture.

    In-sourcing the bus driver

    You work for a small green minded startup. Being green minded and wanting to help your staff, you decide to buy a bus to get everyone to work in the morning. Realising that you don’t actually have a dedicated bus driver, you decide to take it in turns to drive the bus in the morning. Everyone gets involved, everyone knows how to drive the bus, everyone knows which route is best.

    Idea 1

    Don’t over engineer your processes. At this stage you’re already exhibiting DevOps characteristics!

    The talented bus driver

    Your small team has performed really well, people like what you do and now your team isn’t quite so small. There are now over 50 people on your bus in the morning. It wasn’t long until you realised that Sarah is the best bus driver you have. Inevitably, Sarah usually just takes up driving the bus each and every morning.

    If Sarah is off, Mike takes over but he often makes people late for work and tends to hit a lot of parked cars.

    Furthermore the bus recently broke down when Sarah was off, Mike and your bus passengers all took a look at the engine and tried to help but there seemed to be a ‘magic’ piece of string holding two parts together.

    When they consulted the bus manufacturer manual, this piece of string was no where to be seen. Mike had to call Sarah whilst on holiday!

    Mike calls the bus company to ask for a replacement bus, to be told it will be 2 hours until a second bus arrives. The team sit and wait patiently.

    Idea 2

    Try to identify your tipping point. Some key indicators are reliance on individuals, information silo’s, phrases such as ‘oh I didn’t know it functioned like that in production’, production environments start to diverge (without communication) from development environments.

    How many production environments have the web server, app server, database, load balancer et al all running on the same machine? Is this how your development teams work? Does this differ to the view of the operations team?

    Your bus stops

    We’ve identified a few problems but lets focus on the positives. Firstly, let’s celebrate actually spotting the problem, it is very easy to continue with the status quo. Our second success was the reaction of our bus passengers. When the bus broke down, instead of complaining, the bus passengers tried to help mend the bus.

    Idea 3

    When and where your bus stops is less important than who it stops for. It might be stating the obvious, however if you wish to move towards a DevOps culture you need talented team members. You need people that care for the organisation and their profession. A desire to share knowledge for a greater goal.

    If your team are not interested or do not care, then no matter how much you say DevOps you need to boost their interest first.

    Hopefully you’ll start to hear your teams utter phrases such as “What does production look like?”, “You could do this slightly differently you know”…

    What if your talented bus driver has an RTA?

    There are no doubting Sarah’s talents – I mean come on, she was able to fix a bus engine with a piece of string, knew the best bus routes and always got your team to work on time…

    What would we do if Sarah left or was, somewhat ironically, hit by a bus?

    Idea 4

    Could your organisation raise the value of sharing and encourage knowledge share?

    Assuming you’ve got the right people on the bus, you might also find that other staff members have ideas for improvement when you explain the current process. Your Sarah(s) should not fear sharing their knowledge and should be rewarded/encouraged for doing so.

    One of your passengers may even know a faster journey to work or a better way to fix the bus engine.

    The bus seating plan

    Although you didn’t enforce it, you start to find groups forming on your bus journey, all your teenagers are sat together at the back playing music on their phones…..

    Idea 5

    Don’t let job titles restrict conversations. Water cooler chats are incredibly productive and help teams share ideas.

    Replacement bus service

    When the bus broke down Mike was able to order a replacement but it had a 2 hour turnaround time.

    Would it be better if we always had 2 or 3 buses running? If each bus had enough capacity for all 50 passengers we’d probably always make sure people were on time for work.

    It would be even better if all bus drivers knew the routes that Sarah used, all buses had the same piece of magic string and all bus drivers knew what the magic string did.

    Idea 6

    Something my team often hear me say is ‘Design for failure, don’t just react to it‘. When designing your applications and services try to think how the system can accommodate failing services better rather than what actions you might take in the event of failure.

    The same might be said for your human resources. Have your teams got a shared knowledge? What would your bus factor score be?

    Journey times

    Try to keep your passengers informed of your progress – you could let them know their ETA for work. Informed passengers tend to be more engaged and understanding. For example when telephoning a business, do you prefer to know what position you are in ‘the queue’ or would you rather sit on the line waiting without knowledge of when you may be spoken to?

    Idea 7

    Monitoring might be considered a key technical section of DevOps adoption. There are many, many tools that can help with this such as StatsD, Graphite, Librato, New relic, AppDynamics etc but whichever medium you choose, try to make sure your ops stats are in full view of your various teams not simply your ops staff. Get them in front of your dev team, support team, sales team, HR team they impact every member of staff.

    Hopefully, you will be surprised at an increased level of employee engagement. When ops are in war rooms fire fighting problems, each team member will understand the pressures they are under, some team members will also express a desire to help.

    You have arrived at your destination

    At this stage I may have flogged the bus concept a little too far. Are the concepts here a guide for the adoption of a DevOps culture? Does the ‘D’ word have to be attached to the guidelines above? I’ll let the community decide.

  • Q & A with Kevin Behr on DevOps, its impact on security, and the challenges of bringing change to the organization

    Q & A with Kevin Behr on DevOps, its impact on security, and the challenges of bringing change to the organization

    Kevin Behr has been in IT game for 26 years and is the author of seven IT management books, including the bestselling The Visible Ops Handbook, which he coauthored with Gene Kim and George Spafford, and The Definitive Guide to IT Management, published by Hewlett Packard. His most recent book, The Phoenix Project, also coauthored with Kim and Spafford, has helped to spark discussion about DevOps globally. Behr is also well-known for his technology and management keynotes for such organizations as the National Academy of Sciences, Hewlett-Packard, the SANS Institute, AFCOM, and the IT Service Management Forum.

    Behr is also the founder of the Information Technology Process Institute (ITPI) and chief science officer for the CXO and Board Advisory Practice at Praxisflow (formerly Assemblage Pointe), where Behr consults, mentors, and coaches IT organizations and executives on how to increase their business effectiveness and competitive advantage.

    We caught up with Kevin Behr to discuss DevOps, its impact on security, and the challenges of bringing change to the organization.

    devopsy.kinsta.cloud: We’re now hearing a lot about continuous deployment, but what does that mean compared to traditional enterprise thinking about deploys? And what does it mean for IT disciplines like security?
    Behr: With continuous monitoring, you can catch failure quickly, even if the issue that caused the failure makes it into a production state. With today’s SaaS and cloud models, failures don’t matter as much because you are not shipping software as much as you are managing a service. And one fix works for all of your users instantaneously.

    This is the difference between traditional enterprise IT thinking and the thinking around continuous deployment, which really is starting to creep into the enterprise. The new thinking is: how do we do this smaller, faster, and more safe to fail. And organizations aren’t ready to make everything safe to fail, but pushing things in that direction becomes an actual security strategy—versus the notion of, let’s just see what happens.

    That’s what we’re getting into with continuous deployment. You’ve actually made the deployment safe to fail. You almost don’t need very much testing because you can actually deploy, then wait to turn on a new feature; and if it doesn’t work, you can turn that feature off. What we’re starting to see is this kind of radio button style of deployment: they’ll deploy software and it’ll be turned on progressively, versus an old-school Big Bang.

    That’s really kind of neat when you think about it. Software is being treated more like infrastructure. You hear that phrase a lot: Code is infrastructure, and infrastructure is code. You start to see the different meanings of that everywhere. And I think that’s right. I think the security practices have to be a part of the safe to fail approach. You want to have very good monitoring and detecting controls, because they’re going to help you be more resilient.

    devopsy.kinsta.cloud: I think these are concepts that may have challenges when working with some security and IT professionals and others who are used to having control over the process. What are you seeing in regard to the balance between an organization’s need to oversee and protect itself and the benefits of continuous deployment?
    Behr: You’re getting at the fear that security practitioners and others may have, that this breaks down controls that they’ve put in place. They’re realizing that it could actually add more resiliency—consider continuous authorization to do certain things, cross-functional teams, continuous monitoring, and all these things that could very possibly increase resiliency.

    You’ll see more success if security could embrace DevOps, from the standpoint of, “Oh my: This is amazing and so what are the top ten things we must do to make DevOps work? Not “here’s why it can’t work.” Security teams that roadblock like that are going to get steamrolled right now.

    But it’s not good news everywhere. I hate to say that some companies out there are more stupid than others. I’m seeing places use DevOps as a way to excuse all the reduction in force that they’re planning: “We need to get rid of five people because we’re doing DevOps.” What sense does that make? I think mini-backlashes are coming in some organizations. And one of them is going to be where organizations call things DevOps that aren’t, just because it serves their ideas.

    For instance, I’ve been into a couple of banks that have created DevOps departments, which is sacrilege to this whole movement, because the whole idea is to get rid of silos, not to create a new one.

    Some days I get bothered by the notion of organizations calling it DevOps, and then other days I don’t because it’s just the result of the success of an idea.

    devopsy.kinsta.cloud: What steps do you see organizations taking to deal with the change? Are you seeing schisms in organizations where some people are resilient to change?
    Behr: I think that the folks who own these systems that they work in have a long way to go. And I think the people who are having a hard time with DevOps are getting steamrolled right now. People in vice president of operations positions, or the director of operations: They may not love the concept of DevOps because it creates cross-functional challenges for them. The first is: “All of these people who don’t report to me and now are in my data center. What are they doing? Why are they on keyboards? How do I control this?” [Story]

    Think about it. Their whole working role is judged on this notion of reliability. Their heuristics tell them that having lots of unaccountable folks doing stuff isn’t consistent with reliability in the long term. In the shops where DevOps has been successful, the security folks have been able to get in and say: this is awesome. They love the reduced cycle times. They love that the dev guys are coming down and helping with patching. Getting things done whenever they need it. And that they actually know what this stuff means now; it means that their security feedback loops directly into dev.

    I think that’s the other thing that people don’t get. DevOps, when really done right, creates parallel feedback loops, not just serial feedback loops. In the old way, someone asks someone to do something, they do it, and then the provider asks how they liked the service? That’s the feedback loop most operations have today. But when the devs are brought in, all of a sudden the devs are adding another feedback loop. It goes something like, ”Hey, I just watched you do this thing that I don’t do every day. Man, that’s a lot of extra steps. Why are you doing it that way?”

    That’s one feedback loop that comes from just teaching somebody something new, and getting the feedback from the outside. And then there’s the feedback loop of the quality of what you delivered. When you’re getting both of those at the same time, parallel feedback loops are how evolution happens. That’s one of the hallmarks of all evolving systems and adapting systems.

    If security can become one of those parallel feedback loops, and all the documentation of what you’re doing when it happened, and all that related stuff is in one place, then security becomes part of the refactoring.

    I think the other thing is, because the dev and ops people are collaborating more, you’re starting to hear more contextualization happen with the developers. They’ll say things like, “I’ve always wondered why they yell at me whenever I hard-code IP addresses. Now I get it: because you guys have to change these all the time, because we’re now down here changing a bunch of them.” That contextualized understanding goes a long way.

    This is a huge thing. Because there’s a ton of infosec ops: patching, log parsing, and different things that happen at an operation level that are really security driven. And if you can get the devs involved in helping out with some of that stuff, they start to understand why you’re doing it. And in many cases, we’ve got technologies in place that we spend a lot of money on, and security, that are just to prevent devs from doing things that we didn’t think we could change.

    The organizations that get this—and foster it—are the ones that are going to succeed.

  • DevOps Is For Horses: Stop Making Excuses For Starting

    DevOps Is For Horses: Stop Making Excuses For Starting

    While DevOps practices may be spreading throughout organizations both large and small, often times the most visible success stories are also the most exceptional. Organizations like Netflix, Etsy and Google are brought up in conversations about DevOps time and again because they offer the most stark examples of the benefits that the model can bring to businesses. But the scale of their operations can often be intimidating to the everyday IT person.

    “People will tell me, ‘Well, these are unicorns—special companies. We’re not like that. What works there will not work here,’” says Jez Humble, a principal for the consultancy ThoughtWorks and co-author of Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. “I think that’s a fallacy.”

    According to Humble, two phrases that he hates the most as excuses for not adopting DevOps or continuous delivery principles are ‘That can’t work here,’ and ‘But we’ve always done it that way!’ He’s one of a growing contingent of DevOps pundits who argue that it is plenty possible for even the most run-of-the-mill organizations to engage in DevOps.

    “DevOps is also for horses,” says Gene Kim, author of The Phoenix Project. “In fact, every unicorn was once a horse. Amazon in 2001, Google in 2004, Twitter in 2009 and LinkedIn in 2010—they all had huge monolithic code bases and inconsistent deploys.”

    Humble agrees that these so-called unicorns “have the same problems as everyone else.” But no matter how remarkable the scale of their solutions to those problems, other organizations should be able to learn at least one very valuable lesson from their case studies.

    “The critical piece is they’re always working to improve what they do, and that’s something that works everywhere,” Humble says.

    That’s Nick Galbreath’s philosophy. As the former director of engineering at Etsy, Galbreath saw first hand the transformation that the organization was able to make using DevOps patterns. He’ll assure any organization that it didn’t all happen in a day. Which is why when people ask ‘How do you get started with DevOps?’ his answer is simply to start.

    “If you have a sympathetic management ear, ask ‘Why don’t we try to cut deployment time in half?’” says Galbreath, who is now the vice president of engineering at IPONWEB philosophy. “Because the other way is to say ‘Deployment time takes too long, let’s do half as many.’ Wrong direction.”

    Setting up a goal like cutting deployment times in half will get an organization streamlining and weeding out junk processes. It’ll also have the team working to move those processes from inside a select few peoples’ heads and into things like a batch script or even just a simple document so that they’re easily repeatable across the team and that more can be done in parallel.

    “Cutting that work that adds no value in half, you’re really going to empower your group and your team to come up with innovative solutions,” he says.

    In a similar vein, DevOps for normal folks could also be chunked up incrementally on a project-by-project basis or automation tool-by-automation tool basis, says Josh Corman, CTO of Sonatype.

    “You don’t have to do an enterprise-wide shift hard left to DevOps,” he says. “You can do it in little projects or little small automations that free up time so you can achieve more competence.”

    In addition to incrementalism, Humble suggests providing a leadership environment that encourages taking chances and learning from mistakes.

    “Create safe situations so that when people make mistakes we can actually learn from them rather than engage in finger pointing,” he says.