devopsy.kinsta.cloud

Category: Blogs

  • DevOps and The Goal

    DevOps and The Goal

    Reading, “The Goal” was a pivotal event in my career.  More than a decade ago a friend of mine who was an operations professor at Georgia Tech told me it was his favorite book.  We were discussing the challenges of getting work done in a corporate environment.  I was grumbling about leading a web dev team in a company that did not embrace the software development discipline.  Already a fan of Tom DeMarco and Timothy Lister’s, “Peopleware”, and being very vocal about the lack of ability to focus whether due to environmental distractions or the constantly changing priorities and schedules, I was receptive to the message in “The Goal”.  It gave me a vocabulary and a roadmap for improving our throughput issues.

    Our throughput problems were a textbook case for development disaster.  When I joined the project, we had three source code repositories, from none of which could we build production and when we did produce a build it would work in pre-production environments but not production (this just slightly predates the widespread acceptance of continuous integration).  We also had the classic inconsistent release cycles that would lengthen based upon insistence by the business owners to include more features.  Looking back, it is not surprising in the least that we were not successful.

    Not surprisingly, these issues led to a lack of trust with the business.  Overcoming the technical problems was straightforward using brute-force tactics.  We addressed the source code problem by recruiting a senior developer and empowering him to make improvements.  We overcame the build quality issue by using a primitive form of agile and enforcing knowledge transfer; the operations engineer sat between two developers during the build and release process and he was the only one allowed touch the keyboard.  Our tactics were basic, and they worked.

    The bigger challenge facing us was getting our business owners to accept a different SDLC model.  We were saddled with a classic waterfall process, each phase being dependent upon the last with a lot of idle time built-in at each phase gate.  The shortcomings of the waterfall process were amplified because the business lacked confidence about when, if ever, they would get a feature released to production. This lack of confidence led to a constant rearranging of the priorities to get the most important feature du jour in the next release.  The constantly changing priorities created even more idle-time and a lot of work-in-progress (WIP) which Goldratt’s Theory of Constraints (TOC) tells us is at the root of inefficiency.

    Using TOC as a guide, we redefined our SDLC from requirements through QA.  We focused on developing a process that de-coupled the phases, reduced idle-time and work-in-progress (WIP).  De-coupling the phases meant the work could queue between each phase allowing the teams, (requirements, development, QA, and operations) to vary independently; they didn’t have to wait on the team before or after to complete their work which reduced idle-time.  We then focused on improving the slowest areas, which seemed to be requirements and deployments.

    We convinced our business owners to trial a monthly release cycle (this was a long time ago, even before continuous integration was widely accepted!).  The monthly release cycle helped give the business confidence that they would see their features in production at an expected pace and it allowed requirements to cycle on schedule independent of development and QA.  Some features would get done in a month and some would span several months.  Over a period of three months we got into a rhythm that worked for about 14 months until organizational focus changed and we focused elsewhere.

    Reading “The Phoenix Project” reminded me about those days and the challenges Goldratt helped us overcome.  I really liked “The Phoenix Project” and many of the situations rang true.  I have since moved from development to operations (twice!) which gives me a unique perspective as I see both sides.  I see DevOps as the embodiment of Goldratt’s ideas in IT; reducing work-in-progress, reducing cycle-time and always focusing on the most constrained processes.  Add a consistent release schedule and it looks a lot like DevOps.  Everyone has their own ideas about what is DevOps and how best to implement, I suggest we follow Goldratt’s advice and focus our efforts where they will have the most benefit.

  • Don’t let automation be the tail wagging the dog

    Don’t let automation be the tail wagging the dog

    There is quite a bit of talk right now about DevOps and how important automation is.  This could be compared to GM in the 1980s when the company tried to transform the way they built cars through automation and robot.  In the early 1980’s GM spent $90B dollars in an attempt to speed up the way cars were delivered to the market.  This failed with famous stories of robots painting each other and others welding doors shut.  Ultimately this project failed and the robots were scraped.  It was later reported that the money lost on the project could have been spent to acquire both Toyota and Nissan at the time.

    Automation dominates the conversation when it comes to DevOps.  If you do not learn how to build cookbooks you are not doing DevOps.  I fear this is going to set companies trying to transform IT off on the wrong track, focused on the outcome and not on the scientific process to get there.    Alan Sharp-Paul recently wrote that you cannot automate what you don’t understand.  The best way to understand it is in two key ways, change and performance.

    Let Science set you free

    This was the secret that Toyota used to dominate the automobile industry.  They were able to transform all of their employees into little scientist.  Constantly tinkering to squeeze our inefficiencies in the system.  Every worker was empowered to think how things could be done better.

    At the heart of the agile and DevOps processes is the practical application of the scientific method.  Think back to your 8th grade science fairs.  No not the volcanos but the ones that won best of show.  These had a clear hypothesis that stated what they were going to try and prove.  They were able to understand what they were going to change and measure the difference between the before and after.  They could clearly state what changed and if it was a successful experiment or a failure.

    This thinking is key for companies trying to transform to a DevOps IT organization.  If a company does not understand the experiments that are running and what the outcomes should be then automation will just make the problem worse. That is why change management and performance management are foundational for transforming to a DevOps culture.

    Tracking your experiments

    Any good scientist knows exactly what he is working on.  He knows every experiment that is running anticipating the outcome.  He knows all the details about each change that has been made to see how they impact the outcome.  This is the same way that IT should be tracking every change happening.

    Every change done in IT is a scientific experiment running. Tracking every change allows for understanding of the outcome of changes within the application.  Developers and Operations will be more informed about the cause and effect of their changes.  There are multiple different ways that you can track changes but the key is that you are doing it very well before you start to apply automation.

    Measuring the outcome

    There is no need to track the experiments if there are not being measured.  A scientist is very deliberate in how they measure the outcomes.  Good enough just does not get this done.  This is beyond just monitoring.  Monitoring in a lot of cases is after the fact, in production.  This is about measuring the difference for every experiment regardless of where it is being tests.  This creates the need to test changes all the way to when the first functional tests are done, measuring the changes of every code update.  Following it through all of testing and to production.  Understanding how each experiment impacts the system as a whole.

    Efficiency through automation  

    Once there is good understanding of all the experiments and a standard platform for measuring the outcome then automation starts to return exponential benefit.  Because every experiment it known the lower priority ones can be identified and automated.  Risk is better assessed.  Processes can start to speed up because there is visibility and accountability.  Applications are no longer classified as high risk or low risk only change is classified that way.  Determining what can be automated just becomes another experiment that is tried and measured for success and and failure.  Automation becomes just another outcome of DevOps to the business and not the route to achieve it.

  • Is DevOps a Title?

    Is DevOps a Title?

    So, I’ve wanted to write on this topic for a while as I think it deserves a little attention! I’ve heard numerous times that you shouldn’t have DevOps in your title or that job reqs shouldn’t be “DevOps Engineers”. This came up again at the DevOps State of the Union event that we hosted in Boston recently. There were definitely some very vocal folks saying that it just didn’t make sense to look for a DevOps engineer or hire a DevOps engineer. These folks make the following points:

    –          DevOps is a methodology not a job description – we don’t call our developers Agile engineers, they are just developers. They happen to follow the Agile methodology but they their job description isn’t to do Agile, it’s to create awesome products.

    –          DevOps should permeate an organization – having a DevOps group sort of missed the point of DevOps. DevOps should be embedded into the fabric of the company, not an adjunct to the development process. If it isn’t embedded how can you actually practice DevOps?

    –          DevOps does not equal tools – asking for a DevOps engineer with 5 years of experience in Chef or Puppet is sort of an oxymoron. Very few people have long-term experience with new, up-and-coming toolsets. So asking for that doesn’t make sense and again, it’s a methodology, not a toolset.

    It was an interesting discussion and I’m curious to hear what other people think about this perspective. We see a number of job descriptions and cards that say DevOps engineer on them. I can completely understand why people would do that, but I also understand the criticism of that approach as well.

    Before I sign-off, I’ll weigh in with my thoughts and hopefully we can add some discussion in the comments. As a proponent of DevOps, I’d like to see as many people adopt DevOps as possible. Strategically for businesses, I think it will make a significant difference in their operations. They will move faster, understand customers better, and be more efficient with the resources that they have. If it takes assigning a group or a title to bring DevOps into an organization, I think that they should do it. It’s a way to get started and work at embedding DevOps into their culture. I fully agree with the statements that DevOps isn’t a title or a tool, but I guess I’m practical about it too – some organizations will take different views on how to bring it into their company and have it take hold. In the end, I’m pro-DevOps, so if that’s how you need to make it happen, go for it! My guess is that over time, you’ll end up building DevOps into the fabric of the company and you won’t necessarily think of it as a job description or title, but just as a part of everybody’s job.

    What are your thoughts – is DevOps a title? If so, why? If not, why not? I’m looking forward to hearing your thoughts in the comments!

  • Don’t Be Afraid of the “R” Word

    It seems that many people are afraid of the word refactor.  I think in many cases, if someone admits they need to refactor, it means they either didn’t do their job right.  They either missed requirements, wrote inefficient or buggy code, etc.  Alternatively, there is something fundamentally wrong with the product or system.  The fact of the matter is that sometimes there are big problems and a major refactor/rewrite is required.  I think that is rarely the case, but if embraced by all and done judicially, the outcome can be significantly better and the major refactor avoided.

    It doesn’t matter if it is a feature, deployment script or architectural decision, at some point, everyone must admit they are not clairvoyant and cannot predict every way end-users or systems will behave.  Therefore, there will be missed/new requirements and unforeseen problems.  As such, I very much embrace the philosophy of build for what you know now and adapt as you go.  Typically, that means the output is simpler, it got done faster and enhancements and bugs are easier to address.  That philosophy also means you are constantly in a state of refactoring, or said differently, incorporating new learnings, making individual methods more efficient, improving the readability of the code, filling in missing tests, etc.

    Now, just because we have embraced refactoring and we have noble intentions of, in general, leaving the code better than we find it, that doesn’t give us the license to go crazy and start rewriting everything we see. I think there are right and wrong times to refactor.

    First and foremost, don’t let those yucky bits of code distract you from your real purpose which is to deliver business value by building those new systems, features and fixing bugs.  So instead of just jumping right into refactoring, quickly note the yuck as a TODO in the code, story or in your notepad so you don’t forget about it, but also stay on task.  That said, bad code that is directly in your way is fair game and I even think that bad code near the edges of your task are also fair game.

    Secondly, follow good engineering practices.  Make changes in small, bite size chunks, test and deploy your changes often and communicate them with the team.  In the end, you will end up with a better product, happier end-users and engineers which should all translate into a more successful business.

    If you are one of those clairvoyants, then email me and let’s talk over a beer because I have lots of questions.  Otherwise, if you are a mere mortal like myself, then embrace the “R” word and seek out companies and teams that also embrace it.  At the end of the day or after a few releases, it is pretty impressive to reflect back on how adaptable the team and code can be while solving real problems very quickly.

  • Why Baking Security Into the DevOps Framework is Vital to Its Success

    Why Baking Security Into the DevOps Framework is Vital to Its Success

    I attended my first official DevOps conference in Israel in September 2013.  But I was initially introduced to DevOps by my customers – firewall and network managers who, through their efforts to automate  security policy management, found themselves dealing with a new and like-minded set of IT stakeholders  –  DevOps teams.

    They said DevOps was also all about automation – automation that allowed IT to increase their business contributions by enabling new applications and other IT services to be rolled out faster with greater control and efficiency.  

    Conceptually, it was clear to me that the DevOps value proposition is just as relevant to Security as it is to Operations and Development groups, especially since Security teams apply automation with the same fervor as DevOps.  

    In fact, using automation to accelerate IT has caught hold in the network security world to fuel a new movement and class of solutions, called Security Policy Orchestration.  As the CTO of Tufin, a Security Policy Orchestration company, I could not help but recognize the synergies.  I was (and still am) intrigued by DevOps, so when I heard about the conference, I decided to go.

    And I am so glad I did.

    The alignment between DevOps and Security Policy Orchestration is obvious. They both champion the use of automation to accelerate the pace of business. We Security Policy Orchestration folks are just coming at it from a different area within the organization.  Kindred spirits, right?

    In theory, absolutely, but in reality…not so much.

    The conference as a whole was great.  It completely validated my theories regarding the synergies between Security Policy Orchestration and DevOp.  But what surprised me was that security was a topic that was not being discussed at all at the conference.   Maybe a hat-tip to application security, but network security was not even on the DevOps radar, at least not at this particular conference.

    It needs to be.  For DevOps, to really be able to move the needle, we have to work together.  If history has taught us anything, it’s that when new applications and services are not created or rolled out with security in mind, at some point we will have to go back and retrofit it back in.

    Envision the typical experience:   the development team builds a new application, brings IT Operations in to implement it.  Once that is done, IT Ops prides itself on all the corkscrewing it took to make the application work on the actual infrastructure.  Except for once the app is in production, it doesn’t work.  The troubleshooting begins.  Invariably, issues of network connectivity come up.  Solving them requires app, dev and security managers to break down the cultural and organizational barriers that exist within their organizations to work together to ensure the application is architected correctly into the network.  This starts and the process of change requests between AppDev and IT to get the app up and running. 

    If that process is not automated, it can take months longer than it needs to resolve everything.   Not to mention any company subject to PCI has to consider terms like secure zoning, a concept that is not second nature to DevOps folks.   If PCI is not an issue, there are plenty of other internal and regulatory mandates and security –related considerations that, if for no other reason than common sense, can exponentially reduce the chances of a company ending up in the headlines due to an APT.    

    These are just a couple of many scenarios that can be avoided if development, IT and security work together from the get go.

    The net/net is that Dev and ops have the same goal but they are siloed, which is slowing down the pace of business – inhibiting it.  Collaboration is not some warm and fuzzy team-building thing – it is a fundamental business requirement.

    And that’s why I am so excited to have the opportunity to do this blog.  I think I can add a unique perspective on how DevOps and Security can work together to reach our common goals.

    My next post will explain will dig into what Security Policy Orchestration actually is, so we can begin to tease out the similarities and determine how to build on that to true alignment and partnership. 

    Stay tuned as the drama unfolds! This should be fun….

  • DevOps Culture Hacks

    DevOps Culture Hacks

    opscode_logo_final1

    Talk by Jesse Robbins (@jesserobbins), Chairman and CEO of OpsCode

    Blogged live from DevOpsDays Boston 2011 by @martinjlogan

    Tricks for getting DevOps to work in your company, from the technical to the social, taken from experiences at Amazon.

    Jesse Robbins was the “Master of Disaster for Amazon”. As an ops guy, he worked multiple 72 hour sessions, took site up time very seriously, and eventually turned into the stereotypical ops guy who said “no” all the time. What’s worse is he discovered that he was proud of it. Jesse started to notice that he was taking site outages personally. Amazon at the time, 2002-2003, was doing standard ops. They deployed in large monolithic fashion which is an absolutely painful process prone to error. Managing such a process and also finding yourself emotionally involved with the work to a high degree is not a productive situation to be in.

    A turning point for Jesse in terms of moving from an obstacle in the way of change to someone that really knew how to add value with ops practice stemmed from a battle he got into with the “VP of Awesome” at Amazon. This was the nickname of this particular VP because it seemed that pretty much any highly interesting project at Amazon was under this man’s purview. What happened was that Jesse did not want to let out a piece of software because he knew, for sure, that it would bring the site down. The VP overrode him by saying that the site may go down, but the stock price will go up. So, the software went out, and it brought the site down. Two days of firefighting and the site came back up, and so did the stock price, and so did the volume of orders.

    The dev team went on and had a party, they were rewarded for job well done, new and profitable functionality released. At the end of the year, Ops got penalized for the outage! Amazon rewarded development for releasing software and providing value and operations was not a part of that. They were in fact penalized for something that was out of their control.

    This of course did not sit well and as a result of this and other similar situations Jesse actually got famous for saying no. Who in their right mind would want to release software and go through that over and over? When the business would put up a sign advertising new functionality that was to be developed, something they were presumably excited about, Jesse would write the word “No” on it.

    Operations naturally wanted to protect itself and came up with all kinds of artifice in order to do so. Root cause analysis so that blame could be assigned efficiently. Software freezes that prevented software from being delivered to the site during peak times of the year. This seemed like progress, but clearly was not looking back on it.

    (This all sounded fairly familiar to the folks in the room)

    On to DevOps

    Now to talk about DevOps. We have at least the beginnings of an idea of what to do about the situation described above. Prior to addressing this situation correctly, or at least more correctly, Ops looked at their output as a waste. The best thing they could do was to cost the site $0. Instead we need to be looking at this another way, bringing value through the function of ops. DevOps is about creating a competitive advantage around the things Ops does every day.

    Why does the break occur? Historically Ops creates value by reducing change and getting paged when things break. Dev is about value creation and Ops is about protecting that value. This creates a “misalignment of incentives” meaning that different organizations are rewarded for different behaviors. This creates something called local optimization. Knowing these terms will help you talk to MBAs about DevOps!

    We have a fundamental misalignment of incentives and in fact a conflict in incentives. Development is exclusively aligned to releasing software and not at all focused on maintaining it. Ops, is the opposite. Each group optimizes locally around this which creates conflict. Operations is focused on minimizing change because that reduces outage where as Dev is entirely focused on maximizing change.

    Solving this problem is what DevOps is all about.

    The unproductive way of thinking mentioned earlier came about was in an environment with 4000 devs and significantly fewer ops folks. In order to alleviate the problems caused by misaligned incentives and local optimizations were to come up with those punitive changes that are incredibly satisfying to Ops folks but really don’t help solve the problem. Those changes in the form of meetings and review boards that are around to punishing people into releasing the software the way you the Ops person wants. These are the kind of measures that control oriented people gravitate towards, and it feels like progress for them. To be more DevOps don’t try to fight them in this: “Don’t fight stupid, make more awesome”.

    One initial thing that changed, that started the real progress, was to align dev and ops in a way that prevented local optimization; putting devs on call for their own software. This started to shift ops from being the people that just dealt with all the problems to people that became experts on all the services that allow the software to run. Ops started to become tier 2, escalation for devs. The way you got there, was to offer devs deployment options and permissions, if they passed some training and were willing to be on call. Initially this caused a fair amount of chaos. Devs had a load of pagers and got messages that confused the heck out of them. There was pain and frustration. This pain and frustration and the fact that devs were now playing with tools in actual production environments, really changed the culture quickly.

    Through trial and error, top down fiat, audits, and every carrot and stick approach the formula for this class of organizational change was developed. This what Jesse uses even today to accomplish these changes and what we will concern ourselves with for the rest of this talk.

    • Start small and build on trust and safety.
    • Create champions
    • Use metrics to build confidence
    • Celebrate success
    • Exploit compelling events

    Start Small and Build on Trust and Safety

    We tend to want to take on the entire org up front. We want to throw everything out, starting at the top and working down. This simply does not work, Jesse had failed multiple times before realizing that this was a failing strategy. Continuous deployment for example seems like something you want to deploy widely. Instead you should start with a small motivated team and build some success there.

    Another thing to consider when thinking about starting small and building trust is that when introducing disruptive change in an organization you should lead with questions to garner buy in instead of just telling people that you have the solution and such and such is what to do. Don’t even use the word DevOps, just focus on the problems and get permission to start to change it.

    You have to make the experiment of changing things safe. Jesse tells people that he will take 100% of the blame if things go wrong, in exchange for the space needed to make the desired changes. Creating safety is critical in pushing through organizational change. Crucial Conversations by Kerry Patterson is a book that covers this really well and the Jesse thinks is one of the most critical books to read if you want to create organizational change.

    Create Champions

    “You can accomplish anything you want so long as you don’t require credit or compensation”. It is amazing what you can accomplish when you give away that part of your self that requires recognition. You must shine the spotlight on those in your organization that get it. Get those that are recognizing the need and acting upon what you are pushing forward to talk. Make your boss a champion; this is critical. It is really important that your boss can explain what you are doing and why or at least be able to provide air cover for you.

    The second part of this is to give people status; special status. SRE “site reliability engineers” walk around Google with leather bomber jackets. They get hazard pay, they have special parties, they are considered to have elevated status around the organization. When you find your champions do something that makes them stand out. Wikis are quite powerful for this. Write down and explain in very powerful language what these champions do.

    At Amazon Jesse created this thing called the call leader program. They trained people to handle high impact events. After a while there evolved a pressure to join. Eventually you become the person that people have to go to in order to get a certain status which gives you personally more organizational power – not the point but helpful in furthering the change you want to further.

    Use metrics to build trust

    Get as many metrics as you can. Begin to look at them for KPI’s (key performance indicators). These are the things you will use to prove your case. What you are looking for is a story, and a set of metrics to prove you have one. John Allspaw talks about things like MTTR, mean time to recover. These are great for story telling. You want to capture metrics early and use them to tell your story. “Having devs on call will be a great thing for us, and oh, by the way, here is the data that proves it!”

    Make sure that you are good at telling your story with the data that you capture. Here Hans Rosling gives a TED talk about how to tell a story with data that Jesse recommends for everyone. This can be used for inspiration. Ideally have your champions tell your story.

    Celebrating your Successes

    This comes back to “you can accomplish anything you want so long as you don’t require credit or compensation”. Create moments in time where you celebrate the success of the change you created. Have parties when you reduce MTTR by 15 percent. Give people to a moment in time where they recognize the change the created and that the change was good. This gives people a moment in time to look back on and judge progress. This is of critical importance.

    Exploit Compelling Events

    Compelling events are those big company issues, big or small, it does not actually matter. They are the events that create cultural permission to make important change. An example is the executive mandate toward cloud computing. This is a compelling event that allows you to make a whole bunch of procedural change. Big outages are compelling events, they give you cultural permission to make significant change that would be otherwise impossible in normal times.

    When you have a compelling event you don’t encounter resistance, but instead, permission to make large change. If you don’t have such an event, then create it. Jesse had something called “game day” at Amazon. Creating outages to test failure recovery. Non-recovery became a compelling event. Big deployment pushes are examples of compelling events. If you are in the middle of a serious problem with process it is hard to propose chucking it out in flight but if you offer to own the postmortem process you can direct it toward the change you want to make – though start small as indicated by the first point in this list.

    The next time you want to create change in your org particularly DevOps change keep in mind:

    • Start small and build on trust and safety.
    • Create champions
    • Use metrics to build confidence
    • Celebrate success
    • Exploit compelling events

    What are your thoughts on DevOps culture hacks? Anything to add to the list?