Tag: change

  • The DevOps Scorecard

    The DevOps Scorecard

    Mid-last year our team switched from doing Agile to doing DevOps. As we forayed into the journey trying to learn about DevOps and practice it at the same time,  a lot of questions arose in the team.  How was this different from agile and most importantly How were we going to be successful?

    That’s when we wrote down what would be the success criteria for our team: “Ship code frequently without causing a customer outage“.

    As the team matured we started evaluating a more granular way to track success.  Could the team mantra be broken down into quantifiable success metrics that could be represented in a scorecard? Based on our experience the DevOps scorecard should contain these 9 metrics to track DevOps team success:

    DevOps Scorecrd

    1. Deployment frequency: How often were we deploying code and getting new code in the hands of our customers? This metric should trend up or remain stable from week to week. Example: Twice a week, 50 times a day
    2. Change volume: For each deployment how many user stories and new lines of code were we shipping?  Example: 3 new features per day, Average 500 lines of new code per week. Another parameter to consider in addition to volume is complexity of change.
    3. Lead Time (from Dev to Deploy): How long does it take on an average to get the code from development complete through a cycle of A/B testing to 100% deploy and upgrade on production? Lead time should reduce as the team gets a better hold of the lifecycle.
    4. Percentage of failed deployments: What percentage of deployments failed causing an outage or a negative user reaction? This metric should decrease over time. Example: 9% deployments failed this month as opposed to 15% last month. This metric should be reviewed in combination with the change volume. If the change volume is low or remained the same but the percent of failed deployments increased, then there maybe a disfunction somewhere.
    5. Mean time to recovery: When we did fail, how long did it take us to recover? This is a true indicator of how good we are getting with handling change and this should ideally reduce over time.  You can expect some spikes in this number due to complex issues not encountered before. Example: On an average it took the team 15 minutes to resolve each last week, 14 minutes this week.
    6. Customer Ticket Volume: Number of alerts generated by customers to indicate issues in the service. This is a basic indicator of customer satisfaction. Example: 54 tickets were generated this week as opposed to 38 while the user volume remained steady is not a good thing
    7. % Change in User Volume: Number of new users signing up, interacting with my service and generating traffic. As new users sign up is my infrastructure able to handle the demand? Example: This week the number of customers spiked by 30% due to an external event causing volume of requests to go up
    8. Availability: What is the overall uptime for my service and did I violate any SLAs? Example: 99.9% uptime consistently for the last 3 months even with change in user volume
    9. Performance (Response Time): Is my service performing within my predetermined thresholds? This metric should remain stable irrespective of % change in user volume or any new deployment. Example: Sub 5 second response time from all geographies and devices

    Would love to hear thoughts on what other critical metrics DevOps teams are using

  • Changing Organizational Culture – A Sweaty Use Case

    Changing Organizational Culture – A Sweaty Use Case

    When we think about changing organizational culture our immediate reaction is usually – yeah right, go fight city hall – and we take the Homer Simpson route, and don’t even bother trying.

    Over the course of the last year, I managed to achieve what felt like an impossible change within my organization, and I took some valuable lessons from the experience.

    It dawned on me afterwards that there were a lot of parallels in this process that we also underwent when trying to instill a DevOps culture in our organization, and I thought I’d share what I learned.

    The Epiphany

     I love riding my bikes – mostly mountain biking, and I usually am only able to do this over the weekend.  It’s something I really love to do, but in addition to my full time day job – Head of Product at my company, I’m a full time dad – a proud father of three.  All of this doesn’t leave me a lot of time to do the things I love.

    On a good day I spend more than an hour stuck in traffic, and on bad days more than an hour and a half.  This is basically dead wasted time.

    So I had an epiphany one day. Why shouldn’t I take advantage of this time to my favorite thing?  I could easily bike to work – and use this wasted time to my benefit, doing the thing I love most.  The problem lies in the way I look after an hour bike ride.  Pretty sweaty.  Here’s a pic.

    And, I know what you’re thinking – the answer is – no it wasn’t raining that day, in case you were wondering.

    The challenge though, is that I work with people, so I needed to find a way to clean up after my rides.  But, alas alack, we don’t have a shower in the office.  The plot thickens.

    So you might be wondering at this point, how this all relates to DevOps.

    DevOps is about pushing change in organizations – and not necessarily just technical ones. There are cultural organizational practices that often times need to be changed before you can even start to think about technical changes.

    So I believe this story represents a real parallel case study on the organizational changes often needed, to create a DevOps culture within an organization.  These processes many times involve a change in hardware – in this case, the shower itself.  As well as require a change in culture, actually motivating the management to realize the importance of it, and have people start choosing to do athletics before work (and preferably also ride to work).

    The Method

    And then I ran into Jesse’s Robbins’ rules. For those who don’t know, Jesse is one of the founding fathers of DevOps and web operations, and happens to also be the founder of Chef (OpsCode at the time). In one of the recent CloudExpo conferences in Europe, Jesse gave an awesome keynote about how to hack organizational culture to drive change. So working on the premise of “Jesse’s Rules” – I got to work:

    • #1 – Start small – Build trust

    Until the new shower was built, I’d shower daily with a HelioPressure shower  (you can find it on Amazon for less than $100), and I’d shower every day on Nati Shalom’s [LINK TO DEVOPS BLOG] balcony in the office.  Demonstrating I was serious and dedicated to my biking to work. Yes, it definitely raised a few eyebrows, but then again, better to shower on a balcony than walk around smelling like a sweaty rug all day…

    • #2 – Create Champions

    Doing crazy stuff yourself is the easy part. Finding like-minded individuals that will do the same, well that’s a different story. But after working hard at convincing others, I managed to find at least two people that would occasionally use my improvised spa, and this really helped me drive change in my company.

    • #3 – Use Metrics to Build Confidence

    For me there were actually a few metrics that contributed to the driver for change.  The first was that, I right off the bat, lost quite a few pounds – which was an incentive for a number of people to jump on the bandwagon – or rather bike – in this context.  The next metric was hard research that shows that people that are in shape tend to be absent from work 80% less often than employees that are not in shape. That helped me in convincing my CEO that it’s actually an investment and not just thoughtless spending…

    • #4 – Celebrate Success

    After committing to our makeshift shower for months, budget was officially allocated, and a shower was built for all the employees looking to participate in sport activity on their way to work.  Needless to say that I’m not the only one riding my bikes to work today 🙂

    Lessons Learned

    So the lessons I learned throughout the process, and my key take aways as I see them are:

    •  #1 You’ll run into a lot of skeptics along the way. IGNORE THEM.

    Many people will attempt to discourage you from your goals along the way – don’t listen to them. If I had a dollar for the amount of times I heard “it’s never gonna happen”, I wouldn’t need company budget to build a shower.  But I stuck with the plan, and it ultimately paid off.

    •  #2 Change takes time

    Rome wasn’t built in a day.  It’s a cliche – but cliches are born from common experience.  We’d been discussing building a shower for 2-3 years at the least.  It’s true that until I didn’t demonstrate that I was truly dedicated, rule #1, it wasn’t taken seriously. It took time for the company to come around, and digest the idea, until it gradually happened.   So again, if you stick to your guns, and have faith – a change ‘gone come.

    •  #3 Take it one small step at a time

    Basically start small, and be creative. Until you can achieve the big results, celebrate the small achievements too.  Adding champions to your cause, increasing awareness, possibly a meeting to discuss the changes you propose.

    •  #4 Be prepared to improvise

    I owe a lot to my colleague that came up with the pressure shower idea.  I think until I actually demonstrated that there was a really dire need for such a change, it wasn’t really considered seriously.

    •  #5 Find a management sponsor

    Luckily our CEO is athletic in orientation, and is an avid bike rider as well, and he was committed to help making this happen.  Many times it won’t necessarily be the CEO, but find a management champion that believes in your cause, and is willing to help you bring about the change.

     Bottom Line

    No matter what change you’re trying to push, these lessons and rules have proven to hold true for me in any circumstances, and I often times use this method to push significant changes in work and in life.

    Oh, and here’s a pic of the new shower.

    If you’re ever in Israel, and need a place to work, and want to bike there…you’re welcome.