devopsy.kinsta.cloud

Author: Katie Sotheran

Lean for DevOps: Measuring the meaningful
At a recent Lean project report out to stakeholders, a piece of analysis was presented which showed the quantity of metrics being captured, in various areas, at different organizational levels. IBM’s Watson had been used to generate the chart, and it told an interesting story. But that story needed some translating.

‘Look at all those metrics under ‘Managers”, said one Director, with enthusiasm. ‘Seems like they’re looking at a lot of different performance measures for their people’.
‘Yes,’ said the Lean Coach, ‘but is that a good thing in itself?’
Another Director chimed in, ‘Well, when I look at how few metrics are being captured on timeliness I wonder how anyone knows how they’re doing.’

As an IBMer, and Six Sigma Blackbelt, I’m pre-disposed towards data. Information can be power. But with too much being reported, it can be hard to see the wood from the trees. Performance metrics are intended to allow teams and leaders to improve their performance. For many of those people, reading charts and interpreting data may not be a core skill and it certainly isn’t how the organization wants them to spend their time. So those teams generating masses of performance metrics may not be doing themselves, or their leadership, a service.

The key is to identify the critical few metrics. Ideally leading indicators of performance issues, rather than lagging ones. Leaving aside areas like SLA adherence and budget management to those responsible for governing them, these are the handful of metrics I would suggest a DevOps team needs to watch on a daily basis:
- Number of bug fixes for top 20 systems
- Number of critical issues, average time to close and longest time to close – looking just at the average may mask longer running issues
- WIP: number of projects in each phase of the SDLC
- Customer feedback for top 5 systems – preferably actual user feedback; failing that whatever you can infer from app usage. Top 5 systems may change over time as you launch new functionality or as usage changes according to the business calendar.
- Percentage of projects who deliver 100% of their agreed scope
- Core application availability – actual availability to users, not SLA adherence
What do you think? What do you measure?
October 27, 2015
Lean for DevOps: Is it OK to move the banana?
There is an urban legend about a Lean deployment which, in the process of applying the 5S to a workshop, drew lines on the desk to indicate where each worker should keep their banana.

5S, for the uninitiated, is a set of principles by which Lean practitioners organize a workspace – which might apply to a virtual, shared workspace, or Sharepoint repository just as well as a back office space or factory floor.
- Sort – identify and get rid of unnecessary items
- Set in order – organize what’s left so that everything has a logical place – including deciding on the optimum spot for banana placement
- Shine – tidy up and clean your new workspace
- Standardize – document and put boundaries around your newly organized workspace so that it’s obvious where everything should go – including drawing a line around that banana
- Sustain – establish systems, tasks and behaviors to ensure your work place remains neat, organized and free from clutter
These are great principles to apply to any shared workspace – and standardization will speed your adoption of DevOps principles. Standardizing the way we organize information allows for fungibility, so a Quality Manager can switch teams at short notice and quickly pick up the way their new team works, and access the information they need. Standardizing storage in a way is the principle behind Docker, and other modular code concepts. Standardization can reduce errors: if user requirements are captured in a consistent way it is easier to see at a glance if any essentially information is missing before we start work. And standardization ensures consistency, for example, applying a pre-determined decision tree and criteria to enhancement requests will allow prioritization decisions to be made quickly and equitably.

But you can have too much of a good thing, Creative activities and problem solving tasks are often intuitive or require people to take a variety of approaches until they find one that fits. A checklist just may not help here. Furthermore, one of the aims of DevOps is to increase flexibility and speed to market. So if your standard start to get in the way of that, you’ve lost your way.

As a Lean Coach I have struggled with the amount of churn I see in requirements during the development lifecycle: at first glance it reads like rework. But it is, in fact, a healthy feature of an iterative, responsive process. Many IT departments have sought to manage churn by trying to standardize it out – setting deadlines for new intakes, and arduous change control procedures. As soon as you layer governance on top of a process, you create a need for a shortcut, and an expedited route.

I recently ran a Lean workshop for an IT team during which we identified that 30% of change requests were going through an ’emergency’ process. Clearly the standard process wasn’t working.

DevOps requires standardization on an ‘as needed’ basis. This includes a standard way to handle rapid change, and variation. That standard approach must be light touch governance: make the right way the easiest way. And I’ll stick my neck out and say that standardizing where a worker keeps their snack is a step too far.
September 28, 2015
Lean for DevOps: Mapping the tributaries and deltas of demand
Here at IBM’s Lean practice we use a diagram called ‘Mississippi’ to analyze demand. It is not so-called because it originates in Mississippi – in fact I’ve been given to understand that this tool was developed, and named, in our UK practice*. Instead, it’s called the Mississippi because, like that great river, demand comes from many different sources (or tributaries) and can leave a process through a number of different points (the delta). And, apparently, at times the Mississippi delta reflects tidal patterns and flows upstream. In the same way, demand can route back into, or upstream within, a process – for example, in the shape of rework.

We want to analyze the volume and frequency of demand for a number of reasons. Understanding demand helps us to:
- Identify ‘runners’ (everyday requests), ‘repeaters’ (less frequent, but not unusual requests) and ‘rarities’ (infrequent requests). This might be in terms of user groups, applications, or type of request (for example – small changes vs major project). When we design our improved future state, we will design it for runners and repeaters. Sometimes the impact of exceptions can be so painful, team’s lose sight of how infrequent they are, and design a process to prevent or manage those rarities – which will be inefficient for all those runners and repeaters.
- Identify failure demand – Where are additional requests re-entering the process, as rework? Where is there process activity which doesn’t ‘go anywhere’?
- Illustrate channels in and out of the process, so we can identify potential control points, or illuminate a lack of filter, and see where demand might be routed in via unexpected channels.
- Tell us what we don’t know: which elements of demand are we unaware of? We need to understand all demand so that we can build and resource (load level) a process which can meet that demand. Sometimes finding out what we don’t know is a lesson in itself.
These benefits are as relevant to Lean for DevOps as in any other area.

It’s hard to say what you’ll learn from conducting this analysis, until you’ve done it, as every Mississippi is different. But to demonstrate some of the valuable lessons I’ve learned from drawing up Mississippis, here are some real life examples:

Lightbulb moment: ‘Churn is a fact of life – we need a better way to handle it’

Lightbulb moment: ‘We need to start saying no to enhancement requests’

Lightbulb moment: ‘We have way too many changes taking the emergency route: a faster ‘normal’ route will reduce that demand and the associated risk.’

In their different ways, each of these helped make the case for a shortening of the SDLC, closer coupling of development to business demand, and a more nimble deployment process.

*If you’ve seen this in use outside of an IBM engagement, I’d love to hear about it! Find me on Twitter to set the record straight.
July 14, 2015
Why do DevOps? Winning hearts and minds with a Lean boxscore

Change is difficult. It can be painful, disruptive, and costly. Making a case for change requires us to win over hearts and minds. In my experience, we need to win the hearts of the ‘do-ers’, the people whose daily work is going to be disrupted. And the minds of the executives, whose financial support and personal sponsorship we need to make change happen.

A Lean boxscore allows you to address both the hearts, and the minds. Boxscore is a simple tool which we use to compare the current state with the future state, and turn that comparison into a compelling case for change.

Step One: Current State Baseline
Let’s start with where we are today, by capturing performance metrics that tell an accurate story of today’s development lifecycle. This may mean looking beyond the project plan to the reality, by the way. By interviewing or workshopping with development and operations teams, we get a true picture of the way the development process currently performs.

We’re always interested in end-to-end leadtime, of course – from the moment the business provides a prioritized set of requirements, to the moment that functionality is in production and working as intended. We’re also interested in how much wait time there is within the process – how much time between steps, waiting for an approval, a developer, or a tester. We do this by creating a value stream map, capturing all those metrics and validated by the teams who work within this process every day.

We want to look beyond leadtime and wait time too. We capture the number of resources involved in each release, the number of handoffs, and the number of bugs caught at each stage of development and deployment. If we can get to it, we might capture the percentage of time a developer spends writing code for their current release, as opposed to firefighting issues from the last release, arguing over schedules, trying to secure an appropriate environment, or other ‘non value adding’ activities. We might also capture some usage statistics, or other measure of customer satisfaction, and any relevant performance indicators, like server response time.

Before moving on, it’s crucial to ask stakeholders: ‘is this a fair reflection of the way things currently work?’

Step Two: Future State Target
This next step requires some imagination, and some bold thinking. Imagine a world where you’ve adopted DevOps: you’ve removed the waste within the development lifecycle and instilled the ‘shift left’ principle in all your working practices, your ‘4 in a box’ teams are rapidly developing, testing and deploying code, Cloud-based environments give you immediate provisioning, and automation and smart tooling have taken the bellyache out of testing. What does that future look like? How quickly can you develop – if provisioning is immediate? How many bugs do you expect to put live – if testing is automated and your production environment matches your test environment? How many changes to scope – if you’re working with smaller batches and faster delivery?

We create a new value stream map, which details the way this brave new process will work in practice. This allows us to set sensible but challenging targets for our future state.

Before moving on, be sure to ask stakeholders: ‘are these targets achievable, if we make the changes we have outlined?

Step Three: So What?
Finally, we get to the money question. What’s it all worth? For each improvement there will be an impact. Many of these will be financial. Ask your stakeholder ‘if we can get this new functionality out to market 2 months faster, what does that mean for revenue?’ Or ‘if we reduce the number of defects in production by 90%, can we reduce our operating costs?

Putting a dollar value on the financial impact will win over those minds with a logical, fact-based argument. Now, investing in a DevOps pilot seems like something they can’t afford NOT to do.

And the case for change for hardworking developers, testers and production resources? Metrics like number of handoffs, bugs and rework, and batch sizes can easily be translated into ‘how much pain you have to live with each day at work’ to win those hearts over.

April 9, 2015
Think you’ve established Kanban? Why you might only be halfway there
As a Lean Coach working in IT I often hear development teams refer proudly to their scrumboards as an example of Kanban. To which I say, ‘that’s a great start… but it’s really only just the start’.

Let’s start with what a scrumboard is, and does. A scrumboard is a visual display of work, showing the progression of user stories in development. As you see some stories moving through the phases of development, and others getting stuck, you huddle to figure out how to get them unstuck. Now you’re using your visual display for visual management, which is a great Lean solution.

Change management boards, like the one used in The Pheonix Project, are another good example of visual management. The physical display of changes, shown by release date and system, is a great way to get a handle on the amount of change and risk you’re throwing into live production. When you start moving them around to limit the risk, again, you’re using visual management.

The word ‘kanban’ roughly translates as ‘signboard’, and a scrumboard really is a good start. But it’s only half the story. Until you limit the number of stickies on your change management plan, or the number of story cards on your scrumboard, you’re really only half way to kanban nirvana.

True kanban requires you to limit the amount of work in progress and to establish the pull of work. In a manufacturing plant, a kanban card is attached to each unit in production. When a completed unit is, say, taken off the shelf by a customer, the card is sent back into the production line to signal that there is now an empty slot on that shelf which needs replenishing. Limiting the kanban cards sets a constraint around the amount of work in progress – and this limit should reflect the capacity of the process. In the same way, you can limit the number of story cards, boxes on your scrumboard or spaces in your change management plan. When a scrumteam complete a story that frees up a slot for them to pull the next story into their workload: the visual display should reflect this by showing each team’s capacity. In addition to visual management, you can create rules around work in progress, requiring each scrumteam to limit their intake so they have no more than, say, two workpackages in progress at a time.

This is all well and good, but what do you do when demand from the business requires a greater capacity? Historically you’d have thrown that extra demand into the mix, asking your teams to do more with the the same amount of resource, maybe pushing back a release date to accommodate the extra work.

Kanban requires you to create additional capacity in the system to allow for the additional demand either by:
- rebalancing the load, for example switching some resources from a work package that is ahead of schedule
- removing or deprioritizing other work and putting work on hold
- increasing resources by bringing in new team members
All of these actions will generate some churn, so it’s going to be worth a tough conversation with your customer first. ‘Are you sure you want me to disrupt the current sprint, or the next release, to squeeze this in? How about we pick it up next week, when we’ll have some new capacity?’ By limiting your work in progress, you’ll be speeding up the flow of development into production, so you can make that ‘next week’ promise with increasing confidence.

So the next time you walk over to that scrumboard, why not take the next step, and limit the work in progress to truly establish kanban.
April 8, 2015