devopsy.kinsta.cloud

Author: Pauly Comtois

  • Kanban for Operations and Traffic Court

    Kanban for Operations and Traffic Court

    Howard stepped off the elevator and into the lobby. His eyes were puffy and felt like someone had scrubbed them with sand paper. Looking around the office, he made a beeline for the coffee machine. It was one of those fancy jobs where you use a touch screen to select all the different flavors, textures and aromas. Howard thought they all came out tasting like boiled socks. As the last few drops of black sludge dripped into the mug, Lisa peeked her head around the side of the machine.

    “Getting ready for stand up?” Lisa asked, eyeing Howard’s coffee.

    “Getting ready for life.” Howard grumbled as he headed towards the stairs. “You coming?”

    “To stand up, yes. To life, not until I have some of this delicious coffee-flavored oatmeal. See you upstairs.” Lisa said, putting a cup into the machine and starting to make her first of 30 mandatory selections.

    Howard strolled into the Operations area, starting to feel a little more human with some coffee in him. Looking around, he spotted Paul hunched over his keyboard, typing furiously. His desk was in its standard form of disarray. Empty Diet Coke cans covered the desk, with a few crumpled in the middle like fallen soldiers on a battlefield, the other cans standing around the crumpled ones in mock solemnity. The keyboard was one of those mechanical DAS keyboards with the blue keys, so it made clicking noises as Paul’s fingers danced across them.

    “Good morning Paul.” Howard said as he lumbered to a stop beside Paul. Paul raised one hand in a warding off gesture.

    “Can’t have any more interruptions or I’ll never get this done. See you in five minutes at the stand up.” Paul said without looking up. With that, his hand flew back to the keyboard to join the other, like a juggler taking in another bowling pin and seamlessly integrating it with the ones he is already juggling.

    Howard nodded and shuffled off to large Kanban board on the wall of the Operations bay. It was a large screen projecting the digital board onto the wall via a small Raspberry Pi. Howard was quite proud of this board. It had taken some time to convince the team that they could use Kanban in such an interrupt-driven world as Product Operations. The team went through several iterations before coming to the current board on the wall and even now they were willing to make changes.

    All new tickets for the team funneled into the backlog column on the left side of the board. These were automatically marked as untriaged. The team would review these tickets in the morning stand up and then prioritize them as a team. This excluded emergency tickets which were closely inspected. Emergency tickets would enter directly into the Expedited Lane of the board, and the team would swarm them until completed. Those tickets were almost exclusively the domain of outages and customer impacting events. New tickets filtered in through either email or the BOT in the Operations Slack room.

    The team began gathering around the board to review and prioritize the tickets in the backlog. Howard noticed that there were a few tickets in the active column that he didn’t recognize. He made a mental note to ask the team about this after they “walked the board.” The team took turns reading the new tickets in the backlog, and they discussed the work effort, blockers, dependencies and perceived priority of each. The tickets were then rearranged to be in order of priority with highest on top. Next they worked from left to right, with each person explaining the tickets they were responsible for.

    “I noticed that there are a couple tickets in here that were not there yesterday and were not covered in the triage. How did that happen?” Howard asked before everyone drifted back to their desks.

    Chris Fallon, known simply as Snark to everyone else, spoke up. “Those are the ‘important people’ tickets, boss. They are the ones we get from the constant ‘drive-bys’. You know, when an exec or a sales person steps into your cube and tells you that this is going to be the next big thing that makes millions.”

    “Wait a minute.” Howard said, turning to look at Snark. “I thought we all agreed that we would only take tickets in via the backlog, unless it’s an emergency.”

    Snark laughed, “Yeah, well, they seem to think their poor planning and ineptitude is somehow synonymous with being my emergency.”

    Howard took a deep breath and said, “OK, we can’t do this or it undermines the process. When this happens, it’s like the person standing over your desk is saying their time is more valuable than yours. That is simply not true. Let’s take the guidelines we all agreed upon for expedited tickets and print them out. We will put them up on the wall and the next time someone comes up with a ‘favor to ask’ point them to the rules. If they still don’t let up on you, point them to me. We have to be fair, though.”

    Howard thought for a long moment and said, “OK, here is what we will do. We will hold daily traffic court for anyone that puts in a ticket. This is completely optional, of course, but if the person feels like it is an emergency, and yet doesn’t meet the qualifications, they can come to the stand up the next morning and plead their case. If it seems like the need is real, the team can agree to put that item above all the other items to be next pulled into the Kanban process. This doesn’t mean we swarm it like an outage, but they do get to the top of the pile. Thoughts?”

    Everyone silent for a moment, and then the agreed. Everyone but Snark, who continued to look up at the ceiling, apparently deep in thought. At last he lowered his eyes to the group and said, “This will work, IF you get the right people on board first AND you support all of us in sticking to the rules.”

    Howard nodded and said, “I already have a mental list of the people we will need on board. I’ll get that taken care of today. Can you write this up and get it up next to the Kanban board?”

    “I’m on it.” Snark said, and turned with the rest of the team to leave.

    The concept of the traffic court ended up paying dividends in ways Howard hadn’t counted on. It turned out that, much like regular traffic tickets, people didn’t want to invest their time to come in and argue the case with the court. Instead, they decided to take the easier route of just paying the fine. This was their opportunity cost; time over money was the same as time over prioritization. There were cases where people would show up and win or lose their case. In the end, Howard and the Operations team began to understand who was just wanting to jump line and who had a genuine need. It was no surprise that each type tended to be repeat offenders. Over time, the ones exhibiting negative recidivism, began to learn that their behaviors were time-consuming and not being validated. Their behavior didn’t change but how they actively interacted with the team did and thus the outcome was still the one Howard’s team wanted.

    — Pauly Comtois

  • Your DevOps Subculture

    Your DevOps Subculture

    I’m often asked how to create a DevOps Culture and there is no single answer to this question. The focus initially for me in any cultural movement is to understand how the organization incentivizes and what its core beliefs are. When it comes to DevOps, I am normally looking for ways to bring these two teams together from a fundamental aspect first. One method I have found that works well when I’m consulting with these two teams is a Culture Session.

    Teams often end up with a subculture that incorporates aspects of the macro culture. These subcultures can evolve into the silos that DevOps is designed to break down. A Culture Session is one where you bring these teams together en masse and work on defining a new unified subculture.

    Culture Session

    The session starts before we even meet, with buy-in from the leadership. There will likely be detractors that will not want to attend or contribute. Having leadership not only broadcast the meeting, but support the change will be critical to success. We want to address this issue from the top down as well as from the bottom up.

    With both teams in the same room we will begin to address the culture we want. Language should focus on the fact that the team is made up of all engineers. Creating a sense of a single team spanning Development and Operations is a key focal point in DevOps. In order to build a unified new culture, we must discover the core beliefs of the new team. Ask the team what they consider our core beliefs of their culture is. Don’t restrict the statements that come out of this part of the session, everything has merit and should be included.

    It can be difficult sometimes to get a team to think in terms of beliefs. If we are greeted with blank stares or eye-rolls when asked what we believe in, try a different tactic. Ask the team what would be an ideal company to work for and build beliefs off of these statements. Here are some examples from my sessions, converted into beliefs.

    I want to work for a company that… Belief
    is fun to work for We value a fun work environment
    let’s me work from home We value results over where work is accomplished
    wants to hear my ideas We value everyone’s input always
    helps me to grow my career We value the growth of people as much as the business
    has a diverse workforce We value equality in all aspects of our culture
    has a community that supports each other We value consistent adherence

     

    Once you have a fair number of items (based on the number of participants) we can begin to order the list. It’s acceptable to write these on yellow sticky notes as well if you don’t have a whiteboard. The ordering is accomplished via a process similar to voting in Lean Coffee. Everyone gets three votes; dispersed any way they desire. I tend to use dots on the cards and then add up each card. List the top 5-10 cards from highest to lowest, and these are your top cultural values. Discuss the results with the team, are they surprising?

    Aligned Goals

    I believe that aligned goals create a more comfortable work environment and encourage the building of trust. Your teams goals should not only be aligned with the business, but also with your culture. This will make introducing and reinforcing the proper behaviors easier. Some of the highest performing teams I’ve worked with had alignment on all three fronts: Business, Behaviors and Culture.

    Behavior and Community

    Now that you have a core set of beliefs, it is time to begin talking about the behaviors we wish to see that drive those beliefs. The leaders at all levels must continuously encourage the community to reduce behaviors that are not aligned with core beliefs, while encouraging those that do. Over time the community will govern its own members, solidifying the positive culture and beliefs. The ability of the community to manage its own members requires the alignment I spoke of earlier.

    DevOps Language

    What I call DevOps Language is key to building and maintaining a strong community. A major component to DevOps is a culture that is accepting, open, honest, collaborative and trusting. We tend to get defensive when we feel that our abilities or intentions are being called into question. The following shifts may appear simple and non-significant in the beginning, but these simple changes in how we communicate greatly enhance our ability to clearly get our message across.

    Why to How

    Moving from “why” to “how” is a great first step in adopting a DevOps Language. Often we are faced with a lack of understanding that we desire to fill. Perhaps you are looking to get a better handle on the design or structure of a block of code or infrastructure. Maybe you are attempting to learn a process on a new team you’ve just joined. Regardless of the scenario, you are going to have questions. Like many “trigger” words, “why” tends to send some people into defense mode. Why can (and tone of voice is crucial here) make people feel as though you are questioning their intelligence and intent or even implying that you would have done it different (aka better). Moving the language from why to how you did something invites the listener to share their reasoning and experiences, not defend their position.

    I Know to I Agree

    Another simple change that I have found and adopted into my DevOps Language is “I agree”. I had a bad habit of saying “I know” when someone would tell me something I… well that I already knew! Again, the issue here is subtle, but with all things subtle they build over time. Saying “I know” sounds condescending at times and can be seen as an attempt to exert your superiority. In other words, this is not an collaborate language tactic. When we say “I agree” instead, we are saying that we are aligned in our thinking. This is including all parties and doesn’t shut the conversation down.

    Should’ve to Will

    DevOps is all about collaborating to make our future better. We often need to review events of the past to identify how to create a better future. The key to this review is to not focus on counter factuals rather we use forward thinking language. Concentrate on moving from “we should or could have” to “we will”. Over time this will help the team to maintain a forward thinking bias rather than dwelling too long on the past.

    Closing

    Creating systemic long term positive change within your organization takes time, effort and alignment. Creating a unified culture and belief structure through collaboration and inclusion paves the way for your DevOps future. Focus on the language and behaviors that reinforce your culture. Encourage members of the community to help govern and guide the culture, using the agreed upon beliefs. Include the beliefs and DevOps language in the interview and hiring process, in order to include new employees early and often.

    Let’s collaborate! I am always looking for ways to learn and grow. How have you aligned your teams around culture? How can I make this exercise more effective?