Tag: devops

I went seeking honesty at a DevOps conference and met Diogenes.

Diogenes of Sinope was a controversial figure in ancient Athens who was known for walking around the famed ancient city—some say in the nude— during daytime with a lamp in his hand. He parried inquisitors who questioned his strange habit of using a lamp by day by insisting that he was looking for an honest man. At the recent DevOpsDays Boston, I actually met Diogenes. True story. See picture below. (Thankfully, he was wearing clothes.) But, as you probably guessed, this was not the man with the lamp, but rather one of the many conference organizers looking to help less-acquainted individuals find their way around the many meetings and speakers that filled the two-day conclave. Indeed, my Diogenes was instrumental in organizing the conversations and lectures that proved both honest and helpful.  Rather than a self-congratulatory summit, the two-day meeting was an honest conversation highlighting the advantages as well as many of the stresses befalling organizations that have adopted DevOps.

I found the most honest discussions were a floor up and a bit removed from the topics discussed in the main ballroom. These discussions were given the informal title of “Open Spaces” and were run in a typical un-conference format. These conversations were the real opportunities to learn about the stresses of implementing the modern portmanteau of DevOps.

Tensions Between Dev and Ops: Why Can’t We All Get Along?

One of the most interesting discussions in the Open Spaces looked at the dynamics between Devs and Ops. Like with any two groups that need to work together to accomplish larger organizational goals (think Sales and Marketing or Design and Engineering), there are inherent tensions that need to be overcome. The tensions between Dev and Ops are based on the different mission each pursues. Dev is striving to get software out the door as quickly as possible and push software updates forward. Ops is guided by the need to keep the ship as steady as possible, which often means limiting updates, as every update opens the door to potential bugs and deployment challenges. Starting with this premise of conflicting motivations, issues of trust and respect further complicate the relationship as each sees the other as trying to control the process.

Yet, participants in this Open Spaces discussion were hopeful that tensions could be dissolved if management played an active role in creating cross-functional teams and managing workloads better. It was suggested that by essentially seeding Ops with Devs and vice versa, a greater community of understanding could be created. By promoting this cross-pollination, there would be greater understanding on both sides the implications (from the Dev side) of pushing software over the wall to Ops too quickly. From the Ops side, there would be greater understanding that 100 percent uptime is incompatible with dynamic product cycles that encourage innovation. A common and acceptable level of uptime needs to be agreed upon by both sides.

For example, creating demos and code walk-throughs so both Dev and Ops can comment on potential problems and issues allows both Dev and Ops to see issues at the beginning rather than at the time of deployment.

Blameless Post-Mortems in DevOps: Can They Exist?

Another fascinating discussion in the Open Spaces focused on blameless post-mortems. Postmortems first and foremost suffer from an image problem. I believe it starts with the root word “mort” and it doesn’t get much better. To be clear, there was a lot of back and forth during this Open Space about the difference between a “postmortem” and a “retrospective.” Retrospectives are usually a weekly or bi-weekly event that bring to light the multiple root causes that are leading to defects or causing success through a review of RCA. In typical DevOps, one-week sprints are followed by a retrospective. Usually, the more data you can bring to a retrospective, the better. Post-mortems, on the other hand, result from an incendiary incident that was unexpected and undesired.

Since post-mortems inevitably occur due to human oversight or lack of planning, there is no amount of thinking or planning that can prevent a crisis and thus prevent a post-mortem. As such, effective management will make post-mortems as painless as possible. The goal is to have blameless post-mortems balanced with accountability. According to one participant,

“No one wants central casting to send in a screaming manager in a time of crisis who is only interested in assigning blame. That doesn’t help anything.”

Importantly, you cannot fire yourself out of a post-mortem. Post-mortems are a necessary instrument to get at the root causes of issues. And while several different events can cause one issue, the post-mortem will at least begin to look at the process which caused the chaos. By not making the goal of post-mortems to find a scapegoat, the participants in one of these sessions can actually work on effective problem-solving.

The goal of post-mortems needs to be effective problem-solving with the intent of committing the lessons learned to an actual document. It is not enough to let the lessons learned remain as an informal corpus that lives only in the heads of the engineers involved. Instead, documentation is required and it has to be in accessible place. Furthermore, it needs to be summarized and presented to management in either a lunch-and-learn or seminar so that the lessons learned then become actionable.

Furthermore—and this was perhaps the most interesting part of the discussion on post-mortems—effective post-mortems are about getting Dev and Ops out of silos. There is no automation tool that will make Dev and Ops work together. Dev and Ops cannot just be about finding another automation tool. Instead, management needs to step in to help the process. Management needs to take engineers out of their silos and get them to work together if post-mortems are going to be really effective.

Are You Feeling the Burnout Yet?

After so many hours of interesting discussion in the DevOpsDays Open Spaces, I too was beginning to tire a bit. So perhaps it was well-planned to move on to a discussion on avoiding burnout.

The artifacts of burnout are seen throughout the DevOps world as engineers constantly change jobs. This might be the phenomena of the “shiny orb,” where engineers always feel that there is a better, higher-paying job on the other side of the fence. However, as much as that might be true, it is equally the case that engineers are trying to escape burnout.

According to the kanban methodology, which is woven into DevOps, you only pick up work as you can process it. Yet, DevOps practitioners often pick up work because they want to feel like they are team players or invincible or because they underestimate the work’s demands. They also fear the “imposter syndrome,” whereby colleagues seem less genuine if they don’t accept a piece of work offered to them. Inevitably, because the work is more than the engineer can handle, he or she becomes overwhelmed and is too proud to reach out for help.

In a well-managed workplace, admitting overwork would not be an admission of failure. Instead, there would be a supportive culture to help the engineer or better mentoring on how to get through a challenging issue. “One-on-ones,” where the manager and individual contributor sit down together, would effectively solve the issue.

But given the idiosyncrasies of workplaces, good management is not a given and engineers are often left to find their own coping techniques to help avoid burnout. The participants in this particular Open Space came up with the following gems:

• Brain dumping. Empty your brain by putting all the thoughts and tasks that remain on your list and in your brain at the end of the day onto post-it notes. This helps avoid burnout because you are able to clear your brain prior to leaving work.
• Create a Slack-based or bullet journal as a way to write yourself notes and create a list of priorities.
• Follow the Pomodoro technique. Take a five-minute break every 25 minutes and a 15-minute break every two hours to better monitor how much time you are devoting to an issue and also to ensure you don’t forget to get a bit of movement into your day.
• Learn how to say ‘no’ to requests. It’s a very simple word, but uttering ‘no’ once in a while will ensure that you mark boundaries and don’t take on more responsibility than you can handle.
• Enlist management to make them aware of time you are spending beyond the normal working day and try to figure out ways to better manage the schedule and avoid burnout.

The participants of this Open Space left with some new techniques to consider in avoiding engineer burnout. Hopefully, they will be implemented to practical advantage.

Conclusion

My first DevOpsDays conference was a true eye-opener.  I was lucky to get my education among such thoughtful and well-spoken individuals. I look forward to future opportunities to learn at the foot of Diogenes.

About the Author / Orlee Berlove

Orlee Berlove is the Marketing director at OnPage Corporation located in Waltham, Massachusetts. OnPage is a cloud base incident alert and management system that elevates notifications so they continue to “ALERT-UNTIL-READ” so it’s not confused with email, SMS or phone calls. Incidents can be programmed to arrive to the person On-Call and can be escalated if they are not attended to promptly. Follow OnPage on Twitter and LinkedIn.