Category: Doin’ DevOps

Get it together! Why is your service always down?!”

We think we’re doing the whole DevOps thing right — new hires can deploy on day one, Travis CI is humming along, and we own the code we ship. But then something breaks, something doesn’t go according to plan, tempers flare up, and all that warm, fuzzy collaboration seems to evaporate. What’s going on? What happened to #HugOps?

A large part of software engineering doesn’t involve code at all— it’s talking and collaborating with our teammates. Soft skills are harder to measure and quantify than performance or reliability, but they’re just as important— and while we’ll readily spend hundreds to thousands of dollars on books, courses, and training to improve our software skills, companies rarely invest in creating the culture of empathy and compassion that was behind DevOps in the first place.

Empathy works great…until it doesn’t. It’s easy to be compassionate and collaborative when stuff is going great and everyone’s having an easy time, but it’s stressful, high-stakes situations that show you whether you’ve actually built a culture of empathy. Empathy skills for DevOps cannot be stressed enough.

Let’s look at some specific strategies and tactics teams can use to improve organizational empathy.

Learn to Communicate

Conflict mediators, couples therapists, hostage negotiators, policemen, and others in high-conflict situations often use specialized communications frameworks to help de-escalate situations and get parties in conflict talking to each other in a productive way.

I’m a fan of a communications framework called Nonviolent Communication, popular in education (I used to be a schoolteacher), parenting, mediation, as well as peace programs in war zones. Another popular guideline is Crucial Conversations, a more business-focused structure for handling conversations where emotions are charged and stakes are high. There are plenty of books, trainings, and other resources about empathetic communication out there, but they all have some common themes:

• When dealing with a tough situation, separate — explicitly— what happened from how you feel about it. So for the example above, you might say something like “When your service went down for the third time in the past two weeks, I got worried, because our service depends on yours, and we’re trying to hit four nines of uptime this quarter.” Often just the simple step of decomposing reactions from what’s being reacted to defuses the majority of the speaker’s anger— this is used in blameless postmortems to move from “blame and shame” to an honest look at the facts of the situation.
• When somebody shares a feeling, especially if emotions are high, take time to make sure they feel heard and validated…before you reply. That doesn’t mean you agree with them, just that you ensure that they know that you listened and are trying to understand. In our conflict above, this might be something like “It sounds like you’re really frustrated that our service went down, because it prevented your team from hitting your uptime goal.” Most of the time, the other person will instantly soften, relieved to feel heard. Sometimes you’ll get a “Well, duh!”, but that’s far preferable to the other person leaving the conversation feeling unheard and unacknowledged.
• If you have a request of the other person, make it positive, concrete, and explicit. And “Could you not?” doesn’t count. Without a concrete request, it’s hard for the other party to know how to meet your needs or improve the situation, and hard to tell whether that effort has been made. For our situation above, this could be something like “Can you please do a postmortem on the outage, and let me know by the end of next week what you’re going to do to prevent a future outage and anything you need from me to help?”

Again, there are all sorts of frameworks available, and like many tooling choices, it matters less the specific one you use than getting the organization on board with improving everybody’s communication and conflict resolution.

Align on Common Values

Cross-departmental collaboration gets much easier if you’re both trying to do the same thing. That doesn’t mean your day-to-day work has to be the same, but the more of your company that is driving toward the same goals, the easier it is to turn conflict into simple questions about sequencing or prioritization, rather than fundamental directional issues.

LinkedIn, Google, Twitter, and other big tech companies favor the “Objective + Key Result” (OKR) system for this purpose, where a big, ambitious goal and set of results at the company level trickles down into corresponding goals and results for the department and the individual.

We use OKRs here at PagerDuty, but there are plenty of other goal-setting systems that work. The important thing is agreeing on the meta-important stuff in plain language, so that when someone asks you to change your priorities or randomize what you’re working on, you can have a conversation about which of the options best supports the common goal, rather than the much hairier discussion about what the end-goal even is.

Mind Your Medium

Particularly with distributed teams, text-based communication can make tone hard to get across. A request said with a warm smile sounds quite different from one in flat text over email (or to everybody in the company HipChat room).

Where possible, I prefer video chat to text chat— in fact, some of our teams use perpetual Hangouts so that they can all see each others’ pretty faces, and feel like they’re in the same room together. Where that’s not feasible, I like to over communicate: if the tone of a sentence might be ambiguous, I explicitly state what I mean and what I don’t mean. And while it’s a little weird to me that this has become business normal, emoticons also go a long way in communicating feels over the wire.

Build Relationships, Inclusively

Ultimately, all the communications frameworks and shared goals in the world won’t take you to #hugopsland if you don’t have a human relationship with the person you’re trying to work with. An HBR study found that while people generally claim to prefer competence over likability, in practice they behave oppositely — likability is much more important for collaboration than competence.

And while a few people might be naturally gifted with irresistible charisma, for most likability isn’t a static, unchanging thing: it’s a quality built on familiarity and connection. If two people are going to be collaborating on something, get them some face time — not just screen face time— with each other. We have an engineering office in Toronto, and try to make sure new employees quickly get a chance to spend time with the team in the other office as well as their own. At DevOpsDays Minneapolis this year, Katherine Daniels shared the concept of “bootcamping,” where each new Etsy hire spends a period of time embedded in a different team, learning about how they work, what their goals and challenges are, and making meaningful contributions to their codebase.

Plan off-sites, social events, time to chill out and get away from the laptops. But keep in mind that not everybody on your team necessarily drinks alcohol, and some of them might (gasp!) have families. That doesn’t mean that you can’t ever plan a team gathering at a bar or do nighttime activities, just try to be aware of the different constraints, preferences, and physical abilities your team members may have. Providing hangout spaces at work is a simple but often overlooked way to help build relationships while respecting those who may have out-of-work obligations or just really long commutes. And while a shared lunchroom can be a good way to cross-pollinate, it’s also sometimes nice to get out of the building— PagerDuty pairs all new employees up with a “buddy,” and sponsors lunch out in the neighborhood to help them get better acquainted.

Being inclusive of different learning and personality types is important as well. Even if you don’t buy into specific personality tests such as Meyers-Briggs, introverts and extroverts handle relationships and communications differently, and it’s important not to assume that everybody handles information or makes decisions the same way you do. When doing a sprint retrospective, consider starting with a silent brainstorm, to make sure that people’s willingness to talk in a group doesn’t determine their ability to have their opinions heard.

Finally, and this shouldn’t have to be said, but be respectful of differences in religion, diet, sexual orientation, gender identity, family structure, and anything else that’s not directly connected with their performance as your team member. When somebody expresses their preferred pronouns, a dietary preference, or a constraint on their time, recognize that can be a pretty vulnerable thing, especially if they feel like they’re being asked to justify it.

Measure and Improve

While empathy might be tougher to measure than uptime, error rates, or other performance metrics, there are still steps you can take to track how you’re doing.

Our scrum teams each do “health checks” using Google Forms at the end of each sprint, where we smiley-face-rate our feelings about various factors including teamwork and support. Consistent sad-faces tell us there’s something systemic we need to work on. Managers can also specifically ask in 1:1s about how communication is going within and outside the team, to identify patterns.

One additional measure (and benefit) of strong empathy is the amount of diversity you see in your organization. If you can’t connect and communicate with people from different backgrounds, you’re unlikely to be able to recruit or retain diverse talent. On the other hand, when you’re able to see through the beer test, or the hijab, and make potential team members feel included and valued, your pool of awesome candidates gets much bigger.

One final reading recommendation here: Stalling for time: My life as a FBI hostage negotiator. Not only is Noesner’s story a thrilling read, but it takes many of the communication principles above and shows how a master of nonviolent communication permanently changed the FBI’s attitude about the place of empathy in rescuing hostages and defusing violent, high-stakes situations.