devopsy.kinsta.cloud

Author: David Shackelford

  • Multi-Tiered Escalation Policies are Paramount

    Multi-Tiered Escalation Policies are Paramount

    As any team charged with keeping systems up and running will tell you, having one person on-call isn’t enough. What happens if your on-call engineer sleeps through his alert? What happens if his phone’s battery dies without him knowing, or if he gets an alert at an inconvenient time, like when stuck on a bus or in traffic? It will happen.

    In today’s always-on world, companies can’t afford for that to happen without a backup plan. One or more people should always be waiting in the wings, ready to spring into action if a primary on-call is unable to perform his or her duties to the best of his or her abilities at any given time.

    Engineers who perform backup duty don’t need to be as “on-call” as a primary engineer — no pagers needed here — but what backup engineers lack in readiness they can make up for in numbers. It often makes sense to have multiple backups or an entire team of backups.

    Operationally mature companies often group the currently-on-call engineer and all of his or her backups into an escalation policy. Escalation policies set the order in which each person is alerted and the delays between alerts. While there are many ways to organize these escalation policies, here are some of the patterns most often used:

    Primaries Aren’t Enough; Determine a Secondary, Too
    Most companies have a primary on-call engineer, usually determined by a calendar that designates when members of an ops team are on call.

    Operationally mature companies take their on-call practice one step further and supplement this rotation with a rotation of secondary on-call engineers. Why a secondary rotation instead of just paging the whole team when the primary falls through? If a large team is paged, you get a ‘tragedy of the commons’ effect— it’s urgent that somebody respond to the issue, but not clear how urgent the issue is that each individual person respond. Having a designated secondary removes that confusion.

    The secondary rotation is set up to shadow the primary rotation and generally has the same members as the primary rotation. The difference is that the order of the engineers in the secondary rotation is usually offset from that of primary rotation— if your primary and secondary are the same person, you’re going to have a bad time. For example, if a primary rotation contains Alex, Bob, then Charlie, then a secondary rotation may contain Bob, Charlie, then Alex in that order. Companies often find that if last week’s primary is this week’s secondary, the it helps make sure there’s somebody with recent context who can help if the primary runs into trouble. It is becoming more commonplace for organizations to have both a primary and secondary on-call rotation; but, in an environment where customers expect zero downtime, secondary rotations are no longer an optional tool.

    Enable a Strong Support System
    Larger companies and companies with significant numbers of operations tasks often deploy a separate support team that handles all basic operations tasks before the organization elevates the tasks to an on-call engineer. This team — the first tier support team — usually knows how to handle all of the smaller, repetitive problems that crop up often and have clear resolution procedures. These first-tier teams often offer their support to multiple engineering teams and are placed first in an escalation policy.

    If first-tier teams cannot resolve the issue, escalation policies typically dictate that primary then secondary on-call engineers must complete the task. However, there is one more support network operationally mature organizations must implement: management. Management is ultimately responsible for a company’s systems, and needs to be in the know when severe problems happen— both to hold their teams accountable and to muster additional support. At PagerDuty many of our EPs have a rotating manager schedule at the top of multiple escalation policies, and many of our customers use this pattern as well.

    Put Your Tiered Teams to Work
    In practice, an organization policy for a hypothetical “Database Ops” team might look like this:

    • Assign the incident to the user who is on-call in the First-Tier Ops Team schedule
    • Assign the incident to the user who is on-call in the Primary DBA schedule
    • Assign the incident to the user who is on-call in the Secondary DBA schedule
    • Assign the incident to the team lead
    • Assign the incident to the dev manager

    This escalation policy would unfold over time, with timeouts of between 10 and 30 minutes before each subsequent escalation, depending on a team’s needs.

    Implementing an escalation policy that involves multiple tiers of resolution ensures that devops teams address all alerts — whether business-critical or minor — in a timely manner. Limited downtime is paramount in today’s fast-pace business environment. Customers expect their applications to run smoothly, or they expect to switch providers if devops teams cannot resolve issues fast enough. Organizations will be much less likely to see their processes fail, and much more likely to see happy customers, if they implement a multi-tiered escalation policy.

  • Four Metrics for Driving Cultural Change in DevOps Teams

    Four Metrics for Driving Cultural Change in DevOps Teams

    Everyone wants to optimize a team’s performance, but it’s not enough to just talk about it. To truly impact your team’s performance, you need to take a page from operationally mature DevOps teams and use metrics to gain valuable insight into your team’s work, enhance their capacity and drive cultural change.

    The right data allows managers to make quick decisions with confidence and minimal risk. It also lets managers see the actual outcome of those decisions, in turn allowing them to shape their team’s direction going forward and to create a happier, healthier and more productive team dynamic. However, making the shift to data-driven decision making requires more than just increased monitoring. It requires a cultural change.

    Below are four key metrics DevOps teams should be monitoring, as well as the impact each metric can have on a team’s culture and performance.

    1. Use Time to Response to Establish a Culture of High Achievement
      Time to response (also called time to acknowledgement) depends on individual team members. Incident responders might not always have control over the root cause of a particular incident, but they are always in control of how quickly they acknowledge and respond. Hold team members accountable for their response time by setting internal targets and having high expectations.Most operationally mature companies use IT operations management tools to enforce a response time target. These tools allow managers to set an incident response time window, making sure the next person in line is alerted if an incident isn’t responded to. Tracking escalations also gives managers valuable data about their team.
    2. Manage Expectations With Escalations
      For an organization using an IT operations management tool, an escalation is often an exception – it’s a sign that a responder either wasn’t able to acknowledge an incident in time or that the responder didn’t have the tools or skills to address it. While escalation policies are a necessary and valuable component of incident management, teams should ideally focus on driving the number of escalations down. Managers can gauge their team’s performance by tracking the number of incidents escalated over time, and using that data to determine whether their targets need to be adjusted.That said, there are some situations where an escalation is part of standard operating practice. For example, you might have a NOC, first-tier support team or even auto-remediation tool that triages or escalates incoming incidents based on their content. For these situations, it’s important for managers to track what types of alerts should be escalated and what normal numbers should look like for those alerts.
    3. Combat Alert Fatigue With Raw Incident Count
      As an organization grows, incident counts often grow, as well. However, as a team becomes more efficient and mature, the incidents per responder should lower or, at the very least, stay constant.Implementing IT operations management tools can help teams lower their raw incident counts by helping to weed out low-quality alerts, automate common fixes and build runbooks. They can also help break down incident count by team or service, putting incidents in context with the rest of the organization. This ensures that team members are maximizing their time on attacking alerts that matter and building new features, instead of getting mired in alert fatigue.
    4. Gauge Operational Readiness With Mean Time to Resolution
      Time to resolution, or how long it takes a team to resolve an incident, is the highest standard in operational metrics. The baseline for time to resolution can vary from organization to organization, depending on complexity environment, organization of responsibility, even the industry in which a company operates.Tracking time to response allows managers to determine their norm mean time to resolution and ensure their teams are able to work through the challenges of a major incident.

    Use These Metrics to Foster Intelligent Change

    It’s no secret that downtime is expensive, both in loss of revenue and customer trust. Metrics like the ones discussed are put in place to combat downtime and increase reliability, yet it can be easy for teams to become too focused on numbers and past performance.

    It’s important for team members to not lose sight of the business goals and why they’re tracking those metrics in the first place. It’s also important for managers to avoid over-analyzing the past. Metrics measure what has already happened, and, while they can tell a lot about a team’s previous performance, they should be used as a tool to a better future and not a means of assigning blame.

    Metrics are a means to an end, and having more information than you need won’t help you improve your team and refine your business. Keeping the emphasis on subsequent action is the key to using metrics to drive cultural changes.

  • Do You Need Your Ticketing System for Real-Time Incident Management?

    Do You Need Your Ticketing System for Real-Time Incident Management?

    The term “operationally mature” characterizes companies that understand how incident management affects both employee well-being and business impacts. Operationally mature companies have a process in place with built-in reliability and accountability that limits high-severity incidents because of bugs. The more mature a company’s operations, the better it can maintain agility and adapt to change. So how should operationally mature organizations handle incident management? Are ticketing systems still the gold standard?

    Ticketing systems keep track of issues, assign items to other people and order requests. They are a great way to manage a queue of ongoing requests, yet many operationally mature companies shy away from ticketing systems for their real-time incident management.

    Likely, it’s because ticketing systems can be fraught with issues. A clunky workflow, mired in process, means that users can’t always move and adapt quickly. Every minute they spend updating their workflow in the ticketing system equates to less time for dealing with critical issues. Ticketing systems often require a rigorous, structured, manual approach that can slow down work in the heat of the moment.

    Leading DevOps companies are proving that you don’t need a process-heavy system to get work done. In fact, for real-time incident management, the less process, the better. Companies are migrating from a system where a user manually opens a ticket when they see an outage, to monitoring systems that automatically generate alerts as problems arise. As soon as something goes wrong, the system generates an alert. There is no delay waiting for someone to notice the issue and fill out a form. Teams get immediate visibility into what’s going on with the system and can begin work or escalate accordingly.

    Operationally mature companies really need a lighter-weight solution. They need a solution that focuses on automation, allows them to be more agile and fixes incidents faster. The decision is not an either/or proposition, however.

    Many companies still have a place for ticketing systems, even if they aren’t for real-time incident management. Organizations can combine ticketing systems with more agile solutions. A more lightweight system still allows users to open manual alerts if necessary, much like a ticketing system does. Some operationally mature companies use a ticketing system to track incident follow-up and generate work estimates for lower priority tasks. Once the critical incident is resolved and service is restored, companies can create tickets to fix the root causes and make sure the problem doesn’t happen again.

    Ticketing systems are not necessarily outdated or obsolete. Unfortunately, one of the benefits of a ticket system is also a weakness. The plethora of fields, for example, allows users to track various aspects of an issue, but those fields are cumbersome and unnecessary for high-priority items. Ticketing systems can be clunky when DevOps needs something sleek. Combining ticketing with a lightweight, automated, incident management system is a sure-fire way to streamline some of the extra processes.

    Smart companies ask, “How much process is necessary to accomplish our tasks?” Answer that question, and your IT team will thank you. The answer will help them feel less like a help desk and more like an integral member of the DevOps team. And that helps your organization move faster and get closer to achieving DevOps Nirvana.

  • The Pros and Cons of Remote Work in Ops

    The Pros and Cons of Remote Work in Ops

    Remote work in Ops is divisive. The perks of working from home, your local coffee shop or wherever can be quite obvious to engineers and contributors. But some team leaders and company execs aren’t buying into it. One faction sees it as a way to do better quality work. Another has watched deadlines and motivation fall by the wayside because of unresponsive remote workers.

    What are the advantages and disadvantages of remote work? What should you consider before you take the plunge into flexible hours and asynchronous communication?

    Why Do Companies Go Remote?
    Company leaders worry that remote work erodes culture. Workers fret that colleagues and reports don’t get enough done outside the office. And team leaders lose sleep chasing down time-sensitive updates and deliverables.

    Remote work can definitely present some challenges. So, why do companies go remote? Several find it can provide serious benefits when implemented properly.

    By considering remote workers, businesses are able to attract high- quality talent from a larger pool of candidates – that’s especially important in the technology and software world where recruiting top-notch engineers is often a company-level initiative. If your company is based in a geographically competitive market, this opens doors and extends your reach to premier candidates outside of that area. It also goes the other way around – engineers who don’t live in tech-focused cities are given opportunities to be a part of those companies and cultures and aren’t penalized simply for where they live.

    Some contend remote policies produce better work. GitHub argues that code doesn’t follow a 9-to-5 schedule. Remote work can enable engineers to write great code during their peak hours of energy, focus and productivity, rather than on a set schedule that doesn’t play to their strengths. Because all employees – and engineers – are different, remote work provides the option to hone that creativity and focus outside of normal working hours.

    Since we’re so used to operating online, both in our work and personal lives, remote work embraces that to make employees more efficient. Remote workers communicate asynchronously.In other words, they respond to messages and transcripts when it makes logical sense, without interrupting serious tasks or disrupting intense focus. Another plus taking communication online is that it naturally leads itself to fewer meetings, and more doing. You’d be surprised how much you can cover in a three-minute chat via Skype, Slack, Campfire or another communication tool.

    That’s to say nothing of fostering a more enjoyable work environment. Plenty of people believe remote work improves morale. Who doesn’t want to work flexible hours and code when they’re in the zone?

    Finally, remote work supports inclusivity and diverse teams. What are the chances that every single person who can meaningfully contribute to your company just happens to live within a short commute? Not only do distributed teams open up opportunities for skilled workers who live outside of tech hubs, they also make it easier for people to work around family and other obligations.

    For some close-knit teams, there’s no difference between chatting in Google Hangouts and meeting on-site in a conference room. Going remote helps everyone get more done, more efficiently and on their terms.

    How Do You Get Started?
    Remote work may not be for every company. Your business needs or culture might not work best in a distributed environment. But if you do want to give it a try, there are some basic ways to start.

    Using the right tools is key. 37signals, the company behind Basecamp, literally wrote the book on remote work, and most of their team works remotely, using the company’s Campfire chat client to aggregate company communications and keep everyone on the same page. Email works for some conversations, but project management tools like Asana, Trello, Basecamp and JIRA let you add state, metadata, and delicious permalinks to everything that needs to get done.

    Be judicious about which tools you pick, though. Text, video and voice all have pros and cons. You’ll need to use communication channels that make sense for your company’s work style. For instance, when teams that are used to face-to-face stand-ups switch to typed messages, there can be miscommunication and misinterpretation. You’ll get more value out of the tools that enhance your in-office interaction methods. You might have to adopt several tools for your needs: The developer team at Hanno, a user experience company, uses nine different tools to work remotely, including Basecamp, Slack and Google Hangouts.

    While tools are important, they’ll only take you so far. You need to develop strong foundational relationships among teams, colleagues and management before implementing remote work policies. In-person relationships set the tone for remote ones. Try to grab a beer or lunch with your colleagues – it will help form healthy relationships and ensure remote work is truly effective. If your team is completely distributed, try to get together once every quarter or two to reconnect, put faces to Slack handles, and build culture. The plane tickets and hotels may seem a little pricey, but they’ll pay off in productivity and morale.

    And as with many other tests, start small first. Maybe you have a smaller engineering team that’s used to working this way already – let them test drive with a couple days per week so one piece of the company has a chance to hash things out before rolling out more widely. As GitHub points out in this post on how the company works remote, if something doesn’t work, just ditch it in favor of something that does.

    What Should You Consider When You Go Remote?
    Just because everyone’s excited to go remote doesn’t mean you have a bright out-of-office future ahead. Remote work comes with its fair share of challenges and considerations.

    If remote work isn’t ingrained into company culture, problems won’t be far behind. Projects can be delayed because dispersed teams aren’t used to communicating in a timely fashion with out-of-office colleagues. Discontent might simmer if in-office workers view remote teams as slackers. Subcultures can also easily form among remote and non-remote employees.

    Communication presents another daunting obstacle. Teams outside the office might over-communicate, while on-site workers bristle at a deluge of messages. Conversely, remote workers might fail to communicate enough, furthering the perception they’re not getting much done. As mentioned at the USENIX LISA 14 conference remote work panel, refrains of “your team’s never in the office” might become all too common.

    In the end, interpretation matters. For remote policies to work, employees in-office and outside need to understand that they’re both working hard toward a common goal, even if their methods differ. But that doesn’t happen on its own: you need to implement the tools, processes and culture to make it a reality.

    The bottom line is that remote policies work for some organizations and not others. What matters is finding a sustainable organizational culture that leads to business success, whether you do it from behind a desk or at home in your pajamas.

  • #HugOps in Practice: Empathy Skills for DevOps

    #HugOps in Practice: Empathy Skills for DevOps

    Get it together! Why is your service always down?!”

    We think we’re doing the whole DevOps thing right — new hires can deploy on day one, Travis CI is humming along, and we own the code we ship. But then something breaks, something doesn’t go according to plan, tempers flare up, and all that warm, fuzzy collaboration seems to evaporate. What’s going on? What happened to #HugOps?

    A large part of software engineering doesn’t involve code at all— it’s talking and collaborating with our teammates. Soft skills are harder to measure and quantify than performance or reliability, but they’re just as important— and while we’ll readily spend hundreds to thousands of dollars on books, courses, and training to improve our software skills, companies rarely invest in creating the culture of empathy and compassion that was behind DevOps in the first place.

    Empathy works great…until it doesn’t. It’s easy to be compassionate and collaborative when stuff is going great and everyone’s having an easy time, but it’s stressful, high-stakes situations that show you whether you’ve actually built a culture of empathy. Empathy skills for DevOps cannot be stressed enough.

    Let’s look at some specific strategies and tactics teams can use to improve organizational empathy.

    Learn to Communicate

    Conflict mediators, couples therapists, hostage negotiators, policemen, and others in high-conflict situations often use specialized communications frameworks to help de-escalate situations and get parties in conflict talking to each other in a productive way.

    I’m a fan of a communications framework called Nonviolent Communication, popular in education (I used to be a schoolteacher), parenting, mediation, as well as peace programs in war zones. Another popular guideline is Crucial Conversations, a more business-focused structure for handling conversations where emotions are charged and stakes are high. There are plenty of books, trainings, and other resources about empathetic communication out there, but they all have some common themes:

    • When dealing with a tough situation, separate — explicitly— what happened from how you feel about it. So for the example above, you might say something like “When your service went down for the third time in the past two weeks, I got worried, because our service depends on yours, and we’re trying to hit four nines of uptime this quarter.” Often just the simple step of decomposing reactions from what’s being reacted to defuses the majority of the speaker’s anger— this is used in blameless postmortems to move from “blame and shame” to an honest look at the facts of the situation.
    • When somebody shares a feeling, especially if emotions are high, take time to make sure they feel heard and validated…before you reply. That doesn’t mean you agree with them, just that you ensure that they know that you listened and are trying to understand. In our conflict above, this might be something like “It sounds like you’re really frustrated that our service went down, because it prevented your team from hitting your uptime goal.” Most of the time, the other person will instantly soften, relieved to feel heard. Sometimes you’ll get a “Well, duh!”, but that’s far preferable to the other person leaving the conversation feeling unheard and unacknowledged.
    • If you have a request of the other person, make it positive, concrete, and explicit. And “Could you not?” doesn’t count. Without a concrete request, it’s hard for the other party to know how to meet your needs or improve the situation, and hard to tell whether that effort has been made. For our situation above, this could be something like “Can you please do a postmortem on the outage, and let me know by the end of next week what you’re going to do to prevent a future outage and anything you need from me to help?”

    Again, there are all sorts of frameworks available, and like many tooling choices, it matters less the specific one you use than getting the organization on board with improving everybody’s communication and conflict resolution.

    Align on Common Values

    Cross-departmental collaboration gets much easier if you’re both trying to do the same thing. That doesn’t mean your day-to-day work has to be the same, but the more of your company that is driving toward the same goals, the easier it is to turn conflict into simple questions about sequencing or prioritization, rather than fundamental directional issues.

    LinkedIn, Google, Twitter, and other big tech companies favor the “Objective + Key Result” (OKR) system for this purpose, where a big, ambitious goal and set of results at the company level trickles down into corresponding goals and results for the department and the individual.

    We use OKRs here at PagerDuty, but there are plenty of other goal-setting systems that work. The important thing is agreeing on the meta-important stuff in plain language, so that when someone asks you to change your priorities or randomize what you’re working on, you can have a conversation about which of the options best supports the common goal, rather than the much hairier discussion about what the end-goal even is.

    Mind Your Medium

    Particularly with distributed teams, text-based communication can make tone hard to get across. A request said with a warm smile sounds quite different from one in flat text over email (or to everybody in the company HipChat room).

    Where possible, I prefer video chat to text chat— in fact, some of our teams use perpetual Hangouts so that they can all see each others’ pretty faces, and feel like they’re in the same room together. Where that’s not feasible, I like to over communicate: if the tone of a sentence might be ambiguous, I explicitly state what I mean and what I don’t mean. And while it’s a little weird to me that this has become business normal, emoticons also go a long way in communicating feels over the wire.

    Build Relationships, Inclusively

    Ultimately, all the communications frameworks and shared goals in the world won’t take you to #hugopsland if you don’t have a human relationship with the person you’re trying to work with. An HBR study found that while people generally claim to prefer competence over likability, in practice they behave oppositely — likability is much more important for collaboration than competence.

    And while a few people might be naturally gifted with irresistible charisma, for most likability isn’t a static, unchanging thing: it’s a quality built on familiarity and connection. If two people are going to be collaborating on something, get them some face time — not just screen face time— with each other. We have an engineering office in Toronto, and try to make sure new employees quickly get a chance to spend time with the team in the other office as well as their own. At DevOpsDays Minneapolis this year, Katherine Daniels shared the concept of “bootcamping,” where each new Etsy hire spends a period of time embedded in a different team, learning about how they work, what their goals and challenges are, and making meaningful contributions to their codebase.

    Plan off-sites, social events, time to chill out and get away from the laptops. But keep in mind that not everybody on your team necessarily drinks alcohol, and some of them might (gasp!) have families. That doesn’t mean that you can’t ever plan a team gathering at a bar or do nighttime activities, just try to be aware of the different constraints, preferences, and physical abilities your team members may have. Providing hangout spaces at work is a simple but often overlooked way to help build relationships while respecting those who may have out-of-work obligations or just really long commutes. And while a shared lunchroom can be a good way to cross-pollinate, it’s also sometimes nice to get out of the building— PagerDuty pairs all new employees up with a “buddy,” and sponsors lunch out in the neighborhood to help them get better acquainted.

    Being inclusive of different learning and personality types is important as well. Even if you don’t buy into specific personality tests such as Meyers-Briggs, introverts and extroverts handle relationships and communications differently, and it’s important not to assume that everybody handles information or makes decisions the same way you do. When doing a sprint retrospective, consider starting with a silent brainstorm, to make sure that people’s willingness to talk in a group doesn’t determine their ability to have their opinions heard.

    Finally, and this shouldn’t have to be said, but be respectful of differences in religion, diet, sexual orientation, gender identity, family structure, and anything else that’s not directly connected with their performance as your team member. When somebody expresses their preferred pronouns, a dietary preference, or a constraint on their time, recognize that can be a pretty vulnerable thing, especially if they feel like they’re being asked to justify it.

    Measure and Improve

    While empathy might be tougher to measure than uptime, error rates, or other performance metrics, there are still steps you can take to track how you’re doing.

    Our scrum teams each do “health checks” using Google Forms at the end of each sprint, where we smiley-face-rate our feelings about various factors including teamwork and support. Consistent sad-faces tell us there’s something systemic we need to work on. Managers can also specifically ask in 1:1s about how communication is going within and outside the team, to identify patterns.

    One additional measure (and benefit) of strong empathy is the amount of diversity you see in your organization. If you can’t connect and communicate with people from different backgrounds, you’re unlikely to be able to recruit or retain diverse talent. On the other hand, when you’re able to see through the beer test, or the hijab, and make potential team members feel included and valued, your pool of awesome candidates gets much bigger.

    One final reading recommendation here: Stalling for time: My life as a FBI hostage negotiator. Not only is Noesner’s story a thrilling read, but it takes many of the communication principles above and shows how a master of nonviolent communication permanently changed the FBI’s attitude about the place of empathy in rescuing hostages and defusing violent, high-stakes situations.

  • DevOps for non-engineers

    DevOps for non-engineers

    If you’ve scrolled through LinkedIn or companies’ career pages, you’ve probably seen the term “DevOps” used as a part of a job title. DevOps Engineers and DevOps architects are everywhere, and many people are surprised to find that those titles aren’t correct. It sounds innocuous: After all, isn’t DevOps something that you do? If you’re a marketer, hiring manager or non-engineer at your company, it might seem like it.

    But nothing could be further from the truth. DevOps is actually a philosophy and a set of practices that guides how your engineering and IT teams work. And using the term improperly can cause problems, even if you’re speaking with somebody who has “DevOps Engineer” on their LinkedIn profile.

    To clarify what DevOps is and what it means, it helps to know where the movement came from and why it matters.

    Where Did DevOps Come From?

    The rise of DevOps is rooted in the Agile Systems Admin and Enterprise Systems Management movements, as well as an effort by businesses to become more responsive to rapidly evolving markets and problems. Enterprise Systems Management attacked this problem by replacing large vendor solutions with faster, smaller and sometimes open-source tools, while at the same time, the Agile Systems Admin movement was gaining popularity in Europe as forward-thinking companies translated lean and agile manufacturing lessons to IT departments.

    The combination of these agile movements, the rise of SaaS and 24/7 service delivery, and examples like Flickr of excellence in developer-operations collaboration all helped prime the technical community for a change. After a talk by Flickr at the O’Reilly Velocity conference in 2009, highlighting challenges the company faced in continuous deployment and how they solved those problems with focused collaboration across the development and operations silos, Patrick Debois started organizing the first Devopsdays conference, and the name stuck. DevOps has been growing ever since, with conferences around the world, and has become synonymous with collaborative, agile, teams that ship quickly while maintaining high quality. This a very simplified account, of course— I recommend Richard Rapaport’s “A Short History of DevOps”, or Damon Edward’s video on the topic for a detailed history.

    And when you understand that history, it’s easier to understand why engineers bristle at the idea of DevOps as a job title or company role. By referring to and thinking about DevOps properly, you’ll not only make tech talent at your company happier, but will also help internalize DevOps principles within your organization.

    So, What Is DevOps, Then?

    Definitions vary according to who you ask, but The Agile Admin sums it up well:

    DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.

    Put another way by the site, it’s “agile systems administration.” That means DevOps is about software developers and ops engineers working together with shared values, principles, methods, practices and tools to build, maintain and improve systems.  “Works on my box, ops’ problem now” has no place in this movement, and the entire technical organization takes responsibility — together — for delivering great quality and service to customers.

    Engineering organizations often think about problem-solving in terms of technical constraints: do we have enough server capacity? Enough bandwidth? Enough test coverage? But DevOps is also about engaging with the human side of shipping: do responsible teams have the right communication processes? How do we minimize error, and how do we effectively learn from mistakes without compromising morale? Patrick Dubois characterizes effective DevOps champions as “ambassadors, peace makers, facilitators and communicators,” who build bridges across the organization and rally teams around a common cause.

    Why Isn’t DevOps a Job Title?

    For many organizations that are moving to a DevOps model, it represents a fundamental culture shift, and the messaging around it is important.

    The definition above doesn’t fit on a business card. DevOps is not about software developers taking over the job of ops nor ops taking over the job of developers. And it’s not just about shared tools (though that is part of it). It’s about breaking down organizational silos, scaling and automating processes, and leveraging best practices from both the development and operations world to ship better stuff to customers.

    Good DevOps work requires testing, iterating and, even, breaking things. That needs to happen quickly, without friction and, most importantly, without blame. Defining someone as a “DevOps manager”or creating a “DevOps team” could imply that only those individuals are responsible for DevOps, and, should it fail, those failures are due to that individual or team rather than systemic organizational problems.

    As Chip Locke writes, software developers and operations engineers are often different types of people, and trying to combine them into a single role won’t always be effective. When devs and ops are forced into a contrived role instead of encouraged to collaborate across their individual teams, competing values and (sometimes) suboptimal results are the outcome. Additionally, implementing DevOps as a third team that sits between the development and operations teams is a recipe for disaster.

    You’re not in Kansas anymore

    As organizations have to ship faster, more complex software to an increasingly demanding and quickly changing market, the prevalence of devops will only grow— fluency in the movement will help you collaborate with, hire, and retain great engineers, while pigeonholing it can lose you credibility with your technical team that’s hard to get back.

    As the outsider, the onus is on you to educate yourself. Watch the presentations that birthed the movement, read some history, talk with those in the movement. To start you off, here are some of my favorite resources.

    A short history of DevOps

    10 Deploys Per Day: Dev and Ops Cooperation at Flickr

    What is this DevOps thing, anyway?

    The Agile Admin: What is DevOps?

  • Nine Common Ops Mistakes (and How to Prevent Them)

    Nine Common Ops Mistakes (and How to Prevent Them)

    Constant change is a reality for any growing, dynamic organization. Being receptive to change helps us quickly innovate and learn, yet at the same time, poorly-managed change can create instability and downtime. Ask any engineer what the number one cause of downtime in their systems is, and they’ll say change – software changes, network changes, configuration changes. While it would be nice to avoid this instability by working on completely static systems, to cope with the needs of modern technology businesses,  operations teams must learn to better manage change by preventing common ops mistakes before they compromise systems.

    Arup Chakrabarti, operations engineering manager and my colleague at PagerDuty, stopped by Heavybit Industries, a community workspace for developer-focused entrepreneurs, to discuss the biggest mistakes an operations team can make and how to head them off:

    1. Getting It Wrong in Infrastructure Setup

    • Creating Accounts
      A lot of people use personal accounts when setting up enterprise infrastructure deployments. Instead, create new accounts using corporate addresses to enforce consistency.
      Be careful storing passwords. Keeping them in your Git repository could require you to wipe out your entire Git history at a later date. It’s better to save passwords within configuration management so they can be plugged in as needed.
    • Selecting Tools
      Another important consideration with new deployments is selecting your tools wisely. Leverage PaaS tools as long as possible so you can focus on acquiring customers instead of building infrastructure. And don’t be afraid to employ “boring” products like Java. Well-established, tried-and-true tech can let you do some really cool stuff.

    2. Poorly Designed Test Environments

    • Keep Test and Production Separate
      Don’t mingle your test and production environments. Be sure to set up test environments with different hosting and provider accounts than what you use in production. At the same time, make sure your test environments resemble production infrastructure as best as possible.
    • Virtual Machines
      Are you performing local development? There’s no way around it: applications will run differently on local machines and in production. To simulate a production environment as closely as possible, create VMs with a tool like Vagrant.

    3. Incorrect Configuration Management

    • Infrastructure-as-code
      Essentially, infrastructure-as-code is the process of building infrastructure in such a way that it can be spun up or down quickly and consistently. Server configurations are going to pose problems regardless of where your infrastructure is running, so you have to be prepared to restore your servers in as little time as possible. Ansible and Chef are two tools that makes infrastructure-as-code deployment super-simple for ops teams.Whatever tool you use, as a rule of thumb, it’s best to limit the number of automation software tools you’re using. Each one is a source of truth in your infrastructure, which means it’s also a point of failure.

    4. Deploying the Wrong Way

    • Consistency matters
      Every piece of code must be deployed in as similar a fashion as possible. Standardizing deployment practices takes time and effort, but eliminating variability and the potential for human error is essential for success. And that applies to rollbacks, too— anyone with deploy rights should also be able to easily roll back any code they ship.
    • Orchestrate your efforts
      Powerful automation software can certainly help enforce consistency, but automation tools are only appropriate for big deployments. When you’re getting started, Arup suggests running development using Git and employing an orchestration tool, such as Capistrano for Rails, Celery for Python or Ansible and Salt for both orchestration and configuration management.

    5. Not Handling Incidents Correctly

    • Have a process in place
      Creating and documenting an incident management process is absolutely necessary, even if the process isn’t perfect. Be sure to review the incident-management document on an ongoing basis, too. Incident response should be defined relative to the severity of the incident— a minor latency blip and a total downtime event have different response workflows and expectations.
    • Put everyone on-call
      It’s becoming less and less common for companies to have dedicated on-call teams – instead, everyone who touches production code is expected to be reachable in the event of downtime. This requires a platform that can notify different people in different ways. What really matters is getting a hold of the right people at the right time.

    6. Neglecting Monitoring and Alerting

    • Start anywhere
      The specific tool you use for monitoring is less important than just putting something in place. At PagerDuty, we use StatsD in concert with Datadog; open-source tools like Nagios can be just as effective. For application performance management, you might also look at a tool like New Relic or AppDynamics. There are hundreds of great monitoring tools out there, but the important thing is to get something in place and make sure you’re alerting on the exceptions that matter to you.
    • Check externally
      If uptime is important to you (and let’s face it, it always is), make sure you’re also running some sort of external check on your service or site, with a tool like NodePing or Ghost Inspector.

    7. Failing to Maintain Backups

    • Systematizing backups and restores
      Just like monitoring and alerting, backing up your data is non-negotiable. Scheduling regular backups to S3 is an industry-standard practice today, and always have one more backup method than you think you need.And backups are useless if you can’t restore from them! At least once a month, try restoring your production dataset into a test environment to confirm that your backups are working as designed.

    8. Ignoring High Availability Principles

    • “Multiple” is the key.
      Having multiple servers at every layer, multiple stateless app servers and multiple load balancers is a no-brainer. Only with multiple failover options can you truly say you’ve optimized for HA.
    • Datastore design matters, too.
      With multimaster data clusters like Cassandra, individual nodes can be taken out with absolutely no customer-facing impact. Clustered datastores are ideal in fast-moving deployment environments for this reason.

    9. Falling Into Common Security Traps

    • Relying solely on SSH
      Use gateway boxes instead of SSH on your database servers and load balancers. You can run proxies through these gateways and lock traffic down if you suspect an incursion.
    • Not configuring individual user accounts
      When an employee leaves your organization, it’s nice to be able to revoke his or her access expediently. But there are other reasons to set people up with user accounts to your various tools. Someone’s laptop may get lost. An individual might need his password reset. It’s a lot easier to revoke or reset one user password than a master account password.
    • Failing to activate encryption in dev
      Making encryption a part of the development cycle helps you catch security-related bugs early in development. Plus, forcing devs to think constantly about security is simply a good practice.

    These are just a few of the mistakes that operations teams can potentially make when managing change. There are millions more out there that you’re likely to run across, but these nine mistakes tend to be the most commonly seen, even at larger companies like Amazon and Netflix. By putting these best practices in place, you’ll help set your team up to effectively manage change.

  • Reducing Incident Resolution Time

    Reducing Incident Resolution Time

    From raw incident count to response time, there are several key metrics that top Operations teams track to measure and improve their performance. One of the most popular metrics teams track for is mean time to resolution (MTTR). It’s the time between failure and recovery from failure, and is directly linked to your uptime. While MTTR may be the gold standard when it comes to operational readiness, it’s important for teams to look at the bigger picture to effectively decrease incident resolution time.

    Putting MTTR into perspective
    Your overall downtime is a function of the number of outages as well as the length of each. Dan Slimmon does a great job discussing these two factors and how you may want to think about prioritizing them. Depending on your situation, it may be more important to minimize noisy alerts that resolve quickly (meaning your MTTR may actually increase when you do this). But if you’ve identified MTTR as an area for improvement, here are some strategies that may help.

    Working faster won’t solve the problem
    It would be nice if we could fix outages faster simply by working faster, but we all know this isn’t true. To make sustainable, measurable improvements to your MTTR, you need to do a deep investigation into what happens during an outage. True – there will always be variability in your resolution time due to the complexity of incidents. But taking a look at your processes is a good place to start – often the key to shaving minutes lies in how your people and systems work together.

    Check out your RESPONSE time
    Some call it MTTA (mean time to acknowledge) or MTTR (same acronym, but meaning mean time to respond), but the clock starts ticking as soon as an incident is triggered, and with adjustments to your notification processes, you may be able to achieve some quick wins.

    If your response time is on the longer side, you may want to look at how the team is being alerted. Do alerts reliably reach the right person? If the first person notified does not respond, can the alerts automatically be escalated, and how much time do you really need to wait before moving on? Setting the right expectations and goals around response time can help ensure that all team members are responding to their alerts as quickly as possible.

    Establish a process for outages
    An outage is a stressful time, and it’s not when you want to be figuring out how you respond to incidents. Establish a process (even if it’s not perfect at first) so everyone knows what to do. When you’re designing your process for responding to incidents, make sure you have the following elements in place:

    1. Establish a communication protocol – if the incident is something more than one person needs to work on, make sure everyone understands where they need to be. Conference calls or Google Hangouts are a good idea, or a room in HipChat or Slack.
    2. Establish a leader – this is the person who will be directing the work of the team in resolving the outage. They will be taking notes and giving orders. If the rest of the team disagrees, the leader can be voted out, but another leader should be established immediately.
    3. Take great notes – about everything that’s happening during the outage. These notes will be a helpful reference when you look back during the post mortem. At PagerDuty, some of our call leaders like using a paper notebook beside their laptop as a visual reminder that they should be recording everything.
    4. Practice makes perfect – if you’re not having frequent outages practice your incident response plan monthly to make sure the team is well-versed. Also, don’t forget to train new-hires on the process.

    To learn more, check out this talk about incident management from Blake Gentry, a former lead software engineer at Heroku.

    Find and fix the problem
    Finding out what is  actually going wrong is often the lion’s share of your resolution time. It’s critical to have instrumentation and analytics for each of your services, and make sure that information helps you identify what’s going wrong. For problems that are somewhat common and well understood, you may be able to implement automated fixes.

  • Four Steps to Free Your Organization From Scheduled Maintenance

    Four Steps to Free Your Organization From Scheduled Maintenance

    Whether you’re an operations engineer or a customer, scheduled maintenance is often viewed as a given when delivering software as a service. While far better than surprise outages, planned downtime is still tough for everybody involved– customers will always have off-hours access needs, and the low-traffic times you want to perform maintenance are often the same times your operations engineers and admins would like to be sleeping. On top of this, scheduled maintenance implies your system is less reliable than you think, because you’re afraid to change it during the workday.

    How do we make scheduled maintenance better for everyone involved?

    Planned maintenance is a non-starter at PagerDuty, since our customers don’t know when their monitoring tools will fire alerts. We’ve had to focus hard and innovate to deliver continuous uptime for our service, and Doug Barth, my colleague and a member of our operations team, recently did a talk at DevOps Days Chicago on the strategies we used to ditch scheduled maintenance altogether, and replace it with iterative maintenance that doesn’t compromise your entire system.

    1. Deploy in stages

    Your deployments need to be rock-solid. They should be scripted, fast and rolled back quickly, as well as tested periodically to ensure rollbacks don’t lag.

    They also need to be forward and backward compatible by one version. It’s not an option to stop the presses when you push out a new version. Red-blue-green deployments are crucial here, as they ensure only a third of your infrastructure undergoes changes at any given time.

    Lastly, stateless apps must be the norm. You should be able to reboot an app without any effect on the customer (like forced logouts or lost shopping carts).

    2. Avoid knife-edge changes

    Use canary deploys judiciously to test rollouts, judge their integrity and compare results. These test deployments affect only a small segment of your system, so bad code or an unexpected error doesn’t spell disaster for your entire service.

    During his talk, Doug suggested a few practical ways to accomplish this:

    • Gate features so you can put out code dark and slowly apply new features to a subset of customers.
    • Make sure you’re testing the changes on a representative sampling of your customer base: different account sizes, usage patterns, and configurations.
    • Find ways to slowly bleed traffic over from one system to another, to reduce risk from misconfiguration or cold infrastructure.
    • Run critical path code on the side. Execute it and log errors, but don’t depend on it right away.

    3. Make retries your new best friend

    Your system should be loaded with retries. Build them into all service layer hops, and use exponential backoffs to avoid overwhelming the downstream system. The requests between service layers must be idempotent, so that you can reissue requests to new servers without double-applying changes.

    Use queues where you don’t care about the response to decouple the client from the server. If you’re stuck with a request/response flow, use a circuit breaker approach, where your client library delivers back minimal results if a service is down—reducing front-end latency and errors.

    4. Don’t put all of your eggs in one basket

    Distribute your data to many servers, so that no one server is so important you can’t safely work on it.

    At PagerDuty, we use multi-master MySQL clusters, which help with operations and vertical scaling. We also use decentralized, linearly scalable databases like Cassandra for our alerting pipeline: since transactions aren’t gated on the behavior of a single node, we can do operational work during the day.

    Put together, these four strategies set an organization up to ditch annoying weekly downtime and help admins and operational engineers sleep more, worry less and maintain better—all ahead of schedule. Not only is this great for your team’s quality of life, but it also helps you deliver a great customer experience– no matter where your users are or what time it is.

  • There is No ‘I’ in Team: Transitioning Engineering Teams into a DevOps Model

    There is No ‘I’ in Team: Transitioning Engineering Teams into a DevOps Model

    As the pace of development and business continues to scale, teams need an agile and collaborative work environment to succeed. Moving to a DevOps model is a critical part of setting your engineering teams up to succeed, but making the transition can be challenging for many companies.

    What’s the deal? – Why you need a DevOps model

    DevOps is a movement in the software world  that focuses on collaboration between developers and operations, customer empathy and infrastructure automation. In traditional models, developers write code and then hand it to operations teams to deploy and and run in a production environment. This often leads to a lack of ownership between the two teams (“It worked on my local box!”) as well as a slower pace of development. In a DevOps model, by contrast, the two teams work closely with each other, and toward common, customer-facing goals – developers take ownership of their code even while it’s in production, and operations teams build tooling and processes that help developers write, test and ship code faster and more easily.

    By breaking through traditional developer-operations walls, development happens faster and with more customer empathy, ultimately resulting in a better customer experience and a happier team. Rather than seeing the other team as a gate or obstacle, if done right, developers and operations see each other as teammates working to deliver value to the customer and business.

    Unfortunately, introducing DevOps culture and practices to your company isn’t easy. Big changes to long-entrenched silos can generate resistance, and business leaders are often scared of the costs that come with change. To combat these concerns and hesitations, we’ve highlighted a few tips for transitioning to a DevOps model.

    Start with Collaboration – what’s working and what’s not?

    At the heart of DevOps is improved collaboration between your Operations and Development teams. You’re probably not starting from a vacuum, though – look at and talk about the best examples of how your operations and software development teams are collaborating today. Are there particular dev/ops groups that are already working tightly together, championing great communication or doing joint planning? Focusing on what is already working and how to make it work better, rather than what’s broken, keeps things positive and more likely to succeed.

    You could set up a group brainstorming session to discuss, or have one-on-one conversations with individual team members, depending on the format that you think will help people open up more easily. This article highlights opportunities for collaboration across the application delivery lifecycle, and could provide a useful framework for asking questions and identifying opportunities.

    In addition to the state of collaboration, you’ll also want to look for process successes and breakdowns in how you deploy and maintain applications, services and infrastructure. Identify the biggest opportunities for improvement – do your builds and test take hours? Are your 20 development teams all competing for the same three staging environments? Being able to quantify the impact of these pain points can help defend the transition to business stakeholders who may not have as much on-the-ground context about your teams’ pain points.

    It’s also important to build an understanding of the different perspectives operations and development teams may have. If your organization has traditionally been very siloed, thinking about other teams’ goals when planning and prioritizing may be a new mindset. Sharing each team’s objectives and what success looks like for them can help build this understanding and empathy. Focus on transparency (a framework like OKRs can help) to avoid conflict from mismatched assumptions about priorities.

    Create a plan  

    Once you’ve identified opportunities to improve the collaboration and workflow between developers and operations, it’s time to create a specific plan. Pick key opportunities and identify the first few steps to make a difference. Courtney Nash recommends starting with a group of people who are receptive to the vision and picking a small product or service as a test-bed. When implementing big changes, it helps to start small and scale from there after proving success.

    Engineers are not always known for their people skills, but transitioning to DevOps is as much about people as it is about process and tools. Throughout this whole process, you’ll want to be investing key stakeholders in your team and company in the transition. Change management sounds awfully corporate, but at it’s heart, it’s about making sure you build support for your projects and goals by collecting input from key stakeholders: gathering their feedback and goals, sharing and re-shaping your plan based on their input, and communicating your progress with them. It’s always easier to work in silos with people that already agree with you, but for lasting change, you’ll need to invest in building support across teams and throughout your organization.

    Finally, pick realistic timelines and objectives. It’s important to note that some effects of the change may take longer to realize as people ramp up on new tools and ways of working. Don’t discount or neglect the amount of time required for training, and try not to be fazed if the first project you pick doesn’t turn out to be a good one. As Courtney Nash mentions, Nordstrom.com realized their first project was not ultimately the right one to start with. Importantly, they didn’t give up, instead choosing a different, better-suited application.

    Measure and reflect

    While it’s rare for timelines in complex systems to happen exactly as planned, it’s important to timebox your work so that you can measure and reflect on how it went.

    After the specific milestone you’ve chosen, bring stakeholders together and run a retrospective. There are a lot of great resources out there to help with good retrospectives, but regardless of approach, make sure you make an intentional space to look critically at your results, celebrate success, and think about how you’d like to change or iterate the process for the future.

    When reflecting on progress, it’s important to be candid and honest while assuming the best intentions of your teammates. You are guaranteed to make mistakes – that’s inevitable with any new process – but remember (and help others remember) that you’re all working toward the same purpose and share a common set of values and goals.

    Don’t give up

    Adopting DevOps is a big change for an organization, and there will be bumps along the road. At the end of the day, though, the rewards of tighter collaboration among teams and a more empathy-driven engineering culture can be really powerful, and will show in product quality, velocity and team morale.