Tag: pagerduty

  • PagerDuty’s Strong Performance Drives $43.8 Million Series C Investment to Secure Leadership in Digital Operations Management

    SAN FRANCISCO – April 13, 2017 – PagerDuty, the leader in digital operations management, today announced the close of a $43.8 million Series C investment. The investment, led by Accel, with participation from existing investors Andreessen Horowitz, Bessemer Venture Partners, Baseline Ventures and Harrison Metal, comes as a result of PagerDuty’s sustained strong business growth and product innovation. As companies increasingly rely on digital services, many have recognized the need to provide their developer and IT Operations teams with insights that help manage incidents and deliver a superior customer experience.

    “We are thrilled to welcome Accel into the PagerDuty family and appreciate their support and confidence in us. PagerDuty’s strong performance and leadership position in the fast-growing developer solutions market has been built around a mission to help improve customer experience by mobilizing organizations to effectively manage incidents so they have control over their growing digital operations,” said Jennifer Tejada, CEO of PagerDuty. “With growing interest in our solutions, we continue to grow our business thoughtfully and cash-efficiently with a laser-focus on our customers and community of practitioners, as their collective success is our success. This latest round of investment underscores our commitment to cementing our market leadership through solid industry partnerships, continued product innovation and customer success.”

    According to the inaugural PagerDuty State of Digital Operations Report, announced in a separate release today, more than one-third of organizations take an average of 30 minutes or longer to resolve IT incidents that impact consumer-facing digital services. Chances of losing customers increase during this time as IT teams are unable to meet consumers’ high expectations for digital service delivery. The business impact of IT incidents continues to rise—an hour of IT downtime costs one-third of enterprises more than $1 million and in North America alone, organizations spend a combined $700 billion per year.

    This investment comes on the heels of a banner year for PagerDuty’s digital operations management platform which supports more than 8,700 organizations and nearly 200,000 users across the globe to resolve business-impacting IT and digital incidents. Providing full-stack visibility and actionable insights, PagerDuty helps organizations proactively mobilize people and manage issues in their IT  and DevOps environments to deliver seamless customer experiences.

    PagerDuty finished a strong Q1 2017 which included the following recent milestones:

    • Added industry-best talent, including Steven Chung, Senior Vice President of Sales; Steven Gatoff, Chief Financial Officer; Howard Wilson, Chief Commercial Officer; Jeremy Kmet, Vice President of Commercial Sales; Kyle Duffy, Vice President of Solution Consulting; Joe Sexton, former AppDynamics Go-To Market Leader on the Executive Advisory Board
    • Added over 600 new names to its customer roster, including brands across the insurance and marketing industries such as CSAA Insurance Group and Eventbrite
    • Continued focus on the success of over 50 percent of the Fortune 100 customers who use PagerDuty
    • PagerDuty continues to be recognized across the industry with accolades, and most recently the customer support and advocacy team garnered Silver recognition in the Stevie’s award for 2017

    Investors Highlight Critical Need for a Digital Operations Management Platform Across Industries

    “PagerDuty gives development teams of all sizes, including those at the world’s most recognized brands, visibility and actionable insights across their entire infrastructure stack. For the first time, companies now have a tool to both mitigate downtime and efficiently scale their digital operations across the entire organization. PagerDuty joins a strong list of companies in our portfolio with dynamic developer communities, including Atlassian, Dropbox and Slack, and we’re thrilled to have the chance to help Jennifer and the team.” – Andrew Braccia, partner at Accel

    “Over the years, we’ve seen PagerDuty’s customer-centric approach to incident resolution explode in adoption across a diverse set of companies ranging from Panasonic to Instacart. As expectations for ‘always-on’ digital experiences reach an all-time high, PagerDuty’s innovations in digital operations management continue to empower today’s businesses with the full stack visibility needed to deliver on the promise of constant uptime.” – John O’Farrell, partner at Andreessen Horowitz

    “We see in PagerDuty what we’ve seen in our other highly successful portfolio companies focused on developers – a fierce commitment to serving the developer community by continuously creating products and services that enable individuals, teams and companies to deliver on brand promises. Since the beginning, we’ve been proud to watch PagerDuty embed itself into the daily workflow of developers by increasing their productivity and improving the reliability of the systems, services and applications they build. Businesses and consumers alike have come to rely on the product innovations of the talented PagerDuty team.” – Ethan Kurzweil, partner at Bessemer Venture Partners

    Information Technology Intelligence Consulting: http://itic-corp.com/blog/2016/08/cost-of-hourly-downtime-soars-81-of-enterprises-say-it-exceeds-300k-on-average/

    IHS Markit: http://news.ihsmarkit.com/press-release/technology/businesses-losing-700-billion-year-it-downtime-says-ihs

    About PagerDuty

    PagerDuty is the leading digital operations management platform for businesses. Through its SaaS-based platform, PagerDuty empowers developers, DevOps, IT operations and business leaders to prevent and resolve business-impacting incidents for exceptional customer experience. More than 8,000 small, mid-size and enterprise global customers such as NBC Universal, Lululemon, Slack, IBM and Panasonic use and trust PagerDuty to maximize their time and increase their business response and efficiency. When revenue and brand reputation depends on customer satisfaction, PagerDuty arms businesses with the insight to proactively manage incidents and events that may impact customers across their IT environment. Headquartered in San Francisco, the company was recently listed in the 2016 Deloitte Technology Fast 500, Inc. 500 and Forbes Cloud 100 lists.

    — Jules Louis

  • Webinar: ROI is the True Measurement of DevOps Success

    Webinar: ROI is the True Measurement of DevOps Success

    RECORDING

    SLIDES

    [seoslides embed_id=”68ea17c4e993″ script_src=”https://devopsy.kinsta.cloud/embed-script/17871/17900/” overview_src=”https://devopsy.kinsta.cloud/slides/17871/” site_src=”https://devopsy.kinsta.cloud” site_title=”devopsy.kinsta.cloud” title=” ” /]

  • PagerDuty Introduces Response Mobilizer and Response Bridge for Seamless Collaboration and Dynamic Incident Response

    New features facilitate coordinated response by bringing together the right people and information

    San Francisco – July 11, 2016 – PagerDuty, the leader in modern incident management, announces the availability of Response Mobilizer and Response Bridge, two new features that enable seamless collaboration between responders for dynamic incident response and faster resolution. Using PagerDuty’s new Response Moblizer feature, customers can notify and enlist additional experts and critical stakeholders to an incident.  With the PagerDuty Response Bridge, customers can share a conference bridge for real-time collaboration on open incidents. Together, these features allow customers to respond at scale with the right people and contextual information for faster incident resolution.

    Outages in the digital economy have serious customer experience and revenue implications that push ITOps and DevOps teams to constantly seek better tools and processes to mitigate the impact on their business. Critical incidents typically require a greater level of organization because the impact can span across multiple teams and business units, however, delays in enlisting the right subject matter experts cross-functionally can be detrimental for a time-sensitive resolution. Beyond the standard on-call schedules, PagerDuty Response Mobilizer allows customers to rapidly bring the right people together with the right contextual information to address business-critical issues in real-time. By helping customers recruit the right responders, PagerDuty reduces mean-time-to-resolve (MTTR) by minimizing the amount of time spent searching for responders . After gathering the right responders and key stakeholders, PagerDuty Response Bridge enables teams to collaborate via their existing conference bridge or chat tools. With custom messages that include critical incident context for new responders, PagerDuty provides a single incident view to coordinate the ideal response.

    PagerDuty Response Mobilizer and Response Bridge enables customers to:

    • Notify the Right People: Easily select and invite individuals by name or escalation policy to collaborate directly on any PagerDuty incident
    • Mobilize Teams Immediately: Decrease response times by customizing how team members are notified (phone, push notifications, SMS), and identify their response status in real-time and escalate if necessary
    • Access Contextual Information: Include a brief message to responders detailing why they are needed and leverage the rich, contextual details already embedded inside every incident so all responders are immediately aligned
    • Conferencing Support: Include meeting details in any PagerDuty incident and invite responders to collaborate in a meeting, dramatically reducing MTTR
    • Leverage existing Conferencing Platform: PagerDuty supports multiple conferencing providers, including Cisco WebEx, GoToMeeting, and Blue Jeans, so teams can operate within their established communication channels

    “Many companies take 30-60 minutes just to get the right people on a conference bridge during business-impacting outages or incidents,” said Tim Armandpour, senior vice president of product development, PagerDuty. “When down-time is your business’s enemy, how fast you respond to an incident matters to the bottom line. PagerDuty Response Mobilizer and Response Bridge drastically reduce the amount of time and effort needed to collaborate and resolve an incident, but more importantly, mitigate a business issue.”

    PagerDuty Response Mobilizer and Response Bridge are available to all Standard and Enterprise customers today. To learn more, read our blog posts about Response Mobilizer and Response Bridge.

    About PagerDuty

    PagerDuty is an agile incident management solution that integrates with IT Ops and DevOps monitoring stacks to improve operational reliability and agility. From enriching and aggregating events to correlating them into actionable alerts, PagerDuty streamlines the incident management lifecycle by reducing noise and resolution times. With hundreds of native integrations with operations tools, automated scheduling, advance reporting and guaranteed reliability, PagerDuty is trusted by over 7,000 organizations globally to increase business and employee efficiency. The company is headquartered in San Francisco and backed by leading venture capital firms Andreessen Horowitz, Bessemer Venture Partners, Baseline Ventures, and Ignition Partners. Try PagerDuty for free at www.pagerduty.com.

  • PagerDuty Services Group Capability Simplifies Incident Management of Critical Applications and Services

    New Services Group feature simplifies how IT Ops and DevOps teams resolve incidents, delivering increased uptime and performance
    May 11, 2016 09:00 AM Eastern Daylight Time

    SAN FRANCISCO–(BUSINESS WIRE)–PagerDuty, the leader in modern incident management, announces the immediate availability of PagerDuty Services Group. Services Group fundamentally changes the way IT Ops and DevOps teams manage and monitor critical apps and services by mirroring the organization and components of these services in PagerDuty. Services Group aggregates all events related to these individual services from the monitoring stack and renders a comprehensive view of the organization’s application and service infrastructure. This new functionality delivers the fastest way to organize, monitor and manage critical apps and services as they relate to the business.

    “The PagerDuty Services Group feature makes it easier to resolve incidents as they occur in our critical applications and services,” said Andy Domeier, director of systems operations, SPS Commerce, the retail industry’s largest trading community with more than 60,000 customers in over 60 countries. “By re-creating a service in PagerDuty, as it exists in our systems, our team members are more aware of service-level health, allowing us to resolve an incident faster.”

    With the new PagerDuty Services Group, IT Ops and DevOps teams immediately benefit from:

    • Consistent Views – A service in PagerDuty is structured identically to how it is deployed in the IT infrastructure. Teams no longer view the monitoring and management layer of a service, instead they see a comprehensive view as it relates to overall service delivery, such as e-commerce portals, payment processing systems, websites or other key applications.
    • Simplified Management and Collaboration – The monitoring and management layers are aggregated into one view at the service level from multiple vendors. IT service management (ITSM), log management and application performance management (APM) tool views are consolidated, creating better collaboration between different teams and technologies required to support the service.
    • Faster Incident Resolution – Teams work together to resolve disruptions at the service level rather than in technology silos.

    “PagerDuty Services Group greatly simplifies incident management to better support business-critical services,” said Jonathan Wilkinson, vice president of product, PagerDuty. “By bringing together an organization’s application and service infrastructure in a central view, PagerDuty Services Group empowers IT Ops and DevOps teams to resolve incidents as quickly as possible.”

    PagerDuty Services Group is now available to all PagerDuty customers and to learn more, read our blog post.

    About PagerDuty

    PagerDuty is an agile incident management solution that integrates with IT Ops and DevOps monitoring stacks to improve operational reliability and agility. From enriching and aggregating events to correlating them into actionable alerts, PagerDuty streamlines the incident management lifecycle by reducing noise and resolution times. With hundreds of native integrations with operations tools, automated scheduling, advance reporting and guaranteed reliability, PagerDuty is trusted by over 7,000 organizations globally to increase business and employee efficiency. The company is headquartered in San Francisco and backed by leading venture capital firms Andreessen Horowitz, Bessemer Venture Partners, Baseline Ventures, and Ignition Partners. Try PagerDuty for free at www.pagerduty.com.

  • PagerDuty Selects Threat Stack to Enable High-Velocity Security and Compliance in the Cloud

    Threat Stack’s Continuous Security Monitoring Helps Assure Security and Scalability

    BOSTON– March 16, 2016 – Threat Stack, a leader in cloud security management and compliance, today announced that PagerDuty, the pioneer in agile incident management, selected Threat Stack Cloud Security PlatformTM to protect and secure its rapidly growing infrastructure within its Amazon Web Services (AWS) environment. PagerDuty’s decision to adopt Threat Stack was driven by its need for continuous security monitoring and complete visibility into its users, data, and expanding AWS and Microsoft Azure infrastructure.

     

    “As a rapidly growing organization committed to delivering reliability and uptime, it’s critical that we leverage the scalability and flexibility of the cloud, while keeping our resources safe and secure,” said Arup Chakrabarti, Director of Infrastructure Engineering at PagerDuty. “Not only is Threat Stack easy to deploy, it’s also easy to integrate with both new and existing tools. We now have complete visibility into our processes and activities for continuous monitoring and security management across our environment.”

     

    The PagerDuty platform allows IT Ops and DevOps teams to manage and respond to incidents quickly and efficiently, reducing the chance of outages or setbacks of any kind. Threat Stack’s continuous, real-time security monitoring helps the PagerDuty team to quickly identify top-level threats, streamline processes within the operation’s workflow and increase security across the organization.

     

    “We are longtime fans of PagerDuty, and are excited to partner with them to deliver granular, integrated visibility into user activity, processes, files and network connections,” said Venkat Pothamsetty, Vice President, Products and Customer Advocacy at Threat Stack. “We are proud to deepen this relationship by helping PagerDuty scale their cloud-based business while keeping them secure and compliant.”

     

    PagerDuty is using Threat Stack Cloud Security Platform’s full suite of features and special integrations across its AWS environment to provide host-level visibility and other key capabilities including:

    • Autoscaling: Threat Stack’s cloud-native architecture automatically scales up or down with the capacity of PagerDuty’s AWS and Microsoft Azure infrastructure to ensure continuous visibility across its dynamic environment.

     

    • Insider Threat Identification: Continuous monitoring for unauthorized actions, such as escalation of user privileges, unauthorized installs, new users added/deleted, suspicious commands, anomalous behavior, changes to security groups, user session information and process stops.

     

    • External Attacks Early Warning: Automatic detection of abnormal login attempts/failures, wide open security groups, launches of new processes or kernel modules, user session information, process stops, external connections for command and control.

     

    • Data Security Verification: Automated access, editing or copying of customer or personal data intellectual property, as well as internal configuration, passwords, certificates and keys.

     

    • Compliance Reporting: Real-time visibility, detailed audit trails and built-in compliance reporting provide the historical record needed to meet compliance regulations and ensure data and infrastructure are protected.

     

    To learn more about how PagerDuty uses Threat Stack, please visit the Threat Stack blog.

    About Threat Stack

    Threat Stack enables growth-driven companies to scale with confidence by identifying and verifying insider threats, external attacks and data loss in real-time. The only fully integrated, cloud-native continuous monitoring solution that gives customers instant visibility and automatically responds to changes in their environment, Threat Stack provides the coverage needed to run secure and compliant, in all environments, without sacrificing speed and efficiency. For more information, or to start a free trial, visit threatstack.com.

    About PagerDuty

    PagerDuty is an agile incident management solution that integrates with IT Ops and DevOps monitoring stacks to improve operational reliability and agility. From enriching and aggregating events to correlating them into actionable alerts, PagerDuty streamlines the incident management lifecycle by reducing noise and resolution times. With hundreds of native integrations with operations tools, automated scheduling, advance reporting and guaranteed availability, PagerDuty is trusted by over 7,000 organizations globally to increase business and employee efficiency. The company is headquartered in San Francisco and backed by leading venture capital firms Andreessen Horowitz, Bessemer Venture Partners, Baseline Ventures, Ignition Partners and others. To try PagerDuty for free, or to get more information, visit www.pagerduty.com

    Media Contact

    Ashley Crutchfield

    fama PR for Threat Stack

    (617) 986-5025

    threatstack@famapr.com

  • PagerDuty acquires Event Enrichment HQ to speed incident resolution   

    PagerDuty acquires Event Enrichment HQ to speed incident resolution   

    Together PagerDuty and Event Enrichment HQ Integrate Event and Incident Management

    I had a chance to speak yesterday with CEO Alex Solomon, PagerDuty’s Product VP Jonathon Wilkinson and Ophir Ronen, Event Enrichment HQ Co-Founder and now Product Director at PagerDuty.  They all believe that this is a significant move by PagerDuty to help overcome the desensitization that many IT teams suffer as a result of so many alerts about incidents and events.  Trying to turn down the signal to noise ratio and allow IT team members to concentrate on what is most important is a big part of the Event Enrichment platform.

    The company already has begun integration into the PagerDuty and the whole Event Enrichment team is staying on with PagerDuty.

    Below is the press release:

    San Francisco – November 3, 2015 — PagerDuty, the leader in operations performance management, today announced that it has acquired Event Enrichment HQ and their Event Enrichment Platform (EEP). IT operations teams are facing a flood of monitoring data and notifications as the number and sophistication of monitoring tools increases. Integrating the Event Enrichment Platform with PagerDuty reduces noise, surfaces actionable alerts, and delivers context-rich remediation information so businesses resolve critical incidents faster.

    “Event Enrichment HQ realized early that the proliferation of tools within IT operations generates a lot of additional noise and wastes critical time,” said Alex Solomon, PagerDuty CEO and Co-Founder. “The company’s work directly aligns with PagerDuty’s continued commitment to helping businesses deliver uptime and great customer experiences.”

    The integrated solution helps businesses decrease resolution times by reducing noise so incident responders can spend less time on manually triaging events and by providing remediation information within the notification itself as the first source of truth. PagerDuty’s core functionality has addressed the challenge of assembling the right incident responders on the issue, but now teams have even better focus with intelligent control of monitoring system events.

    “Having felt the pain ourselves for many years, we set out to provide the IT operations community with the tools to suppress noise and enrich events” said Ophir Ronen, Event Enrichment Co-Founder and PagerDuty Product Director. “Joining PagerDuty allows us to continue that vision and extend into helping customers streamline the entire incident lifecycle to decrease downtime.”

    PagerDuty now offers a private beta of the Event Enrichment functionality to select customers. Additional details are available here.

    ABOUT PAGERDUTY

    PagerDuty’s operations performance platform reliably increases the uptime of businesses’ applications, services and infrastructure. By connecting people, systems and data into a single view, PagerDuty helps all teams — from DevOps to Enterprise IT — deliver powerful visibility and improved incident resolution times. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Adobe, Airbnb, Etsy, GitHub, Intuit, National Instruments, New Relic, Panasonic and Rackspace. The company is headquartered in San Francisco and backed by leading venture capital firms Andreessen Horowitz, Bessemer Venture Partners, Baseline Ventures, Ignition Partners and others. To try PagerDuty for free, or to get more information, visit www.pagerduty.com.

    ABOUT EVENT ENRICHMENT HQ

    Event Enrichment HQ’s Event Enrichment Platform (EEP) allows IT professionals to significantly reduce their downtime by eliminating noise and enriching actionable events with remediation information. IT teams using EEP have the ability to route, manage, and enrich their alerts without interacting directly with critical production monitoring systems or code. Privately held, Event Enrichment HQ is made up of IT operations veterans who are focused on providing solutions for reducing downtime. The company is headquartered in Seattle, Washington. For more information, or to try EEP for free, visit www.eventenrichment.com.

  • #HugOps in Practice: Empathy Skills for DevOps

    #HugOps in Practice: Empathy Skills for DevOps

    Get it together! Why is your service always down?!”

    We think we’re doing the whole DevOps thing right — new hires can deploy on day one, Travis CI is humming along, and we own the code we ship. But then something breaks, something doesn’t go according to plan, tempers flare up, and all that warm, fuzzy collaboration seems to evaporate. What’s going on? What happened to #HugOps?

    A large part of software engineering doesn’t involve code at all— it’s talking and collaborating with our teammates. Soft skills are harder to measure and quantify than performance or reliability, but they’re just as important— and while we’ll readily spend hundreds to thousands of dollars on books, courses, and training to improve our software skills, companies rarely invest in creating the culture of empathy and compassion that was behind DevOps in the first place.

    Empathy works great…until it doesn’t. It’s easy to be compassionate and collaborative when stuff is going great and everyone’s having an easy time, but it’s stressful, high-stakes situations that show you whether you’ve actually built a culture of empathy. Empathy skills for DevOps cannot be stressed enough.

    Let’s look at some specific strategies and tactics teams can use to improve organizational empathy.

    Learn to Communicate

    Conflict mediators, couples therapists, hostage negotiators, policemen, and others in high-conflict situations often use specialized communications frameworks to help de-escalate situations and get parties in conflict talking to each other in a productive way.

    I’m a fan of a communications framework called Nonviolent Communication, popular in education (I used to be a schoolteacher), parenting, mediation, as well as peace programs in war zones. Another popular guideline is Crucial Conversations, a more business-focused structure for handling conversations where emotions are charged and stakes are high. There are plenty of books, trainings, and other resources about empathetic communication out there, but they all have some common themes:

    • When dealing with a tough situation, separate — explicitly— what happened from how you feel about it. So for the example above, you might say something like “When your service went down for the third time in the past two weeks, I got worried, because our service depends on yours, and we’re trying to hit four nines of uptime this quarter.” Often just the simple step of decomposing reactions from what’s being reacted to defuses the majority of the speaker’s anger— this is used in blameless postmortems to move from “blame and shame” to an honest look at the facts of the situation.
    • When somebody shares a feeling, especially if emotions are high, take time to make sure they feel heard and validated…before you reply. That doesn’t mean you agree with them, just that you ensure that they know that you listened and are trying to understand. In our conflict above, this might be something like “It sounds like you’re really frustrated that our service went down, because it prevented your team from hitting your uptime goal.” Most of the time, the other person will instantly soften, relieved to feel heard. Sometimes you’ll get a “Well, duh!”, but that’s far preferable to the other person leaving the conversation feeling unheard and unacknowledged.
    • If you have a request of the other person, make it positive, concrete, and explicit. And “Could you not?” doesn’t count. Without a concrete request, it’s hard for the other party to know how to meet your needs or improve the situation, and hard to tell whether that effort has been made. For our situation above, this could be something like “Can you please do a postmortem on the outage, and let me know by the end of next week what you’re going to do to prevent a future outage and anything you need from me to help?”

    Again, there are all sorts of frameworks available, and like many tooling choices, it matters less the specific one you use than getting the organization on board with improving everybody’s communication and conflict resolution.

    Align on Common Values

    Cross-departmental collaboration gets much easier if you’re both trying to do the same thing. That doesn’t mean your day-to-day work has to be the same, but the more of your company that is driving toward the same goals, the easier it is to turn conflict into simple questions about sequencing or prioritization, rather than fundamental directional issues.

    LinkedIn, Google, Twitter, and other big tech companies favor the “Objective + Key Result” (OKR) system for this purpose, where a big, ambitious goal and set of results at the company level trickles down into corresponding goals and results for the department and the individual.

    We use OKRs here at PagerDuty, but there are plenty of other goal-setting systems that work. The important thing is agreeing on the meta-important stuff in plain language, so that when someone asks you to change your priorities or randomize what you’re working on, you can have a conversation about which of the options best supports the common goal, rather than the much hairier discussion about what the end-goal even is.

    Mind Your Medium

    Particularly with distributed teams, text-based communication can make tone hard to get across. A request said with a warm smile sounds quite different from one in flat text over email (or to everybody in the company HipChat room).

    Where possible, I prefer video chat to text chat— in fact, some of our teams use perpetual Hangouts so that they can all see each others’ pretty faces, and feel like they’re in the same room together. Where that’s not feasible, I like to over communicate: if the tone of a sentence might be ambiguous, I explicitly state what I mean and what I don’t mean. And while it’s a little weird to me that this has become business normal, emoticons also go a long way in communicating feels over the wire.

    Build Relationships, Inclusively

    Ultimately, all the communications frameworks and shared goals in the world won’t take you to #hugopsland if you don’t have a human relationship with the person you’re trying to work with. An HBR study found that while people generally claim to prefer competence over likability, in practice they behave oppositely — likability is much more important for collaboration than competence.

    And while a few people might be naturally gifted with irresistible charisma, for most likability isn’t a static, unchanging thing: it’s a quality built on familiarity and connection. If two people are going to be collaborating on something, get them some face time — not just screen face time— with each other. We have an engineering office in Toronto, and try to make sure new employees quickly get a chance to spend time with the team in the other office as well as their own. At DevOpsDays Minneapolis this year, Katherine Daniels shared the concept of “bootcamping,” where each new Etsy hire spends a period of time embedded in a different team, learning about how they work, what their goals and challenges are, and making meaningful contributions to their codebase.

    Plan off-sites, social events, time to chill out and get away from the laptops. But keep in mind that not everybody on your team necessarily drinks alcohol, and some of them might (gasp!) have families. That doesn’t mean that you can’t ever plan a team gathering at a bar or do nighttime activities, just try to be aware of the different constraints, preferences, and physical abilities your team members may have. Providing hangout spaces at work is a simple but often overlooked way to help build relationships while respecting those who may have out-of-work obligations or just really long commutes. And while a shared lunchroom can be a good way to cross-pollinate, it’s also sometimes nice to get out of the building— PagerDuty pairs all new employees up with a “buddy,” and sponsors lunch out in the neighborhood to help them get better acquainted.

    Being inclusive of different learning and personality types is important as well. Even if you don’t buy into specific personality tests such as Meyers-Briggs, introverts and extroverts handle relationships and communications differently, and it’s important not to assume that everybody handles information or makes decisions the same way you do. When doing a sprint retrospective, consider starting with a silent brainstorm, to make sure that people’s willingness to talk in a group doesn’t determine their ability to have their opinions heard.

    Finally, and this shouldn’t have to be said, but be respectful of differences in religion, diet, sexual orientation, gender identity, family structure, and anything else that’s not directly connected with their performance as your team member. When somebody expresses their preferred pronouns, a dietary preference, or a constraint on their time, recognize that can be a pretty vulnerable thing, especially if they feel like they’re being asked to justify it.

    Measure and Improve

    While empathy might be tougher to measure than uptime, error rates, or other performance metrics, there are still steps you can take to track how you’re doing.

    Our scrum teams each do “health checks” using Google Forms at the end of each sprint, where we smiley-face-rate our feelings about various factors including teamwork and support. Consistent sad-faces tell us there’s something systemic we need to work on. Managers can also specifically ask in 1:1s about how communication is going within and outside the team, to identify patterns.

    One additional measure (and benefit) of strong empathy is the amount of diversity you see in your organization. If you can’t connect and communicate with people from different backgrounds, you’re unlikely to be able to recruit or retain diverse talent. On the other hand, when you’re able to see through the beer test, or the hijab, and make potential team members feel included and valued, your pool of awesome candidates gets much bigger.

    One final reading recommendation here: Stalling for time: My life as a FBI hostage negotiator. Not only is Noesner’s story a thrilling read, but it takes many of the communication principles above and shows how a master of nonviolent communication permanently changed the FBI’s attitude about the place of empathy in rescuing hostages and defusing violent, high-stakes situations.

  • Four Steps to Free Your Organization From Scheduled Maintenance

    Four Steps to Free Your Organization From Scheduled Maintenance

    Whether you’re an operations engineer or a customer, scheduled maintenance is often viewed as a given when delivering software as a service. While far better than surprise outages, planned downtime is still tough for everybody involved– customers will always have off-hours access needs, and the low-traffic times you want to perform maintenance are often the same times your operations engineers and admins would like to be sleeping. On top of this, scheduled maintenance implies your system is less reliable than you think, because you’re afraid to change it during the workday.

    How do we make scheduled maintenance better for everyone involved?

    Planned maintenance is a non-starter at PagerDuty, since our customers don’t know when their monitoring tools will fire alerts. We’ve had to focus hard and innovate to deliver continuous uptime for our service, and Doug Barth, my colleague and a member of our operations team, recently did a talk at DevOps Days Chicago on the strategies we used to ditch scheduled maintenance altogether, and replace it with iterative maintenance that doesn’t compromise your entire system.

    1. Deploy in stages

    Your deployments need to be rock-solid. They should be scripted, fast and rolled back quickly, as well as tested periodically to ensure rollbacks don’t lag.

    They also need to be forward and backward compatible by one version. It’s not an option to stop the presses when you push out a new version. Red-blue-green deployments are crucial here, as they ensure only a third of your infrastructure undergoes changes at any given time.

    Lastly, stateless apps must be the norm. You should be able to reboot an app without any effect on the customer (like forced logouts or lost shopping carts).

    2. Avoid knife-edge changes

    Use canary deploys judiciously to test rollouts, judge their integrity and compare results. These test deployments affect only a small segment of your system, so bad code or an unexpected error doesn’t spell disaster for your entire service.

    During his talk, Doug suggested a few practical ways to accomplish this:

    • Gate features so you can put out code dark and slowly apply new features to a subset of customers.
    • Make sure you’re testing the changes on a representative sampling of your customer base: different account sizes, usage patterns, and configurations.
    • Find ways to slowly bleed traffic over from one system to another, to reduce risk from misconfiguration or cold infrastructure.
    • Run critical path code on the side. Execute it and log errors, but don’t depend on it right away.

    3. Make retries your new best friend

    Your system should be loaded with retries. Build them into all service layer hops, and use exponential backoffs to avoid overwhelming the downstream system. The requests between service layers must be idempotent, so that you can reissue requests to new servers without double-applying changes.

    Use queues where you don’t care about the response to decouple the client from the server. If you’re stuck with a request/response flow, use a circuit breaker approach, where your client library delivers back minimal results if a service is down—reducing front-end latency and errors.

    4. Don’t put all of your eggs in one basket

    Distribute your data to many servers, so that no one server is so important you can’t safely work on it.

    At PagerDuty, we use multi-master MySQL clusters, which help with operations and vertical scaling. We also use decentralized, linearly scalable databases like Cassandra for our alerting pipeline: since transactions aren’t gated on the behavior of a single node, we can do operational work during the day.

    Put together, these four strategies set an organization up to ditch annoying weekly downtime and help admins and operational engineers sleep more, worry less and maintain better—all ahead of schedule. Not only is this great for your team’s quality of life, but it also helps you deliver a great customer experience– no matter where your users are or what time it is.

  • eHarmony finds its perfect match with PagerDuty and DevOps

    eHarmony finds its perfect match with PagerDuty and DevOps

    Did you know that 4% of all marriages in the US are the result of people “finding their perfect match” on the eHarmony service? Keeping the systems that are making all of those matches up and doing it 24/7/365 is no easy task. Especially at peak load times like Valentine’s Day for instance. The pioneer in making online relationships relies on many DevOps tools including PagerDuty to minimize downtime and keeps its relationship science systems humming along.

    1672a00I had a chance to speak with Hav Mustamandy, Director of System Operations at eHarmony. Since Hav joined the team there about 7 months ago he has been busy modernizing the eHarmony network using a lot of tools that you would expect. They are a big Chef shop for example. PagerDuty is another tool they lean on a lot. More than tools though Hav has been instilling a sense of team between his Dev, Ops and QA teams.  In a short period of time, Hav has made tremendous progress in making the eHarmony IT business more responsive, reliable and faster.

    Frankly, why shouldn’t eHarmony act more like a Unicorn anyway? After all they are a web based business like many of the companies that we think of as being so successful. But before Hav came it was not necessarily so.

    Upon arriving he had to evaluate everything and everyone and start doing things differently. One of those things was the way they use PagerDuty. More than just an update to the old pager, Hav realized that PagerDuty could help him automate their incident resolution processes. Additionally using PagerDuty’s advanced analytics reporting, Hav was able to pinpoint problem areas in both systems and people. With the problem systems he was able to upgrade or change them. With people he realized where re-training was required.

    As a result Hav has seen some amazing results. The uptime of eHarmony’s systems even during peak load times has improved significantly.  Technical issues are identified and dealt with before they rise to critical levels.

    By leveraging APIs that PagerDuty makes available Hav has integrated several of his other systems into PagerDuty so he can gain insight into their performance as well. To him this is the future. Event management with PagerDuty at the core is what he is aiming for.

    For me PagerDuty and ChatOps applications biggest advantage is the ability to move inter-team communications into hyper-speed. If you are relying on email for communication it is probably not fast enough to keep up with your business.

    We have been using Slack here at devopsy.kinsta.cloud for ChatOps and I can tell you that our communication now moves so much quicker than it did when we communicated strictly through email. However, the discussion I have had with my team is if we are just chatting, what is the advantage of Slack to just google messenger or something like that?

    I think the answer lies in integration. Using APIs we should see more than just chatting going on with ChatOps. Systems will communicate with us in real time as well. This is so much more than chatting on a messenger client.

    The name of the game whether it is the dating game, the online media game or whatever game you play is doing it faster, better and with less downtime. For eHarmony it looks like they have “found their perfect match”.

  • Inject failure to make your systems more reliable

    Inject failure to make your systems more reliable

    Injecting failure into your infrastructure to test your services resilience has been gaining popularity. Most people think of Netflix and their use of “Chaos Monkey”, in fact, they have an entire “Simian Army” of tests and drills they run to make their systems more reliable. At PagerDuty, we couldn’t just copy what has been published about their practices. Instead we needed an approach that would work for us. We have a relatively small infrastructure compared to Netflix and wanted to develop a practice that enables a team of our size. In response to this need, we created Failure Friday.

    The idea behind it is simple: We block off an hour of time and deliberately shut down certain pieces of our infrastructure to see how our service holds up. For example, our first attack will test process failure. We’ll typically run something like “sudo service cassandra stop” command, keeping the service down for about 5 minutes. In this time, our service should be resilient enough to continue processing traffic. If not, immediate action will be required. The main goal of injecting failure into our systems, of course, is to establish what will happen in the event of a node, datacenter or network outage. We can then use this knowledge to make our infrastructure more fault-tolerant.

    Benefits of Breaking Your Systems… On Purpose

    On the surface, injecting failure into your systems seems counter intuitive. Why break your systems, especially on a Friday just before everyone is about to go enjoy their weekend? Since beginning our weekly failure testing in June 2013, our results have uncovered a number of benefits, which we believe give reason for all teams to develop failure testing scenarios of their own.

    • Uncover implementation issues that may reduce resiliency. We have the opportunity to fix this issue before they cause an outage resulting in an alert outside of business hours.
    • Proactively discover deficiencies in our infrastructure. These deficiencies may eventually become the root cause of an outage, allowing us to fix them before they break in the wild.
    • Builds a strong team culture. By coming together as a team once a week, we share knowledge. Our Ops team learns how the development teams debug production issues, while Devs gain a better understanding of how their software is deployed.
    • Helps us onboard new employees. Failure Friday has allows us to walk through actual failure and alerting scenarios with new hires so they can shadow and learn to handle outages at 11 AM instead of 3:00 AM.
    • Adjust alerting and on-call schedules. Team members participating in failure testing should receive all the alerts that they would during an unexpected outage. This is a great opportunity to make adjustments to limit noise and also ensure you are receiving all of your critical alerts. When we get an alert, but no action was required we know we need to make adjustments to your monitoring thresholds. Or if we don’t receive an alert, but we need to take action, we need to adjust our thresholds so we don’t miss that or a similar alert in the next time.
    • Reminds us that failures are inevitable. Failure is not a just a random freak occurrence, it can and will happen. All code that engineering teams write is now tested against how it will survive during a Failure Friday scenario.

    Develop Your Own Failure Friday

    When you consider putting together your own version of Failure Friday you must remember that communication between teams is key. Failure Friday will not likely be successful if performed by individual teams.

    Step 1: Bring your teams together.

    Get key stakeholders for each of your teams together to discuss each other’s testing needs and decide what to test. It will not be possible to test every possible scenario, so putting together a list of priorities that is meaningful for everyone will help you accomplish quite a bit in a short session.

    Step 2: Find a time that works best for your team.

    At PagerDuty, we choose Friday for a one-hour session because there tends to be fewer deploys on Friday. This way we aren’t creating any roadblocks for our business with our testing practices. Since it’s a recurring event everyone in the company can expect Failure Friday to occur and be prepared. Though Friday can seem like a risky time, having run failure testing you may catch potential threats, leading to a quieter weekend.

    Step 3: Set Up a War Room

    Having a common place where all your teams can sit together will allow your teams to seamless work together. They will also benefit by observing and learning from members of other teams in how they resolve incidents. For PagerDuty, we set up camp in the corner of our dining area. You don’t need to do anything fancy.

    Step 4: Prepare for your first attack

    Before injecting failure, we’ll disable any non-critical cron jobs that are scheduled to run during the hour. The team whose service we’re going to attack will be ready with their monitoring dashboards to track changes and be take action, if necessary. (note: cron jobs are disabled because we don’t want them to interfere with the attack. For instance, we wouldn’t want a Cassandra repair run in the middle of taking a service down.)

    Step 5: Set up your communication Channels

    Because communication is imperative during Failure Friday you will want to use Failure Friday as a way to also test your communication channels. Using a chat room is an essential useful way to exchange information non-verbally. Chat rooms also provide timestamps so you are able to log and review actions that are taken in order to correlate any actions to metrics you may capture during your testing scenario.

    It’s All About Reliability

    Failure Friday doesn’t end when we resume normal operations. From our learning we will assign action items to team members in our ticketing system (just as we’d do if a piece of our system went down for real), allowing us to take precautions in order to prevent these scenarios from occurring in the wild. By putting people in one room and throwing problems at them, we make both our team and our product stronger.