devopsy.kinsta.cloud

Tag: incident response

  • Leading Effective Incident Response Without Interminable Bridge Calls

    Leading Effective Incident Response Without Interminable Bridge Calls

    There are easier ways to manage incident response without creating war rooms and packing IT staff onto bridge calls

    Your phone vibrates at 11 p.m., and you know that can only mean another major incident with one of the business’ critical systems. You get geared up for the war room, dial into the bridge call and start reviewing the major incident report. You do this instinctively because it’s the third major incident in as many weeks, and you wonder if there’s a faster, easier, less stressful way to keep critical systems running, especially during periods of operational uncertainties.

    You’re not the only one recognizing that having too many major incidents is a significant issue. In a recent survey on the future of monitoring and AIOps, 94% of respondents stated that issue resolution is critical to their business, but only 28% are satisfied with their handling of major incidents. Too many organizations feel that their only path toward improving their incident handling capabilities involves redesigning their critical business applications, facing down their technical debt or building out an SRE practice.

    There are easier ways to manage major incidents without creating war rooms and packing IT staff onto bridge calls. When used well, AIOps techniques can address the pain points in resolving major incidents that drove business leaders and CIOs to establish war rooms and bridge calls in the first place. To understand what AIOps capabilities are needed, it helps to review the history of war rooms and bridge calls and why they are inefficient in solving today’s operational challenges.

    Furthermore, IT organizations can take advantage of AIOps platforms much faster than re-architecting applications, addressing technical debt,or hiring more site reliability engineers.

    How Bridge Calls Became the Status Quo in Incident Response

    No one wants unreliable or poor performing business systems. It’s why IT teams have become extremely proficient at recovering from incidents, especially the ones that are easy to diagnose and easy to resolve. Problems such as web servers going down, databases running out of storage or services stuck in deadlocks are relatively easy to diagnose with today’s monitoring tools. In fact, over the last few years, many IT Ops groups have used tools to automate the recovery from these common issues.

    But more complex issues are harder to solve. Issues such as:

    • Problems causing a cascading failure of dependent systems that are all sending frequent alerts.
    • Issues right after major application deployments or infrastructure changes.
    • Bottlenecks in customer-facing applications that are experiencing unusually heavy loads.

    IT’s history of solving these complex incidents isn’t great. The incident management team responds and calls in help from Tier One support. With so many things going wrong, operational teams have no choice but to call in higher levels of support, including developers. By the time someone communicates to the business on the incident’s status, leaders are irate over the lack of communication, and the time it is taking to recover from the issue.

    CIOs and IT leaders dislike being yelled at and seeing long outages. Their easiest management response is to get all the experts in the room, often called the war room, in the hope that having more people involved is better. War rooms often include bridge calls to allow remote people to attend, and bridge calls are standard practice for major incidents occurring off-hours.

    Bridge Calls Don’t Solve Incident Management Problems Better

    It surely isn’t better for the operational engineers to be responding to off-hour incidents regularly. It’s not better if a seasoned major incident manager is needed to oversee the bridge call and ensure differing opinions don’t escalate into arguments or finger-pointing. It isn’t better if the recovery times miss business objectives and if root causes are never identified. It isn’t better if the resolution times to recurring problems don’t improve.

    It’s also really bad if the number of complex issues is increasing. This is likely to be the case as application architectures based on hybrid cloud or containers and microservices add complexity while frequent releases from DevOps teams add to risk. Also, global events such as COVID-19 create usage uncertainties and network bottlenecks during periods of increased business importance.

    Solving for the Root Cause of Inefficiencies

    Bridge calls and war rooms are inefficient practices to solve complex problems. They require too many people, take too long and require too many more follow-ups to identify the root cause.

    IT has built up an arsenal of tools to better manage myriad operational domains. We’re using one set of monitoring tools for the data center and a separate one for the public cloud. Every database, API, microservice, application component and type of IoT device has different tools to monitor performance. In fact, according to the future of monitoring and AIOps survey, about 20% of respondents reported having 25 or more monitoring tools. In addition, there’s also all the system, network and application log files that often have the most critical information when troubleshooting a complex issue.

    That’s many tools, considerable data and countless alerts to sift through under the pressure of a major incident. These tools often require different skills and subject matter expert participation, which is another reason why bridge calls and war rooms are so crowded.

    Now, AIOps can mean many different things to different people. But if it’s going to improve major incident recoveries without requiring bridge calls or war rooms, then it must:

    • Aggregate all the data from these monitoring tools.
    • Aggregate data from all of your change and topology tools, so you can find out which changes led to an incident or what’s affected by the incident.
    • Sequence the data into a time series showing what issues came first.
    • Correlate the hundreds or even thousands of alerts into discrete events that operators should review.
    • Present the information in an easy console for operators to assess during major incidents.
    • Enable engineers to automate steps in the recovery.
    • Integrate with different workflow, collaboration and communication tools to automate incident response steps.
    • Share the event sequence with engineering and development teams.

    During incidents, the differentiator is that machine learning models have gotten a head start in processing all the data. Instead of dozens of tools requiring lots of people, the start of the incident review can begin by examining information in a single platform. Instead of IT Ops being inundated with too much noisy data and alerts, machine learning has already processed and correlated events into a more easily decipherable storyline. Instead of manual recovery steps, automated steps can be centralized and orchestrated from a single console. Lastly, since the organization’s workflow, collaboration and communication tools are integrated, no one should feel out of the loop on issue status and postmortem steps to make improvements.

    IT teams often celebrate when a legacy system is shut down because operations have fully transitioned to a modernized platform. Isn’t it time to do the same with war rooms and bridge calls?

  • Runtime Security and Incident Response with Loris Degioanni

    Runtime Security and Incident Response with Loris Degioanni

    By adopting DevOps, Kubernetes and containers you can accelerate innovation and deliver a better experience for customers and employees. While Kubernetes adoption simplifies application deployment, security needs to adapt and be cloud-first, open and frictionless as a new set of complexities may surface.

    Check out this interview with Sysdig Founder and CTO Loris Degioanni to find out how you can manage the health of your applications and learn more about runtime security and incident response.

    Loris is a security expert and an open source veteran, and he’s also one of the keynote speakers presenting at this year’s DevOps Experience Virtual Summit. During his presentation, Loris will share his perspective on where security is going and practical advice for managing security risk in a Kubernetes and DevOps world.

    For more insights and tips on Kubernetes security, attend Loris’s keynote session at DevOps Experience 2020 on Oct. 22. To check out the full agenda or to register for free, visit the DevOps Experience website.

  • PagerDuty Applies AIOps to IT Incident Management

    PagerDuty Applies AIOps to IT Incident Management

    PagerDuty, at its online Summit20 conference today, unveiled a bevy of updates that apply machine learning algorithms across a wide range of workflows across the PagerDuty incident response platform.

    In addition, PagerDuty today announced it will acquire Rundeck, a provider of an IT automation platform that auto-remediates incidents by enabling playbooks to automatically execute whenever there is an issue, for $100 million.

    Jonathan Rende, senior vice president of products for PagerDuty, said the overall goal is to make IT teams more efficient by applying AIOps at a time when organizations are trying to reduce to the total cost of managing increasingly complex IT environments.

    At the same time, Rende said PagerDuty is also moving to reduce the fatigue that inevitably ensues when trying to determine the relationship between the thousands of alerts that all the elements of an IT environment can now generate. The goal is to provide IT teams with more context and actionable intelligence, said Rende.

    As part of that effort, PagerDuty is now integrating change events that occur in code repositories via new integrations with GitHub, Puppet and Evolven. The bulk of IT incidents stem from changes made to the IT environment so via these integrations it will become easier to identify the most likely root cause of an issue, said Rende.

    PagerDuty has also added new capabilities to its Service Directory that employs machine learning algorithms to dynamically surface dependencies between people, changes, incidents and services in real-time. As IT environments become more complex thanks to the rise of microservices, he said it’s now more critical than ever to track relationships across workflows spanning multiple IT teams. The challenge is that existing manual processes can keep pace with the rate of change now occurring in IT environments, Rende added.

    PagerDuty is also moving to reduce the number of alerts that are generated by enabling the PagerDuty Event Intelligence module to surface recommendations generated by machine learning algorithms that suggest ways to group alerts to better streamline incident responses. In addition, IT teams can now pause alerts if they decide to allow systems to auto-remediate before they decide to intervene.

    The company is also extending the reach of an application programming interface (API) for analytics it created to enable IT teams to launch queries against the data collected by PagerDuty. In addition, PagerDuty is making available an Analytics Maturity Model that provides access to benchmark data from more than 13,300 customers to both compare their efforts with other IT teams and discover additional best practices.

    PagerDuty today also announced one-click integration with the Zoom video platform to enhance collaboration, as well as extending its existing integration with Microsoft Teams to include a similar one-click capability.

    Finally, PagerDuty also launched an instance of its platform for customer service teams that also provides enhanced integrations with Zendesk and Salesforce Service Cloud. With more organizations dependent on digital business processes to survive, Rende said the need to automate customer service processes has never been more pressing.

    At the same time, IT teams are struggling to keep pace with the rate of tickets being generated by employees who are now working from home. Put it all together and it becomes abundantly clear that continuing to rely on legacy approaches to IT incident management will simply not suffice.

  • When IT Disaster Strikes, Part 3: Conducting a Blameless Post-Mortem

    When IT Disaster Strikes, Part 3: Conducting a Blameless Post-Mortem

    In the first and second parts of this three-part series, we looked at how organizations can effectively resolve incidents and the role of each person involved in the resolution process. In this final piece, we’ll look at what organizations must do post-incident to better understand what went wrong and learn from their response.

    At the time of occurrence, an outage is every IT practitioner’s worst nightmare. However, after the dust settles, each incident presents an opportunity to learn as a team, make an impact on your processes and improve your product. This means that even after you’ve found a resolution, your work isn’t quite done.

    Enter the Incident Post-Mortem

    Once you’ve successfully fixed a major issue and the chaos of the firefight is winding down, it’s time to conduct an incident post-mortem with your team to investigate the problem and figure out what went wrong. Without a post-mortem, you fail to recognize what you’re doing right, where you could improve and, most importantly, how to avoid making the same mistakes in the future.

    Here are my three tried and true steps for a successful post-mortem process:

    Designate a Post-Mortem Owner

    To kick off the post-mortem, the incident commander should assign an owner at the very end of an incident. This owner is responsible for scheduling the post-mortem meeting, which should take place within five business days of an incident. Owners are also responsible for populating the post-mortem document, investigating the incident, pulling in other people to assist and facilitating the post-mortem meeting. In cases where a public blog post is required, the owner will also be responsible for creating the content and putting it through necessary internal review cycles.

    Populate the Post-Mortem Document and Conduct Analysis

    Once an owner has been designated, the team should get to work updating the post-mortem document with available information, including the timeline of status changes and key actions taken by incident responders. Every participant in the incident response should be included in this page, so be sure to go back through all communication and add relevant parties. After you’ve updated the page with the full timeline of actions, you will need to build out details for each event. For each item in the timeline, identify a metric or a third-party page where the data came from. This could be anything from a link to a Datadog graph or Splunk search to a relevant Tweet, as long as it shows the data point you’re trying to illustrate.

    With the timeline in place, you can begin analyzing the incident. Capture all available data about what happened, what caused the issue, how many customers were affected, what impact they experienced, etc., to determine the underlying cause of the incident. This information should be detailed, as it will dictate follow-up action tickets your team will use to prevent recurrence of the incident and improve the incident response process. It also will be examined during the post-mortem meeting, so the more context you can include in your analysis, the better.

    In the case of severe incidents where external communication is required, this is also the time when the post-mortem owner and team should develop customer communications. Pro tip: Avoid using the word “outage” if possible in this communication. Many people see the word outage and assume that your entire system was down, when in reality that was likely not the case. Be precise here, as it is important for both internal learning and sharing within the larger IT community.

    Here is a general template for what to include on the post-mortem page:

    • Overview
    • What Happened
    • Root Cause (or Contributing Factors)
    • Resolution
    • Impact
    • Responders Involved
    • Timeline
    • How’d We Do?
    • Action Items
    • Communication (Internal and External)
    • Hold a Post-Mortem Debrief Meeting

    The final step in the process, the incident post-mortem meeting, should include the incident commander, service owners, key engineers/responders and the customer liaison (for severe incidents). Use your post-mortem document as an agenda to help things run smoothly. The discussion should center around what happened, what the team could have done better and any followup actions that need to be taken. Any disagreements about the facts, analysis or recommended actions should be cleared up during this time, and everyone should walk away from the meeting with a better sense of problems impacting your service’s reliability and having learned more about the service involved.

    The No. 1 rule of running an incident post-mortem is to keep it blameless. The goal of the debriefing process is not to point fingers, but to learn what happened and how you can improve as a team.

    Don’t make the mistake of neglecting the post-mortem process after a major incident. A well-designed, blameless post-mortem empowers your team to continuously learn and offers a way to iteratively improve your infrastructure and incident response process. Not only does this help your team, but it also helps create amazing software and services.

    — Eric Sigler

  • 3 Best Practices for Better Incident Response

    3 Best Practices for Better Incident Response

    It only takes a headline-grabbing mass internet outage or two to drive home the financial and brand incentives of keeping an application up at all times. That’s why in today’s DevOps organizations, incident response is a common topic of conversation.

    There’s a lot of talk within the DevOps community about organizational processes and designs for effective incident response such as the Incident Command model, as discussed by our friends from Fastly in this talk. We also frequently discuss technologies and tools for incident response, spanning monitoring, alerting, logging, mitigation and many other focus areas. At NS1, we make use of the Incident Command model and have invested in the usual array of technologies for incident response, along with developing many of our own.

    Even after investing significant time considering how to respond to an incident and the budget they allocate for related technologies, it’s still possible for organizations to fail to respond effectively when a real incident arises. As a provider of mission-critical DNS and traffic management services for many of the largest internet properties, we know there’s little room for failure. We’ve learned that augmenting our processes and technologies with three simple practices makes a huge difference in real-world incident response situations.

    The Value of a Good Checklist

    A good checklist is a favorite tool of mine when it comes to effective incident response. A checklist is a simple, concise, easy-to-understand tool to help an operator recall key steps to take in a particular situation. In an intense incident response scenario, it can be easy for an operator to skip or forget steps in their response. Rather than being overly prescriptive, a checklist’s job is meant to help an experienced professional who is familiar with the systems they’re operating make the best use of their knowledge in a high-pressure situation.

    A checklist that I have really come to appreciate doesn’t have anything to do with DevOps or networked systems operations. It is the Cessna 172S Emergency Procedure for engine failure during flight:

    In his book, “The Checklist Manifesto,” Atul Gawande discusses this checklist in depth. This example demonstrates some of the key principles behind a good checklist: it’s simple and short, not overly specific or prescriptive; it’s limited to only the most important actions the operator needs to recall during an incident mitigation; and, because of its simplicity and clarity, it helps the operator calmly execute those key actions.

    This Cessna checklist offers another great lesson for operators of today’s IT infrastructure. As Gawande writes, “Because pilots sometimes become so desperate trying to restart their engine, so crushed by the cognitive overload of thinking through what could have gone wrong, they forget this most basic task: FLY THE AIRPLANE.” In IT operations, it’s no different—we can sometimes become so distracted by root cause analysis and the desire to know why an incident has occurred that we forget to focus on mitigation first—analysis can come later, once the incident is resolved.

    Running Drills

    It’s possible to have well-vetted checklists, great tools and technologies and the best organizational models, but they all can be forgotten in an instant of panic if people aren’t used to using them. So, how do we get great at using them all in real-world situations? Through practice.

    Any incident response strategy needs to incorporate fire drills because they ensure we regularly exercise our processes, tools and checklists in simulated scenarios that approximate real-world incident response situations. There are a number of ways to create fire drills with respect to networked systems. One tool we often discuss in the technical operations community is one built by Netflix called Chaos Monkey, a software embodiment of what has become an emerging discipline: chaos engineering. It’s a methodology for improving the resilience of a distributed system by creating failure situations—for example, by randomly “breaking” virtual servers in a distributed application to force failure handling systems to engage. This is a kind of “automated” fire drill that drives resilient architecture.

    A “war game” that simulates DDoS attack situations is another drill we have benefited from. We build tools similar to those used by sophisticated attackers and regularly exercise our DDoS mitigation response by literally attacking production systems. In this type of fire drill, we split our team into attackers and defenders (similar to the red/blue team approaches used by information security organizations for penetration testing). We’ve learned that running good DDoS war games requires us to attack real systems with real attack techniques, which helps us understand the dynamics of our distributed systems and builds muscle memory in our team around key steps needed to respond to real-world incidents. We’ve also found that fire drills such as these are great tools for team-building; they’re almost like running football or soccer scrimmages at the end of a long practice session, helping everyone on the team understand how they can best contribute and what “position” they should play in a real-world event.

    Analyzing After the Fact

    Postmortem or retroactive analysis is a final practice for driving effective incident response.  After any incident, real or simulated, a thorough review of the incident and the response is critical. Asking and carefully answering some simple questions about the response to the incident is the key: What happened and why? What worked well in our response? How could we have responded more effectively?

    The reason to do this type of analysis is to apply learnings from the incident and create a feedback loop, so that in the future, your team’s response to incidents in general—and certainly to similar incidents—is more effective. Obviously, it’s critical to fix any issues that surface during an incident response in your systems, incident response processes and mitigation tools. You should also review relevant checklists after an incident: what was missed, or what was confusing? And finally, feed the outputs of real or simulated incidents into your fire drills so you can practice similar scenarios in the future.

    Be Prepared

    Like death and taxes, incidents are going to occur. Companies that deliver content or applications in networked systems need to be prepared for them. Best practices such as creating a simple checklist and conducting drills helps a team’s incident responses improve over time, helping to reduce stress during an actual incident and increasing the likelihood of a swift resolution.

    About the Author / Kris Beevers

    Kris Beevers is founder and CEO of NS1. He is a recognized authority on DNS and global application delivery, and often speaks and writes about building and deploying high performance, at scale, globally distributed internet infrastructure. He holds a PhD in Computer Science from RPI. Prior to founding and leading NS1, he built CDN, cloud, bare metal and other infrastructure products at Voxel, which was acquired by Internap in 2011.

  • 6 Must-Haves When Using Chat for Incident Management

    6 Must-Haves When Using Chat for Incident Management

    Over the last few years, teams have realized the benefits of sharing and distributing knowledge in chat applications such as Slack. Today, teams are extending the use of these applications beyond collaboration by embracing ChatOps. ChatOps empowers teams by bringing complex day-to-day operational work into shared chat channels. If done correctly, it drastically reduces context switching and increases the speed at which teams can tackle tough challenges.

    Companies are realizing incredible benefits by incorporating ChatOps into their incident management workflows. Chat channels provide hyper-effective means to alert teams of issues, take immediate action and collaborate with clarity and urgency. It is for this reason that companies including OpsGenie, PagerDuty and VictorOps have invested in integrating their incident management applications into chat applications such as Slack, HipChat and Microsoft Teams.

    ChatOps to the Rescue

    Ideally, users of a modern incident management solution need to have access to a multitude of features so that they can address specific challenges directly within the chat tool. During an incident, switching between applications to access and collaborate on the actionable data is a waste of time. Keeping the conversation and action in one place, where team members already are, is the key to a fast and successful incident resolution process. That is why ChatOps has emerged as an alternative to the war room concept.

    To get the most out of the technology and resolve problems more quickly and easily, here are six tips to consider when using ChatOps for incident management:

    1. Post Messages with Clarity

    The basic requirement for integrating any application within chat tools is to simply have messages posted within chat channels. However, there are three additional needs to consider when integrating an incident management tool into chat when increasing efficiency is the goal.

    • The integration must be able to post chat messages regardless where the incident originates and additional information added (monitoring, ticketing, or collaboration tools).
    • Messages need to be customized based on the specifics or severity of the incident. Often the basic message needs to be appended with tags or additional information fields.
    • If possible, include a button in the message that allow responders to take instant, prescriptive actions.

    2.  Control How Information is Shared

    Posting every alert and comment into a shared channel can quickly become overwhelming and is considered bad practice. A strong integration enables the creation of separate channels for specific responders and have direct control over what types of alert and follow-up messages are shown.

    A seamless integration also provides the flexibility to receive only the incident actions that are decidedly relevant. Low-priority issues that don’t require immediate action can be filtered and not sent to the chat stream, ensuring the team stays focused on what really matters.

    3. Accelerate Actions

    ChatOps transforms chat into a vehicle through which users take actions and deploy solutions. This virtually eliminates the need for operations teams to switch between applications when responding to an incident.

    Most integrations enable users to execute incident actions using text commands. This method is valid and often preferred. However, deeper integrations simplify tasks by providing interactive buttons that enable users to begin working on incidents with just a click.

    The value of responding to an alert by clicking a button extends beyond simple alert acknowledgement. Enabling all common responses (i.e. assigning an alert, taking ownership, adding notes, muting incidents, etc.) to be executed in this manner accelerates resolution and unlocks the real power of ChatOps.

    4. Instantly Retrieve Supporting Information

    A challenge of a long chat thread is that critical information can get lost within the communications or additional information may be needed. A user might have to leave the channel to look up important details and related information, such as looking at other open incidents, for example.

    The integration of an incident management system into the chat environment needs to allow users to access supporting information at any time. The ability to query not only additional details about a specific incident but also general information regarding on-call schedules and alert policies is a requirement. By empowering team members to access this data directly within the chat interface, they stay focused and are able to respond in the most efficient manner possible.

    5. Create New Alerts

    A common task when managing an incident is to create manual alerts. Again, requiring people to switch applications to complete this task is a distraction and slows the progress.

    Creating new alerts for a team or service right within the chat channel is a necessity. The incident management system then notifies the person on-call automatically. Having the ability to create an alert directly within chat empowers individuals that may not have direct access to the incident management system to report problems efficiently.

    6. Control Access

    When an incident occurs, it is important that corrective action is taken by only the people who are authorized to respond. Mapping (or linking) the chat users to the users within the incident management system enables this level of control.

    Linking the user profiles between the chat and incident management has additional benefits. As alerts are acknowledged and responded to within the chat environment, the events are recorded within the incident management solution. A clear history describing who took each action is recorded for future reporting and analysis.

    While there are many things to consider when integrating incident management into a chat application, the benefits are clear. Response time, ability to collaborate, and team access to incident details are crucial factors for incident resolution.

    About the Author / Serhat Can

    Serhat Can is a Technical Evangelist for OpsGenie. He has contributed to different parts of OpsGenie as an engineer and now tries to spread the word by coding, writing and talking about DevOps. He is still a proud member of the on-call schedules. Follow him on Twitter and connect with him on LinkedIn.
  • VMware Aims to Advance Application Security

    VMware Aims to Advance Application Security

    Most of the tasks associated with maintaining IT security assume that IT security professionals should discover and neutralize all potential threats to an application. At the VMworld 2017 conference this week, VMware is seeking to turn that notion of IT security on its head by focusing the efforts of IT security professionals on the known good.

    Tom Corn, senior vice president for security products at ware, says instead of chasing all the potential bad things that could adversely impact an application, VMware App Defense employs machine-learning algorithms and data VMware collects from its management console and the host on which the applications are running to determine what elements make up an application workload. Those elements are compared to the manifest that was created when the workload was first deployed and, if they have been altered, VMware App Defense will invoke a preloaded library of incident response routines to enforce policy using VMware network virtualization software. Data collected from the host by VMware App Defense is done by inserting code into the kernel rather than deploying agent software.

    Corn says VMware App Defense uniquely provides DevSecOps teams with access to reliable telemetry into the guest operating system environments running on top of VMware. IBM will feed the telemetry data gathered by VMware App Defense into its portfolio of security analytics software, and eventually that data will be fed into the IBM Watson cognitive computing platform to generate application security recommendations. Other vendors pledging support for VMware App Defense include VMware sister companies RSA and SecureWorks, as well as Carbon Black and Puppet.

    Longer term, Corn says VMware expects to also be able to apply this capability to any production environment attached to a VMware NSX virtual network overlay.

    As potentially compelling as VMware App Defense might be, Corn notes that IT organizations should not attempt to boil the IT security ocean. Rather, they should concentrate on new applications currently under development. Otherwise, managing security policies across hundreds of applications made up on thousands of microservices and communicating over microsegmented network overlays might be too complex to manage.

    VMware has been making a case for some time now to employ NSX to take advantage of microsegmentation to limit the flow of east-west traffic in a data center. The idea is that if one node gets compromised by malware, it can only share that malware with nodes participating on the same microsegment of virtual network. VMware is now expanding on that idea in a way that also provides more visibility all the way up the application stack.

    Priced at $500 per CPU per year, VMware App Defense will induce more IT security professionals to recommend deploying its stack over any other available option, VMware hopes. The degree to which IT operations teams and cybersecurity professionals can influence developers in that direction remains to be seen. But VMware App Defense does fulfill its promise, it’s more than likely developers will appreciate the not having to spend as much time defending existing applications instead of writing new ones.

    — Mike Vizard

  • The Shifting Data Protection Paradigm: Proactive vs. Reactive

    The Shifting Data Protection Paradigm: Proactive vs. Reactive

    The ever-evolving threat landscape over the last few years should have prompted organizations of all sizes to revise their data protection methods. The volume of incoming data is now staggering, and will only continue to increase—and much of that data is sensitive or of great value to cybercriminals. Data may be structured or unstructured, distributed across numerous locations and be encrypted or unencrypted. Thus, many organizations are attempting to protect data without knowing what they have, whether it is valuable or even where the data resides.

    As the era of big data has exploded, so, too, have the number and size of cyberattacks. Hackers have grown increasingly adept, often refining their skills beyond those of many cybersecurity experts. Today’s hacker may be an idealistic “lone wolf,” but he or she is more likely to be an employee of a foreign government or associated with a gang of highly organized criminals. This means that attacks can be carefully planned and executed over many months, probing a target’s vulnerabilities repeatedly until an intrusion is eventually successful.

    Cybersecurity teams often are at a disadvantage when it comes to protecting data. It has become increasingly difficult to find and retain qualified, experienced cybersecurity experts, and this talent gap won’t be narrowing significantly anytime soon. Economic issues have forced many organizations to trim their budgets, which can result in reduced funding for the security operations center (SOC) or reduced staffing levels. However, even major corporations with large cybersecurity budgets and massive teams are finding it difficult to prevent breaches and keep their data secure. The breaches suffered by Home Depot, Sony and other high-profile corporations—as well as breaches at the U.S. Office of Personnel Management and the Pentagon—demonstrate that cybercriminals are winning more than their fair share of the battles.

    It is not that organizations are unaware of the need to protect their data. Virtually all companies have some type of cybersecurity program in place. However, most of them choose to take either a reactive approach to threats or a proactive security stance. But what they truly need is an approach that integrates both methods.

    Reactive vs. Proactive: The Differences

    With a reactive approach, action does not begin until an incident occurs. This does not necessarily mean that a breach has occurred; it simply means that some unusual or threatening activity has triggered an alert. Unfortunately, an organization can receive several thousand alerts in a single day. Many will be “false positives” and others may have been blocked by defenses, but some may represent a successful penetration of the system. Once alerted, the company’s incident response plan is initiated; in the beginning, the goals are to contain the threat and restore service. Forensics, analysis and efforts to prevent a recurrence follow quickly.

    With a proactive approach, organizations try to detect potential threats before an incident occurs. Files from unknown or suspicious sources can be quarantined automatically for investigation. If found to be malicious, the source, signature or other identifying characteristic can be fed automatically back into the system to block any additional attempts at the gate.

    A proactive approach also includes regular, planned hunting exercises to detect threats that may be lurking in the system but not yet detected or perhaps not yet detonated. Although ad hoc hunting can be accomplished to some degree if performed by experienced analysts, the best method is to automate certain steps so the efforts are sustainable and repeatable.

    Why Organizations Need Both Approaches

    It is always better to prevent a breach than recover from one. However, should—or when—an intrusion occurs, every organization needs a mature, workable, detailed plan to react to the incident. All employees need the training and experience to know what they need to do when the incident response plan is activated.

    Despite the need to have a solid plan to react to incidents, blocking potential threats can be more economical, faster and safer. Therefore, organizations need to be both proactive and reactive to provide the best security for their data. The key is to find the proper balance between the two approaches. Finding the right balance requires evaluating the budget, the skills of staff members and other resources that might be available.

    For both proactive and reactive methods, however, automation is essential. Manual processes simply cannot compete with automation when it comes to speed, cost, accuracy and risk mitigation. With the number of attacks growing by the day and hackers becoming increasingly more adept and persistent, automation provides the best way for organizations to stay one step ahead of the attackers.

    About the Author / Rishi Bhargava

    Rishi Bhargava is co-founder and VP of Marketing for Demisto, a cyber security startup with the mission to make security operations “faster, leaner and smarter.” Prior to founding Demisto, he was vice president and general manager of the Software Defined Datacenter Group at Intel Security, and before Intel, he was vice president of product management for Datacenter and Server security products at McAfee, now part of Intel Security. He has more than a dozen patents in the area of computer security. He holds a BS in Computer Science from Indian Institute of Technology, New Delhi, and a Masters in Computer Science from University of Southern California, Los Angeles.

  • When IT Disaster Strikes, Part 1: Resolving Incidents

    When IT Disaster Strikes, Part 1: Resolving Incidents

    As a developer or operations team member, there is nothing quite like the dread you feel when you hear the familiar ringtone of your on-call page at 3 a.m. Being on call means that you may be contacted at any time to investigate and fix issues that arise for the system, but that doesn’t mean you can’t get ready beforehand. There are steps your team can take to prepare for incidents and streamline the process of resolving them, leading to fewer 3 a.m. wake-up calls and better-running software and services.

    Below is an overview of a proven approach your organization can take before, during and after an IT incident to reduce headaches and increase hours of sleep.

    Before: Define the Who, What, When of Incidents

    The first step in any incident prevention and orchestration process is determining how your organization defines an incident. Generally, this is done by defining the varying degrees of severity, with lower-numbered severities more urgent. Operational issues can be classified at one of these severity levels. As a general rule of thumb, if you are unsure which level an incident is, treat it as more severe to ensure it is dealt with timely.

    In addition to aligning your team on defining an incident to communicate the sense of urgency involved, there are several main roles that should be designated ahead of potential IT issues. Certain roles have only one person per incident, while other roles can have multiple people assigned to it. It’s all about coming together as a team, working the problem and getting a solution quickly. Generally speaking, developer, IT Ops and DevOps teams must designate the following roles:

    • A point person or “Incident Commander”—someone who will drive the process forward, but not be involved in the actual remediation;
    • Someone to document a timeline of an incident as it progresses for future analysis and learning and to act as a backup for the point person;
    • One or more subject matter experts who are deeply familiar with the specified component or service and who will take the remediation steps; and
    • One person to manage customer support and communication during and after an incident.

    During: Keep Calm and Communicate

    If you are alerted of a major incident, don’t panic. The first step is to join a previously agreed-upon method of communication to be used during incidents to ensure that communication can run smoothly throughout the resolution process. The incident commander (IC) and the IC deputy should announce the issue in appropriate communication channels and lead the process, and it is best to defer to the subject matter experts assigned to the incident to ensure non-essential communication is kept to a minimum.

    The steps you take to prepare for an incident have a significant impact on how quickly you are able to move when disaster strikes. If incident prep has been done correctly, each member of the incident response team will have a very specific role and set of responsibilities carved out, ranging from someone to provide regular updates in the chat client to someone to modify your company’s status page to keep customers informed. By having these roles defined beforehand, you don’t have to spend valuable time during an incident figuring them out instead of fixing the problem. In a future article, I will break down each member of the incident response team’s specific role and the steps they should follow during an incident.

    After: Conduct a Post-Mortem Without Finger-Pointing

    For every major incident, you should follow up with a post-mortem—a blame-free, detailed description of exactly what went wrong to cause the incident, along with a list of steps to take to prevent a similar incident from occurring again in the future. The incident response process itself should also be included as part of the review.

    As the IT incidents we deal with daily become increasingly tied to larger organizational success and business objectives, streamlining the resolution process is a must. According to a report from IDC, the average hourly cost of an infrastructure failure is $100,000 per hour, and the average cost of a critical application failure per hour is $500,000 to $1 million. In future articles, I will break down the varying roles on an incident response team, specific steps to follow during an incident, a tried-and-true template for conducting a successful post-mortem and more, to ensure effective incident prevention and resolution.

    About the Author / Eric Sigler

    Eric Sigler is the Head of DevOps at PagerDuty, helping protect its customers from the pains of downtime. Before his current role, Eric led infrastructure teams at Minted, Expensify, and the Missouri University of Science and Technology. Connect with him on Twitter.

  • Avoid the DevOps Temptation to Build It Yourself

    Avoid the DevOps Temptation to Build It Yourself

    As tempting as building your own DevOps incident communication tool sounds, you should think twice.

    My colleagues and I recently were on the phone with a company seeking an IT incident communications and collaboration solution (a.k.a. IT alerting) to eliminate the time wasted with inefficient communications between teams when dealing with major IT issues.  We were talking to Jeff, the director of IT operations, and Alec, the incident manager, about how our customers benefit from using IT service alerting solutions to automate the communication to their Dev and Ops teams, the stakeholders and the impacted customers every time a major IT incident is identified.

    We insisted on the fact that communication automation not only would save them a significant amount of time but also provide a consistent, predictable and repeatable process with timely and relevant communications to the different groups of people. We then moved to the ROI discussion; very easily we were able to quantify the hard dollar gains they would realize. In fact, the company estimated that an unplanned downtime of its most mission-critical business application costs the organization an average of $1,700 per minute during business hours, and half of that when issues occur at night. Depending on the nature of the incident, they need to get anywhere between five and 14 people from several teams on a restoration call or war room, and it takes them at least 20 minutes to get the right people.

    Right there, by automating the communication and the escalation, they could reduce that time down to five minutes and save the business at least $25,500 per incident.

    We were 20 minutes into the call when the conversation took a very interesting turn. Based on the cost savings we had calculated, Jeff and Alec start thinking  they may just be able to build their own communication automation solution in-house. “I’m pretty sure we can find a few developers and a DBA for a few weeks and get this done,” Jeff said.

    An IT communication automation solution provides the ability to reach out to the right on-call people via different communication channels (voice, text, SMS, mobile app) until they reply. If they don’t, automated escalation kicks in and the next person is contacted. Once they acknowledge they’ve received the notification, they can jump on the incident restoration call in one click. The solution also enables the incident manager to notify hundreds or thousands of impacted customers within a few minutes so frustrated users don’t start calling the help desk.

    “That’s a possibility, of course,” I said, “but do you really have an idea of what it would take to design the solution, build it, maintain it and support it when it is being used in production?”

    Jeff had raised a very good point, indeed. Why not build your own solution when you have software development resources available internally? The conversation had shifted from talking about our out-of-the-box solution to digging into what it would take for DevOps to build their own.

    What Does it Really Take to Build an IT Communication Automation Solution?

    The problems Jeff and Alec raised are all really big problems, and they are big enough for many IT departments for them to decide to go down the path toward building and deploying their own internal IT alerting solution. This may be the right way to go, but you need to think through the development effort involved and, more importantly, support the maintenance of this new DevOps tool.

    For some small shop, it may seem like a good idea, but building a reliable and economically viable IT alerting solution in-house requires more than just funding a development team for a few years and negotiating a good plan with telecom providers for SMS, voice and conference calls in the United States and in the countries where your IT people might be.

    You will need to consider the cost to develop the solution, host it, support and maintain it.

    To develop the solution, you’ll need to assemble a team comprised of a software architect, a few developers, a mobile app developer, a dba and a UX designer. Also, make sure they stick around for a few years so they can support and maintain their custom code and bring new features together as the tool becomes utilized across the enterprise. You surely want to avoid turnover here. The great benefit of such a custom-built solution is that it should address your specific needs, whereas a solution from a vendor may only accommodate 90 percent of those.

    These seven developers will cost you about $1 million U.S., and this is not including resources for QA and UAT, the infrastructure guys and a technical writer. That’s another few hundred thousand dollars per year, at least, until the tool is ready for prime time!

    In addition, you will need to reserve some operational budget to cover the cost associated with hosting, operating, monitoring, supporting and maintaining the solution:

    • Hosting the tool. Account for the hardware, the databases and the software you’ll need to build, test and run the solution in the production environment. If you want the solution to be highly available, you will also consider active-active clusters and a disaster recovery site. That’s another few thousand dollars U.S. every year.
    • Monitoring the tool. This new IT alerting tool most likely will be flagged as a critical application in your IT ops toolset and will require monitoring by the NOCs along with a runbook and an escalation procedure in case of a problem. Isn’t this a catch-22? When something wrong happens with your brand new IT alerting tool, you won’t be able to use it to reach out to the on-call people. Make sure you have a special emergency runbook for when this occurs. If you include technical maintenance, bug fixing, QA, that may add up to a half full-time employee, or roughly $75,000 per year.
    • Integrating the tool. It is most likely that you will want to integrate your IT alerting tool with other IT infrastructure and IT service management (ITSM) tools you are currently using so that communications can be triggered automatically as soon as a critical incident is detected. You may need to find a few API developers to make this happen. Remember, those integration connectors all must be maintained and tested with every new release of these IT operations management and ITSM tools.
    • Communicating with the tool. Besides voice, text and SMS, if mobile app push notification is a requirement then you need to build a mobile app so people can update their schedule and availability from their smartphone. This is a great feature that also will need to be developed, hosted, supported and maintained.
    • Running the tool. You will need to provision for all your communication costs should they be voice, internal voice, SMS, international SMS and collaboration tools such as conference calls. This can become significantly expensive, especially if you have distributed teams across continents. Jeff said the company had on average two critical incidents per week, with an average of 10 people on each call who would spend an average of three hours per conference call. That’s almost 200,000 minutes per year for conference calls only, not including SMS and voice. Prices range from a few pennies per minutes in the United States to a few dimes in other countries.

    What’s The Alternative to Building it?

    There are several IT alerting solutions available out there. Some are:

    • Designed for small local teams, others for broader and more distributed IT organizations.
    • Built for IT operations teams, others are built for the Service Desk.
    • On premise, others are cloud-based.
    • Only cover the USA, others offer worldwide coverage.
    • Are more expensive than others.
    • Subscription based, others offer perpetual licenses.
    • Mono product, others offer a platform you can leverage for other use cases.

    There’s a large variety of solutions you could choose from which will address most of your current needs. As a result, you may be tempted by a subscription based plan so you are not stuck with perpetual software licenses and lengthy onsite installation and configuration, support and maintenance. In additional these vendors need to meet their SLAs which will give you the assurance of a constant and good quality of service. Remember that they are also incentivized to deliver best in class customer service because subscription models imply they need to keep you as a happy customer year over year.

    How Should I Decide Which is Best for My Organization?

    It should be pretty straightforward. If you are passionate about IT alerting then go for it and build or hire a team, provide them with a roadmap and cross your fingers that everything will go smoothly and on time.

    At the opposite, if IT Alerting is outside your core business, if you don’t have bandwidth to develop this new IT Ops tool or if you need it sooner than later, then I think you are better focused on other IT projects which will have a direct positive impact on the core business of your enterprise. Save the time and budget to better align with the business and bring the new features that the business demands faster to market. If this is the case then you may consider speaking with an IT Alerting vendor. Gartner can provide insight on these vendors and how they integrate with the appropriate IT tools.

    There are so many dependencies that it will always be less expensive to buy the service from a vendor whose focus is on IT Alerting than it would be for you to build one from scratch. Today most companies are undergoing some kind of digital transformation which means more and more of the services they deliver will rely on IT. The odds of something going wrong with IT and impacting the company’s business operations have never been so high and they are only growing exponentially. Add to this the increased mobility of employees and distributed nature of teams due to acquisition, or cost optimization, and you now realize how challenging the Build-it-Yourself ­­option can quickly become.

    At the end of the day, you and your teams have a much greater chance to be rewarded and recognized by your upper management for helping the company being more efficient and gain market share over your competition than for developing a new IT Ops tool which you could have bought from a provider!

    Prior to this call, I doubt Jeff had thought through all the details of building the solution in-house, all the associated costs he would have to engage for building, supporting and maintaining his new IT Ops tool.

    We’ll see what comes out and what Jeff, his team and the CIO finally decide to do, but here’s what he said when we wrapped up: “Good thing we talked today, and I am sure we’ll be speaking again very soon”. We’ll be talking again to Jeff and his team and find out what way they want to go with their IT Alerting solution: Build it or Buy it?  This is the question.

    Can We Remain Sane Without IT Alerting?

    Without IT service alerting, when a critical IT incident occurs, the service desk or the critical incident manager ends up manually going through the following:

    • Internal procedure documents or runbooks to find out “what team(s)” should be responding to the incident.
    • Spreadsheets to identify “who” within the team should be contacted, then through listings to check “who is actually available.”

    And this becomes tricky when the teams are distributed across buildings, across geographies and countries, as you need to account for time zones, vacations, time-off, sick days and anything of this nature that may prevent someone from responding to notifications.

    Let’s say a dozen people from different teams have been identified to investigate a mission-critical application failure. That means you need to pull someone from the database team, two from the infrastructure team, three developers, two network engineers, the incident manager for this case, someone from UAT, two people from the NOC and someone from support. The developers are in Beijing, the service desk is in Atlanta and the two NOCs are located in the U.K. and the United States.

    Now, they need to be contacted as quickly as possible and the war room needs to be assembled so they can start investigating the issue and come up with a resolution action plan. Plus, in some cases, more than a hundred people would be involved in a single restoration call for a few hours, and sometimes for days. I’m not going to throw any company names here, but you can easily find these disastrous stories online as they very often make mainstream press headlines.

    In the meantime, don’t forget, the clock is ticking and every minute that passes means:

    • Bigger revenue losses for the company, particularly  in the case of an e-commerce website failure or slowdown,
    • Employee productivity drop-off when the ERP system fails,
    • Patient safety degradation if your EMR/EHR experiences a full or even a partial outage, and
    • Big mess ahead when an airline reservation system has a hiccup.

    As a reference point, the website Geekwire estimates that Amazon loses around $120,000 per minute when the site is not performing fast enough.

    In these situations, you need all hands on deck to act quickly. Not only do you need to contact every person individually, but also ensure they respond to the call and commit to working on solving the issue. As you know, emails don’t wake people up, especially on a Sunday at 2 a.m. What if not all the IT experts answer the message within the allocated time frame? You now need to escalate to the next person on the list and try reaching out …

    Also, people are not always reachable via the same channel at all time, in all countries. Some may prefer a phone call during the night and a SMS during weekends. For others, the best way to reach out is via mobile app push notifications. Ask yourself, is English OK for everybody, or do you need to take into account possible translation needs for different languages?

    Even though you are able to get the right contact information for all the right people, calling each and every person and explaining the situation can take a very long of time, and the clock is still ticking. By that time, users of the IT service may have already experienced service degradation and have started calling in to the help desk or entering tickets online. Not only do you have to deal with the outage, but your IT department is now drowning in a ticket storm. The clock is ticking and the phone is now ringing!

    If you are facing an e-commerce website slowdown or outage, you will also need to deal with bad reviews and negative comments being Tweeted and shared about your company over social networks.

    Vincent Geffray

    About the Author/Vincent Geffray

    Vincent Geffray is Senior Director of Product Marketing at Everbridge with focus on IT Service Alerting & Communications Automation and IoT.

    He has more than 14 years of experience in the information technology business, designing, promoting and selling Enterprise IT Operations Management solutions, including Critical Communications, Application Performance Management, IT Process and Workload Automation. He also has international experience as he started his career in Europe. Vincent holds a Master of Science (Mechanical Engineering and Computer Science) and executive certificates from the MIT – Sloan School of Management.

    Vincent’s LinkedIn: www.linkedin.com/in/vgeffray

    Vincent’s Twitter: @vgeffray