devopsy.kinsta.cloud

Tag: incident response

  • Blameless Integrates Incident Management Platform With Opsgenie

    Blameless Integrates Incident Management Platform With Opsgenie

    Blameless this week announced it has integrated its namesake incident management solution with the Opsgenie alerting platform from Atlassian.

    Aaron Lober, director of product marketing for Blameless, said the Blameless platform is now integrated with both Opsgenie and PagerDuty, the two most widely used platforms for managing alerts.

    Blameless provides an incident management platform that normalizes data collected from a wide range of DevOps tools and platforms to determine how best to respond to different incident classes. Armed with those insights, it then becomes possible to quickly assemble the best team to respond to an event, said Lober.

    That capability reduces stress because Blameless also surfaces recommendations for resolving incidents based on the data it has collected, he added.

    The Blameless platform is tailored to organizations that have hired site reliability engineers (SREs) to manage IT operations using DevOps best practices, noted Lober. The overarching goal is to eliminate rote processes that, over time, conspire to burn out incident management teams that often experience high turnover rates, he added.

    Opsgenie has been connected to the Blameless platform via the service catalog and alerting functions. Blameless users responding to or managing an incident can escalate issues to any teams defined within Opsgenie. The service catalog and alerting data generated throughout the life cycle of an incident is automatically captured and stored on the Blameless Incident Timeline. That capability makes it possible to identify appropriate service owners and the escalation protocol that was followed by that team. Blameless can also display the actual user name of each responder that has been notified.

    When accessing Blameless via Slack or Microsoft Teams, Opsgenie can automate both when an incident channel is created and pre-select relevant services that should be made available. Alerts can also be manually triggered using commands from the Blameless chatbot for those who prefer to trigger alerts from within Slack or Teams themselves.

    As IT environments become increasingly more complex, more organizations are making a concerted effort to apply DevOps best  practices to incident management. The goal is to reduce downtime and increase overall resiliency at a time when organizations are more dependent on software than ever. As part of that effort, IT teams should identify the repetitive processes that span multiple incidents and then move to automate as many as possible.

    In the longer term, it’s not clear what impact artificial intelligence (AI) will have on incident management. At some point, though, IT teams should expect their efforts to be augmented by machine learning algorithms capable of identifying issues before they balloon into a major crisis. However, there will always be some human intervention required to manage incidents for the foreseeable future, noted Lober.

    In the meantime, most IT teams should continue to expect the unexpected at a time when IT environments have never been more dynamic and change multiple times a day.

  • Will DevSecOps Replace the Security Operations Center?

    Will DevSecOps Replace the Security Operations Center?

    Traditionally, a security operations center (SOC) is a physical facility where an organization performs information security activities. The SOC team analyzes and monitors the organization’s security systems. A SOC aims to protect businesses from security breaches by identifying, analyzing and responding to cybersecurity threats. 

    The SOC team consists of administrators, security analysts and security engineers. They work in collaboration with internal development and IT operations teams.

    A SOC is a proven method to improve threat detection and incident response, reduce the potential for security breaches and help organizations respond appropriately in the event of an incident. The SOC team isolates unusual activity on servers, databases, networks, endpoints and applications, identifies and investigates security threats and responds to security incidents as they occur.

    SOCs were once considered suitable only for very large organizations. But many small organizations today are building lightweight SOCs, such as hybrid SOCs that rely on a combination of part-time in-house staff and outsourced experts or virtual SOCs that function without physical facilities.

    SOC Challenges

    SOCs play a critical role in protecting an organization’s assets from cyber threats and ensuring the security and integrity of its networks and systems. However, SOCs can face a variety of challenges in their efforts to detect and respond to potential threats, including:

    Alert Overload

    Alert overload is a common challenge faced by SOCs. It occurs when the SOC team is inundated with too many security alerts, making it difficult to prioritize and effectively respond to them. This can be caused by a high volume of alerts, a lack of context or proper prioritization, limited visibility, or limited resources. To address the challenge of alert overload, SOC teams may implement strategies such as tuning security tools, implementing a triage process, utilizing automation, and increasing the size of the team.

    Visibility Gaps

    Visibility gaps refer to areas of an organization’s networks and systems that are not adequately monitored or protected by security tools and processes. These gaps create vulnerabilities that cyber threats can exploit, making it difficult for SOC teams to detect and respond to potential threats. 

    Visibility gaps may be caused by complex environments, limited resources, shadow IT, or misconfigured systems. To address visibility gaps, SOC teams may implement strategies such as implementing a security information and event management (SIEM) system, conducting regular assessments, implementing additional security controls, educating employees and ensuring that systems are properly configured.

    Complex Investigations

    Investigating cybersecurity threats can be a challenging task for SOC teams. It requires them to quickly and accurately identify the root cause of the threat and take appropriate action to contain and mitigate it. This can be difficult due to limited visibility, complex environments, a lack of context or limited resources. 

    To aid in the investigation process, SOC teams may use forensics tools to gather and analyze evidence related to potential security incidents. Common tools include network forensics tools, which allow the team to analyze network traffic and identify the source of a threat and digital forensics tools, which allow the team to examine devices and systems to identify evidence of an incident. 

    Slow Detection and Response Times

    Slow detection and response times are a common challenge faced by security operations centers (SOCs). They can impact the organization’s mean-time-to-detection (MTTD) and mean-time-to-response (MTTR), which measure how long it takes the SOC team to detect and respond to potential security incidents. 

    A high MTTD and MTTR can have serious consequences for an organization, as it may allow a cybersecurity threat to remain active for an extended period of time, potentially causing significant damage.

    What Is DevSecOps?

    DevSecOps stands for development, security and operations. It is a new work process that involves integrating security practices into all DevOps processes. The DevSecOps mindset guides all technical experts equal access to security expertise.

    DevSecOps moves security practices and enforcement early in the development process. This creates a culture where security is everyone’s responsibility, not just that of the security team.

    Building security into every phase of the software delivery life cycle enables continuous integration and high development velocity with fewer security issues and lower compliance costs.

    How DevSecOps Improves Security and Quality

    Organizations using the DevSecOps model understand that security should not be an afterthought and should be a core part of the software development life cycle (SDLC).

    DevSecOps collaboration is more difficult than DevOps because it requires achieving two seemingly contradictory goals simultaneously—expediting the delivery process while spending more time to make sure code is secure and free of bugs.

    To implement DevSecOps without compromising product quality, organizations need to create a culture that deals with security-as-code, encouraging developers to consider the security aspects of their projects and automate security tasks. Organizations need to establish ongoing communication and collaboration between IT engineers, software developers and security teams.

    How DevSecOps Processes Are Transforming the SOC

    Traditionally, SOC processes were completely isolated from the rest of the organization. Developers would build systems, IT Ops run systems, and security protected them. Today, it is understood that merging these three functions into one organization that shares responsibility for security improves security and significantly increases operational efficiency.

    SecOps was a second phase, in which operations and security teams became one organization. Security was shifted left, starting from early stages of IT requirements and system design, not at the end of the process.

    SecOps had a major impact on organizations implementing DevOps. It paved the way for the third phase—broader collaboration between security, operations, and software development teams, known as DevSecOps. This moves security even further to the left, building security into the system from the first iteration of development.

    DevSecOps can operate in parallel to a SOC. Here are some ways SOCs can modernize their processes, even if the SOC itself is a silo:

    • Develop a decentralized SOC with DevOps members: Department members familiar with DevOps can help with incident response by gaining insight into IT systems and understanding vulnerabilities and threats to security personnel.
    • Collaborate with threat hunters and DevOps teams: Instead of isolating threats and reporting them to management, threat hunters can communicate directly with development or operations teams to address key security vulnerabilities.
    • Create a Center of Excellence: SOCs can work with development and operations teams to adopt security best practices and spread these positive results throughout the organization to promote DevSecOps practices.
    • Allow the SOC to provide advice and guidance: Everyone involved in security should have easy access to the SOC and be able to work with the organization’s top security experts.

    Conclusion

    In conclusion, while DevSecOps can significantly impact how SOCs operate, it is unlikely to completely replace the SOC. DevSecOps emphasizes the integration of security practices and tools into the development process from the beginning, which can help to prevent security incidents from occurring in the first place. However, it is still important for organizations to have a dedicated team in place to monitor for and respond to potential security incidents.

    The SOC will continue to play a critical role in protecting an organization’s assets from cybersecurity threats and ensuring the security and integrity of its networks and systems. By adopting DevSecOps practices, the SOC can improve the security and quality of the organization’s software applications, reduce the risk of security incidents and improve the efficiency of its operations. However, the SOC will remain a vital part of the organization’s overall security posture and will continue to evolve and adapt to new challenges as they arise.

  • Embrace Change in Production to Improve MTTR

    Embrace Change in Production to Improve MTTR

    DevOps creates value through constant deployment changes. When done right, more frequent deployments allow teams to continuously improve their products or services while making it easier to manage the effects of the change. One way to do this is with continuous integration and continuous deployment (CI/CD). CI/CD helps to reduce the risks to each build and allows teams to release valuable features quickly and safely. But CI/CD alone is not sufficient for understanding change and service impact in production. The recent DEJ State of IT Performance Report found that change is the largest source of production issues. A full 76% of all performance problems can eventually be traced back to changes in the environment but 67% of organizations don’t have the ability to identify changes in their environment that caused performance issues.

    In a modern IT environment, different teams are constantly integrating new features and new technologies and renewing their stack. With many moving parts, it’s impossible to mimic the production environment when testing new features in pre-production. By the time you’ve implemented and tested new code, your branch will be different from what is in production.

    Here’s the reality we’re working in:

    ● Integrating testing in CI/CD pipelines will not keep bugs out of production. Some organizations don’t do true CI/CD, meaning some processes are not even tested and automated in the pipeline.
    ● User configuration errors are common. Here’s a consolidated list of configuration errors that brought down production.
    ● Change anxiety is real: Ops folks get nervous when hitting the deploy button, even for small changes.

    Changes are inevitable in DevOps, but organizations can protect themselves from a performance incident or an outage by integrating change in production safely with a few simple steps.

    Build Data Connections

    Today, data is kept in silos, which is why it’s time-consuming and challenging to make associations and leverage them to find the root causes of performance issues. DevOps teams need to connect their data in meaningful ways to troubleshoot complex issues in distributed systems. Building the “connections” foundation is critical to understanding the state of a system and how the pieces interconnect and interact.

    How well an application performs in production is related to a combination of code, infrastructure and operations. For instance, when an incident occurs, teams should quickly know if that issue is related to code, infrastructure or operational activity. Understanding these connections and correlating the data cannot be performed manually. We have hit the wall in linking together data silos; we just don’t scale as software does. Teams need to consider a monitoring and analytics approach that connects operational data together from the start and models the causal links between the data for real-time root cause diagnosis.

    Understand Change Impact to Avoid Outages

    How well can your DevOps team identify the key changes made to your production environment? If your answer is “not very well,” then you’re missing critical data that helps to speed up troubleshooting. The lack of visibility into changes and how they impact production makes it impossible for operations staff to identify the cause—the very thing that SREs care most about.

    Changes are made to the code or the cloud infrastructure, either directly or through infrastructure-as-code (IaC) templates, for a variety of reasons. The risk of misconfiguration and performance issues increases when these operational activities are not continuously monitored and correlated across the stack.

    Metrics don’t tell the whole story. Logs don’t pinpoint where to act. Simply collecting these data types leaves many companies struggling with knowing what is important and what’s not when it comes to real-time troubleshooting.

    Having a shared “system of record” for changes for all the different streams of activities increases visibility and control of all changes in real-time. Teams can see pre-incidents and be aware of increased risks when changes are made, especially during high-velocity change periods. By seeing how a code change can impact things down the line, developers can proactively troubleshoot and debug code.

    Implement Real-Time Root Cause Diagnosis

    In the same DEJ report, organizations shared that 66% of MTTR is spent on the ‘diagnose’ stage. This correlates to the 67% of teams that are unable to identify the changes in their environment that caused performance issues.

    With a shared “system of record” for changes, DevOps teams can troubleshoot incidents in real-time, dramatically cutting the time spent on the “diagnose” stage of MTTR. This is especially effective when teams can see all the changes and other events that are happening to the system in a single, searchable/filterable real-time feed; this timeline would provide the potential causes of issues.

    The best tool would allow you to start from an effect and work backward, chaining multiple events overlaid with observability data to determine the causes quickly. Building the causal chain from the end helps teams hone in on the connected change and answer each “why” in a straightforward way. This capability makes it easier and faster to prioritize relevant changes and resolve performance degradations or outages with actionable context.

    Communicate Everything

    During diagnosis and resolution, it’s important to see what’s being done without having to ask on Slack, wait for each other or unknowingly interfere with remediation actions. The best option would be leveraging the same timeline to capture all the context of an incident at the moment it occurs, and at any moment in time during diagnosis and resolution, to avoid reverting each other’s changes.

    While an incident postmortem brings teams together to gather data and discuss the details and learnings from it, this process is often inefficient and time-consuming. Each team member involved in the incident response effort would have to recollect what they observed, what actions they took and what they communicated to other team members. What if a system could capture all the actions and communications during an incident response effort? Wouldn’t it be nice to just summarize your thoughts in one sentence and be done with the reporting?

    When preparing for planned and unplanned changes that can impact the bottom line, DevOps teams need to consider these steps to gain true observability of their operations. Focus on finding the connections in the data to explain cause and effect. Understand change impact —as those changes are being made —to avoid outages. Capture and communicate activities as seamlessly as possible and identify and capitalize on opportunities to automate the workflow.

  • Kintaba Creates the First Conference Dedicated to Incident Response – Techstrong TV

    Kintaba Creates the First Conference Dedicated to Incident Response – Techstrong TV

    John Egan, Co-founder and CEO at Kintaba, talks about IRConf, the first conference fully dedicated to Incident Response. The company behind the conference is Kintaba — a modern incident management platform for the entire organization. The video is below followed by a transcript of the conversation.

    Alan Shimel: Hey, everyone. Welcome to Techstrong TV. We have got another segment for you here. I’m happy to be joined by John Egan. John is the CEO of Kintaba Incident Response. 

    Hey, John. Welcome to Techstrong TV. 

    John Egan: Thanks, Alan. Good to be here. 

    Alan Shimel: Nice to have you. So, John, you know, I might have messed up what your position with Kintaba is. 

    John Egan: That’s right. CEO and Co-Founder. 

    Alan Shimel: Okay. And you know what? I mean I’ve heard of Kintaba, but that doesn’t mean our audience knows Kintaba, so if you wouldn’t mind, give us maybe a quick little background on Kintaba and maybe a little bit of your own background? 

    John Egan: Yeah. So, Kintaba is an incident response platform. We’re designed to let any organization be able to easily implement best practices for responding to critical moments within the company, so this tends to be things like site outages, major customer issues, and other sort of technical and non-technical problems that require kind of a group real-time response, you know, separate from like project management where you might have planned things that are happening. You take, like, the outage that happened, I think yesterday, at GitLab was a good example, right? Or a slack outage like that, that has an incident response team that’s responding to that situation and making sure it gets fixed quickly, and they’re probably using some kind of tool, either internally or off the shelf, and we’re that tool that any organization can implement in minutes instead of bringing in a consultant and spending months on this kind of stuff. 

    Alan Shimel: Absolutely. You know what was interesting, John, is I mean look, I’ve seen, for whatever reason since I’ve started devopsy.kinsta.cloud eight years ago, incident response has always been like this, you know, it’s part of the DevOps story, and we’ve seen of course, you know, several companies have very successful exits and be, you know, very successful. They all kind of follow this pattern where they, you know, were replacing the beeper, then they became incident response. Then it becomes a whole platform, and APIs and integrations, and so forth. What makes Kintaba different? 

    John Egan: So, we’re really purists on the incident response side of things. I think a lot of organizations that come into this space end up either kind of pivoting over towards alerting, all right, which is sort of Pagerduty’s world – 

    Alan Shimel: Right. 

    John Egan: Which is more about automated detection, ping someone, it gets closed out, you know, you might get thousands of these things across the organization, or they go broad and start to become, you know, more of like an SRE platform to try to handle all of these different things that the site reliability engineers are really managing inside of the organization. But we really believe that incident response stands on its own as a practice that cuts across all of the organizations inside of your company. So, something unique about Kintaba is normally when you’re rolling it out within the organization you might start small. You might start with five or six seeds, a couple of people in SRE, but really rapidly we tend to get deployed across the entire organization and what we do really well is we’re a collaborative layer that helps everyone from customer success, to your SREs, to even your PR and marketing guys all be connected during these critical moments and all have one place that they can go to see what’s happening and participate in that response. 

    It’s a bit more of, you know, we call it an evolved sense of incident response, right? Versus kind of the old-fashioned way of looking at it, which might be two or three people in your SRE team are responsible for this. It’s really more about the entire community of your organization that responds to these company critical moments and building software that makes it easy for all of those people to participate. So, we’re a very non-technical feeling piece of software. We’re very human driven, everything from our reporting, to our collaboration experience, to the way you go and write documents, which is a new feature we’ve just released, all of this information is designed to be as easy to use as something like Google, as opposed to being, you know, more like a, you know, almost like a service now, kind of like complex enterprise implementation. 

    Alan Shimel: Got it. Good stuff. And, John, we’re going to talk about this conference you guys are putting on, but before we go, before we dive into that, you know, so we understand what Kintaba is. You know, in your go-to market here what are you guys finding? What are you seeing? You know, what’s kind of the state of the market, if you will? 

    John Egan: So, we’re really learning a lot about this community as it’s just starting to form, right? I think when communities are young and they’re new you actually find the best practices are pretty fluid and I think it’s one of the unique things about this space is Google put out a book back in 2016 where they sort of defined how they handle incident response internally. It was the first time anyone really wrote it down in a way that could be absorbed, but it turns out every organization has a slightly different attitude that they take towards this space and when you’re implementing a product, something like Kintaba, it really turns out that that product has to be pretty flexible on its way in to make sure that it’s easy for people to adopt, and then it also has to bring with it sort of almost like a light hand to say these are the very high-level macro steps you’re going to take to be successful, right? 

    And so, what we’ve learned early on is when Kintaba first launched, you know, over a year ago, we were very rigid about the way that you sort of had to come in and step through, you know, each part of incident response. What we’ve really learned and what the products evolve towards is to have rigidity around the high level steps report openness around visibility to when incidents are happening, like easy to use collaborative spaces, post mortems that editable and changeable, but also, be flexible in terms of who’s getting access to this thing, how are you giving people titles that are non-technical in the, like, customer success roles, you know, and in the marketing and PR roles, and how do you bring all of that together at once. We thought when we came in that, okay, maybe this would be a couple of seat experience for people, you know, you come in and maybe you need two or three seats, greater, four or five, and what we’ve really learned is these are company practices, like I was saying before, and I think that’s been kind of our big learning about the space is as the community grows this is really a cross-functional community inside of organizations rather than just a technical practice inside of DevOps and I think that’s really exciting. 

    Alan Shimel: Yeah. Absolutely. Absolutely. So, John, you guys are coming in behind the first dedicated incident response virtual conference coming up in April. Why don’t you – well, I already said it I guess, but why don’t you take it from there? Tell us more. 

    John Egan: Yeah. So, IR Conf, or Incident Response Conference, was really built out of the fact that we just couldn’t find another one to be part of that was focusing purely on the incident responder. When we talk about communities internally, you know, you think of communities as what makes people put their hands up and say, “I’m part of that,” right? I think DevOps, and SRE, and a lot of these spaces have done a good job, right? We have crowds and people will say, “Okay. I’m part of that.” Incident response has always  been one of those ones where I think people sort of nervously said, “Well, I think I’m part of that.” What IR Conf really does is it focuses not on sort of that statement of I’m in the incident response department, but instead focuses on the idea that I’m a responder, right? I’ve been part of responding to a major incident and when you ask that question it turns out a lot of hands go up and it turns out that there’s a community there that’s really looking for other practitioners to come together and talk about these things I was just talking about, right, best practices, right? What are other companies experiencing? How are we defining this role and the people who are involved? And so, what’s really cool about this conference is we get to focus really directly on that incident response process, you know, not even necessarily as it relates to other parts of DevOps, but just how we deal with this very specific problem that faces all organizations. 

    So, I’m really excited we have one our keynote is Emily Freeman, who’s out of AWS – 

    Alan Shimel: Sure. 

    John Egan: And has done all kinds of great talks about kind of the wider, softer development lifecycle inside of DevOps, and here she can speak much more directly, you know, to this response, incident response space. I think you can see the excitement around it. You know, we have sponsors, basically everyone across the DevOps world, Honeycomb, Commodore, Rookout, BridgeCrew, Gremlin, Teleport. All of these folks, you know, really came running in and said, “Yes. This is great. We need a space to come in and talk to our sub-communities that are already using our products that care about this space,” and so really, we just felt like there was a hole in the market and so this is a free conference. Anyone can join. This is a virtual conference. It’s on April 1st. It’s at IRCONF.IO if you’re out there looking for it. We’re really hoping to just get everyone to come along who’s ever been involved in an incident before and has experienced that fire moment inside of a company of, you know, hopefully not panic, but maybe panic, and wants to come together and talk to other folks, who have had the same experience. 

    Alan Shimel: Sure. I’m thrown back to my cyber – we didn’t call it cyber then – to my infosec days in the war room, you know, when stuff hit the fan. You know, best practices that were to set up a war room on how to respond to a breach, you know, some sort of attack. 

    John Egan: And that’s really where a lot of this grew out of, right? 

    Alan Shimel: Sure. 

    John Egan: We have incident response really lived in the security world – 

    Alan Shimel: Yeah. 

    John Egan: And it’s one of these parts of the evolution of the space, right, that now we’re adopting, you know, that process that really got its feet inside of the security incident response space across the organization to say, “Wow. These guys are doing a great job,” like, “We should be doing similar work across the org.” 

    Alan Shimel: Not all companies had such a great job on their war room, but that’s a whole other story. 

    Hey, John, you mentioned Emily Freeman speaking. Who are some of the other speakers? 

    John Egan: We’ve got two more who are announced right now and it’s Emily Freeman, and it’s Dave Rensin. Dave Rensin was previously at Google and ran a good chunk of the SRE Team there. He’s at a new organization now and he’s spoken at a couple of other conferences in the DevOps space, specifically about the process of incident response. I’m really excited about him being able to come in as well and talk, both, from a large company standpoint, right, a company like Google, where he really pushed a lot of customer metrics, right, for dealing with incidents, not just internal technical metrics, as well as the smaller company, like where he’s moved to now is a startup.

    Those two are kind of our cornerstones. Then we’ve got a series of other speakers, who are announcing over the next couple of weeks – 

    Alan Shimel: Who?

    John Egan: Who are coming in from all over industry, so I’m really excited. I’m can’t announce them here yet. They’re not finalized yet. 

    Alan Shimel: Okay. Good. 

    John Egan: Really exciting. Great organizations. 

    Alan Shimel: Can’t blame me for asking. All right. Hey, John, for people who are interested, that was IRCONF.IO? 

    John Egan: That’s right. IRCONF.IO. You go in there and just click Attend and register. We’re still at early bird, so everything is free right now. 

    Alan Shimel: And it’s April 1st. Do you have times on that? 

    John Egan: I believe we’re planning to start it early, around 8:30 is the beginning of it. 

    Alan Shimel: East Coast time? 

    John Egan: It’s technically a half-day, but it runs until about 2:00, I believe. 

    Alan Shimel: So, that’s East Coast time though? 

    John Egan: East Coast time. Yes. 

    Alan Shimel: Yep. We’ve got a global audience here, so that’s East Coast U.S. time. 

    John, sounds great. You know, if there’s a need in the market, you know, the market has a vacuum, so it’s good to see it getting filled here. I want to wish you the best with it, with the conference and with Kintaba. 

    John Egan: Great. Thanks, Alan. 

    Alan Shimel: It was great having you. John Egan, CEO, Co-Founder, Kintaba, the folks behind IRCONF and you can find out more about that at IRCONF.IO. That’s coming April 1st. Check it out. 

    We’re going to take a break here on Techstrong TV and we’ll be right back. 

  • The Rogers Outage of 2022: Takeaways for SREs

    The Rogers Outage of 2022: Takeaways for SREs

    When, eight years from now, folks are creating lists of the top IT incidents of the 2020s, there’s a good chance that they’ll include the Rogers outage of 2022. The failure, which made internet and cellular network service unavailable for more than 12 million users across Canada, was one of the most significant outages in memory, in terms of both the number of affected accounts and the level of service disruption.

    For site reliability engineers, there’s a lot to learn from this incident, from both a technical and a crisis-management perspective. Here’s a look at our main takeaways.

    The Rogers Outage: What Happened?

    Before diving into lessons for SREs, let’s go over what happened during the Rogers outage and what we know about the cause of the incident.

    Rogers is a major telecommunications provider that serves around 12 million customers in Canada. Early on July 8, 2022, it suffered an incident with its infrastructure that made internet and wireless phone service unavailable for most or all of its customers. The outage was so complete that users couldn’t even make 911 calls or withdraw money from ATMs–which means the failure brought down not just ordinary services, but also what you might consider resources that are essential for normal daily life.

    Connectivity was restored for most users within about 15 hours. However, some customers reported continuing disruptions for several days after the beginning of the outage.

    What Caused the Rogers Failure?

    At first, Rogers did not reveal much technical detail about what triggered the incident. A statement attributed to the company’s CEO the day after the outage said that a “maintenance update in our core network … caused some of our routers to malfunction.” But that didn’t explain specifically which type of maintenance update the company was trying to apply, why the update didn’t go as planned or which types of routers were affected.

    It wasn’t until several weeks later, when Rogers submitted a response to questions from the Canadian Radio-Television and Telecommunications Commission, that more technical detail came to light. Rogers explained to the CRTC that, during an update process, the company ran code that deleted a routing filter. It’s still unclear exactly how the bad code made its way into the update routine.

    Lessons for SREs

    For SREs, the Rogers outage is an example of a worst-case scenario. It was an extremely disruptive incident that took a relatively long time to mitigate, at least when you consider how severe the outage was. That’s why SREs should use Rogers’ experience to assess what SRE teams can do to avoid issues like this–and manage them more effectively when they do happen.

    Test Your Updates

    From a technical perspective, one of the biggest takeaways from the Rogers outage for SREs is that you should test updates thoroughly before applying them. If engineers had tested their update routine more carefully, they would presumably have detected the bad code that deleted the routing filter before the deletion actually took place.

    To be fair, Rogers is hardly the first company to experience a major outage due to a bad update. Atlassian, for instance, had a similar issue earlier this year, when a buggy migration process caused a long-lasting outage.

    If you’re an SRE, you don’t want your company to be the next to make headlines when something that should be a simple change, like an update, triggers a massive outage. Test, test and test again before bringing your changes into production.

    Build in Redundancy

    It also seems reasonable to conclude that Rogers didn’t have redundancy built into its infrastructure in the form of backup routers that could take over when the primary routers crashed due to the bad update.

    This is another reminder to SREs about why it’s so critical to avoid putting all of your eggs in one basket or creating single points of failure. Redundancy may cost more, and it may not totally prevent disruptions. But it can make them less severe and easier to recover from. It’s not hard to imagine the Rogers incident being a lot less severe if even just 20% or 30% of its traffic could have been automatically redirected to backup routers. That might have been enough to keep critical services operational, at least.

    Have an External Crisis Communications Plan

    Rogers has received more than a little criticism for not communicating well with its customers during the outage. The company waited hours before releasing an initial statement about the failure. It was not very responsive to media queries early on, and, as noted above, it took several weeks before Rogers finally released technical details about what went wrong (and even then, the details emerged only because of a government inquiry into the incident).

    The lack of rapid, transparent communication probably wasn’t deliberate. We have to believe that Rogers didn’t want to leave customers in the dark. Instead, the slow communication likely reflects the absence of a pre-planned strategy for external incident communication in a situation where Rogers’ own networks were non-operational. Without a communications plan, Rogers presumably struggled to decide which information to share, or how to share it, in the midst of a major crisis.

    If Rogers had a playbook in place for communicating with the public and media about this type of issue before its network went down, it probably would have been able to share information more effectively–and it may not have faced as much of a backlash from customers and regulators.

    Internal communications, at least, presumably were smoother. The company did manage to get service back up for most customers within about fifteen hours, which probably would not have happened if its engineers lacked a means of communicating with each other as they worked to resolve the incident.

    Conclusion

    For SREs, the Rogers outage of 2022 is a lesson in the importance of testing, designing redundant infrastructure and preparing for transparent communications before crisis strikes. While more careful planning in these respects may not have totally prevented the incident, it would likely have at least reduced its severity and mitigated the backlash that Rogers now faces from angry customers and regulators.

  • Why More Incidents Are Better

    Why More Incidents Are Better

    Ask most SREs how many incidents they’d have to respond to in a perfect world, and their answer would probably be ‘zero.’ After all, making software and infrastructure so reliable that incidents never occur is the dream that SREs are theoretically chasing.

    Reducing the number of actual incidents as much as possible is a noble goal. However, it’s important to recognize that incidents aren’t an SRE’s number-one enemy. What matters more than the number of incidents you experience is how effectively you respond to each one.

    Plus, there’s value in incidents. They are a learning opportunity. If your business never experienced them, it would arguably be facing more risk, not less.

    We know: These ideas may sound a little counterintuitive. You might even accuse us of being “pro-incident”—which we sort of are. Allow us to explain.

    The Silver Lining

    In many respects, incidents are inherently bad. When one occurs, it means something broke. That’s bad. It may also mean that users were disrupted, operations halted or money was lost. Those things are even worse.

    On the other hand, incidents aren’t all bad. They actually benefit SRE teams for several reasons:

    • Learning opportunities: Incidents are opportunities to figure out what went wrong and prevent it from recurring. They can also help teams learn how to react more quickly or efficiently the next time something fails.
    • Get ahead of bigger issues: Sometimes, working through one incident means you can avoid another that’s even worse. Perhaps one server fails, for example, and your response revealed that the failure was due to a larger issue that would have eventually caused a worse outage if left unaddressed. But thanks to the incident, you detected the larger issue before it triggered a more massive failure.
    • Reinforce team culture: Nothing breeds camaraderie or a spirit of collaboration like working alongside other engineers to respond to a crisis in the middle of the night. Although being in this setting may not be anyone’s first choice, it does often have a positive impact on your team’s culture and esprit de corps.
    • Demonstrating value: Assuming you handle them well, incidents are an opportunity for SREs to prove how valuable they are to the organization. If incidents never happened, it’s a safe bet that some bosses would start to wonder why they need SREs in the first place. (It would be a flawed train of thought, of course, because SREs would deserve credit for preventing incidents, but it’s a thought that may float around some C-level brains nonetheless.)

    We could go on, but the point is clear: Although incidents cause problems in some respects, they actually create value in others.

    Focus on Response, Not Avoidance

    This is not to say that you should welcome incidents with open arms. Obviously, any decent SRE should focus first and foremost on being proactive and preventing incidents from happening whenever possible. They should use chaos engineering to identify problems that could be lurking unseen in production environments. They should leverage IaC to minimize risks. And so on.

    That said, what ultimately matters more than incident frequency is the effectiveness of incident response. It’s better to experience ten incidents that you resolve in under an hour each than one incident that takes mission-critical systems offline for a week.

    So, in addition to investing in tools and processes that mitigate the risk of incidents, SRE teams should place equal emphasis on ensuring that they can react quickly and effectively when an incident happens. This means having the ability to share information efficiently, define clear roles, know what to prioritize when working through complex incidents and have clear plans in place that spell out how you’ll handle a problem as soon as you detect it. Without these abilities, you’re at risk of letting incidents that should be small turn into major outages.

    ‘Zero Incidents’ is not Realistic

    It’s important to recognize, too, that while it can be fun to imagine a world where zero incidents occur, the reality is that such a world will never exist. If it could, we wouldn’t see new records set each year for the number of security incidents that businesses collectively suffer, for example.

    Nor would we see headlines about major outages at huge enterprises like Facebook or AWS on a recurring basis. If those companies, which have world-class reliability teams and virtually endless resources at their disposal, can’t reduce incidents to zero, neither can anyone else.

    Conclusion

    The bottom line: There is no such thing as total incident prevention, no matter how hard you try. And even if there were, that wouldn’t actually be a good thing, for the reasons explained above.

    So, by all means, undertake reasonable proactive efforts to prevent as many incidents as you can from happening. But don’t let investment in prevention cause under-investment in response. Being prepared to handle incidents when they happen—which they inevitably will—is what matters most.

  • Improving Observability With ML-Enabled Anomaly Detection

    Improving Observability With ML-Enabled Anomaly Detection

    Nowadays, DevOps and SRE teams have many tools to access and analyze logging data. However, there are two challenges that prevent these teams from resolving issues in a timely manner: 

    • They aren’t equipped with all the data they need
    • Detecting and resolving issues is reactive and manual

    In this article, I’m going to break down why these challenges persist and propose a new approach to observability that will help you overcome them.

    Observability Obstacles

    Challenge: You’re not analyzing all your log data

    DevOps and SRE teams need to understand the behavior of their applications and services to spot anomalies as they occur and keep them up and running. However, the cost of traditional logging platforms prevents these teams from analyzing 100% of their observability data. 

    Instead, they have adopted mechanisms to reduce ingestion volumes. In some cases, this means truncating or filtering out datasets. In others, it means moving data to a less active storage tier/target where analytics are not applied. Furthermore, engineers have to predict upfront which datasets are important and meet the criteria to index. It’s these datasets that teams have the ability to analyze in real-time. The rest are ultimately neglected.

    The net impact is that teams are left with an incomplete picture of their environment, making it difficult to understand how applications and services are behaving. As a result, they cannot pinpoint every issue as it occurs.

    Challenge: Traditional platforms create reactive and manual processes

    When working with traditional observability platforms, teams must constantly define logic to detect issues in their environment. And often, they are only able to detect the issues that they are aware of and have had time to build logic for. To keep alerts firing correctly, you have to stay on top of different data structures, changing data shapes, new schemas and libraries, and fluctuating baselines. 

    However, not every team is equipped to detect issues they haven’t seen before. When teams are faced with unknown behavior, they are left conducting a post-mortem debug which is inherently a reactive process. Furthermore, it is increasingly hard to resolve the issue quickly because logging data is often unstructured and difficult to search. Teams have to manually hunt through loglines to pinpoint the activity and affected systems or components. This approach can add hours or even days to an investigation.

    Improving Incident Response

    The solution: Machine learning-enabled anomaly detection and resolution

    To catch every issue and speed up incident investigations, teams can apply machine learning at the data source to detect potential threats and equip teams with the information they need to keep their applications and services running. This kind of approach allows engineers to analyze 100% of their log data and compile it in a consumable and actionable view so that they can spot changes in behavior and resolve anomalies quicker than the traditional approach.

    As loglines are created, they can be automatically analyzed and converted into insightful metrics that inform application and service behavior. From there, federated machine learning can be used to automatically detect and alert on an issue as it occurs. It does so by baselining key metrics, understanding when they’re outside of typical ranges, and determining the likelihood of anomalous behavior—all through machine learning.

    There is no need to continually build and refine logic, effectively eliminating overly manual engineering practices.

    When an issue occurs, developers and DevOps engineers will receive a detailed report of the specific changes in activity within their systems. In this report, they’ll see the exact window of time of the incident and the affected systems or metadata. This report also includes a pinpointed capture of the relevant raw data that contributed to the anomaly, as well as systems, components, or services that were involved.

    Finally, the relevant full-fidelity logs can be dynamically streamed to a central observability platform (Splunk, Datadog, etc.) before, during and after the anomaly. This gives teams the exact data they need to resolve the issue in a timely manner.

    In all, this kind of edge observability allows system administrators to analyze all of their logs all of the time—not just ones that match the pre-defined query in a centralized setup, saving tons of time when it comes to incident response.

  • 5 Tips If You’re the First SRE Hire

    5 Tips If You’re the First SRE Hire

    Site reliability engineers (SREs) have a considerable set of tasks to juggle no matter where they work or how long their company has had an SRE practice. But if you’re the very first one to join an organization—as many SREs are these days, given that the trend is trickling down into smaller and smaller companies—you face a special group of challenges.

    You may find it difficult to get buy-in for SRE from other technical teams. You may struggle to know how to convince your boss that the company’s decision to hire an SRE is paying off. You probably worry that you’ll end up being overextended because you are the only SRE. And so on.

    These five tips are for site reliability engineers who find themselves in the position of being the pioneer at their companies. Whether you’re an experienced SRE who has just been hired at a company that had none previously or you’re brand-new to SRE and your first job happens to be at a company that is also brand-new to the SRE role, the following best practices will help you make the transition as smooth as possible for all involved.

    Collect SRE Metrics Early and Often

    For starters, you’ll want to be prepared to justify the business’s investment in your new role as soon as a boss or fellow engineer asks.

    Do this by collecting metrics about the technical and business impact of SREs starting as early as you can. Your goal should be to quantify how you are benefitting the organization so that you can encourage more support for yourself and any others who may be hired after you.

    On top of this, collecting metrics is also important for establishing early feedback about which SRE strategies are succeeding and which could be improved. Since you’re building an SRE practice from scratch, getting this insight early and often is critical.

    Treat Devs and IT Engineers Equally

    As you start the role, you may find some uncertainty among existing engineers about whose ‘side’ you are on. Are you there to support developers or IT engineers?

    The answer, of course, is both. Although SREs tend to borrow from software engineers’ playbooks more than they do from those of IT Ops, an SRE’s goal should be to empower and support both of these teams. So, unless you’re given explicit instructions to interface with only one group or the other, make yourself everybody’s buddy within the IT organization.

    Introduce Yourself to Non-Technical Teams

    SREs spend most of their time doing technical things with technical tools. But ultimately, the only reason they are doing what they do, in most cases, is to support non-technical users by improving the reliability of the systems they use.

    As the first SRE at a company, it may be well worth your time to get to know these non-technical stakeholders and ask them what they see as the greatest reliability issues in the organization. Building these relationships early on will help you communicate the value of your role across the organization. It’s also likely to come in handy during incident management routines, when you may need to interface with non-technical stakeholders as well as other engineers.

    Build Your SRE Tool Set

    Deciding which tools to use is an obvious early step for any SRE who is building a practice from scratch. But we didn’t put it first on this list because it’s important to get to know the rest of the organization before committing to a particular set of tools.

    As you meet the rest of the organization, ask detailed questions about which tools they use to address needs like observability, security and incident management. Then, by factoring in feedback about which tools work well and which don’t, choose which tools you’ll deploy to support SRE. Be prepared for those tools not to match your own personal preferences perfectly. You may need to prioritize the preferences of the rest of the organizations.

    Plan to Grow the Team

    Unless you have been hired at a very small company, it’s likely that the business plans to hire more SREs over time (assuming that you prove the value of the position, of course).

    Toward that end, plan for how the team will expand. Think about things like whether to use embedded SREs or create a dedicated SRE team. Think, too, about how you’ll document your company’s SRE practice and ensure it keeps operating smoothly as individual engineers come and go.

    Conclusion

    The job of a newly hired SRE is not an easy one, especially if you’re the first person at your company to fill those shoes. But by being deliberate and strategic about how you work, which data you collect and how you introduce the company to SRE, you set yourself – and your future colleagues – up for success.

  • SRE Vs. DevOps: The Wrong Question?

    SRE Vs. DevOps: The Wrong Question?

    The age-old question about the competition between DevOps and SRE sets up a false dichotomy. DevOps is a methodology while SRE is a team within operations. Although the two are often pitted against one another, developers also embody the skills and the capabilities to implement fixes. 

    Traditionally, application changes were reactive. The process of SRE formalizes problem resolution playbooks for better efficiency within the development life cycle. This has changed substantially over the past few years with DevOps seeing more traction and mainstream adoption.

    So what is the modern relationship between DevOps and SRE and how can they best be used together? Let’s understand DevOps first.

    Defining DevOps 

    According to Chef’s Embrace DevOps guide, DevOps is about transforming the way companies run; part of that transformation is understanding that companies should focus on managing people rather than products. DevOps evolved over the last decade and was originally born into the world of software. The adoption of cloud technologies, changes in the tech stack architecture, SaaS platforms, and security regulations have all impacted DevOps culture

    Some of the fundamentals of modern DevOps are:

    Everything-as-code: The current state of IT architecture has adopted an “everything-as-code” approach. Infrastructure, configuration, compliance and security can all be transformed into code.

    Automate everything: Codifying everything enables standardization of processes and policies across the organization. The processes are simplified into repeatable functional code, and this makes it easier to automate the development and deployment processes while significantly reducing human errors.

    Validation: There are many tools used for continuous integration, delivery and deployment in addition to other tools used for maintaining infrastructure and compliance. Without validation, there would be no way to know if everything is functioning as needed and it would be difficult to ensure seamless delivery in a highly agile environment.

    Above all, teams adopting DevOps methodologies take end-to-end responsibility of managing the application development life cycle (develop, deploy and maintain). They bridge the responsibilities gap between Dev and Ops into one team. 

    SRE Focus Areas 

    Successful site reliability engineering (SRE) teams focus on the volume of performance metrics and the process of shifting left the responsibility for managing and mitigating risk. Fundamentally, they share the same values as developers in creating efficient applications. However, SRE teams focus on introducing efficiencies based on incidents that occur and sometimes need development teams’ intervention to shift the risk left in the product development cycle. 

    Troubleshooting is one of the main responsibilities of SRE teams since they have to manage incidents. As a result, SRE teams have a strong knowledge of product design which is essential for product reliability, rollouts and reducing recovery time. Scalable automation is also a big part of the challenge SRE teams face. Oftentimes, tasks are done manually and this can increase the likelihood of human error. A solution is implementing a policy-as-code approach in concert with developer teams.

    It should be noted that when it comes to the responsibility and scope of work between SRE and DevOps, the lines are blurred—especially with the proliferation of APIs and when evaluating microservices. The types of tools being used are a differentiator since SRE teams measure efficiency through monitoring, detection and incident reporting systems. 

    Breaking the Silos 

    Modern organizations will fail when divided into silos. Developers understand that to optimize their internal processes, they need to use a DevOps methodology and involve SRE teams. For example, a startup growing and looking to scale needs development teams that are performing SRE functions with strong operational knowledge. It requires buy-in and understanding of every team member to know their role and the needs of the organization as it grows and responsibilities shift. 

    These fundamental values are crucial for an enterprise looking to achieve an efficient CI/CD pipeline. In a reactive cycle, SRE teams step in and are responsible for some parts of operations, but the DevOps paradigm is different because it integrates the responsibility of deployment, monitoring and remediation throughout the entire team. 

    Organizations embracing DevOps practices should embrace SREs’ expertise and welcome their practices into developer teams. SREs can leverage automation to fix issues as they occur and also make changes in software architecture to avoid such issues in the future. Compared to the reactive approach, developer teams are incentivized to keep things constantly running. 

    Scaling DevOps practices requires shared responsibility. Of note, developers have to answer when the phone rings and this can cause chaos in engineering teams responsible for large enterprise-grade applications with fast time-to-market and frequent update cycles.

    A Modern Perspective

    Efficiency is a core value shared by both SRE teams and the DevOps methodology. When organizations can fully integrate operations within the DevOps cycle, those silos can be broken down and integrate a shift left mentality. The traditional reactive remediation cycles won’t cut it anymore. Proactivity is essential for today’s lightning-fast application needs and continuous updates. 

    Application problems can be addressed immediately when approached with modern tooling and a policy-as-code approach enabling developers and SREs teams to work together. The modern relationship between DevOps and SRE teams should be one of coexistence. Both are needed for successful scalability within the enterprise. Many once thought SRE teams and developers couldn’t coexist in harmony, but debunking that myth helps organizations optimize their collaboration efforts. 

  • Prioritizing Scalability, Reliability and Security in Engineering

    Prioritizing Scalability, Reliability and Security in Engineering

    As digital products and services are more deeply embedded in critical industries and infrastructure and the implications of problems grow in scope, engineering organizations are renewing their focus on building platforms that are scalable, reliable and secure. More importantly, they are recognizing that this isn’t the responsibility of an SRE or CSO, it’s an ethos that must be carried each day by every engineer on the team. This is an important shift—from teams and cultures that valued speed of delivery above all else to teams and cultures that make high-quality systems the foundation of everything they do.

    This isn’t easy, and big cultural changes don’t happen overnight. There are many things that can go wrong when the team is operating in the legacy mode of prioritizing quick time-to-market but there are also some strategies and best practices for beginning the process of cultural change across your engineering organization.

    The Old Way of Doing Things

    You’re the engineer on call, and you get woken up at 3:00 a.m. by an alert that tells you an app is down. Groggy and annoyed, you look at the message and have no idea what service it’s referring to or where to begin.

    After nearly an hour, you can’t figure out who owns the service or how to get it up and running. So, you call that senior engineer on your team—you know the one—the person who gets called every time there’s a tricky question or problem. Thankfully, they answer your call in the middle of the night and help you bring the app back up. But in a month, the same scenario happens again; only this time, when you go to call that senior engineer, they’re no longer there. They quit to go work at a company that cares about reliability.

    Best Practices for Today

    How can you avoid this vicious cycle? Here are some best practices you can implement to begin shifting focus away from speed and toward scalability, reliability and security.

    1. Evangelize the Narrative That the Systems you Work on Matter

    At the end of the day, your systems are there to serve your customers and users. Leadership and teammates who embody these values are incredibly important. People should worry about letting their customers down, not about being yelled at by their manager or receiving a bad performance review.

    2. Set Explicit Expectations about Uptime

    When you say that your app needs to have 99.99% uptime, it doesn’t have the same impact as saying 52.6 minutes of downtime a year. Translate uptime numbers so they’re tangible—and on people’s minds every time there’s an outage.

    3. Don’t Sweep Problems Under the Rug

    After the dust settles from an incident, it’s important to get together and talk about it. Be transparent about what happened, which person or team made the error and what the implications to users were. The idea isn’t to point fingers or assign blame but to bring the team together so that you can learn, evolve and figure out how to prevent this from happening in the future.

    Here are some questions you can use:

    1. Who wrote the code?
    2. How was it tested?
    3. Should our automated testing have caught this?
    4. Who reviewed this code?
    5. Were they the right people to review the code?
    6. Why wasn’t it caught in the staging environment?
    7. Who released it and who went through the release validation checklist?

    Formalizing this process lets everyone on the team know that mistakes are part of the job and that there’s a standard way that problems are handled. This is a great way to help engineers feel free to do their best work without fear or avoidance.

    4. Define Individual Owners for Every Piece of the System

    If there’s any confusion about who owns what within your engineering org, you need to address that right away; it’s a massive problem. Not only do you need an owner for every part of the system, but they must also be empowered to fix issues and know that they’re the ones on the hook.

    5. Invest in Documentation, Runbooks and Logs

    The owner of each piece of the application is also responsible for making sure that documentation is up to date, runbooks are clear and easy to follow and logs are readily accessible. When you’re getting paged at 3:00 a.m. about a service you’ve never heard of before, these resources are key.

    6. Prioritize the Delivery Pipeline Like it’s a Key Feature

    You should always be thinking about how to build safeguards around your workflows, like automated testing or new tools. This may mean you need to go toe-to-toe with product managers to make sure the work required to build a robust delivery system isn’t getting kicked down to the backlog.

    7. Get Business Buy-In by Using Terms They Understand

    Unreliable code is technical debt. And debt is something all leaders understand, regardless of their technical abilities. Stripe published a survey in 2018 that found the average developer spends more than 17 hours a week dealing with maintenance issues, costing $300 billion per year globally. If you’re pushing to invest in SRE practices, those numbers might help you make your case.

    Culture comes first, but the right tools can reinforce the culture and drive accountability over time. Tools that help teams keep track of critical information such as service owners, documentation and runbooks can help tremendously. Additionally, having easily available dependency maps of the architecture are key to quickly understanding complex systems. Lastly, setting organization-wide standards for best practices and providing templates for spinning up new services that follow these can help ensure that these things are not forgotten.

    This article was based on a talk given by the author at Conf42: SRE 2021.