Tag: Blameless

  • Blameless Integrates Incident Management Platform With Opsgenie

    Blameless Integrates Incident Management Platform With Opsgenie

    Blameless this week announced it has integrated its namesake incident management solution with the Opsgenie alerting platform from Atlassian.

    Aaron Lober, director of product marketing for Blameless, said the Blameless platform is now integrated with both Opsgenie and PagerDuty, the two most widely used platforms for managing alerts.

    Blameless provides an incident management platform that normalizes data collected from a wide range of DevOps tools and platforms to determine how best to respond to different incident classes. Armed with those insights, it then becomes possible to quickly assemble the best team to respond to an event, said Lober.

    That capability reduces stress because Blameless also surfaces recommendations for resolving incidents based on the data it has collected, he added.

    The Blameless platform is tailored to organizations that have hired site reliability engineers (SREs) to manage IT operations using DevOps best practices, noted Lober. The overarching goal is to eliminate rote processes that, over time, conspire to burn out incident management teams that often experience high turnover rates, he added.

    Opsgenie has been connected to the Blameless platform via the service catalog and alerting functions. Blameless users responding to or managing an incident can escalate issues to any teams defined within Opsgenie. The service catalog and alerting data generated throughout the life cycle of an incident is automatically captured and stored on the Blameless Incident Timeline. That capability makes it possible to identify appropriate service owners and the escalation protocol that was followed by that team. Blameless can also display the actual user name of each responder that has been notified.

    When accessing Blameless via Slack or Microsoft Teams, Opsgenie can automate both when an incident channel is created and pre-select relevant services that should be made available. Alerts can also be manually triggered using commands from the Blameless chatbot for those who prefer to trigger alerts from within Slack or Teams themselves.

    As IT environments become increasingly more complex, more organizations are making a concerted effort to apply DevOps best  practices to incident management. The goal is to reduce downtime and increase overall resiliency at a time when organizations are more dependent on software than ever. As part of that effort, IT teams should identify the repetitive processes that span multiple incidents and then move to automate as many as possible.

    In the longer term, it’s not clear what impact artificial intelligence (AI) will have on incident management. At some point, though, IT teams should expect their efforts to be augmented by machine learning algorithms capable of identifying issues before they balloon into a major crisis. However, there will always be some human intervention required to manage incidents for the foreseeable future, noted Lober.

    In the meantime, most IT teams should continue to expect the unexpected at a time when IT environments have never been more dynamic and change multiple times a day.

  • Kintaba Creates the First Conference Dedicated to Incident Response – Techstrong TV

    Kintaba Creates the First Conference Dedicated to Incident Response – Techstrong TV

    John Egan, Co-founder and CEO at Kintaba, talks about IRConf, the first conference fully dedicated to Incident Response. The company behind the conference is Kintaba — a modern incident management platform for the entire organization. The video is below followed by a transcript of the conversation.

    Alan Shimel: Hey, everyone. Welcome to Techstrong TV. We have got another segment for you here. I’m happy to be joined by John Egan. John is the CEO of Kintaba Incident Response. 

    Hey, John. Welcome to Techstrong TV. 

    John Egan: Thanks, Alan. Good to be here. 

    Alan Shimel: Nice to have you. So, John, you know, I might have messed up what your position with Kintaba is. 

    John Egan: That’s right. CEO and Co-Founder. 

    Alan Shimel: Okay. And you know what? I mean I’ve heard of Kintaba, but that doesn’t mean our audience knows Kintaba, so if you wouldn’t mind, give us maybe a quick little background on Kintaba and maybe a little bit of your own background? 

    John Egan: Yeah. So, Kintaba is an incident response platform. We’re designed to let any organization be able to easily implement best practices for responding to critical moments within the company, so this tends to be things like site outages, major customer issues, and other sort of technical and non-technical problems that require kind of a group real-time response, you know, separate from like project management where you might have planned things that are happening. You take, like, the outage that happened, I think yesterday, at GitLab was a good example, right? Or a slack outage like that, that has an incident response team that’s responding to that situation and making sure it gets fixed quickly, and they’re probably using some kind of tool, either internally or off the shelf, and we’re that tool that any organization can implement in minutes instead of bringing in a consultant and spending months on this kind of stuff. 

    Alan Shimel: Absolutely. You know what was interesting, John, is I mean look, I’ve seen, for whatever reason since I’ve started devopsy.kinsta.cloud eight years ago, incident response has always been like this, you know, it’s part of the DevOps story, and we’ve seen of course, you know, several companies have very successful exits and be, you know, very successful. They all kind of follow this pattern where they, you know, were replacing the beeper, then they became incident response. Then it becomes a whole platform, and APIs and integrations, and so forth. What makes Kintaba different? 

    John Egan: So, we’re really purists on the incident response side of things. I think a lot of organizations that come into this space end up either kind of pivoting over towards alerting, all right, which is sort of Pagerduty’s world – 

    Alan Shimel: Right. 

    John Egan: Which is more about automated detection, ping someone, it gets closed out, you know, you might get thousands of these things across the organization, or they go broad and start to become, you know, more of like an SRE platform to try to handle all of these different things that the site reliability engineers are really managing inside of the organization. But we really believe that incident response stands on its own as a practice that cuts across all of the organizations inside of your company. So, something unique about Kintaba is normally when you’re rolling it out within the organization you might start small. You might start with five or six seeds, a couple of people in SRE, but really rapidly we tend to get deployed across the entire organization and what we do really well is we’re a collaborative layer that helps everyone from customer success, to your SREs, to even your PR and marketing guys all be connected during these critical moments and all have one place that they can go to see what’s happening and participate in that response. 

    It’s a bit more of, you know, we call it an evolved sense of incident response, right? Versus kind of the old-fashioned way of looking at it, which might be two or three people in your SRE team are responsible for this. It’s really more about the entire community of your organization that responds to these company critical moments and building software that makes it easy for all of those people to participate. So, we’re a very non-technical feeling piece of software. We’re very human driven, everything from our reporting, to our collaboration experience, to the way you go and write documents, which is a new feature we’ve just released, all of this information is designed to be as easy to use as something like Google, as opposed to being, you know, more like a, you know, almost like a service now, kind of like complex enterprise implementation. 

    Alan Shimel: Got it. Good stuff. And, John, we’re going to talk about this conference you guys are putting on, but before we go, before we dive into that, you know, so we understand what Kintaba is. You know, in your go-to market here what are you guys finding? What are you seeing? You know, what’s kind of the state of the market, if you will? 

    John Egan: So, we’re really learning a lot about this community as it’s just starting to form, right? I think when communities are young and they’re new you actually find the best practices are pretty fluid and I think it’s one of the unique things about this space is Google put out a book back in 2016 where they sort of defined how they handle incident response internally. It was the first time anyone really wrote it down in a way that could be absorbed, but it turns out every organization has a slightly different attitude that they take towards this space and when you’re implementing a product, something like Kintaba, it really turns out that that product has to be pretty flexible on its way in to make sure that it’s easy for people to adopt, and then it also has to bring with it sort of almost like a light hand to say these are the very high-level macro steps you’re going to take to be successful, right? 

    And so, what we’ve learned early on is when Kintaba first launched, you know, over a year ago, we were very rigid about the way that you sort of had to come in and step through, you know, each part of incident response. What we’ve really learned and what the products evolve towards is to have rigidity around the high level steps report openness around visibility to when incidents are happening, like easy to use collaborative spaces, post mortems that editable and changeable, but also, be flexible in terms of who’s getting access to this thing, how are you giving people titles that are non-technical in the, like, customer success roles, you know, and in the marketing and PR roles, and how do you bring all of that together at once. We thought when we came in that, okay, maybe this would be a couple of seat experience for people, you know, you come in and maybe you need two or three seats, greater, four or five, and what we’ve really learned is these are company practices, like I was saying before, and I think that’s been kind of our big learning about the space is as the community grows this is really a cross-functional community inside of organizations rather than just a technical practice inside of DevOps and I think that’s really exciting. 

    Alan Shimel: Yeah. Absolutely. Absolutely. So, John, you guys are coming in behind the first dedicated incident response virtual conference coming up in April. Why don’t you – well, I already said it I guess, but why don’t you take it from there? Tell us more. 

    John Egan: Yeah. So, IR Conf, or Incident Response Conference, was really built out of the fact that we just couldn’t find another one to be part of that was focusing purely on the incident responder. When we talk about communities internally, you know, you think of communities as what makes people put their hands up and say, “I’m part of that,” right? I think DevOps, and SRE, and a lot of these spaces have done a good job, right? We have crowds and people will say, “Okay. I’m part of that.” Incident response has always  been one of those ones where I think people sort of nervously said, “Well, I think I’m part of that.” What IR Conf really does is it focuses not on sort of that statement of I’m in the incident response department, but instead focuses on the idea that I’m a responder, right? I’ve been part of responding to a major incident and when you ask that question it turns out a lot of hands go up and it turns out that there’s a community there that’s really looking for other practitioners to come together and talk about these things I was just talking about, right, best practices, right? What are other companies experiencing? How are we defining this role and the people who are involved? And so, what’s really cool about this conference is we get to focus really directly on that incident response process, you know, not even necessarily as it relates to other parts of DevOps, but just how we deal with this very specific problem that faces all organizations. 

    So, I’m really excited we have one our keynote is Emily Freeman, who’s out of AWS – 

    Alan Shimel: Sure. 

    John Egan: And has done all kinds of great talks about kind of the wider, softer development lifecycle inside of DevOps, and here she can speak much more directly, you know, to this response, incident response space. I think you can see the excitement around it. You know, we have sponsors, basically everyone across the DevOps world, Honeycomb, Commodore, Rookout, BridgeCrew, Gremlin, Teleport. All of these folks, you know, really came running in and said, “Yes. This is great. We need a space to come in and talk to our sub-communities that are already using our products that care about this space,” and so really, we just felt like there was a hole in the market and so this is a free conference. Anyone can join. This is a virtual conference. It’s on April 1st. It’s at IRCONF.IO if you’re out there looking for it. We’re really hoping to just get everyone to come along who’s ever been involved in an incident before and has experienced that fire moment inside of a company of, you know, hopefully not panic, but maybe panic, and wants to come together and talk to other folks, who have had the same experience. 

    Alan Shimel: Sure. I’m thrown back to my cyber – we didn’t call it cyber then – to my infosec days in the war room, you know, when stuff hit the fan. You know, best practices that were to set up a war room on how to respond to a breach, you know, some sort of attack. 

    John Egan: And that’s really where a lot of this grew out of, right? 

    Alan Shimel: Sure. 

    John Egan: We have incident response really lived in the security world – 

    Alan Shimel: Yeah. 

    John Egan: And it’s one of these parts of the evolution of the space, right, that now we’re adopting, you know, that process that really got its feet inside of the security incident response space across the organization to say, “Wow. These guys are doing a great job,” like, “We should be doing similar work across the org.” 

    Alan Shimel: Not all companies had such a great job on their war room, but that’s a whole other story. 

    Hey, John, you mentioned Emily Freeman speaking. Who are some of the other speakers? 

    John Egan: We’ve got two more who are announced right now and it’s Emily Freeman, and it’s Dave Rensin. Dave Rensin was previously at Google and ran a good chunk of the SRE Team there. He’s at a new organization now and he’s spoken at a couple of other conferences in the DevOps space, specifically about the process of incident response. I’m really excited about him being able to come in as well and talk, both, from a large company standpoint, right, a company like Google, where he really pushed a lot of customer metrics, right, for dealing with incidents, not just internal technical metrics, as well as the smaller company, like where he’s moved to now is a startup.

    Those two are kind of our cornerstones. Then we’ve got a series of other speakers, who are announcing over the next couple of weeks – 

    Alan Shimel: Who?

    John Egan: Who are coming in from all over industry, so I’m really excited. I’m can’t announce them here yet. They’re not finalized yet. 

    Alan Shimel: Okay. Good. 

    John Egan: Really exciting. Great organizations. 

    Alan Shimel: Can’t blame me for asking. All right. Hey, John, for people who are interested, that was IRCONF.IO? 

    John Egan: That’s right. IRCONF.IO. You go in there and just click Attend and register. We’re still at early bird, so everything is free right now. 

    Alan Shimel: And it’s April 1st. Do you have times on that? 

    John Egan: I believe we’re planning to start it early, around 8:30 is the beginning of it. 

    Alan Shimel: East Coast time? 

    John Egan: It’s technically a half-day, but it runs until about 2:00, I believe. 

    Alan Shimel: So, that’s East Coast time though? 

    John Egan: East Coast time. Yes. 

    Alan Shimel: Yep. We’ve got a global audience here, so that’s East Coast U.S. time. 

    John, sounds great. You know, if there’s a need in the market, you know, the market has a vacuum, so it’s good to see it getting filled here. I want to wish you the best with it, with the conference and with Kintaba. 

    John Egan: Great. Thanks, Alan. 

    Alan Shimel: It was great having you. John Egan, CEO, Co-Founder, Kintaba, the folks behind IRCONF and you can find out more about that at IRCONF.IO. That’s coming April 1st. Check it out. 

    We’re going to take a break here on Techstrong TV and we’ll be right back. 

  • SRE: How Do You Get to Blameless?

    SRE: How Do You Get to Blameless?

    Site reliability engineering (SRE) is the companion to DevOps. If we take my normal divisions of DEVops versus devOPS, traditional DevOps is mostly the developer side, and SRE is mostly the Ops side. This isn’t a perfect comparison, but it is a good starting point, and for this blog, I’ll run with it.

    While DevOps focuses on getting releases out consistently, quickly and with minimum disruption, SRE focuses on putting those releases into a robust infrastructure. Tracking application issues, regardless of whether you’re using DevOps, always requires information about the software and hardware infrastructure. SRE takes the proactive steps to try and make sure the environment is stable and designed to host the applications being thrown onto it. When that process fails, SRE attempts to identify the code/architectural problem and resolve it without blaming staff members—hence, the idea of “blameless.” It’s about the environment and improving it, not about the individual and punishing them.

    There are a lot of interesting parts of SRE that most IT groups could look at and see as a good place to start taking advantage of. Closely tracking errors in a cross-section of IT interest areas—by application, by infrastructure, by toolset—means problem areas can be definitively identified and fixed. But there is more behind SRE. In this example, once tracking of where errors occur more frequently is in place, a selective reduction in deployments using that piece of the IT puzzle can go into effect until the issues are resolved. This is not unlike what we are doing today, but it is formalized and has measurements around it that generally are not in place.

    Which brings us to Blameless, a company that just came out of stealth to champion SRE. While some large organizations with large application footprints are using SRE already, Blameless is offering a platform to make SRE more accessible to companies without large staff and application footprint.

    We took a bit of time to talk with Blameless last week, and their vision is a strong one: Address directly what DevOps addresses more indirectly. Build and environment that can measure and improve the entire application ecosystem.

    If nothing else, the idea of error budgets is intriguing. Counterintuitive at first glance, the idea of saying, “Piece X gets Y amount of issues before we start backing off of it or raise the priority to fix the underlying issues,” is a solid one.

    If your organization is looking for a mechanism to improve reliability and stability while reducing time to resolve issues, SRE in general and Blameless specifically are worth a closer look.

    — Don Macvittie

  • Blameless Launches DevOps Platform Based on SRE Principles

    Blameless Launches DevOps Platform Based on SRE Principles

    Blameless has emerged from stealth to launch a platform for managing DevOps based on the site reliability engineering (SRE) best practices defined by Google.

    Fresh off raising an additional $20 million in funding, the Blameless platform provides across-the-board visibility into the complex network of application programming interfaces (APIs) and applications distributed across an organization, said CEO Ashar Rizqi. Specific modules within the Blameless platform address incident resolution, reliability insights dashboards, postmortems, service level objectives (SLOs) and automatic error budget calculation.

    Blameless takes its name from an SRE principle that no one should be held at fault when complex systems fail. Rather, the assumption should be that inherent complexity of the IT environment is at fault. While DevOps is more of a philosophy surrounding the management of DevOps, SRE is emerging a more prescriptive instance of implementing DevOps that Google is now promoting. The goal is to make sure that whatever the issue, the right people to address it are brought together at the right time, said Rizqi.

    The Blameless platform was created by Lyon Wong, who now serves as the COO of Blameless. Previously, Wong was a partner at Lightspeed Venture Partners, founder of S28 Capital and a product manager for Microsoft Windows. Rizqi is a former head of core platform at MuleSoft and SRE manager at Box.

    Rizqi noted that while most organizations are trying to embrace DevOps principles, they often lack the tools needed to build the workflow processes required. All too often, organizations rely on little more than a Wiki document to manage their response to an incident. The Blameless platform is designed to provide access to an integrated set of tools for managing everything from the incident itself to the impact that event had on the business, he said.

    It’s still too early to say with any certainty how widely SRE practices will be adopted. This far, most organizations that have adopted DevOps have tended to prefer to define a set of processes that are customized to their environment. But there are just as many, if not more, organizations that have found it daunting to get started. Platforms such as Blameless represent an opportunity for those organizations to embrace DevOps using a more prescriptive approach.

    Of course, a big part of the reason that DevOps has been embraced unevenly by various departments within the enterprise is the lack of a consistent approach to implementing DevOps at scale. Teams that have embraced DevOps with enterprises have tended to create their own individual processes using a wide variety of tools. Many of those organizations have also found it challenging to get IT operations teams to embrace those processes because they require an approach that can be implemented consistently across multiple projects at scale. SRE provides IT operations teams with a framework that Google has already proven works at scale.

    Regardless of the path chosen, the one thing that is for certain is more agile approaches to managing IT operations will be required to keep pace with the rate at which applications are now being developed and deployed.

    — Mike Vizard