Tag: SRE

  • DevOps Chats: Jim Gochee, Chief Product Officer, New Relic

    DevOps Chats: Jim Gochee, Chief Product Officer, New Relic

    In this DevOps Chat we speak with Jim Gochee, Chief Product Officer, New Relic. In his role at New Relic, Jim has seen years and years of evolution in digital transformations. Jim and I spoke about a number of topics including about how to succeed with DevOps, the rise of SRE and APM. Great discussion all around.

    As usual, the streaming audio of our conversation is below, the transcript follows that. Enjoy.

     Audio

    Transcript

    [su_pullquote align=”right”][su_custom_gallery source=”media: 122025″ link=”post” target=”blank” width=”200″ height=”200″ title=”never”]Jim Gochee
    Jim is New Relic’s Chief Product Officer, overseeing the teams that design, build, and deliver New Relic’s products. He has a passion for building world-class products and loves technology – especially when it’s well architected, scalable, and fast. He takes great pleasure in helping New Relic’s customers analyze their technology and build great digital experiences. Prior to New Relic, Jim was the Chief Architect for the Wily Technology division of CA, where he was the technical lead for Introscope, an industry-leading web application performance monitoring solution. Prior to that, he was Chief Technology Officer for an internet-based trading system for the wholesale food industry. Jim also spent four years at Apple, Inc. and three years at Connectix Corporation working on VirtualPC. He holds an A.B. in Computer Science from Dartmouth College.[/su_pullquote]
    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud and you’re listening to another DevOps chat.  Today’s DevOps chat features Jim Goci of New Relic.  Jim, welcome to DevOps chat

    Jim Gochee: Hey, Alan, how are you?  It’s good to be here.

    Shimel: Thank you, it’s good to have you here.  So, Jim, let’s get some stuff out of the way here, a little housekeeping.  What exactly is your role at new relic?

    Gochee: Yeah, Alan, so I oversee all of product development and that includes engineering, product management, design as well as operations.  So pretty much the whole shooting match.

    Shimel: Fantastic.  And Jim, I was talking to you off-mic before we started today, we haven’t done a good job frankly of covering New Relic at devopsy.kinsta.cloud but when we first launched, which is going on five years ago, New Relic was already a well-established player in the DevOps market and is certainly even more established now.

    But why don’t you give our audience maybe just a quick little history to bring us up to where New Relic is today.

    Gochee: Yeah, absolutely.  We’ve been around actually for ten years, Alan. So we just had our tenth-year anniversary party just a couple of weeks ago if you can believe that. So, 2008, and when we started, we were focused on Ribyon Rails, monitoring the backend services, grew the product portfolio over time to include other languages.

    But mostly sold to small companies, startups, companies that got bigger like Air B&B. But really, it was a couple years ago we started to pivot more into the enterprise and that was a very exciting time for us.

    We were already getting pulled into bigger and bigger companies at a project level and then we found ourselves being kind of drawn in organically to more and more groups to the point where these big companies were wanting to have more of a formal relationship with us, from a sales perspective.

    So we built out an enterprise sales team and have been really focused there. So maybe, in one way, while you haven’t seen a lot of us in the last couple of years as we’ve been pretty heads down, focusing on the needs of the enterprise market. But what I’ll also say is we’ve been releasing a lot of product as well in the last a couple of years and the biggest thing that we did is we rounded out our offering to include an infrastructure monitoring component.

    And that really complicates, not complicates, but it complements our offering at the application layer as well as the end user layer. So at this point, we have a pretty comprehensive portfolio of offerings that cover that full stack.

    Shimel: Very cool, very cool indeed. So Jim, I would imagine in your role around product and everything you’re doing at New Relic, you get to talk to more than a few customers, yeah?

    Gochee: Well, I do and I just feel so blessed, Alan, because this is an incredible time to be in technology. So I’ve seen a few really interesting times in tech. I think the first was when the Internet really started to get mainstream, back in the mid-’90s, but what’s really interesting now is this thing that you hear a lot about which is digital transformation.

    And we all feel it, which is we want a relationship with the companies that we do business with, we want a relationship via our mobile phones or the web browser. And so a lot of companies are trying to figure out how do they have that new customer relationship, right? And how do they come up with new ways of engaging with their customer?

    And then, of course, again, companies like Air B&B completely rethinking business models, companies like Uber, and so it’s just a really fascinating time to be in the tooling space that supports and assists a lot of these digital transformations that are going on. So, yeah, we definitely see a lot of interesting things.

    Shimel: Excellent. So, Jim, one of the themes that we have run into here recently at devopsy.kinsta.cloud is this idea that somehow people thought digital transformation and DevOps transformation was gonna be easy. And they’re a little bit taken aback.

    You know, you speak to them, the first kind of foray into it it’s like, you know, this is hard. And I feel like saying no one promised you a rose garden, right?  If it was that easy everyone would be doing it already. Are you seeing any of that?

    Gochee: Well, you know, Alan, in fact, we do see a lot of that but I would say is this, I guess I see the full continuum. I see the early adopters and the enterprise to DevOps. I mean, DevOps, you guys have been around for five years and we’ve been doing it for 10 and it’s early to do when you start a company that way.

    In other words, when you start a company and you have a DevOps organization already, it’s easy to grow and  keep doing that. But when you’re a traditional enterprise and your operations team is very siloed from your development teams, to bring those two together is a pretty big task, right? And you hear a lot about the cultural challenge involved, and that’s very real.

    The thing I’d like to ask, the question I ask when I go into one of our customers and I’m not familiar yet with really where they’re at in the journey, I ask them one simple question. Who takes pager call? And if it’s still the operations team that’s taking pager call, then normally they’re still complaining that the developers don’t care, the developers are very isolated, and you see those silos still in place.

    And it’s really not until a development team, then with the help of operations, people like SREs embedded on that team, it’s really not until that happens where I would really see the crossing of the chasm into more of a true DevOps model.

    Shimel: Absolutely. You know, you mentioned SREs, Jim, and that’s another area where we’re just seeing it, man, it’s blowing up, right, in terms of companies hiring SREs, incorporating SREs on to their ops team. Many people have gone as far to say SRE puts the ops in DevOps. What are your feelings around the SREs and that?

    Gochee: Well, what I like about the SRE role is Google wrote a book about what they believe it is. And so while that probably just applies to Google and nothing’s ever as perfect as it seems, they clearly define these are the things that this role does.

    And this is why these things are important, and it’s a real philosophy and it’s a real mindset, and they wrote it down and these are all things that other people had been doing in various ways. But I think the SRE role, it got really clarified what exactly it is.

    And so I am starting to see this more and more as well, and it’s not like SRE is a new concept; again, Google’s been doing it for a very, very a long time but I just feel like it’s kind of catching on.

    I can tell you ourselves we’ve been doing DevOps for a long time but just in the last year, we’ve really formalized around SREs as well and we got a vice president of site reliability engineering, we converted people’s titles over, we got really crisp about exactly what are you doing.

    We’re trying to figure out right now whether central versus embedded or a hybrid of the two approaches, so you know, how to organize SREs and how do they really attach into development teams? There’s still plenty of work to do even that we’re looking at, but I think this is helping the industry really figure out a model that they can work with.

    Because that’s the biggest thing I see, Alan, is people want to transform to be more agile into DevOps and to use the Cloud, but they don’t really know how to do it, right? They don’t really have a good playbook for it yet.

    Shimel: I agree with you 100 percent. It’s interesting you guys are eating your own dog food with that.

    So, Jim, in many ways, New Relic, sort of, if you will, pioneered what we call APM, you know, and we recently did—it got too big but it was a DevOps tools primer, just trying to at least draw it so that the taxonomy of categories—and certainly APM, but APM itself has become so intertangled in this whole software development life cycle. Is that still where New Relics thinks they play?

    Gochee: Yeah, so it’s a great question. We like to say that we’ve democratized APM; in that, we created a service that anybody could subscribe to, download our agent, install it into an application code, relaunch your application and now you’ve got your data showing up on our screens.

    The technology around APM, though, has been around a lot longer but we were the first to bring it to everybody in a very transparent selling model, right, and again try it before you buy it. And we have by far, in a way, the most users in this APM category. These categories are funny things though, yeah, applications here but there’s also end user, there’s synthetics, now there’s infrastructure tier.

    It’s all kind of collapsing and probably the markets sort of get renamed after a while. One of the things that we like to say that we think we have is a digital intelligence platform, digital intelligence just means data about everything that’s part of your digital system, all the code running, all the infrastructure that’s running.

    And so at the end of the day, if you want to be agile and deploy a lot code and make a lot production change, you need to have data that tells you that things are still working correctly about all the different pieces of your infrastructure, right?

    And without that data, you’re flying blind and so, flying blind, it’s okay if you deploy code once every six months, you can fly blind for a little bit while you sort out any issues. But if you’re deploying code every day, you’ve got to have real-time data about what’s happening in order for that to be successful.

    Shimel: Sure do, sure do. And Jim, when we talk about the move to the enterprise, I think that’s one of—not that startups or non-enterprise customers don’t use data or that they do fly blind. They don’t. That’s not the case at all. But when you go to the enterprise and you start thinking about enterprise-class functionality, right, is when you start realizing what—really, things have got to be locked down, right?

    There’s got to be sort of—I mean, sometimes it can be overprocessed and overcomplicated but you can’t just fly by the seat of your pants as much. And I think products like New Relic, they can’t help but bring some of that regimentation to a company.

    Gochee: Absolutely, no, I mean your point’s really well-taken. When you’re a smaller company, you can take more risk, you don’t have to do things the right way, you know. That’s common. When you’re bigger company, you have a lot more experienced people who are used to having better tooling, better processes, better governance, right?

    Because there’s a lot at stake. I mean, these are some of the largest companies in the world who are betting their future on their digital projects, and those projects have to function and they have to work. So, I don’t know, can I tell you one story, Alan?

    Shimel: Sure.

    Gochee: It’s a few years old now but do you remember when healthcare.gov first launched?

    Shimel: Yes, I do.

    Gochee: It first launched and the thing didn’t really work. It was hung and broken and the president was like apologizing to the nation, right? This is an example of we looked at each other and we’re like this is the highest-profile example of a web project gone awry.

    Shimel: [Laughs]

    Gochee: Yeah, right, I mean the president has to apologize for poor web performance and –

    Shimel: At least we had a president who apologized then, but go ahead.

    Gochee: That’s another story altogether.

    Shimel: That’s a whole other story.

    Gochee: The president brought in a team from Silicon Valley to help fix that site. I forget Mikey’s last name from Google. Anyways, they brought New Relic in and then we gave them the data to help sort that out, and then from now on, we have been the tool of record for how that system has been performing. And they’re just 1 of 15,000 customers that we have.

    Shimel: Interesting.  But that’s a classic example of it, too. That was done probably, frankly, without the necessary requirements in laying it all out there. Jim, let me ask you a question. From your catbird seat, what do you see at the future of the market here?

    Gochee: Yeah, so just in terms of digital transformation and digital customer experience, that’s absolutely gonna be powering the whole economy and tech sector in particular for the next five to ten years. Cloud, the move to Cloud, the opportunities you can get by some of the Cloud services.

    In particular, Amazon is really blazing the trail in just the sheer number of services they offer the market, changing how people can build and deploy software, it’s incredible. So we see a continued explosion in software and digital experiences, and then that drives companies like us, we offer tooling in this area that really helps companies move more quickly, have a better customer experience. And so I mean, I think the whole category of tech and DevOps and Agile and tooling, observability is a trend, SREs, all of this is gonna continue to have a lot of strength and a lot growing over the next five or ten years. I just absolutely believe that.

    Shimel: Excellent, Jim, as I mentioned to you, when we first got started here, 15 minutes goes really quick. I think we’re probably closer to 20 at this point but it’s okay. Maybe we can have you back on here at New Relic and a DevOps chat because we would like to stay on board with everything that’s going on there. But in the meantime, thanks for being a guest on this episode of DevOps chat, Jim.

    Gochee: Alan, it’s been enjoyable, thank you so much for having me.  I’m happy to come back on as well 

    Shimel: Absolutely, we’ll make you a regular. Hey, that was Jim Gochee, Jim runs products and a bunch of other things over at New Relic and this was another episode of DevOps Chat. This is Alan Shimel for devopsy.kinsta.cloud. I hope to see you soon on another chat soon, everyone. Take care.

    — Alan Shimel

  • DevOps Careers: Report Examines What It Takes To Be an SRE

    DevOps Careers: Report Examines What It Takes To Be an SRE

    Do you want to be a site reliability engineer, or SRE? Sixteen years ago, the role didn’t actually exist, but now you can find more than 1,000 opportunities for a career as an SRE on LinkedIn. Before you decide if you want to explore such a career, though, you should check out the new survey from Catchpoint, which examines the role and what exactly SREs do.

    Catchpoint launched its survey in January with the goal of determining what exactly it means to be an SRE. The survey takes a closer look at the types of organizations, skill sets and organizational cultures in place where SREs work. Catchpoint wanted to build a profile of the SRE role and determine whether there is a core set of principals or skills common to the SRE role.

    The report includes responses from 416 participants representing a range of roles, industries and company sizes. Nearly 4 out of 10 respondents actually identify as SREs. More than half work for companies in technology-related industries, and more than 40 percent indicated that they work for a software-as-a-service (SaaS), infrastructure-as-a-service (IaaS) or other as-a-service (XaaS) provider. More than 50 percent represent companies with at least 1,000 employees, while just under 40 percent are from larger organizations of 5,000 or more employees.

    What is an SRE?

    The SRE role originated at Google in 2003. The SRE straddles the line between the “Dev” and “Ops” sides of DevOps. SREs foster collaboration, productivity and software performance and—ultimately—gives developers greater responsibility for the performance of the applications they produce.

    SREs typically both write code and support existing systems. Nearly two-thirds of SREs report they are deploying code daily—with almost half indicating that they deploy code multiple times per day. According to the report, most SREs report to software engineering despite the fact that a large percentage of SREs actually come from an IT operations background.

    Coding skills and a solid knowledge of IT operations are both valuable resources for an SRE, but there are also a variety of “soft skills” that seem to be crucial to success. Problem-solving, teamwork, composure under pressure and strong written and verbal communications regarding incident resolution are all necessary to be effective as an SRE.

    Novices Need Not Apply

    The Catchpoint survey contains some bad news for fresh college graduates looking for their first gig, and excellent news for experienced IT professionals: The SRE role is not an entry-level job. Eight out of 10 SREs have a college degree and have worked in IT for six or more years. A college degree in computer science or information technology isn’t necessarily a requirement, but 73 percent of SREs did, in fact, get degrees in a technical field of study.

    Although a majority of SREs surveyed report to the software engineering department, most come from an operations or infrastructure background. Just under 65 percent of survey participants reported that they held a role as a sys admin, while only 53 percent previously held a role as a developer or software engineer.

    To learn more about the site reliability engineer role, check out the full 2018 SRE Report from Catchpoint.

    — Tony Bradley

  • SRE vs. DevOps — a False Distinction?

    SRE vs. DevOps — a False Distinction?

    Just a few days before he died at the beginning of the 1990s, a wise man taught us that “the show must go on.” Freddie Mercury’s parting words have long provided the guiding light for many, if not all, ops teams. In their eyes, the production environment should be exposed to minimum risk, even at the expense of new features and problem resolution.

    About 10 years ago, Google decided to change its approach to production management. It took the company only a few years to realize that while R&D focused on creating new features and pushing them to production, the Operations group was trying to keep production as stable as possible—the two teams were pulling in opposite directions. This tension arose due to the groups’ different backgrounds, skill sets, incentives and metrics by which they were measured.

    Trying to bridge this gap between the two groups, one of Google’s ops leaders, Ben Treynor, thought of an innovative solution. Instead of having an Ops team built solely from system administrators, software engineers—with an R&D background and mentality—could enrich the way the team worked with the development group, change its goals and help with automating solutions.

    Site Reliability Engineering, or SRE

    And so, the Site Reliability Engineering position was created. According to Google, SRE engineers are responsible for the stability of the production environment, but at the same time are committed to new features and operational improvement. Google decided its SRE teams should be composed of 50 percent software engineers and 50 percent system administrators. The engineers were driven to use software as a way of solving problems and perfecting what had historically been solved by hand. They integrated easily with the development team, and encouraged code quality improvements and automation testing.

    Understanding the value of SRE, several organizations of various sizes decided to embrace its principles. Some, such as Dropbox, Netflix and Github, are well-known for being at the forefront of technology leadership.

    Wait, Isn’t That DevOps?

    DevOps is a more recent movement, designed to help organizations’ IT department move in agile and performant ways. It builds a healthy working relationship between the Operations staff and Dev team, allowing each to see how their work influences and affects the other. By combining knowledge and effort, DevOps should produce a more robust, reliable, agile product.

    Both SRE and DevOps are methodologies addressing organizations’ needs for production operation management. But the differences between the two doctrines are quite significant: While DevOps raise problems and dispatch them to Dev to solve, the SRE approach is to find problems and solve some of them themselves. While DevOps teams would usually choose the more conservative approach, leaving the production environment untouched unless absolutely necessary, SREs are more confident in their ability to maintain a stable production environment and push for rapid changes and software updates. Not unlike the DevOps team, SREs also thrive on a stable production environment, but one of the SRE team’s goals is to improve performance and operational efficiency.

    Google tried a few approaches to implementing the SRE process before finding the one that suited them best. One of these approaches attempted to tie the number of permitted releases to the product’s stability. The principle underlying this process is that new releases are green-lighted based on current product performance.

    For each service, the SRE team sets a service-level agreement (SLA) that defines how reliable the system needs to be to end users. If the team agrees on a 99.9 percent SLA, that gives them an error budget of 0.1 percent. An error budget is exactly what the name suggests: the maximum allowable threshold for errors and outages. Here’s the interesting thing: the development team can “spend” this error budget in any way they like. If the product is currently running flawlessly, with few or no errors, they can launch whatever they want, whenever they want. Conversely, if they have met or exceeded the error budget, and are operating at or below the defined SLA, all new releases are frozen until they reduce the number of errors to a level that allows the launch to proceed. This process ensures that both the SREs and developers have a strong incentive to minimize the number of errors in production.

    Another interesting approach recommended by Treynor, which is more related to SRE professionalism and efficiency, is allowing SREs to move between projects. Moreover, he suggests allowing SRE engineers to move to development, and even the other way around. As the work done by both teams is similar, these transitions help the Ops team gain better and deeper knowledge of the product and code, and bring the Dev teams into the production space to help them understand its challenges. This strongly promotes a team atmosphere, rather than one in which an individual feels that “I’m on the SRE team for this product.” As part of this approach, Treynor had Dev teams handle 5 percent of the operations workload. This, according to many organizations, adds to the SRE team’s motivation and effectiveness.

    Riddles in the Dark

    Well, SRE seems perfect. As already stated, several large-scale organizations have chosen to move some of their production operations from old Ops to SRE. However, there are still a few questions that need to be asked:

    Is the position of SysAdmin (production/ops) no longer relevant for SysAdmins?

    Historically, almost all system administrators have come into their roles through tech support and similar work, or even just running Linux on their desktops and then transitioning into server work. It should be pretty clear that the same path is not available into SRE. To retain their positions, SysAdmins should now be more code-oriented, have better technological knowledge and be receptive to new methods of conducting the work they already do.

    Can an SRE team prevent production incidents?

    One of the strengths of a professional site reliability engineer is the ability to handle the growth of production load and traffic in-house. Monitoring and analyzing processes and logs with platforms such as the ELK Stack is part of the day-to-day workload, and the team should be able to identify problems as they occur, and even foresee risks to software stability. The power of this position—based on the skill set it requires—lies in developing solutions for these problems and risks. As Ciara, a software engineer in Google’s cloud storage SRE team, described it in a “Life in Google” post, “We solve cooler problems in cooler ways.”

    Are we doing DevOps or are we doing SRE?

    This question is usually asked by people trying to position themselves in the Operations world. According to many companies that implemented SRE in a slightly different way than Google, you don’t have to decide. At Reddit, ops engineers work on reducing toil, improving deployment and scaling processes, but they are referred to as “DevOps.”

    At Logz.io, we bridge the gap between developers and production through the use of automated monitoring and performance stress-testing. And we personally refer to it as “DevOps” rather than “SRE.”

    So, is it Just a Question of Context?

    In her blog, Charity Majors talks about SRE and DevOps being two different operational approaches that any organization can choose to work by, but insists on emphasizing that there is no “correct” approach. Although Google and Dropbox have decided on SRE, this does not mean the rest of the world should do so as well. What fits Google’s needs and organizational philosophy does not necessarily work for other orgs—at any scale. Moreover, Charity believes that DevOps, having grown and evolved within a broader variety of software organizations, is the more flexible, collaborative and adaptable approach, and will work better for most software organizations at all stages of development.

    On the other hand, SoundCloud senior engineer, Matthias Rampke issued a series of tweets on how SRE and DevOps were basically the same, with only one difference—high management support. Though in contradiction to her blog, Charity lent weight to this opinion during an AskMeAnything event that revolved around DevOps and SRE. She shared a story about a friend who was hiring for a startup, which, as an experiment, posted the exact same job description twice, the only difference being that one listing was titled “DevOps engineer,” and the other “SRE.” At the time of relating the story, DevOps was winning by 10 percent or 20 percent. However, all this means is that the job title is irrelevant, and naming an SRE team does not magically bestow qualities upon it. Rather than focusing on the title, organizations must focus on the work being done.

    To summarize, we can quote Matt Simmons, a technologist with many insights into this topic, who says, “Not every infrastructure needs an SRE, but every infrastructure could use an administrator who acted more like one.”

    — Asaf Yigal

  • SRE vs. DevOps vs. Cloud Native: The Server Cage Match

    SRE vs. DevOps vs. Cloud Native: The Server Cage Match

    This post supports my DevOpsDays Austin Presentation on May 4

    I don’t believe in DevOps shaming. Our community seems compelled to correct use of DevOps as an adjective for tools, teams and teapots. The frustration is reasonable: DevOps clearly taps into head space for both devs and operators who see a brighter automated future together. For example, check out this excellent DevOps discourse by Cindy Sridharan.

    As an industry, we crave artificial conflict so it’s natural to try and distill site reliability engineering (SRE), DevOps and cloud native into warring factions when they are not. They all share a focus on Lean process.

    Very simply: SRE is a job function, DevOps is a process approach and Cloud Native is an architecture.

    These three concepts have tremendous alignment because they all have the same fundamental goal: increase product delivery through improved flow. It’s really that simple. We are looking to enhance productivity using system thinking concepts from Lean manufacturing.

    SREs improve productivity by eliminating manual operations, reducing errors and improving infrastructure quality. They do this starting from the production infrastructure and working backwards. Highly automated delivery, integrated logs, systemwide monitoring and blameless root cause analysis are key tools. Before we get wrapped up in tools, the distinguishing factor for SRE teams is that they take ownership of the production infrastructure.

    There is tremendous power in assigning ownership for delivery. That’s why I see industry benefit in defining an SRE job function. A DevOps strategy is a system approach and it becomes an anti-pattern to making an individual or team responsible to own “the DevOps.”

    In DevOps, we look for ways to streamline the entire product creation flow. That often drives toward tools and techniques that SREs require; however, our goals are not department-specific. We want to work with all stakeholders including developers, product managers, testers, SREs, executives and customers. This system focus for developers becomes CI/CD pipelines, platforms, work tracking tools and other workflow elements. These tools are not specifically DevOps tools, but confusion is natural since they are often pulled into organizations due to DevOps thinking.

    This SREs should expect to collaborate deeply with other functions under a DevOps process.

    Cloud-native architecture also falls under the DevOps mandate because it strives to decompose large application stacks into highly automated collections of services. There is no magic “DevOps platform” in cloud-native design; however, teams using the architectural approach get modularity, strong APIs, service orientation, simplified configuration and build automation practices that fit perfectly into DevOps thinking.

    For example, using containers does not mean a dev or SRE team is following DevOps process or using cloud-native architecture; however, it is an indicator that they are leaning toward smaller deliverables and improved dev-production fidelity.

    Yet, SRE, Cloud Native and DevOps do have a shared enemy: Complexity.

    We’re doing a great job creating best practices and abstractions that accelerate developers and creating innovative infrastructure options in both cloud and physical. That means we are seeing dual explosions in developer productivity and infrastructure complexity. Unfortunately, this wealth has not made it into operations automation, which has been delivering incremental improvements at best. Even worse, the added workload and complexity adds pressure to SRE functions.

    Ultimately, all three of these concepts are aspects of Lean business practices; however, acceleration from DevOps and cloud native is outpacing progress by SREs. This imbalance creates risk for everyone involved because Lean is about system performance. Delays and failures cannot be isolated to a single silo.

    The mistake of seeing Dev and Ops as silos means creating false contention between SRE, DevOps and cloud native.

    If you are hearing that in your organization, then take it as a warning about how your teams are integrating Lean systems thinking into your processes.

     

    — Rob Hirschfeld

  • Site Reliability Engineering: How to Make the Operations Side of DevOps Actually Work

    Site Reliability Engineering: How to Make the Operations Side of DevOps Actually Work

    The DevOps movement often has been accused of focusing too much on the first half (Development) and not enough on the second half (Operations). Certainly there has been more attention paid to deployment of payloads than to operating running systems, leading to the dismissal of the handover between Dev and Ops as “throwing it over the wall.”.

    Lately, we have seen the emergence of a new focus on the stability and reliability of the systems that are the targets of those deployments, with the creation of the new practice of site reliability engineering (SRE). This has been a welcome addition to the IT toolbox, but it can still seem to put all the onus on Ops teams to catch whatever comes over the wall.

    The Problem with Site Reliability Engineering = IT Infrastructure

    In SRE, the IT infrastructure is expected to be highly automatic and self-healing in the face of any events. Here is the problem: That approach works well for foreseeable events, but less well for the unforeseen ones. Because of that factor, it works well in environments that run few types of individual workload types, but do so at very large scale. If you’re thinking that sounds like Google and Facebook, you’d be right.

    What this amounts to is massive engineering of the infrastructure to withstand known or foreseeable problems. However, typical enterprise IT environments are not like that. A large bank might have thousands of applications, each of which must accommodate changes on a pace that is dictated from outside the IT environment. Applying assumptions from one environment to the other is asking for trouble.

    To illustrate this, let’s look at an example from outside IT. Recently was the anniversary of the sinking of the Titanic, on April 15, 1912. The RMS Titanic itself embodies the principles of SRE, engineered to prevent or survive all manner of emergencies. The ship was equipped with state-of-the-art everything to transport its passengers in comfort and safety across the Atlantic.

    As we all know, that plan did not quite work out. A combination of unexpected changes in the environment (more icebergs than normal), business imperatives to maintain speed and mishandled warnings about the ice from other ships led the Titanic to disaster.

    While the consequences of an IT failure are rarely quite as dramatic as the sinking of an ocean liner, many of the same factors apply. IT administrators may believe that in the worst case their backups and disaster recovery plans will be sufficient to handle any problems, but these plans are designed around known and foreseeable problems. Unexpected circumstances can easily lead to cascading failures, which is why it is critical to be ready for the unexpected when it—inevitably— occurs.

    The Problem with Traditional IT Ops = Static Models

    The key flaw in IT Ops is reliance on static models of the world. These models come in many shapes – the CMDB is the “model” that many think of first, but static rulesets are also a model. The most dangerous models, though, are the ones that are invisible: namely, the filters that are put in place to determine which alerts and events are even worth considering.

    This is the flaw that ultimately sank the Titanic: The crew were unable to correctly priorities events and react to them in time to avoid failure. It is also the flaw that causes countless much more minor issues every day in data centers everywhere: the filesystem that filled up with logs, taking down the database that the critical business application relied on—and the filter that had not forwarded that alert because it usually was harmless. Or the loss of one leg of the redundant network link, which was ignored because the other leg was still up, so there was still a link—until that one failed too, and an entire site went dark.

    The key factor is that the filters were not wrong to suppress those alerts. By definition, informational or warning alerts are not the same as major or critical ones, and most of the time they can be safely ignored, to be dealt with later.

    Every now and then, though, a pattern of those alerts, if understood correctly, can identify a future problem in the making. These developing issues could be nipped in the bud, if only Ops teams had enough hours in the day to look at them and review them. But, of course, they don’t—every Ops team I have ever met is drowning in issues, and behind those there is a long and lengthening to-do list.

    What Can DevOps Do to Make Operations Better

    New approaches are emerging to make the principles behind SRE more widely accessible and applicable. In particular, more dynamic noise reduction and correlation is now possible, to sift those important alerts from the constant background noise and put them together into a picture of what is really happening. The key factor is to be able to do this in real-time and without a human having to plan out laboriously what are all the possible scenarios that they might need to know about in the future.

    Gartner has called this new approach Algorithmic IT Operations, or AIOps. The idea is to bring together all possible sources of events, whether those are alerts from the compute or network infrastructure, transaction slowdowns reported by an APM tool, automated deployments being run from a CI/CD toolchain, or anything else the might conceivably be relevant. All of this information then can be sifted by algorithms to understand what is actually important to Ops and brought to the attention of the right specialists who can work on the issues and get them solved fast. Part of that process is also the integration with systems of record (which generally means IT service management), and with automation and orchestration tools that can accelerate remediation activities.

    This is how we can get the Ops side of DevOps to where they need to be to be able to accommodate the ever-accelerating pace of change—whether from Dev teams wanting to run ever more frequent deployments, or from business users needing help with their own goals—or from the next unexpected issue to come down the line.

    — Dominic Wellington

  • Is Site Reliability Engineering the True ‘Ops’ in DevOps?

    Is Site Reliability Engineering the True ‘Ops’ in DevOps?

    I recently had the privilege of attending the SRECon conference in San Francisco. It’s in its third year and 650 attendees gathered to learn and share their experiences, insights and site reliability engineering (SRE) practices.

    Most SRECon attendees either identified as SREs or were there to learn more about this new role and approach. While there was strong representation from the “unicorn” community (Google, Shopify, Twitter, Netflix and Linkedin), enterprises such as Capital One and Comcast were among the many exhibitors actively seeking to recruit SRE talent. The conference spirit was infused with enthusiasm, a willingness to share, innovative thinking and grassroots advice. It reminded me very much of my first DevOps Days in 2012. There is no doubt that interest in SRE is rapidly growing and worthy of attention by enterprise IT.

    The practice of site reliability engineering was born out of the book by the same name from Google. Interestingly, the SRE movement emerged separately from the DevOps movement—although there is little doubt that they are part of the same IT spectrum with similar customer value-driven goals. DevOps focuses on engineering continuous delivery to the point of  deployment; SRE focuses on engineering continuous operations at the point of customer consumption. Both domains rely on sharing, culture, metrics and automation. Both require human and automated resources to ensure a seamless value stream and exceptional customer experience.

    The role of the SRE is rapidly gaining momentum. Part systems administrator, part second tier support and part developer, SREs are encouraged to be inquisitive, to acquire new skills, to ask questions, to solve problems and to embrace automation. I asked several of the exhibitors what types of skills they were looking for in new SREs. The answers almost unilaterally included the ability to code as well as an unwavering thirst for knowledge and improvement—to find answers, solve problems and ensure the highest levels of reliability and availability.

    I also can see a clear bridge between SRE with IT service management (ITSM). Many of the presenters spoke about the handling and management of late-night incidents, escalations, root cause analysis, service level objectives, availability metrics and other topics addressed in ITSM frameworks such as ITIL. The tools and process definitions may be different, the opportunity to instill agility into many of the ITSM processes to meet changing demands is certainly apparent.

    What are the risks? To my mind the biggest risk is if SRE, DevOps, ITSM, Agile SDLC and other frameworks do not align and remain siloed outposts. SRE and DevOps are two sides of the same coin. Let’s hope that the SRE momentum catches up to DevOps and that enterprises begin to integrate both sets of practices as the norm.

    — Jayne Groll

  • Making Performance More Than a Best Practice

    Making Performance More Than a Best Practice

    Yes, you read that correctly: SRE means that strong application performance—speed, reliability and availability—no longer is simply a best practice. It is a must. Site reliability engineering, or SRE, is a term Google coined several years ago, and while there is no universal definition, SRE can be thought of as a discipline that blends software engineering with IT operations management. At the heart of the SRE model is a site reliability engineer who knows how to code as well as handle ops. The SRE concept is similar to DevOps, which is intended to bridge the divide between development and operations teams, but until now has not caught on the same way DevOps has.

    The Need for SRE

    The rise of the digital economy is now propelling end-user application performance to the top of the IT priority list. According to a recent EMA Survey, 55 percent of IT professionals rank end-user experience monitoring as their top application management priority. At the same time, growing IT complexity, including the cloud and multi-platform applications, are making it more difficult than ever to ensure strong user performance and identify the root source of problems before they escalate. We believe this performance imperative is driving the SRE resurgence, giving DevOps teams the necessary “teeth” to govern and resolve performance-related disconnects, swiftly and decisively.

    Performance issues traditionally have been a lightning rod of conflict for even the most well-oiled DevOps teams. Developers, relentlessly focused on rolling out new products and features, keep throwing their “completed” work over the fence to operations. Operations teams, conversely, are focused on minimizing change and maintaining stability as much as possible. These opposing desires naturally create friction.

    Performance problems only exacerbate this. Developers, who have already moved on to the next must-have feature, say it’s operations’ fault. Operations points their fingers at the developers, saying that poorly written code is to blame. No one is assuming responsibility and the developers and operations teams are stuck in a stalemate. Meanwhile, users are suffering, the call center is being inundated, revenues likely are circling the proverbial drain and the situation is quickly reaching a boiling point.

    SRE is valuable because it finally brings an end to this suffering through a set of clear, irrefutable, mathematically based rules to govern all DevOps efforts and decisions. If a new piece of software achieves a targeted performance level, further rollouts can proceed. If not, future releases will halt until the first piece of software is fixed. The site reliability engineer, who is well-versed in both development and operations, serves as an independent arbiter who provides a clear assessment of where the root cause lies—on the development or operations side of the house. With guidelines firmly established and a central, unbiased person in charge of determining the right course of action, the developer/operations friction can be avoided and problem resolution can commence.

    In an SRE model, developers and IT operations also share a staffing pool. If there’s budget for one hire, and a product is not delivering an agreed-upon performance level, the hire goes to the IT operations. If the product is delivering the required performance level, the hire can go to development, which will help the development team roll out new features and products.

    The real beauty of SRE is that both sides have a vested interest in achieving strong application performance, ideally right out of the gate. If whichever side owns the problem does not fix it, they can forfeit any hopes of adding manpower to their team, at least in the near-term. There’s another more serious implication: for developers, there will be no further development; and for operations teams, less software to support may translate to lower headcount requirements.

    An SRE approach has many advantages, including fostering a vested interest in performance across the entire DevOps team, and holding feet to the fire. Performance cannot be an afterthought—performance must be considered and measured from the very beginning phases of ideation and development all the way through production and post-production modifications. Users don’t care how fast or efficient your DevOps team was if the ultimate work product does not perform—in fact, this means the whole effort likely was in vain. Like a strict adjudicator, SRE may just be what DevOps teams need to bring to an end to performance-related squabbles once and for all.

    About the Author / Mehdi Daoudi

    mehdidaoudiMehdi Daoudi is CEO and co-founder of Catchpoint Systems, provider of web performance testing and monitoring solutions. Before Catchpoint Systems, he spent 10+ years at DoubleClick and Google, where he was responsible for Quality of Services, buying, building, deploying, and using monitoring solutions to keep an eye on an infrastructure that delivered billions of transactions daily. Connect with him on LinkedIn and Twitter.