devopsy.kinsta.cloud

Tag: site reliability engineering

  • Defining Availability, Maintainability and Reliability in SRE

    Defining Availability, Maintainability and Reliability in SRE

    In the world of reliability engineering, you’ll frequently encounter the three “-ability” words: Availability, maintainability and reliability. They sound similar and have similar meanings. In fact, these words may seem so similar that it can be tempting to use them interchangeably.

    That would be a mistake. Availability, maintainability and reliability all have distinct—if related—meanings, and they each play different roles in reliability operations.

    Definitions of Availability, Maintainability and Reliability

    Let’s start by succinctly defining each of the terms.

    Availability, Defined

    Availability is the extent to which an IT resource is ready to perform a task when requested.

    Thus, an application or server that is responding to requests is available (even if it takes longer to respond than desired or otherwise operates suboptimally, in which case it may have a reliability issue but not an availability issue).

    Availability is usually measured as a percentage. A resource that has 99% availability, for example, is one that is up and responding 99% of the time.

    Maintainability, Defined

    Maintainability is a measure of how quickly and easily a resource can be fixed when something goes wrong.

    If a buggy application release can be quickly fixed by rolling back to a stable version, the application would have a high degree of maintainability. On the other hand, if you have a server that needs to be rebuilt manually after it fails, it’s not very maintainable.

    Reliability, Defined

    Reliability is the extent to which a resource functions as required upon request (as opposed to simply being available).

    For example, if your users require an application to process each transaction within one second, and it does this 99%of the time while also maintaining a 99% availability level, that would be a relatively reliable application. In contrast, an application that responds to almost all requests but that suffers from high latency or error rates would not be very reliable, although it might be highly available.

    Differences Between Availability, Maintainability and Reliability

    The simplest way to spell out the differences between availability, maintainability and reliability is to highlight what’s unique about each concept.

    Availability

    Unlike maintainability and reliability, availability is an essentially binary metric, in the sense that a system is either available or it’s not. Although availability status can change over time, there is no such thing as varying degrees of availability.

    Availability also stands out because it is often the most important metric in defining SLAs and SLOs. Contracts typically specify that resources will achieve a certain level of availability, defined in percentage terms. They may sometimes also specify metrics like response times, which are a reflection of reliability, but availability is more likely to be the most important metric within service agreements.

    Maintainability

    Maintainability is unique in that it’s a pretty subjective concept. A system that one SRE considers easy to maintain could seem difficult to maintain to another SRE. The methodologies that engineers leverage to optimize maintainability could vary, too; for instance, someone from a DevOps background may be more likely to focus on optimizing maintainability within the software delivery chain than someone trained in classic site reliability engineering, which focuses on automating IT operations through code more than on the software delivery process.

    That said, automation tools help virtually every team maximize maintainability, no matter their preferences or background.

    Reliability

    Reliability stands out because it reflects how well a system performs. Here again, this is a somewhat subjective metric, because performance requirements may vary from one user to another. Nonetheless, reliability is the most effective means of measuring whether a system meets the performance levels it needs to, even if those requirements change over time.

    Why do availability, maintainability and reliability matter?

    Because availability, maintainability and reliability each measure different aspects of a system’s status, putting them together is a useful means of gaining insight into the overall reliability of a system.

    In order to be reliable, a system requires both availability and maintainability. A system can’t be reliable if it’s not available. It’s also unlikely to be highly reliable if it takes a long time to fix issues due to low maintainability.

    At the same time, by focusing on availability, maintainability and reliability individually, you can drill down into specific issues within the IT resources you manage. For instance, you might have systems that are high-performing when they’re available, but that have low reliability rates because of availability issues. In that case, you’ll know that an investment in increased availability is likely to yield the greatest reward for increasing overall reliability.

    The Bottom Line

    The bottom line: While availability, maintainability and reliability are all reflections of the quality of an IT resource, they measure quality in different ways. Tracking each category separately is important for ensuring that you know where your weakest links exist within overall system performance and health.

    At the same time, however, it’s important to compare and correlate availability, maintainability and reliability data so that you can achieve continuous insight into a resource’s status. When you can track each item separately but also combine them together to gain a complete picture of your system, you are in the best position to optimize reliability operations.

  • Where Do SREs Go From Here?

    Where Do SREs Go From Here?

    Charlene O’Hanlon talks with Leo Vasiliou, director of product marketing at Catchpoint, about the results of a study the company fielded with VMWare Tanzu and DevOps Institute of nearly 300 site reliability engineers (SREs). This year’s report underscores the challenges of multi-cloud, calls out the underutilization of AIOps and shows a systemic shift in core baselining data. The video is below, followed by a transcript of the conversation.

    Announcer: This is Digital Anarchist.

    Charlene O’Hanlon: Hey, everybody. Welcome back to TechStrong TV. I’m Charlene O’Hanlon, and I’m here now with Leo Vasiliou who is the Director of Product Marketing over at Catchpoint. Leo, thanks so much for joining me today. Really appreciate it.

    Leo Vasiliou: Oh, of course, Charlene. Thank you very much for allowing us to have the conversation today.

    O’Hanlon: Absolutely, absolutely. I want to talk to you about a recent survey study that you guys put out in conjunction with VMware Tanzu and DevOps Institute. What were you guys looking at?

    Vasiliou: Well, from an existential perspective, we hope that the survey that we do leads into the annual report and we’d like for it to be the most data-backed set of insights in its community. There are sort of three underlying constructional essences of the report. One is to provide a core baselining data for the key SRE tenants. So DevOps, who had an amount of time, would call in reported levels of toil. The other essence is to look at what’s trending. And then the third one will generally be around the IT to business relationship. And then with each one of those constructional essences, the context and line items within each section will change from year to year.

    O’Hanlon: Gotcha. Alright. So this one focuses on site reliability engineering and what they’re seeing and what they’re dealing with on kind of a daily basis. Is that right?

    Vasiliou: Yes, yes.

    O’Hanlon: Okay. Alright, great. So tell me a little bit about some of the findings in the study. I know you guys do these, as you just said, reports yearly. So I’m interested in knowing what’s changed from year to year and what really surprised you in this year’s report.

    Vasiliou: Well, I like to say SREs are double struck with challenges because the SRE is essentially two jobs in one, right? They have to increase the efficiency of their operational activities while mitigating the risk of their transformational activities. So one of the key findings this year, which by the way this may be the most provocative dataset we’ve published just because some of the findings conflict with some popular theories in the market, but take the first one just as an example of that, is as part of the baselining set of data there was a self-reported drop in toil, which is the word they use to define repeated, mundane, busywork that doesn’t have any inherent value, of 15 percent. And one of the reasons this is such an insightful nugget to lead us into our various conspiracy theories is it was across the entire distribution of data, across all of their percentiles, as well as across all of the geographies that we looked at.

    O’Hanlon: Wow.

    Vasiliou: And so for me, when I had the blessed opportunity to do the research and sit down and co-write the report, having been stuck in the house for, let’s just call it a year and a half at this point, there was just something different about working on a piece like this and it just felt a bit more meaningful. And the conspiracy theory there or the hypothesis is that if people return to work, or return to the office I should say or some type of hybrid environment, will those reported levels of toil increase again next year as people adjust to the hustle and bustle of commutes just adding to frustration in how they think about things?

    O’Hanlon: That is a really, really fascinating thought process there. We could do an entire conversation about just that and whether what happens this year is going to be indicative of what happens in the future or if this year is going to be like a reset year. So from an SRE perspective though, the fact that the level of toil decreased 15 percent, that seems pretty significant. If you look at the entire spectrum of what they have to deal with on a daily basis, 15 percent, good lord, that’s huge in the SRE space.

    Vasiliou: Yeah. It is huge. I agree. It’s also based on the response of hundreds of survey participants year over year. And what’s interesting, Charlene, is one of the unique tactics we took in this year’s report is before we publish the reports and post survey pre report, we actually showed the results to various industry, let’s just call them, leaders. One of them we shared this particular section to had an alternative theory on why levels of toil dropped, and it had to do essentially with the fact that if people are not in the office, maybe does the very act of them not being observed compel them to do less busywork.

    So there was an alternative theory. But bringing us back to and leading into one of the other essences of the report, which is kind of the trends piece, it had to do with the amount of monitoring data sources, like the volume, variety, velocity of data, and the number of multi same-service platforms, e.g. multi CDN, multi DNS, et cetera, in use. And so will the return to work and the promise of AI ops help them keep toil levels low? Which leads us into the next sort of surprising finding, which is that SREs and their implementation or shift to AI ops is quite slow to maybe what we might be reading about in the industry, which was probably the second most provocative finding in the report.

    O’Hanlon: Yeah. So why do you think it is so slow then? Were there any trends that kind of came out of that? Any reasons why organizations are kind of slow to adopt AI ops?

    Vasiliou: Well, there is obviously a deluge of information and rhetoric in the market on the internet today. We need to seriously consider this idea of an information diet, which is a concept I first heard about a couple years ago. When we think about the idea of the hype cycle versus the challenges SREs face on a day-to-day basis, sometimes the market and the promise of what these types of solutions will offer might outpace actual implementation capabilities from, hey, we’re an SRE, let’s go implement this. I think part of the challenge here is probably the concept of AI ops is too broad and too large for consumption. It’s the idea of just saying, “Hey, let’s go solve world hunger or, hey, let’s go solve or implement AI ops,” right?

    O’Hanlon: Right, right.

    Vasiliou: You’ve got to break it down into smaller components and incrementally develop capabilities from there. Otherwise, how the heck are you going to baseline your progress for when you’re done? I think it’s too big, too broad, and some of the vendor promises may be of self-healing, auto remediation, maybe they were a little more on the hype side, where some of the other AI ops of components like root cause analysis or even correlation, maybe those are truthfully more on the hope and promise side.

    O’Hanlon: Okay. Do you think that AI ops – to your point, it is broad – do you think it just means different things to different people and so somebody actually may be implementing some type of AI ops and actually not realizing it?

    Vasiliou: The short answer is yes. So when we asked the question what monitoring tools or practices are used today, they were presented with seven options, ranging from application, infrastructure, experience monitoring, AI ops, and it was the fifth lowest, meaning that almost 40 percent of respondents said they’d never used it in the first place. The second question we asked is very simply what is the received value from AI ops, where one was the low value, nine was the high, and the range evenly spanned the value scale that was presented in the survey. What was interesting is there were a subset of people who said they never used AI ops but that they still received value, which was an additional dimension we didn’t think about when writing the survey.

    But it kind of brings me back to the essence of your question, which is are you user versus a consumer? And then go back to the previous comments about how big and broad this quote/unquote AI ops idea is, and there’s absolutely a propensity that people hear and think completely different things, unfortunately.

    O’Hanlon: Interesting. Interesting. So one of the things that we’ve been talking about in the SRE space over the last couple years is really how the role is kind of morphing and changing along with the market in general and to address a lot of organizational needs as well. Are there any findings from this year’s study that kind of bear out how an SRE role is changing and even if it is changing?

    Vasiliou: That will go back to what we were saying just a moment ago about mitigating the risk of their transformational activities. When you were asking the question, I’ll just cite the stat from our very first SRE report. We said that 85 percent of respondents said that in order to be successful in this role you have to commute into an office. So that got topsy-turvy turned upside down on its head.

    O’Hanlon: Right. Yeah. [Laughs]

    Vasiliou: But to get back to the essence of the question here, when we looked at the quantity of data sources and the prolific rise of the use of multiplatform providers, we sort of looked at this correlation between how SRE orgs are structured. Are you decentralized, are you centralized, or is there some type of hybrid in between? And then the decentralized piece was broken down even further by stack component, by platform component, or by business product or service. The vast majority of SREs said they are currently decentralized by business product or service, but I think the transformational piece here, which is something we’ll look at next year when we write the report, is because of the prolific rise of multiple third party and multiple same-service third party providers, there is a huge, gigantic spotlight to shift the primary dimension of org structure from product and service to a platform basis.

    And we talk about it like this. If you’ve got 10 different teams assigned to 10 different products or services and one member from each of those SRE teams developing capabilities that happen to be the exact same capabilities in development by those other nine teams, that’s the scalability and efficiency _____ that we start to lead in and talk about in this year’s report. So I think the transformational piece will be the assignment of SRE development engineering resources from product and service as a primary dimension to a platform – for example, cloud, CDN, DNS, API – as the primary dimension where those capabilities will be normalized and developed. Even though underlying platform structures will be different, those capabilities will then be offered as part of, let’s call it, an internal marketplace for use by multiple teams to kind of get away from that concept of silos and being segmented by product or service. So a bit of a long answer but all contextual to the key part there, which is the trend and the transformational piece that we wrote about this year.

     O’Hanlon: Right. Yeah, well it really is food for thought there if you consider how the SRE role was established and how it’s kind of – if you go down that path, how it seems to be changing and evolving as the more technology is adopted and more things are moved out to the edge and the need for SREs at basically all points on the network if you think about it. So great stuff there. Well, Leo, I know we could talk about all of the findings and have a much larger conversation, but unfortunately that’s all the time that we have for this one. But I do want to thank you very much for going through some of the survey findings. If folks are interested in taking a look at the study itself, is it readily available?

    Vasiliou: It is, it is. So it’s available at Catchpoint.com. Just head over to our learning and research section and enjoy the read. Charlene, one of the sentiments we wanted to leave readers with this year was this concept of baselining against Google’s definition of SRE, but we hereby declare by the power vested in us by absolutely nobody at all that you are hereby free to be an SRE.

    O’Hanlon: Oh, wow. Just one more thing to add to my plate. [Laughs] Well Leo, thanks again. I really do appreciate your time, and it’s a pleasure talking to you. I’m sure that we’ll talk again in the future about future studies that you guys have going on. So, thanks again. Really appreciate it.

    Vasiliou: You are quite welcome. Thank you for your time, Charlene.

    O’Hanlon: Alright, everybody. Please stick around. We’ve got lost more TechStrong TV coming up, so stay tuned.

    [End of Audio]

  • 4 Ways to Overcome Agile Transformation Obstacles

    4 Ways to Overcome Agile Transformation Obstacles

    If an organization desires a successful Agile transformation, the first step is to shift its thinking from ‘doing Agile’ to ‘being agile.’ Unfortunately, a commonly shared belief that Agile is just a software development methodology keeps too many companies stuck in the ‘doing Agile’ mindset.

    This belief ties back to the principles of Agile thinking from frameworks like Scrum or Lean. Most of these methodologies were focused on teams of developers. Naturally, the vocabulary and tools derived from these methods are unique to software development. While development has evolved away from waterfall approaches, the rest of the organization tends to run into cultural challenges downstream because the vocabulary and ways of working are foreign.

    When an organization begins to lose progress downstream, it is too focused ‘doing Agile’ rather than ‘being agile.’ In order to shift the mindset toward ‘being agile,’ a new mindset must be adopted across the organization. These four insights offer ways to overcome obstacles along the way:

    Instill Agile Thinking Across the Organization

    The first step toward instilling agile thinking is to establish common vocabulary and define what it means to be agile. A common vocabulary and understanding of agility helps teams across the organization understand and see that agility is more than doing sprints.

    As an organizational strategy, agility equates to flexibility – the ability to adapt to changing business requirements. It also means being collaborative and making complex processes more lightweight. Instilling this type of agile thinking into everyone at the organization leads to success. DevOps, for example, successfully instills this type of thinking through value stream management.

    Essentially, this hurdle boils down to setting common goals and creating a common understanding of what ‘agile’ means across the organization.

    Achieve Autonomy Through Self-Organization

    Agile frameworks encourage self-organization. When teams self-organize they are given a task to do. However, no one tells them how they’re going to accomplish it. A self-organizing team gains a certain amount of autonomy, which leads to empowerment.

    With self-organization also comes risk. The organization must learn how to embrace risk, how to look at failure as an opportunity to improve and how to empower teams in a more self-regulating way. When teams self-regulate and self-organize, it’s a team win when they are successful. This develops a sense of communal pride.

    When breaches or too many failures do happen in a self-regulating system, the policies and consequences naturally occur. However, the key differentiator is that these environments are not driven by the policies. Rather, the objectives drive the behavior. When objectives are being met, empowerment stands.

    Agile software development, DevOps and site reliability engineering (SRE) are intended to be self-regulating systems, but all first require leadership to create psychologically safe environments, blameless postmortems and more. There is no room for a blame culture in a self-regulating system, and the organization must be agile enough to meet the demands of the system.

    Leverage Intelligent Lightweight Process

    Agility does not equate to a complete absence of process. It means there is enough process in place to ensure consistency, improve flow between people and teams and that automation replicates the process simply and effectively.

    IT organizations must understand that it is possible to move away from complex and authority-based processes and continue to leverage an element of process.

    Intelligent Process Means Intelligent Automation

    Although humans are the driving force behind agility, automation helps them go faster with more consistency. However, there is no such thing as ‘being agile’ with automation alone. Agile cannot be bought, and it’s important to get teams thinking that way. Automation should complement Agile, but not define it.

    When investing in automation, a cost/benefit analysis is critical. When considering automation, think in terms of two key metrics: time and quality. With two equally important metrics, the organization cannot become dependent on one aspect or another.

    For example, you can’t go faster with a manual process. You also can’t go faster with only automation, because the automation alone doesn’t replicate your intelligent lightweight process. And, you can’t go faster just by having a process, because the process can become very bureaucratic.

    Process is just a means of getting there, but it needs people and automation to make the processes come alive. People, process and automation all play an essential role in creating an intelligent process.

    Successful Agile transformation is more a way of thinking and behaving than simply following a framework. From a human perspective, an organization that experiences true agility, as we’ve defined it today, allows humans to exercise their competencies, grow skills, collaboratively celebrate success and learn from failures.

    If you’d like to learn more about Agile Transformation, especially in regards to Agile Testing, you can register for the next SKILup Day on April 22. Please register for SKILup Day: Agile Test Management for the Enterprise here.

  • SREs: Stop Asking Your Product Managers for SLOs

    SREs: Stop Asking Your Product Managers for SLOs

    One of the fundamental premises of software reliability engineering is that you should base your reliability goals—i.e., your service level objectives (SLOs)—on the level of service that keeps your customers happy. The problem is, defining what makes your customers happy requires communication between software reliability engineers (SREs) and product managers (PMs) (aka business stakeholders), and that can be a challenge. Let’s just say that SREs and PMs have different goals and speak slightly different languages.

    It’s not that PMs fail to appreciate the value that SREs bring to the table. Today, in the era of software as a service, features such as security, reliability and data privacy (which used to be considered “non-functional requirements”) are respected as critical features of the service-product a SaaS company delivers. Modern application users and customers of software services care a lot about data privacy, cybersecurity and uptime; therefore, PMs care, too. In fact, it’s not uncommon to see these features touted prominently on a company’s website (I saw one today claiming “99.99% uptime for every customer!”) because the folks in marketing know that customers are making purchasing decisions based on whether the company can deliver reliability, speed, security and performance quality. So, yes, PMs do care.

    The difficulty lies in the negotiation between the PM and the SRE about the value of providing that level of service and the cost associated with it:

    The people and infrastructure costs associated with providing the next level of reliability increase geometrically for a host of reasons, but the expected value to a $10 million business of losing out on that one in 1,000 transactions versus one in 10,000 is only $1,000.

    But translating “customer happiness” into actual objectives derived from observable metrics is easier said than done, particularly for companies that are just getting started with SLOs. Let’s face it, all the talk about “nines” can be difficult to conceptualize. Fortunately, there’s an easy way for PMs and SREs to get on the same wavelength.

    Engineers, stop asking your PMs for SLOs!  Instead, convert user stories into reliability goals.

    User stories are a great way for PMs to express a customer’s experience and expectations to the engineering team. SREs can then translate the user stories into SLOs in a few simple steps.

    What is a User Story?

    User stories are narratives that describe how a particular type of user engages with the service. The first-person plotline might go like this: I am this type of person, and this is what I need and expect to be happy, and if I’m not happy, this is what it’s going to cost you.

    Note: In this user story, we’ve taken the liberty of including empathy for the reliability expectations of the user.

    Retail example: I’m a shopper on your online store on Black Friday, and I want to take advantage of your Black Friday sale. I want to log on, peruse your Black Friday deals, pick the products and features I prefer, and make my purchase in just a moment or two. If it takes too long to log on, or if I get hung up in the system, I’ll easily get frustrated and move on to other retailers. I don’t want to miss the deals!

    An SRE can take that user story, ask the PM a few questions to gather just a few more critical bits of information, then do some simple math to derive the SLO for your Black Friday.

    Ask Your PM These Questions

    To determine the reliability goals for a user story, you need to add three pieces of context:

    Step 1 – Identify the Top Critical User Transactions

    First, determine the relative priority or criticality of the transaction being successful. Not all transactions are created equally, so we need to understand what kind of transaction we’re talking about here. For example, browsing a catalog page (which could be refreshed if unsuccessful) is different from the shopper’s checkout experience.

    Retail example: We have several transactions in a retail business, including:

    1. Inventory catalog search.
    2. Order fulfillment.
    3. Payments processing.
    4. Price and discount.
    5. Checkout experience/shopping cart abandonment.

    For our example, let’s develop the SLO for payments processing, which is a throughput metric—the total number of transactions a retailer can handle in a given period of time.

    Step 2 – Estimate the Volume of Transactions at Various Times

    Now that we know the transaction we’re talking about (payment processing throughput), we can estimate the rough volume of transactions or user interactions that will occur in each of those categories, rounded to the closest order of magnitude (see Fermi Estimation). How much of your business happens during this time? Do you have a critical time of day? Do you have a critical season? Do you have a critical day(s) of the year?

    The same transaction at a different time may be more or less valuable. Think about an ad playing during the Superbowl or stock prices while the market is open. Or when your CEO is giving a critical customer presentation. Timing matters.

    Retail example: Our hypothetical retailer has critical periods in the holiday shopping season and a critical day called Black Friday. Typically, 4% of annual business happens during Black Friday weekend, and 16% of annual business happens during the remaining holiday season.

    Adobe Analytics estimates that sales for the full weekend (Thanksgiving through Cyber Monday) is 20% of total revenue for the full holiday season. According to the National Retail Federation, 2019 holiday sales were $730.2 billion, accounting for 20% of annual sales.

    Step 3 – Estimate the Business Value of Transactions in Each Circumstance

    Finally, we can now understand how the reliability of this transaction affects business outcomes. This could be the risk of lost revenue or perhaps a brand reputation metric such as reduced CSAT. Now you have a pretty clear picture of the business impact of unreliability.

    Retail example: If our imaginary retailer has revenue of $10 million, it would make $400,000 of that during Black Friday weekend, and $1.6 million during the rest of the holiday season. On Black Friday shopping carts are worth $100, $80 during the holiday season and $50 the rest of the time. There are 800 transactions per day on Black Friday, 625 per holiday season day and 488 any other day.

    Now that you have this input, you can start to calculate appropriate SLOs based on the value at risk and the cost to meet that reliability goal.

    If possible, you can estimate the cost to deliver each level of reliability. Consider not only the cloud computing and direct operational costs, but also the reliability investments in testing, performance, security, chaos engineering, release management, rollback, monitoring, on-call, automation, etc., that would be required to achieve the reliability level at each point. By plotting this cost (at least to the order of magnitude) you can find a break-even point where the value at risk and cost of reliability crossover. This is the business-justifiable SLO for these transactions.

    Plot the value-at-risk as a function of value x volume x SLO. The value-at-risk will decline sharply as the reliability is increased; however, the cost to achieve this reliability also increases exponentially. It’s simply not worth investing more than that at-risk amount for any given user story, so the SLO determination is clear. Note that the SLO may not be achievable for the team currently, but you have a precise idea of what matters to the business and the break-even point of your reliability investments.

    Instead of asking PMs to “speak SRE,” span the communication gap by using the common language of user stories to build business-cogent SLOs.

  • Why It’s Time for Site Reliability Engineering to Shift Left

    By adopting a multilevel approach to site reliability engineering and arming your team with the right tools, you can unleash benefits that impact the entire service-delivery continuum

    In today’s application-driven economy, the infrastructure supporting business-critical applications has never been more important. In response, many companies are recruiting site reliability engineering (SRE) specialists to help them improve reliability, availability, latency and a host of other metrics that impact customer satisfaction. These vital new people resources are focused on keeping sites and services running in peak form.

    There are clear limitations to this operations focus, though. A service is made up of many application components that can impact business-critical services. How do you ensure reliability is baked into each app and each code change if your engineers focus only on backend performance?

    Some trailblazers are beginning to tackle this issue head-on, tasking their SREs with taking on new roles that support key stages of the DevOps life cycle. The goal is to take early, proactive steps to ensure quality and reliability are built in from the beginning.

    A Path Pioneered by Test Teams

    To understand why this “shift left” is so important, consider recent revolutionary changes in the software testing function. Not that long ago, testing was considered a distinct phase in the software life cycle, timed to begin well after the work of application developers was done. Testing Centers of Excellence uncovered defects and passed code back to the development team for rework.

    As developers moved to agile sprints, though, this approach to testing quickly became a bottleneck. By the time Centers of Excellence uncovered issues, developers had moved on to other projects. In response, new continuous testing programs emerged, staffed by a new breed of software development engineers who had honed their testing skills.

    These individuals are now embedded with development teams to ensure applications are built better from the start. Open source test automation tools help them test early and often, without impeding agile delivery schedules.

    It’s important to note that Testing Centers of Excellence haven’t gone away. Instead, they have taken on a broader leadership role and are now responsible for coordinating testing across the enterprise in support of continuous software integration and delivery.

    A 3-Level Model for SRE

    There is much that SRE teams can learn from the experiences of their testing colleagues. To produce optimal results, it is important to integrate reliability experts into your organization’s overarching agile framework. Consider adopting the following three-level model that shifts the function to the left and integrates skilled engineers at key points in the DevOps life cycle:

    1. Application level. Embed SREs with development and testing teams to ensure application components and new releases won’t disrupt reliability. These individuals focus on application-level service objectives, error budgets and integration into the DevOps pipeline.
    2. System level. Assign SREs to focus on the applications in your organization’s release train. These individuals support release and launch coordination, evaluate system architecture readiness and ensure your organization meets systemwide service-level objectives.
    3. Enterprise level. Establish a Site Reliability Engineering Center of Excellence, staffed by engineers who oversee the governance of your enterprise architecture. These individuals establish best practices and select the tools and resources required to support your site reliability function companywide.

    An Expanded Toolset

    To support this broadened role, you will need tools that can support your engineers across the entire DevOps life cycle. You will need to be able to see and track issues engineers uncover and to evaluate system-level readiness and performance against your service-level objectives. You will also need a big-picture view that spans activity across the entire enterprise.

    Fortunately, a new generation of solutions is now available to help, powered by artificial intelligence, machine learning and intelligent automation. These resources can help your team process massive amounts of data from disparate toolsets and turn that data into actional insights. As a result, you’ll have all the lifecycle analytics you need to be effective in your new and broadened role.

    SRE and Shift Left: Broad New Benefits

    By adopting a multilevel approach to sire reliability engineering and arming your team with the right tools, you can unleash benefits that impact the entire service-delivery continuum.

    You can build a continuous cycle of feedback, collaboration and governance that spans design, development and the launch and operation of new services. You can better manage configuration changes, service levels and error budgets. You can get a clearer view of application reliability and production readiness to guide planning initiatives and to help you prioritize investments.  And you can build better-informed business cases to move your organization forward. Shifting left is clearly a win-win move.

  • SREs Tasked With More To Improve Digital Experiences

    SREs Tasked With More To Improve Digital Experiences

    The digitization of the enterprise has never moved so swiftly, and never have enterprise environments been as complex to manage. And, never have more businesses been as dependent on their infrastructure and operations being unfaltering.

    According to a new survey and accompanying report from Catchpoint and the DevOps Institute, site reliability engineers (SREs) and teams still struggle with effective SRE processes that would help these teams improve their system reliability, incident management and end user/customer satisfaction.

    For instance, the survey found that 53% of respondents reported being brought in too late into the application life cycle. Most respondents said that they are currently spending 75% of their time focused on operations and very little of their time on development objectives.

    That heavy focus on ops comes at a cost by increasing the ongoing costs associated with owning and operating their systems and infrastructure. Moreover, in recent months, perhaps because of increased remote work, Ops-related activities have increased by 10% overall among those surveyed.

    Since the start of the COVID-19 pandemic, the number of SREs who believe they will be working remotely in the next year jumped to 50% from only 19% who thought so before the outbreak of the novel coronavirus.

    When it comes to performing their duties, more than half of the respondents cited the change to remote working causing challenges when it comes to attaining a balance between their work and their life.

    Additionally, while 9% of respondents felt incident management has improved, 41% said that half or more of their work is focused on managing mostly manual, repetitive and tactical tasks that could likely be automated. Another 52% said they spend too much time debugging.

    The report provided a few suggestions for teams:

    • Be sure to include consideration for not only your code but also the networks, third-party services and delivery chain components to evaluate how well the three observability pillars are applied through this new digital experience observability lens.
    • Work to be included earlier in the development process should shift reliability further left to reduce cost, increase team alignment and identify constraints that can be removed.
    • Turn newly surfaced or previously ignored challenges into strategic differentiators. Focusing on challenges such as morale, employee experience, work/life balance and employee engagement and sentiment may showcase a company’s employee-first mentality to attract or retain top talent.

    As we covered in “Digital Transformation Efforts Hindered by Lack of Upskilling,” there are no digital transformations without staff transformations and the same is true for practices such as DevOps and SRE. And it’s that last point that may be one of the biggest inhibitors to SRE teams getting from where they are today in maturity to where they want or need to be. An earlier survey from the DevOps Institute, the “2020 Upskilling: Enterprise DevOps Skills Report” found that 58% of enterprises have a tough time actually finding those with the right DevOps skills, and 48% said it’s difficult to retain skilled DevOps professionals.

  • Why is Site Reliability Engineering Important?

    Why is Site Reliability Engineering Important?

    Site reliability engineering (SRE) is one of the fastest-growing enterprise roles and set of operational practices for managing services at scale.

    During the Site Reliability Engineering SKILup Day June 18, I asked a group of the DevOps Institute Ambassadors why SRE was important to them.

    Below you will find some insightful responses, tips and calls to action. If you missed SRE SKILup Day or wish to rewatch any of the sessions on-demand, visit the conference landing page and access the videos for free: https://devopsinstitute.com/site-reliability-engineering-skilup-day/.

    Happy Reading!

    Anshul Lalit
    DevOps Institute Ambassador from Kristiansand, Norway

    “The term SRE surely has been introduced by Google, but directly or indirectly several companies have been doing stuff related to SRE for a long time, though I must say that Google gave it a new direction after coining the term ‘SRE.’ I have a clear view on SRE as I believe it walks hand-in-hand with DevOps. All your infrastructure, operations, monitoring, performance, scalability and reliability factors are accounted for in a nice, lean and automated system (preferably); however this is not enough. Culture is an important aspect driving the SRE aspects, along with business needs. As the norm ‘to each, his own’ goes, SRE is no different. It is easy to get inspired from pioneer companies, but it’s impossible to copy their culture and means to replicate the success, especially with your ‘anti-patterns’ and ‘traditional’ remedial baggage. Do you have similar infrastructure and business needs as the company showcasing brilliant success with SRE? No. Can it help you? Absolutely. The key factor here is to recognize what is important to your success blueprint after understanding the fundamentals of it and find your own success factors considering your cultural needs. Your strategy and culture need to walk together, just like your guiding (strategy) and driving (culture) factors.”

     

    Maciej Jarosz
    DevOps Institute Ambassador from Warsaw, Poland

    “With all of those methodologies and frameworks out there, one can only ponder: ‘Which one should I choose?’ Well, all of those frameworks and methodologies provide something of value, as long as you have an opportunity to actually test some of those concepts in your own local environment. Consider SRE for example: It worked for Google, sure. If anyone doubts it there are at least three books on the topic with the materials, proof and philosophy behind SRE. Even if your company is not the scale of Google you still should take a look at best practices that worked in 2017, 2018, 2019 and that are still working now. Considering the crazy times that we live in right now and the plethora of various free materials, conferences, blog posts, research whitepapers and videos on the topic, what are we waiting for? The knowledge is right here, ripe for the taking. Advancing the humanity of DevOps is just a matter of experimentation, trust and continuous improvement. And with events such as the SKILup Day: Site Reliability Engineering where you can network, exchange field stories, brag about your ‘battle scars’ and participate in some splendid speeches, it’s only easier to jump into the fascinating world of SRE discipline.

     

    Helen Beal
    DevOps Institute Ambassador from Chichester, England

    “I might be a bit biased since I’ve been focused on DevOps for nearly a decade as a consultant, coach and learning facilitator, but for me there are a number of systems of thought that contribute to the DevOps body of knowledge. I stole (with permission!) a phrase I heard from DevOps Institute’s CEO, Jayne Groll, many years ago that DevOps is the ‘harmonious, polygamous marriage of ITSM, lean and agile’ and I’ve also presented at several events over years around the correlations between DevOps and Holacracy. At an event in Munich a few years back, John Willis and I started a conversation about a DevOps Superpattern as he also added his ‘three-legged stool’ of The Theory of Constraints, Safety Culture and Learning Organizations’. For me, SRE is another valuable system of thought that we can learn from and use to make practical changes to the way teams and organizations work to drive higher organizational performance.”

     

    Mark Peters
    DevOps Institute Ambassador from Texas, USA

    “I recently saw a social media image for a shirt which broke out SRE as ‘Simply Restart Everything.’ While not entirely true, the concept of a site reliability engineer is important as another way to link Ops and Dev in the DevOps construct. SREs formalize a position whose job is to connect, to link folks together, and to streamline processes. Increasing reliability means to increase simplicity across multiple workflows. The bigger your organization, the more simple the process needs to be. Unfortunately, many organizations increase complexity, reduce transparency, and create wasted effort as they add more individuals. SREs are important in trying to understand processes from end to end and emphasizing outcomes rather than any particular process stage.”

     

    Sean Davis
    DevOps Institute Ambassador from Georgia, USA

    “Site reliability engineering bridges the gaps between platform design, development and operational execution by providing new perspectives on system reliability. When you link software engineering and systems engineering mindsets with operational engineering, you streamline the support of a product-focused on business outcomes not ticket output. It can be a valuable gambit for the traditional approach that pays off big for companies. By trading off specializations in focusing on operational burden, you gain a larger view of the platform and reduce the efforts required to support the product and spend more time automating and innovating to move towards self-healing and autoscaling products instead of manual, reactionary troubleshooting.”

     

    MIke D. Kail
    DevOps Institute Ambassador from California, USA

    “Instead of the word reliability, I prefer to use and approach it as resiliency, which then includes approaches and contributing members of the chaos engineering group that are hyper-focused on continually increasing the ability to recover from any and all failures, which we all know happen in both software and the underlying hardware. Resiliency also should resonate with the core Development teams as they should strive to continuously improve and collaborate with the SRE teams.”

     

    James Wasson
    DevOps Institute Ambassador from Oklahoma, USA

    “We create pipelines, version control, and infrastructure as code like breathing now. That means we are done, right? The beauty of SRE is that it takes us right to where the value is made. It requires us to go back to the beginning and ask questions about decisions we need to make regarding a system. Site reliability engineers help us remove “resulting” from our processes. With the help of these principles, we begin to make near-real-time decisions based on the impacts to our customers, rather than the results of the past. We spend so much time creating amazing features, determining the velocity of our teams, and implementing the newest framework from that great book, but one of the most helpful questions we get from site reliability engineering is ‘How fast is too fast?’ If our organizations can answer this question early, we are able to make more informed decisions that create extended and meaningful value to our customers.“

     

    Biswajit Mohapatra
    DevOps Institute Ambassador from Pune, India

    “The ever-changing digital and cloud landscape brings in an unprecedented need for collaboration, transparency, resiliency, stability, performance, reliability and correctness. SRE sets measurable engineering goals mapping to service level objectives (SLOs) and enables monitoring and tracking of these quality of service (QoS) parameters. SRE is a set of practices focused on reducing silos by shared ownership, planning for failures using error budgets, small-batch changes with focus on stability, automation of manual tasks and introducing a culture of measurement, monitoring and tracking. The fundamental goal of SRE is aimed at depicting a prescriptive approach to plan, build, implement, measure and achieve DevOps objectives with a focus on reliability and automation at every opportunity. SRE covers several critical areas of cloud platform architecture such as orchestrated automation, responsive operation, optimized performance, just-in-time scaling, modernized environment and predictive event management. It’s time to look at operations through the lens of software engineering practices. Leveraging SRE to design, build, operate and enhance software systems is critical to building the culture of risk-managed and reliable digital footprint for enterprises.”

     

    Mitesh Soni
    DevOps Institute Ambassador from Mumbai, India

    “What vs How
    Interface vs Class
    Abstract Class vs Class
    DevOps vs SRE

    Similarity and difference lie in their positions. Ryan Foland has given a wonderful example to describe What and How: ‘What’ begins at the surface level, and usually describes things in a general sense while ‘How’ describes the process of something or the underlying steps and actions that are done to get results. I find this example a perfect fit for the relationship between DevOps and site reliability engineering (SRE). Both DevOps and SRE have created buzz and confusion and will continue to do so. DevOps is cultural change to transform existing practices with people, processes, and tools for a better quality outcome with speed. SRE is a concrete implementation of DevOps practices or principles. I would like to leave one question here. Let me form the base first: Agile and DevOps complement each other. Agile is like a train compartment and DevOps is like an engine. Is SRE like a train track?”

     

    Marc Hornbeek
    DevOps Institute Ambassador from Puerto Vallarta, Mexico

    “Site reliability engineering (SRE) compliments DevOps by measuring and achieving reliability of applications and services working on production and DevOps infrastructure in a prescribed manner using error budgets, team relationships brokered by an error budget, Ops-as-code, and the use of reliability to control deployments.  SRE emphasizes the following objectives that are important to DevOps engineering:

    • Assure availability of applications and services.
    • Assure error rate and latency meet Service Level Agreements (SLAs).
    • Enable large-scale systems while controlling risk.
    • Create operational cost savings by automation and policies.
    • Improve skills of developers and SREs.
    • Resolve conflicts between Dev and SRE.
    • Improve capacity planning and provisioning.”

    Suma Puligella
    DevOps Institute Ambassador from Virginia, USA

    “SRE encompasses both software engineering and IT operations and bridges the gap between them. It ensures that the organization’s systems are scalable, stable, predictable and automated. SRE and DevOps seem like two sides of the same coin with a unified goal of enhancing the release cycle without any compromises. However, there are some differences between SRE and DevOps:

    • SRE covers architectural and design patterns along with reliability implementation techniques such as reactive programming. Whereas DevOps is a set of practices that combines software development (Dev) and IT Operations (Ops). DevOps also aims at continuous software delivery with high quality, reliability and shared ownership among software stakeholders.
    • DevOps raises the risks for the development team(s) to solve. Whereas SRE’s approach is to find problems and solve some of them themselves. SRE teams set service-level agreements (SLA) that define how reliable the systems need to be to end users. Their main goal is to improve performance and operational efficiency.
    • DevOps is all about the “What” needs to be done, SRE talks about “How” this can be done.

    The bottom line is that it’s all about the data. You need information in order to understand how to measure success and failure and how to gain continuous reliability across the applications.”

     

    Tiffany Jachja
    DevOps Institute Ambassador from Maryland, USA

    “Organizations are finding the best practices for delivering faster and meeting customer demands. Site reliability engineering is about building confidence that our systems work as expected. SRE defines the methods, concerns, and responsibilities needed to maintain or increase the reliability of a system. It improves our ability not only to deliver a product to end customers but also to do so in a sustainable way as an organization. Many workloads can be costly, complicated, or time-consuming for different reasons. These practices provide a foundation for the root analysis of our systems, at the end of the day we want to be able to set key goals, measure systems, and ensure a product or service fulfills its objectives. This is important for anyone with a stake in software development and delivery.”

     

    Uchit Vyas
    DevOps Institute Ambassador from Singapore

    “Site reliability engineering is a discipline that seeks to create reliable IT systems by managing the rate and quality of change, ensuring clients protect themselves from any break in service and avoid being subject to fines and penalties.

    As per my experience, SRE practices are mainly based around the following elements:

    • SLIs, SLOs and SLAs.
    • Monitoring.
    • Toil Definition.
    • Error Budgets.
    • Test Automation.
    • Capacity planning and measuring risk factors.

    SRE implements a lot of the same practices as DevOps, but from an operations perspective focuses predominantly on operational stability and resilience of the production estate.”

     

    Zuzana Homolyova
    DevOps Institute Ambassador from Slovakia

    “SRE works as a kind of connection bridge between development and operations as it cares about every process from the source code to deployment. It really focuses on how to bring scalable and reliable software systems into the real infrastructure, way, application, performance and security. To describe the difference between DevOps and SRE I would say: ‘DevOps asks what needs to be done but SRE asks how it can be done.’

    What is really important is that DevOps and SRE do overlap each other. They share goals to understand how to measure success or failure and how to gain continuous reliability across every application.”

     

    Craig Pearson
    DevOps Institute Ambassador from Newcastle, England

    “I’ve walked the line between DevOps and SRE in organizations and witnessed first-hand both good and bad practices. While there are a lot of SRE success stories, I’d like to focus on the occasional use of SRE as a justification to rebuild the walls around operations/production that DevOps strived for so long to take down. While adopting  SLOs and focusing effort on toil reduction are obviously beneficial, taking a defensive position over the production environment is not. The trigger for this is often an issue impacting production—sometimes major, sometimes minor—or the perception that more audit rigor is needed in this new continuous delivery world. Whatever the trigger, sometimes SRE is proposed as the answer (R stands for reliability, right?) SRE has to shift-left and spread more of the wisdom of production to teams delivering value-adding products and services to that production environment. Deploying value rapidly is crucial. Doing this in a reliable manner is key. Both are valid reasons for introducing site reliability engineering—the future. However, keeping the wisdom of production in a separate (silo?) SRE team and withholding access to production is a backward step.  A touch of back to the future maybe …”

     

    Nadeem Augustine
    DevOps Institute Ambassador from Cape Town, South Africa

    “Having been exposed to DevOps and SRE, I do see a difference between the two roles. I often find the DevOps way of working focuses on how to bring the issues into development and a way to process it between teams. This creates a collaborative way of working in the organization. That is a pivotal part in rolling out DevOps. DevOps is the ‘what‘ that needs to be done in order to bridge the gap between development. Having played both roles as a ‘developer’ and ‘operations,’ you now see how SRE plays a deeper role in assisting the organization in achieving the end goal. As an SRE you focus on the ‘how‘ to resolve issues end to end with the quickest most stable resolution. Having resolved the issues you create a more reliable and repeatable process.”

     

    Daniel Oh
    DevOps Institute Ambassador and CNCF Ambassador from Boston, USA

    “Who would be the most valuable resource to stabilize the cloud platform built on an immutable infrastructure (i.e. Kubernetes) for the Ops team? Here is the evidence that many recruiters continue to seek the roles that are named site reliability engineers (SRE) but also more IT Ops folks want to have SRE responsibilities in their careers.“

     

    Hugo Lourenço
    DevOps Institute Ambassador from Lisbon, Portugal

    “How long does it take to find the product you need? How long does it take to get the product delivered? Does the product/service require training or expert assistance? Do you need other calls +, products +, services +, to make this product / service work? Does the product require external maintenance and you will be accountable to remember the provider when it’s needed? To get support assistance or any other contact with the provider consumes your valuable time. Your time? All these reasons were the reasons why I embraced agile, lean and automation = DevOps, DevOps is the only weapon in a digital era. I’m passionate about building products and services where consumers will pay to keep your company to exist and deliver such amazing products/services!”

     

    Felipe Duenas
    DevOps Institute Ambassador from Quito, Ecuador

    “SRE is more than buzzwords and trends. It is far more about the people and process than the technology that underlies them. While technology certainly can (and likely will) adapt to the concept as it matures and more teams adopt it, you don’t need new tools to align your development and operations organizations around the principles of Site Reliability Engineering. SRE ’embodies the philosophies of DevOps with a greater focus on measuring and achieving reliability through engineering and operations work.’ SREs also ensure that everyone in the company agrees on how to measure reliability, and what to do when availability falls out of specification. This includes contributors at every level, from developers, through team managers and all the way up to VPs and executives.”

     

    Suresh GP
    DevOps Institute Ambassador from Singapore

    “Site reliability engineering is redefining the value of infrastructure operations and its relevance in the wake of DevOps toolchain and deployment pipeline. After a long time, we see that this area of discipline is creating a profound impact on the holistic value chain by building world-class resilience and reliability of products and services with an end-end view. It also focuses on creating a sense of shared ownership and experimentation in challenging the status quo and delivering streamlined customer experience. I see a lot of synergy with Agile, DevOps, Lean, ITSM with SRE that compliments the dimension brilliantly. Last but not the least, cultural change is of phenomenal importance to make this transition happen and be sustainable. The principles and practices are so sound that the legacy environments can still take advantage of it and contribute to velocity increase and improved reliability.”

     

    BMK
    DevOps Institute Ambassador and CNCF Ambassador from Wellington, New Zealand

    “Enterprises who run or want to run a cloud application at scale, need to adapt to SRE practices and disciplines. Merely stating a 99.99% SLA does not help, if you care about your customers and your customers’ experiences with your systems then, yes, we need to level-up our game, we need to understand the nitty-gritty of reliability of our systems, operations and more importantly understand the limitations and constraints within which we need to operate. Some useful pointers: 1) Caution: Do not simply rename your ‘system admins’ to ‘SRE’  2) Upskill your ops folks to code 3) Upskill your dev folks to learn and know more about infrastructure and operations.”

     

    Aparna Balasundar
    DevOps Institute Ambassador, India

    “If DevOps is the extension of agile to build continuous delivery pipelines, SRE is a natural extension of DevOps to build resilient, repeatable, reliable, scalable production systems. Shift left to include the SRE as part of the architecture design, build, deploy favoring long term viable systems. With the cultural mind shift, if SREs have to provide safe and reliable systems, developers have to dedicate building such systems. These joint working partnerships will benefit the overall business goal.”

     

    Simone Jo Moore
    DevOps Institute Ambassador from Occitanie, France

    “Site Reliability Engineering has as many touchpoints across the service value stream as any other practice. I challenge you to map your interactions across your enterprise so the impacts of SRE to the value outcome are clearly visible. Those companies that treat these touchpoints as part of their beating heart competencies versus peripheral consideration, will continue to leap forward. The socio aspect is important here. Technology is a catalyst—people’s interaction with the tech has shifted, is shifting and continuing to shift. Security is embedded; there is no point at which practices (human and automated) do not exist in SRE enterprise-wide. The resources we apply depends on how big and complicated the system is.

    It goes beyond technical feedback to being an advantageous feedback loop for improvement across business objectives with engagement and financial benefit. Think of SRE as a movie, yes I’m a sci-fi fan but think about movies like “Mission to Mars” or “The Martian” and even “Ad Astra.” In each of these, reliability also includes our trust in something being resilient. Even when things don’t go as expected, there are ways in which the system can be adapted to solve problems. Resiliency is about humans rather than technology, but think about the progression of AI. We want AI to become resilient in not about coming back to a baseline but move forward to a new baseline of its own accord. Its point of stasis is never the same yet reliable based on the quality of the engineering. Our point of stasis is never the same either, and the cycle continues.”

  • A Thousand Salutes to SREs: Unsung Heroes of the Pandemic

    A Thousand Salutes to SREs: Unsung Heroes of the Pandemic

    Applications are the lifeblood of a modern business, and uptime is crucial — even more so during a pandemic. The COVID-19 crisis has shown us that we can adapt to a new normal. It has also shown us that customers will take their business elsewhere if an application experience does not meet their expectations.

    The pandemic has put business-critical applications under unprecedented strain, and has forced organizations across all industries to rethink their business continuity plans when it comes to keeping their applications up and running. In these troubled times, it has often fallen on the humble site reliability engineers (SREs) to keep applications available and to ensure that they are delivering the best possible user experience. 

    SREs as First Responders: Paramedics for Your Applications

    When unforeseen events occur that cause issues with your applications, it’s the SREs who come to your aid. SREs are your first responders — the paramedics for your applications. They are the team on call to administer life-saving CPR to your applications to keep them alive.

    During outages, SREs know what to look for, how to interpret what they see and quickly understand what’s wrong. By monitoring your application’s vital signs, SREs can relay crucial information to your developers and infrastructure team so that they can decide which corrective surgery to perform to restore the application’s health.

    SREs are the sutures that connect the developers and infrastructure teams with the product team to ensure that your applications are available and provide the intended functionality for your end users — always.

    SREs as Wellness Coaches: Preventive Care Providers for Your Applications

    Thomas Edison said that genius is 1% inspiration and 99% perspiration. It’s the same for SREs: They don’t become the heroes of the hour without extensive preparation. SREs are not only the paramedics for your applications during sudden and unexpected events, but they are also the wellness coaches who take preventative healthcare seriously.

    Like wellness coaches, SREs define and develop strategies and action plans to improve the overall wellness of applications to prevent them from crashing. They set up regular application monitoring for early detection of issues so that they can proactively formulate a plan to restore application health.

    The skill of SREs lies in their ability to wear multiple hats and translate business goals into corresponding technical details. They understand the business risk of application failure and work tirelessly on models to quantify it. And then they work with developers and product teams to understand what should be measured as a precursor to failure. They set and maintain service-level objectives (SLOs) aligned with business needs and monitor them constantly to spot and analyze trends.

    SREs determine error budgets so that they will know how bad things can be before an issue becomes a problem. And, when something is trending out of scope, they help diagnose which services are causing the issue, define corrective actions and set the thresholds that enact auto-scaling to meet fluctuation in demand. 

    A Thousand Salutes in a Hundred Languages 

    SREs are, by definition, the proactive enforcers of business continuity. Their role is quite literally to prevent disasters from affecting the business by anticipating the unanticipated. They spend their days (and nights) looking at architectures and applications to assess what can go wrong and to determine how problems can be mitigated. In a pandemic, it is so often the emergency and preventive work of the SRE team that saves the business.

    SREs are, indeed, the unsung heroes, and we should salute them. So, SREs across the world — thank you, dankie, gracias, merci, danke, dhanyavād, grazie, Xièxiè, dziękuję, спасибо, ευχαριστώ, arigatô, obrigado, mulţumesc. Have you thanked your SRE today?

  • Measuring DevOps Performance

    Measuring DevOps Performance

    As IT becomes increasingly central to our organizations, it is increasingly important to improve our ability to deliver innovations efficiently and safely. DevOps is a movement to reimagine the way we deliver software, with an emphasis on delivering value to end users through automation and collaboration. In the midst of complex changes to complex processes, it’s easy to lose sight of the most important point: Our “improvements” must deliver actual improvements. Measuring the performance of a software delivery team is the basic foundation on which you can assess the impact of changes.

    One of the main contributions of the State of DevOps Report has been to focus consistently on the same key metrics year after year. Although the questions in their survey have evolved and new conclusions have emerged over time, the four key metrics used as benchmarks have remained in place:

    1. Lead time (from code committed to code deployed)
    2. Deployment frequency (to production)
    3. Change fail percentage (for production deployments)
    4. Mean time to restore (from a production failure)

    The book “Accelerate” provides a detailed explanation of each of these metrics, and why they were chosen; those points are summarized here.

    The first two of these metrics pertain to innovation, and the fast release of new capabilities. The third and fourth metrics pertain to stability, and the reduction of defects and downtime. As such, these metrics align with the dual goals of DevOps, to “move fast, and not break things.”

    These also align with the two core principles of lean management, derived from the Toyota Production System: “Just in time” and “Stop the line.” “Just in time” is the principle that maximum efficiency comes from reducing waste in the system of work; and that the way to reduce waste is to optimize the system to handle smaller and smaller batches, and to deliver them with increasing speed. “Stop the line” means the system of work is tuned not just to expedite delivery, but also to immediately identify defects to prevent them from being released, thus increasing the quality of the product and reducing the likelihood of production failures.

    Lead time is important because the shorter the lead time, the more quickly feedback can be received on the software, and thus the faster innovation and improvements can be released.  The book “Accelerate” revealed that one challenge in measuring lead time is it consists of two parts: time to develop a feature, and time to deliver it.

    The time to develop a feature begins from the moment a feature is requested, but there are some legitimate reasons why a feature might be deprioritized and remain in a product’s backlog for months or years. There is a high inherent variability in the amount of time it takes to go from feature requested to feature developed. Thus, lead time in the State of DevOps Report focuses on measuring only the time to deliver a feature once it has been developed.

    The software delivery part of the lifecycle is an important part of total lead time, and is also much more consistent. By measuring the lead time from code committed to code deployed, you can begin to experiment with process improvements that will reduce waiting and inefficiency, and thus enable faster feedback.

    Deployment frequency is the frequency of how often code or configuration changes are deployed to production. Deployment frequency is important since it is inversely related to batch size. Teams that deploy to production once per month deploy a larger batch of changes in each deployment than teams who deploy once per week. All changes are not created equal. Within any batch of changes there will be some which are extremely valuable, and others that are almost insignificant.

    Large batch sizes imply that valuable features are waiting in line with all the other changes, thus delaying the delivery of value and benefit. Large batches also increase the risk of deployment failures, and make it much harder to diagnose which of the many changes was responsible if a failure occurs. Teams naturally tend to batch changes together when deployments are painful and tedious. By measuring deployment frequency you can track your team’s progress as you work on making deployments less painful and enabling smaller batch sizes.

    Change fail percentage measures how frequently a deployment to production fails. Failure here means that a deployment causes a system outage or degradation, or requires a subsequent hotfix or rollback. Modern software systems are complex, fast-changing systems, so some amount of failure is inevitable. Traditionally it’s been felt that there’s a trade-off between frequency of changes and stability of systems, but the highly-effective teams identified in the State of DevOps Report are characterized by both a high rate of innovation and a low rate of failures. Measuring failure rate allows the team to track and tune their processes to ensure that their testing processes weed out most failures before they occur.

    Mean time to restore (MTTR) is closely related to the lead time to release features. In effect, teams that can quickly release features can also quickly release patches. Time to restore indicates the amount of time that a production system remains down, in a degraded state, or with non-working functionality. Such incidents are typically stressful situations, and often have financial implications. Resolving such incidents quickly is a key priority for operations teams. Measuring this metric allows your team to set a baseline on time to recover, and to work to resolve incidents with increasing speed.

    In 2018, the State of DevOps Report added a fifth metric, system uptime, which is inversely related to how much time teams spend recovering from failures. The system uptime metric is an important addition for several reasons. First of all, it aligns with the traditional priorities and key performance indicators of sysadmins (the operations team). The number one goal of sysadmins is keeping the lights on or ensuring that systems remain available. The reason for this is simple: the business depends on these systems and when the systems go down, the business goes down. Outages are expensive.

    Tracking system uptime is also central to the discipline of site reliability engineering (SRE). SRE is the evolution of the traditional sysadmin role, expanded to encompass web-scale or cloud-scale systems where one engineer might be responsible for managing 10,000 servers. SRE emerged from Google, who shared their practices in the influential book Site Reliability Engineering. One innovation shared in that book is the concept of an error budget, which is the recognition that there is a trade-off between reliability and innovation, and that there are acceptable levels of downtime.

    According to the Site Reliability Engineering Book, Chapter 3, “Put simply, a user on a 99% reliable smartphone cannot tell the difference between 99.99% and 99.999% service reliability. With this in mind, rather than simply maximizing uptime, Site Reliability Engineering seeks to balance the risk of unavailability with the goals of rapid innovation and efficient service operations, so that users’ overall happiness — with features, service, and performance — is optimized.”

    The State of DevOps Report shows how these five metrics are interrelated (See Figure 1). The timer starts on lead time the moment a developer finishes and commits a feature to version control. How quickly that feature is released depends on the team’s deployment frequency. While frequent deployments are key to fast innovation, they also increase the risk of failures in production. Change fail percentage measures this risk, although frequent small deployments tends to reduce the risk of any given change. If a change fails, the key issue is then the mean time to restore service. The final metric on availability captures the net stability of the production system.

    Figure 1: Correlation between the five metrics.

    How the Five Key Software Delivery and Operations Performance Metrics Tie Together

    Together, these metrics constitute a team’s software delivery performance. The goal of any DevOps initiative should be to improve software delivery performance by strategically developing specific capabilities such as continuous delivery and the use of automated testing.

    How your team measures these capabilities is another challenge. But “Accelerate” makes a compelling argument for the validity of surveys. Automated metrics can be implemented over time, although the mechanism to do this will depend on how you do your deployments. Salesforce production organizations track past deployments, but it’s not currently possible to query those deployments, so you would need to measure deployment frequency (for example) using the tools you use to perform the deployments. Salesforce publishes their own service uptime on Trust, but that gives no indication of whether critical custom services that customers build on Salesforce are in a working state or not.

    Surveys provide a reasonable proxy for these metrics, especially if responses are given by members of the team in different roles. Guidelines for administering such surveys are beyond the scope of this book, but your teams’ honest responses are the most critical factor. Avoid any policies that could incent the team to exaggerate their answers up or down. Never use these surveys to reward or punish; they should be used simply to inform. Allow teams to track their own progress and to challenge themselves to improve for their own benefit and for the benefit of the organization. As it says in the Agile Manifesto “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”

    Metrics provide a reliable, long-term indicator of how your software delivery team is performing. They open the door for your team to experiment with different approaches and assess their impact using a common standard. The key metrics described here are important because they emphasize end-to-end performance, and thus incent teams to focus on collaboration towards this common goal. Balancing velocity with reliability is critical, thus these metrics should be viewed together, to ensure that one goal is never emphasized at the expense of the other.

  • Site Reliability Engineering 101: DevOps Versus SRE

    Site Reliability Engineering 101: DevOps Versus SRE

    Consider the following scenario.

    An Independent Software Provider (ISV) developed a financial application for a global investment firm that serves global conglomerates, leading central banks, asset managers, brokerage firms and governmental bodies. The development strategy for the application encompassed a thought through DevOps plan with cutting-edge agile tools. This has ensured zero downtime deployment at maximum productivity. The app now handles financial transactions in real-time at an enormous scale, while safeguarding sensitive customer data and facilitating uninterrupted workflow. One unfortunate day, the application crashed and this investment firm suffered a severe backlash (monetarily and morally) from its customers.  

    Here is the backstory: Application’s workflow exchange had crossed its transactional threshold limit, and lack of responsive remedial action crippled the infrastructure. The intelligent automation brought forth by DevOps was confined mainly to the development and deployment environment. The IT operations, thus, remained susceptible to challenges. 

    Decoupling DevOps and RunOps — The Genesis of Site Reliability Engineering (SRE)

    A decade or two ago, companies operated with a legacy IT mindset. IT operations consisted mostly of administrative jobs without automation. This was the time when the code writing, application testing and deploying was done manually. Around 2008 to 2010, automation started getting prominence. Now developers and ops teams worked concurrently toward continuous integration and continuous deployment — backed by the agile software movement. The production team was mainly in charge of the runtime environment. However, they lacked skillsets to manage IT operations, which resulted in application instability, as depicted in the scenario above. 

    Thus, DevOps and RunOps were decoupled, paving the way for SRE — a preventive technique to infuse stability in IT operations.

    Software-First Approach: Brain Stem of SRE

    “SRE is what happens when you ask a software engineer to design an operations team,” said Benjamin Treynor Sloss, VP of engineering at Google.

    This means an SRE function is run by IT operational specialists who code. These engineers implement a software-first approach to automate IT operations and preempt failures. They apply cutting edge software practices to integrate dev and Oops on a single platform, and execute test codes across the continuous environment. Therefore, they carry advanced software skills, including DNS configuration, remediating server, network and infrastructure problems, and fixing application glitches. 

    The software approach codifies every aspect of IT operations to build resiliency within infrastructure and applications. Thus, changes are managed via version control tools and checked for issues leveraging test frameworks, while following the principle of observability.

    SRE, DevOps

    The Principle of Error Budget

    SRE engineers verify the code quality of changes in the application by asking the development team to produce evidence via automated test results. SRE managers can fix service level objectives (SLOs) to gauge the performance of changes in the application. They should set a threshold for permissible minimum application downtime, also known as error budget. If a downtime during any changes in the application is within the scale of the error budget, then SRE teams can approve it. If not, then the changes should be rolled back for improvements to fall within the error budget formula.

    Calculating Error Budget

    A simple formula to calculate error budget is: (system availability percentage) minus (SLO benchmark percentage). Please refer to the System Availability Calculator cheat sheet below.

    Suppose the system availability is 95% and your SLO threshold is 80%. 

    Error budget: 95% – 80% = 15%

    Error budget/month: 108 hours. (At 5% downtime, per day downtime hours is 1.2 hours. Therefore, for 15% it is 1.2 x 3 = 3.6. So, for 30 days it will be 30 x 3.6 = 108 hours).

    Error budget/quarter: 108*3 = 324 hours.

    Quick trivia: Breaking monolithic applications lets us derive SLOs at a granular level.

    Cultural Shift: A Right Step Toward Reliability and Scalability 

    Popular SRE engagement models such as Kitchen Sink (aka Everything SRE), infrastructure teams and embedded SREs, tend to build dedicated teams that lead to a “Silo” SRE environment. The problem with the Silo environment is that it promotes a hands-off approach, which results in a lack of standardization and coordination between teams. So, a sensible approach is shelving off a project-oriented mindset and allowing SRE to grow organically within the whole organization. It starts by appraising the teams of customer principles and instilling a data-driven method for ensuring application reliability and scalability. 

    Organizations must identify a change agent who would create and promote a culture of maximum system availability. The agent can champion this change by practicing the principle of observability, where monitoring is a subset. Observability essentially requires engineering teams to be vigilant of common and complex problems hindering the attendance of reliability and scalability in the application. See the principles of observability below.

    The principles of observability follow a cyclical approach, which ensures maximum application uptime.

    Step Zero: Unlocking Potential of Pyramid of Observability

    Step zero is making employees aware of end-to-end product detail — technical and functional. Until an operational specialist knows what to observe, the subsequent steps in the pyramid of observability remain futile. 

    Also, remember that this culture shift isn’t achievable overnight. It will be successful when practiced sincerely over a few months. 

    DevOps Versus SRE

    People often confuse SRE with DevOps. SRE and DevOps are complementary practices that drive quality in the software development process and maintain application stability. 

    Let’s analyze four key fundamental differences between DevOps and SRE. 

    SRE, DevOps

    Conclusion – SRE Teams as Value Center

    A software product is expected to deliver uninterrupted services. The ideal and optimal condition is maximum uptime with 24/7 service availability. This requires unmatched reliability and ultra-scalability. 

    Therefore, the right mindset will be to treat SRE teams as a value center, which carries a combination of customer-facing skills and sharp technical acumen. Lastly, for SRE to be successful, it is necessary to create SLI driven SLOs, augment capabilities around cloud infrastructure, a smooth inter-team coordination and thrust automation and AI within IT operations.