devopsy.kinsta.cloud

Tag: site reliability engineering

Why SREs Are Critical to DevOps
Although a relatively new concept, site reliability engineers (SREs) have become crucial for DevOps teams, helping to solve an array of operational problems such as network availability and user experience. However, in previous years, some people have questioned the longevity of such a role.

This article will discuss several reasons why site reliability engineering roles are here to stay and why they have become critical to DevOps.

What Are Site Reliability Engineers (SREs)?

A site reliability engineer helps connect development teams and IT operations by completing tasks previously assigned to operational roles. SREs use a range of automation tools for troubleshooting purposes, creating software systems that are both scalable and reliable.

The role revolves around automation and standardization, as cloud systems become increasingly prevalent. SREs also play a vital role in ensuring the user experience of systems is of the highest standard.

Why Site Reliability Engineering Roles Are Here to Stay

Let’s take a look at two of the primary reasons SRE roles are here to stay.

User Expectations

The key reason for the rise in popularity and the expected longevity of site reliability engineering roles is user experience. In the modern climate, user expectations are very high and more dedicated roles are required to help deliver a service that is not only acceptable but attempts to exceed user expectations.

Even back in the 2010s, users were much more patient in terms of website load speeds and how quickly they could access information. Even crashes and load failures were tolerated without the user exiting a website.

However, that has all changed due to the competitiveness of the market, as well as the growth of remote working, which has increased dependency on online resources. This means that if a service is unavailable for any period, users are unable to do their job.

An SRE’s role includes optimizing systems, ensuring they perform at the required level and that resources are always available to those that need them. This makes SREs critical to DevOps teams that may not have the capacity to focus on meeting user expectations sufficiently.

The Complexity of Software Architectures

Another factor that has added a lot of value to the role of an SRE is their ability to provide much-needed support for software environments and architectures that continue to develop in terms of complexity. Kubernetes is one such architecture that requires dedicated attention.

Applications that are distributed across systems such as Kubernetes and other cloud-native platforms require specialized personnel that have the necessary knowledge and experience. Machine learning is also becoming more prevalent and requires special focus from a performance perspective. In 2020, just more than 22% of companies had machine learning models in production for one to two years.

This level of complexity is only expected to grow further over the coming years as cloud-native systems become the norm. This is why new roles must be created that go beyond that of a normal IT engineer.

Challenges for SRE Roles

Like any IT role, the role of a site reliability engineer always faces challenges that have led people to question whether SREs are a long-term solution or whether it is a niche specialization.

A Lack Of Opportunity Away From Hyperscale Companies

The role of an SRE was created by hyperscale companies like Google that manage extremely large IT systems. For companies that do not have such a wide and varied IT infrastructure on a global scale, there are questions about whether such a role is necessary.

However, as previously mentioned, the growth of cloud-native technology shows no sign of slowing and, with this growth, opportunities for SREs at smaller organizations may arise.

The Role of an SRE is Considered Obscure

The ambiguity of an SRE role can be considered a downfall in some cases, with many businesses unsure of what these professionals offer when compared to a traditional systems engineer. In many cases, SREs are deemed to be a hybrid of a software engineer and an IT operations role.

Fortunately, DevOps has realized the potential of SREs and their ability to provide an important bridge between developers and engineers, helping to achieve dedicated tasks to a high standard.

SREs Vs. DevOps

Although both roles are connected and share some similarities, there are some clear differences.
- Processes – DevOps has full visibility into the development environment which allows them to make changes from the initial development stages through to production. SREs, on the other hand, only have visibility into the production process, allowing them to make suggestions to ensure performance levels are maintained.
- Implementation – The task of implementing new features on a system falls on the DevOps team. Meanwhile, SREs are tasked with ensuring the new features do not cause any system failures or impact performance during the production stage.
- Key Focus – The primary focus of a site reliability engineer is to ensure the system’s reliability, availability and scalability. DevOps focuses on how quickly product development is completed, as well as its continuity.
- Structure – The structure of an SRE team and a DevOps team have obvious differences. SRE teams are made up of individuals who have similar skill sets relating to development and operations. Whereas DevOps teams consist of professionals who have specific roles related to individual aspects of the project. These roles can include the team leader, product owner, cloud architect, software developer, QA engineer, system administrator, release manager and more.
- Tools – DevOps teams often use tools such as integrated development environments (IDEs) when developing a product. These can include Jenkins (continuous integration and continuous development), JIRA (change management), Splunk (log monitoring) and GitHub (distributed version control). SREs commonly use tools such as Prometheus and Grafana (collection and visualization of metrics, such as CPU usage and available disk space), OP5 and PagerDuty (incident alerts), Ansible and Kubernetes (container orchestration) as well as a range of cloud platforms.
- Bug Reporting – DevOps debug code whenever a bug is reported in the end product. SREs are not involved with debugging from a development perspective and are only required to perform such tasks if there is a production outage, there are infrastructure issues or perhaps if they are dealing with common AWS misconfigurations.
- Measuring Performance – The typical metrics that DevOps uses to measure performance are deployment frequency and deployment failure rate. SREs measure service level objectives (SLOs), service level indicators (SLIs), service level agreements (SLAs) and error budgets.
- Incident Handling – In the case of incident handling, DevOps teams work to mitigate an issue based on the feedback provided. SREs then analyze the issues post-incident, including the root cause of the issue. This is then documented and provided to the developers to fix.
Why SRE Roles Are Critical to DevOps – Conclusion

As you can see, the two roles differ significantly but SREs can play a critical role in ensuring the DevOps team delivers a product that offers maximum performance.

The core benefits of employing professionals in a site reliability engineer capacity are high levels of product performance and reliability to meet user expectations while helping to manage complex IT architecture.

In the future, organizations that manage a large amount of cloud-based IT assets may struggle to deliver efficient products without such professionals. As such, demand for SRE roles is likely to grow.
January 5, 2023
Best of 2022: Day in the Life of a Site Reliability Engineer (SRE)

As we close out 2022, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

By now, most of us are familiar with the concept of site reliability engineering (SRE). The term was originally coined by Google and SRE has been gaining traction in recent years as a role dedicated to increasing the resilience of digital ecosystems. To accomplish this, a big part of the SRE doctrine is “automating your job away.”

Observability and monitoring tools are extremely important to DevOps and for SRE. But what is the day-to-day experience actually like for your average SRE? What activities does an SRE do? And what percentage of their time does an SRE spend on particular tasks?

I recently met with James Curtis, lead site reliability engineer at a large multinational company. According to Curtis, the SRE approach takes a certain type of person—someone with Ops in their blood and a passion for eradicating repetitive tasks. Below, we’ll use his input and the input from other sources to better understand what it’s like being in the shoes of an SRE.

Understanding the SRE Role

Organizations have interpreted the role in different ways. But in a general sense, SREs attempt to maintain high reliability and availability for software applications and respond to incidents as they occur. To aid their efforts, an SRE tries to streamline and automate as many operations as possible to remove opportunities for human error.

A hallmark SRE goal is to reduce “toil.” Curtis defines toil as “tedious actions that really have no enduring value.” For example, say an admin must manually restart a service in Microsoft Exchange every time a triggering event interrupts the service—this action could certainly be automated away. SREs spend much of their time eliminating toil by coding automation and configuring internal tools to better interact with software infrastructure.

SREs usually are also in charge of logs and setting benchmarks using a tool like Splunk or Datadog to observe and ingest data. Curtis himself uses Cribl, which offers an observability pipeline to parse and route log data. Since SREs oversee internal service-level indicators, they are typically in charge of normalizing behavior and setting SLOs and SLAs.

Common SRE Activities

An SRE juggles a lot of tasks. For proof, read the hour-by-hour day in the life by Yonatan Schultz, SRE at New Relic. Schultz’s average day is spent configuring infrastructure, jumping from project to project, and, of course, hopping into many meetings. Here are some other tasks an SRE might perform on a daily basis:

Monitoring service-level indicators (SLIs). An SLI could be the number of successful requests out of total requests. Having a high SLI, in this case, would be a target. SREs track other metrics such as availability, uptime performance, latency, error count and throughput. Regularly monitoring systems is essential to ensure proper resource utilization of containers and to avoid out-of-memory (OOM) errors.

Setting SLOs and SLAs and determining error budgets. Once you have determined baseline system performance, you can set service-level objectives (SLOs). These are typically internal targets like 99.99% availability. While SREs typically oversee functional metrics, some teams set goals for non-functional metrics, as well. SREs help determine service-level agreements (SLAs), which are more legally binding and typically partner-facing.

Responding to incidents. On-call SREs will be tasked with finding the root cause of issues as they arise. When triaging an incident, it’s helpful to have all the necessary logs and tools immediately at hand. This is one area where automation can assist by pulling relevant details to instantly build a case, said Curtis.

Writing postmortems. After an incident has been dealt with, it’s important to learn from it. Postmortems are common in cybersecurity practice and often fall under the responsibility of an SRE. These reviews seek to answer set criteria to get to the heart of an incident and identify the root cause(s) of an issue to prevent it from happening again.

Automating other system tasks. SREs will spend significant time coding and building tools for engineers to interact with infrastructure. For instance, an SRE might generate reliability reports that consider performance over long time periods.

Cross-department collaboration. SREs don’t tend to own application code. Instead, they support multiple software divisions. This means checking in with other developers, disseminating best practices and reviewing new architectures to represent the reliability side of the equation.

As you can see above, the SRE role might blend many different activities, and keeping track of them all may be part of the job itself. Anika Mukherji, an SRE at Pinterest, shared that, at Pinterest, there is a weekly meeting where SREs share what they spent time on. For another helpful “day in the life” story, take this account of an average OpenShift SRE’s day. Nikita spends her day responding to open JIRA cards, handling incidents, pushing code to GitHub and syncing with SREs in other regions when shifts change.

Time Well Spent

So, how does an SRE allocate their time? As Curtis explained, the ideal goal is a 50/50 split between time spent in work mode and time trying to automate that work away. Of course, this is more like a sliding scale, he admits. When things are broken, attention naturally shifts toward more manual work, says Curtis. While there may be less automation upfront, the scale balances out the more you build.

This is similar at other institutions too. For example, in an interview with DZone, Paul Greig explained that half of his time is spent on service reliability upkeep and half on toil reduction. John Turner, SRE at Squarespace, said some 70% of his time is spent writing code—much of it automation code.

Who’s Fit to Be an SRE?

As mentioned above, the SRE job requires a specific attitude toward solving operational problems. Curtis described this person as “someone who hates the monotony of doing something over and over.” This person should have a drive to continuously solve new problems because as soon as you’ve automated one thing, you move on to the next, he said.

Don’t assume a role with high automation equals an easy paycheck. We’ve all heard the story of the system admin who secretly automated their job away and never told a soul. While this may fly in other roles, the job of an SRE is never finished. “You’re never going to run out of stuff to optimize,” Curtis said.

Again, the SRE role is a blend of many different activities. You will likely interface with different developer teams, too. Since the role involves this type of interaction, communications skills and understanding are a must.

Final Thoughts

The SRE practice is wide and varied—each company may have its unique flavor. Looking to the future, there are plenty of use cases for machine learning to further empower SRE practices, said Curtis. This is especially relevant in security automation, where algorithms could be trained against real attacks to flag suspicious behavior. Or, with the right amount of data, predictive analytics could be applied to anticipate high CPU peaks, informing server utilizations.

In both scenarios, Curtis stressed the importance of observability. “It really gives you the ability to look at data and to ask a question later you didn’t realize you needed to ask,” he said. This power hinges on easy data transformation to make things speak the same language. That’s why his team opted for Cribl, that allowed them to normalize that data on the fly and replay it later. “The ability to change data and morph it—that’s the power observability gives you.”

December 23, 2022
SRE Survey Reveals Major Technical and Cultural Challenges

Catchpoint, in partnership with Blameless, today published an annual survey of 559 site reliability engineers (SREs) that found 59% of respondents didn’t view tool sprawl to be a major concern. Another 40% said tool sprawl is a moderate (33%) to serious problem (8%).

More than half of respondents said they build as much as 30% of the tools they use themselves, with more than half (54%) noting their organization tracks three or more different types of telemetry data to drive observability. The top three types of telemetry data collected are infrastructure (62%), application (58%) and network monitoring (55%) and, on average, the survey suggested the median toil level of SREs dropped 5% year-over-year.

Just under half of respondents (46%) also noted they saw no or low value from artificial intelligence for IT operations (AIOps) compared to 30% that have seen moderate to high value. Another 25% are still unsure.

Finally, the survey concluded that organizations that have adopted a “just culture”—where organizations ensure that incident reporting is not met with punitive measures—are 500% more likely to be Elite performers in terms of the DevOps Research and Assessment (DORA) metrics defined by Google.

Leo Vasilou, director of product marketing for Catchpoint, said that the survey results provided some insights into the challenges SREs face as they strike a balance between trying to achieve DevOps goals and the fact that hiring has become more constrained during an economic downturn.

That stress, however, has less to do with managing toolchains than it does ensuring that applications driving critical business processes are consistently available, he noted.

SREs have emerged as a distinct DevOps function in recent years, focused almost exclusively on programmatically managing the dependencies that exist between a wide range of applications and the underlying infrastructure upon which they depend. Optimizing applications in the cloud-native era has become more challenging because the number of dependencies that exist between the microservices that make up a modern distributed computing environment continues to exponentially increase. In many cases, applications are continually added to IT environments without a commensurate increase in the overall size of the SRE team.

Regardless of the economic climate, hiring and retaining SREs remains a challenge given the demand for IT professionals with those skills. Of course, there is no formal SRE certificate that is widely recognized so who qualifies to be an SRE tends to vary from one organization to the next. The one thing that is generally agreed upon is that SRE automate a wide range of tasks that would otherwise require multiple IT administrators to perform. Glassdoor also reports that, on average, an SRE makes north of about $125,000, which ranks among the highest salary level within any IT organization.

Each organization, depending on their geographic region, would need to determine how many IT administrators they could replace if they hired an SRE. That doesn’t mean IT administrators that rely on a wide range of graphical tools to manage IT tasks are going away any time soon, but as DevOps continues to evolve, the dynamics between SRE and IT administrators also will continue to evolve.

November 8, 2022
Scaling Predictive Analytics With AIOps to Drive Next-Gen SRE

Enterprise systems are only as valuable as they are reliable, in the sense that they don’t suffer excessive breakdowns. Otherwise, companies experience costly downtime and added stress for engineers due to the additional burden of managing issues. This critical function of ensuring systems run reliably and optimally, at production scale and with minimum human intervention, is the purview of site reliability engineering (SRE) teams.

The SRE professional’s job is to reduce human toil by developing and implementing reliable and highly scalable systems that optimize software and applications. Their goal is to proactively identify and, where possible, anticipate potential disruptions to minimize risk and ensure optimal system performance. However, to be successful, SRE teams require a level of enterprise-wide visibility that can be difficult to provide without the right tools to capture and analyze data at the appropriate level of detail.

Let’s examine how artificial intelligence for IT operations (AIOps) can support the SRE mission by connecting data enterprise-wide for unprecedented visibility and control of systems and processes, enabling auto resolution of most issues and more efficient triaging for the few remaining cases where SRE teams must collaborate around a fix.

Challenges of Complexity and Scale in Pursuing SRE Excellence

As more enterprises digitize their operations and move to greater automation, their IT operations must leverage all data assets skillfully in order to improve reliability and reduce human toil. The SRE profession arose to satisfy this need, with a focus on monitoring systems, accessorizing automated releases, understanding change impacts and automating some of the most common system processes.

To do their job effectively, SRE teams need a wide variety of data and analytics capabilities at their disposal. This includes the ability to deep dive into descriptive and diagnostic analytics to look backward and at present conditions to discover what happened in the past and why and to baseline current operations. But this is just the beginning. The true value of SRE comes with the scaling of predictive and prescriptive analytics to draw on that historic data and apply detailed analysis to generate predictive insights into what is most likely to happen in the future; these insights are the basis for identifying proactive measures that can serve to address any potential problems and optimize those future outcomes, minimizing adverse impacts on operations

These analytic capabilities must be fed by robust data that comes from across the entire IT estate. Developers assigned to the SRE role face an especially strong mandate to have this holistic visibility as they plan, build, test, release, monitor and secure systems; their job is hampered to the extent that organizational silos or problems of scale get in the way of achieving that visibility. It is here that AIOps can help evolve IT operations to become more proactive and autonomous by scaling the power and reach of predictive and prescriptive analytics.

AIOps Empowers the SRE Mission Enterprise-Wide

AIOps is an essential tool for the SRE community in the battle to reduce operator stress, configure IT systems to be more stable and run efficiently with less human intervention. AIOps employs artificial intelligence (AI) and machine learning (ML) for observability, context, normal behavior analysis and automated health diagnostics. This, in turn, enables anomaly detection in real-time and closed-loop automatic resolution of most issues.

For example, AIOps can uncover patterns showing that, 90% of the time, a particular alert in the organization’s payment system triggers a seemingly unrelated alert within 15 minutes. Moving forward, this gives SRE teams a 15-minute head start on addressing that secondary alert; the discovery forms the basis of an auto-resolution or triaging scenario that uncovers the underlying cause of both alerts and provides a permanent fix.

That’s just one illustration of how AIOps uses advanced analytics to discern subtle patterns in data to predict when and where problems may occur, so proactive fixes can automatically be prescribed to head off those problems. In this way, AIOps is the technology backbone that scales and automates the SRE team’s insight and control across all system assets and dependencies, a critical tool for continuous optimization of performance with minimal human intervention.

Conclusion: The Game-Changing Role for AIOps in SRE

AIOps is a game-changer for the critical function of site reliability engineering. Powered by a potent blend of advanced analytics on robust data coming from systems across the enterprise, AIOps delivers unprecedented visibility and control for SRE teams in their mission to reduce toil and ensure the reliability and resiliency of enterprise systems. The result is less human intervention and enhanced value from IT systems that are made more robust, more stable and more efficient.

September 26, 2022
5 Ways to Prevent an Outage

In today’s always-on, ever-connected world, we all expect 100% availability.

What gets in the way of this? The devil is in the details. Over time, everything breaks: Disks, nodes, containers, networks, DNS servers and configuration issues can all lead to major outages.

Amazon’s 13 minutes of downtime in August 2021 translated to almost $5 million in lost revenue and Amazon Web Services’ three outages in December of last year cost millions more. And it was widely reported that the October 2021 Facebook (including related products: Messenger, Instagram and WhatsApp) outage resulted in an estimated $100 million in lost revenue due to a network configuration error.

There have been many outages of varying scope and duration—some attributed to technology breaking, others traced back to people making mistakes. In fact, a 2017 Amazon outage was attributed to “human error”–or more specifically, one employee’s typo. The employee intended to take a small number of servers offline to fix a billing system issue but the incorrect command led to several servers being affected and a major outage.

So, what can your company do to reduce the risk, duration and impact of a potential outage? For one, companies can’t pretend that if they only try harder, they can prevent humans from making mistakes. Do you have dozens of people manually keying in hundreds of commands every day? If so, a mistake is inevitable. Companies should instead investigate how and why one small blunder in a command line could do widespread damage. Guardrails and redundancies should be in place to protect against and minimize these types of incidents.

As the enterprise continues its digital transformation toward a multi-cloud, hybrid and distributed world, here are five guidelines to consider—guidelines that can help you preempt and prevent your next big outage.

1. Always Consider Your Blast Radius

Blast radius measures how one error, typo or malfunction can impact other parts of a system.

What is the chain of components or services that is impacted by one error?

What are the hidden dependencies?

How many customers would be affected and for how long?

More is connected than you think; discover how it’s all related in advance to eliminate surprises.

If too many systems in a network are mutually dependent and interconnected, the result is deadlock. In other words, when one breaks, the other also fails. We have seen this in past Facebook outages. A single erroneous command sparked a domino effect that shut down the backbone connecting all of Facebook’s data centers globally.

How can you prevent this scenario? Plan for the worst possible scenario. When it comes to mission-critical systems, take an intentional approach to situation failover and backup. What if A, B and C all happen simultaneously? Also, think about your cost versus reliability trade-offs. Maybe you can’t afford two full sets of servers but can afford a full set of archived code.

2. Smooth Out Rough Edges

Consider that core infrastructure and code get used (and tested) a lot more than edge code/infrastructure. Basically, the core is hardened while edge infrastructure is more vulnerable. You should be more proactive at the edge around identifying vulnerabilities and doing explicit testing to counter this risk.

The 2014 Google disk erase/satellite outage is a cautionary tale involving the edge. In Google’s case, the edge networks offered a key advantage: A better connection for the customer due to lower latency since data is accessed from a source closer to the user as opposed to the core network. Google had a CDN/edge network for TCP connection termination, caching for various apps, DNS regulation, etc. Teams needed to routinely upgrade a satellite rack and run decommission automation—and this automation failed part of the way through. Error message. They retried the decommission process again to debug the failure. The manual re-execution of the decommission automation meant thousands of satellite machines were decommissioned and, within minutes, the disks of all satellite machines globally were erased. The result: Customers experienced an increase in latency. Behind the scenes, it required two days of all-hands-on-deck to reinstall these machines.

3. Think Escalator, Not Elevator

A good system has both defense and depth. Considering that it takes three or four things to (really) go wrong, think about failure cascades. Bluntly put, limit dependencies. Measure the longest dependency failure path and mean-time-to-recovery (MTTR). When you do this, think escalator not elevator. What do I mean? Well, when an elevator fails, you are left with a shaft; an escalator that fails still works as a staircase.

In 2016, severe weather in Sydney, Australia resulted in an AWS EC2/EBS outage. First, the utility provider suffered a loss of power at a regional substation. This failure resulted in a total loss of utility power to multiple AWS facilities. In one of the facilities, AWS power redundancy didn’t work as designed and power was lost to a significant number of instances in that availability zone. To make matters worse, the instances that lost power lost access to both their primary and secondary power as several power delivery lineups failed to transfer load to their generators. Also, a latent bug in AWS instance management software resulted in a slower-than-expected recovery of the remaining instances.

The lesson? This correlated power delivery lineup failure illustrated that failures are not independent. You must design for the largest correlated failure in combination with the background noise of ongoing independent failure. Recovery code needs to be scale-tested against the largest expected failure.

4. Stay Ahead of Digital Certificate Rotations

Don’t underestimate the importance of digital certificates—and their expiration dates. A few years ago, LinkedIn allowed one of its SSL certificates to expire and it knocked out LinkedIn sites in the U.S., UK and Canada. In 2017, Equifax missed a breach for 76 days because of an expired certificate. The average annual cost of unplanned outages due to certificate expirations is $11.1 million.

For example, you could have a load balancer sitting in front of several servers. The load balancer could have an up-to-date certificate while the servers behind it are expiring. It is important to check for expiring servers in multiple ways. Ideally, you check the boxes in two ways: Ping the HTTP endpoints and also have scripts that check the certificate files themselves.

Expired certificates (TLS for example) can lead to encryption and authentication risks as well as lost internet connectivity and larger outages involving websites and applications. To avoid this, you continuously check, renew and deploy certificates—and this can be achieved via automation. Think about automatically rotating expiring certificates.

5. Tackle Toil to Empower Your Teams

What’s the biggest headache for site reliability engineers (SREs) and those who manage production operations? Toil.

We can define toil as manual, tedious tasks that engineers perform within production environments. Basically, toil wastes time and slows down operations. As a result, most SREs and engineering teams strive to minimize toil within their workflows.

In production operations, you should work to limit toil. Make sure your team is regularly investing time in proactive work; the reactive work shouldn’t take most of their time. Toil reduction gives developers, DevOps engineers and SREs more time to focus on projects that create net-new value for the business (new application features). Google stated that it has an “advertised goal” of keeping toil “below 50% of each SRE’s time.” Basically, no engineer should spend more than half of their time on manual, repetitive work. But the ultimate goal should be to get toil as close to zero as possible. Automating incident remediation can help SREs spend less time on toil and less time on-call.

Also, when you implement fixes—especially people and process fixes—ask yourself if this is a systemic fix or one that is relying on best intentions. Look for tools to safely empower your team (with guardrails). As you work to identify major vulnerabilities and dependencies (as discussed above), you should automate away the most common issues while planning for the “corner cases” (or the 1% of 1%) that may lead to major outages (a.k.a. the worst-case scenarios).

As you consider these guidelines, make sure you focus on putting actual mechanisms in place—not one-off, “hopeful” directives. Do not rely on people to remember best practices from a training session—these are not mechanisms. Instead, design processes that can be systematically implemented. When you do a post-mortem, look for concrete programs you can put into place that will prevent the same mistake from happening again. Automation can help you achieve this level of confidence, management and quality control—and it may just keep you one step ahead of a potential future outage.

August 15, 2022
Why More Incidents Are Better
Ask most SREs how many incidents they’d have to respond to in a perfect world, and their answer would probably be ‘zero.’ After all, making software and infrastructure so reliable that incidents never occur is the dream that SREs are theoretically chasing.

Reducing the number of actual incidents as much as possible is a noble goal. However, it’s important to recognize that incidents aren’t an SRE’s number-one enemy. What matters more than the number of incidents you experience is how effectively you respond to each one.

Plus, there’s value in incidents. They are a learning opportunity. If your business never experienced them, it would arguably be facing more risk, not less.

We know: These ideas may sound a little counterintuitive. You might even accuse us of being “pro-incident”—which we sort of are. Allow us to explain.

The Silver Lining

In many respects, incidents are inherently bad. When one occurs, it means something broke. That’s bad. It may also mean that users were disrupted, operations halted or money was lost. Those things are even worse.

On the other hand, incidents aren’t all bad. They actually benefit SRE teams for several reasons:
- Learning opportunities: Incidents are opportunities to figure out what went wrong and prevent it from recurring. They can also help teams learn how to react more quickly or efficiently the next time something fails.
- Get ahead of bigger issues: Sometimes, working through one incident means you can avoid another that’s even worse. Perhaps one server fails, for example, and your response revealed that the failure was due to a larger issue that would have eventually caused a worse outage if left unaddressed. But thanks to the incident, you detected the larger issue before it triggered a more massive failure.
- Reinforce team culture: Nothing breeds camaraderie or a spirit of collaboration like working alongside other engineers to respond to a crisis in the middle of the night. Although being in this setting may not be anyone’s first choice, it does often have a positive impact on your team’s culture and esprit de corps.
- Demonstrating value: Assuming you handle them well, incidents are an opportunity for SREs to prove how valuable they are to the organization. If incidents never happened, it’s a safe bet that some bosses would start to wonder why they need SREs in the first place. (It would be a flawed train of thought, of course, because SREs would deserve credit for preventing incidents, but it’s a thought that may float around some C-level brains nonetheless.)
We could go on, but the point is clear: Although incidents cause problems in some respects, they actually create value in others.

Focus on Response, Not Avoidance

This is not to say that you should welcome incidents with open arms. Obviously, any decent SRE should focus first and foremost on being proactive and preventing incidents from happening whenever possible. They should use chaos engineering to identify problems that could be lurking unseen in production environments. They should leverage IaC to minimize risks. And so on.

That said, what ultimately matters more than incident frequency is the effectiveness of incident response. It’s better to experience ten incidents that you resolve in under an hour each than one incident that takes mission-critical systems offline for a week.

So, in addition to investing in tools and processes that mitigate the risk of incidents, SRE teams should place equal emphasis on ensuring that they can react quickly and effectively when an incident happens. This means having the ability to share information efficiently, define clear roles, know what to prioritize when working through complex incidents and have clear plans in place that spell out how you’ll handle a problem as soon as you detect it. Without these abilities, you’re at risk of letting incidents that should be small turn into major outages.

‘Zero Incidents’ is not Realistic

It’s important to recognize, too, that while it can be fun to imagine a world where zero incidents occur, the reality is that such a world will never exist. If it could, we wouldn’t see new records set each year for the number of security incidents that businesses collectively suffer, for example.

Nor would we see headlines about major outages at huge enterprises like Facebook or AWS on a recurring basis. If those companies, which have world-class reliability teams and virtually endless resources at their disposal, can’t reduce incidents to zero, neither can anyone else.

Conclusion

The bottom line: There is no such thing as total incident prevention, no matter how hard you try. And even if there were, that wouldn’t actually be a good thing, for the reasons explained above.

So, by all means, undertake reasonable proactive efforts to prevent as many incidents as you can from happening. But don’t let investment in prevention cause under-investment in response. Being prepared to handle incidents when they happen—which they inevitably will—is what matters most.
July 15, 2022
How to Adopt an SRE Practice (When You’re not Google)
Site reliability engineering (SRE) isn’t a new term or practice. The practice of applying software engineering skills and principles to operations problems and tasks happened even before site reliability engineer was a defined job title. But organizing a proactive approach to building and maintaining software drives long-term success in improving operational efficiency, data-driven roadmap planning and general uptime and reliability. All these advantages account for the widespread adoption of SRE.

In this article, I’ll dive into what it takes to get into site reliability engineering, how to adopt it within your own organization and some of the core principles and best practices you’ll need to keep in mind as you move forward in your SRE maturity journey.

What is SRE?

SRE, or site reliability engineering, is the practice of applying software engineering expertise to DevOps and operations problems. Often, this means proactively writing code and developing internal applications or services to combat reliability and performance concerns. SRE has been a practice for many years but has been more recently popularized by Google when they first published Site Reliability Engineering: How Google Runs Production Systems in March 2016.

SRE is a complicated acronym because it both relates to a job title, site reliability engineer, as well as a general practice adopted by development and IT teams, site reliability engineering. SRE teams are often organized differently depending on the organization and the services they support. Sometimes site reliability engineers (SREs) are sprinkled among development teams. This means they can vocalize concerns during roadmap planning and work closely with QA teams, collaborating on everything from shipping features to staging to managing production environments.

Another approach organizes an SRE team as a standalone group. The SRE team as a whole is focused on responsibilities including monitoring applications and infrastructure, establishing reliability metrics, tracking issues with new releases and handling on-call duties. No matter how they are organized, what remains the same for all these SREs is their initiative—reliability and performance.

DevOps Vs. SRE

The rapid adoption of DevOps practices may have you wondering, “How is SRE different from DevOps?” Commonly, the two terms go hand-in-hand. SRE is a practice that doesn’t necessarily have to be part of DevOps culture and adoption but is often implemented by organizations that are further along in their DevOps journey. The further along on your DevOps maturity model, the more likely you are to have bought into SRE practices.

The conversation isn’t ‘SRE versus DevOps; the conversation is about establishing a proactive SRE team and operating model to help strengthen your DevOps practice. Whether you choose an SRE framework integrated into your development and IT teams, or you want to make SRE a standalone business unit, you need to understand their general responsibilities.

SRE Team Responsibilities

Site reliability engineers are responsible for defining what performant and reliable really mean when you talk about your applications, services and infrastructure. The day-to-day tasks can range anywhere from instrumenting new monitoring solutions to building custom apps for technical support teams. They might ship new code to production to fix bugs or deliver new features, or they might respond to incidents in real-time and work closely with support teams to deliver positive customer experiences.

At the end of the day, site reliability engineers are the key to understanding exactly how customers experience your product and how to track system performance and reliability through your customers’ eyes. SRE teams need to figure out exactly where service boundaries exist between what development teams ship and what customers experience. From there, they need to find ways to monitor reliability and performance concerns in a way that helps your internal teams proactively identify risks and deliver better software.

SRE teams spread knowledge across the product and development teams to consistently define reliability across the entire organization. With everyone on the same page, the engineering teams can make data-driven decisions when it comes to releasing new features or improving current experiences in production.

Site Reliability Engineering Principles and Practices

At a very high level, Google defines the core of SRE principles and practices as an ability to ’embrace risk.’ Site reliability engineers balance the organizational need for constant innovation and delivery of new software with the reliability and performance of production environments.

The practice of SRE grows as the adoption of DevOps grows because they both help balance the sometimes opposing needs of the development and operations teams. Site reliability engineers inject processes into the CI/CD and software delivery workflows to improve performance and reliability but they will know when to sacrifice stability for speed. By working closely with DevOps teams to understand critical components of their applications and infrastructure, SREs can also learn the non-critical components.

Creating transparency across all teams about the health of their applications and systems can help site reliability engineers determine a level of risk they can feel comfortable with. The level of desired service availability and acceptable performance issues that you can reasonably allow will depend on the type of service you support as well. SRE principles and practices embrace experimentation and require a dedication to proactively understanding the health of the services they support.

An SRE Operating and Maturity Model

You can perform a lot of the responsibilities asked of a site reliability engineer and still have a software engineer job title. So, how do you know how mature your site reliability engineering practice is? Luckily for you, we’ll lay out a quick way to build an effective SRE operating model and track your maturity against it. An SRE operating model typically includes three elements, which you might achieve in phases:
1. A team (or at least one person) dedicated to the practice of SRE.
2. Deep integration and influence across product, development and operations teams.
3. Autonomy to automate workflows and write code for nearly any part of your application or system.
Your SRE maturity depends on where your organization falls along these three elements of an SRE operating model. If you’ve taken the steps to make an SRE team or hire your first site reliability engineer, you’re at the beginning of your journey. If you have a team and they are a valued part of roadmap discussions, QA, deployment workflows, incident management processes, then you have a somewhat mature SRE practice.

An organization only reaches complete SRE maturity when the SRE business unit has the autonomy to automate workflows, build applications, own monitoring and alerting solutions, or interject themselves into nearly any conversation. Vocalizing performance and reliability concerns upfront and having a proactive discussion about those concerns is always better than simply ignoring them until it’s too late.

Monitoring, CI/CD and Organizational Automation

Site reliability engineers can and will automate just about anything and everything. If it can proactively detect, remediate or resolve an issue, it needs to be automated. From continuous integration and delivery practices to production environment monitoring, SREs should have some visibility into all of it. If they can identify ways to proactively uncover performance and reliability issues, then they need to have the authority to implement those changes.

Today’s DevOps and IT capabilities around automation, monitoring, artificial intelligence and machine learning give SRE teams a huge advantage when identifying issues, responding to them and fixing them. Organizations with mature DevOps and SRE practices can catch problems in staging and they can also build automated incident management workflows and self-correcting systems. By determining critical components in your applications and infrastructure, SREs can narrow down the scope of things that can cause major problems.

The Practice of Service Levels (SLIs, SLOs and SLAs)

Service levels come into play to help SRE teams communicate the true health of digital products and services to all stakeholders. This is done by identifying and measuring critical components that are key to delivering positive customer experiences. In particular, they need to know when one or more components expose functionality to external customers. We call these intersection points system boundaries. System boundaries are the place where site reliability engineers need to apply service-level indicators and objectives to their metrics in order to tell the real story of system performance and reliability.
- Service-level indicators (SLIs) are the key measurements to determine the availability of a system.
- Service-level objectives (SLOs) are goals you set for how much availability you expect out of a system.
- Service-level agreements (SLAs) are the legal contracts that explain what happens if the system doesn’t meet its SLO.
While SREs aren’t always responsible for managing service levels, it often falls within their purview. By tracking SLIs and tying them to SLOs, you can set goals around the performance of a system. Google’s SRE book defines the four golden signals of service levels as latency, traffic, errors and saturation. So, for example, you could look at an API call and track its number of successful/failed requests (the SLI) against a general percentage of requests that need to be successful for customers to have a good experience (the SLO).

SRE teams will often set strict SLOs on critical components within their applications and services to better understand how strict of an SLA they can agree to with customers. From here, the team can apply error budgets as a way to understand how quickly they must resolve issues in order to stay compliant with their SLOs. Service levels allow teams to aggregate metrics and create a transparent view of uptime, performance, and reliability across the entire organization. At a glance, business leaders can use service levels to monitor compliance across multiple teams, applications, services, etc. to gain a comprehensive understanding of their system’s health.

Adopting SRE Best Practices

Adopting SRE best practices and principles won’t happen overnight. It takes time and effort to proactively monitor your teams and systems for performance and reliability concerns. But, in the end, your DevOps teams and especially your customers will thank you for deciding to take advantage of site reliability engineering.
June 14, 2022
The Pros and Cons of Embedded SREs

To embed or not to embed: That is the question.

At least, that’s one of the questions that companies have to answer as they decide how to implement site reliability engineering. They can either embed SREs into existing teams, or they can build a new, separate team.

Both approaches have their pros and cons. The right strategy for your company or team depends, of course, on your needs and priorities.

What is an Embedded SRE?

An embedded SRE works as part of a non-SRE team. They might join development or IT operations teams, for example—although they could also be part of quality assurance teams, security teams or any other unit within the organization that can benefit from their expertise.

Embedded SREs are the opposite of creating a dedicated, general-purpose SRE team. In the latter approach, an organization hires talent that collaborates with non-SRE teams but who aren’t integrated directly into them.

There are several other organizational models that fall somewhere between these two poles. But typically, the two main types of structures are embedded or dedicated teams.

Advantages of Embedded SREs

Embedding SREs into other teams offers several benefits.

Arguably the most important advantage is that embedded SREs ensure the highest level of collaboration between site reliability engineers and other stakeholders. When they work directly alongside other engineers on a continuous basis, it’s a pretty safe bet that their perspective will inform every decision that the other engineers make. Separate teams will still generally collaborate with other teams, but not necessarily with the same level of dedication as embedded SREs.

Embedded SREs are also beneficial for organizations where the site reliability engineering concept remains new, and where stakeholders still need to learn what SREs do and the value they can bring. It may be a bit challenging to get buy-in when you propose creating an entirely separate team; indeed, developers or IT engineers may even perceive such a team as a threat to their own jobs (even though in reality, of course, they complement other teams instead of competing with them). But when you add SREs to larger teams, it’s likely to be easier to get everyone on board with the concept and for other types of engineers to appreciate how SREs make their jobs easier.

Finally, embedded SREs work well for organizations that are on the small side and don’t need an entire team. If you have just several dozen engineers on staff instead of several hundred, hiring a few SREs to embed into your existing teams probably makes more sense than investing in a brand-new team.

The Challenges of Embedded SRE

The embedded model has its drawbacks, too.

One major risk is that the SRE’s influence within a larger team may not be strong enough to have a major impact on how that team operates. A single SRE working alongside a dozen developers or IT engineers, for example, may struggle to integrate site reliability engineering techniques and tools into the existing team’s workflow—especially if it’s a team that already has strong opinions about how it should operate.

Along similar lines, a single SRE working within a team using the embedded model could end up being stretched too thin to do his or her job well. That’s especially true if other team members expect them to “own” functions that should really be a collective, team responsibility—like responding to incidents or using chaos engineering to test reliability. The latter processes may be led by SREs, but they shouldn’t be their responsibility alone, and the function can start to break down when other team members think of them as the person on whom they can simply “dump” all of their reliability-related tasks.

Finally, the embedded model can be difficult to scale, which is one reason why it doesn’t work as well at large organizations. If you have dozens of different engineering teams, embedding an SRE into each one requires a lot of management effort. It also deprives the various SREs within your organization of the ability to work closely with each other. In that type of scenario, it probably makes more sense to create a dedicated team and have those SREs interface with other teams as needed, rather than trying to distribute site reliability engineers across the business by embedding them directly into other teams.

Conclusion

In general, then, embedded SREs make most sense for companies that have relatively small engineering organizations, and in which the concept of the function has not yet received complete buy-in. For large businesses that have numerous engineering teams, it will likely be easier to manage SREs by creating a separate team just for them.

That said, your mileage may vary from the norm, and it may be worth experimenting a bit to determine which model—embedded or not—makes the most sense for your company. Keep in mind that you can also use a hybrid approach wherein you embed SREs into some teams, while simultaneously maintaining a dedicated team that leads site reliability initiatives for the organization as a whole.

After all, site reliability engineering is all about thinking creatively and coming up with innovative solutions to challenges—even the challenge of figuring out how to structure the SREs within your organization.

April 21, 2022
The Evolution of Incident Management
Have you ever thought about the history of incident management?

If you’re an SRE, you might be so caught up in the day-to-day work of managing reliability and responding to incidents that you never take time to step back and reflect on the evolution of your role and your responsibilities. And that’s a shame because SREs didn’t invent incident management concepts and strategies on their own.

On the contrary, the way SREs think about incident response, structure incident management teams and rank the priority of incidents owes much to incident management strategies developed in the offline world decades ago. To understand fully what it means to be an SRE today, you have to appreciate this deep history of incident management and response.

So, let’s take a look at that history and examine where modern incident response concepts originated.

Historical Problems in Incident Management

Societies have always had incidents, of course. Fires, floods, infrastructure breakdowns and similar crises have been happening for millennia.

For most of history, however, humans lacked an efficient, purposeful way to manage these sorts of incidents. Response efforts were ad hoc, and their effectiveness owed more than a little of their success to sheer luck.

Particular challenges included:
- Lack of effective, consistent communication between stakeholders.
- Varying organizational structures which made it hard to identify leaders, coordinate response efforts and delegate tasks.
- Inconsistent response strategies.
- Differing approaches to assessing the priority of incidents.
Historically, organizations may have been able to handle incidents well enough if the incidents required response from only one, small group. But the more stakeholders involved, the harder it was to respond quickly and effectively.

Putting Out Fires: The Birth of the ICS

Things began to change for the better when stakeholders started thinking about better ways to put out fires—literally.

By the 1960s, fire chiefs in California realized that they were struggling to respond effectively to the wildfires that broke out every summer. Each year brought worse blazes than the last, with more land burned and more buildings lost. The Laguna fire of 1970 brought matters to a head and was the catalyst for a new approach to incident response for fire agencies.

After assessing what was going wrong, the fire chiefs determined that it wasn’t a lack of equipment or personnel. It was poor coordination among the various firefighting agencies that responded to blazes. Lacking a clear chain of command and a systematic approach to firefighting, the agencies struggled to deploy their resources rapidly and, thus, effectively.

To fix the problem, California fire chiefs developed what became known as the Incident Command System, or ICS. The ICS defined a hierarchy for incident response with an incident commander at the top. It also defined several categories of incident response processes including operations, planning, logistics and finance. And it established a consistent set of terms that stakeholders can use to describe their actions during incident response, which makes it easier to communicate clearly.

Although the ICS was initially conceived to fight fires, it became the de facto standard for organizing incident response strategies of all types.

From ICS to NIMS

The history of incident response doesn’t end with the ICS. A new chapter began in the early 2000s when the U.S. federal government developed an even more comprehensive approach to incident management called the National Incident Management System, or NIMS.

NIMS was born in the wake of the September 11, 2001, terrorist attacks, which underlined the importance of efficient communication not just between different agencies of the same type (like fire departments), but of entirely separate organizations. To achieve this, NIMS expanded upon the principles of the ICS.

In addition to adopting most of the incident command principles and practices included in the ICS, NIMS included standards for coordinating the distribution of resources. It also embraced the concept of the emergency operations center, which is in some ways similar to a network operations center in the digital world.

In some respects, NIMS resembled a compliance framework (although to be clear, that’s not what it is). It includes fourteen management principles, similar to compliance controls, that organizations must implement to manage incidents using a NIMS approach.

Incident Management Today

Obviously, putting out forest fires and responding to terrorist attacks is pretty different from dealing with data center failures or a buggy application deployment. ICS and NIMS weren’t designed for site reliability engineering or IT teams specifically.

Still, the influence of ICS and NIMS on the way SREs think is clear enough. Terminology like “incident commander” comes from these frameworks. So do concepts like shared accountability for incident response processes and the importance of involving all stakeholders—not just technical teams—in incident response.

ICS and NIMS may not be acronyms familiar to most SREs. But they should be, because they are the historical sources of the incident management philosophies that form the foundation for SRE work today and offer valuable lessons for any SRE on the job today.
February 8, 2022
Site Reliability Engineering (SRE) Comes of Age in 2022

The site reliability engineer (SRE) role is still gathering steam across organizations. In January 2022, LinkedIn listed SRE as the 21st job with the highest global demand throughout the past five years. That’s pretty high for such a specific tech role. And, looking to the future, it appears the SRE practice will only continue to gain adoption as a method to support high availability, reliability and improved digital customer experiences. The SRE approach is also essential to meet service-level agreements (SLAs) and internal service-level objectives (SLOs).

I recently met with Kurt Anderson, SRE architect, and Emily Arnott, content writer from Blameless to get their take on the top trends and predictions for the SRE practice in 2022. According to the Blameless team, 2022 will likely see increased adoption of the SRE role throughout companies and across internal departments.

Just as manufacturing sets high standards for its quality assurance, digital applications now require the same high reliability to meet inflated user expectations. To address these new realities, SREs have their work cut out for them—they must confront how to unify disparate DevOps tools and must drill farther down to unearth specific user experience qualms—all without adding to alert fatigue.

Increasing SRE Adoption

As mentioned above, the SRE role has seen tremendous growth in recent years and doesn’t show signs of slowing. SRE as a practice was first popularized by Google in 2015. Today, they define it simply as “what happens when you ask a software engineer to solve an operational problem.” This equates to maintaining high levels of reliability for digital services by creating errors budgets, designing SLAs/SLOs and optimizing process automation over time.

To Arnott, we’re still seeing SRE adoption take off at an exponential rate as more and more companies embrace Google’s SRE doctrine. “During the pandemic, more and more companies were forced to adopt a digital strategy. Reliability has to be fundamental to that strategy.” Arnott foresees SRE becoming a fruitful role at startups, which may initially have been intimidated by the prospect.

“The practice of SRE is a scale issue,” Anderson explained. It can be difficult for time-pressed companies to enact an SRE program when their attention is already so divided. However, as new requirements force warp-speed innovation, we’re seeing greater DevOps adoption. And the SRE role, consequentially, is becoming more of an imperative.

More Unification of Multi-Tool Condition

Most DevOps professionals would agree—we live in a multi-tool world. For example, multi-cluster management concerns are bringing a plethora of new tools to the fold. And computing environments are just as diverse, often combining multiple clouds and composed of hybrid estates that house various clusters of microservices. Even within the same organizations, different teams may opt for differing DevOps tooling. Anderson believes that SREs can bring “more unification to a fractured DevOps adoption across a single company.”

As companies adopt more software, there’s inevitably some degree of tool sprawl, Anderson said. For example, there may be competing methods to monitor the use of digital services. The platform team may be using Datadog, an app team may be using New Relic and another team may be using Prometheus. These tools don’t speak the same language and generate different logs, creating silos.

For Arnott, the future involves accepting the innate complexity of the microservices era and using an agnostic layer that can receive data from anywhere to identify meaningful incidents.

Observability Gets a Bigger UX Emphasis

The SRE role is definitely a trend that has been growing and will continue to grow in importance, Anderson said. And as more organizations turn digital, it’s becoming necessary; with this shift comes enormous pressure to exceed user experience expectations. “Competition and people’s expectations are so high right now,” said Arnott. “If you are unreliable, it’s unacceptable these days.”

Whereas DevOps focuses more on getting applications into production, SRE takes a more UX-focused approach. Take Netflix, for example, which has invested heavily in a core SRE team that ensures robust and reliable user-facing services. “A key underpinning to keeping our customers happy and streaming is a strong focus on reliability,” according to the Netflix Technology Blog.

To cater to this new paradigm, teams will require a deeper understanding of observability based on particular cohorts of users, said Arnott. SREs require precise user experience contexts, such as an uptick of lags or increased error responses. But they also must discover the contributing factors behind these bugs. “SLOs and error budgets are like taking your temperature—they tell you if something is wrong. They don’t tell you why you have a fever.”

SRE Takes Cues From Manufacturing

As reliability becomes more and more central to a company’s brand presence, it’s only natural that something that important would be elevated into the executive suite, Anderson explained. As a reference example, he pointed to the fact that manufacturing industries often appoint a chief reliability officer. In traditional manufacturing, product lines undergo a statistical process control with rigorous testing to ensure quality. Analogous positions exist across industries, such as chief quality officers in financial services.

The takeaway is that as reliability becomes more of defining competitive metric, tech companies may require a similarly high-level C-suite position to align on that idea and to direct teams to implement SRE practices. Of course, the necessity for such a title will entirely depend on organizational size and makeup.

Greater Reliance on SLOs to Avoid Broken SLAs

A service-level agreement (SLA) is typically a formal contract built around the reliability of a digital service with an external partner, which, if breached, may result in financial penalties or other forms of restitution. This agreement could exist between a software provider and an end user but is more strictly enforced in a B2B context.

On the other hand, a service-level objective (SLO) is more of an internal target. SLOs are often set at a lower threshold than SLAs. For example, perhaps an SLA ensures a user login fails no more than 10 out of one million attempts, but the SLO is set at two out of one million.

Setting internal SLOs helps keep engineers abreast of modest customer experience issues before they become a major incident that results in fines. “Some companies treat SLOs as just another alert; they should be representative of user experience,” said Anderson. It makes sense that as SRE initiatives begin to mature, companies will slowly start to adopt even more competitive goals to deliver superior user experiences and avoid unsatisfied partners.

Final Thought: Don’t Play The Blame Game

A big part of the SRE doctrine is the concept of blameless postmortems. Because when you’re in the middle of an incident, the worst thing to do is point fingers. “Blaming triggers a defense mechanism and when you’re in a defensive mode, you can’t respond well,” said Anderson.

Instead, it’s better to keep an open mind when performing these retrospectives—accept that what occurred happened for a reason, and seek to discover that reason to make positive improvements. “What can we change at a systemic level to become more resilient?” inquires Arnott. “What is the origin of that mistake? What information was lacking? What communication should have happened to prevent this?”

Typically, accidents don’t occur intentionally or due to incompetence but rather due to holes in the process. By not playing the blame game, companies are better positioned to fix issues as they progressively release new features. Then, hopefully, with the right SRE approach in place, teams can advance their digital services to meet the high stakes of today’s modern user experience expectations.

January 25, 2022