devopsy.kinsta.cloud

Tag: SRE

5 Mean-Time Reliability Metrics To Follow

Most folks working in DevOps or SRE roles are familiar with metrics like mean-time-to-recovery (MTTR). Keeping track of the average time a team takes to respond to incidents is crucial to identifying bottlenecks in the support process. It’s also something executives like to show higher-ups when sharing a snapshot of overall platform performance. However, focusing on one single metric might be missing the greater picture.

For example, how long did it take to discover the incident? How long did it take from discovery until action was taken? What was the timeframe between filing a ticket and updating all clients that had a bug? When can you say you’ve completely resolved an issue? As you can see, there are many potential metrics that could inform software reliability and the platform engineering process. “One number is never going to tell you a complete story,” said Emily Arnott, community relations manager, Blameless.

I recently met with the Blameless team for a closer look into mean-time reliability metrics. Below, we’ll explore the nuances behind five different types of MTTX metrics and consider the business value of keeping tabs on each type.

1. Mean-Time-To-Detect

Mean-time-to-detect is a measurement of how long it takes, on average, to detect that an incident is present. This often happens automatically within a monitoring system. Perhaps a tool sends out an alert when latency meets a certain threshold, for example. But, detection could also come from other sources, such as a customer complaint.

For example, consider Log4j. Perhaps a runtime vulnerability scanning tool noticed one of your components is impacted by a novel CVE and automatically sent a notification to the appropriate team channel.

2. Mean-Time-To-Acknowledge

Now, just because an incident is detected doesn’t mean it’s immediately acknowledged. Mean-time-to-acknowledge, then, is a measurement of how long it takes a human being to realize the incident and begin to act on it.

In our CVE example, this would be the time between vulnerability detection and initial response. For example, the on-call incident manager received a vulnerability notification, read the exposure details and then filed a JIRA ticket and contacted the relevant team members.

Many factors might extend mean-time-to-acknowledge. Perhaps someone isn’t logged into Slack, or a hardware issue stalls the notification. Alert fatigue may also get in the way, or there may be a reluctance to file a ticket and introduce more toil. Depending on the severity of the incident, someone simply might not believe the alert is worth responding to.

3. Mean-Time-To-Recover

Now, the following metrics are a little more open to interpretation, but we’ll do our best to define each. Mean-time-to-recover is the average time it takes to introduce a temporary fix after discovering an incident.

For example, if a particular region is experiencing outages, engineers might temporarily divert traffic to a more stable server. This is not a permanent fix, but the system recovers and operations are generally unaffected. This helps maintain the status quo while a more permanent solution is ideated, tested and applied.

4. Mean-Time-To-Repair

Mean-time-to-repair (or restore) is the mean time it takes to issue a permanent repair to a system after discovering an incident. In order for a system to be considered fully repaired, it has to not just be working, but working robustly.

For example, let’s say the incident in question involves performance issues. Perhaps a patch is issued to the core branch to remove bulky code that’s causing slow load times for clients. Mean-time-to-repair introduces a more permanent solution but still might not be the complete resolution to the problem.

5. Mean-Time-To-Resolve

Mean-time-to-resolve can be thought of as the average time between when an incident occurs to when the issue is completely resolved. Not only is the core codebase patched, but all clients reliant upon the software have been updated, as well. Lessons are learned and mitigation plans are set to respond to similar incidents or vulnerabilities in the future.

Mean-time-to-resolve is about resolving the incident entirely, said Jake Englund, senior site reliability engineer, Blameless. This includes addressing the underlying fundamental contributors, completing all logs that remain on the back burner and following up with a retrospective.

Using Mean-Time Metrics

Mean-time metrics can provide a quantitative picture of incident response performance, which can be valuable for overall business operations. Mean-time-to-acknowledge, for example, can expose gaps in the remediation process, such as cognitive strife in reporting incidents, said Matt Davis, intuition engineer, Blameless. Understanding these technical and human factors are the first step to making the resolution process swifter.

Of course, the above metrics rely on incident data, which may not always be a top priority. According to Davis, encouraging a culture that declares incidents—even minor ones such as a configuration change—can improve knowledge sharing within a team. “If you declare an incident, you could enact more systemic change,” added Arnott.

Limitations of Mean-Time Metrics

These mean-time metrics do have some limitations. “A number is only one part of the story,” said Davis. As a result, teams might struggle with deciding precisely what to detect. MTTR can be a helpful metric, but it’s the context that matters, he explained. Therefore, tracking multiple metrics can help provide a more sophisticated, nuanced picture. This involves looking beyond averages to consider outlier events, added Englund.

There are also semantic nuances between the MTTX metrics defined above. “There’s a lot of ambiguity around these words,” said Davis. As a result, organizations might compute each figure differently. Some of these figures use time-markers which may not be technically possible to track consistently, especially since each incident is unique. Demarcating the precise moment an incident began might require guestimation. You might know when a service is fully restored, but the lasting customer perception is harder to gauge.

Also, another potential downside is that mean-time metrics could easily be manipulated or misinterpreted, whether deliberately or inadvertently. Operations leads might selectively recall specific windows when showing MTTX metrics to higher-ups, leaving out other statistics that paint a different picture.

Treat MTTX As A Guidepost

Improving incident response is becoming mission-critical to maintain fully-functional systems as things like outages, downtime, slow speeds and zero-day vulnerabilities can negatively impact user experience. Sometimes, these issues must be addressed immediately to maintain SLAs.

But tracking reliability averages isn’t all that simple, and they will likely mean something different to each organization. “It’s about embracing complexity and asking the right questions,” said Davis.

In summary, mean-time reliability metrics provide helpful insight into the ongoing state of incident response. Yet, such metrics shouldn’t be imposed as a strict target — instead, they should be viewed as an informative guidepost. “Metrics can help you find what is discussion-worthy, but it’s not a discussion itself,” said Arnott.

July 7, 2022
How to Adopt an SRE Practice (When You’re not Google)
Site reliability engineering (SRE) isn’t a new term or practice. The practice of applying software engineering skills and principles to operations problems and tasks happened even before site reliability engineer was a defined job title. But organizing a proactive approach to building and maintaining software drives long-term success in improving operational efficiency, data-driven roadmap planning and general uptime and reliability. All these advantages account for the widespread adoption of SRE.

In this article, I’ll dive into what it takes to get into site reliability engineering, how to adopt it within your own organization and some of the core principles and best practices you’ll need to keep in mind as you move forward in your SRE maturity journey.

What is SRE?

SRE, or site reliability engineering, is the practice of applying software engineering expertise to DevOps and operations problems. Often, this means proactively writing code and developing internal applications or services to combat reliability and performance concerns. SRE has been a practice for many years but has been more recently popularized by Google when they first published Site Reliability Engineering: How Google Runs Production Systems in March 2016.

SRE is a complicated acronym because it both relates to a job title, site reliability engineer, as well as a general practice adopted by development and IT teams, site reliability engineering. SRE teams are often organized differently depending on the organization and the services they support. Sometimes site reliability engineers (SREs) are sprinkled among development teams. This means they can vocalize concerns during roadmap planning and work closely with QA teams, collaborating on everything from shipping features to staging to managing production environments.

Another approach organizes an SRE team as a standalone group. The SRE team as a whole is focused on responsibilities including monitoring applications and infrastructure, establishing reliability metrics, tracking issues with new releases and handling on-call duties. No matter how they are organized, what remains the same for all these SREs is their initiative—reliability and performance.

DevOps Vs. SRE

The rapid adoption of DevOps practices may have you wondering, “How is SRE different from DevOps?” Commonly, the two terms go hand-in-hand. SRE is a practice that doesn’t necessarily have to be part of DevOps culture and adoption but is often implemented by organizations that are further along in their DevOps journey. The further along on your DevOps maturity model, the more likely you are to have bought into SRE practices.

The conversation isn’t ‘SRE versus DevOps; the conversation is about establishing a proactive SRE team and operating model to help strengthen your DevOps practice. Whether you choose an SRE framework integrated into your development and IT teams, or you want to make SRE a standalone business unit, you need to understand their general responsibilities.

SRE Team Responsibilities

Site reliability engineers are responsible for defining what performant and reliable really mean when you talk about your applications, services and infrastructure. The day-to-day tasks can range anywhere from instrumenting new monitoring solutions to building custom apps for technical support teams. They might ship new code to production to fix bugs or deliver new features, or they might respond to incidents in real-time and work closely with support teams to deliver positive customer experiences.

At the end of the day, site reliability engineers are the key to understanding exactly how customers experience your product and how to track system performance and reliability through your customers’ eyes. SRE teams need to figure out exactly where service boundaries exist between what development teams ship and what customers experience. From there, they need to find ways to monitor reliability and performance concerns in a way that helps your internal teams proactively identify risks and deliver better software.

SRE teams spread knowledge across the product and development teams to consistently define reliability across the entire organization. With everyone on the same page, the engineering teams can make data-driven decisions when it comes to releasing new features or improving current experiences in production.

Site Reliability Engineering Principles and Practices

At a very high level, Google defines the core of SRE principles and practices as an ability to ’embrace risk.’ Site reliability engineers balance the organizational need for constant innovation and delivery of new software with the reliability and performance of production environments.

The practice of SRE grows as the adoption of DevOps grows because they both help balance the sometimes opposing needs of the development and operations teams. Site reliability engineers inject processes into the CI/CD and software delivery workflows to improve performance and reliability but they will know when to sacrifice stability for speed. By working closely with DevOps teams to understand critical components of their applications and infrastructure, SREs can also learn the non-critical components.

Creating transparency across all teams about the health of their applications and systems can help site reliability engineers determine a level of risk they can feel comfortable with. The level of desired service availability and acceptable performance issues that you can reasonably allow will depend on the type of service you support as well. SRE principles and practices embrace experimentation and require a dedication to proactively understanding the health of the services they support.

An SRE Operating and Maturity Model

You can perform a lot of the responsibilities asked of a site reliability engineer and still have a software engineer job title. So, how do you know how mature your site reliability engineering practice is? Luckily for you, we’ll lay out a quick way to build an effective SRE operating model and track your maturity against it. An SRE operating model typically includes three elements, which you might achieve in phases:
1. A team (or at least one person) dedicated to the practice of SRE.
2. Deep integration and influence across product, development and operations teams.
3. Autonomy to automate workflows and write code for nearly any part of your application or system.
Your SRE maturity depends on where your organization falls along these three elements of an SRE operating model. If you’ve taken the steps to make an SRE team or hire your first site reliability engineer, you’re at the beginning of your journey. If you have a team and they are a valued part of roadmap discussions, QA, deployment workflows, incident management processes, then you have a somewhat mature SRE practice.

An organization only reaches complete SRE maturity when the SRE business unit has the autonomy to automate workflows, build applications, own monitoring and alerting solutions, or interject themselves into nearly any conversation. Vocalizing performance and reliability concerns upfront and having a proactive discussion about those concerns is always better than simply ignoring them until it’s too late.

Monitoring, CI/CD and Organizational Automation

Site reliability engineers can and will automate just about anything and everything. If it can proactively detect, remediate or resolve an issue, it needs to be automated. From continuous integration and delivery practices to production environment monitoring, SREs should have some visibility into all of it. If they can identify ways to proactively uncover performance and reliability issues, then they need to have the authority to implement those changes.

Today’s DevOps and IT capabilities around automation, monitoring, artificial intelligence and machine learning give SRE teams a huge advantage when identifying issues, responding to them and fixing them. Organizations with mature DevOps and SRE practices can catch problems in staging and they can also build automated incident management workflows and self-correcting systems. By determining critical components in your applications and infrastructure, SREs can narrow down the scope of things that can cause major problems.

The Practice of Service Levels (SLIs, SLOs and SLAs)

Service levels come into play to help SRE teams communicate the true health of digital products and services to all stakeholders. This is done by identifying and measuring critical components that are key to delivering positive customer experiences. In particular, they need to know when one or more components expose functionality to external customers. We call these intersection points system boundaries. System boundaries are the place where site reliability engineers need to apply service-level indicators and objectives to their metrics in order to tell the real story of system performance and reliability.
- Service-level indicators (SLIs) are the key measurements to determine the availability of a system.
- Service-level objectives (SLOs) are goals you set for how much availability you expect out of a system.
- Service-level agreements (SLAs) are the legal contracts that explain what happens if the system doesn’t meet its SLO.
While SREs aren’t always responsible for managing service levels, it often falls within their purview. By tracking SLIs and tying them to SLOs, you can set goals around the performance of a system. Google’s SRE book defines the four golden signals of service levels as latency, traffic, errors and saturation. So, for example, you could look at an API call and track its number of successful/failed requests (the SLI) against a general percentage of requests that need to be successful for customers to have a good experience (the SLO).

SRE teams will often set strict SLOs on critical components within their applications and services to better understand how strict of an SLA they can agree to with customers. From here, the team can apply error budgets as a way to understand how quickly they must resolve issues in order to stay compliant with their SLOs. Service levels allow teams to aggregate metrics and create a transparent view of uptime, performance, and reliability across the entire organization. At a glance, business leaders can use service levels to monitor compliance across multiple teams, applications, services, etc. to gain a comprehensive understanding of their system’s health.

Adopting SRE Best Practices

Adopting SRE best practices and principles won’t happen overnight. It takes time and effort to proactively monitor your teams and systems for performance and reliability concerns. But, in the end, your DevOps teams and especially your customers will thank you for deciding to take advantage of site reliability engineering.
June 14, 2022
5 Tips If You’re the First SRE Hire

Site reliability engineers (SREs) have a considerable set of tasks to juggle no matter where they work or how long their company has had an SRE practice. But if you’re the very first one to join an organization—as many SREs are these days, given that the trend is trickling down into smaller and smaller companies—you face a special group of challenges.

You may find it difficult to get buy-in for SRE from other technical teams. You may struggle to know how to convince your boss that the company’s decision to hire an SRE is paying off. You probably worry that you’ll end up being overextended because you are the only SRE. And so on.

These five tips are for site reliability engineers who find themselves in the position of being the pioneer at their companies. Whether you’re an experienced SRE who has just been hired at a company that had none previously or you’re brand-new to SRE and your first job happens to be at a company that is also brand-new to the SRE role, the following best practices will help you make the transition as smooth as possible for all involved.

Collect SRE Metrics Early and Often

For starters, you’ll want to be prepared to justify the business’s investment in your new role as soon as a boss or fellow engineer asks.

Do this by collecting metrics about the technical and business impact of SREs starting as early as you can. Your goal should be to quantify how you are benefitting the organization so that you can encourage more support for yourself and any others who may be hired after you.

On top of this, collecting metrics is also important for establishing early feedback about which SRE strategies are succeeding and which could be improved. Since you’re building an SRE practice from scratch, getting this insight early and often is critical.

Treat Devs and IT Engineers Equally

As you start the role, you may find some uncertainty among existing engineers about whose ‘side’ you are on. Are you there to support developers or IT engineers?

The answer, of course, is both. Although SREs tend to borrow from software engineers’ playbooks more than they do from those of IT Ops, an SRE’s goal should be to empower and support both of these teams. So, unless you’re given explicit instructions to interface with only one group or the other, make yourself everybody’s buddy within the IT organization.

Introduce Yourself to Non-Technical Teams

SREs spend most of their time doing technical things with technical tools. But ultimately, the only reason they are doing what they do, in most cases, is to support non-technical users by improving the reliability of the systems they use.

As the first SRE at a company, it may be well worth your time to get to know these non-technical stakeholders and ask them what they see as the greatest reliability issues in the organization. Building these relationships early on will help you communicate the value of your role across the organization. It’s also likely to come in handy during incident management routines, when you may need to interface with non-technical stakeholders as well as other engineers.

Build Your SRE Tool Set

Deciding which tools to use is an obvious early step for any SRE who is building a practice from scratch. But we didn’t put it first on this list because it’s important to get to know the rest of the organization before committing to a particular set of tools.

As you meet the rest of the organization, ask detailed questions about which tools they use to address needs like observability, security and incident management. Then, by factoring in feedback about which tools work well and which don’t, choose which tools you’ll deploy to support SRE. Be prepared for those tools not to match your own personal preferences perfectly. You may need to prioritize the preferences of the rest of the organizations.

Plan to Grow the Team

Unless you have been hired at a very small company, it’s likely that the business plans to hire more SREs over time (assuming that you prove the value of the position, of course).

Toward that end, plan for how the team will expand. Think about things like whether to use embedded SREs or create a dedicated SRE team. Think, too, about how you’ll document your company’s SRE practice and ensure it keeps operating smoothly as individual engineers come and go.

Conclusion

The job of a newly hired SRE is not an easy one, especially if you’re the first person at your company to fill those shoes. But by being deliberate and strategic about how you work, which data you collect and how you introduce the company to SRE, you set yourself – and your future colleagues – up for success.

June 13, 2022
Survey Surfaces Challenges Ahead on National DevOps Day

A survey published today for National DevOps Day found nearly two-thirds (63%) have seen an increase in the frequency of service incidents that have affected their customers over the course of the last 14 months.

The survey polled 1,046 engineering, IT operations, DevOps and site reliability engineering (SRE) professionals at organizations with more than 300 employees. The research was conducted by Transposit, a provider of a platform for connecting DevOps workflows, and found well over half (58%) of respondents reported the cost of downtime for their organization could be, on average, up to $499,999 per hour, with 40% of respondents noting that the cost of downtime has increased in the last year.

Respondents said the top contributing factors to the increase in service incidents were digital transformation (61%), new product rollouts or product updates (55%) and methods and tools for collaboration that do not adequately support their team working remotely (49%). A full 90% of respondents reported there has been an increased focus on digital transformation during the last year with 73% having expanded the technology stack they support. More than half (55%) have increased the size of their DevOps teams.

Overall, 46% of organizations reported they have experienced between six and 19 major incidents over the course of the last 12 months. More than half (53%) also reported an increase in the amount of time it takes to resolve incidents over the course of the last year, even though three-quarters of respondents (75%) said their organization increased their adoption of SRE practices in the last 12 months. Additionally, 65.1% of those organizations planned to hire site reliability engineers in the next 12 months. However, only just over a third of those respondents said their organization planned to expand their SRE efforts in 2022.

A full 83% said up to 50% of their engineering operations processes are automated, but over half of the respondents that use SRE practices noted they still manually enter data into an IT service management (ITSM) application or other system-of-record to keep track of actions that were taken by humans during the resolution of an incident. Nearly two-thirds said their approach to automation was incremental, in which they begin by codifying processes and worked up to more advanced, fully automated scenarios.

The top three tasks respondents would like to automate were service requests (53%), change requests (43%) and user provisioning (40%).The top barriers to implementing automation were cited as inadequate documentation of institutional knowledge and existing processes (56%), lack of clarity about what to automate (55%) and an inability to share knowledge (52%).

Investments to improve their incident management processes over the next 12 months are focused on automation tools or applications (48%), communications/collaboration tools or applications (42%) and integration tools or applications (41%). Only a quarter (25%) said their tools are integrated through one tool or platform.

Lack of unified communication with teammates (45%), processes that have changed or are harder to follow while working remotely (42%) and lack of visibility into dependencies and what teams or people are responsible for the code or infrastructure (38%) are all cited as reasons why serviced requests are now more complex.

Transposit CEO Divanny Lamas said despite all the challenges facing DevOps teams, it’s fortunate that advances in low-code automation are making it simpler to more widely implement DevOps best practices. The challenge is finding a way to implement those best practices faster at a time when IT environments are becoming more complex.

May 24, 2022
What SREs Can Learn From the Atlassian Outage of 2022
What happens when the tools and services you depend on to drive site reliability engineering turns out to be susceptible to reliability failures of their own? That’s the question teams at about 400 businesses presumably asked themselves in the wake of a major outage in Atlassian Cloud. The incident offers a number of insights for SREs about reliability risks within reliability management software itself—as well as how to work through complex outages efficiently and transparently.

What Caused the Atlassian Cloud Outage?

The outage, which began on April 4 and finally resolved April 18, affected about 400 Atlassian Cloud customer accounts. Atlassian Cloud is a hosted suite of popular Atlassian products including Jira and OpsGenie. The outage meant that affected customers could no longer access these tools or the data they managed in them.

According to Atlassian, the problem was triggered by a faulty application migration process. Engineers wrote a script to deactivate an obsolete version of an application. However, due to what Atlassian called a “communication gap” between teams, the script was written in such a way that it deactivated all Atlassian Cloud products, not just the obsolete application.

To make matters worse, the script was apparently configured to delete data permanently, rather than mark it for deletion, which was the intention. As a result, data in affected accounts was removed permanently from production environments.

The Good, the Bad and the Ugly

The Atlassian Cloud outage may not be the very worst type of incident imaginable—failures like Facebook’s 2021 outage were arguably worse because they affected more people and because service restoration was complicated by physical access issues—but it was still pretty bad. Production data was permanently deleted and hundreds of enterprise customers experienced total service disruptions that have lasted several days and counting.

Given the seriousness of the incident, it’s tempting to point fingers at Atlassian engineers for letting an incident like this happen in the first place. They seem to have written a script with some serious issues, then presumably deployed it without testing it first—which is exactly the opposite of what you might call an SRE best practice.

On the other hand, Atlassian deserves credit for responding to the incident efficiently and transparently. Although the company was silent at first, it ultimately shared details about what happened and why even though those details were a bit embarrassing to its engineers.
Crucially, Atlassian also had backups and failover environments in place, which it used to speed the recovery process. The major reason why the outage lasted so long, the company said, is that restoring data from backups to production requires integrating backup data for individual customers into storage that is shared by multiple customers, a tedious process that Atlassian apparently can’t perform automatically (or doesn’t want to, presumably because it would be too risky to automate).

Unfortunately for impacted customers, it does not appear that any fallback tools or services were made available while they waited for Atlassian to restore operations. We imagine this poses more than minor issues for teams that rely on tools like Jira to manage projects and OpsGenie to handle incidents. Perhaps those teams stood up alternative tools in the meantime – or perhaps they crossed their fingers, hoping their project and reliability management tools would come back online ASAP.

Takeaways for SREs From the Atlassian Outage

For SREs, then, the key takeaways from this incident would seem to be:

Always perform dry runs of migration processes in testing environments before putting them into production. Presumably, if Atlassian engineers had tested their application migration script first, they would have noticed its flaws before it took out live customer environments.

Back up, back up, back up—and make sure you have failover environments where you can rebuild failed services based on backups. While this outage is bad, it would be 100 times worse if Atlassian couldn’t restore service based on backups and data had been lost permanently.

Ideally, each customer’s data should be stored separately. As we noted above, the fact that Atlassian used shared storage seems to have been a factor in delaying recovery. That said, it’s hard to fault Atlassian too much on this point; it’s not always practical to isolate data for each user due to the cost and administrative complexity of doing so.

SRE teams would do well to think about how they’ll respond if their reliability management software itself goes offline. For example, it might be worth extracting and backing up data from your reliability management tools so you can still access it if your tool provider experiences an incident like this.

Over-communicate with your customers often and early. In this case, there was initial radio silence that left customers in the dark. Most of this chatter eventually took place in public forums.
Conclusion

The Atlassian Cloud outage is notable both for its length and for the fact that, ironically, it took out software that teams use to help prevent these types of issues from happening at their own businesses.

The good news is that Atlassian had the necessary resources in place to restore service as quickly as possible. A shared data storage architecture led to slow recovery, which is unfortunate; again, it’s hard to blame Atlassian too much for not setting up dedicated storage for each customer.
May 19, 2022
Increasing Use of SLOs to Enable Observability
Observability is a growing discipline among most IT and operations departments. To release stable software faster, operators need continuous visibility into metrics like performance, uptime and availability. As a result, engineers are increasing their use of service-level objectives (SLOs) across the board—a recent study found that 82% of companies are increasing their use of SLOs.

SLOs enable deep visibility into specific applications’ performance and are often used by site reliability engineers (SREs) to ensure quality and avoid service interruptions. Not only that, but various teams are correlating SLOs to aspects of the business to reduce cost and help direct decision-making. But while many environments have visibility, there are still gaps present.

Nobl9 recently released a global survey of IT professionals and executives that tracked the state of service-level objectives in 2022. Below, we’ll look at the key findings from the report and consider what they mean for the state of SLOs and observability in general.

State of Observability and SRE

In general, site reliability engineering is still maturing across organizations. Though only 31% of companies have adopted SRE, it’s slated for much future growth, as 46% said they plan to embrace SRE in the future.

These operators are now faced with many cloud-native observability tools, which are producing a sea of data in the form of metrics, logs and traces. A full 39% of companies use anywhere from six to 10 observability and monitoring tools and 35% use more than 11.

With so many tools now deployed, who is using this observability and monitoring data? Namely, at 74% of companies, observability data support operational requirements. Operations teams are the ones most likely to use SLOs to monitor uptime, performance and overall efficiency. After operations teams, security teams also use this observability data (71%), which makes sense as SLOs can inform incident response. Other areas that follow are customer support, compliance and capacity planning.

One interesting finding is that only 42% of companies use SLOs for service-level agreement (SLA) adherence. This indicates that SLOs are most often applied to internal optimizations and decision-making.

Hybrid Environments Complicate Visibility

Companies are mostly tracking SLOs to increase their visibility into networks (83%), databases (76%) and applications (75%). Other top areas include private cloud environments and legacy computing arrangements. But although observability is trending, organizations still lack complete visibility across the entire stack.

Of note is that nearly half (46%) of respondents said their monitoring and observability tools don’t provide full visibility into all of their company’s IT assets. For example, only 45% of companies have visibility into their containers and just 35% have visibility into their microservices architecture.

A lack of full-stack visibility may be complicated due to rising hybrid and multi-cloud conditions, as 78% say a hybrid cloud environment makes monitoring infrastructure more difficult.

Benefits of Tracking SLOs

The report defined SLO as “a performance and availability target set for a given system, application [or] service over time.” And the benefits are readily apparent for organizations following such targets—they can help increase performances, direct decision-making and help avoid outages.

As a result, 70% of companies are currently using SLOs in some fashion. Here are some benefits of tracking SLOs:
- Increase microservice performance: 87% said using SLOs for microservices architecture would improve service performance.
- Enable full-stack observability: 58% said some of their company SLOs are mapped to business operations.
- Improve business decision-making: 91% agreed using SLOs can help drive better business decisions.
- Prevent service interruption: 67% said their company prevented business interruptions thanks to SLO thresholds alerts.
- Reduce expenses: 90% indicated SLOs saved their company money.
Final Thoughts

More teams are tracking service-level objectives than ever before. And most companies (71%) that aren’t using SLOs now plan on adopting them soon. The data demonstrated that the observability market is an evolving area with room to grow. And, the same can be said about the SRE role.

Yet, it’s good to point out that not all companies will adopt precisely the same roles or monitoring procedures. Therefore, there will likely continue to be variance in who is using SLOs and how they are applied. Regardless, the study suggested that investigating SLOs has the potential to benefit the software life cycle in many ways.

The study, “Adoption of SLOs Grow to Increase Visibility and Drive Business Improvements” surveyed 309 participants around the globe in various sectors. The study was sponsored by Nobl9 and conducted by Dimensional Research. For more information, you can download the report here.
May 17, 2022
Shedding Light On Toil: Ways Engineers Can Reduce Toil

It comes in many shapes and sizes and is embraced as a necessary evil. It lurks in the shadows, emerging now and again to stealthily creep into our workflows, where it feasts on our perceived shame. If not pruned, it grows and wraps its tendrils around the workforce, draining the energy out of every digital worker across departments. We all have it and most of us believe we suffer alone.

I’m talking, of course, about toil.

Dramatics aside, toil is common throughout most programming and DevOps positions, whether we like it or not. And when it comes to platform engineering, the chances of encountering toil is high. You can think of toil as those tedious workarounds that should be automated but aren’t. This could be due to a lack of standard configurations for deployments. Or, perhaps engineers must copy and paste data from one module to the other—an integration that has not yet been automated.

One of the hallmarks of the SRE role is to spot and reduce such toil—hopefully before it gets out of hand. But, what exactly is toil? And, how can SREs reduce it?

I recently met with Jake Englund, senior site reliability engineer, Blameless, and Matt Davis, intuition engineer, Blameless, to uncover some tips on spotting and eliminating toil in the daily life of an SRE. Below, we’ll attempt to define toil and explore some techniques to defeat this ghoulish foe.

First, What is Toil?

Toil is tedious. Englund defines toil as repetitive manual tasks that are, ideally, automatable. “It’s not gratifying work; it lacks enduring value.” James Curtis similarly described toil as “tedious actions that really have no enduring value.” Toil is similar to technical debt in that, if left unchecked, it can hinder productivity and contribute to burnout.

Toil is invisible. Toil is the invisible work that goes on behind the scenes, says Davis. For example, at your team meeting or stand-up you might say something like, “I’m deploying this fix.” There’s usually no need to explain the nitty-gritty details of your exact step-by-step process, as it would be too far in the weeds. As a result, manual and repetitive tasks are often not given the spotlight.

Toil is persistent. Toil might be readily apparent, but it’s not as simple to eliminate; attempts at fixing toil could potentially create new toil. Tedious tasks may also be essential to keeping core infrastructure pieces up-to-date. According to Englund, what often ends up happening is that only the specific engineer knows the niche mechanics of what they’re maintaining, making them irreplaceable—a power that some employees might abuse. “If you’re the only one who knows how to do something and you’re hoarding that information, it leads to inertia against resolving toil,” said Englund.

Toil is pervasive. Toil is not limited to SRE or engineering. From UX to management or marketing, within most positions, you’ll encounter a degree of toil. This is because the ‘work as prescribed; is often much different than ‘work as done’, said Davis.

Four Tips To Reduce Toil

So, what can organizations do to conquer the nightmarish reality that is toil? Especially those that don’t have a dedicated engineering team to reduce drudgery?

1. Find It!

To reduce toil, you first have to find it. Sounds simple enough, right? Well, discovering toil might not be as obvious as it sounds. “You get into these toil situations where you might not even know you’re in it. How do you know that toil is encompassing what you’re doing?” asked Davis.

To find toil, the Blameless team has a weekly session open to the whole company to examine ‘work as done.’ According to Davis, this has been very beneficial for discovering where toil occurs. It also helps garner perspectives from roles throughout the entire company.

2. Standardize and Automate

Next, a proven method to reduce toil is standardizing your engineering processes. Standardization, said Davis, is “another word for removing ambiguity.” For example, SREs can benefit from having a common series of steps to follow when responding to alerts. Runbooks that aggregate data to respond to issues can reduce toil in the data collection process while also helping improve mean-time-to-discovery and recovery rates.

Standardizing change management is another way to reduce toil. If engineers are approaching change management in various ways, this could cause additional review and slow down the release process. Removing ambiguity from the change lifecycle can alleviate toil down the line.

When you find yourself having to retain mental models for configurations, it’s a good sign that standardization is required, explained Englund. For example, perhaps some configurations are needed in some environments but are commented out in others. Documenting the specific configurations for different development environments (and automatically populating configurations) is one way to reduce toil and the chance for human error.

3. Proactively Monitor

More proactive monitoring is another way to reduce toil, according to Englund and Davis. “Responding to a crash loop is responding too late,” added Davis. Instead, he advocated that SREs look toward leading indicators that suggest the potential for failure so that teams can make adjustments well before anything drastic occurs.

If SLIs like error rate and latency are getting bad, you must take reactive measures to fix them, causing more toil. Instead, proactive monitoring is best to see the cresting wave before the flood. Leading indicators could arise from following things like data queue operations connected to servers or the saturation of a particular resource. “If you can figure out when you’re about to fail, you can be prepared to adapt,” said Davis.

4. Practice How to Improvise

One major caveat of standardization is that you’re inevitably going to encounter edge cases that require flexibility. And when an outage or issue does arise, the remediation process is often very unique from case to case. As a result, not all investment into standardization pays out.

Alternatively, teams that know how to improvise together are proven to be better equipped for unforeseen incidents—one study on Arctic Sea emergencies found collective improvisation to be imperative for emergency response situations. Similarly, collaboration is vital for software teams. Davis calls this the “practice of practice.” If you don’t practice backup restores, for example, teams may flounder in the case of a database outage. Yet, since every remediation effort is unique, it really boils down to learning how to collaborate effectively as a team. “We can gain a lot from getting together to practice,” added Davis.

All the Small Things

Toil is like manually typing an email signature every time you are about to hit “Send.” Maybe you’ve just been putting off configuring your signature in your email client. Or, perhaps you like to write a custom sign-off for each recipient. Regardless of the reason, the longer this repetitive motion is required, it’s a stick in the spokes of productivity.

In the grand scope of things, this tedium can be a drain on time and resources. For every extra keystroke or click, every un-automated workflow or every copied-and-pasted code block, a small degree of toil is piled onto the daily workload of an engineer, dragging them away from focusing on what really matters.

Standardization, automation, proactive monitoring and collaboration are all helpful elements to consider on your journey toward reducing toil. But of course, there is the trap of increased software development automation—the time saved building automation must outweigh the upfront development cost (and ongoing maintenance effort). Otherwise, you might end up creating even more toil.

May 2, 2022
SRE Vs. DevOps: The Wrong Question?

The age-old question about the competition between DevOps and SRE sets up a false dichotomy. DevOps is a methodology while SRE is a team within operations. Although the two are often pitted against one another, developers also embody the skills and the capabilities to implement fixes.

Traditionally, application changes were reactive. The process of SRE formalizes problem resolution playbooks for better efficiency within the development life cycle. This has changed substantially over the past few years with DevOps seeing more traction and mainstream adoption.

So what is the modern relationship between DevOps and SRE and how can they best be used together? Let’s understand DevOps first.

Defining DevOps

According to Chef’s Embrace DevOps guide, DevOps is about transforming the way companies run; part of that transformation is understanding that companies should focus on managing people rather than products. DevOps evolved over the last decade and was originally born into the world of software. The adoption of cloud technologies, changes in the tech stack architecture, SaaS platforms, and security regulations have all impacted DevOps culture

Some of the fundamentals of modern DevOps are:

Everything-as-code: The current state of IT architecture has adopted an “everything-as-code” approach. Infrastructure, configuration, compliance and security can all be transformed into code.

Automate everything: Codifying everything enables standardization of processes and policies across the organization. The processes are simplified into repeatable functional code, and this makes it easier to automate the development and deployment processes while significantly reducing human errors.

Validation: There are many tools used for continuous integration, delivery and deployment in addition to other tools used for maintaining infrastructure and compliance. Without validation, there would be no way to know if everything is functioning as needed and it would be difficult to ensure seamless delivery in a highly agile environment.

Above all, teams adopting DevOps methodologies take end-to-end responsibility of managing the application development life cycle (develop, deploy and maintain). They bridge the responsibilities gap between Dev and Ops into one team.

SRE Focus Areas

Successful site reliability engineering (SRE) teams focus on the volume of performance metrics and the process of shifting left the responsibility for managing and mitigating risk. Fundamentally, they share the same values as developers in creating efficient applications. However, SRE teams focus on introducing efficiencies based on incidents that occur and sometimes need development teams’ intervention to shift the risk left in the product development cycle.

Troubleshooting is one of the main responsibilities of SRE teams since they have to manage incidents. As a result, SRE teams have a strong knowledge of product design which is essential for product reliability, rollouts and reducing recovery time. Scalable automation is also a big part of the challenge SRE teams face. Oftentimes, tasks are done manually and this can increase the likelihood of human error. A solution is implementing a policy-as-code approach in concert with developer teams.

It should be noted that when it comes to the responsibility and scope of work between SRE and DevOps, the lines are blurred—especially with the proliferation of APIs and when evaluating microservices. The types of tools being used are a differentiator since SRE teams measure efficiency through monitoring, detection and incident reporting systems.

Breaking the Silos

Modern organizations will fail when divided into silos. Developers understand that to optimize their internal processes, they need to use a DevOps methodology and involve SRE teams. For example, a startup growing and looking to scale needs development teams that are performing SRE functions with strong operational knowledge. It requires buy-in and understanding of every team member to know their role and the needs of the organization as it grows and responsibilities shift.

These fundamental values are crucial for an enterprise looking to achieve an efficient CI/CD pipeline. In a reactive cycle, SRE teams step in and are responsible for some parts of operations, but the DevOps paradigm is different because it integrates the responsibility of deployment, monitoring and remediation throughout the entire team.

Organizations embracing DevOps practices should embrace SREs’ expertise and welcome their practices into developer teams. SREs can leverage automation to fix issues as they occur and also make changes in software architecture to avoid such issues in the future. Compared to the reactive approach, developer teams are incentivized to keep things constantly running.

Scaling DevOps practices requires shared responsibility. Of note, developers have to answer when the phone rings and this can cause chaos in engineering teams responsible for large enterprise-grade applications with fast time-to-market and frequent update cycles.

A Modern Perspective

Efficiency is a core value shared by both SRE teams and the DevOps methodology. When organizations can fully integrate operations within the DevOps cycle, those silos can be broken down and integrate a shift left mentality. The traditional reactive remediation cycles won’t cut it anymore. Proactivity is essential for today’s lightning-fast application needs and continuous updates.

Application problems can be addressed immediately when approached with modern tooling and a policy-as-code approach enabling developers and SREs teams to work together. The modern relationship between DevOps and SRE teams should be one of coexistence. Both are needed for successful scalability within the enterprise. Many once thought SRE teams and developers couldn’t coexist in harmony, but debunking that myth helps organizations optimize their collaboration efforts.

April 28, 2022
The Pros and Cons of Embedded SREs

To embed or not to embed: That is the question.

At least, that’s one of the questions that companies have to answer as they decide how to implement site reliability engineering. They can either embed SREs into existing teams, or they can build a new, separate team.

Both approaches have their pros and cons. The right strategy for your company or team depends, of course, on your needs and priorities.

What is an Embedded SRE?

An embedded SRE works as part of a non-SRE team. They might join development or IT operations teams, for example—although they could also be part of quality assurance teams, security teams or any other unit within the organization that can benefit from their expertise.

Embedded SREs are the opposite of creating a dedicated, general-purpose SRE team. In the latter approach, an organization hires talent that collaborates with non-SRE teams but who aren’t integrated directly into them.

There are several other organizational models that fall somewhere between these two poles. But typically, the two main types of structures are embedded or dedicated teams.

Advantages of Embedded SREs

Embedding SREs into other teams offers several benefits.

Arguably the most important advantage is that embedded SREs ensure the highest level of collaboration between site reliability engineers and other stakeholders. When they work directly alongside other engineers on a continuous basis, it’s a pretty safe bet that their perspective will inform every decision that the other engineers make. Separate teams will still generally collaborate with other teams, but not necessarily with the same level of dedication as embedded SREs.

Embedded SREs are also beneficial for organizations where the site reliability engineering concept remains new, and where stakeholders still need to learn what SREs do and the value they can bring. It may be a bit challenging to get buy-in when you propose creating an entirely separate team; indeed, developers or IT engineers may even perceive such a team as a threat to their own jobs (even though in reality, of course, they complement other teams instead of competing with them). But when you add SREs to larger teams, it’s likely to be easier to get everyone on board with the concept and for other types of engineers to appreciate how SREs make their jobs easier.

Finally, embedded SREs work well for organizations that are on the small side and don’t need an entire team. If you have just several dozen engineers on staff instead of several hundred, hiring a few SREs to embed into your existing teams probably makes more sense than investing in a brand-new team.

The Challenges of Embedded SRE

The embedded model has its drawbacks, too.

One major risk is that the SRE’s influence within a larger team may not be strong enough to have a major impact on how that team operates. A single SRE working alongside a dozen developers or IT engineers, for example, may struggle to integrate site reliability engineering techniques and tools into the existing team’s workflow—especially if it’s a team that already has strong opinions about how it should operate.

Along similar lines, a single SRE working within a team using the embedded model could end up being stretched too thin to do his or her job well. That’s especially true if other team members expect them to “own” functions that should really be a collective, team responsibility—like responding to incidents or using chaos engineering to test reliability. The latter processes may be led by SREs, but they shouldn’t be their responsibility alone, and the function can start to break down when other team members think of them as the person on whom they can simply “dump” all of their reliability-related tasks.

Finally, the embedded model can be difficult to scale, which is one reason why it doesn’t work as well at large organizations. If you have dozens of different engineering teams, embedding an SRE into each one requires a lot of management effort. It also deprives the various SREs within your organization of the ability to work closely with each other. In that type of scenario, it probably makes more sense to create a dedicated team and have those SREs interface with other teams as needed, rather than trying to distribute site reliability engineers across the business by embedding them directly into other teams.

Conclusion

In general, then, embedded SREs make most sense for companies that have relatively small engineering organizations, and in which the concept of the function has not yet received complete buy-in. For large businesses that have numerous engineering teams, it will likely be easier to manage SREs by creating a separate team just for them.

That said, your mileage may vary from the norm, and it may be worth experimenting a bit to determine which model—embedded or not—makes the most sense for your company. Keep in mind that you can also use a hybrid approach wherein you embed SREs into some teams, while simultaneously maintaining a dedicated team that leads site reliability initiatives for the organization as a whole.

After all, site reliability engineering is all about thinking creatively and coming up with innovative solutions to challenges—even the challenge of figuring out how to structure the SREs within your organization.

April 21, 2022
Survey Shows Shift to SRE Principles to Automate IT Management

A global survey of 1,500 IT decision-makers conducted by F5 found that, on average, organizations are managing more than 200 applications deployed across an extended enterprise as well as more than 20 different security and delivery technologies.

Lori MacVittie, principal technical evangelist in the Office of the CTO for F5, said that level of complexity makes it apparent that IT teams will need to embrace higher levels of automation to manage the IT environments that are now at the core of a wide range of digital business transformation initiatives. The survey revealed that more than three-quarters of respondents (77%) were embracing site reliability engineering (SRE) principles to automate the management of IT.

A full 95% of respondents also planned to mine operational data for insights they could use to improve the customer experience and drive business growth. However, 98% of respondents also said they are currently unable to extract those needed insights from their existing systems. A total of 90% said they planned to implement some form of artificial intelligence (AI) to achieve that goal. More than two-thirds of respondents said their organizations realized that creating superior digital interactions for customers will require modernizing business processes and back-office functions.

As part of those efforts, the survey suggested that application workloads are being shifted across highly distributed computing environments. In addition to the 84% of respondents that are planning to move workloads to the edge, the survey found two-thirds of organizations (67%) are currently or planning to repatriate applications from the cloud to a data center environment in the next 12 months. A total of 93% of respondents reported using some type of as-a-service offering delivered via the cloud today.

Naturally, securing that many platforms also represents a major challenge. However, despite inherent risks, more than three-quarters (76%) of respondents said, given the choice, they’d turn off security measures to improve performance. That lack of willingness to sacrifice performance suggests that attitudes to security may not have changed much despite all the breaches that now regularly occur, said MacVittie.

It’s not clear to what degree managing enterprise IT environments has become untenable; however, it’s clear that legacy manual processes will need to become automated. Organizations can’t afford to keep hiring IT staff to manage IT platforms that span everything from the network edge to public clouds. In fact, the survey makes clear that DevOps practices—in the form of the SRE principles first defined by Google—are gaining traction. That doesn’t necessarily mean everyone managing IT is becoming an SRE or an engineer, but it does suggest that automation frameworks are being more widely used.

It may be a while before those automation frameworks are ubiquitous. However, one way or another, many of the tasks that have historically conspired to make working in IT tedious are going to disappear. The issue, of course, is to not wind up with islands of disconnected automation that introduce more complexity.

April 12, 2022