devopsy.kinsta.cloud

Tag: site reliability engineering

Site Reliability Engineering 101: DevOps Versus SRE

Consider the following scenario.

An Independent Software Provider (ISV) developed a financial application for a global investment firm that serves global conglomerates, leading central banks, asset managers, brokerage firms and governmental bodies. The development strategy for the application encompassed a thought through DevOps plan with cutting-edge agile tools. This has ensured zero downtime deployment at maximum productivity. The app now handles financial transactions in real-time at an enormous scale, while safeguarding sensitive customer data and facilitating uninterrupted workflow. One unfortunate day, the application crashed and this investment firm suffered a severe backlash (monetarily and morally) from its customers.

Here is the backstory: Application’s workflow exchange had crossed its transactional threshold limit, and lack of responsive remedial action crippled the infrastructure. The intelligent automation brought forth by DevOps was confined mainly to the development and deployment environment. The IT operations, thus, remained susceptible to challenges.

Decoupling DevOps and RunOps — The Genesis of Site Reliability Engineering (SRE)

A decade or two ago, companies operated with a legacy IT mindset. IT operations consisted mostly of administrative jobs without automation. This was the time when the code writing, application testing and deploying was done manually. Around 2008 to 2010, automation started getting prominence. Now developers and ops teams worked concurrently toward continuous integration and continuous deployment — backed by the agile software movement. The production team was mainly in charge of the runtime environment. However, they lacked skillsets to manage IT operations, which resulted in application instability, as depicted in the scenario above.

Thus, DevOps and RunOps were decoupled, paving the way for SRE — a preventive technique to infuse stability in IT operations.

Software-First Approach: Brain Stem of SRE

“SRE is what happens when you ask a software engineer to design an operations team,” said Benjamin Treynor Sloss, VP of engineering at Google.

This means an SRE function is run by IT operational specialists who code. These engineers implement a software-first approach to automate IT operations and preempt failures. They apply cutting edge software practices to integrate dev and Oops on a single platform, and execute test codes across the continuous environment. Therefore, they carry advanced software skills, including DNS configuration, remediating server, network and infrastructure problems, and fixing application glitches.

The software approach codifies every aspect of IT operations to build resiliency within infrastructure and applications. Thus, changes are managed via version control tools and checked for issues leveraging test frameworks, while following the principle of observability.

The Principle of Error Budget

SRE engineers verify the code quality of changes in the application by asking the development team to produce evidence via automated test results. SRE managers can fix service level objectives (SLOs) to gauge the performance of changes in the application. They should set a threshold for permissible minimum application downtime, also known as error budget. If a downtime during any changes in the application is within the scale of the error budget, then SRE teams can approve it. If not, then the changes should be rolled back for improvements to fall within the error budget formula.

Calculating Error Budget

A simple formula to calculate error budget is: (system availability percentage) minus (SLO benchmark percentage). Please refer to the System Availability Calculator cheat sheet below.

Suppose the system availability is 95% and your SLO threshold is 80%.

Error budget: 95% – 80% = 15%

Error budget/month: 108 hours. (At 5% downtime, per day downtime hours is 1.2 hours. Therefore, for 15% it is 1.2 x 3 = 3.6. So, for 30 days it will be 30 x 3.6 = 108 hours).

Error budget/quarter: 108*3 = 324 hours.

Quick trivia: Breaking monolithic applications lets us derive SLOs at a granular level.

Cultural Shift: A Right Step Toward Reliability and Scalability

Popular SRE engagement models such as Kitchen Sink (aka Everything SRE), infrastructure teams and embedded SREs, tend to build dedicated teams that lead to a “Silo” SRE environment. The problem with the Silo environment is that it promotes a hands-off approach, which results in a lack of standardization and coordination between teams. So, a sensible approach is shelving off a project-oriented mindset and allowing SRE to grow organically within the whole organization. It starts by appraising the teams of customer principles and instilling a data-driven method for ensuring application reliability and scalability.

Organizations must identify a change agent who would create and promote a culture of maximum system availability. The agent can champion this change by practicing the principle of observability, where monitoring is a subset. Observability essentially requires engineering teams to be vigilant of common and complex problems hindering the attendance of reliability and scalability in the application. See the principles of observability below.

The principles of observability follow a cyclical approach, which ensures maximum application uptime.

Step Zero: Unlocking Potential of Pyramid of Observability

Step zero is making employees aware of end-to-end product detail — technical and functional. Until an operational specialist knows what to observe, the subsequent steps in the pyramid of observability remain futile.

Also, remember that this culture shift isn’t achievable overnight. It will be successful when practiced sincerely over a few months.

DevOps Versus SRE

People often confuse SRE with DevOps. SRE and DevOps are complementary practices that drive quality in the software development process and maintain application stability.

Let’s analyze four key fundamental differences between DevOps and SRE.

Conclusion – SRE Teams as Value Center

A software product is expected to deliver uninterrupted services. The ideal and optimal condition is maximum uptime with 24/7 service availability. This requires unmatched reliability and ultra-scalability.

Therefore, the right mindset will be to treat SRE teams as a value center, which carries a combination of customer-facing skills and sharp technical acumen. Lastly, for SRE to be successful, it is necessary to create SLI driven SLOs, augment capabilities around cloud infrastructure, a smooth inter-team coordination and thrust automation and AI within IT operations.

May 21, 2020
SRE (Part 1): A Modern Overview
Site Reliability Engineering (SRE) is a topic that over the last several years has become a popular discussion across many companies I have interacted with. This article is the first of two that I’m passionate to share, as I’ve come across many organizations who are trying to build and implement this ideology, while all are asking the same questions—What is SRE? Who are SREs? How do we get it? Where do we start?

I—just like everyone else—have opinions around this topic. However, there is a common ground between us all. SRE is not only about tooling and tech, it’s not only about the unicorn hires you can make, but it is primarily a cultural shift within companies. It changes the way developers, product owners, admins, general engineers, etc., all interact with one another. You have to shift your operating model and how you approach building in your organization.

Now, just as a disclaimer these are just my opinions and experiences with building organizations like this and talking to other companies who have implemented or are currently implementing. There is no prescription for building SRE. Each and every company will implement at their own cadence and own internal philosophies. You will, over time, build it in a way that fits your organization and internal operating mode. Do not move too fast and attempt to force the model into your organization. Fail but fail gracefully and learn from those failure. You’ll get to your Narnia at some point.

Definitions

Let’s start with some quick definitions of the acronyms we will be using. These will be short explanations but give you the general outline.
- SRE: Site Reliability Engineer, an individual who possesses an interest in infrastructure and operations, with a software engineering background.
- SWE: Software Engineer, someone who writes the application/service software.
- Infrastructure SRE: a variation of SRE that works on infrastructure related projects (monitoring, provisioning, IaaS, etc.).
- Application/Service SRE: a variation of SRE that is dedicated to specific application/service group. These are the groups who are building the products that generate revenue for your company (hopefully).
The definition of SRE is quite scattered and I don’t think anyone truly understands/knows what it means—and no, unicorn’s will not show up below.

This is what SRE means to me:
- SRE is a core group of individuals who have a wide array of skills. They are more of a technology generalist than specialist. These skill sets range from operations, infrastructure, networking, development, hardware, distributed systems, monitoring, stability, capacity planning, software engineering, etc.
- SREs are responsible for architecture and implementation of technical infrastructure and supporting services, focusing on stability, security and scalability of those platforms.
- SREs should be building standard best practices, platforms, services, infra, etc., that reach the global organization.
- SRE is not just about the technology. SRE is a mindset, thought process and cultural shift.
- SRE shouldn’t be made mandatory for everyone in your company. Teams will have the choice to fund SRE support if they need/want it.
Areas of Responsibility

Technology has so many different aspects that it may seem hard to narrow down what SREs should actually be spending their time and energy on. Since these individuals may not be directly building public facing products or features (depending on your company, of course), what exactly will they be doing?

This by no means is limited to the list below, but should be the general areas in which you start to focus your SRE organization. Without these underlying infrastructure plumbing being properly built, monitored and scaled, it will be difficult to preach this new cultural mindset to the outside team members. The items below may seem very infrastructure heavy—and some may be. Many of these can be split between application SRE and infrastructure SRE. One is the primary producer, while the other is the primary consumer.
- Monitoring.
- Configuration management and automation.
- Infrastructure services/networking/platforms/architecture (including hardware and gen compute).
- Infrastructure tooling and capacity planning.
- Big data/data warehousing/data analytics.
- Documentation and runbooks.
- Incident response and incident management process.
Is SRE a Single Team or Multiple Teams?

There is no prescription, and results will vary depending on multiple factors. However, I recommend that this is split logically into multiple teams if you have the staff to accommodate. You can start small and have a generalist team that covers multiple areas and eventually break into smaller, more manageable pieces. The areas are:
- Infrastructure (hypervisor, storage, operating systems, containers, automation, SDN).
- Observability (monitoring, telemetry, event correlation, trend analysis, IN&IM).
- Tooling (custom tools, configuration management, developer experience tools).
- Services (databases, message queues, orchestration, micro-services).
- Apps (mostly application side knowledge and ability to support/troubleshoot).
Even though they are split in some sense, the reporting chain must remain centralized throughout the organization. Decentralizing in my opinion will break the effectiveness and communication of the team and result in an unsuccessful SRE implementation. SREs should report to like-minded individuals whose mission statement is to push SRE agenda and philosophies. Agenda is not the only reason. A central hierarchy allows for governance around tooling, best practices and general technical/architectural guidance. From that, you can allow those folks to cross pollinate in other groups and teams and spread those standards across the board. A breakdown in structure and communication will inevitably cause a deterioration in the mission.

Another reason for centralization is that it allows you to make sure your SRE hires do not become a dumping ground for operational work the app teams do not want to do or have pushed off. I have seen this happen in the past; it discourages the new hires and isolates them. They do not have a mission or a central team to work with and end up building things off on their own that only fit the needs of who they are supporting, instead of seeing the bigger picture.

The application SWEs should still be doing operational related work, since they are the ones who created the need for it. The application SRE can help offload some of that work and find ways through the central chain on what tools others are using to offload. Operational issues can quickly turn into error budgeting for the application teams.

Your SREs will likely be split into two disciplines: application SRE and infrastructure SRE. This allows you to cover both sides while pushing the same mission throughout all the teams. Infrastructure SREs are out there, and there are plenty of them. I have seen it come as a natural evolution of DevOps, system engineers and system administrators. You should focus on building your internal talent pool and then going outside for hires.

Having some of the SWEs on the team volunteer and shift work over to SRE helps speed up the process for that specific application. You can leverage their knowledge of the code base and have them learn and understand the centralized mission and tooling to bring down to their stack. These individuals can also train the central team on the app stack.

However, not every team in your company will need dedicated SRE support. Part two of this series will be a deeper dive into application and infrastructure SRE, and how to organize your teams from a management perspective.

— Anthony Caiafa
April 6, 2020
Top Six Open Source Tools for Monitoring Kubernetes and Docker
Kubernetes and Docker are two of the most commonly heard buzzwords in modern DevOps conversations. Docker is a tool that enables you to containerize and run your applications, and Kubernetes provides you with a platform to orchestrate or manage these containers—since managing thousands of containers manually using the Docker CLI would be a practical nightmare.

However, simply running thousands of containers and managing them through Kubernetes is not enough. You have to observe and analyze them properly to ensure that your services are up and running optimally. This process is called Site Reliability Engineering (SRE), a term initiated and popularized by Google. Observability and analysis are a major element of SRE. It can be subdivided into the following three areas:
- Monitoring allows you to extract numerical metrics from your applications and resources that can later be visualized and analyzed to present the current state of your resources. Once the metrics are extracted, they can be used to set up alert rules, facilitate profiling and debugging, and make better SRE decisions.
- Logging enables developers to debug their containers in case of a failure. Containers are short-lived, and so are logs. Kubernetes and Docker do provide a native way of browsing container logs, but it is very limited in its functionality. As a result, a centralized log plane is a must in any container-oriented environment.
- Tracing allows you to debug services running on a network and follow a request trail until the source of a problem can be determined. In a microservices architecture, when multiple services/containers send requests to each other to perform one business task, a proper tracing solution is necessary.
This article will list and describe the most effective open source tools for monitoring and analyzing services running as containers.

Prometheus

Prometheus is the first tool that comes to mind when discussing open source monitoring solutions. It is very popular in the development community, and it is a graduate of the CNCF project, which focuses on the monitoring side of SRE. Originally created and open sourced by SoundCloud, Prometheus simplifies the process of pulling numerical metrics based on a time series from a given metrics endpoint. It was built to monitor highly dynamic container environments.

Prometheus is subdivided into three elements: Prometheus server, Alertmanager and exporters. Exporters are stand-alone, independent processes/containers that can be run on your target resource to generate and export metrics via a metrics API. The Prometheus server then takes care of service discovery as well as pulling metrics from the exporters to be stored in the Prometheus DB and later used for visualization or alerting. Alertmanager is responsible for setting up alert rules, analyzing the data in the Prometheus DB and sending alert messages to multiple receivers if a certain rule is triggered. There are a huge number of exporters, listed here, which are both officially supported by Prometheus and maintained by the community.

Prometheus has become the industry standard for monitoring cloud-native architectures. While it is known for its simplicity with service discovery, its ease of use, its alerting capacity and its integration with Kubernetes, its polling architecture is not ideal. Currently, the metrics endpoint has to be reachable by the Prometheus server. However, there is a pushgateway implemented in Prometheus that enables metric pushing rather than polling. Another downside of Prometheus is that it doesn’t scale well. This problem can be fixed in the Thanos adaptation of Prometheus.

Related tools and technologies: Grafana, Cortex, Thanos, Prometheus Exporters, Alert Manager, Istio, Prometheus Operator.

Figure 1: Prometheus Graphs. Source: SoundCloudDev.

Grafana

Grafana is an open source metric analytics and visualization suite. It allows you to create custom dashboards using data taken from multiple sources, such as Prometheus, Elasticsearch, MySQL, Postgres and Redis. Furthermore, Grafana has its own alerting system and a role-based access control (RBAC) system for software. As a data visualization tool, Grafana is famous among Prometheus users for enabling them to effectively visualize the metrics stored in Prometheus. There’s a wide array of official and community-built custom dashboards for various data sources in Grafana, allowing users to easily set up dashboards (found at this link) and get on with monitoring. Grafana offers another associated product called Loki, which aggregates logs in Kubernetes and integrates well with the Grafana UI.

Related tools and technologies: Loki, Prometheus.

Figure 2: Kubernetes Resources Dashboard in Grafana. Source: Replex.

Elastic Stack

Elastic Stack is a group of open source products from Elastic designed to help users search, analyze, and visualize data from any type of source, in any format, in real time. This product was previously recognized as the ELK stack, where each letter of the acronym stood for one of the company’s main products: Elasticsearch, Logstash and Kibana. Elastic Stack provides both monitoring and logging solutions with the help of its big data database, Elasticsearch.

In order to aggregate logs, people tend to use Elasticsearch for storage, Logstash or Fluentd for log streaming, and Kibana for visualization. Fluentd is not part of the Elastic Stack, but it is widely used with Kubernetes to stream logs in place of Logstash, the tool provided by Elastic Stack. Similarly, Metricbeat is used to scrape metrics and set up visualizations on Kibana. The enterprise version of Elastic Stack ships with X-Pack, a set of additional tools that enables reporting, alerting and role-based access control (RBAC), among other features. By default, the Elastic Stack GUI, Kibana, does not support RBAC. You have to use the previously mentioned enterprise version of Elastic Stack to enable it.

Related tools and technologies: X-pack, Metricbeat, Logstash, Kibana.

Figure 3: Discover View in Elasticsearch. Source: XpresServers.

Sensu Go

Sensu Go is a telemetry and service health checking solution for multi-cloud monitoring at scale. It allows you to gain visibility into servers, containers, services, applications, functions and connected devices across any public or private cloud. Sensu can be run side-by-side with Prometheus to get the best out of both solutions, or it can be run natively without Prometheus. Sensu works best alongside Prometheus, since exporting application level metrics to Prometheus requires loading the Prometheus SDK to your application codebase and exposing a metrics endpoint. The endpoint is then scraped and stored in the Prometheus Server. This may sound like a lot of work—and, sometimes, it is. Sensu avoids this complexity by using a sidecar concept. A Sensu agent is deployed alongside your application. The agent constantly collects and exposes metrics to the Prometheus server without your having to change the application codebase.

Sensu works without Prometheus too. It can operate natively in Kubernetes, where it has its own server to store and visualize metric data exposed by the previously mentioned Sensu agent.

Related tools and technologies: Prometheus.

Figure 4: Events in Sensu. Source: becon.

Sysdig Inspect

Sysdig has two open source products: Sysdig Inspect and Falco. Here, we’ll focus on Inspect, which monitors and captures the container processes running in your system and allows you to drill into those processes for post-incident forensics. This enables you to analyze your application performance, troubleshoot errors and monitor any processors that could be misbehaving. Furthermore, if your system is breached, Sysdig allows you to understand how the breach occurred and what data was taken in the process. Sysdig Inspect is a very powerful tool that focuses on the performance tuning and security investigation of your system.

Related tools and technologies: Grafana, Sysdig, Sysdig Falco.

Figure 5: Sysdig Inspect Overview. Source: Sysdig.

Jaeger

Jaeger is an end-to-end distributed tracing solution that was open sourced by Uber Engineering. It allows you to monitor and troubleshoot transactions in complex distributed systems. In modern microservice architectures, the majority of operational problems fall under the umbrellas of networking and observability. When a service failure occurs, you have no idea how the requests have gone from service to service over the network to complete a single business transaction. This makes debugging extremely difficult. Currently incubating under CNCF, Jaeger uses tracing to enable root cause analysis, performance and latency optimization, and distributed transaction monitoring. Jaeger works out of the box with Istio, a popular service mesh implementation open sourced by Google.

Related tools and technologies: Prometheus, Jaeger, Zipkin, Istio.

Figure 6: Jaeger Find Traces. Source: Jaeger.

Conclusion

These tools are widely used in the tech industry, and they all have their benefits. Most of these solutions, however, require skilled implementation and ongoing manual maintenance that can be a burden for DevOps teams and a distraction from the business. There’s no one solution that can cater to all of your requirements, since each tool focuses on one or two specific aspects of observability and analysis. By mixing these tools together, you can derive a unique solution for your individual business needs.

For ease of comparison, the chart below outlines the features offered by each of the tools discussed in this article.

— Ran Ribenzaft
March 9, 2020
SRE Is the Most Innovative Approach to ITSM Since ITIL

For over a decade, ITIL has been the leading ITSM framework adopted by enterprises across the globe. So, what is driving a rapidly increasing interest in Site Reliability Engineering (SRE) as a service management alternative?

In its own words, Google refers to SRE as its approach to service management: “The SRE team is responsible for the availability, latency, performance, efficiency, change management, monitoring, emergency response and capacity planning.”

In traditional ITSM terms, the role of the SRE is responsible for service level, change, availability, event, incident, problem, capacity, performance, infrastructure and platform management. While the operational practice areas may be similar, there are significant differences in how the practices are approached.

ITIL4 Framework Compared to SRE

Released in 2019, the newest update to ITIL4 remains a complex governance model with four dimensions, seven guiding principles, a Service Value System and 34 processes (now renamed as practices). While ITIL4 pays homage to Agile and DevOps, there is little depth in the publications released to date—to be fair, several other publications in the series are in queue for gradual release. The framework spans just about every aspect of software delivery and operations and seemingly is trying to be the single source of truth in both principle and practice for IT management.

SRE Is More Closely Aligned to Agile and DevOps

By comparison, SRE focuses specifically on the reliability and resilience of complex production system operations. As an engineering discipline, SRE more closely aligns with the Agile and DevOps patterns that are being adopted by product development teams including continuous integration, testing, delivery and deployment. SREs bring the wisdom of production into the teams thereby breaking down many of the silo walls that have impeded IT for so long.

As a service management alternative, SRE also updates traditional ITSM activities with innovative and self-organizing concepts such as management to service level objectives, error budgets, toil reduction, release engineering, monitoring/observability and embracing risk as neutral approaches to service management. The core SRE book provides practical and actionable guidance for Site Reliability Engineers on managing incidents, learning from failure, testing for reliability, load balancing, handling different types of emergencies, software engineering and capacity planning. Related publications such as the Site Reliability Engineering Workbook and Seeking SRE provide additional insight.

Most importantly, a Site Reliability Engineer is an actual hireable job with a defined role, set of responsibilities and skills. SREs and SRE teams are encouraged to be creative, accountable and must spend 50% of their time reducing toil by engineering automation in order to make tomorrow better than today. Like Agile and DevOps, SRE supports self-regulation with policies and consequences. In fact, some consider SRE to be the third piece of the Develop (Agile), Deploy (DevOps), Operate (SRE) feedback loop.

What This Means for the “Humans of DevOps” in Operations and ITSM

SRE is breathing new life and opening new career paths for operations and ITSM professionals who a few years ago were battling against the mantra of “NoOps.” According to Linkedin’s 2020 Emerging Jobs Report, SRE is the fifth fastest-growing job role with 34% growth.

Services will always need to be managed. However, competing in a digital age requires new ways of working and thinking with speed and quality as key metrics. Agility must be instilled across the value stream spectrum in order to increase flow and deliver an exemplary customer experience. Your organization may not be as complex as Google’s, but the principles and practices of SRE are applicable to all environments.

You can read the Site Reliability Engineering and Site Reliability Engineering Workbook publications for free from Google. For those wanting to learn more about the practices and patterns associated with SRE, DevOps Institute recently released its Site Reliability Foundation certification with accredited training being offered globally by its Global Registered Education Partner network. Either way, if you are an operations or ITSM professional, I would highly recommend learning more about SRE. It is the future of ITSM as we cross the digital divide.

— Jayne Groll

January 31, 2020
Emerging Patterns in SRE Implementation

First we were told that every company will become a technology company. Then the tides shifted to every company will become a DevOps company. Now the zeitgeist has a new obsession: Every company will become a Site Reliability Engineering (SRE) company.

Have you been keeping pace? If your company survived the ruthless restructuring of digital transformation and is savvy enough to have surfed the DevOps wave, you might already have heard SRE is the next industry standard on the horizon. And you might be bracing yourself for yet another cycle of emerging best practices. Is the dread setting in?

Fear not, throwing out the welcome mat for SRE does not require kicking DevOps to the curb. On the contrary, the two are complementary philosophies, which, when harnessed correctly, can usher in a new era of value delivery. Essentially, SRE is a specific implementation of the core DevOps pillars of success that applies a software engineering mindset toward solving traditional operations problems–with a focus on creating reliable and scalable technology. Together, DevOps and SRE work in harmony, striving toward a shared goal of breaking down organizational barriers and delivering better software faster.

If you’re ready to give SRE a shot but you are unsure how it best fits within your team’s existing structure, let’s break down the top five team organizational structures that emerge with clients. However, feel free to view these as loose guidelines rather than an exact blueprint. Similar to DevOps, there are diverging views and practices of integrating SRE into a company’s structure.

Discovering the unique way SRE fits into your team is the most rewarding and fun part of adopting this new system. What matters most here are outcomes. Creating a role for SRE means alleviating some of the pressure on your developers, allowing them to be more productive, and more efficiently and effectively implement the DevOps paradigm. Soon enough you’ll be reaping the benefits of what happens when, as Ben Treynor, Google vice president of engineering said, “a software engineer is tasked with what used to be called operations.”

The OG(oogle)

Let’s give credit where it’s due. We wouldn’t have the SRE acronym, let alone the entire discipline, if it weren’t for Google’s innovative ops team. With the aim of bridging the gap between its production and ops teams, Google ops leader, Ben Treynor proposed eschewing the norm of building an ops team solely from system administrators and instead bringing in software engineers who possessed a systems background.

This model entails creating a dedicated engineering team focused on running and scaling a product or platform. The team should be comprised of highly skilled software and systems engineers who are on-call for the product, and simultaneously directly updating the product code base for reliability and building associated tooling to support the product. Most importantly, the product development team is still responsible for a portion of these operational tasks and may not be allowed to ship new features if reliability falls below agreed upon thresholds (the Error Budget).

Why Change When You Can Rebrand?

If your ops, DevOps and platform teams are on board with the SRE philosophy but you either don’t want or don’t need to make significant changes to your product code, this is the approach for you. In this model, your teams can simply rebrand as Site Reliability Engineers, while adopting and implementing the tenets that come with the title. That means shifting the concentration of engineering to focus on improving reliability and scalability. This team would also be on-call, playing a prominent role in the underlying infrastructure, tooling, platforms or day to day support of the product.

Beware: You need to take seriously the underlying tenets and need for strong software engineering skills on these teams in order to realize the benefits.

The Inside Job

Perhaps you’re down with SRE but don’t want it to co-mingle with product accountability. A viable option then would be creating an internal consulting arm–a centralized team solely dedicated to creating and advocating for reliability tools and processes. One of the prominent benefits of this structure is that your team would be able to not only see the forest for the trees but also serve as your SRE gurus, keeping your team abreast of the latest technologies, trends and research in the SRE space.

You Get an SRE Engineer!

For those who want to weave in SRE engineers throughout their entire organization, this approach applies a holistic mindset to the SRE implementation proposition. That means SRE engineers are introduced into cross-functional teams that own the end-to-end lifecycle of a product, from build through decommissioning. From here the road diverges: your choice is to either create an SRE team of engineers who belong to a single capability and are also embedded full time within a product team, or allow product teams to hire their own dedicated SRE engineers. As Cloud and SRE Practice Lead at Slalom Build, I took the former. For us, this approach guarantees bespoke SRE implementation, ensuring right-sized scalability and reliability throughout the product lifecycle.

SRE? New to Me

Despite the compounding hype around SRE, it has yet to hit universal name-recognition. If it’s still unfamiliar to you, that’s okay. It’s not too late to raise your hand and get up to speed during this period in which we are still in the early stages of adoption. I assure you that if you reach out to expand your knowledge of SRE and even bring it back to your team, you will be met with enthusiasm and welcoming arms.

— Rob Cummings

December 11, 2019
The SRE Pressure Cooker: Balancing Velocity Against Risk

Delivering fast, reliable digital services today is a lot like Olympian alpine skiing. These services must deftly maneuver a series of perilous passages en route to end users, all while maintaining the astounding speed we now take for granted.

In an SRE’s world, those passages are today’s increasingly complex and interconnected internet infrastructure through which they must deliver and support cutting-edge application functionality–all while rolling out updates at a relentless pace. It may appear easy to end users, but SREs are working incredibly hard behind the scenes, and as the internet grows more intertwined and treacherous, the risk for catastrophe increases.

Judging from the results of our second annual SRE survey, striking this critical balance between supporting development velocity while maintaining site reliability is an onerous and stressful task. Here’s a look at some of the key findings, and takeaways for organizations increasingly reliant on this emerging role.

The SRE Role Is Relatively New

Even though Google coined the term in 2003, 64% of respondents noted their SRE disciplines have only been in existence for three years or less. This means many SREs and SRE teams—those professionals dually knowledgeable in software development and adapting IT systems to meet the needs of particular software—are still adjusting to their roles and responsibility. Many SRE teams are also small relative to the size and scope of the infrastructure to be managed, and they remain a scarce resource, with many organizations highlighting skills shortages in this area.

IT Incidents Are the “New Normal”

Almost half (49%) of survey respondents reported they had worked on a service incident over the course of the past week. In the month of June alone, Google—which is known for having a very mature SRE practice—had two significant outages: One that hit YouTube, G Suite, and several popular third-party apps relying on Google Cloud for their back-ends (including Discord and Snapchat), and the Google Calendar outage just a few weeks later.

Catchpoint has detected a significant uptick in outages in recent months, likely the result of growing internet complexity which naturally increases the potential for problems impacting digital service reliability. This is a challenge for all companies and their SRE teams, including the biggest and the strongest—the last few weeks have shown us no one is immune. Organizations can help their SREs maintain a healthy perspective by bearing this in mind.

Many SREs Don’t Have Clearly Defined Service Level Objectives (SLOs)

This is problematic, because without SLOs, it is nearly impossible to identify what is an incident in the first place. Twenty-seven percent of SREs reported they don’t have any SLOs, and when flags are raised on everything, this leads to more alerts, a greater number of false positives, and naturally, greater SRE fatigue and stress. Our own conversations with SREs have revealed that even among those with SLOs in place, these are often unrealistic or not correlated to end-user satisfaction and happiness—for example, targeted at 100% availability (impossible perfection) when 98% availability may suit end users just fine. If you’re striving for perfection, every little thing that goes wrong constitutes an emergency, even if customers don’t notice. Setting unrealistic targets and not letting customer satisfaction define SLAs is a recipe for SRE burnout.

Many SREs Feel Burnt Out and Stressed

According to our survey, 21% of respondents indicated they never experience post-incident stress. However, it was interesting to note the answers to the following question, where we asked what types of changes SREs notice after an incident—in concentration, ability to sleep, mood and more. One would suspect if an SRE never experienced post-incident stress, he/she would select “none” for the second question—but that wasn’t the case. Rather, one third of SREs said they have noticed such symptoms. Sixty-seven percent of SREs who reported feeling stress after each incident, indicated they don’t believe their company cares about their well-being.

We are aware of other surveys showing SREs are quite happy, particularly given their impressive compensation. While we don’t believe being an SRE automatically makes someone stressed and unhappy, organizations should never assume these professionals know and understand how appreciated they are. There is often a reluctance to discuss topics such as job-related stress, as evidenced by the fact that 50% of the SREs we targeted opted not to complete our survey—perhaps a result of SREs’ predominant hero culture. In addition to compensation, organizations should consider other ways to show SREs—whether they vocalize their stress levels or not—they care about their well-being, such as offering an extra vacation day or other perks after a particularly trying incident.

SREs Could Benefit from Automation, but Don’t Have Enough

Another thing we consistently heard in our conversations with SREs is they feel so busy putting out fires today they don’t get a chance to create a better tomorrow—in other words, they lack the opportunity to be proactive and preemptive in their approaches. A key factor in this is the amount of toil SREs face daily—meaning manual, repetitive, automatable, tactical work. Our survey found 59% of SREs believe there is too much of it in their jobs, and too little automation. Nobody strongly agreed with the statement, “We have used automation to reduce toil,” while almost half disagreed or strongly disagreed.

Considering SRE teams are often small and the talent pool tight, it becomes critical for organizations to automate as much work as possible, including areas such as application performance monitoring. The SRE team at Alaska Airlines recently automated performance monitoring and deep-dive diagnostics, and as a result has been able to reduce their alert noise by 92%. Ultimately, this has freed up the team to focus on real issues versus wasting time investigating false positives. This has helped them bring down the average mean time to detection for real issues from hours to less than 10 minutes.

Conclusion

The stage has been set for SREs to experience significant stress, at a time when organizations need their exceedingly rare skillsets the most. As internet complexity increases, it is critical for organizations to continually nurture, encourage and motivate these valuable team members. This means understanding and empathizing with the intense pressures they’re under; having SLOs well-defined, realistic and meaningful; leveraging soft skills to communicate effectively and non-monetary incentives to show appreciation; and providing them with needed automation so they can focus on proactive improvements that advance the business.

— Mehdi Daoudi

August 5, 2019
Blameless Launches DevOps Platform Based on SRE Principles

Blameless has emerged from stealth to launch a platform for managing DevOps based on the site reliability engineering (SRE) best practices defined by Google.

Fresh off raising an additional $20 million in funding, the Blameless platform provides across-the-board visibility into the complex network of application programming interfaces (APIs) and applications distributed across an organization, said CEO Ashar Rizqi. Specific modules within the Blameless platform address incident resolution, reliability insights dashboards, postmortems, service level objectives (SLOs) and automatic error budget calculation.

Blameless takes its name from an SRE principle that no one should be held at fault when complex systems fail. Rather, the assumption should be that inherent complexity of the IT environment is at fault. While DevOps is more of a philosophy surrounding the management of DevOps, SRE is emerging a more prescriptive instance of implementing DevOps that Google is now promoting. The goal is to make sure that whatever the issue, the right people to address it are brought together at the right time, said Rizqi.

The Blameless platform was created by Lyon Wong, who now serves as the COO of Blameless. Previously, Wong was a partner at Lightspeed Venture Partners, founder of S28 Capital and a product manager for Microsoft Windows. Rizqi is a former head of core platform at MuleSoft and SRE manager at Box.

Rizqi noted that while most organizations are trying to embrace DevOps principles, they often lack the tools needed to build the workflow processes required. All too often, organizations rely on little more than a Wiki document to manage their response to an incident. The Blameless platform is designed to provide access to an integrated set of tools for managing everything from the incident itself to the impact that event had on the business, he said.

It’s still too early to say with any certainty how widely SRE practices will be adopted. This far, most organizations that have adopted DevOps have tended to prefer to define a set of processes that are customized to their environment. But there are just as many, if not more, organizations that have found it daunting to get started. Platforms such as Blameless represent an opportunity for those organizations to embrace DevOps using a more prescriptive approach.

Of course, a big part of the reason that DevOps has been embraced unevenly by various departments within the enterprise is the lack of a consistent approach to implementing DevOps at scale. Teams that have embraced DevOps with enterprises have tended to create their own individual processes using a wide variety of tools. Many of those organizations have also found it challenging to get IT operations teams to embrace those processes because they require an approach that can be implemented consistently across multiple projects at scale. SRE provides IT operations teams with a framework that Google has already proven works at scale.

Regardless of the path chosen, the one thing that is for certain is more agile approaches to managing IT operations will be required to keep pace with the rate at which applications are now being developed and deployed.

— Mike Vizard

March 20, 2019
How SRE Creates a Blameless Culture

People in tech love to praise failure as essential to innovation. But at far too many organizations, when failure happens, it is punished. When an IT system goes down, an individual or a team is immediately ID’d, blamed and shamed. This is counterproductive. Blame damages companies by creating a play-it-safe atmosphere and stifling innovation; even worse, when mistakes are made, they are often hidden.

Fortunately, more and more organizations are recognizing the damage that blame can do and embracing a blameless culture. This is a big shift, yes, but companies can make the move to blameless—and make it stick—if they bring site reliability engineering (SRE) best practices to their technology teams.

Why Blame is Bad

I’ve worked at several fast-growth companies that suffered under a culture of blame. At one, we were scaling quite rapidly. Change was constantly being introduced to the infrastructure, which meant that it broke all the time.

Because we had customers running their core business on our systems, our reliability requirements were very high—so high that for a while we stopped feature development altogether, in fear that new features would bring our systems down.

This was difficult enough, but the culture of blame made it worse. There was no psychological safety net and fingers were pointed constantly. There was an us-versus-them mentality—engineers versus non-engineers, developers versus operations, developers versus developers. Developers dedicated way too much mindshare to protecting themselves and their domain.

In a blame-rich environment, developers who should be delivering concrete results spend too much of their time in meetings trying to justify their work. There is a culture of retribution for mistakes. This leaves little room to learn and grow from failure because developers get harassed by management when they do make mistakes, or they get fired.

Your best people—those empowered to make decisions—see their responsibility undermined and go elsewhere. Creativity and innovation slow to a crawl. Meanwhile, competitors run ahead.

Why Blameless is Better

In a blameless culture, everyone feels safe and no one is afraid to make mistakes. It’s a psychologically secure environment, where true DevOps can happen. Developers feel confident enough to express their ideas and take chances. Development folks and operations folks collaborate well, and everyone is aligned on the problem.

The work environment is positive and workers are assumed to be doing their best. When mistakes happen, people aren’t blamed. Instead, any error is viewed as a manifestation of an underlying vulnerability in the systems. Attention is focused on fixing those vulnerabilities and, as a result, systems are constantly improving.

How to Get to Blameless

Let’s say you just had an outage and your database crashed. Don’t start by placing blame on a team or individual. Rather, start by conducting a postmortem. Ask questions and find the answers: Why did the database crash? Because Alex pressed the wrong button. Well, why? Because we didn’t have a system in place to enable automatic checking or a review process. So why did we not have that?

Now you’re getting to blameless and you’re reaping the benefits. You’re having a productive conversation and you’re focused on fixing your systems, not affixing blame. It’s not about Alex pressing the wrong button; it’s about how your organization is not set up to ensure success. It’s about this vital opportunity to learn from a mistake and get better. Don’t fire Alex. Put a solution in place so the problem doesn’t happen again.

How SRE Can Help

To get to a truly blameless environment, you need to implement SRE best practices. Why? Because SRE creates ultra-scalable and highly reliable software systems, by building in automation and tooling—moving the human element out of the picture and focusing on the systems.

One of the founding principles of the SRE movement is blamelessness. By implementing an SRE team and establishing SRE practices, your organization can move from the traditional operational model of break-fix to an environment that applies a development approach to IT operations, with the goal of improving reliability via automation, continuous integration and delivery.

For this to happen, you will need a SRE champion to quarterback your SRE effort. You will also need organizational buy-in to blamelessness. It has to be practiced top-down and have the full support of all leaders in the organization.

Tooling is critical, too. Look for tools and dashboards that can quickly do postmortems and find action items that need attention, that can identify the most commonly impacted services, products and customers, and the contributing factors that most often lead to incidents. The right tools will help you resolve incidents faster, by providing a clear overview of metrics related to incident identification and the actions that need to be taken.

You want to move fast and be innovative. But can you do it reliably? With a blameless culture, you can. You’ll have a safe environment that attracts and retains the best talent. You’ll move quicker, because you’ll be focused on fixing systems that break and finding out why—not the blame game. By delivering a higher degree of reliability, you’ll build greater trust with your customers.

You’ve heard the saying: Failure is not an option. In a blameless environment, failure is always an option—because it means that systems are always improving and innovation is always happening.

— Ashar Rizqi

February 27, 2019
4 Predictive Analytics Challenges Facing SRE Teams
“Predictive analytics” refers to the many techniques used to analyze data to make predictions about the future, such as data mining, statistics, modeling, machine learning and artificial intelligence. Being able to predict technical and business outcomes is one of the main promises of big data. Indeed, many businesses have invested heavily in infrastructure to anticipate key performance indicators such as demand, pricing or maintenance. The investment required is substantial but, when it works, it yields a very positive ROI.

In this post, we’ll examine why implementing predictive analytics in a site reliability engineering (SRE) context requires a significant investment in both setting up the right data sets and applying the right logic to that data.

SRE should in theory be a prime target area for predictive analytics. Why? Because companies have already widely adopted monitoring tools over the past decade. In doing so, they’ve amassed extensive system health and behavior data that is ripe for analysis. SRE teams have also increased their ability to make sense of this data as data science tools have become more accessible. Basic data science aptitude is increasingly become a desired skill when hiring engineers.

Additionally, the SRE team is responsible for a mission-critical KPI: system availability. This single metric is directly tied to customer SLAs in digitally dependent businesses. SRE teams have a strong incentive to implement predictive analytics. Why? Having the ability to prevent downtime by addressing potential issues before they hit the production environment is hugely valuable. More system availability translates into more revenue, higher customer satisfaction and lower costs of problem mitigation.

However, the reality is that an SREs job is still mostly focused on the diagnostic, mitigation and “fixing” parts, as opposed to more proactive tasks. This is in big part because predictive analytics applied to SRE/DevOps has proven difficult to implement and perform in a disciplined manner. Therefore, it is still an aspirational goal at most companies.

SRE Challenges in Predictive Analytics

Up next, let’s explore the four challenges that need to be overcome to implement high-quality predictive analytics practice in an SRE context, including:
- Data collection
- Data quality
- Data volume
- Model usability
Challenge 1: Data Collection

First, the working data sets need to fit what you are trying to predict. In this case, it is challenging to get the right data set because the SRE team’s mission is very broad; specifically, ensuring the reliability of the entire system. The difficulty lies in the fact that there is no such thing as “full system” data readily available. All the relevant data is distributed across siloed, or overlapping repositories and monitoring tools. Centralizing the data requires either a full system agent (which takes a lot of work to set up) or through APIs, which requires mastering a multitude of API integrations. An additional obstacle in getting the right data lies in the inconsistent nature of events, time series data and logs. These disparate data types can’t be analyzed together unless they can be transformed into a uniform data set.

Challenge 2: Data Quality

SRE environments emit lots of data and some won’t make sense for the purpose of the analysis. Whichever data collection method is used, the information tracked often reflects some level of bias on the part of whoever set up the tracking mechanism in the first place. This leads to capturing both irrelevant data or inadvertently leaving out important information. In other words, the data is often both noisy and lacking at the same time.

The key to successful modeling lies in the selection of prediction variables with data that is obtained before the event predicted happens (in this case, a failure). Defining the variables that lead to the determination of a subsystem failure is critical and requires specific domain knowledge. It is difficult to define and extract a pure data set sanitized of noise. The more data you leverage, the higher the chances that you are actually “feeding the beast” with irrelevant information. Too much irrelevant data leads to inaccurate models and false positives.

Challenge 3: Data Volume

Here you can have two different types of challenges: too little data and too much.

Not enough data

You typically need at least 100 instances of what you are trying to predict and probably at least 100 where it didn’t work to train a model. This assumes the data set is clean and directly relevant to a very narrow, specifically defined problem. In the case of SRE, the scope of issues that can affect a system is such that the data set required is very substantial. Likely, it’s several months worth of a wide variety of system behavior measurements. For example, if you collect data only through passive collection, think about the lag time between t-zero (when you start collecting data) and the time when you would have results. During that time, you have a bootstrapping problem—your model can’t be trained to predict accurately what you want.

Too much data

There are two practical points to make around the notion of “too much data.”
- Lack of standardization: Processing large volumes of data for analysis remains a difficult engineering challenge. There is no standardization for data pipeline; every method has a limit in terms of how many inputs it can ingest on a per-second basis, and ingest rates often end up becoming bottlenecks in large enterprise environments.
- Volume does not equal accuracy: There is a point beyond which you get diminishing returns: statistical theorems show that after a certain point, feeding more data into a predictive analytics model will not provide more accurate results.
Overall, getting the quality and quantity of the training data right is the best investment you can make in building your predictive capabilities.

Challenge 4: Model Usability

From a pure “mechanics” standpoint, the analytical concepts and methods are there to reflect the complex thinking required to predict the behavior of an enterprise’s combination of technology, information and infrastructure. Predictive software relies heavily on advanced algorithms and methodologies such as logic regressions, time series analysis and decision trees. The problem is that the workings of the kinds of systems that SREs deal with are very hard to understand in the first place, let alone predict. It is not uncommon even for SREs to not entirely understand how a system works if they were not the ones who initially set it up. In addition, every company’s system is different.

When data scientists design predictive models without years of experience in an SRE capacity in multiple environments—in other words, without leveraging subject matter expertise—it rarely works perfectly. Often the models they create lack usability, even after the validation process to confirm its quality. A modeler will split the available data into a training set where the model is run to generate predictions and compare performance against the actual outcomes. That will require multiple iterations to generate a model that has high overall predictive accuracy, and 9 out of 10 times the result is overfitting, or perfect for the training data. A subject matter expert in the domain of technical operations can cut the cycles of iteration and get to a predictive model that works with an acceptable level of risk.

In short, implementing predictive analytics in an SRE context takes a significant investment both in setting up the right data set and applying the right logic. Stay tuned for a future blog post that will provide some pointers on how to achieve the best prediction results.

About the Author / JP Emelie Marcos

JP Emelie Marcos is the co-founder and CEO of SignifAI. Previously, JP served as the Chief Operating Officer of Tango, a leading mobile consumer application that reached 400 million regular users in five years. JP’s career spans 20+ years leading teams in technology startups and large companies as an investor, GM and CEO.
August 23, 2017
SRE vs. DevOps — a False Distinction?

Just a few days before he died at the beginning of the 1990s, a wise man taught us that “the show must go on.” Freddie Mercury’s parting words have long provided the guiding light for many, if not all, ops teams. In their eyes, the production environment should be exposed to minimum risk, even at the expense of new features and problem resolution.

About 10 years ago, Google decided to change its approach to production management. It took the company only a few years to realize that while R&D focused on creating new features and pushing them to production, the Operations group was trying to keep production as stable as possible—the two teams were pulling in opposite directions. This tension arose due to the groups’ different backgrounds, skill sets, incentives and metrics by which they were measured.

Trying to bridge this gap between the two groups, one of Google’s ops leaders, Ben Treynor, thought of an innovative solution. Instead of having an Ops team built solely from system administrators, software engineers—with an R&D background and mentality—could enrich the way the team worked with the development group, change its goals and help with automating solutions.

Site Reliability Engineering, or SRE

And so, the Site Reliability Engineering position was created. According to Google, SRE engineers are responsible for the stability of the production environment, but at the same time are committed to new features and operational improvement. Google decided its SRE teams should be composed of 50 percent software engineers and 50 percent system administrators. The engineers were driven to use software as a way of solving problems and perfecting what had historically been solved by hand. They integrated easily with the development team, and encouraged code quality improvements and automation testing.

Understanding the value of SRE, several organizations of various sizes decided to embrace its principles. Some, such as Dropbox, Netflix and Github, are well-known for being at the forefront of technology leadership.

Wait, Isn’t That DevOps?

DevOps is a more recent movement, designed to help organizations’ IT department move in agile and performant ways. It builds a healthy working relationship between the Operations staff and Dev team, allowing each to see how their work influences and affects the other. By combining knowledge and effort, DevOps should produce a more robust, reliable, agile product.

Both SRE and DevOps are methodologies addressing organizations’ needs for production operation management. But the differences between the two doctrines are quite significant: While DevOps raise problems and dispatch them to Dev to solve, the SRE approach is to find problems and solve some of them themselves. While DevOps teams would usually choose the more conservative approach, leaving the production environment untouched unless absolutely necessary, SREs are more confident in their ability to maintain a stable production environment and push for rapid changes and software updates. Not unlike the DevOps team, SREs also thrive on a stable production environment, but one of the SRE team’s goals is to improve performance and operational efficiency.

Google tried a few approaches to implementing the SRE process before finding the one that suited them best. One of these approaches attempted to tie the number of permitted releases to the product’s stability. The principle underlying this process is that new releases are green-lighted based on current product performance.

For each service, the SRE team sets a service-level agreement (SLA) that defines how reliable the system needs to be to end users. If the team agrees on a 99.9 percent SLA, that gives them an error budget of 0.1 percent. An error budget is exactly what the name suggests: the maximum allowable threshold for errors and outages. Here’s the interesting thing: the development team can “spend” this error budget in any way they like. If the product is currently running flawlessly, with few or no errors, they can launch whatever they want, whenever they want. Conversely, if they have met or exceeded the error budget, and are operating at or below the defined SLA, all new releases are frozen until they reduce the number of errors to a level that allows the launch to proceed. This process ensures that both the SREs and developers have a strong incentive to minimize the number of errors in production.

Another interesting approach recommended by Treynor, which is more related to SRE professionalism and efficiency, is allowing SREs to move between projects. Moreover, he suggests allowing SRE engineers to move to development, and even the other way around. As the work done by both teams is similar, these transitions help the Ops team gain better and deeper knowledge of the product and code, and bring the Dev teams into the production space to help them understand its challenges. This strongly promotes a team atmosphere, rather than one in which an individual feels that “I’m on the SRE team for this product.” As part of this approach, Treynor had Dev teams handle 5 percent of the operations workload. This, according to many organizations, adds to the SRE team’s motivation and effectiveness.

Riddles in the Dark

Well, SRE seems perfect. As already stated, several large-scale organizations have chosen to move some of their production operations from old Ops to SRE. However, there are still a few questions that need to be asked:

Is the position of SysAdmin (production/ops) no longer relevant for SysAdmins?

Historically, almost all system administrators have come into their roles through tech support and similar work, or even just running Linux on their desktops and then transitioning into server work. It should be pretty clear that the same path is not available into SRE. To retain their positions, SysAdmins should now be more code-oriented, have better technological knowledge and be receptive to new methods of conducting the work they already do.

Can an SRE team prevent production incidents?

One of the strengths of a professional site reliability engineer is the ability to handle the growth of production load and traffic in-house. Monitoring and analyzing processes and logs with platforms such as the ELK Stack is part of the day-to-day workload, and the team should be able to identify problems as they occur, and even foresee risks to software stability. The power of this position—based on the skill set it requires—lies in developing solutions for these problems and risks. As Ciara, a software engineer in Google’s cloud storage SRE team, described it in a “Life in Google” post, “We solve cooler problems in cooler ways.”

Are we doing DevOps or are we doing SRE?

This question is usually asked by people trying to position themselves in the Operations world. According to many companies that implemented SRE in a slightly different way than Google, you don’t have to decide. At Reddit, ops engineers work on reducing toil, improving deployment and scaling processes, but they are referred to as “DevOps.”

At Logz.io, we bridge the gap between developers and production through the use of automated monitoring and performance stress-testing. And we personally refer to it as “DevOps” rather than “SRE.”

So, is it Just a Question of Context?

In her blog, Charity Majors talks about SRE and DevOps being two different operational approaches that any organization can choose to work by, but insists on emphasizing that there is no “correct” approach. Although Google and Dropbox have decided on SRE, this does not mean the rest of the world should do so as well. What fits Google’s needs and organizational philosophy does not necessarily work for other orgs—at any scale. Moreover, Charity believes that DevOps, having grown and evolved within a broader variety of software organizations, is the more flexible, collaborative and adaptable approach, and will work better for most software organizations at all stages of development.

On the other hand, SoundCloud senior engineer, Matthias Rampke issued a series of tweets on how SRE and DevOps were basically the same, with only one difference—high management support. Though in contradiction to her blog, Charity lent weight to this opinion during an AskMeAnything event that revolved around DevOps and SRE. She shared a story about a friend who was hiring for a startup, which, as an experiment, posted the exact same job description twice, the only difference being that one listing was titled “DevOps engineer,” and the other “SRE.” At the time of relating the story, DevOps was winning by 10 percent or 20 percent. However, all this means is that the job title is irrelevant, and naming an SRE team does not magically bestow qualities upon it. Rather than focusing on the title, organizations must focus on the work being done.

To summarize, we can quote Matt Simmons, a technologist with many insights into this topic, who says, “Not every infrastructure needs an SRE, but every infrastructure could use an administrator who acted more like one.”

— Asaf Yigal

May 18, 2017