devopsy.kinsta.cloud

Author: Quentin Rousseau

  • 5 Tips If You’re the First SRE Hire

    5 Tips If You’re the First SRE Hire

    Site reliability engineers (SREs) have a considerable set of tasks to juggle no matter where they work or how long their company has had an SRE practice. But if you’re the very first one to join an organization—as many SREs are these days, given that the trend is trickling down into smaller and smaller companies—you face a special group of challenges.

    You may find it difficult to get buy-in for SRE from other technical teams. You may struggle to know how to convince your boss that the company’s decision to hire an SRE is paying off. You probably worry that you’ll end up being overextended because you are the only SRE. And so on.

    These five tips are for site reliability engineers who find themselves in the position of being the pioneer at their companies. Whether you’re an experienced SRE who has just been hired at a company that had none previously or you’re brand-new to SRE and your first job happens to be at a company that is also brand-new to the SRE role, the following best practices will help you make the transition as smooth as possible for all involved.

    Collect SRE Metrics Early and Often

    For starters, you’ll want to be prepared to justify the business’s investment in your new role as soon as a boss or fellow engineer asks.

    Do this by collecting metrics about the technical and business impact of SREs starting as early as you can. Your goal should be to quantify how you are benefitting the organization so that you can encourage more support for yourself and any others who may be hired after you.

    On top of this, collecting metrics is also important for establishing early feedback about which SRE strategies are succeeding and which could be improved. Since you’re building an SRE practice from scratch, getting this insight early and often is critical.

    Treat Devs and IT Engineers Equally

    As you start the role, you may find some uncertainty among existing engineers about whose ‘side’ you are on. Are you there to support developers or IT engineers?

    The answer, of course, is both. Although SREs tend to borrow from software engineers’ playbooks more than they do from those of IT Ops, an SRE’s goal should be to empower and support both of these teams. So, unless you’re given explicit instructions to interface with only one group or the other, make yourself everybody’s buddy within the IT organization.

    Introduce Yourself to Non-Technical Teams

    SREs spend most of their time doing technical things with technical tools. But ultimately, the only reason they are doing what they do, in most cases, is to support non-technical users by improving the reliability of the systems they use.

    As the first SRE at a company, it may be well worth your time to get to know these non-technical stakeholders and ask them what they see as the greatest reliability issues in the organization. Building these relationships early on will help you communicate the value of your role across the organization. It’s also likely to come in handy during incident management routines, when you may need to interface with non-technical stakeholders as well as other engineers.

    Build Your SRE Tool Set

    Deciding which tools to use is an obvious early step for any SRE who is building a practice from scratch. But we didn’t put it first on this list because it’s important to get to know the rest of the organization before committing to a particular set of tools.

    As you meet the rest of the organization, ask detailed questions about which tools they use to address needs like observability, security and incident management. Then, by factoring in feedback about which tools work well and which don’t, choose which tools you’ll deploy to support SRE. Be prepared for those tools not to match your own personal preferences perfectly. You may need to prioritize the preferences of the rest of the organizations.

    Plan to Grow the Team

    Unless you have been hired at a very small company, it’s likely that the business plans to hire more SREs over time (assuming that you prove the value of the position, of course).

    Toward that end, plan for how the team will expand. Think about things like whether to use embedded SREs or create a dedicated SRE team. Think, too, about how you’ll document your company’s SRE practice and ensure it keeps operating smoothly as individual engineers come and go.

    Conclusion

    The job of a newly hired SRE is not an easy one, especially if you’re the first person at your company to fill those shoes. But by being deliberate and strategic about how you work, which data you collect and how you introduce the company to SRE, you set yourself – and your future colleagues – up for success.

  • The Pros and Cons of Embedded SREs

    The Pros and Cons of Embedded SREs

    To embed or not to embed: That is the question.

    At least, that’s one of the questions that companies have to answer as they decide how to implement site reliability engineering. They can either embed SREs into existing teams, or they can build a new, separate team.

    Both approaches have their pros and cons. The right strategy for your company or team depends, of course, on your needs and priorities.

    What is an Embedded SRE?

    An embedded SRE works as part of a non-SRE team. They might join development or IT operations teams, for example—although they could also be part of quality assurance teams, security teams or any other unit within the organization that can benefit from their expertise.

    Embedded SREs are the opposite of creating a dedicated, general-purpose SRE team. In the latter approach, an organization hires talent that collaborates with non-SRE teams but who aren’t integrated directly into them.

    There are several other organizational models that fall somewhere between these two poles. But typically, the two main types of structures are embedded or dedicated teams.

    Advantages of Embedded SREs

    Embedding SREs into other teams offers several benefits.

    Arguably the most important advantage is that embedded SREs ensure the highest level of collaboration between site reliability engineers and other stakeholders. When they work directly alongside other engineers on a continuous basis, it’s a pretty safe bet that their perspective will inform every decision that the other engineers make. Separate teams will still generally collaborate with other teams, but not necessarily with the same level of dedication as embedded SREs.

    Embedded SREs are also beneficial for organizations where the site reliability engineering concept remains new, and where stakeholders still need to learn what SREs do and the value they can bring. It may be a bit challenging to get buy-in when you propose creating an entirely separate team; indeed, developers or IT engineers may even perceive such a team as a threat to their own jobs (even though in reality, of course, they complement other teams instead of competing with them). But when you add SREs to larger teams, it’s likely to be easier to get everyone on board with the concept and for other types of engineers to appreciate how SREs make their jobs easier.

    Finally, embedded SREs work well for organizations that are on the small side and don’t need an entire team. If you have just several dozen engineers on staff instead of several hundred, hiring a few SREs to embed into your existing teams probably makes more sense than investing in a brand-new team.

    The Challenges of Embedded SRE

    The embedded model has its drawbacks, too.

    One major risk is that the SRE’s influence within a larger team may not be strong enough to have a major impact on how that team operates. A single SRE working alongside a dozen developers or IT engineers, for example, may struggle to integrate site reliability engineering techniques and tools into the existing team’s workflow—especially if it’s a team that already has strong opinions about how it should operate.

    Along similar lines, a single SRE working within a team using the embedded model could end up being stretched too thin to do his or her job well. That’s especially true if other team members expect them to “own” functions that should really be a collective, team responsibility—like responding to incidents or using chaos engineering to test reliability. The latter processes may be led by SREs, but they shouldn’t be their responsibility alone, and the function can start to break down when other team members think of them as the person on whom they can simply “dump” all of their reliability-related tasks.

    Finally, the embedded model can be difficult to scale, which is one reason why it doesn’t work as well at large organizations. If you have dozens of different engineering teams, embedding an SRE into each one requires a lot of management effort. It also deprives the various SREs within your organization of the ability to work closely with each other. In that type of scenario, it probably makes more sense to create a dedicated team and have those SREs interface with other teams as needed, rather than trying to distribute site reliability engineers across the business by embedding them directly into other teams.

    Conclusion

    In general, then, embedded SREs make most sense for companies that have relatively small engineering organizations, and in which the concept of the function has not yet received complete buy-in. For large businesses that have numerous engineering teams, it will likely be easier to manage SREs by creating a separate team just for them.

    That said, your mileage may vary from the norm, and it may be worth experimenting a bit to determine which model—embedded or not—makes the most sense for your company. Keep in mind that you can also use a hybrid approach wherein you embed SREs into some teams, while simultaneously maintaining a dedicated team that leads site reliability initiatives for the organization as a whole.

    After all, site reliability engineering is all about thinking creatively and coming up with innovative solutions to challenges—even the challenge of figuring out how to structure the SREs within your organization.

  • Top Nine Skills for SREs to Master

    Top Nine Skills for SREs to Master

    It’s easy to talk at a high level about what site reliability engineers do: They ensure that IT systems achieve availability and performance requirements.

    But which skills, exactly, do SREs need to do to perform their jobs? That’s a more complicated question.

    To answer, let’s look at the top nine site reliability engineer skills that modern SREs (or aspiring SREs) should master. Although SRE skills may vary from one team to the next depending on the types of systems managed and the main types of reliability challenges faced, virtually all SREs need a core set of standard skills that allow them to understand and manage the type of complex, distributed systems they will have to support at the typical organization today.

    Here’s a breakdown of top SRE skills.

    Networking Expertise for SREs

    The network plays a pivotal role in connecting modern, distributed environments. As such, it’s often the culprit when something goes wrong—a lesson that Facebook, for example, learned when a networking problem brought down its entire global infrastructure.

    Situations like this are why SREs should master networking concepts. Even if their organization also employs networking engineers, site reliability engineers need a deep understanding of networking themselves to know when the network is the root cause of an incident and how to resolve network-caused issues effectively.

    Linux and Unix

    If you come from a Windows background but you want to be an SRE, there’s no getting around it: You’ll need to learn how to work with Linux and other Unix-like systems in addition to Windows.

    That’s because, even at organizations that don’t rely heavily on Linux servers, you’re likely to find that Linux and Unix concepts are deeply embedded within other systems that you have to work with. Most public cloud management tools follow the conventions of Linux CLI tools, for example. So do systems like Docker and Kubernetes, even if you run them in a Windows environment.

    Cloud Computing

    Like Linux and networking, cloud computing is another category of skill that modern SREs can’t live without.

    The reason why is almost self-explanatory: Around 90% of businesses use the cloud, and you can’t manage reliability for cloud environments very well if you don’t understand cloud architectures, cloud networking, cloud data storage, cloud observability and so on.

    CI/CD Pipelines

    SREs don’t typically help to develop software, but they nonetheless need a deep understanding of how software is written and deployed—which, at most organizations, today, is a process that happens via a CI/CD pipeline.

    It’s hard to engineer reliability if you don’t know how to address reliability problems that emerge from application source code or deployment processes. Understanding how CI/CD processes work and which tools drive them is key for virtually every SRE today.

    Quality Assurance and Software Testing Automation

    SREs also don’t usually help to test software pre-deployment. That task falls to developers or quality assurance engineers.

    Nonetheless, understanding how software is tested—and how to use test automation to speed tests and expand test coverage—is a vital SRE skill. After all, the more thoroughly and efficiently your team can test software, the greater your chances of catching reliability problems pre-deployment, when they are easier to fix and pose a much lower risk to the business.

    Security Engineering and Response

    Securing is another domain that SREs don’t “own,” but where they nonetheless require significant skills.

    Indeed, good reliability engineering makes security a priority, and vice versa. SREs who don’t understand security fundamentals are at risk of implementing reliability solutions that are effective from a reliability standpoint, but not necessarily secure.

    DevOps

    Although SREs are not DevOps engineers, SRE and DevOps are closely related domains. SREs at most organizations today will be expected to understand DevOps concepts and, in many cases, work alongside DevOps teams.

    So, plan to master DevOps skills as part of your site reliability engineer skills acquisition strategy.

    Incident Management

    Perhaps the single most important type of skill for SREs to learn is incident management. Although many roles may participate in incident response, SREs usually take the lead in organizing the incident response team, communicating with stakeholders and devising the best strategy for resolving each incident as quickly as possible.

    This means SREs should know how incident response roles are structured and understand incident response concepts. They should also be familiar with incident response platforms that automate the complex processes required to ensure rapid, effective incident resolution.

    Managing Postmortems

    In addition to overseeing incident response, SREs may be tasked with managing post mortems. Knowing how to run a postmortem—as well as when a postmortem is necessary, and when it makes sense to use a “blameless” postmortem approach—is an essential SRE skill.

    The list of SRE skills could certainly go on. Above are only the most fundamental types of skills SREs will need for most modern environments. But if you’re just starting out on your journey to becoming an SRE, the nine skill domains described above are a good place to begin acquiring the knowledge you’ll need to excel in an SRE career.