devopsy.kinsta.cloud

Tag: SRE

  • Emerging Patterns in SRE Implementation

    Emerging Patterns in SRE Implementation

    First we were told that every company will become a technology company. Then the tides shifted to every company will become a DevOps company. Now the zeitgeist has a new obsession: Every company will become a Site Reliability Engineering (SRE) company.

    Have you been keeping pace? If your company survived the ruthless restructuring of digital transformation and is savvy enough to have surfed the DevOps wave, you might already have heard SRE is the next industry standard on the horizon. And you might be bracing yourself for yet another cycle of emerging best practices. Is the dread setting in?

    Fear not, throwing out the welcome mat for SRE does not require kicking DevOps to the curb. On the contrary, the two are complementary philosophies, which, when harnessed correctly, can usher in a new era of value delivery. Essentially, SRE is a specific implementation of the core DevOps pillars of success that applies a software engineering mindset toward solving traditional operations problems–with a focus on creating reliable and scalable technology. Together, DevOps and SRE work in harmony, striving toward a shared goal of breaking down organizational barriers and delivering better software faster. 

    If you’re ready to give SRE a shot but you are unsure how it best fits within your team’s existing structure, let’s break down the top five team organizational structures that emerge with clients. However, feel free to view these as loose guidelines rather than an exact blueprint. Similar to DevOps, there are diverging views and practices of integrating SRE into a company’s structure.

    Discovering the unique way SRE fits into your team is the most rewarding and fun part of adopting this new system. What matters most here are outcomes. Creating a role for SRE means alleviating some of the pressure on your developers, allowing them to be more productive, and more efficiently and effectively implement the DevOps paradigm. Soon enough you’ll be reaping the benefits of what happens when, as Ben Treynor, Google vice president of engineering said, “a software engineer is tasked with what used to be called operations.”

    The OG(oogle)

    Let’s give credit where it’s due. We wouldn’t have the SRE acronym, let alone the entire discipline, if it weren’t for Google’s innovative ops team. With the aim of bridging the gap between its production and ops teams, Google ops leader, Ben Treynor proposed eschewing the norm of building an ops team solely from system administrators and instead bringing in software engineers who possessed a systems background.

    This model entails creating a dedicated engineering team focused on running and scaling a product or platform. The team should be comprised of highly skilled software and systems engineers who are on-call for the product, and simultaneously directly updating the product code base for reliability and building associated tooling to support the product. Most importantly, the product development team is still responsible for a portion of these operational tasks and may not be allowed to ship new features if reliability falls below agreed upon thresholds (the Error Budget).

    Why Change When You Can Rebrand?

    If your ops, DevOps and platform teams are on board with the SRE philosophy but you either don’t want or don’t need to make significant changes to your product code, this is the approach for you. In this model, your teams can simply rebrand as Site Reliability Engineers, while adopting and implementing the tenets that come with the title. That means shifting the concentration of engineering to focus on improving reliability and scalability. This team would also be on-call, playing a prominent role in the underlying infrastructure, tooling, platforms or day to day support of the product.

    Beware: You need to take seriously the underlying tenets and need for strong software engineering skills on these teams in order to realize the benefits.

    The Inside Job

    Perhaps you’re down with SRE but don’t want it to co-mingle with product accountability. A viable option then would be creating an internal consulting arm–a centralized team solely dedicated to creating and advocating for reliability tools and processes. One of the prominent benefits of this structure is that your team would be able to not only see the forest for the trees but also serve as your SRE gurus, keeping your team abreast of the latest technologies, trends and research in the SRE space. 

    You Get an SRE Engineer!

    For those who want to weave in SRE engineers throughout their entire organization, this approach applies a holistic mindset to the SRE implementation proposition. That means SRE engineers are introduced into cross-functional teams that own the end-to-end lifecycle of a product, from build through decommissioning. From here the road diverges: your choice is to either create an SRE team of engineers who belong to a single capability and are also embedded full time within a product team, or allow product teams to hire their own dedicated SRE engineers. As Cloud and SRE Practice Lead at Slalom Build, I took the former. For us, this approach guarantees bespoke SRE implementation, ensuring right-sized scalability and reliability throughout the product lifecycle.

    SRE? New to Me

    Despite the compounding hype around SRE, it has yet to hit universal name-recognition. If it’s still unfamiliar to you, that’s okay. It’s not too late to raise your hand and get up to speed during this period in which we are still in the early stages of adoption. I assure you that if you reach out to expand your knowledge of SRE and even bring it back to your team, you will be met with enthusiasm and welcoming arms.

    — Rob Cummings

  • The SRE Pressure Cooker: Balancing Velocity Against Risk

    The SRE Pressure Cooker: Balancing Velocity Against Risk

    Delivering fast, reliable digital services today is a lot like Olympian alpine skiing. These services must deftly maneuver a series of perilous passages en route to end users, all while maintaining the astounding speed we now take for granted.

    In an SRE’s world, those passages are today’s increasingly complex and interconnected internet infrastructure through which they must deliver and support cutting-edge application functionality–all while rolling out updates at a relentless pace. It may appear easy to end users, but SREs are working incredibly hard behind the scenes, and as the internet grows more intertwined and treacherous, the risk for catastrophe increases.

    Judging from the results of our second annual SRE survey, striking this critical balance between supporting development velocity while maintaining site reliability is an onerous and stressful task. Here’s a look at some of the key findings, and takeaways for organizations increasingly reliant on this emerging role.

    The SRE Role Is Relatively New

    Even though Google coined the term in 2003, 64% of respondents noted their SRE disciplines have only been in existence for three years or less. This means many SREs and SRE teams—those professionals dually knowledgeable in software development and adapting IT systems to meet the needs of particular software—are still adjusting to their roles and responsibility. Many SRE teams are also small relative to the size and scope of the infrastructure to be managed, and they remain a scarce resource, with many organizations highlighting skills shortages in this area.

    IT Incidents Are the “New Normal”

    Almost half (49%) of survey respondents reported they had worked on a service incident over the course of the past week. In the month of June alone, Google—which is known for having a very mature SRE practice—had two significant outages: One that hit YouTube, G Suite, and several popular third-party apps relying on Google Cloud for their back-ends (including Discord and Snapchat), and the Google Calendar outage just a few weeks later.

    Catchpoint has detected a significant uptick in outages in recent months, likely the result of growing internet complexity which naturally increases the potential for problems impacting digital service reliability. This is a challenge for all companies and their SRE teams, including the biggest and the strongest—the last few weeks have shown us no one is immune. Organizations can help their SREs maintain a healthy perspective by bearing this in mind.

    Many SREs Don’t Have Clearly Defined Service Level Objectives (SLOs)

    This is problematic, because without SLOs, it is nearly impossible to identify what is an incident in the first place. Twenty-seven percent of SREs reported they don’t have any SLOs, and when flags are raised on everything, this leads to more alerts, a greater number of false positives, and naturally, greater SRE fatigue and stress. Our own conversations with SREs have revealed that even among those with SLOs in place, these are often unrealistic or not correlated to end-user satisfaction and happiness—for example, targeted at 100% availability (impossible perfection) when 98% availability may suit end users just fine. If you’re striving for perfection, every little thing that goes wrong constitutes an emergency, even if customers don’t notice. Setting unrealistic targets and not letting customer satisfaction define SLAs is a recipe for SRE burnout.

    Many SREs Feel Burnt Out and Stressed

    According to our survey, 21% of respondents indicated they never experience post-incident stress. However, it was interesting to note the answers to the following question, where we asked what types of changes SREs notice after an incident—in concentration, ability to sleep, mood and more. One would suspect if an SRE never experienced post-incident stress, he/she would select “none” for the second question—but that wasn’t the case. Rather, one third of SREs said they have noticed such symptoms. Sixty-seven percent of SREs who reported feeling stress after each incident, indicated they don’t believe their company cares about their well-being.

    We are aware of other surveys showing SREs are quite happy, particularly given their impressive compensation. While we don’t believe being an SRE automatically makes someone stressed and unhappy, organizations should never assume these professionals know and understand how appreciated they are. There is often a reluctance to discuss topics such as job-related stress, as evidenced by the fact that 50% of the SREs we targeted opted not to complete our survey—perhaps a result of SREs’ predominant hero culture. In addition to compensation, organizations should consider other ways to show SREs—whether they vocalize their stress levels or not—they care about their well-being, such as offering an extra vacation day or other perks after a particularly trying incident.

    SREs Could Benefit from Automation, but Don’t Have Enough

    Another thing we consistently heard in our conversations with SREs is they feel so busy putting out fires today they don’t get a chance to create a better tomorrow—in other words, they lack the opportunity to be proactive and preemptive in their approaches. A key factor in this is the amount of toil SREs face daily—meaning manual, repetitive, automatable, tactical work. Our survey found 59% of SREs believe there is too much of it in their jobs, and too little automation. Nobody strongly agreed with the statement, “We have used automation to reduce toil,” while almost half disagreed or strongly disagreed.

    Considering SRE teams are often small and the talent pool tight, it becomes critical for organizations to automate as much work as possible, including areas such as application performance monitoring. The SRE team at Alaska Airlines recently automated performance monitoring and deep-dive diagnostics, and as a result has been able to reduce their alert noise by 92%. Ultimately, this has freed up the team to focus on real issues versus wasting time investigating false positives. This has helped them bring down the average mean time to detection for real issues from hours to less than 10 minutes.

    Conclusion

    The stage has been set for SREs to experience significant stress, at a time when organizations need their exceedingly rare skillsets the most. As internet complexity increases, it is critical for organizations to continually nurture, encourage and motivate these valuable team members. This means understanding and empathizing with the intense pressures they’re under; having SLOs well-defined, realistic and meaningful; leveraging soft skills to communicate effectively and non-monetary incentives to show appreciation; and providing them with needed automation so they can focus on proactive improvements that advance the business.

    — Mehdi Daoudi

  • SREs: The Happiest – and Highest Paid – in the Industry

    SREs: The Happiest – and Highest Paid – in the Industry

    According to the 2019 StackOverflow survey completed by 90,000 developers around the world, whether you are already a Site Reliability Engineer (SRE) or thinking about becoming one, looks like you are onto something.

    SREs, also called DevOps specialists, are a pretty unique type of talent; one that has knowledge in both software development and system infrastructure. And because the world of infrastructure is so vast, these professionals also need to juggle networking, security, system administration, hardware and anything else that could make an infrastructure unstable or down (yes, I am looking at you, A/C). That makes any day at work unique.

    SREs, for the most part, are in charge of making sure that your favorite website apps are loading, and fast. Their work is not customer-facing but highly critical. According to Gartner, the average cost of IT downtime is going anywhere from $5,600 per minute to—when you work for Amazon—$2 million for every minute down.

    That’s one of the reasons why SREs are among the highest paid in the industry, well, right behind their fellow managers! They also rank among the ones with the most coding experience—it takes time and hard work to get there.

    And all of this leads to happiness! With an interesting job and high pay, SREs satisfaction is among the highest in the industry, and they rank in the top three of the ones not actively looking for a job. SRE for a day, SRE forever!

    But here’s the problem: There aren’t a lot of SREs out there and hardly any computer science programs cover the topic. A recent tweet from Target software engineer Jessica Schalz illustrates the point very well:

    As TechRepublic reported, these professionals are among the most wanted in the tech industry and 33% of IT leaders report having a hard time hiring profiles with these skills. A potential reason for this lack of talent might come from the fact that the profession is failing at attracting about half of our population: women. SREs are 30 times more likely to be men than women. Training non-bearded SRE talent is possible. The Holberton School, a coding school I co-founded where 40% of the students are women, is leading a lot of students to be hired as SREs.

    And the good news is that we are not alone: CloudFoundry and Infrastructure Masons are also taking action to move the needle. And industry leaders such as Tammy Bütow and Abby Kearns are showing the way.

    As former software architect Eduards Sizovs pegged it: Great developers are raised, not hired. The industry is spending tons of money and energy and trying to attract the best talent when it would often be more efficient to take juniors that can grow to their need.

    “There is a new generation of developers of all races, genders, backgrounds coming up who can do these jobs,” said Max Johnson, DevOps engineer at Pypestream. “Businesses should widen their hiring scope and consider non-traditional sources that train as good, if not better, talent and coach them further for these highly skilled jobs.”

    LinkedIn is among the first to understand the value of investing in people—so it’s probably not a coincidence that it’s among the best places to work in the U.S. From an internal, employee-led perspective, the company has developed a number of in-house programs for both career and technical development. For example, its Women in Tech (WIT) organization leads a number of programs to connect, inspire and grow women technologists. From an external standpoint, with its REACH engineering apprenticeship program, LinkedIn is redefining the way it identifies, recruits and trains talent. REACH is an apprenticeship program that creates opportunity for every individual with the passion and potential to either develop or restart their career in engineering. The program guides participants through various levels of technical roles at the company, with the end goal being that each individual builds their skill set and gains the experience needed to become an engineer.

    “Reinventing our approach to recruiting and developing junior talent has been a great way to expand our funnel and has brought forward more diverse perspectives, forming a more robust team,” said Casey Brown, senior manager, SRE at LinkedIn and core team member of the REACH program.

    As you have probably understood, SREs are a niche talent (43% of them even picked Vim as their favorite editor). But as I recently tweeted, they are ruling the world, and more of them are needed. Let’s get creative to train and hire more of these amazing professionals!

    — Sylvain Kalache

  • SRE: How Do You Get to Blameless?

    SRE: How Do You Get to Blameless?

    Site reliability engineering (SRE) is the companion to DevOps. If we take my normal divisions of DEVops versus devOPS, traditional DevOps is mostly the developer side, and SRE is mostly the Ops side. This isn’t a perfect comparison, but it is a good starting point, and for this blog, I’ll run with it.

    While DevOps focuses on getting releases out consistently, quickly and with minimum disruption, SRE focuses on putting those releases into a robust infrastructure. Tracking application issues, regardless of whether you’re using DevOps, always requires information about the software and hardware infrastructure. SRE takes the proactive steps to try and make sure the environment is stable and designed to host the applications being thrown onto it. When that process fails, SRE attempts to identify the code/architectural problem and resolve it without blaming staff members—hence, the idea of “blameless.” It’s about the environment and improving it, not about the individual and punishing them.

    There are a lot of interesting parts of SRE that most IT groups could look at and see as a good place to start taking advantage of. Closely tracking errors in a cross-section of IT interest areas—by application, by infrastructure, by toolset—means problem areas can be definitively identified and fixed. But there is more behind SRE. In this example, once tracking of where errors occur more frequently is in place, a selective reduction in deployments using that piece of the IT puzzle can go into effect until the issues are resolved. This is not unlike what we are doing today, but it is formalized and has measurements around it that generally are not in place.

    Which brings us to Blameless, a company that just came out of stealth to champion SRE. While some large organizations with large application footprints are using SRE already, Blameless is offering a platform to make SRE more accessible to companies without large staff and application footprint.

    We took a bit of time to talk with Blameless last week, and their vision is a strong one: Address directly what DevOps addresses more indirectly. Build and environment that can measure and improve the entire application ecosystem.

    If nothing else, the idea of error budgets is intriguing. Counterintuitive at first glance, the idea of saying, “Piece X gets Y amount of issues before we start backing off of it or raise the priority to fix the underlying issues,” is a solid one.

    If your organization is looking for a mechanism to improve reliability and stability while reducing time to resolve issues, SRE in general and Blameless specifically are worth a closer look.

    — Don Macvittie

  • Blameless Launches DevOps Platform Based on SRE Principles

    Blameless Launches DevOps Platform Based on SRE Principles

    Blameless has emerged from stealth to launch a platform for managing DevOps based on the site reliability engineering (SRE) best practices defined by Google.

    Fresh off raising an additional $20 million in funding, the Blameless platform provides across-the-board visibility into the complex network of application programming interfaces (APIs) and applications distributed across an organization, said CEO Ashar Rizqi. Specific modules within the Blameless platform address incident resolution, reliability insights dashboards, postmortems, service level objectives (SLOs) and automatic error budget calculation.

    Blameless takes its name from an SRE principle that no one should be held at fault when complex systems fail. Rather, the assumption should be that inherent complexity of the IT environment is at fault. While DevOps is more of a philosophy surrounding the management of DevOps, SRE is emerging a more prescriptive instance of implementing DevOps that Google is now promoting. The goal is to make sure that whatever the issue, the right people to address it are brought together at the right time, said Rizqi.

    The Blameless platform was created by Lyon Wong, who now serves as the COO of Blameless. Previously, Wong was a partner at Lightspeed Venture Partners, founder of S28 Capital and a product manager for Microsoft Windows. Rizqi is a former head of core platform at MuleSoft and SRE manager at Box.

    Rizqi noted that while most organizations are trying to embrace DevOps principles, they often lack the tools needed to build the workflow processes required. All too often, organizations rely on little more than a Wiki document to manage their response to an incident. The Blameless platform is designed to provide access to an integrated set of tools for managing everything from the incident itself to the impact that event had on the business, he said.

    It’s still too early to say with any certainty how widely SRE practices will be adopted. This far, most organizations that have adopted DevOps have tended to prefer to define a set of processes that are customized to their environment. But there are just as many, if not more, organizations that have found it daunting to get started. Platforms such as Blameless represent an opportunity for those organizations to embrace DevOps using a more prescriptive approach.

    Of course, a big part of the reason that DevOps has been embraced unevenly by various departments within the enterprise is the lack of a consistent approach to implementing DevOps at scale. Teams that have embraced DevOps with enterprises have tended to create their own individual processes using a wide variety of tools. Many of those organizations have also found it challenging to get IT operations teams to embrace those processes because they require an approach that can be implemented consistently across multiple projects at scale. SRE provides IT operations teams with a framework that Google has already proven works at scale.

    Regardless of the path chosen, the one thing that is for certain is more agile approaches to managing IT operations will be required to keep pace with the rate at which applications are now being developed and deployed.

    — Mike Vizard

  • How SRE Creates a Blameless Culture

    How SRE Creates a Blameless Culture

    People in tech love to praise failure as essential to innovation. But at far too many organizations, when failure happens, it is punished. When an IT system goes down, an individual or a team is immediately ID’d, blamed and shamed. This is counterproductive. Blame damages companies by creating a play-it-safe atmosphere and stifling innovation; even worse, when mistakes are made, they are often hidden.

    Fortunately, more and more organizations are recognizing the damage that blame can do and embracing a blameless culture. This is a big shift, yes, but companies can make the move to blameless—and make it stick—if they bring site reliability engineering (SRE) best practices to their technology teams.

    Why Blame is Bad

    I’ve worked at several fast-growth companies that suffered under a culture of blame. At one, we were scaling quite rapidly. Change was constantly being introduced to the infrastructure, which meant that it broke all the time.

    Because we had customers running their core business on our systems, our reliability requirements were very high—so high that for a while we stopped feature development altogether, in fear that new features would bring our systems down.

    This was difficult enough, but the culture of blame made it worse. There was no psychological safety net and fingers were pointed constantly. There was an us-versus-them mentality—engineers versus non-engineers, developers versus operations, developers versus developers. Developers dedicated way too much mindshare to protecting themselves and their domain.

    In a blame-rich environment, developers who should be delivering concrete results spend too much of their time in meetings trying to justify their work. There is a culture of retribution for mistakes. This leaves little room to learn and grow from failure because developers get harassed by management when they do make mistakes, or they get fired.

    Your best people—those empowered to make decisions—see their responsibility undermined and go elsewhere. Creativity and innovation slow to a crawl. Meanwhile, competitors run ahead.

    Why Blameless is Better

    In a blameless culture, everyone feels safe and no one is afraid to make mistakes. It’s a psychologically secure environment, where true DevOps can happen. Developers feel confident enough to express their ideas and take chances. Development folks and operations folks collaborate well, and everyone is aligned on the problem.

    The work environment is positive and workers are assumed to be doing their best. When mistakes happen, people aren’t blamed. Instead, any error is viewed as a manifestation of an underlying vulnerability in the systems. Attention is focused on fixing those vulnerabilities and, as a result, systems are constantly improving.

    How to Get to Blameless

    Let’s say you just had an outage and your database crashed. Don’t start by placing blame on a team or individual. Rather, start by conducting a postmortem. Ask questions and find the answers: Why did the database crash? Because Alex pressed the wrong button. Well, why? Because we didn’t have a system in place to enable automatic checking or a review process. So why did we not have that?

    Now you’re getting to blameless and you’re reaping the benefits. You’re having a productive conversation and you’re focused on fixing your systems, not affixing blame. It’s not about Alex pressing the wrong button; it’s about how your organization is not set up to ensure success. It’s about this vital opportunity to learn from a mistake and get better. Don’t fire Alex. Put a solution in place so the problem doesn’t happen again.

    How SRE Can Help

    To get to a truly blameless environment, you need to implement SRE best practices. Why? Because SRE creates ultra-scalable and highly reliable software systems, by building in automation and tooling—moving the human element out of the picture and focusing on the systems.

    One of the founding principles of the SRE movement is blamelessness. By implementing an SRE team and establishing SRE practices, your organization can move from the traditional operational model of break-fix to an environment that applies a development approach to IT operations, with the goal of improving reliability via automation, continuous integration and delivery.

    For this to happen, you will need a SRE champion to quarterback your SRE effort. You will also need organizational buy-in to blamelessness. It has to be practiced top-down and have the full support of all leaders in the organization.

    Tooling is critical, too. Look for tools and dashboards that can quickly do postmortems and find action items that need attention, that can identify the most commonly impacted services, products and customers, and the contributing factors that most often lead to incidents. The right tools will help you resolve incidents faster, by providing a clear overview of metrics related to incident identification and the actions that need to be taken.

    You want to move fast and be innovative. But can you do it reliably? With a blameless culture, you can. You’ll have a safe environment that attracts and retains the best talent. You’ll move quicker, because you’ll be focused on fixing systems that break and finding out why—not the blame game. By delivering a higher degree of reliability, you’ll build greater trust with your customers.

    You’ve heard the saying: Failure is not an option. In a blameless environment, failure is always an option—because it means that systems are always improving and innovation is always happening.

    — Ashar Rizqi

  • Survey Finds Wide Gap Between DevOps Adoption and Success

    Survey Finds Wide Gap Between DevOps Adoption and Success

    A recent HBR survey shows there is work to be done to advance DevOps in organizations

    A survey of 654 subscribers to the Harvard Business Review found that while 86 percent of respondents said it is important for their organizations to be able to build and deploy software quickly, only 10 percent said they are successful at it.

    Conducted by Harvard Business Review Analytic Services on behalf of Google, the survey found nearly half (48 percent) of respondents said their organization always relies on DevOps methodologies when deploying software, while another 21 percent rely on DevOps selectively. An additional 8 percent said their organization plans to adopt DevOps practices.

    Among those respondents employing DevOps practices, the top benefits cited were increased speed to market (70 percent), productivity (67 percent), customer relevance (67 percent), innovation (66 percent) and product/service quality (64 percent). Despite those benefits, however, most organizations still have a significant amount of work to do in terms of implementing best DevOps practices—the three biggest hurdles organizations face are organizational silos (50 percent), legacy technologies (49 percent) and resistance to change (46 percent).

    Melody Meckfessel, vice president of engineering for Google Cloud Platform (GCP), said these issues are precisely why Google is sharing how it operates in the form of site reliability engineering (SRE) principles, which are a prescriptive set of guidelines for implementing best DevOps practices. Not every organization needs to implement those principles to the same level, but just about every organization can benefit from adopting a set of well-defined engineering principles, said Meckfessel.

    The challenge many organizations face is the fact that most of the IT staff are not engineers by trade. Most IT staffs are made up of administrators who tend to have limited programming skills. But Meckfessel said even IT organizations that don’t necessarily have a lot of engineering expertise still can benefit from automating operations based on the SRE principles.

    Many of those organizations will need to slow down the rate at which they are building applications today to be able to speed up tomorrow. It takes time for organizations to master the cultural change required to implement DevOps practices. But once they begin to master DevOps, the benefits to the organization compound quickly. Meckfessel said two of the most important attributes of DevOps is adopting an immersive approach to training and establishing a no-blame culture within the IT organization when it comes time to conduct post-mortems. That latter goal is especially difficult to achieve sometimes, because most interactions between developers and IT operations teams historically have been less than positive.

    Meckfessel also noted it’s critical now for organizations to incorporate cybersecurity into its DevOps processes as well, which requires not only additional training for developers but also closer alignment with cybersecurity teams.

    It’s now only a matter of time before the complexity associated with emerging technologies such as microservices based on containers force many organizations to dive deeper in DevOps. The real challenge now is charting a path of least resistance that also generates the maximum level of benefit for all concerned.

    — Mike Vizard

  • Google Rolls Out Additional Cloud Certifications

    Google Rolls Out Additional Cloud Certifications

    Google is moving to address a chronic shortage of cloud skills necessary to advance DevOps.

    Specifically, Google plans to add Professional Cloud Developer (GA), Professional Cloud Network Engineer (Beta), Professional Cloud Security Engineer (Beta) and G Suite certifications (GA) alongside Professional Cloud Architect, Professional Data Engineer and Associate Cloud Engineer certifications that already exist.

    Google also plans to localize its cloud learning and certification programs in Japanese, French, German, Spanish and Portuguese to make them accessible to more people.

    Rochana Golani, director of Google Cloud learning and enablement for Google, said Google is trying to increase the number of qualified engineers available to manage IT infrastructure as code. Most of the IT operations community is made up of IT administrators that don’t have much in the way of actual programming skills. The Google cloud certifications are all steeped in the site reliability engineer (SRE) principles that Google defined in 2003. As a more prescriptive instance of a DevOps practice, the goal is to make the skills needed to successfully transition to DevOps more widely accessible, said Golani.

    Google is especially keen to increase the number of networking and security professionals that have demonstrated cloud skills, as the critical role these functions play in cloud application environments is often underestimated, added Golani.

    Those challenges likely will only increase as organizations increasingly embrace microservices based on containers running on Kubernetes clusters that eventually will be tightly coupled to serverless computing frameworks. In fact, the first era of the cloud was defined by a comparatively simple replacement of virtual machines from on-premises IT environments with similar virtual machines residing on a shared public infrastructure. The next wave of the cloud is being defined more by so-called cloud native applications constructed using microservices. That approach is inherently more flexible from a business perspective, while paradoxically being more difficult to manage from an IT perspective.

    On the upside, however, DevOps in general and SRE disciplines more specifically enable IT organizations to operate more hand in glove with the business as it grows and evolves.

    Of course, none of that is achievable without the skills required to achieve that goal. Savvy IT professionals naturally will pursue certifications that enable them to earn higher salaries. But organizations that encourage their personnel to achieve those certifications should be able to at the very least engender more loyalty among IT professionals they help advance their skills.

    In the meantime, it should be apparent that transformation is all but inevitable. Not every workload will move into to the cloud. But the principles applied to managing workloads in the cloud will be increasingly applied within on-premises IT environments. Those on-premises IT environments may not be as programmable and resilient as a public cloud, but the need to process data locally is not going to go away anytime soon. The challenge IT organizations are facing today is how best to define a set of best DevOps practices that can span both environments.

    — Mike Vizard

  • The Importance of Soft Skills in Engineering

    The Importance of Soft Skills in Engineering

    In college, professors regularly told us that in the real world, we need to be able to work in groups. As a young skeptical undergraduate, I naturally rolled my eyes every time I heard this. Most of us hated group projects. More often than not, people would go missing or not do their part with little to no consequence.

    After nearly five years of reflection on my college learning journey, I realize that while my professors were right about group work being important in the professional world, they were wrong about the way we go about working in groups. Instead of teaching us what is commonly called group work, I’ve found that honing my soft skills has best served me in group project settings.

    Google defines soft skills as, “Personal attributes that enable someone to interact effectively and harmoniously with other people.”

    Group projects in university teach us about scrum and agile principles—the concept of creating tasks and delegating them with everyone on the hook (to a certain degree) for the delivery of the project. It’s very clean, it’s very simple. It’s almost like a business contract; there is very little human emotion involved. It’s entirely possible to successfully deliver a project while being completely disgruntled at your team and the process.

    I’ve found that being forced to “learn” group work skills often leads to it being devalued as you leave school and step into your first professional role. This is ultimately detrimental over time, as soft skills are vital to successful engineers (especially as they become more senior).

    When I joined LinkedIn right after graduation, I knew it was my role to help build reliable infrastructure. I instinctively had an “us versus them” mentality with developers. Over time (and through several unfortunate experiences), I realized that this doesn’t actually work: I found myself in need of building some non-technical skills.

    Unfortunately, the “real world” is not cut-and-dry business transactions, as I came to learn. Saying “no” to developers and then asking them for things that site reliability engineers (SREs) consider sensible just doesn’t cut it. You have to be able to effectively communicate with others. Over the last five years, I learned a few soft skills can go a long way in working with others.

    Empathy

    One of the greatest traits you can have as an engineer is empathy. There’s a great quote from theologian Ian McLaren, who said, “Be kind, for everyone you meet is fighting a hard battle.” As invincible as we think we are, at the end of the day, we’re all human and it’s easy to forget that. Co-workers may have family or personal issues that they don’t speak about, or they may struggle to understand a concept. Empathy can help you better communicate with others. Approaching your colleagues with a kind tone rather than a grumpy or aggressive tone goes a long way in building a harmonious work environment.

    During university, I had a team member in a group project who was not able to fulfill his commitments by the required time. My immediate reaction was to let him know I was upset at him. Little did I know that his wife was ill and he was struggling to keep up with his study commitments. My frustration toward him didn’t help the situation and, given the extra context, is actually insensitive. Asking your peers how you can help them is far better than lashing out when things don’t go according to the plan. 

    Clear Communication

    One thing that struck me as an engineer is how many disagreements are based on communication failures. This includes assumptions, misreading intentions or not being specific enough with asks. Many of us work in poor-quality conference calls and text-based conversations. It’s very easy to lose context and intent in digital communication and equally easy to disagree because something wasn’t as clear as it was intended to be.

    Part of my role at LinkedIn is to review all site issues on a daily basis and ensure that we’ve addressed all concerns. When raising these concerns (usually during a phone conference), good intent can be lost quickly. Over time, I learned to ensure that my colleagues understand where my concerns come from and offer support to resolve any outstanding questions.

    Very simply put: Over-communicate. Make sure your tone and intent are very clear to the recipient and be sure to add as much detail to the feedback or request that you’re making.

    Humility

    One important thing I’ve come to learn is that, as SREs, we are expected to have all the answers; after all, our job skills are multi-faceted.

    Over the last four years at LinkedIn, I’ve worked on many different teams, from the traffic stack to the database platform, and amassed a wealth of knowledge on how those systems work—to the point where I could almost be considered an expert for all of those pieces of infrastructure. Naturally, people come to me asking questions about these subjects and unfortunately, I don’t always have all of the answers. It’s very easy to end up in an imposter syndrome position where you feel inadequate because you don’t have the answer to every question.

    This is actually counterintuitive. While great engineers have a solid understanding of the infrastructure, tools and teams around them, the best engineers will learn who the best subject matter experts are and will direct relevant queries to that person.

    This becomes a critical skill as you scale as an engineer. If 30 people are coming to you asking you about things that aren’t in your subject matter expertise, you can get slowed down and may not be able to give the best possible advice, effectively slowing the person asking the question. Knowing when to refer someone and whom to refer them to is part of being a great engineer.

    Mentorship

    The general career growth chart for SREs roughly looks like this:

    • You do toil tasks
    • You work on projects
    • You lead projects and delegate to others
    • You invisage projects and ask people to lead them
    • You create a team or company vision and get groups of people to lead them

    As you become more senior, your role becomes less dependent on day-to-day execution of toil and more focused on empowering those around you. You become less directly in control of the outcome of a project and your success comes from the success of those to whom you’ve delegated. This is where soft skills become important.

    Over the past two and a half years, I’ve been lucky to have a mentor who has helped my personal development. He has been a great role model, especially in teaching me how to lead and influence others. He has often said to me that you need to break through the eight-hour barrier. What he means is that you need to up-level those around you so that your impact on the business goes beyond just your work. Your contributions are also to others to whom you’ve scaled your knowledge and skills. As well as technically leading my team, I have the privilege of being a mentor to others at LinkedIn.

    These four skills are just a small set of a large base of soft skills. Mastering each of these takes time and practice, but ultimately makes you both a better engineer and person.

    — Michael Kehoe

  • DevOps Chat: SRE w/ Stig Sorensen of Bloomberg

    DevOps Chat: SRE w/ Stig Sorensen of Bloomberg

    Many people say that site reliability engineering (SRE) is “putting the Ops in DevOps.” In this DevOps Chat, we speak with Stig Sorensen of Bloomberg’s SRE team. Stig tells us how SRE is helping Bloomberg keep pace with the fast moving world of news and information.

    Great discussion with a real-life practitioner about how SRE is being deployed in a real-world company. As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Audio

    Transcript

    Alan Shimel: Hey, everyone. It’s Alan Shimel and you’re listening to another DevOps Chat. We’ve got a really good DevOps Chat lined up today on a really hot topic, and it’s SRE, site reliability engineering. And my guest is Stig Sorensen, who heads up – a member of the SRE team over at Bloomberg. Stig, welcome to DevOps Chat. 

    Stig Sorensen: Thank you very much. I’m glad to be on.  

    Shimel: So let’s get formalities out of the way. Stig, give our people your title and background here so that we can get started.  

    Sorensen: Okay, so my name is Stig Sorensen. I am the manager of the production visibility group at Bloomberg, responsible for building our CMDB tools and also building telemetry assay as a platform. I’m also one of the main evangelists for rolling out SRE as a role and as a – a new movement across Bloomberg. 

    Shimel: And Bloomberg is, of course, no small company, so rolling out this new role, this new way of looking at Ops and so forth, is – is of course not a – you know, it’s not something we should take lightly or think it’s easy. 

    Sorensen: Well, that’s very true. It’s – across our 5,000 engineers, it’s – that supports everything from our commercial website to – to market data feeds, to obviously our main product, the Bloomberg Professional Terminal which hundreds of thousands of the key influents around the world rely on. And, with various states and various different tech stacks, it’s actually quite – quite a challenge. 

    Shimel: Excellent. So, so Stig, why SRE, why now at Bloomberg? 

    Sorensen: So SRE in, like, what we do isn’t really new to Bloomberg. It’s always been sort of the engineers that own production and sort of own the – the availability of production, and own the process around it.

    What we’re sort of trying to do now is sort of really more divide the responsibility between the application engineers and the SREs such that we can have people that are experts on – either of them, but it’s sort of unfair in today’s complex world for everyone to know everything from how a – how an instrument trades on the street to figuring out how to – how to run and manage a high availability system. Right? So it’s more fitting it into infrastructure and making people experts. 

    Shimel: So an interesting point you bring up, Stig, and you said, we’ve always been doing this kind of thing. And it’s true. A lot of what we now today are calling SRE isn’t necessarily brand new in terms of task and responsibility. It – it may be there’s more of the packaging and some of that mixed in.

    When – when did Bloomberg, let’s say, start referring to this role, to this, you know, needs as, you know, quote/unquote “SRE”? 

    Sorensen: So we sort of started small about two years ago and it’s really sort of – really caught fire in, I would say, the last year where we really made it sort of a more firm-wide. It’s always been pockets of where it’s sort of been done, but then we really, like, part of the puzzle is really sort of internal branding as well in terms of labeling the – not labeling the people, but identifying who is there creating the right environment for these people to learn, to share ideas. Right?

    I think, as you said, the – it’s not necessarily the new thing to do. What’s new for us is sort of really trying to do it in a more – more standard way and a more persistent way across the – across the firm. 

    Shimel: Yep. So Stig, you know, 5,000 engineers. How many are, you know, quote/unquote “on the SRE team”? 

    Sorensen: So they – we don’t actually have a centralized SRE team today. We sort of rolled it out in a way where we aligned the SRE teams with the application teams. The reason for that is really a lot of – in order for the SRE teams to be successful, the applications also have to change. Sort of separating the responsibility too far away makes that harder.

    So we sort of put them parallel, side-by-side, two equal teams with the different responsibilities and – and that’s really sort of how we kick start the teams. Most of the SRE – a lot – quite a few of the SRE teams are seeded from – from the application teams and a lot of, like, the – the core infrastructure teams and so on.

    So all-in-all we have about 75 people that are in sort of the – in the sort of teams that we have sort of – the teams that are SRE teams. As I said, we have a lot more people doing it across the firm, but they’re doing it as a part-time – part-time job. 

    Shimel: Got it. And so Stig, you know, SRE – this concept of an SRE – as you said, though they – we’ve been doing these kinds of things forever, the idea of calling them SRE and labeling them as such is relatively new.

    Is there any particular training or skill sets that you’re asking people to – to make sure they’re, you know, proficient in in order to be quote/unquote “an SRE”? 

    Sorensen: That – I think – I think sort of – the answer is that we are hiring and we’re trying to hire many, many more. They sort of come from two different camps. It’s either people within operational or system admin background that’s sort of picked up programming and moving that way, or we have application engineers with an – with a more active view towards systems and towards availability – because we see SREs as software engineers just doing something – building a different type of software.

    And I sort of jokingly tried to sort of classify SREs versus application developers in a way, sort of – I see application developers as inherently optimistic, they think everything will work all the time, and SREs are sort of the opposite, they think everything will break all the time. And that sort of relates – I think it’s more of a mindset than necessarily the skills. Skills you can always learn. And then I think some – as long as you have the right mindset, the right aptitude to do it, that – well, that’s what’s key to us. Right?

    And I think there is – I think there’s way more SRE jobs than actually people that have done SRE before out there now. So we can’t go out and try to hire the – the ready SRE. We need to figure out how we get smart people with the right aptitude and training them.

    And people mention training, we sort of try to do it in sort of like a lot of meet-ups, a lot of sort of internal meet-ups, external meet-ups, attending conferences to figure out what’s happening in industry, talking at conferences such that we also show what we’re doing.

    And I think it’s – I think it’s really sort of a lot of on-the-job training, working with the application team, working with other SRE teams, and you realize that you’re not alone. There’s always – there’s always someone doing something very similar to you. You don’t need to invent the wheel every single time.  

    Shimel: Yeah, and Stig, we should not underestimate the importance of what you just said there. With a new field like SRE – and we saw this in DevOps, early on in DevOps – because there’s no sort of codified best practices, right? There’s the book from the Google team, and there’s other stuff coming out now but there really isn’t a – a best practices. There’s emerging practices. And so the idea of peer-to-peer sharing and – shared learning is – is really important to continue advancing, you know, the state of the art, if you will.

    In addition to my DevOps dot com duties, I’m also one of the co-founders of DevOps Institute and this is exactly what we – the fact pattern that we saw in DevOps. And it’s the same fact pattern we’re seeing in SRE and frankly it’s one of the reasons why we are working, and expect to have out soon, an SRE course that kind of starts, you know, pulling together all of these emerging practices and having a common body of knowledge around it. 

    Sorensen: I think that’s great, that – that we’re sort of trying to standardize these things, but – because a lot of it also becomes sort of – you have to be flexible when you roll it out because not every application or every system can be rolled out – can – can get an SRE team and roll out in the same way because there – there’s different maturity of the systems, there’s different problems. Right?

    Like, in terms of, like, do you focus on monitoring and a production environment or do you focus on more sort of the classic DevOps things and how you get things into production. Like, you really have to look at it from a holistic point of view. The goal of the SRE team is to improve availability of production. There’s multiple ways to do it. You could pick one and make it better, you’d probably – you’d probably improve it. Right?

    And the – thinking – like, it has to do everything to make a difference is also probably not the right thing to think about. 

    Shimel: Yeah. Absolutely. So, Stig, let me – let me kind of pivot here a little bit and let’s talk about – what’s in it for Bloomberg with SRE? Why are they supporting you? 

    Sorensen: Well they – it’s – I think it’s really just looking at the services we provide. Right? Like, we’re providing real-time financial information to 300,000 of the most influential people around the world and it’s not really just about availability, it’s also about performance and latency of this – the data going around the world. Right? We – we process something – what, like 100 billion market data picks a day and one-and-a-half million news stories a day. Right? So it’s – so in order to keep all these pipelines going, it is actually quite a big job. Right? And I think just – just making sure that we are available and don’t influence the financial markets by not being there. 

    Shimel: Yeah. I mean, that – you know, and that brings home the, you know, we’re not – sorry, Bloomberg doesn’t sell widgets that no-one cares about. Downtime and performance there is – is, you know, teamed up or, you know, ingrained with financial market performance and reporting. You can’t afford – you can’t afford downtime. You can’t afford, kind of, mess-ups.

    But Stig, we spoke a little off-mic about this and it’s a topic I want to bring up is, you know, at Bloomberg, you guys have been doing this SRE thing now and you think it – you’re making progress and it’s good, but we really haven’t done any sort of formal ROI analysis or, you know, similar type of work. Correct? 

    Sorensen: Yeah, no – that’s very true. It’s – it’s probably one of the biggest challenge in terms of, we have support and everyone believes it’s the right thing to do. You have the gut feel that this is – we have to do this to make it better or to be – or to keep the service going we’ve always had.

    But I think it’s sort of a – for us it’s – for us, like, measuring – _____ measuring availability is actually not black and white. It’s not like, how many – how many failures you had on a website. Right? Because if you are a certain market player and the real-time market data is delayed by a few – by one millisecond or hundreds of milliseconds, it could make a big difference for you. Right? And it’s – is that complete failure or partial failure? And, sort of, all these things in between.

    It’s sort of something we’re – we’re really working on to try to define better metrics around it such that we can track our progress effectively to see if – if we had – if we – if we are succeeding. It’s – if you go by the gut feel, it only lasts so long but at some point you need to really ask yourself, you need to make sure that the return is there 

    Shimel: Absolutely. Stig, I mean, and let’s be clear. This is not a Bloomberg issue at all. I – you know, we haven’t really seen it across the board yet. You know, formal ROI type of analyses for SRE, but as you said, it – you know, it obviously makes sense or, you know, logical to think that it’s good for – for organizations and I think this is why we’re seeing so many organizations starting to adopt this.

    Another thing I wanted to touch on, though, Stig, was the idea of, again, paralleling the DevOps experience, originally everyone said, oh, DevOps is great for unicorns. It’s great for Google, it’s great for Facebook, it’s great for the Twitters of the world, but it doesn’t really work on the horses. Right? On the just regular, large enterprises.

    SRE, well, yeah, that’s a Google thing and it’s great for Google but is it going to work for Bloomberg. Right? Or similar kinds of things. And, you know, you’re living proof, if you will. But every road, Stig, there’s – it’s – you know, I call it the DevOps cha-cha. It’s never linear, one step in front of the other and it’s just, you know, smooth sailing. There’s always a failure here, a lesson learned here, a step back, two steps forward, one-two, one-two-three.

    Give us, you know, I’m sure you could give us plenty of successes. Give us a failure or two that you guys really learned from. Lessons learned. 

    Sorensen: Cool, yes, it’s – I think sort of in – the lesson learned in failure of the transformation was really, I think – some of the teams that was – that was labeled SRE was given too much operational work such that they didn’t necessarily have enough bandwidth to get themselves out of the quagmire they were in. And that sort of created, like, a new pattern in terms of that, like, SREs are operations, and they’re not. SREs are developers. But developing things to get rid of operations work, but if you don’t have time to develop, you’re never going to get out of it.

    So I think that’s probably about the biggest lesson learned in terms of – in terms of the building the culture and getting sort of the – the optics to be right internally. And sort of in the same way, I mean, sort of like, it was also, sort of, SREs within the application teams and not having dedicated teams was also quite a – sort of an anti-pattern that we saw a bit where it’s easier to steal the SRE’s time to do application work. And then again, we don’t actually move forward. So the – so that’s sort of probably the biggest failure.

    I think also we need to make sure that we have a good set of base tools and base things that – that can be shared between the teams such that _____ everyone needs to – needs to build everything from the ground up. And we probably – we probably started sort of in silos where sort of parallel things were built but it improved. It improved – it improved the production quality but it didn’t necessarily make us go faster in terms of rolling out SRE and so it – we sort of had to step back and sort of standardize a bit more of tools and make sure we had central ownership of some of these tools rather than individual teams owning them. 

    Shimel: Excellent. Stig, we’re coming up on the end of our time here. I wanted to kind of just ask you, so, you know good progress, off to a heck of a start, a fine beginning. Where do you see the future of SRE at Bloomberg specifically and in the greater market, let’s say, if you had a crystal ball, over the next three to five years? 

    Sorensen: I think – I think internally here, right, like we’re – we’re trying to grow significantly so for us to just be able to – to both convert people internally from an application to an SRE and also hire the people.

    And I would hope for us and for larger companies like us where there’s a transformation, that it becomes – becomes more standardized such that we can have a centralized org that is the SRE org, and have sort of more of a – where we can – where we can scale – scale and support more teams whereas now it’s really one-to-one and – not necessarily one-to-one in terms of people but one-to-one in terms of teams.

    And I would hope the industry in general also starts to think about the people that do not run on the big cloud providers and sort of actually own everything from the ground up, like us, and I would hope that more companies that are like us get into this such that –

    [Inaudible due to radio noise]

    – that it would be more pools and more things to support us. Right? 

    Shimel: Sure. Sure. Well, Stig Sorensen of the SRE team at Bloomberg, I want to thank you for being our guest on today’s DevOps Chat, and, you know, this is really – this whole SRE thing is certainly an area where we are watching closely. Maybe we could have you back on in a couple months and get an update. 

    Sorensen: I would love to, that would be great. 

    Shimel: All right. Stig, thanks again. This is Alan Shimel for DevOps dot com and you’ve just listened to another DevOps Chat. Have a great day, everyone. 

    — Alan Shimel