devopsy.kinsta.cloud

Tag: SRE

  • How the SRE Role Is Evolving

    How the SRE Role Is Evolving

    In recent years, site reliability engineering (SRE) has garnered much interest. In 2019, LinkedIn listed site reliability engineer as the second most promising job in the United States. Now, in 2021, the role continues to grow and evolve within many organizations.

    Initially spearheaded at Google and credited to engineer Ben Treynor, the strategy seeks smarter accountability for application reliability. An SRE team sets service level indicators, makes error budgets for new features and uses tools like application performance monitoring (APM) to visualize performance insights, among other tasks. SREs are quickly becoming a key check to increase business output in response to new digital innovations.

    I recently met with AppDynamics Regional CTO Gregg Ostrowski to discuss the emergence of SRE and how the approach is evolving. In short, it appears SREs will only continue to grow in importance across more companies, adopting new tactics like chaos engineering, and diversifying their teams with new domain knowledge in response to increasing technological complexity.

    What is an SRE?

    Traditionally, organizations employed system administrators, or sysadmins, to maintain operations for large computing systems and services. However, this typically produced a dichotomy: product developers want to push new features, while operations teams (sysadmins) want to make sure the existing service doesn’t break. The split between developers and operations can be unhealthy and cause friction.

    Whereas sysadmin positions are detached from development and involve a lot of manual work, an SRE approach, on the other hand, takes a more engineering-focused approach to automate operations. Google’s SRE book, which has become a bible of sorts, defines the role as “What happens when you ask a software engineer to design an operations team.”

    So, what does an SRE do? Well, at Google, they are in charge of availability, latency, performance, efficiency, change management, monitoring, emergency response and capacity planning for the services they operate. At Google, they spend no more than 50% of their time on traditional “ops,” like tickets, on-call and manual tasks, and are encouraged to use the remaining 50% of their time to develop automation that will replace human operations labor.

    5 Ways the SRE Approach is Evolving

    It’s been a few years since Google introduced the SRE concept, and naturally, unique iterations have popped up within other enterprises. But, on the whole, where is the concept heading? Ostrowski sees SREs evolving in a few key areas: growth and maturity, increased diversification with domain-specific experts and new monitoring tactics.

    1. Increased Adoption

    First, not all companies have embraced an SRE model. A recent study by Blameless found “… 50% of respondents employ an SRE model with dedicated engineers focused on infrastructure and tooling, or an embedded model where full-time SREs are assigned to a service.”

    The SRE model is gaining momentum, but there is still room for greater adoption. There is also room for internal growth. Ostrowski sees a single SRE team as a single point of failure. “It needs to be a whole department,” he said.

    In addition, SREs are gaining a more prominent voice at the table, influencing feature rollout. “With proper and mature SRE involvement, teams can’t willy-nilly deploy,” he said. Ostrowski views these teams as maintaining a critical balance between business risk and introducing new technology.

    2. Larger, Diversified SRE Departments

    Many companies are experiencing rising user demands, and thus must rapidly scale their application networks. Simultaneously, there has been a Cambrian explosion of deployment types — systems could be using any assortment of legacy infrastructure, mainframe, microservices, cloud environments and multiple cloud vendors. “The complexity and topology of the IT space has grown substantially, with many interdependencies,” Ostrowski said.

    Due to increasingly complex technical stacks, SRE must now cover many domains. Thus, SRE departments will likely require a more diversified team of domain experts. Ostrowski likens SREs to “Navy SEALs of the IT team.” For example, assigning them with experience navigating the nuances behind Google Cloud Platform (GCP) or Microsoft Azure could maintain service reliability in respective clouds.

    Individuals’ personality matters for suitability to SRE, too, said Ostrowski. The role requires a specific type of innovator who considers the repercussions of code, and loves improving processes and performances.

    3. New Testing Tactics Will Emerge

    Ostrowski also foresees SRE departments introducing new monitoring and testing approaches to maintain reliability. One of these is chaos engineering, which champions the idea of intentionally breaking application systems. Different strategies will undoubtedly emerge to help drive user experience (UX) and ensure that performance is always top of mind.

    4. Businesses Rely on SREs to Mitigate Risk

    All things will inevitably fail, at some point. SREs accept failure and learn to manage it, designing repeatable operational mitigation structures. “As companies become reliant on the consumerization of IT, business is dependent on SREs to drive business,” said Ostrowski. This reliance will likely increase as businesses tap SREs to maintain stability.

    Whether it’s reducing mean time to repair (MTTR), programming service level indicators to monitor website load time, or forecasting error budgets for new feature introductions, SREs will be increasingly relied upon to maintain business stability and high performance.

    “The SRE mindset is about coming to terms with a blameless environment,” said Ostrowski. They can help “balance a risk between the business and application team,” described Ostrowski.

    5. SREs Steer UX

    In this digital-only environment, “The application has become the business,” described Ostrowski. If the application is core to the business, monitoring the user journey is necessary to improve it. Ostrowski believes SREs oversee a unique territory that could produce valuable business insights too.

    In addition to monitoring uptime to ensure URL response times meet SLAs, SREs could track UX-related insights, such as conversion rates or cart abandonment percentages. Tracking such analytics and setting standards baselines could help pinpoint problems affecting UX. It could also assist product development in designing better-performing software.

    Final Thoughts

    “Anything that you do more than twice has to be automated.”
    -Adam Stone, CEO, D-Tools

    Traditional operations teams are typically detached from product development and are culturally very different. However, within an SRE approach, operations instead aim to run systems that automate work typically performed manually by sysadmins.

    In this brief introduction, we barely scratched the surface of the topic. And of course, each company is unique in its approach to hybridizing product development and operations. DevOps tenants are closely aligned with SRE — often they are intermixed or one and the same.

    Ultimately, truly reaping business value from this approach will depend on breaking down silos and opening conversations — or at least automated notifications — between disparate units. Only then can operations and development coexist in the most productive way.

  • Rapid Strategic SRE Assessments Accelerate IT Transformations

    Rapid Strategic SRE Assessments Accelerate IT Transformations

    A Strategic SRE Assessment is critical in helping organizations successfully transform traditional IT operations to SRE

    Site reliability engineering (SRE) is rapidly becoming more important to organizations that need to scale IT operations while maintaining or improving reliability and security, and to modernize operations to match the growth of DevOps in the organization. Without sufficiently thorough consideration given to best practices for people, processes and technologies, the transformation from traditional IT operations to SRE is unlikely to succeed. Despite challenges, the transformation to SRE is compelling because it provides an opportunity to reduce total operations costs and improve security and reliability while scaling services.

    To be successful, IT leaders need a comprehensive understanding of SRE best practices and need a well-engineered, yet affordable strategy and road map that effectively implements best practices and balances the many aspects needed to transform people, proces and technology. A Strategic SRE Assessment approach, based on best practices, is a key tool to help leaders understand, obtain the strategy and create a road map that will successfully transform traditional IT operations to SRE.

    There is no “standard” SRE assessment approach in the industry. This article enumerates a practical and proven Strategic SRE Assessment approach. The heart of the Strategic SRE Assessment approach is a survey of SRE best practices categorized under nine pillars: Culture, Work Sharing, Toil Reduction, SLAs/SLOs/SLIs, Measurements, Anti-Fragility, Deployments, Performance Management and Incidents Management. The survey is used to collect importance and capability scores for each of the SRE practices. As part of the assessment, a survey determines the current state and preferences of SRE practices for an organization for each of the nine SRE pillars.

    The following are examples of best practices for each pillar that are part of the Strategic SRE assessment survey:

    Culture – Leaders need to understand and sponsor a clear vision for SRE.

    • The culture embraces working practices to proactively shifts left the “Wisdom of Production.”
    • The culture encourages learning from failure and continuous improvement.
    • The SRE role is clearly defined and understood and SREs are considered a vital part of the organization.
    • Rules of engagement and principles for how SRE teams interact with their environment are codified—not only the production environment but also the product development teams, testing teams, users and so on. Those rules and work practices help to maintain SRE focus on engineering work as opposed to operations work.

    Work Sharing – Successful SRE requires that work be shared between SRE teams and development.

    • SRE teams are composed of software developers with strong operations knowledge or IT operations people with strong software development skills.
    • SRE workloads are managed to a budget with typically 50% or less spent on operations work.
    • SRE on-call work budgets are typically 25% or less. Developers share on-call work.
    • Technical debt is worked in small increments.

    Toil Reduction – SREs work to reduce non-value-added work through automation and standards.

    • Work standardization and automation—of both tools and processes—is employed to improve scalability, repeatability and other important goals.
    • Work standardization is used to reduce time spent on non-value-added work.
    • Automation is used to reduce time spent on non-value-added work.

    SLAs/SLOs/SLIs – Service level metrics, based on user perspectives, combined with error budget policies, are critical to managing flow and reliability.

    • SLOs are selected to measure availability, performance and other metrics in terms that matter to an end user.
    • An error budget (= 1-availability) internal service level objective (SLO) is established for each service. SLOs are set to be more stringent than external service level agreements (SLAs) promised to customers. The SLO budget considers what level users will accept, alternatives for users who are dissatisfied, and usage patterns of the product at different availability levels.
    • An error budget policy prescribes consequences if an error budget is spent. For example, the service freezes changes (except urgent security and bug fixes addressing any cause of the increased errors) until either the service has earned back room in the budget or the count period resets.

    Measurements – Smart monitoring implements observability to accelerate response times to incidents.

    • Monitoring solutions are designed to produce service level indicators (SLIs) from which SLOs can be calculated automatically.
    • SLIs do not require a human to interpret an alerting domain. Instead, software observability algorithms do the interpreting, and humans are notified only when they need to act.
    • Monitoring systems use telemetry and instrumentation to ensure the data being monitored is accurate, relevant and available in real-time.

    Anti-Fragility – Systems of people, processes and technologies are continuously tested and improved to assure they are resilient enough to serve applications as they scale.

    • Anti-fragility strategies are practiced to proactively assure the resilience of applications, infrastructures and pipeline services.
    • Fire drills are periodically conducted to proactively determine weaknesses in systems of people, processes and technologies that could potentially impact SLOs.
    • Chaos monkey or equivalent infrastructure testing strategies are used to proactively determine weaknesses in infrastructure systems that could potentially impact SLOs.
    • Security testing and DevSecOps practices reduce the risk of security vulnerabilities.

    Deployments – Deployment strategies reduce the risk of deployment failures.

    • Non-emergency deployments use progressive rollouts in which changes are applied to small fractions of traffic and capacity at one time. If unexpected behavior is detected, the changes are rolled back first and diagnosed afterward to minimize mean time to recovery.
    • Automation is used to implementing progressive rollouts, detect problems and implement rollback changes safely when problems arise, because outages are primarily caused by changes in a live system.
    • Deployment strategies such as blue-green, feature flag rollouts, A/B testing and canary rollouts are used to reduce the blast radius of failed deployments.

    Performance Monitoring – Proactive testing and monitoring ensure applications and infrastructures have the capacity necessary to flex and scale when needed.

    • The SRE team is responsible for provisioning and monitoring utilization because utilization is a function of how a given service works and how it is provisioned.
    • The SRE team is responsible for capacity planning and provisioning. Regular load testing of the system is used to correlate raw capacity (servers, disks and so on) to service capacity.
    • Provision to handle a simultaneous planned and unplanned outage without making the user experience unacceptable. This results in an “N + 2” configuration, in which peak traffic can be handled by N instances (possibly in degraded mode) while two instances are unavailable.

    Incident Management – Incidents are managed in an efficient, blameless manner with a goal to use incidents to improve responses.

    • Playbooks record the best practices for human response actions because playbooks produce roughly a 3x improvement in mean time to recovery.
    • Emergency response performance is measured using mean time to repair (MTTR) metrics instead of mean time to failure (MTTF) because the most relevant metric in evaluating the effectiveness of emergency response is how quickly the response team can bring the system back to health—that is, the MTTR.
    • Postmortems are blameless and focus on process and technology, not people.
    • Postmortems are conducted for significant incidents regardless of whether they resulted in a page; postmortems that did not trigger a page are even more valuable, as they likely point to clear monitoring gaps. Find all root causes of the event and assign actions to correct the problem or improve how it is addressed next time.

    SRE Assessment Surveys Are Not Enough

    The survey alone is an essential tool to collect information about the current state and preferences for each of the SRE practices. However, this is only part of the comprehensive Strategic SRE Assessment approach.

    The Strategic SRE Assessment approach starts with workshops to get alignment between leaders and teams regarding big-picture goals prior to conducting the survey. This is necessary so that everyone who answers the survey has a common perspective and can better respond with considered views of importance and capability scores for each of the practices. The results of the survey include an analysis that points out specific pillars and individual practices that have a high gap score in which the practices are scored as important but not yet practiced well. The high-priority gaps are then used to organize a value-stream map working session, that is designed to find the highest priority bottlenecks in people, processes and technology bottlenecks in the value steam, relative to the organization’s goals and gaps. With the goals, gaps, and data from the value stream map in hand, the next step is to formulate an implementation road map that directly addresses the goals, gaps and bottlenecks in the value stream.

    The final step is to obtain strategic alignment of the big-picture plan by reviewing the findings and road map with the leadership and implementation teams. This entire Strategic SRE Assessment approach, from goal alignment through surveys, value stream mapping, road map creation and strategic alignment, varies depending on several factors such as the size of the organization, number of applications and size of the deployment domains to be included in the assessment.

    A Rapid Strategic SRE Assessment for a sample set of model applications typically is conducted in 21 days. More complex Strategic SRE assessments involving many applications may take up to 90 days.

    In most cases the Rapid Strategic SRE Assessment is the best place to start. Most often, it is not a good idea and is too expensive to “boil the ocean” and make changes across all applications and deployment domains at once. By starting with a small number of model applications, it is easier to implement, measure and demonstrate success and use the results at each step to justify ongoing investments needed to complete the transformation across the organization. Another advantage of the Rapid Strategic SRE Assessment is that it is a quick and affordable approach to check-point progress as SRE is implemented progressively across applications and deployment domains.

    Key Takeaways for SRE Transformation Leaders

    SRE offers immense value for modernizing and scaling IT operations reliably and securely and to complement DevOps transformation. Adherence to SRE best practices is essential to reduce risk and assure security as the transformation is implemented.

    Each organization is different and has different requirements and priorities for its applications and deployment domains. The specific SRE practices that are the highest priority vary according to the priorities of applications and deployment domains. The Rapid Strategic SRE assessment approach, based on nine pillars of SRE best practices, is quick, effective and affordable.

    The Rapid Strategic SRE Assessment is a preferred approach to kick-start the SRE transformation. This approach is also a preferred and affordable tool to reassess the direction as SRE is implemented progressively across the organization.

    Summary

    There is no “standard” SRE assessment approach in the industry. The Rapid Strategic SRE Assessment approach described in this article has been tried and proven with large and smaller organizations. The author invites your comments regarding this approach and the SRE best practices.

  • Why It’s Time for Site Reliability Engineering to Shift Left

    By adopting a multilevel approach to site reliability engineering and arming your team with the right tools, you can unleash benefits that impact the entire service-delivery continuum

    In today’s application-driven economy, the infrastructure supporting business-critical applications has never been more important. In response, many companies are recruiting site reliability engineering (SRE) specialists to help them improve reliability, availability, latency and a host of other metrics that impact customer satisfaction. These vital new people resources are focused on keeping sites and services running in peak form.

    There are clear limitations to this operations focus, though. A service is made up of many application components that can impact business-critical services. How do you ensure reliability is baked into each app and each code change if your engineers focus only on backend performance?

    Some trailblazers are beginning to tackle this issue head-on, tasking their SREs with taking on new roles that support key stages of the DevOps life cycle. The goal is to take early, proactive steps to ensure quality and reliability are built in from the beginning.

    A Path Pioneered by Test Teams

    To understand why this “shift left” is so important, consider recent revolutionary changes in the software testing function. Not that long ago, testing was considered a distinct phase in the software life cycle, timed to begin well after the work of application developers was done. Testing Centers of Excellence uncovered defects and passed code back to the development team for rework.

    As developers moved to agile sprints, though, this approach to testing quickly became a bottleneck. By the time Centers of Excellence uncovered issues, developers had moved on to other projects. In response, new continuous testing programs emerged, staffed by a new breed of software development engineers who had honed their testing skills.

    These individuals are now embedded with development teams to ensure applications are built better from the start. Open source test automation tools help them test early and often, without impeding agile delivery schedules.

    It’s important to note that Testing Centers of Excellence haven’t gone away. Instead, they have taken on a broader leadership role and are now responsible for coordinating testing across the enterprise in support of continuous software integration and delivery.

    A 3-Level Model for SRE

    There is much that SRE teams can learn from the experiences of their testing colleagues. To produce optimal results, it is important to integrate reliability experts into your organization’s overarching agile framework. Consider adopting the following three-level model that shifts the function to the left and integrates skilled engineers at key points in the DevOps life cycle:

    1. Application level. Embed SREs with development and testing teams to ensure application components and new releases won’t disrupt reliability. These individuals focus on application-level service objectives, error budgets and integration into the DevOps pipeline.
    2. System level. Assign SREs to focus on the applications in your organization’s release train. These individuals support release and launch coordination, evaluate system architecture readiness and ensure your organization meets systemwide service-level objectives.
    3. Enterprise level. Establish a Site Reliability Engineering Center of Excellence, staffed by engineers who oversee the governance of your enterprise architecture. These individuals establish best practices and select the tools and resources required to support your site reliability function companywide.

    An Expanded Toolset

    To support this broadened role, you will need tools that can support your engineers across the entire DevOps life cycle. You will need to be able to see and track issues engineers uncover and to evaluate system-level readiness and performance against your service-level objectives. You will also need a big-picture view that spans activity across the entire enterprise.

    Fortunately, a new generation of solutions is now available to help, powered by artificial intelligence, machine learning and intelligent automation. These resources can help your team process massive amounts of data from disparate toolsets and turn that data into actional insights. As a result, you’ll have all the lifecycle analytics you need to be effective in your new and broadened role.

    SRE and Shift Left: Broad New Benefits

    By adopting a multilevel approach to sire reliability engineering and arming your team with the right tools, you can unleash benefits that impact the entire service-delivery continuum.

    You can build a continuous cycle of feedback, collaboration and governance that spans design, development and the launch and operation of new services. You can better manage configuration changes, service levels and error budgets. You can get a clearer view of application reliability and production readiness to guide planning initiatives and to help you prioritize investments.  And you can build better-informed business cases to move your organization forward. Shifting left is clearly a win-win move.

  • What Is Resilience Engineering?

    What Is Resilience Engineering?

    Admitting things will go wrong isn’t easy for anyone or any team. But modern engineering practices have moved beyond this fear, giving birth to a new practice in DevOps and site reliability engineering (SRE) known as resilience engineering.

    Resilience engineering today isn’t thought of as a function. However, just as DevOps was a description of culture before it was a role and site reliability was an extension of operations before it was a focus, I wouldn’t be surprised if resilience engineering became a function in the new future. The first question most will ask however is, “Isn’t this just SRE?” The purpose of the term is to change the focus from simply reacting to incidents to developing long-term response strategies for them.

    Because the expectation in these environments is that things will break, resilience is the responsibility of existing DevOps and cloud operations teams. When applications and services do break, a “fly by the seat of your pants” response strategy will not work.

    A Focus on Frameworks

    Resilience engineering, while rooted in engineering practices, is largely focused on building strategies and a framework for their execution. This leaves the process of building resilience into a largely unestablished system in part because each system is unique. And, how you respond to issues in that system will likely be unique, even if the management plane that reports issues is not.

    The job of resilience engineering is to:

    Establish procedures, habits and decision trees

    When things break, fight is the only option. Operators and on-call engineers need to address issues in a systematic and repeatable way and do their best to remove emotion and fear from the equation. This not only helps triage and resolve issues, but it also makes sure the activity associated with the issue leads to meaningful insights in post mortems and future collaboration. Part of that is establishing habits and decision-making processes for those who are on-call. The processes help prioritize what to focus on, help catch details because details are critical.

    Be data-driven

    Resilience engineering must rely on data. This is another place where traditional SRE practices grow with a focus on resilience. In typical SRE environments, the focus is on the now, using real-time dashboards of the current state. When you think about resilience you are thinking about the past’s impact on the future, not the now. The only way to do that is to make sure the data supports it; thus, part of resilience engineering is making sure the data is there. Instead of data silos that most organizations have across their delivery chain. Resilience engineering should ensure that telemetry across the entire delivery chain is captured, correlated and shared. The reason this is critical is that what happens early in the delivery chain directly impacts incidents. That activity can be the source of answers, it can be the triggers for rollback, or it can be the clarity needed to prevent similar issues in the future. Without continuity between each stage of the delivery chain, it’s easy to miss correlated events that can lead to more systemic problems.

    Engineering out of reproducible incidents

    For most, the best part of resilience engineering is taking what is learned from previous incidents and finding ways to automate future resolution. Learning from data and having consistency in habit leads to the ability to create runbooks and automate remediation for known issues. Often incident response audit trails can read like playbooks for addressing issues of a particular type. When the resolution is not directly related to code and the potential of issues to surface again in the future is inevitable, being able to build intelligence to address it saves waking someone up at midnight and much shorter impact on customers.

    The Resilience Stack

    Organizations looking to embrace resilience engineering need to have a toolkit built for it. It’s pretty straightforward but worth highlighting. The resilience stack will include:

    1. Observability and/or monitoring tool.
    2. Incident response tool.
    3. On-call strategy documentation.
    4. Post mortem process and documentation.
    5. Documented processes including recommend response steps.
    6. Path to automation.

    For those with a relatively mature and automated environment, the next step is chaos engineering—embracing chaos as a way to get ahead of incidents before they happen in the wild.

    Get your markdown skills to maximum level. There is a lot of documenting that needs to happen with comprehensive resilience engineering. But these documents should not be shelf-ware—they should be living and ultimately lead to the implementation of automation or feedback to development.

    As for monitoring, observability and incident response tools, it’s not enough to simply implement them. Knowing how data will be collected, consumed and actualized is also necessary. So is establishing an on-call strategy with purpose, not just because having everyone on-call is the “cool thing to do.”

    It may seem obvious to say that organizations want system resiliency. What is not obvious is how to execute it. When organizations see the gaps (and are often embarrassed by them), they understand that resilience is a focus for either current functions or new ones in the future. That is why it’s worthwhile to talk about resilience engineering and what makes it effective.

  • SREs Tasked With More To Improve Digital Experiences

    SREs Tasked With More To Improve Digital Experiences

    The digitization of the enterprise has never moved so swiftly, and never have enterprise environments been as complex to manage. And, never have more businesses been as dependent on their infrastructure and operations being unfaltering.

    According to a new survey and accompanying report from Catchpoint and the DevOps Institute, site reliability engineers (SREs) and teams still struggle with effective SRE processes that would help these teams improve their system reliability, incident management and end user/customer satisfaction.

    For instance, the survey found that 53% of respondents reported being brought in too late into the application life cycle. Most respondents said that they are currently spending 75% of their time focused on operations and very little of their time on development objectives.

    That heavy focus on ops comes at a cost by increasing the ongoing costs associated with owning and operating their systems and infrastructure. Moreover, in recent months, perhaps because of increased remote work, Ops-related activities have increased by 10% overall among those surveyed.

    Since the start of the COVID-19 pandemic, the number of SREs who believe they will be working remotely in the next year jumped to 50% from only 19% who thought so before the outbreak of the novel coronavirus.

    When it comes to performing their duties, more than half of the respondents cited the change to remote working causing challenges when it comes to attaining a balance between their work and their life.

    Additionally, while 9% of respondents felt incident management has improved, 41% said that half or more of their work is focused on managing mostly manual, repetitive and tactical tasks that could likely be automated. Another 52% said they spend too much time debugging.

    The report provided a few suggestions for teams:

    • Be sure to include consideration for not only your code but also the networks, third-party services and delivery chain components to evaluate how well the three observability pillars are applied through this new digital experience observability lens.
    • Work to be included earlier in the development process should shift reliability further left to reduce cost, increase team alignment and identify constraints that can be removed.
    • Turn newly surfaced or previously ignored challenges into strategic differentiators. Focusing on challenges such as morale, employee experience, work/life balance and employee engagement and sentiment may showcase a company’s employee-first mentality to attract or retain top talent.

    As we covered in “Digital Transformation Efforts Hindered by Lack of Upskilling,” there are no digital transformations without staff transformations and the same is true for practices such as DevOps and SRE. And it’s that last point that may be one of the biggest inhibitors to SRE teams getting from where they are today in maturity to where they want or need to be. An earlier survey from the DevOps Institute, the “2020 Upskilling: Enterprise DevOps Skills Report” found that 58% of enterprises have a tough time actually finding those with the right DevOps skills, and 48% said it’s difficult to retain skilled DevOps professionals.

  • How To Build a Culture of Resilience Through Good Habits

    How To Build a Culture of Resilience Through Good Habits

    Good habits are hard to form. I’ve been listening to the audiobook “Atomic Habits“ by James Clear on my morning runs, and something struck me. At Gremlin, along with our software, what we’re trying to promote are positive new habits for our customers. According to the author, one of the primary reasons new habits don’t stick is because there’s often sacrifice without immediate gratification.

    Psychologically, we’re wired to want instant gratification. But not all habits give immediate rewards; in fact, many delay our gratification for some time. So how do we help ourselves pursue good habits? Put simply: We need to make them obvious, attractive and easy.

    One of the ways to make them easy is to focus on what the author calls “gateway habits”—the smallest piece of the habit that can reasonably be achieved in two minutes. So if your goal is to eventually run a marathon, the gateway habit is putting on your running shoes every day.

    To build a culture of resilience at your company, start small and create getaway habits. If a team runs a GameDay once a month (time dedicated to experimenting on your systems) or even simply runs their first single chaos engineering experiment, then award that person or team with immediate recognition.

    Here are some other ways to build a culture of resilience at your organization:

    • Recognize the change to new habits.
    • Create DNR (do not repeat) items.
    • Adopt “You build it, you own it.”
    • Track the four golden signals.

    Recognize the Change to New Habits

    Incentives are a great way of kick-starting a new habit, but they don’t necessarily sustain the good behavior. It’s identifying the improvements that result from the new habit that really makes it stick. We’ll get to some specific metrics you can track later in the article, but notice for now that identifying the improvements is when gratification starts to drive enthusiasm. Ideally, that enthusiasm grows until the new habit becomes part of your identity.

    In our example of running a marathon, the moment of most significant change is when the person starts to self-identify as a runner. Then, the habit is no longer a chore, but rather part of who they are. Ideally, we want all computer engineers to adopt a specific set of habits until they consider themselves site reliability engineers (SREs) as part of their identity. That’s when the habit is solidified. 

    Create DNR (Do Not Repeat) Items

    Engineers and product managers want to ship new products and features. There’s nothing quite as satisfying in software development as deploying new code and seeing what you built running out in the world. But, if what you built is consistently breaking or providing a bad user experience, then you are hurting your customers and ultimately your business.

    To make sure we are always learning and getting better, at Gremlin we have what we call “DNR,” or Do Not Repeat work. This work consists of action items from outages and incidents that must not ever be repeated, lest we fail to learn our lesson from these failures. Practically, what this means is all feature work is halted until the issues highlighted as DNR work are remedied and the fixes are verified. In other words, you don’t get to write new code until your old code is fixed. We all know that many teams struggle with the trade-offs of moving fast, but ultimately, if you don’t have strict guidelines in place, then more often than not engineers will prioritize shipping something over making sure it’s reliable.

    Creating a DNR item is an easy way to incentivize the behavior you want to see internally by appealing to the engineer’s desire to produce new features. We convince them to write better code because better code means they get to spend less time fixing things.

    Adopt “You Build It, You Own It”

    This is the driving principle of DevOps. It is the reason behind shifting left. When the team that develops the software is different from the team that operates it, then there’s a misalignment of incentives. If I am a developer being tracked (and promoted) solely on the amount of code I ship, my focus will be on getting more bits out the door and not on ensuring the features I release will withstand the burdens of operation. That’s another team’s concern.

    That’s the motivation behind the proliferation of the “you build it, you own it” mindset at top-performing organizations. Hell, my first day at Amazon, they tossed me a pager and said “Good luck.” And while that may sound daunting (and it was), I can tell you that it not only motivated me to make sure I built systems to last, but it also fundamentally changed the way I thought about architecting systems. Spoiler: I’m not a big fan of my pager going off at 3 in the morning.

    In other words, the person or team building the system needs to be the same person or team that feels the pain if that system is failing. But it’s not just about punishment and pain, that team also needs to be recognized and rewarded when their system is running reliably. This creates an alignment of incentives that promotes the kind of habits seen across top-performing teams.

    Track the Four Golden Signals

    In monitoring distributed systems, Google’s SRE book outlines the four golden signals of monitoring as latency, traffic, errors and saturation. If I could wave a magic wand and immediately improve the culture of an organization, then I would have service-level objectives (SLOs) tied to these four metrics. But going back to habit formation: If you make acquiring a habit too difficult upfront, then an engineering team will ultimately reject it. So if your organization isn’t mature enough to create SLOs, simply beginning to track these metrics will up-level your game tremendously. They will give you an understanding of what is not working well and help guide your priorities.

    To Summarize

    Imagine a world where, after a major incident happens, the points of failure involved become DNR work. The same failure is not allowed to happen again and no new feature work will be completed until the fixes are implemented. And more importantly, no new feature work will be completed until those fixes are verified via a chaos engineering experiment, which is then cataloged and run continuously against your system. Then you take that knowledge and share it with other teams so they can run the same experiments and make sure they are immune to those failures as well.

    This is how you build a culture of resilience.

  • Survey Finds SRE Role Remains Largely Aspirational

    Survey Finds SRE Role Remains Largely Aspirational

    Monitoring tools provider Catchpoint, in collaboration with the DevOps Institute, has published a survey that finds there is a significant gap between the tasks and duties that define the role of what a site reliability engineer (SRE) should be versus what is actually occurring within IT organizations that have hired SREs.

    Based on a survey of 600 IT professionals who have SRE titles or individuals that identify themselves as performing SRE-like tasks, the “Catchpoint SRE 2020 Report” finds the bulk of respondents are still squarely focused on managing IT infrastructure.

    Google, which is widely credited with defining the SRE role, maintains SREs should be spending as much time managing applications as they do infrastructure. Survey respondents on average are spending 75% of their time on operations. Only 14% of respondents said they are spending 50% or more of their time on application development processes. More than have of respondents (53%) also noted they are becoming involved too late in the application life cycle.

    The Catchpoint survey finds the top three tasks survey respondents associated with their roles were monitoring and alerting (93%), dashboarding (73%) and managing infrastructure as code (71%).

    Catchpoint CEO Mehdi Daoudi said one of the most telling aspects of the survey is only 53% of respondents identify observability as a task associated with their role, even though that’s a core tenet for best DevOps practices. Further down the list of DevOps tasks were testing (41%), telemetry (38%) and chaos engineering (26%).

    Daoudi said it’s probable many IT organizations are adding SREs in anticipation of furthering their DevOps ambitions. Not every SRE has all the skills required, but as more of them gain hands-on experience it’s likely many will grow into those roles. In fact, 41% of respondents said half or more of their work revolves around mostly manual, repetitive tasks that could be automated. More than half of respondents (52%) said they spend too much time on debugging-related tasks.

    Part of the challenges many IT organizations face derives from not putting enough focus on automating the management of legacy IT infrastructure, said Daoudi. Every time a new platform is adopted there’s a rush to operationalize it. However, IT teams frequently overlook the need to automate the management of rote tasks to free up more time to learn how to manage new platforms such as Kubernetes.

    Going forward, the survey finds nearly 50% of respondents said they believe they will be working remotely post COVID-19. More than half of respondents also said personal challenges included staying focused and having a good work/life balance while working from home.

    The degree to which SRE titles are being adopted to enable IT professionals to justify a raise or simply pad their resumes is unknown. However, the focus on the SRE role is likely to increase in the wake of the economic downturn brought on by the COVID-19 pandemic. In theory, a single SRE should be able to manage all the tasks that previously might have required as many as 10 IT administrators. At a time when many organizations are looking to reduce the total cost of IT, the number of full-time employees required to manage IT environments has clearly become a major concern.

  • A Thousand Salutes to SREs: Unsung Heroes of the Pandemic

    A Thousand Salutes to SREs: Unsung Heroes of the Pandemic

    Applications are the lifeblood of a modern business, and uptime is crucial — even more so during a pandemic. The COVID-19 crisis has shown us that we can adapt to a new normal. It has also shown us that customers will take their business elsewhere if an application experience does not meet their expectations.

    The pandemic has put business-critical applications under unprecedented strain, and has forced organizations across all industries to rethink their business continuity plans when it comes to keeping their applications up and running. In these troubled times, it has often fallen on the humble site reliability engineers (SREs) to keep applications available and to ensure that they are delivering the best possible user experience. 

    SREs as First Responders: Paramedics for Your Applications

    When unforeseen events occur that cause issues with your applications, it’s the SREs who come to your aid. SREs are your first responders — the paramedics for your applications. They are the team on call to administer life-saving CPR to your applications to keep them alive.

    During outages, SREs know what to look for, how to interpret what they see and quickly understand what’s wrong. By monitoring your application’s vital signs, SREs can relay crucial information to your developers and infrastructure team so that they can decide which corrective surgery to perform to restore the application’s health.

    SREs are the sutures that connect the developers and infrastructure teams with the product team to ensure that your applications are available and provide the intended functionality for your end users — always.

    SREs as Wellness Coaches: Preventive Care Providers for Your Applications

    Thomas Edison said that genius is 1% inspiration and 99% perspiration. It’s the same for SREs: They don’t become the heroes of the hour without extensive preparation. SREs are not only the paramedics for your applications during sudden and unexpected events, but they are also the wellness coaches who take preventative healthcare seriously.

    Like wellness coaches, SREs define and develop strategies and action plans to improve the overall wellness of applications to prevent them from crashing. They set up regular application monitoring for early detection of issues so that they can proactively formulate a plan to restore application health.

    The skill of SREs lies in their ability to wear multiple hats and translate business goals into corresponding technical details. They understand the business risk of application failure and work tirelessly on models to quantify it. And then they work with developers and product teams to understand what should be measured as a precursor to failure. They set and maintain service-level objectives (SLOs) aligned with business needs and monitor them constantly to spot and analyze trends.

    SREs determine error budgets so that they will know how bad things can be before an issue becomes a problem. And, when something is trending out of scope, they help diagnose which services are causing the issue, define corrective actions and set the thresholds that enact auto-scaling to meet fluctuation in demand. 

    A Thousand Salutes in a Hundred Languages 

    SREs are, by definition, the proactive enforcers of business continuity. Their role is quite literally to prevent disasters from affecting the business by anticipating the unanticipated. They spend their days (and nights) looking at architectures and applications to assess what can go wrong and to determine how problems can be mitigated. In a pandemic, it is so often the emergency and preventive work of the SRE team that saves the business.

    SREs are, indeed, the unsung heroes, and we should salute them. So, SREs across the world — thank you, dankie, gracias, merci, danke, dhanyavād, grazie, Xièxiè, dziękuję, спасибо, ευχαριστώ, arigatô, obrigado, mulţumesc. Have you thanked your SRE today?

  • Measuring DevOps Performance

    Measuring DevOps Performance

    As IT becomes increasingly central to our organizations, it is increasingly important to improve our ability to deliver innovations efficiently and safely. DevOps is a movement to reimagine the way we deliver software, with an emphasis on delivering value to end users through automation and collaboration. In the midst of complex changes to complex processes, it’s easy to lose sight of the most important point: Our “improvements” must deliver actual improvements. Measuring the performance of a software delivery team is the basic foundation on which you can assess the impact of changes.

    One of the main contributions of the State of DevOps Report has been to focus consistently on the same key metrics year after year. Although the questions in their survey have evolved and new conclusions have emerged over time, the four key metrics used as benchmarks have remained in place:

    1. Lead time (from code committed to code deployed)
    2. Deployment frequency (to production)
    3. Change fail percentage (for production deployments)
    4. Mean time to restore (from a production failure)

    The book “Accelerate” provides a detailed explanation of each of these metrics, and why they were chosen; those points are summarized here.

    The first two of these metrics pertain to innovation, and the fast release of new capabilities. The third and fourth metrics pertain to stability, and the reduction of defects and downtime. As such, these metrics align with the dual goals of DevOps, to “move fast, and not break things.”

    These also align with the two core principles of lean management, derived from the Toyota Production System: “Just in time” and “Stop the line.” “Just in time” is the principle that maximum efficiency comes from reducing waste in the system of work; and that the way to reduce waste is to optimize the system to handle smaller and smaller batches, and to deliver them with increasing speed. “Stop the line” means the system of work is tuned not just to expedite delivery, but also to immediately identify defects to prevent them from being released, thus increasing the quality of the product and reducing the likelihood of production failures.

    Lead time is important because the shorter the lead time, the more quickly feedback can be received on the software, and thus the faster innovation and improvements can be released.  The book “Accelerate” revealed that one challenge in measuring lead time is it consists of two parts: time to develop a feature, and time to deliver it.

    The time to develop a feature begins from the moment a feature is requested, but there are some legitimate reasons why a feature might be deprioritized and remain in a product’s backlog for months or years. There is a high inherent variability in the amount of time it takes to go from feature requested to feature developed. Thus, lead time in the State of DevOps Report focuses on measuring only the time to deliver a feature once it has been developed.

    The software delivery part of the lifecycle is an important part of total lead time, and is also much more consistent. By measuring the lead time from code committed to code deployed, you can begin to experiment with process improvements that will reduce waiting and inefficiency, and thus enable faster feedback.

    Deployment frequency is the frequency of how often code or configuration changes are deployed to production. Deployment frequency is important since it is inversely related to batch size. Teams that deploy to production once per month deploy a larger batch of changes in each deployment than teams who deploy once per week. All changes are not created equal. Within any batch of changes there will be some which are extremely valuable, and others that are almost insignificant.

    Large batch sizes imply that valuable features are waiting in line with all the other changes, thus delaying the delivery of value and benefit. Large batches also increase the risk of deployment failures, and make it much harder to diagnose which of the many changes was responsible if a failure occurs. Teams naturally tend to batch changes together when deployments are painful and tedious. By measuring deployment frequency you can track your team’s progress as you work on making deployments less painful and enabling smaller batch sizes.

    Change fail percentage measures how frequently a deployment to production fails. Failure here means that a deployment causes a system outage or degradation, or requires a subsequent hotfix or rollback. Modern software systems are complex, fast-changing systems, so some amount of failure is inevitable. Traditionally it’s been felt that there’s a trade-off between frequency of changes and stability of systems, but the highly-effective teams identified in the State of DevOps Report are characterized by both a high rate of innovation and a low rate of failures. Measuring failure rate allows the team to track and tune their processes to ensure that their testing processes weed out most failures before they occur.

    Mean time to restore (MTTR) is closely related to the lead time to release features. In effect, teams that can quickly release features can also quickly release patches. Time to restore indicates the amount of time that a production system remains down, in a degraded state, or with non-working functionality. Such incidents are typically stressful situations, and often have financial implications. Resolving such incidents quickly is a key priority for operations teams. Measuring this metric allows your team to set a baseline on time to recover, and to work to resolve incidents with increasing speed.

    In 2018, the State of DevOps Report added a fifth metric, system uptime, which is inversely related to how much time teams spend recovering from failures. The system uptime metric is an important addition for several reasons. First of all, it aligns with the traditional priorities and key performance indicators of sysadmins (the operations team). The number one goal of sysadmins is keeping the lights on or ensuring that systems remain available. The reason for this is simple: the business depends on these systems and when the systems go down, the business goes down. Outages are expensive.

    Tracking system uptime is also central to the discipline of site reliability engineering (SRE). SRE is the evolution of the traditional sysadmin role, expanded to encompass web-scale or cloud-scale systems where one engineer might be responsible for managing 10,000 servers. SRE emerged from Google, who shared their practices in the influential book Site Reliability Engineering. One innovation shared in that book is the concept of an error budget, which is the recognition that there is a trade-off between reliability and innovation, and that there are acceptable levels of downtime.

    According to the Site Reliability Engineering Book, Chapter 3, “Put simply, a user on a 99% reliable smartphone cannot tell the difference between 99.99% and 99.999% service reliability. With this in mind, rather than simply maximizing uptime, Site Reliability Engineering seeks to balance the risk of unavailability with the goals of rapid innovation and efficient service operations, so that users’ overall happiness — with features, service, and performance — is optimized.”

    The State of DevOps Report shows how these five metrics are interrelated (See Figure 1). The timer starts on lead time the moment a developer finishes and commits a feature to version control. How quickly that feature is released depends on the team’s deployment frequency. While frequent deployments are key to fast innovation, they also increase the risk of failures in production. Change fail percentage measures this risk, although frequent small deployments tends to reduce the risk of any given change. If a change fails, the key issue is then the mean time to restore service. The final metric on availability captures the net stability of the production system.

    Figure 1: Correlation between the five metrics.

    How the Five Key Software Delivery and Operations Performance Metrics Tie Together

    Together, these metrics constitute a team’s software delivery performance. The goal of any DevOps initiative should be to improve software delivery performance by strategically developing specific capabilities such as continuous delivery and the use of automated testing.

    How your team measures these capabilities is another challenge. But “Accelerate” makes a compelling argument for the validity of surveys. Automated metrics can be implemented over time, although the mechanism to do this will depend on how you do your deployments. Salesforce production organizations track past deployments, but it’s not currently possible to query those deployments, so you would need to measure deployment frequency (for example) using the tools you use to perform the deployments. Salesforce publishes their own service uptime on Trust, but that gives no indication of whether critical custom services that customers build on Salesforce are in a working state or not.

    Surveys provide a reasonable proxy for these metrics, especially if responses are given by members of the team in different roles. Guidelines for administering such surveys are beyond the scope of this book, but your teams’ honest responses are the most critical factor. Avoid any policies that could incent the team to exaggerate their answers up or down. Never use these surveys to reward or punish; they should be used simply to inform. Allow teams to track their own progress and to challenge themselves to improve for their own benefit and for the benefit of the organization. As it says in the Agile Manifesto “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”

    Metrics provide a reliable, long-term indicator of how your software delivery team is performing. They open the door for your team to experiment with different approaches and assess their impact using a common standard. The key metrics described here are important because they emphasize end-to-end performance, and thus incent teams to focus on collaboration towards this common goal. Balancing velocity with reliability is critical, thus these metrics should be viewed together, to ensure that one goal is never emphasized at the expense of the other.

  • Site Reliability Engineering 101: DevOps Versus SRE

    Site Reliability Engineering 101: DevOps Versus SRE

    Consider the following scenario.

    An Independent Software Provider (ISV) developed a financial application for a global investment firm that serves global conglomerates, leading central banks, asset managers, brokerage firms and governmental bodies. The development strategy for the application encompassed a thought through DevOps plan with cutting-edge agile tools. This has ensured zero downtime deployment at maximum productivity. The app now handles financial transactions in real-time at an enormous scale, while safeguarding sensitive customer data and facilitating uninterrupted workflow. One unfortunate day, the application crashed and this investment firm suffered a severe backlash (monetarily and morally) from its customers.  

    Here is the backstory: Application’s workflow exchange had crossed its transactional threshold limit, and lack of responsive remedial action crippled the infrastructure. The intelligent automation brought forth by DevOps was confined mainly to the development and deployment environment. The IT operations, thus, remained susceptible to challenges. 

    Decoupling DevOps and RunOps — The Genesis of Site Reliability Engineering (SRE)

    A decade or two ago, companies operated with a legacy IT mindset. IT operations consisted mostly of administrative jobs without automation. This was the time when the code writing, application testing and deploying was done manually. Around 2008 to 2010, automation started getting prominence. Now developers and ops teams worked concurrently toward continuous integration and continuous deployment — backed by the agile software movement. The production team was mainly in charge of the runtime environment. However, they lacked skillsets to manage IT operations, which resulted in application instability, as depicted in the scenario above. 

    Thus, DevOps and RunOps were decoupled, paving the way for SRE — a preventive technique to infuse stability in IT operations.

    Software-First Approach: Brain Stem of SRE

    “SRE is what happens when you ask a software engineer to design an operations team,” said Benjamin Treynor Sloss, VP of engineering at Google.

    This means an SRE function is run by IT operational specialists who code. These engineers implement a software-first approach to automate IT operations and preempt failures. They apply cutting edge software practices to integrate dev and Oops on a single platform, and execute test codes across the continuous environment. Therefore, they carry advanced software skills, including DNS configuration, remediating server, network and infrastructure problems, and fixing application glitches. 

    The software approach codifies every aspect of IT operations to build resiliency within infrastructure and applications. Thus, changes are managed via version control tools and checked for issues leveraging test frameworks, while following the principle of observability.

    SRE, DevOps

    The Principle of Error Budget

    SRE engineers verify the code quality of changes in the application by asking the development team to produce evidence via automated test results. SRE managers can fix service level objectives (SLOs) to gauge the performance of changes in the application. They should set a threshold for permissible minimum application downtime, also known as error budget. If a downtime during any changes in the application is within the scale of the error budget, then SRE teams can approve it. If not, then the changes should be rolled back for improvements to fall within the error budget formula.

    Calculating Error Budget

    A simple formula to calculate error budget is: (system availability percentage) minus (SLO benchmark percentage). Please refer to the System Availability Calculator cheat sheet below.

    Suppose the system availability is 95% and your SLO threshold is 80%. 

    Error budget: 95% – 80% = 15%

    Error budget/month: 108 hours. (At 5% downtime, per day downtime hours is 1.2 hours. Therefore, for 15% it is 1.2 x 3 = 3.6. So, for 30 days it will be 30 x 3.6 = 108 hours).

    Error budget/quarter: 108*3 = 324 hours.

    Quick trivia: Breaking monolithic applications lets us derive SLOs at a granular level.

    Cultural Shift: A Right Step Toward Reliability and Scalability 

    Popular SRE engagement models such as Kitchen Sink (aka Everything SRE), infrastructure teams and embedded SREs, tend to build dedicated teams that lead to a “Silo” SRE environment. The problem with the Silo environment is that it promotes a hands-off approach, which results in a lack of standardization and coordination between teams. So, a sensible approach is shelving off a project-oriented mindset and allowing SRE to grow organically within the whole organization. It starts by appraising the teams of customer principles and instilling a data-driven method for ensuring application reliability and scalability. 

    Organizations must identify a change agent who would create and promote a culture of maximum system availability. The agent can champion this change by practicing the principle of observability, where monitoring is a subset. Observability essentially requires engineering teams to be vigilant of common and complex problems hindering the attendance of reliability and scalability in the application. See the principles of observability below.

    The principles of observability follow a cyclical approach, which ensures maximum application uptime.

    Step Zero: Unlocking Potential of Pyramid of Observability

    Step zero is making employees aware of end-to-end product detail — technical and functional. Until an operational specialist knows what to observe, the subsequent steps in the pyramid of observability remain futile. 

    Also, remember that this culture shift isn’t achievable overnight. It will be successful when practiced sincerely over a few months. 

    DevOps Versus SRE

    People often confuse SRE with DevOps. SRE and DevOps are complementary practices that drive quality in the software development process and maintain application stability. 

    Let’s analyze four key fundamental differences between DevOps and SRE. 

    SRE, DevOps

    Conclusion – SRE Teams as Value Center

    A software product is expected to deliver uninterrupted services. The ideal and optimal condition is maximum uptime with 24/7 service availability. This requires unmatched reliability and ultra-scalability. 

    Therefore, the right mindset will be to treat SRE teams as a value center, which carries a combination of customer-facing skills and sharp technical acumen. Lastly, for SRE to be successful, it is necessary to create SLI driven SLOs, augment capabilities around cloud infrastructure, a smooth inter-team coordination and thrust automation and AI within IT operations.