devopsy.kinsta.cloud

Tag: toil

  • Leverage Empirical Data to Avoid DevOps Burnout

    Leverage Empirical Data to Avoid DevOps Burnout

    Burnout is possible in any industry. But it seems to be especially rampant in tech, a space bursting at the seams with inflated expectations and ever-accelerating trends. Nearly 60% of tech workers said they were currently feeling burned out at their workplace, found a study by TeamBlind.

    Limiting burnout and supporting employee mental health is essential, not only for the workers’ sake but for the business as well. During this period, which commentators call the Great Resignation, if employees don’t feel respected, they will likely seek new opportunities elsewhere. Ensuring the needs of engineers are met is a vital target amid an ongoing cloud skills drought, a talent gap to which the DevOps space is certainly not immune.

    I recently met with Eric Mader, senior solutions architect, AHEAD, to understand the root causes of burnout as it relates to DevOps. According to Mader, burnout can be caused by many factors, including work-life imbalances and conflicting egos in the workplace. But above else, burnout is most often caused by management setting unrealistic targets that aren’t based on empirical team performance data.

    What Factors Contribute to DevOps Burnout?

    So, what can cause burnout in DevOps? First, the workplace culture can significantly contribute to burnout. In most remote working models, explained Mader, it isn’t easy to ascertain the correct work-life balance. For example, there may be an unwritten expectation to immediately respond to Slack messages or quickly get back to emails after-hours or over the weekends.

    An “always on” attitude can lead to unhealthy work habits. And while this isn’t necessarily unique to DevOps, it is sometimes unavoidable due to the high connectivity required to perform the job. Another potential contributor to burnout is the team dynamic, noted Mader. “Every team seems to be unique in its own way,” he said.

    Although Agile and DevOps involve very technical work, at times, ego can get in the way of a healthy team dynamic. For example, in many tech teams, there is often a “guru,” such as a senior developer or tech lead, that everyone goes to for advice, Mader said. This person might end up doing most of the work (both technical and interpersonal) and usually feels the most burned out. Or, other wanna-be leaders might take on an unnecessary command-and-control role.

    Personality differences with coworkers could contribute to burnout, but other factors like dealing with toil or managing legacy technical debt can be just as tiresome.

    ‘Move Fast And Break Things’

    According to Mader, perhaps the biggest contributor to burnout is the insistence from executives that tech workers consistently outperform their previous output—especially when past team performance data does not support these goals. Setting achievable goals is great, but when deliverables are out of scope, it’s a recipe for burnout.

    Part of the pressure to increase velocity is the constant illusion that “faster and more is better,” said Mader. Value is often attributed to the number of items achieved in a sprint instead of the quality of each point. For example, a team might deliver 200 story points of work in a sprint, but only 30 of these points carry a significant amount of value. Or, a leader might see a team accomplish 60 points of work in a sprint and then encourage 200 in the next one.

    These sorts of high expectations suffer from basic capacity problems. And the added stress quickly realizes itself as a burden, contributing to burnout, explained Mader. He encourages technical leads not to force more work but instead, “focus on the highest-value work based on the empirical data you have.”

    Five Tips To Avoid Burnout

    Mader shared some tips on how to avoid burnout in DevOps teams. Here are some simple yet effective ideas to maintain happier engineers and a more productive, healthier team relationship.

    Snuff out the always-on attitude. Organizations need to walk the talk when it comes to mental health awareness. Therefore, ensure employees know that their life boundaries are respected. Encourage this by honoring time off for holidays, paternity/maternity leave or personal vacation. Discourage emailing on the weekends and encourage turning on ‘Do Not Disturb’ alerts on team chats and work email accounts.

    Prioritize what gets automated. There’s a misnomer that any automation is good automation. In fact, over-prescription of software development automation can lead to some negative consequences. “I’ve seen many DevOps teams that focus so much time on automating low-impact business processes,” said Mader. Instead, prioritize automation projects that affect more people and those that would be continually used.

    Try out the ‘Fist of Five.’ Ensuring everyone’s voice is heard helps avoid a toxic working environment. One tactic to quickly gauge reaction in team meetings is to use what Mader calls The Fist of Five: One if an idea is totally off base and five if an idea makes total sense. This simple practice could ensure everyone gets to contribute input on new projects.

    Take the quietest person in the room aside. Only the squeaky wheel gets the oil, Mader noted. With that in mind, take aside the quietest person post-meetings to gather their input—you might be surprised. Junior folks who may be less apt to speak their minds typically have a less-institutionalized mindset and could suggest more innovative ideas.

    Use empirical data to inform future goalposts. Finally, Mader encouraged DevOps teams to “use empirical evidence to back up on where you’re going in the future.” Specifically, in the DevOps sphere, empirical data takes the form of metrics like mean-time-to-recovery, or SLAs, SLIs and SLOs.

    Remember that even if a team did accomplish a certain number of goals in a sprint, it doesn’t mean they can produce the exact same results the next time around—they could have been pushed to their limit to produce past results. Thus, managers should aim to move the needle on metrics progressively, not in giant strides that upset morale.

    Take an Opinionated Approach to DevOps

    Nowadays, DevOps teams are spoiled for choice—there are countless tools on the market and an endless stream of best practices and methods. Even within a single company, the DevOps toolset may vary considerably from team to team. To help make sense of it all, Mader encouraged DevOps leads to simplify the transformation journey by setting guardrails and adopting a more opinionated approach around what tools and processes you take on.

  • Let Me Reiterate – Don’t Rush to Iterate

    Let Me Reiterate – Don’t Rush to Iterate

    We in the DevOps world spend an inordinate amount of time talking about tools and roles—and there are always new tools and modifications of roles to talk about.

    Do you know what we don’t discuss as an industry? The day-to-day grind of being in DevOps. Oh, you can go to forums like Reddit and have those discussions with other practitioners, but pundits don’t discuss what it’s like to actually do the work. I secretly suspect this is because most of us pundits no longer develop code in general, let alone have a working DevOps environment. I certainly don’t code as much as I used to, and I develop more than many of my peers in punditry. For the record, we do have a working DevOps environment (two, but one almost never gets spun up) at my company, but I’ll be nice and won’t get sidetracked by talking about it here.

    We’ve talked about it tangentially, but I wanted to be direct with you all. The vast majority of your organizations do not need to iterate non-stop. And the mentality that you cannot iterate too much is simply false. While the guard gates and German Shepherds that protected production from rogue developers in the past were overreactions, there should be a happy medium. Every time change is introduced in production, there is some amount of risk. And we have a ton of tools to limit that risk these days—feature flags to turn new functionality on/off, auto-run testing before the move to production, etc.—but there is still measurable risk. And “we can just roll it back” only works in small, unsegmented projects. Once you get into nested dependencies (microservices are a great example), then versioning starts to make rollbacks an issue. Knowing the interconnected relationships between what is being changed is a big deal. So it is not as simple as the ::shrug:: “Just rebuild and deploy without your changes,” crowd would have you believe.

    And to a great extent, that is why this pundit doesn’t come out and blather about iteration speeds—the decision to iterate once every six months or every time Bob checks in code is very much an organizational/application one. How fast can the organization handle a full iteration, and how often does the application require it? The answer is rarely the insane number that vendors like to throw out to show you how great they are. There are a whole lot of organization/application combinations whose rate of change is slow enough that they could honestly schedule full-on iterations and plan updates around vulnerability patches in the underlying composite pieces. The majority only need to iterate over months. Very (and I stress very) few need to iterate even daily, let alone more than a hundred times a day. And they are indeed special snowflakes with super-tweaked environments.

    It is kind of stunning to me that a portion of the rise of Agile and DevOps was to free up IT time because we were overloaded. And then once that was done, we decided to crank up the rate of change so that IT was … overloaded. A release takes DevOps time, DevOps time and security time. It takes resources to build, test and deploy. Don’t overdo it.

    Build when you need to. Deploy when it is a benefit to customers. Keep rocking it. We’re out here using your apps, or your apps are grinding behind the scenes, keeping the business running. Either way, you know better than anyone what your app and your org need. Don’t follow the “10 billion deploys a day!” crowd; push for what works. And you are the SME in that department.

  • Shedding Light On Toil: Ways Engineers Can Reduce Toil

    Shedding Light On Toil: Ways Engineers Can Reduce Toil

    It comes in many shapes and sizes and is embraced as a necessary evil. It lurks in the shadows, emerging now and again to stealthily creep into our workflows, where it feasts on our perceived shame. If not pruned, it grows and wraps its tendrils around the workforce, draining the energy out of every digital worker across departments. We all have it and most of us believe we suffer alone.

    I’m talking, of course, about toil.

    Dramatics aside, toil is common throughout most programming and DevOps positions, whether we like it or not. And when it comes to platform engineering, the chances of encountering toil is high. You can think of toil as those tedious workarounds that should be automated but aren’t. This could be due to a lack of standard configurations for deployments. Or, perhaps engineers must copy and paste data from one module to the other—an integration that has not yet been automated.

    One of the hallmarks of the SRE role is to spot and reduce such toil—hopefully before it gets out of hand. But, what exactly is toil? And, how can SREs reduce it?

    I recently met with Jake Englund, senior site reliability engineer, Blameless, and Matt Davis, intuition engineer, Blameless, to uncover some tips on spotting and eliminating toil in the daily life of an SRE. Below, we’ll attempt to define toil and explore some techniques to defeat this ghoulish foe.

    First, What is Toil?

    Toil is tedious. Englund defines toil as repetitive manual tasks that are, ideally, automatable. “It’s not gratifying work; it lacks enduring value.” James Curtis similarly described toil as “tedious actions that really have no enduring value.” Toil is similar to technical debt in that, if left unchecked, it can hinder productivity and contribute to burnout.

    Toil is invisible. Toil is the invisible work that goes on behind the scenes, says Davis. For example, at your team meeting or stand-up you might say something like, “I’m deploying this fix.” There’s usually no need to explain the nitty-gritty details of your exact step-by-step process, as it would be too far in the weeds. As a result, manual and repetitive tasks are often not given the spotlight.

    Toil is persistent. Toil might be readily apparent, but it’s not as simple to eliminate; attempts at fixing toil could potentially create new toil. Tedious tasks may also be essential to keeping core infrastructure pieces up-to-date. According to Englund, what often ends up happening is that only the specific engineer knows the niche mechanics of what they’re maintaining, making them irreplaceable—a power that some employees might abuse. “If you’re the only one who knows how to do something and you’re hoarding that information, it leads to inertia against resolving toil,” said Englund.

    Toil is pervasive. Toil is not limited to SRE or engineering. From UX to management or marketing, within most positions, you’ll encounter a degree of toil. This is because the ‘work as prescribed; is often much different than ‘work as done’, said Davis.

    Four Tips To Reduce Toil

    So, what can organizations do to conquer the nightmarish reality that is toil? Especially those that don’t have a dedicated engineering team to reduce drudgery?

    1. Find It!

    To reduce toil, you first have to find it. Sounds simple enough, right? Well, discovering toil might not be as obvious as it sounds. “You get into these toil situations where you might not even know you’re in it. How do you know that toil is encompassing what you’re doing?” asked Davis.

    To find toil, the Blameless team has a weekly session open to the whole company to examine ‘work as done.’ According to Davis, this has been very beneficial for discovering where toil occurs. It also helps garner perspectives from roles throughout the entire company.

    2. Standardize and Automate

    Next, a proven method to reduce toil is standardizing your engineering processes. Standardization, said Davis, is “another word for removing ambiguity.” For example, SREs can benefit from having a common series of steps to follow when responding to alerts. Runbooks that aggregate data to respond to issues can reduce toil in the data collection process while also helping improve mean-time-to-discovery and recovery rates.

    Standardizing change management is another way to reduce toil. If engineers are approaching change management in various ways, this could cause additional review and slow down the release process. Removing ambiguity from the change lifecycle can alleviate toil down the line.

    When you find yourself having to retain mental models for configurations, it’s a good sign that standardization is required, explained Englund. For example, perhaps some configurations are needed in some environments but are commented out in others. Documenting the specific configurations for different development environments (and automatically populating configurations) is one way to reduce toil and the chance for human error.

    3. Proactively Monitor

    More proactive monitoring is another way to reduce toil, according to Englund and Davis. “Responding to a crash loop is responding too late,” added Davis. Instead, he advocated that SREs look toward leading indicators that suggest the potential for failure so that teams can make adjustments well before anything drastic occurs.

    If SLIs like error rate and latency are getting bad, you must take reactive measures to fix them, causing more toil. Instead, proactive monitoring is best to see the cresting wave before the flood. Leading indicators could arise from following things like data queue operations connected to servers or the saturation of a particular resource. “If you can figure out when you’re about to fail, you can be prepared to adapt,” said Davis.

    4. Practice How to Improvise

    One major caveat of standardization is that you’re inevitably going to encounter edge cases that require flexibility. And when an outage or issue does arise, the remediation process is often very unique from case to case. As a result, not all investment into standardization pays out.

    Alternatively, teams that know how to improvise together are proven to be better equipped for unforeseen incidents—one study on Arctic Sea emergencies found collective improvisation to be imperative for emergency response situations. Similarly, collaboration is vital for software teams. Davis calls this the “practice of practice.” If you don’t practice backup restores, for example, teams may flounder in the case of a database outage. Yet, since every remediation effort is unique, it really boils down to learning how to collaborate effectively as a team. “We can gain a lot from getting together to practice,” added Davis.

    All the Small Things

    Toil is like manually typing an email signature every time you are about to hit “Send.” Maybe you’ve just been putting off configuring your signature in your email client. Or, perhaps you like to write a custom sign-off for each recipient. Regardless of the reason, the longer this repetitive motion is required, it’s a stick in the spokes of productivity.

    In the grand scope of things, this tedium can be a drain on time and resources. For every extra keystroke or click, every un-automated workflow or every copied-and-pasted code block, a small degree of toil is piled onto the daily workload of an engineer, dragging them away from focusing on what really matters.

    Standardization, automation, proactive monitoring and collaboration are all helpful elements to consider on your journey toward reducing toil. But of course, there is the trap of increased software development automation—the time saved building automation must outweigh the upfront development cost (and ongoing maintenance effort). Otherwise, you might end up creating even more toil.

  • 5 Ways to Reduce DevOps Toil

    5 Ways to Reduce DevOps Toil

    Over the last several years, DevOps has become a bit of a buzzword. It has become simultaneously a practice, a culture, a team, a job title and a vendor product. You can hire some DevOps, buy some DevOps, adopt DevOps and sprinkle a little bit of DevOps on top for good measure.

    But, at its core, DevOps is about service ownership: having a single team with end-to-end ownership of software running in production. You should not have one team building software and another team deploying and operating it. That’s slow. DevOps is literally combining development and operations into a single consolidated team. DevOps means embracing the mantra of “you build it, you own it.”

    The primary benefit of DevOps is increased speed and agility of engineering teams. With DevOps, exactly one team is responsible for a given piece of production software. There are no handoffs or knowledge silos. Practices like continuous delivery can emerge. Features can be put in front of your customers’ eyes with a shorter turnaround time. A secondary benefit is increased reliability and security. With “you build it, you own it,” it is no longer someone else’s job to ensure that software is correct or performant or reliable or secure. These responsibilities are “shifted left” onto exactly one team.

    So, where do things go off the rails?

    Service ownership and DevOps mean giving teams the agency to make whatever changes are necessary to their software. That can and will come at the expense of product functionality at times. So, the first place things go off the rails is when there is not a healthy balance between product needs and engineering needs. If teams are constantly working on new features but do not have the capacity to fix operational issues that are affecting them, product quality will eventually suffer. Ultimately, product-engineering imbalance generally comes from misaligned incentives or poorly defined goals from management.

    The second place where things go off the rails is insufficient training. Especially with organizations early in their journey to adopting DevOps, which may have separate development and operations teams, there is a lot of training on new technologies and processes. Developers need to learn about observability, responding to incidents and debugging production applications. Operators and sysadmins need to learn about coding and the principles of software design. These are new skills for both camps and investment in training and mentorship is required.

    What are some best practices for keeping DevOps on track?

    There are lots of these best practices. It really depends on your starting point as an engineering organization, but here are five important ones:

    1. Track Service Ownership

    If DevOps is about “you build it, you own it,” you need an authoritative list of a) what’s running in production and b) which team owns it. If a piece of software goes down, you do not want to find out that the last person to work on it was the intern who left three years ago. Many folks start tracking this kind of information in spreadsheets or wikis, but keeping the data up-to-date becomes challenging. SaaS service catalogs have emerged as the easiest way to automatically keep this data accurate, complete and consistent.

    2. Adopt Continuous Integration, Continuous Delivery and Feature Flags

    Committing small changes multiple times per day is superior to release trains where one large change is committed on an infrequent basis. CI/CD allows for new features and improvements to be tested faster and put in front of customers more quickly. Feature flag tools let you easily test new functionality and product hypotheses for different segments of customers.

    3. Embrace Incident Management

    Software breaks sometimes. Having a robust process for how to triage and respond to both alerts and incidents is important. Within the context of DevOps, the team that owns a piece of software should be responsible for being on-call to respond to associated incidents. If the software breaks or behaves in some unexpected way, the team is expected to fix it. The team is also responsible for identifying what happened and investing in ways to ensure the same error does not occur again.

    4. Treat Everything as Code, Then Automate all the Things

    Applications and microservices? That’s code. Infrastructure? That’s code, too. Security policies? Also code. Deployments? You guessed it–code.

    Treating everything as code allows for processes like “code review” to emerge beyond just your applications. You can have multiple eyes on every change, which catches bugs sooner and spreads knowledge to others. As well, version control systems (like Git) are a natural audit log, allowing anyone to see what was changed, when, and by whom.

    With everything defined as code, the next step is to invest in automation. How many steps does it take for a developer to deploy new code to production? Or to spin up new infrastructure? Or to hook up monitoring or error tracking tools?

    Your goal should be “one step:” run a command. It can be a shell script, a Slack bot or a button with a nice web UI. But it should always be one step. If there are more, you’re now in “manual steps” territory, and that’s brittle. You have no guarantees that successive individuals will run whatever sequence of steps in the same order, which can lead to divergence in how various parts of production are set up.

    Another example is maintaining a list of a) what’s running in production and b) who owns it. Manually keeping that list up-to-date is impossible as the size of your production infrastructure or engineering team scales. You’re better off investing in service catalog tooling that can automatically integrate and capture this information.

    5. Invest in Reducing Toil

    Service ownership and DevOps are two-way streets between management and their engineering teams. On one hand, consolidated ownership will lead to faster delivery and more reliable software. That’s good for management. But, on the other hand, teams require actual empowerment and agency to make decisions about their software. Part of that is having capacity on their roadmaps to make improvements to issues that are causing toil.

    For example, if a particular service is paging a lot or experiencing load issues, service ownership means giving the owning team the time to make any necessary changes. That will come at the short-term expense of new production functionality.

    There’s no free lunch on this last point. Engineering management can’t just combine dev and ops responsibilities and magically expect results. Service ownership is a two-way street. Otherwise, toil will kill your teams in the long term.

  • Survey Reveals Slight Decline in Level of SRE Toil

    Survey Reveals Slight Decline in Level of SRE Toil

    The amount of routine toil that site reliability engineers (SREs) perform declined slightly in the last year even though IT environments in general are becoming more complex to manage.

    An annual survey of 300 SREs conducted by Catchpoint, a provider of an IT monitoring platform, in collaboration with VMware and the DevOps Institute suggests that the amount of toil—defined as low-level manual tasks—declined 15% in the last year.

    Only 22% of respondents claim to be scientifically measuring toil, so, to a degree, the survey results are apocryphal. Overall, the survey, published this week, finds SREs are, not surprisingly, spending the bulk of their time on responding to incidents and outages, followed by participating in postmortem analysis and on-call rotations. The primary sources of toil cited in the survey are too much technical debt, lack of alignment of priorities and goals, lack of appreciation of business value, lack of training and support and lack of collaboration.

    In terms of automation adoption, the most common use cases cited are release management, infrastructure management and application management. The tools most commonly employed by SREs are used to monitor infrastructure, networks and application performance. Adoption of artificial intelligence (AI) tools to manage IT operations (AIOps) remains low. The survey also finds respondents are roughly split evenly between centralized (42%) and decentralized (38%) approaches to managing IT. Another 20% said they have adopted a hybrid approach. Lack of visibility across the stack (53%) was the most cited cloud application monitoring challenge.

    The survey also finds only half of respondents continually refine their service level objectives (SLOs).

    Leo Vasiliou, director of product marketing for Catchpoint, said it’s clear that SREs are becoming more proactive in terms of preventing and containing the impact of IT incidents. The challenge they face is lack of visibility and context as those IT environments become more complex. Complexity is increasing, in part, because of the rise of cloud-native computing platforms running microservices-based applications that have lots of dependencies.

    Less clear right now is the percentage of IT environments managed by SREs today. In theory, a single SRE can do the same amount of work as multiple IT administrators by automating processes using various DevOps tools rather than relying on traditional graphical tools. However, SREs are hard to find and retain. It’s also not always apparent to what level of automation needs to be attained to qualify to be an SRE or, for that matter, what certifications may be required. Interest in becoming an SRE is rising, in part, because they typically command higher salaries.

    In the meantime, it’s clear organizations will need to find ways to automate the management of IT to a much greater extent. Most organizations cannot afford to hire a small army of IT professionals to manage IT environments that become more complex with each passing day. Everything from DevOps best practices and next-generation observability tools that promise to provide more context to AI platforms will all be part of the mix. The challenge is finding a way to implement those tools before IT management spins further out of control.