devopsy.kinsta.cloud

Tag: burnout

Don’t Let Developer Toil Affect the Business Value of Your Apps

It is no surprise that the world today is driven by apps. Organizations all over the world are basing their business goals on their capacity to integrate new apps quickly, and they need to constantly reimagine how they interact with and communicate with their consumers and employees to fulfill changing expectations. To do this effectively, businesses need to operate at a high level of speed and agility. But in many cases, this isn’t happening. The phrase ‘move fast and break things’ could not be further from the truth when it comes to how enterprises approach innovation today. In fact, according to Forrester Consulting, 48% of executives say they haven’t changed their applications in a year or more. One reason for this may be that organizations are dealing with a high amount of developer toil and technical debt. Developer toil is the repetitive, predictable, constant stream of tasks that support the creation of software and how it’s run, but don’t actually affect the features in the software that people use. Or in other words “any activities that don’t directly create business value.”

Typically, developers spend too much time on processes that could be automated or even eliminated. Ironically, this toil builds up over time as their organizations prioritize shipping features rather than addressing these problems in their overall software process. While developers can change their software quickly at first, just like debt in real life, if this toil and tech debt are neglected, it takes over and consumes the organization’s ability to grow. This results in long, infrequent release cycles. A feature that seemed simple and once took just 15 minutes now takes weeks, even months, to get in front of users.

Conducting a Developer Toil Audit

Anything you can do to speed up your software release cycle will improve the quality, resilience and business value of your software. Continuously addressing developer toil provides a significant boost to your organization’s software capabilities. For one of VMware’s customers, addressing developer toil reduced the time taken to introduce a new feature significantly–from 400 hours to just 24 by doing a developer toil audit and addressing key factors contributing to this technical debt.

So how can organizations overcome developer toil? One way to do this is through a developer toil audit. This is a systematic process for finding, valuing and prioritizing fixing waste in your software process. First, the process finds wasted time and process debt in how you build and release software. Second, the process then helps you justify delaying working on features in favor of eliminating and automating your software creation process. The result is speeding up your software release cycle. The more frequently you can change and release software, even with just small changes, the more opportunities you must learn what works and put those features in front of customers, employees, and other users.

The process of conducting a Developer Toil audit includes:

1. Asking the right questions: A great way to uncover developer toil is to ask about people’s struggles, frustrations, and even boredom in the form of a tailored question set. We advise using questions that are specific to your organization in addition to general questions like “how long does it take to perform a full build?” For example, highly regulated organizations should ask about tasks involving compliance. Once the survey is completed, you can do some quick analysis to locate and prioritize developer toil.

2. Combine into usable metrics: You now have a single metric for each type of developer toil. And by combining all the questions you create a single metric to track overall developer toil. Extra points for creating dashboards with visualizations! As with all such aggregate metrics, these are more directional than exacting—their job is to point you to problems that should be solved.

3. Link these metrics to business value: A simple ranking of developer toil is better than nothing. However, before deciding which developer toil to address, we recommend linking each type of developer toil to the business value created by fixing the toil. Put briefly, the value you get will be related to the time saved, and, thus, the ability to ship more features in the future.

The less time developers spend on toil, the more time they can spend on tasks that directly benefit the business. Another way of looking at this is plain old productivity: Developers can now do more with the same amount of time.

Another important outcome is increased staff morale. The less time developers spend on boring, repetitive work—toil—the happier they’ll be. Stronger employee hiring ability and retention are (or should be) a strategic imperative for any organization that depends on software. Morale also increases software quality: happier people make better software.

Slow Down to Speed Up

With the survey results in hand, you should have a pretty good idea of which developer toil to fix. The last step is to do the usual product management prioritizing to weight fixing these items with other tasks you could do. These are strategic decisions that product managers need to make. To fix toil, you need to stop shipping features to free up time. You need to slow down the business. At the start, before product managers have been empowered to make these kinds of decisions, higher-level management should be involved to calibrate how much slow down you’re willing to put up with for future agility. The calibration you’re doing is this: If I fix one item of toil and ship just one feature (instead of two) this release cycle, then in the next release cycle I can ship four features. Humans are not great at that kind of bird-in-the-hand versus birds-in-the-bush thinking, so you’ll need to figure out what works in your organization.

Through our experience as consultants working with product teams in different industries, we’ve used Developer Toil Audits to focus and motivate product managers and developers to fix toil and speed up their release cycles and, thus, improve how their organizations build software. This directly improves the business by adding more capabilities and capacity to deliver more features, even in the short term. Each time you fix any technical debt, you gain more capacity and capabilities to create new features and improve your software. This is how you use a modern software culture to increase business agility. Or, put more simply: Less developer toil leads to better software, and better software leads to better business.

August 8, 2022
Leverage Empirical Data to Avoid DevOps Burnout

Burnout is possible in any industry. But it seems to be especially rampant in tech, a space bursting at the seams with inflated expectations and ever-accelerating trends. Nearly 60% of tech workers said they were currently feeling burned out at their workplace, found a study by TeamBlind.

Limiting burnout and supporting employee mental health is essential, not only for the workers’ sake but for the business as well. During this period, which commentators call the Great Resignation, if employees don’t feel respected, they will likely seek new opportunities elsewhere. Ensuring the needs of engineers are met is a vital target amid an ongoing cloud skills drought, a talent gap to which the DevOps space is certainly not immune.

I recently met with Eric Mader, senior solutions architect, AHEAD, to understand the root causes of burnout as it relates to DevOps. According to Mader, burnout can be caused by many factors, including work-life imbalances and conflicting egos in the workplace. But above else, burnout is most often caused by management setting unrealistic targets that aren’t based on empirical team performance data.

What Factors Contribute to DevOps Burnout?

So, what can cause burnout in DevOps? First, the workplace culture can significantly contribute to burnout. In most remote working models, explained Mader, it isn’t easy to ascertain the correct work-life balance. For example, there may be an unwritten expectation to immediately respond to Slack messages or quickly get back to emails after-hours or over the weekends.

An “always on” attitude can lead to unhealthy work habits. And while this isn’t necessarily unique to DevOps, it is sometimes unavoidable due to the high connectivity required to perform the job. Another potential contributor to burnout is the team dynamic, noted Mader. “Every team seems to be unique in its own way,” he said.

Although Agile and DevOps involve very technical work, at times, ego can get in the way of a healthy team dynamic. For example, in many tech teams, there is often a “guru,” such as a senior developer or tech lead, that everyone goes to for advice, Mader said. This person might end up doing most of the work (both technical and interpersonal) and usually feels the most burned out. Or, other wanna-be leaders might take on an unnecessary command-and-control role.

Personality differences with coworkers could contribute to burnout, but other factors like dealing with toil or managing legacy technical debt can be just as tiresome.

‘Move Fast And Break Things’

According to Mader, perhaps the biggest contributor to burnout is the insistence from executives that tech workers consistently outperform their previous output—especially when past team performance data does not support these goals. Setting achievable goals is great, but when deliverables are out of scope, it’s a recipe for burnout.

Part of the pressure to increase velocity is the constant illusion that “faster and more is better,” said Mader. Value is often attributed to the number of items achieved in a sprint instead of the quality of each point. For example, a team might deliver 200 story points of work in a sprint, but only 30 of these points carry a significant amount of value. Or, a leader might see a team accomplish 60 points of work in a sprint and then encourage 200 in the next one.

These sorts of high expectations suffer from basic capacity problems. And the added stress quickly realizes itself as a burden, contributing to burnout, explained Mader. He encourages technical leads not to force more work but instead, “focus on the highest-value work based on the empirical data you have.”

Five Tips To Avoid Burnout

Mader shared some tips on how to avoid burnout in DevOps teams. Here are some simple yet effective ideas to maintain happier engineers and a more productive, healthier team relationship.

Snuff out the always-on attitude. Organizations need to walk the talk when it comes to mental health awareness. Therefore, ensure employees know that their life boundaries are respected. Encourage this by honoring time off for holidays, paternity/maternity leave or personal vacation. Discourage emailing on the weekends and encourage turning on ‘Do Not Disturb’ alerts on team chats and work email accounts.

Prioritize what gets automated. There’s a misnomer that any automation is good automation. In fact, over-prescription of software development automation can lead to some negative consequences. “I’ve seen many DevOps teams that focus so much time on automating low-impact business processes,” said Mader. Instead, prioritize automation projects that affect more people and those that would be continually used.

Try out the ‘Fist of Five.’ Ensuring everyone’s voice is heard helps avoid a toxic working environment. One tactic to quickly gauge reaction in team meetings is to use what Mader calls The Fist of Five: One if an idea is totally off base and five if an idea makes total sense. This simple practice could ensure everyone gets to contribute input on new projects.

Take the quietest person in the room aside. Only the squeaky wheel gets the oil, Mader noted. With that in mind, take aside the quietest person post-meetings to gather their input—you might be surprised. Junior folks who may be less apt to speak their minds typically have a less-institutionalized mindset and could suggest more innovative ideas.

Use empirical data to inform future goalposts. Finally, Mader encouraged DevOps teams to “use empirical evidence to back up on where you’re going in the future.” Specifically, in the DevOps sphere, empirical data takes the form of metrics like mean-time-to-recovery, or SLAs, SLIs and SLOs.

Remember that even if a team did accomplish a certain number of goals in a sprint, it doesn’t mean they can produce the exact same results the next time around—they could have been pushed to their limit to produce past results. Thus, managers should aim to move the needle on metrics progressively, not in giant strides that upset morale.

Take an Opinionated Approach to DevOps

Nowadays, DevOps teams are spoiled for choice—there are countless tools on the market and an endless stream of best practices and methods. Even within a single company, the DevOps toolset may vary considerably from team to team. To help make sense of it all, Mader encouraged DevOps leads to simplify the transformation journey by setting guardrails and adopting a more opinionated approach around what tools and processes you take on.

August 8, 2022
Majority of Open Source Devs Open to Change Jobs in 2022

Open source is the foundation of most modern software applications. Now more than ever, retaining developers with experience working on and with open source projects is critical to maintaining software development agility. Yet amid the Great Resignation, it can be challenging to keep talent happy—organizations must demonstrate a commitment to employee benefits, mentorship and cutting-edge technologies to remain competitive.

The EDB Open Source Talent Survey 2022 surveyed over 1,400 developers and DevOps professionals working on open source projects. Below, we’ll highlight the key takeaways about their volume of work, the quality of on-the-job training and top motivators that could influence workers to take a job elsewhere.

The Great Resignation Persists

It’s not news that IT is under mounting pressure to deliver new features and applications to meet surging digital demands. As a result, the amount of work has increased across the board. The report found that 67% of respondents said their volume of assigned work has increased in the past 12 months.

Employer favorability has also declined slightly—28% have a less favorable view of their employer over the last 12 months. Although 46% are satisfied with their current job, 44% are satisfied but would be open to new opportunities. And 10% said they were not satisfied with their current position. Taken together, that means 54% are open to or actively looking for new roles.

Mentoring: An Enticing Benefit

So, what are the main drivers that would influence IT workers to change jobs? It’s no surprise that increased pay and benefits are a top reason behind most resignations. However, salary is not the only factor—43% say improved career opportunities influence a job shift, a steep rise from 21% in 2021.

On-the-job mentorship and training are also becoming more desirable. A little over a third (38%) said that better access to mentorship would make them consider another job opportunity. This was followed by working with more cutting-edge technologies (32%), greater access to training and certifications (30%) and allotting more time for open source projects (19%).

Interestingly, one-fifth of responders who said they were satisfied with their current employer said their organization introduced a mentoring program in the past year. This indicates that investment in employee career advancement helps increase morale and retain skilled workers.

Career Advancement And Training

Out of the above reasons developers are open to switching jobs, one factor in particular is increasingly important: Working with cutting-edge technologies. This figure doubled in the last year, rising from 16% to 32%. Most engineers prefer not to work with cumbersome legacy technologies or manage rising technical debt. And if an organization isn’t evolving its software infrastructure, employee skillsets become stale.

Working with new open source or cloud-native technologies can help retain workers, but will likely require training. In terms of specific categories, 41% said that training in emerging technologies, like cloud, is a favorable factor in considering another job opportunity.

From a security perspective, ongoing training is also vital to staying on top of emerging exploits. As a result, 43% said security training is an interesting perk when considering a job shift. These areas are followed by training on core technologies like database and infrastructure (38%) and training on compliance (37%).

Final Thoughts

Open source software is pervasive and at the heart of today’s digital innovation. Nearly all applications depend on some form of open source software, but it’s no less prevalent in the DevOps sphere, where open source projects are used in continuous delivery, container orchestration, observability and a multitude of other stages of the software life cycle.

Supporting the needs of skilled labor will be necessary to leverage the benefits of open source projects. As such, organizations will require not only competitive wages but also relevant training and investment into cutting-edge technologies to retain such developers.

Another factor that could drive workers away is unnecessary toil, which can negatively affect morale. A company’s environmental, social and governance (ESG) initiatives can also affect its talent pool. Furthermore, a commitment to high-quality developer experience can ensure the needs of developers are heard and respected.

The EDB Open Source Talent Survey 2022 surveyed over 1,400 application developers and IT professionals from around the world. It targeted open source technology users, specifically those holding application development, information technology and operations titles. For additional insights, you can read the full report here.

July 29, 2022
Eradicating Developer Burnout With a Flexible Workforce

Burnout can affect people no matter what their job role, but software developers are particularly vulnerable. Too often, businesses chase profits by chronically relying on heavy workloads delivered by employees in short turnaround times. Developers often undertake intense work and find themselves pushed to meet deadlines under great pressure over and over again. But without proper care, burnout is almost inevitable. After all, developers are humans first before they are developers.

Programming is a cognitively stressful job that requires individuals to constantly solve complex problems. Working for hours on end day after day with no break is tough for anyone, and this pattern over a period of weeks is all too familiar for developers.

But slowly, the face of the workplace is changing. We are seeing a post-pandemic focus on employee well-being, and the current job market conditions mean that professionals have choices and the right to demand better care or even explore self-employment or non-traditional job paths (thanks, Great Resignation!). In this context, businesses are also thinking differently about how they work with talent to use workers more efficiently. There is an opportunity here to lower the burnout risk facing developers for the benefit of employees and employers.

Organizations Can’t Afford Burnout

Developers are key to most digital transformation projects, so organizations can ill afford for their developer teams to suffer from burnout. This can significantly delay the rollout of new products, features or apps and slow down DevOps initiatives, which can’t progress without the tech teams responsible working consistently and productively.

This risk to developers is particularly high in the wake of the pandemic, with technical staff under additional pressure to play catch-up and deliver projects that might have stalled. During this period, there has been little consideration for employee well-being; a 2021 study found that 83% of software developers were experiencing burnout. It’s not just about writing code—technology teams also struggle with inefficient engineering processes, backlog and technical debt and are more likely to encounter friction when trying to deliver work quickly, leading to a greater risk of burnout.

Low quality in-house software is also a point of frustration for development teams, with 83% of engineers reporting being concerned about software reliability in their organisation. Prioritizing the right tools for these teams is one way that businesses can help ensure projects run more smoothly. It’s a simple fix, but one that is frequently overlooked, though empowering technical teams is increasingly top-of-mind for businesses.

Developers Aren’t Easy to Replace

Great developers come with years of experience, learning their trade through practical problem-solving. They’ve never been easy to replace, but the state of the current global job market means it is more difficult than ever. This is inherently linked to developer well-being; some are changing jobs and moving to other companies in search of better working conditions while others are creating their own. For example, many are switching to freelance roles to gain independence and control of their own workloads.

However, freelancing can be lonely and admin heavy. Despite the improved flexibility, the lack of support can also lead to burnout. Talent clouds offer a potential solution to this through closed communities offering structured work for freelancers and regular interaction with peers. They have seen a surge in membership and have driven wider uptake of self-employment. In fact, 11.2% of developers were freelancing in 2021, up from 9.5% in 2020, as they sought to balance the freedom of freelancing and some of the perks of permanent work.

And that’s crucial. For a role that’s considered ‘always on’, the flexibility of freelancing to decide when or where developers work could be crucial to delivering quality, secure code. Without wasting hours a day commuting—that could otherwise be spent working or, crucially, decompressing from their intense workload—it’s been the option of choice for overworked developers all over the world. IT is the foundation on which modern businesses are built—giving the architects of these businesses some of this flexibility is a small price to pay to ensure their well-being. And, as a result, the timely delivery of products, projects, features and apps that accelerate digital transformation initiatives.

It’s a Two-Way Street

We’ve already seen how the increased workplace flexibility afforded to freelancers translates into better productivity and lowers the risk of burnout. Equally, 31% of developers worry about unclear goals and processes, so there is also something to be said for improving project management of digital initiatives to ease the burden on technical staff. In fact, 58% of devs feel that burnout is not inevitable, and that better working practices and management are key tools to its prevention.

For businesses wanting to deliver projects quickly, harnessing on-demand workers from the increasing pool of freelancers means they can engage specialists depending on the type of work or coding language being used, but also that they are fresh to tackle each new challenge—a key concern among developers who have experienced burnout. Doing so via a private talent cloud also ensures that the burden of managing and vetting freelancers is taken away from the organisation and nixing potential concerns about bringing in external talent.

Eradicating Burnout

Well-being is in the public eye more than ever. On one hand, developers should not have to face the high risk of burnout; on the other, businesses can’t afford to lose the talent fuelling their digital future.

Both parties can help solve the issue. Developers are taking professional well-being into their own hands by exploring freelance careers that give them flexibility. On their end, businesses need to consider creative ways to access talent. In part, this is about ensuring internal technical teams have the right support in place so they do not become overworked. It’s not easy to find this support in the current job market, but with developers flocking to freelancing, there is an opportunity to rapidly engage highly skilled workers on an on-demand basis at pinch points. After all, a well-managed workload significantly reduces the risk of burnout.

We have the resources and available skills to eradicate burnout among developers—we just need to arrange them in the right way.

July 12, 2022
Essential Ingredients For a Remote DevOps Team
Enterprise business processes are run on evolving applications, and the DevOps team needs to be as efficient as the applications they create. Unfortunately, the COVID-19 pandemic has had a profound impact on DevOps productivity—and that impact hasn’t always been positive.

Tracking the performance of the DevOps team has become more challenging in a remote and hybrid world. Like everyone else, developers have become remote workers throughout the pandemic which means adapting to a new way of working. There has been a lot of research done on the impact of work-from-home on productivity. Of course, measuring any developer’s productivity was already a complex task. It’s not as though you can use a simple metric like the number of lines of code produced per day; it’s a lot more complicated than that.

Working from home has made it even more challenging for developers to stay productive. There are dozens of distractions when you work from home, and research shows that it can take even the most skilled developer 15 to 20 minutes to resume work once they have been interrupted. There’s also more stress associated with balancing work and home life when working remotely. According to Catalyst, 92% of workers say they are feeling burnout from the stress of remote work.

Software Development Thrives on Remote Work

In many ways, the trend toward remote work has been a windfall for developers. The nature of coding has always lent itself to a hybrid or remote work environment, but unfounded concerns about securing intellectual property have always been an impediment.

A recent Gartner survey revealed that employees are three times more likely to become high performers when given flexibility as to where, how, and when they work. Remote work also enhances flexibility and eliminates wasted time commuting. And work-from-home reduces childcare responsibilities. Catalyst found that 32% of women with child-care concerns are less likely to leave their jobs if they have work-from-home as an option.

Remote work also eliminates geographic barriers, which is a well-understood benefit in the software industry. Being able to hire programmers working from home anywhere in the world increases the size of the talent pool and makes it easier to find developers with the right skillset.

Of course, there are challenges for DevOps teams that embrace a remote work model. Communication is probably the biggest challenge. Zoom meetings may not be as effective as in-person meetings since it’s harder to read facial expressions and pick up non-verbal cues. It’s up to the team leader or senior manager to promote effective communications. It’s also important to have the right tools, have the right communications channels and maintain respectful interactions. Even with a remote work environment, the team dynamic can be adversely affected by the misuse of responsibility, behavioral issues and disrespectful behavior.

Building a Remote DevOps Team

Supporting collaboration in a hybrid work environment requires new processes, new guidelines and new tools. The best place to start is by choosing the right collaboration platforms and procedures. For example, email is a poor channel for immediate requests, but you don’t want to be answering continuous phone calls. Establish protocols for using email, chat, teleconferences and meetings. Adopting cloud communications tools also gives you more agility and scalability.

When choosing the right collaboration tools consider security. Working off-site gives you less control over enterprise access. Remote workers access enterprise systems using different devices and public and private networks. To reduce vulnerability to cyberattacks provide safe and secure network access with multiple security layers. Adopt virtual private networking (VPNs), remote desktop protocol (RDP) clients, and malware protection. Workers tend to be less security conscious when working remotely so take what steps you can to reduce risk.

In addition to having the right tools, you must have the right approach. To continuously improve efficiency you need transparency, inspection and adaptation, just as you do with DevOps. Start by establishing and documenting a clear set of expectations for the remote team. Also, create a leadership hierarchy. Even though they are working from home, everyone needs a work structure. It’s up to the team leaders to create a stable work environment. Lack of clarity removes that sense of structure and safety and can undermine the team.

Maintaining transparency is the best approach for any remote work culture. Establishing a workplace fueled by honesty and openness and where every voice is heard ensures everyone is working to achieve a common goal and keeps morale high.

Maintaining a successful remote work environment depends on respecting the four Cs: Communication, collaboration, connectivity and culture. The four Cs are interdependent and work together to create an effective digital work environment, providing an employee experience across locations, devices, channels and situations.

Creating a Productive Environment

Once you have laid the foundation for remote collaboration, the next step is to monitor productivity. There are a few ways to look at DevOps team productivity. You can look at the final output of the overall effort, assessing the application that is the end product. You also can measure overall team productivity.

To monitor key metrics to assess team productivity, project managers must be prepared to drill down to evaluate hot spots:
- Measure output for specific developers and see how it stacks up against the productivity of other team members.
- Maintain leaderboards highlighting top contributors. Using gamification creates healthy competition and can inspire greater productivity when top performance is recognized.
- Create a conducive environment by providing the right set of tools and motivating engineers to be at their best.
- Monitor distractions. Coding is an intensive activity and disruptions harm productivity.
Don’t measure productivity based on coding. Developer contributions also include designing new features, functional discussions, code review, documentation, testing and bug fixes, meeting with customers and other tasks.

Improving DevOps productivity is also a product of automation. In addition to streamlining real-time collaboration with tools such as instant messaging and Microsoft Teams, shortening development time with automated testing tools and AI-powered test frameworks eliminates manual processes and frees time for more productive tasks.

If you are looking for a silver lining from the pandemic, consider the evolution of the remote work environment. Supporting remote DevOps developers is nothing new, but forcing employees to work from home has made the hybrid work environment more efficient. DevOps teams continue to invent better ways of working that promote flexible work hours, higher productivity and a better work-life balance.
April 1, 2022
White House Meeting Puts Spotlight on OSS Sustainability

A recent meeting between IT industry leaders and White House officials highlighted open source software sustainability concerns as high-profile breaches and zero-day attacks have many organizations reviewing their software supply chains.

The White House published a statement describing, among other things, how participants had a “substantive and constructive” discussion on how to make a difference in the security of open source software while continuing to effectively engage and support the open source community.

Meeting participants included Deputy National Security Advisor for Cyber and Emerging Technology Anne Neuberger, National Cyber Director Chris Inglis, and officials from the Office of the National Cyber Director, Office of Science and Technology Policy, the Department of Defense, the Department of Commerce, the Department of Energy, the Department of Homeland Security, the Cybersecurity and Infrastructure Security Agency (CISA), the National Institute of Standards and Technology and the National Science Foundation. Akamai, Amazon, Apache Software Foundation, Apple, Cloudflare, Facebook/Meta, GitHub, Google, IBM, the Linux Foundation, the Open Source Security Foundation, Microsoft, Oracle, Red Hat and VMware all sent representatives.

The White House reported that the discussion focused on preventing security defects and vulnerabilities in code and open source packages, improving the process for finding defects and fixing them and shortening the response time for distributing and implementing fixes.

There were few specific recommendations, but the White House is clearly bringing more pressure to bear after the disclosure of the zero-day Log4j vulnerability in Java applications wreaked havoc in enterprise IT environments and government agencies. That vulnerability made it clear just how dependent organizations are on open source software projects and brought to attention the fact that many are created and maintained by just a handful of volunteer maintainers and contributors. The individuals that created those projects don’t always have a lot of cybersecurity expertise. In fact, many of them would argue that the onus for securing open source software is on the organizations that use what amounts to free software. It’s not the responsibility of the contributors and maintainers of open source software to drop everything and create a patch on-demand to address a zero-day vulnerability.

The federal government, however, has made it clear via executive order that it expects IT vendors and large enterprises that depend on open source software to do more to secure it. In the meantime, IT teams will need to evaluate their dependence on open source software; especially if, from a security perspective, that software isn’t sustainable simply because there are not enough contributors with the necessary expertise.

Naturally, this is a complex issue. In many cases, organizations are relying on open source components without even realizing it. Those components have been embedded within an application by an independent software vendor (ISV) that typically doesn’t disclose how that application was constructed. When a zero-day vulnerability is disclosed, cybersecurity teams can spend weeks looking for all the ways the instances of an application they either built themselves or licensed are impacted.

“There’s no way to really know,” said Mitch Ashley, principal for Techstrong Research, an arm of Techstrong Group, the parent company that publishes devopsy.kinsta.cloud. “A vulnerability can be anywhere.”

It’s not abundantly clear how critical an issue the sustainability of open source software security is just yet. Relative to the amount of open source software currently employed, the number of security issues that have been encountered is comparatively small, noted Ashley.

“The innovation benefits enabled by open source software far outweigh the risks,” he said.

It may be a while before the open source community comes to terms with rising software supply chain concerns. However, it’s clear that many more members of the IT community are about to be held accountable for it.

January 18, 2022
Incident Resolution for Remote Teams
People working in IT support and incident management right now are faced with unusual difficulties supporting large remote workforces and managing unpredictable workloads. On Reddit, system admins and other IT pros are bemoaning the hiccups and hassles of working in isolation while trying to resolve issues and maintain high SLAs. You can’t go grab your indispensable SME for troubleshooting, because that person is also home and inundated with messages and alerts from many different tools.

Networks, VPNs and servers are being hammered extra hard right now from higher usage of SaaS apps and collaboration tools. User expectations are even more grandiose than normal: The home-based IT environment is a lifeline to whatever remains of human interaction. IT organizations are expected to make applications and services work flawlessly–overnight. Undoubtedly, IT pros are working with assiduity to keep the wheels of work moving for their employers during a time of crisis. Yet, working longer hours isn’t the answer. We need new ways of working—ones which may prove better than the current status quo for the long term.

In many organizations, the process of incident management and resolution begins with people looking at screens full of metrics. You see a spike in something, such as network I/O, and immediately begin to investigate it, assuming the worst. A more adverse scenario is when users call out problems on Twitter, Reddit or other popular tech communities, creating a noxious PR situation which IT and the business must swiftly respond to and manage. This ad hoc, reactive fire drill takes time, and resolving that particular issue may not be the most important item of the day. In software development, it’s akin to the bad practice of designing the architecture of a system first and then the user experience.

Given everything that’s at stake in the foreseeable future of stay-at-home mandates, SREs and IT support technicians need to flip the incident management process. Rather than beginning with the metrics and working backwards to uncover the problem, if there even is one, start by looking for the most important problems to solve. Then, discover the resources and components which support that end-user process and finally, analyze the metrics which are most relevant. For example:
1. A B-to-C provider of a streaming video service is now seeing a 200% increase in daytime traffic. This is resulting in sluggish load times for users and glitches.
2. IT has tagged the components beneath that service, so IT operators know exactly which VMs, databases and load balancers come into play.
3. Next, the operations team can identify the most relevant metrics, for instance CPU idle time, to measure and take steps to correct.
The Problem-First Approach

It sounds so simple: flip the process. Yet, this is not the case. IT operators and SREs are not accustomed to thinking in terms of which problems to solve or prevent. Being proactive or predictive hasn’t been the common workflow.

Here is how IT operations and DevOps teams can take a proactive approach to managing the availability and health of enterprise services.

Coordinate Across Departments

Identifying the most critical problems to solve right now is the hardest step and the one which will take the longest. Common problems these days are related to cost and scale: How do I right-size my environment to reduce costs or handle more workloads? Broaden your field of inputs to narrow down to the specific issue, such as slow page loads on the financial reporting website. Product managers, account managers and business unit leads who are the closest to the customer experience can deliver feedback on the top issues affecting customer/user satisfaction. Teams should also review recent support tickets to identify common themes of pain.

Fine-Tune Your Approach to Metrics

The Utilization Saturation and Errors (USE) Method is one way to approach the problem-first incident management process. As detailed by Brendan Gregg, a senior performance architect at Netflix, this methodology begins by posing questions, seeking answers and working backwards to the metrics. For each resource that you want to measure, identify three metrics: one for utilization, one for saturation and one for errors.

“The USE Method has made you aware of what you didn’t check: What were once unknown-unknowns are now known-unknowns,” Gregg explained.

Standardize Workflows

Create a common process for incident management across all your teams. Without the advantage of having almost everyone in the same room to huddle together in an ad hoc fashion when big issues crop up, it’s imperative to institute clear steps and roles. Doing so will prevent the frustrations, confusion and oversights that needlessly delay resolution. Since most incidents are composed of multiple contributing factors, teams need to adopt a small number of user-friendly tools to document and organize the information. At our company, we’re now using tools such as Miro, an online whiteboard application, to replace our physical whiteboarding sessions. Of course, there’s also Zoom, Slack, Jira and a host of other cloud-based tools already in place at many organizations. Mandate which tools everyone should use, with some guidelines on how to use them.

Increase Automation

In some organizations, scaling requirements in response to demand have increased tenfold. Automation is playing a critical role now; moving away from a web GUI, for example, is more scalable and aligns with modern tools such as Chef and Puppet. User tickets can be auto generated, for instance, from emails and linked to code management systems like GitHub. Modern development and operations teams are also expanding automation in unit testing and provisioning.

Watch for Burnout

Whether there’s more work and/or a need to fill the hours during long quarantine days, many software engineers, testers and architects are working longer days right now. Yet, exhaustion and burnout can lead to errors and oversights along with low morale. It’s up to managers to make sure that employees are taking breaks, working reasonable days and having the time and energy to attend to personal needs.
April 21, 2020
DOES 2019: Understanding the Effects of Burnout
If you work in the IT field, cybersecurity, the startup world or any other high-pressure job, you or someone you know has probably suffered from burnout in one form or another. If so, you already know that burnout is a real condition with significant consequences. In fact, the World Health Organization (WHO) recently recognized burnout as an occupational phenomenon.

The WHO describes burnout as follows:

Burn-out (sic) is a syndrome conceptualized as resulting from chronic workplace stress that has not been successfully managed. It is characterized by three dimensions:
- - feelings of energy depletion or exhaustion;
  - increased mental distance from one’s job, or feelings of negativism or cynicism related to one’s job; and
  - reduced professional efficacy.
I have been on the road pretty much constantly for the last three or four months (no, I am not trying to say I suffer from burnout). We have covered lots of great conferences. One of the best, as usual, was Gene Kim’s DevOps Enterprise Summit in London. We have about 30 videos from the conference up on the Digital Anarchist site. Of those interviews, my favorite is the one in which I had a chance to sit down with three PhDs to discuss the issue of burnout. I say three PhDs not just to throw around titles—these are three recognized experts in their field.

The trio of panelists are:

Dr. Christina Maslach — Probably the foremost authority in the world on burnout.

Dr. Steven Spear — Senior Lecturer at MIT Sloan School and the Engineering Systems Division at MIT.

Dr. Nicole Forsgren — Founder of DevOps Research and Assessment, now part of Google.

This interview is a little bit longer than most, slightly more than 30 minutes. But I think it is well worth your time to hear what these three experts have to say on the issue of burnout and workplace fatigue.

In addition to our interview on the Digital Anarchist set, I am also happy to include the panel discussion these three folks had on the main stage of DOES London with Gene Kim.

More importantly, if you or someone you know is suffering from burnout symptoms, please reach out for help and do something about it. You are not alone.

Don’t forget! DevOps Enterprise Summit Las Vegas 2019 is happening Oct. 28-30 at The Cosmopolitan Hotel. Be sure to register here if you haven’t already and join us for the premier event for technology leaders of large, complex organizations implementing DevOps principles and practices. You can also preview the event by watching presentations from past DevOps Enterprise Summits.

— Alan Shimel
August 2, 2019