Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Agile is bad, but “Wagile” is worse.
Plus: This prod is even worse than yours. (more…)
Category: DevOps Culture
-
Agile Sucks (Redux) | Plus: DevOps on Mars
-
Building a Platform for DevOps Evolution, Part One
Winston Churchill once said, “To improve is to change; to be perfect is to change often.” While nearly everyone will confess that change is necessary to grow and succeed, we all know it’s much harder, especially when human behavior is involved. Professionals who work in DevOps environments know this all too well.
Adoption of DevOps principles and behaviors should be viewed as a gradual process. Many organizations make false starts or attempt to implement a change only to see it fail. Sometimes this is due to changes in leadership, market conditions or technology, but then organizations catch their second wind, build momentum and reaccelerate their evolution.
DevOps can Change
With a shared vision, mutual goals and hard work, DevOps can change to keep up with user demands. DevOps professionals will always confront two potent barriers: Cultural and process debt and technical debt.
Cultural and process debt occurs when leaders refuse to delegate authority, distrust their teams and lack a value stream approach to work. Technical debt develops when monolithic systems are tightly coupled, automation is delayed for testing, environments and version control and there is no access to cloud technologies.
Although these obstacles are challenging and complicated, many organizations still meet them head-on. Evolution depends on a habit of making improvements every day through the use of an improvement kata such as PDCA (plan, do, check, act), which involves long-term strategies and objectives and continual experiments between the current and next target states.
An evolving DevOps approach maintains moving pieces across a large organization of hundreds, thousands or even tens of thousands of people who cannot change overnight. Many lack the will and the cognitive capacity to embrace and enact change.
This environment requires release managers to delegate power and transition from centralized commander to an enabler, understanding different value streams have varied skills and requirements. Mark Peters, a cybersecurity specialist and DevOps leader with years of military operations experience, outlined his top release management traits in the context of DevOps ways of working as automated processes, distributed accountability and accelerated delivery.
PDCA Improvement Kata
DevOps proposes that we adopt an improvement kata in our day-to-day efforts as an improvement habit, consisting of four steps that are repeated:
1. Set the long-term vision
2. Understand the current state
3. Identify the next target state
4. PDCA between the statesExperimentation is at the heart of the PDCA cycle. The most practical experiments are empirical, data-driven and date-specific. Using a value stream management platform to give data to support the experiments leads to data-driven discussions on real and measurable success.
In the next part of this series, we will begin exploring different experiments DevOps organizations can conduct to drive change and improve their release management procedures.
-
The Rogers Outage of 2022: Takeaways for SREs
When, eight years from now, folks are creating lists of the top IT incidents of the 2020s, there’s a good chance that they’ll include the Rogers outage of 2022. The failure, which made internet and cellular network service unavailable for more than 12 million users across Canada, was one of the most significant outages in memory, in terms of both the number of affected accounts and the level of service disruption.
For site reliability engineers, there’s a lot to learn from this incident, from both a technical and a crisis-management perspective. Here’s a look at our main takeaways.
The Rogers Outage: What Happened?
Before diving into lessons for SREs, let’s go over what happened during the Rogers outage and what we know about the cause of the incident.
Rogers is a major telecommunications provider that serves around 12 million customers in Canada. Early on July 8, 2022, it suffered an incident with its infrastructure that made internet and wireless phone service unavailable for most or all of its customers. The outage was so complete that users couldn’t even make 911 calls or withdraw money from ATMs–which means the failure brought down not just ordinary services, but also what you might consider resources that are essential for normal daily life.
Connectivity was restored for most users within about 15 hours. However, some customers reported continuing disruptions for several days after the beginning of the outage.
What Caused the Rogers Failure?
At first, Rogers did not reveal much technical detail about what triggered the incident. A statement attributed to the company’s CEO the day after the outage said that a “maintenance update in our core network … caused some of our routers to malfunction.” But that didn’t explain specifically which type of maintenance update the company was trying to apply, why the update didn’t go as planned or which types of routers were affected.
It wasn’t until several weeks later, when Rogers submitted a response to questions from the Canadian Radio-Television and Telecommunications Commission, that more technical detail came to light. Rogers explained to the CRTC that, during an update process, the company ran code that deleted a routing filter. It’s still unclear exactly how the bad code made its way into the update routine.
Lessons for SREs
For SREs, the Rogers outage is an example of a worst-case scenario. It was an extremely disruptive incident that took a relatively long time to mitigate, at least when you consider how severe the outage was. That’s why SREs should use Rogers’ experience to assess what SRE teams can do to avoid issues like this–and manage them more effectively when they do happen.
Test Your Updates
From a technical perspective, one of the biggest takeaways from the Rogers outage for SREs is that you should test updates thoroughly before applying them. If engineers had tested their update routine more carefully, they would presumably have detected the bad code that deleted the routing filter before the deletion actually took place.
To be fair, Rogers is hardly the first company to experience a major outage due to a bad update. Atlassian, for instance, had a similar issue earlier this year, when a buggy migration process caused a long-lasting outage.
If you’re an SRE, you don’t want your company to be the next to make headlines when something that should be a simple change, like an update, triggers a massive outage. Test, test and test again before bringing your changes into production.
Build in Redundancy
It also seems reasonable to conclude that Rogers didn’t have redundancy built into its infrastructure in the form of backup routers that could take over when the primary routers crashed due to the bad update.
This is another reminder to SREs about why it’s so critical to avoid putting all of your eggs in one basket or creating single points of failure. Redundancy may cost more, and it may not totally prevent disruptions. But it can make them less severe and easier to recover from. It’s not hard to imagine the Rogers incident being a lot less severe if even just 20% or 30% of its traffic could have been automatically redirected to backup routers. That might have been enough to keep critical services operational, at least.
Have an External Crisis Communications Plan
Rogers has received more than a little criticism for not communicating well with its customers during the outage. The company waited hours before releasing an initial statement about the failure. It was not very responsive to media queries early on, and, as noted above, it took several weeks before Rogers finally released technical details about what went wrong (and even then, the details emerged only because of a government inquiry into the incident).
The lack of rapid, transparent communication probably wasn’t deliberate. We have to believe that Rogers didn’t want to leave customers in the dark. Instead, the slow communication likely reflects the absence of a pre-planned strategy for external incident communication in a situation where Rogers’ own networks were non-operational. Without a communications plan, Rogers presumably struggled to decide which information to share, or how to share it, in the midst of a major crisis.
If Rogers had a playbook in place for communicating with the public and media about this type of issue before its network went down, it probably would have been able to share information more effectively–and it may not have faced as much of a backlash from customers and regulators.
Internal communications, at least, presumably were smoother. The company did manage to get service back up for most customers within about fifteen hours, which probably would not have happened if its engineers lacked a means of communicating with each other as they worked to resolve the incident.
Conclusion
For SREs, the Rogers outage of 2022 is a lesson in the importance of testing, designing redundant infrastructure and preparing for transparent communications before crisis strikes. While more careful planning in these respects may not have totally prevented the incident, it would likely have at least reduced its severity and mitigated the backlash that Rogers now faces from angry customers and regulators.
-
5 Ways to Prevent an Outage
In today’s always-on, ever-connected world, we all expect 100% availability.
What gets in the way of this? The devil is in the details. Over time, everything breaks: Disks, nodes, containers, networks, DNS servers and configuration issues can all lead to major outages.
Amazon’s 13 minutes of downtime in August 2021 translated to almost $5 million in lost revenue and Amazon Web Services’ three outages in December of last year cost millions more. And it was widely reported that the October 2021 Facebook (including related products: Messenger, Instagram and WhatsApp) outage resulted in an estimated $100 million in lost revenue due to a network configuration error.
There have been many outages of varying scope and duration—some attributed to technology breaking, others traced back to people making mistakes. In fact, a 2017 Amazon outage was attributed to “human error”–or more specifically, one employee’s typo. The employee intended to take a small number of servers offline to fix a billing system issue but the incorrect command led to several servers being affected and a major outage.
So, what can your company do to reduce the risk, duration and impact of a potential outage? For one, companies can’t pretend that if they only try harder, they can prevent humans from making mistakes. Do you have dozens of people manually keying in hundreds of commands every day? If so, a mistake is inevitable. Companies should instead investigate how and why one small blunder in a command line could do widespread damage. Guardrails and redundancies should be in place to protect against and minimize these types of incidents.
As the enterprise continues its digital transformation toward a multi-cloud, hybrid and distributed world, here are five guidelines to consider—guidelines that can help you preempt and prevent your next big outage.
1. Always Consider Your Blast Radius
Blast radius measures how one error, typo or malfunction can impact other parts of a system.
What is the chain of components or services that is impacted by one error?
What are the hidden dependencies?
How many customers would be affected and for how long?
More is connected than you think; discover how it’s all related in advance to eliminate surprises.
If too many systems in a network are mutually dependent and interconnected, the result is deadlock. In other words, when one breaks, the other also fails. We have seen this in past Facebook outages. A single erroneous command sparked a domino effect that shut down the backbone connecting all of Facebook’s data centers globally.
How can you prevent this scenario? Plan for the worst possible scenario. When it comes to mission-critical systems, take an intentional approach to situation failover and backup. What if A, B and C all happen simultaneously? Also, think about your cost versus reliability trade-offs. Maybe you can’t afford two full sets of servers but can afford a full set of archived code.
2. Smooth Out Rough Edges
Consider that core infrastructure and code get used (and tested) a lot more than edge code/infrastructure. Basically, the core is hardened while edge infrastructure is more vulnerable. You should be more proactive at the edge around identifying vulnerabilities and doing explicit testing to counter this risk.
The 2014 Google disk erase/satellite outage is a cautionary tale involving the edge. In Google’s case, the edge networks offered a key advantage: A better connection for the customer due to lower latency since data is accessed from a source closer to the user as opposed to the core network. Google had a CDN/edge network for TCP connection termination, caching for various apps, DNS regulation, etc. Teams needed to routinely upgrade a satellite rack and run decommission automation—and this automation failed part of the way through. Error message. They retried the decommission process again to debug the failure. The manual re-execution of the decommission automation meant thousands of satellite machines were decommissioned and, within minutes, the disks of all satellite machines globally were erased. The result: Customers experienced an increase in latency. Behind the scenes, it required two days of all-hands-on-deck to reinstall these machines.
3. Think Escalator, Not Elevator
A good system has both defense and depth. Considering that it takes three or four things to (really) go wrong, think about failure cascades. Bluntly put, limit dependencies. Measure the longest dependency failure path and mean-time-to-recovery (MTTR). When you do this, think escalator not elevator. What do I mean? Well, when an elevator fails, you are left with a shaft; an escalator that fails still works as a staircase.
In 2016, severe weather in Sydney, Australia resulted in an AWS EC2/EBS outage. First, the utility provider suffered a loss of power at a regional substation. This failure resulted in a total loss of utility power to multiple AWS facilities. In one of the facilities, AWS power redundancy didn’t work as designed and power was lost to a significant number of instances in that availability zone. To make matters worse, the instances that lost power lost access to both their primary and secondary power as several power delivery lineups failed to transfer load to their generators. Also, a latent bug in AWS instance management software resulted in a slower-than-expected recovery of the remaining instances.
The lesson? This correlated power delivery lineup failure illustrated that failures are not independent. You must design for the largest correlated failure in combination with the background noise of ongoing independent failure. Recovery code needs to be scale-tested against the largest expected failure.
4. Stay Ahead of Digital Certificate Rotations
Don’t underestimate the importance of digital certificates—and their expiration dates. A few years ago, LinkedIn allowed one of its SSL certificates to expire and it knocked out LinkedIn sites in the U.S., UK and Canada. In 2017, Equifax missed a breach for 76 days because of an expired certificate. The average annual cost of unplanned outages due to certificate expirations is $11.1 million.
For example, you could have a load balancer sitting in front of several servers. The load balancer could have an up-to-date certificate while the servers behind it are expiring. It is important to check for expiring servers in multiple ways. Ideally, you check the boxes in two ways: Ping the HTTP endpoints and also have scripts that check the certificate files themselves.
Expired certificates (TLS for example) can lead to encryption and authentication risks as well as lost internet connectivity and larger outages involving websites and applications. To avoid this, you continuously check, renew and deploy certificates—and this can be achieved via automation. Think about automatically rotating expiring certificates.
5. Tackle Toil to Empower Your Teams
What’s the biggest headache for site reliability engineers (SREs) and those who manage production operations? Toil.
We can define toil as manual, tedious tasks that engineers perform within production environments. Basically, toil wastes time and slows down operations. As a result, most SREs and engineering teams strive to minimize toil within their workflows.
In production operations, you should work to limit toil. Make sure your team is regularly investing time in proactive work; the reactive work shouldn’t take most of their time. Toil reduction gives developers, DevOps engineers and SREs more time to focus on projects that create net-new value for the business (new application features). Google stated that it has an “advertised goal” of keeping toil “below 50% of each SRE’s time.” Basically, no engineer should spend more than half of their time on manual, repetitive work. But the ultimate goal should be to get toil as close to zero as possible. Automating incident remediation can help SREs spend less time on toil and less time on-call.
Also, when you implement fixes—especially people and process fixes—ask yourself if this is a systemic fix or one that is relying on best intentions. Look for tools to safely empower your team (with guardrails). As you work to identify major vulnerabilities and dependencies (as discussed above), you should automate away the most common issues while planning for the “corner cases” (or the 1% of 1%) that may lead to major outages (a.k.a. the worst-case scenarios).
As you consider these guidelines, make sure you focus on putting actual mechanisms in place—not one-off, “hopeful” directives. Do not rely on people to remember best practices from a training session—these are not mechanisms. Instead, design processes that can be systematically implemented. When you do a post-mortem, look for concrete programs you can put into place that will prevent the same mistake from happening again. Automation can help you achieve this level of confidence, management and quality control—and it may just keep you one step ahead of a potential future outage.
-
Next-Level Tech: DevOps Meets CSOps
Operations are some of the most important aspects of an organization. In tech, DevOps has dominated the conversation due to its necessity in keeping up with the rate of building products, applications and infrastructure with code. DevOps encompasses much more than a team of developers; it is a culture, a set of practices and tools that allow organizations to deliver applications with speed and scale and then effectively operate. This philosophy has been incredibly useful over the last decade but has lacked a connection with other parts of the organization. Developer teams are connected, but within their silo, their work is lacking integration with the end user, diluting the opportunity to deliver the best solutions for their customers. More immediately, developers, DevOps and ITops cannot see the breadth of impact when working in an autonomous environment, leading to poor prioritization and remediation.
Enter customer service. Another pillar of organizational strength, customer service is not only a necessity but has risen to become a strategic function for the business. They are the first responders of a business when digital services don’t work as expected or their customer expectations are not met. Customer service agents are highly trained, well-funded and armed with the latest technology so they can be the most effective first point of contact for customers. Customer service teams can learn something from developers in their organization and work within the same kind of ‘operations’ model that dictates philosophies and practices: in other words, CSOps or customer service operations.
Why CSOps?
Customer service is an incredibly powerful element of business operations, as it is tied to user experience. Consumers are not shy about announcing when they have experienced poor customer service, and the opposite is also true. End users also loudly trumpet the moments that they have received strong customer service; that goes a long way in creating loyalty, repeat business and retention. In a study from Bain & Company, it was noted that customers were four times more likely to buy from a competitor if they experienced a service-related problem rather than an issue based on price or product. Leading with customer service, like many e-commerce platforms have done, is a new standard of operating and is critical to meeting the expectations of customers. In a B2B setting, customer service is no different. Customers or users are drawn to organizations that can quickly remediate their concerns in a frictionless and efficient manner.
Customer service is now at the forefront of corporate initiatives and isn’t going anywhere. In a recent article from the Wall Street Journal, analysts and executives weighed in on CIO budgets and forecasted priorities amidst growing recession concerns. With cuts coming to some areas, customer service was not one of them. The overall sentiment is that “businesses see more value than ever in tech that will enhance the customer experience and give them an edge in an increasingly tight market.”
In today’s organizations, you cannot compete with DevOps if you do not have CSOps. They are symbiotic in their relationship to the bottom line—if products or applications do not meet the customer’s desired outcome, communication from customer service teams must connect with developers and convey what’s wrong or missing. Then, developers can work to provide the proper solutions. Customer service operations feed the DevOps pipeline to create products and services that address customer needs. In other words, you cannot succeed with DevOps if you don’t have CSOps.
Creating the DevOps and CSOps Relationship
The truth is that most CSOps and back office teams like SREs or ITOps don’t have a good relationship, nor do they have effective processes to communicate with each other. This disconnect not only creates problems with customers, but the lack of internal communication creates fault lines across teams and an inability to produce results efficiently. With the trajectory of technology organizations over the last decade ignited by the SaaS model, flexible work and the rise of applications, it is important to look at best practices within an organization and understand how teams can work better together.
While developer teams continue to iterate and produce code, customer service teams are constantly fielding customer feedback, concerns and complaints. Customer service agents are often the first to learn if something is broken and where—if organizations are not using their front-line workers as a tool in remediation and incident response, they are missing out on valuable information that can help them find a solution.
Beyond just fielding complaints and reports about product and service issues, customer service teams are able to aggregate and correlate reports, as well as gather additional information from reporting users. When something goes wrong, mobilizing the required rapid response can often be difficult. If there is a lack of communication between the customer service teams and the developer or engineering teams, a customer complaint about a product or service could take too long to make it through remediation. Creating a rapid connection from the customer service team to the developer team allows customer service agents to have visibility into the problem and the solution, offering feedback to the customer in real-time as the solution is being implemented. It is this ‘Ops’ culture on both the customer service and development teams that supports the organization from end to end.
The symbiotic relationship of DevOps and CSOps can benefit any organization, but adoption requires visibility and bi-directional communication. Development teams that are in close contact with their customer service teams benefit from having better, real-world user information that can then be incorporated into new features and improvements. When customer service teams have a clear path to escalate customer-reported issues—with the right context to the appropriate teams—time to resolution decreases by eliminating silos and breaking down the walls between CSOps and DevOps.
It is time that every organization integrates customer service teams as part of delivering on customer satisfaction instead of just reporting on it. The integration with engineering and back office teams creates a culture of service-first operations.
-
Why You Should Rip Up Your Org Chart and Reorganize Around Value Streams
To compete in the digital economy, teams must optimize both efficiency and effectiveness–it’s a tough ask. The traditional dominator hierarchy is a problem because organizations that use command and control create silos and handoffs that stifle innovation at the team level. Coordinated autonomy releases the creative spirit of individuals–but how can leaders assure alignment with their vision?
One way to eliminate the delays and waste associated with silos and foster autonomy is to arrange around value streams. A value stream is anything that has a customer or anything that delivers a product or a service. All value streams begin with ideas and end when a change is delivered. Or, even better, when our cherished customers realize the intended value outcome. The original intent of Agile was to provide “value” to our customers faster … but we hardly ever talk about what this value is or measure its existence.
Traditional organizational charts arrange resources according to expertise. This means that work is handed off to different teams as it progresses through the value stream. Value streams exist whether or not people call them that and whether or not they actively manage them. Every time a hand-off occurs, it creates an opportunity for delays or errors. It also means it’s hard for people to understand the end-to-end system. Instead, they do as they are told to optimize their piece of the puzzle without really understanding how their actions impact work that’s upstream or downstream.
More progressive ways of working, like Agile and DevOps, do promote cross-functional, autonomous teams, but they typically leave a big gap between planning and delivery and are not always smart at connecting feedback from customer experience back into planning. Value stream orientation promotes team diversity by recognizing the individual background (cross-functionally) into the value delivered to the customer.
Organizing around value streams is the foundation for nurturing the value stream-centric ways of thinking and working that drive both efficiency and effectiveness. Then, customer experience is improved and higher levels of organizational performance are achieved. This is because:
- Putting customer experience front and center brings higher retention and referral rates
- Elevating the concept of flow (see below) gives teams permission to prioritize removing waste
- Balancing innovation with improvements reduces technical debt, toil and unplanned work so employee experience is better, too
- Connecting planning to delivery to feedback in a continuous loop aligns teams and helps with decision-making
Flow is about how fast ideas can move through a value stream and be made available to the people they’re designed for. It’s about outputs and efficiency. Customer experience tells teams about the outcomes of their work and their effectiveness. Used together, these measures ensure the teams are doing the right things right and moving in the right direction.
In this model, leaders share their authority and empower their teams. They support teams’ self-discovery of improvements and make sure those local learnings go global. They help their teams understand how much time they spend innovating compared to reacting or improving. Then the leaders can help their teams reduce distracting and stressful unplanned work and instead invest in increasing capacity for innovation in the future.
Follow these steps to become a value stream-centric organization:
- Identify your value streams
- Identify who is involved in each step of your value stream
- Make your value stream visible by visually collaborating with your value stream team
- Use tooling to gather value stream data and generate insights for your team
- Meet regularly to inspect and adapt your value stream
All organizations are composed of value streams, regardless of whether they have been made visible, and most large organizations have complex networks of interconnecting value streams. These dependencies are not easily decoupled. They’re not easily visible, either. But once you’ve mapped them, you can start seeing where they connect and manage the dependencies. Ultimately the goal is to break these dependencies since they cause delays and risk, but it’s hard to do that when they can’t be seen.
Connecting the organizational vision with the value streams becomes much easier with this model. Organizational structures are flattened. Leaders understand which value streams they are accountable for and how they work. The time between having an idea and getting feedback on how it’s received is shorter and amplified. They can see the work being done, measure what impact it’s having and make better decisions about what to innovate or invest in next.
-
What GitHub’s 2FA Mandate Means for Devs Everywhere
Multifactor authentication (MFA) is becoming increasingly standard within software development organizations, with GitHub recently announcing that two-factor authentication (2FA) will be mandatory for all code contributors by the end of 2023.
This is a smart move. In recent years, bad actors have frequently initiated attacks by accessing source code through the use of stolen developer credentials. Once inside, they quickly penetrate organizations’ entire code repositories and elevate access to company-wide systems. This was the case with the 2021 SolarWinds compromise that ultimately jeopardized federal agencies, corporations and governmental institutions.
But for some developers with lofty goals and concentrated timelines, MFA is an extra step that has the potential to slow them down. As organizations continue to implement MFA and other foundational security controls, they may face pushback—but they shouldn’t be deterred. MFA is a critical attack barrier that improves individual protections and can contribute to tightened source code security.
If organizations approach implementation thoughtfully, they can lay the foundation for subsequent security controls—and build credibility among their users and within the larger development community.
The Risks of Lax Individual Developer Security
Organizations’ source code typically powers a host of upstream programs and products, so it’s valuable for bad actors to launch attacks by infiltrating their repositories. Attackers can also hide malware in source code that can be escalated to breach entire systems and third-party organizations. This risk is heightened by the fact that many companies rely solely on antivirus scans and other surface-level monitoring tools to flag source code vulnerabilities instead of combining them with dynamic testing and penetration testing.
Attackers typically gain access to source code through stolen developer credentials. While single authentication systems—i.e., usernames and passwords—are the easiest to compromise, MFA creates an additional barrier that’s more difficult for attackers to overcome. While this tool does create additional friction in the development process, it’s worth the extra step. Even for hobbyist or open source developers, compromised source code can cause larger repercussions that put other developers at risk. When it comes to cyberattacks, it’s best to assume any entity is a potential target.
Beyond individual security, MFA and other foundational controls are critical to safeguarding organizational reputations and adhering to legal privacy standards. By 2023, 65% of the global population will have personal data covered by privacy regulations. Security measures that protect authentication credentials will become even more important in the coming years, especially for organizations producing source code that powers consumer-facing products and programs. With an intentional approach to implementation, MFA can help mitigate potential source code attacks—and form the basis of comprehensive security programs.
Four Steps to Successful MFA Implementation
The benefits of MFA far outweigh the cost of user friction and developer pushback. Get the most out of this authentication technology by following these four steps to implementation.
- Start small and build credibility. Approach implementation gradually to help developers understand MFA’s value. Successful implementation will give your leadership and security teams credibility for future security initiatives, and it will improve processes for security integration into your larger software development life cycle.
- Examine your security program’s value chain. Implementing foundational security controls like MFA provides an opportunity to examine your security program’s entire value chain. Take time to look for potential vulnerabilities beyond your source code repository and flag areas within your development lifecycle that may create risk.
- Layer subsequent security controls. With MFA under your belt, build up your security program with other tools and controls—like dynamic and penetration testing, source code scanning, software composition analysis (SCA) and vulnerability management. The more security tools you layer, the harder you make it for attackers to succeed.
- Plan for continuous optimization. Attackers will devise more ways to thwart MFA as more organizations mandate it. For example, the popularity of using SMS as a second form of authentication declined when attackers developed methods to gain control of mobile phones. To keep up with evolving attacker behaviors, continuously optimize your MFA technology and subsequent security controls.
MFA Will Become the Industry Standard
As bad actors evolve attack methods and privacy regulations strengthen, MFA will undoubtedly become an industry standard for protecting source code repositories.
To stay ahead of the curve, focus on the concept of MFA throughout your network by ensuring it protects every system entry point. This extra step is an easy price to pay for improved source code protection—and ultimately safeguards every asset, product and program your code powers. Thoughtfully implementing MFA can lay the foundation for comprehensive source code protection.
-
Don’t Let Developer Toil Affect the Business Value of Your Apps
It is no surprise that the world today is driven by apps. Organizations all over the world are basing their business goals on their capacity to integrate new apps quickly, and they need to constantly reimagine how they interact with and communicate with their consumers and employees to fulfill changing expectations. To do this effectively, businesses need to operate at a high level of speed and agility. But in many cases, this isn’t happening. The phrase ‘move fast and break things’ could not be further from the truth when it comes to how enterprises approach innovation today. In fact, according to Forrester Consulting, 48% of executives say they haven’t changed their applications in a year or more. One reason for this may be that organizations are dealing with a high amount of developer toil and technical debt. Developer toil is the repetitive, predictable, constant stream of tasks that support the creation of software and how it’s run, but don’t actually affect the features in the software that people use. Or in other words “any activities that don’t directly create business value.”
Typically, developers spend too much time on processes that could be automated or even eliminated. Ironically, this toil builds up over time as their organizations prioritize shipping features rather than addressing these problems in their overall software process. While developers can change their software quickly at first, just like debt in real life, if this toil and tech debt are neglected, it takes over and consumes the organization’s ability to grow. This results in long, infrequent release cycles. A feature that seemed simple and once took just 15 minutes now takes weeks, even months, to get in front of users.
Conducting a Developer Toil Audit
Anything you can do to speed up your software release cycle will improve the quality, resilience and business value of your software. Continuously addressing developer toil provides a significant boost to your organization’s software capabilities. For one of VMware’s customers, addressing developer toil reduced the time taken to introduce a new feature significantly–from 400 hours to just 24 by doing a developer toil audit and addressing key factors contributing to this technical debt.
So how can organizations overcome developer toil? One way to do this is through a developer toil audit. This is a systematic process for finding, valuing and prioritizing fixing waste in your software process. First, the process finds wasted time and process debt in how you build and release software. Second, the process then helps you justify delaying working on features in favor of eliminating and automating your software creation process. The result is speeding up your software release cycle. The more frequently you can change and release software, even with just small changes, the more opportunities you must learn what works and put those features in front of customers, employees, and other users.
The process of conducting a Developer Toil audit includes:
1. Asking the right questions: A great way to uncover developer toil is to ask about people’s struggles, frustrations, and even boredom in the form of a tailored question set. We advise using questions that are specific to your organization in addition to general questions like “how long does it take to perform a full build?” For example, highly regulated organizations should ask about tasks involving compliance. Once the survey is completed, you can do some quick analysis to locate and prioritize developer toil.
2. Combine into usable metrics: You now have a single metric for each type of developer toil. And by combining all the questions you create a single metric to track overall developer toil. Extra points for creating dashboards with visualizations! As with all such aggregate metrics, these are more directional than exacting—their job is to point you to problems that should be solved.
3. Link these metrics to business value: A simple ranking of developer toil is better than nothing. However, before deciding which developer toil to address, we recommend linking each type of developer toil to the business value created by fixing the toil. Put briefly, the value you get will be related to the time saved, and, thus, the ability to ship more features in the future.
The less time developers spend on toil, the more time they can spend on tasks that directly benefit the business. Another way of looking at this is plain old productivity: Developers can now do more with the same amount of time.
Another important outcome is increased staff morale. The less time developers spend on boring, repetitive work—toil—the happier they’ll be. Stronger employee hiring ability and retention are (or should be) a strategic imperative for any organization that depends on software. Morale also increases software quality: happier people make better software.
Slow Down to Speed Up
With the survey results in hand, you should have a pretty good idea of which developer toil to fix. The last step is to do the usual product management prioritizing to weight fixing these items with other tasks you could do. These are strategic decisions that product managers need to make. To fix toil, you need to stop shipping features to free up time. You need to slow down the business. At the start, before product managers have been empowered to make these kinds of decisions, higher-level management should be involved to calibrate how much slow down you’re willing to put up with for future agility. The calibration you’re doing is this: If I fix one item of toil and ship just one feature (instead of two) this release cycle, then in the next release cycle I can ship four features. Humans are not great at that kind of bird-in-the-hand versus birds-in-the-bush thinking, so you’ll need to figure out what works in your organization.
Through our experience as consultants working with product teams in different industries, we’ve used Developer Toil Audits to focus and motivate product managers and developers to fix toil and speed up their release cycles and, thus, improve how their organizations build software. This directly improves the business by adding more capabilities and capacity to deliver more features, even in the short term. Each time you fix any technical debt, you gain more capacity and capabilities to create new features and improve your software. This is how you use a modern software culture to increase business agility. Or, put more simply: Less developer toil leads to better software, and better software leads to better business.
-
Leverage Empirical Data to Avoid DevOps Burnout
Burnout is possible in any industry. But it seems to be especially rampant in tech, a space bursting at the seams with inflated expectations and ever-accelerating trends. Nearly 60% of tech workers said they were currently feeling burned out at their workplace, found a study by TeamBlind.
Limiting burnout and supporting employee mental health is essential, not only for the workers’ sake but for the business as well. During this period, which commentators call the Great Resignation, if employees don’t feel respected, they will likely seek new opportunities elsewhere. Ensuring the needs of engineers are met is a vital target amid an ongoing cloud skills drought, a talent gap to which the DevOps space is certainly not immune.
I recently met with Eric Mader, senior solutions architect, AHEAD, to understand the root causes of burnout as it relates to DevOps. According to Mader, burnout can be caused by many factors, including work-life imbalances and conflicting egos in the workplace. But above else, burnout is most often caused by management setting unrealistic targets that aren’t based on empirical team performance data.
What Factors Contribute to DevOps Burnout?
So, what can cause burnout in DevOps? First, the workplace culture can significantly contribute to burnout. In most remote working models, explained Mader, it isn’t easy to ascertain the correct work-life balance. For example, there may be an unwritten expectation to immediately respond to Slack messages or quickly get back to emails after-hours or over the weekends.
An “always on” attitude can lead to unhealthy work habits. And while this isn’t necessarily unique to DevOps, it is sometimes unavoidable due to the high connectivity required to perform the job. Another potential contributor to burnout is the team dynamic, noted Mader. “Every team seems to be unique in its own way,” he said.
Although Agile and DevOps involve very technical work, at times, ego can get in the way of a healthy team dynamic. For example, in many tech teams, there is often a “guru,” such as a senior developer or tech lead, that everyone goes to for advice, Mader said. This person might end up doing most of the work (both technical and interpersonal) and usually feels the most burned out. Or, other wanna-be leaders might take on an unnecessary command-and-control role.
Personality differences with coworkers could contribute to burnout, but other factors like dealing with toil or managing legacy technical debt can be just as tiresome.
‘Move Fast And Break Things’
According to Mader, perhaps the biggest contributor to burnout is the insistence from executives that tech workers consistently outperform their previous output—especially when past team performance data does not support these goals. Setting achievable goals is great, but when deliverables are out of scope, it’s a recipe for burnout.
Part of the pressure to increase velocity is the constant illusion that “faster and more is better,” said Mader. Value is often attributed to the number of items achieved in a sprint instead of the quality of each point. For example, a team might deliver 200 story points of work in a sprint, but only 30 of these points carry a significant amount of value. Or, a leader might see a team accomplish 60 points of work in a sprint and then encourage 200 in the next one.
These sorts of high expectations suffer from basic capacity problems. And the added stress quickly realizes itself as a burden, contributing to burnout, explained Mader. He encourages technical leads not to force more work but instead, “focus on the highest-value work based on the empirical data you have.”
Five Tips To Avoid Burnout
Mader shared some tips on how to avoid burnout in DevOps teams. Here are some simple yet effective ideas to maintain happier engineers and a more productive, healthier team relationship.
Snuff out the always-on attitude. Organizations need to walk the talk when it comes to mental health awareness. Therefore, ensure employees know that their life boundaries are respected. Encourage this by honoring time off for holidays, paternity/maternity leave or personal vacation. Discourage emailing on the weekends and encourage turning on ‘Do Not Disturb’ alerts on team chats and work email accounts.
Prioritize what gets automated. There’s a misnomer that any automation is good automation. In fact, over-prescription of software development automation can lead to some negative consequences. “I’ve seen many DevOps teams that focus so much time on automating low-impact business processes,” said Mader. Instead, prioritize automation projects that affect more people and those that would be continually used.
Try out the ‘Fist of Five.’ Ensuring everyone’s voice is heard helps avoid a toxic working environment. One tactic to quickly gauge reaction in team meetings is to use what Mader calls The Fist of Five: One if an idea is totally off base and five if an idea makes total sense. This simple practice could ensure everyone gets to contribute input on new projects.
Take the quietest person in the room aside. Only the squeaky wheel gets the oil, Mader noted. With that in mind, take aside the quietest person post-meetings to gather their input—you might be surprised. Junior folks who may be less apt to speak their minds typically have a less-institutionalized mindset and could suggest more innovative ideas.
Use empirical data to inform future goalposts. Finally, Mader encouraged DevOps teams to “use empirical evidence to back up on where you’re going in the future.” Specifically, in the DevOps sphere, empirical data takes the form of metrics like mean-time-to-recovery, or SLAs, SLIs and SLOs.
Remember that even if a team did accomplish a certain number of goals in a sprint, it doesn’t mean they can produce the exact same results the next time around—they could have been pushed to their limit to produce past results. Thus, managers should aim to move the needle on metrics progressively, not in giant strides that upset morale.
Take an Opinionated Approach to DevOps
Nowadays, DevOps teams are spoiled for choice—there are countless tools on the market and an endless stream of best practices and methods. Even within a single company, the DevOps toolset may vary considerably from team to team. To help make sense of it all, Mader encouraged DevOps leads to simplify the transformation journey by setting guardrails and adopting a more opinionated approach around what tools and processes you take on.
-
Recession! DevOps Hiring Freeze | Data Centers Suck (Power) | Intel to ‘be’ Wi-Fi 7
Welcome to
The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Engineer jobs are being cut, cloud infrastructure is using too much energy, and Intel’s 802.11be silicon is alive. (more…)