devopsy.kinsta.cloud

Tag: developer

How to Supercharge Your Engineering Teams

With the economy still bleak, data and engineering teams that were already struggling to keep pace with business needs are now facing a potential hiring freeze or worse in 2023. That’s why there’s never been a more important time to maximize the productivity and impact of your teams, but the question is how to identify the big changes that move the needle in performance. Incremental adjustments that save a sliver of time here and there are not hard to come by, but what teams need are the breakthroughs that yield giant leaps in productivity and efficiency. Rather than a 10% improvement, it’s the 10x improvements that drive lasting gains.

As an engineer and CEO of a software company, I’m constantly looking for ways to boost the productivity of our teams. Our guiding principle is simple: Maximize the impact per hour invested for every individual. To do that, we must ensure that individuals maximize the time spent flexing their superpowers, minimize busy work and reduce dependencies on others that delay work or tie them up in meetings. While counterintuitive at first, we consistently find that collections of well-aligned, independently executing individuals produce a far greater impact than a large team swarming on a project. Put another way; the individual impact goes up when the number of people you depend on to deliver the result goes down.

I’ve developed an approach over the years that has allowed me to identify several opportunities for step-change advancements in productivity and efficiency, and I’d like to share it here, along with two concrete examples. I will also share an exercise we run with our teams to help identify and execute these opportunities.

First, here are my two main criteria for identifying the projects and tasks where we can effect changes that will have an outsized impact on the company:

● Big resource-consumers: Work that’s essential to the business but consumes a large amount of engineering and other resources is a top candidate for optimization.
● Big impact, easy win: Areas that will have a significant impact on company performance if we can do them faster and with higher frequency are also great candidates—even if they’re not consuming a ton of resources.

Here are two examples that illustrate each of those criteria and where we were able to see massive gains in productivity, quality of service, or both.

Tackling Big Resource-Consumers

At Ascend.io, we provide a platform for building and automating data pipelines, which requires us to build connectors into a huge variety of data sources. At one time, building a new connector required multiple engineers to spend two to three weeks writing more than 5,000 lines of code that spanned four microservices in three different languages. As we added more customers and needed to support more data sources, this quickly became unsustainable.

When we looked at this process, we saw two issues: Engineers were writing custom code for each connector in a way that would be better served by a common framework. Moreover, the process required engineers from multiple teams and disciplines to work together and coordinate their priorities. As soon as you get dependencies that require engineers to coordinate with others and wait on the outcomes of other people’s work, you’re wasting time.

We addressed this by designing a new development framework that exploits commonalities across the connectors we build. This now requires far less unique code and removes the need for multiple engineers to work together. In most cases, one person can build a new connector in days with 100 lines of code. This made us far more nimble and freed up engineering time to do other valuable work.

Achieving High Impact

Like most companies building software, we need to convey the capabilities of our product to new and existing users to ensure they receive maximum benefit from their use of Ascend. With new features and enhancements being released on a weekly basis, there is a lot to keep up with. This is high-impact work, however, and if we can do it in a vastly better way, it drives an outsized impact for Ascend and our customers.

For us, building these automated product tours typically required a product marketer or product manager to design the customer journey, and one or more engineers on a sustained basis to build the actions into the product. This again creates dependencies that slow progress and require significant manual coding.

The breakthrough for us was a tool called Intercom, which allows you to build sophisticated product tours without needing to write code. Our product team can now build more interesting and complex product tours for customers, and our engineers can stay focused on building new features instead of the code needed to showcase them. What’s more, they are able to design, test and deploy these tours independent of our product and code deployment cycle, further freeing them to iterate as fast as possible.

A Thought Experiment

So how do you actually find these opportunities for 10x improvement? I run the following thought experiment with our teams several times a year to uncover these opportunities:

Imagine your workload scales 10x overnight and you absolutely must meet this increased need, but without hiring more engineers or working more hours. How do you do it?

It’s amazing what this exercise can uncover—but there are ground rules and it can take time to see results. Coming up with these big ideas is a skill like any other, and most people are not great at it the first time they try it. Your first iteration may produce no options to pursue—some may have high-impact ideas but with even higher investment requirements, while others may reject the premise outright. That’s normal. So, don’t insist they sit in a room until they dream up that 10x breakthrough, because it will be demoralizing. Instead, plant the seeds by establishing the above framework and then let the ideas start to grow.

More likely, a few days later, they’ll be out for a hike or chatting with an old co-worker and they’ll suddenly have the germ of an idea. That may still not be the winning solution, but someone else will build on it, and then someone else and, finally, you iterate your way to a breakthrough. Pulling the team back together and embracing the “yes, and” approach can be both energizing and highly productive.

Brainstorming like this is fast and free, and you only need one great idea to have a material impact. We run this exercise a few times a year and have come up with several big ideas that were game-changers for the company. Besides the examples above, we also moved key parts of our software development from Scala and Go to Python a few years ago, which is a much more efficient language (as measured by developer time) and allows our engineers to build things much faster.

Software engineering is a complex beast that often spans many teams working on multiple processes in parallel. But that complexity is also your opportunity: Every complex system has room for optimization, especially at companies that moved fast in the beginning to build a product and find market fit. Now is the time to lift your gaze, look around and ask where significant gains can be made. They almost certainly exist, and a 10x improvement could be what allows you to not only survive the turbulent year ahead but emerge with a thriving team and business on the other side.

March 27, 2023
Optimizing the Developer Experience in 2023

2022 was a banner year for innovation, but not only in the traditional “churning out new features” way. Many organizations started thinking about their processes and how to amplify them to maximize efficiency and employee retention. And when they started digging, companies realized that the developer experience was a major pain point that could be rectified–and a ripple effect of positive gains would result.

Companies started looking more closely at how the tools and processes developers use support and enhance their work. Great tools and thoughtful processes boost developer happiness and satisfaction. And when those levels rise, guess what else increased? Productivity, efficiency and employee retention rates.

We saw this trend of developer support take hold last year, but with more work to be done across many companies, we’ll see it continue in 2023 and beyond, impacting just about every facet of tech in new ways. Many companies have realized what does work by trying what doesn’t work and are making big changes from there.

Shifting Results

A major player in the changing developer experience has been the shift left approach that’s taken hold in recent years. About 80% of organizations have said they are adopting this idea of moving more of the responsibility for security and compliance to the front of the pipeline to avoid a compliance scramble at the point of delivery. The result, however, is a compliance process that ends up being more complex and less efficient, putting additional burdens on developers.

Involving more people creates more opportunities for security breaches and slows down the entire software delivery life cycle. It also shifts security responsibilities to developers and requires them to do significantly more than is listed in their job description. In fact, 60% of those shifting left said their development team is bearing the burden of the change. Rather than focusing on writing code, developers have been called on to understand regulatory policies, build regulatory controls into their applications or pipelines, inspect massive log files generated by more and more diagnostic tools for security info and spend more time in meetings to discuss audits. That’s not a recipe for fulfilled developers; it’s the path to a loss of innovation.

Stop Paying the Compliance Tax

When developers–or anyone on your team–are tied up in compliance tasks, they aren’t spending that time in profit-earning efforts. This “compliance tax” costs companies an estimated 15% each year–that’s 15% of their time spent managing compliance instead of innovating or providing services that add value. That’s not to say that compliance isn’t important–it’s essential, in fact–but limiting the burden of compliance improves the overall experience of developers.

When developers aren’t innovating, they’re not happy. They want to be writing great code, not translating regulatory governance policies. In today’s culture of quiet quitting, companies simply can’t afford to turn over compliance tasks to developers. It’s a waste of their talent and a surefire way to lose great developers and the chance to attract new ones.

Shift Left, but Do it Right

The good news is the compliance tax is largely avoidable. Instead of shifting responsibilities left, shift compliance everywhere. No one person or group, including developers, should be in charge of these tasks–that approach just isn’t comprehensive enough to ensure security and compliance throughout the software delivery life cycle.

Instead, security and compliance should be moved away from a point-in-time part of the cycle to a continuous process built into every step of the life cycle. CI/CD reduces the burden on any one person or group and is proven to be more efficient at catching anything that is non-compliant. When the tools are doing the heavy lifting, the people aren’t making as many mistakes and non-compliant code isn’t slipping through.

Baking in compliance allows developers to spend more time doing what they do best: Creating amazing features. This, more than anything, will boost the developer experience. Developers want to create and, when they experience the right environment and the right resources, innovation tends to happen.

Beyond the Developer Experience

Focusing on the developer experience pays off in more ways than just creating happy employees (though that alone is reason enough to make a change). Innovative developers are the backbone of leading tech companies. When they’re churning out new features that are getting public recognition and are touting a culture with a positive developer experience, companies attract more talented developers and the innovation loop continues.

So, if you’re looking for more innovation in 2023, start where you’ve always started: Your developers. Set them up for success by automating what can be automated and baking in the mundane – yet essential – tasks to truly optimize the developer experience. Then sit back and watch what incredible capabilities your developers can generate.

January 19, 2023
Why Can’t We Be Friends? Improving Developer-Tester Alignment

Sometimes coding can be such an art form that it deserves to be a masterpiece in a museum. Developers are proud of their code—at least until the so-called art critics (aka testers) step in with their critiques. And then the rapport can swiftly become as messy as a painter’s palette. You have probably seen this scenario play out time and again at your organization. In the mad dash of a software release, the relationship between developers and testers can quickly become stressful and strained. And while your teams know about helpful tools that can streamline collaboration, you may not be aware of other steps you can take that go beyond the screen to ensure an even better developer-tester relationship.

Full disclosure: At my organization, we’re in the software testing business and we’re not playing favorites when it comes to testers versus developers. We’ve just learned a thing or two about what makes a happy collaboration for all the parties involved. And we’ve seen that organizations that improved their developer-tester alignment not only accelerate a smoother release cycle but also leave no hard feelings on the table for the next time.

Leave the Keyboard, Take the Cannoli

Developers and testers will find that if they move their interactions away from the keyboard, they’ll discover that sweet spot of improved communication and transparency. Whether at your daily standup, over a cup of coffee or by simply picking up the phone, spending a few minutes chatting can address miscommunications before they become major problems that can delay your release.

If something seems off, bring it up as early as possible. Take, for example, the classic case of missed requirements. One of our customers recently shared a story with us that really hit home. At the top of the sprint, a particular user story included only seven acceptance criteria–which caught the attention of one of the developers. He pointed it out in a conversation with the testers and business analysts on the project. Together, the group identified four additional acceptance criteria missed during the initial scoping.

Without that conversation, the team would have assumed those seven acceptance criteria were the only ones that tests needed to be written against. A number of defects would have slipped into production, and it wouldn’t have been the software developers’ or the testers’ fault. Instead, the testing team realized significant gaps in test coverage and their test plan grew from seven to 40 test cases.

The takeaway? Make sure that developers, testers and business owners are clear on how business requirements align with what software developers plan to build, as well as how testers plan to test against them. Have this conversation before the sprint begins—and don’t forget the cannoli.

You Complete Me: In-App Test Management

“I love documenting test plans in spreadsheets!” said no one, ever. All that manual work just takes time, introduces errors and leads to frustrations among developers and testers who have to go digging for the information they need. Adding a Jira app, for example, for test management can be the key to completing the alignment between teams.

In-app test management gives both teams real-time data for decision-making and collaboration with built-in reporting on testing. That means that software testers can fully plan, design, track, and analyze testing activity within the same workflows that developers and testers are already using. For developers, there’s no need to ask testers for status updates or results. And testers can quickly see how tests align to requirements, which helps them brainstorm additional tests and provides developers better context and information to develop around.

The cherry on top? The entire team can maintain true traceability between requirements and test coverage, which both minimizes finger-pointing and ensures that everyone has a clear picture of what’s required before the release is ready to ship.

There’s an Awful Lot you can Tell About a Person by Their Software

As Mama always said, shoes tell where someone is going and where they’ve been (mic drop, Forrest). By talking the talk and walking the walk in a counterpart’s shoes, developers and testers can better understand each other’s worlds. In-app test management can give developers a window into the tester’s thought process, because it shows them not only test status but also how a tester arrived at a particular defect. After gaining this familiarity, developers may be more open to inviting testers to sit in on traditionally dev-focused activities.

Take unit testing or code reviews, for example. Most QA teams tend to steer clear. But by joining in, testers can ask questions and grasp which pieces will be validated at the unit, integration, and functional levels. A bonus is that testers get an inside look at the codebase, helping them develop more technical knowledge, as well as collaborate with developers on how the unit tests they plan to run–gasp!–fit into the overall picture of quality and release readiness.

Remember, the true purpose of QA is to provide information that verifies that the application works well enough to be put in front of customers. By understanding the testing that’s going on at the unit level, QA can be more efficient by focusing on what’s not being covered in unit tests.

The result? A more cohesive, effective test plan and a release that the whole team–developers, testers and business owners–can be proud of. In other words, a release masterpiece.

January 18, 2023
The Curious Connection Between Cloud Repatriation and SRE Ops

I have a fondness for philosophy. I’m about three classes short of a degree, and every few years I tell myself one day I’ll finish it. Thus, I am very familiar with what is known in statistics—and logic—as a post hoc fallacy, from which we get the saying “correlation is not causation.” This is the logical error of assuming that if event Y followed event X, event Y must have been caused by event X. The most famous callout of this fallacy came from Bobby Henderson, who illustrated the absurdity of assuming causation from correlation with his chart demonstrating that global warming was caused by the declining number of pirates in the world. Yeah, that doesn’t make sense, but neither do a lot of charts from which people draw causality. Just because two data points are mapped against each other does not mean one caused the other. In many cases, it doesn’t even make logical sense to correlate the two. After all, pirates and global warming? No one actually takes that seriously. But it’s an important point to make as we dive into the question of the relationship between site reliability engineering (SRE) operations and cloud repatriation.

To be clear, I’m not suggesting that adopting SRE practices causes cloud repatriation. But I am suggesting that there is a close and meaningful relationship between the two. The fact that Google—a cloud provider—created SRE as a practice is not a mistake. The model, mindset and skillsets associated with SRE are integral to successfully operating cloud infrastructure and services.

Cloud Repatriation Is Real

Public cloud repatriation itself is a somewhat taboo topic in certain circles. Consider the controversy raised by Andreessen Horowitz when it published “The Cost of Cloud, a Trillion Dollar Paradox” and suggested companies were repatriating from the cloud and realizing significant cost savings as a result. Some would have you believe it’s not happening, but there’s enough data and anecdotal evidence to indicate that yes, it is.

F5’s 2021 state of application strategy report asked the market about public cloud repatriation. A mere 13% had repatriated apps and another 14% were planning to. One year later, that combined total rose 40 percentage points to 37% and 30%, respectively. This is not an anomaly, as there are multiple credible analyst firms reporting similar results. Interestingly, the rate of repatriation is not globally universal. APCJ and LATAM are both far less likely to repatriate than EMEA and NA.

I maintain that companies are repatriating apps from the public cloud—and the question isn’t ‘Are they?’ but rather ‘How many workloads are they pulling out—and where are they going?’ That’s a question for the State of Application Strategy 2023 research.

For now, let’s consider a possible enabler of repatriation—SRE operations. Because even if the increasing cost of cloud is a driver of the desire to repatriate if you don’t have the skills to operate as efficiently elsewhere—and thus benefit from lower cost—then why would you repatriate?

And we posit that it is SRE operational practices and skills that enable companies to repatriate and maintain the efficiency and cost savings needed to justify the decision—whether they’re moving those workloads to another public cloud, on-premises or to the edge.

Digging Into the Data

On the surface, there is a strong correlation between the adoption and application of SRE practices with cloud repatriation that seems to indicate that organizations with the ability to operate in a cloud-like manner, i.e., they’ve adopted SRE practices, effectively pick up their toys (apps) and go home (on-premises or elsewhere) because they can.

Cut another way, only 4% of organizations that have not adopted SRE practices have repatriated apps from the public cloud. A whopping 73% of those who have adopted SRE practices have also repatriated apps.

Of course, adopting practices does not necessarily mean applying practices. So, we looked at how organizations are actually operating applications, systems, and infrastructure. Specifically, we looked at the percentage of their operations that use SRE practices. Perhaps unsurprisingly, that generated similar results.

Of those who operate 0% of their apps, systems, and infrastructure using SRE practices, 81% are not repatriating. Conversely, of those who use SRE practices for 76%–99% of apps, systems and infrastructure operations, 54% have repatriated. The point at which repatriation appears to begin picking up steam is when organizations surpass using SRE practices to operate more than one-quarter (25%) of their apps, systems and infrastructure.

Remember I noted that APCJ and LATAM were far less likely to repatriate? They’re also far less likely to be leveraging SRE practices to operate their apps, systems, and infrastructure. In fact, over one-quarter (26%) of respondents in LATAM and APCJ (29%) were operating ZERO percent of apps, systems and infrastructure using SRE practices. In EMEA? That’s only 5%. And in NA, even lower at 2%.

Meaningful Relationship or Curious Coincidence?

There appears to be an inarguable correlation between organizations embracing SRE as an operational practice and public cloud repatriation rates. But is it a meaningful relationship or merely a curious coincidence?

I’m going to argue that it’s a meaningful relationship.

The practices and skillsets associated with SRE are wholly suited to operating a cloudy environment—at scale. As I said before, it’s no mistake that it was Google who created SRE and has literally written the book on it. The value of cloud is in its operational model, which can dramatically lower the cost per transaction, whether measured by HTTP exchanges or customer sessions. That enables a cost-efficient scale of applications and digital services.

The use of automation and practices that tend to focus on meaningful incidents rather than non-disruptive occurrences provides the cost-efficient scale of the people (and thus their expertise) who are tasked with maintaining a high level of availability and performance.

The adoption and use of SRE practices enable organizations to efficiently scale operations whether in the public cloud or on-premises or at the edge. And what the data tells us is that organizations appear to be using the capability to do just that.

September 13, 2022
Don’t Let Developer Toil Affect the Business Value of Your Apps

It is no surprise that the world today is driven by apps. Organizations all over the world are basing their business goals on their capacity to integrate new apps quickly, and they need to constantly reimagine how they interact with and communicate with their consumers and employees to fulfill changing expectations. To do this effectively, businesses need to operate at a high level of speed and agility. But in many cases, this isn’t happening. The phrase ‘move fast and break things’ could not be further from the truth when it comes to how enterprises approach innovation today. In fact, according to Forrester Consulting, 48% of executives say they haven’t changed their applications in a year or more. One reason for this may be that organizations are dealing with a high amount of developer toil and technical debt. Developer toil is the repetitive, predictable, constant stream of tasks that support the creation of software and how it’s run, but don’t actually affect the features in the software that people use. Or in other words “any activities that don’t directly create business value.”

Typically, developers spend too much time on processes that could be automated or even eliminated. Ironically, this toil builds up over time as their organizations prioritize shipping features rather than addressing these problems in their overall software process. While developers can change their software quickly at first, just like debt in real life, if this toil and tech debt are neglected, it takes over and consumes the organization’s ability to grow. This results in long, infrequent release cycles. A feature that seemed simple and once took just 15 minutes now takes weeks, even months, to get in front of users.

Conducting a Developer Toil Audit

Anything you can do to speed up your software release cycle will improve the quality, resilience and business value of your software. Continuously addressing developer toil provides a significant boost to your organization’s software capabilities. For one of VMware’s customers, addressing developer toil reduced the time taken to introduce a new feature significantly–from 400 hours to just 24 by doing a developer toil audit and addressing key factors contributing to this technical debt.

So how can organizations overcome developer toil? One way to do this is through a developer toil audit. This is a systematic process for finding, valuing and prioritizing fixing waste in your software process. First, the process finds wasted time and process debt in how you build and release software. Second, the process then helps you justify delaying working on features in favor of eliminating and automating your software creation process. The result is speeding up your software release cycle. The more frequently you can change and release software, even with just small changes, the more opportunities you must learn what works and put those features in front of customers, employees, and other users.

The process of conducting a Developer Toil audit includes:

1. Asking the right questions: A great way to uncover developer toil is to ask about people’s struggles, frustrations, and even boredom in the form of a tailored question set. We advise using questions that are specific to your organization in addition to general questions like “how long does it take to perform a full build?” For example, highly regulated organizations should ask about tasks involving compliance. Once the survey is completed, you can do some quick analysis to locate and prioritize developer toil.

2. Combine into usable metrics: You now have a single metric for each type of developer toil. And by combining all the questions you create a single metric to track overall developer toil. Extra points for creating dashboards with visualizations! As with all such aggregate metrics, these are more directional than exacting—their job is to point you to problems that should be solved.

3. Link these metrics to business value: A simple ranking of developer toil is better than nothing. However, before deciding which developer toil to address, we recommend linking each type of developer toil to the business value created by fixing the toil. Put briefly, the value you get will be related to the time saved, and, thus, the ability to ship more features in the future.

The less time developers spend on toil, the more time they can spend on tasks that directly benefit the business. Another way of looking at this is plain old productivity: Developers can now do more with the same amount of time.

Another important outcome is increased staff morale. The less time developers spend on boring, repetitive work—toil—the happier they’ll be. Stronger employee hiring ability and retention are (or should be) a strategic imperative for any organization that depends on software. Morale also increases software quality: happier people make better software.

Slow Down to Speed Up

With the survey results in hand, you should have a pretty good idea of which developer toil to fix. The last step is to do the usual product management prioritizing to weight fixing these items with other tasks you could do. These are strategic decisions that product managers need to make. To fix toil, you need to stop shipping features to free up time. You need to slow down the business. At the start, before product managers have been empowered to make these kinds of decisions, higher-level management should be involved to calibrate how much slow down you’re willing to put up with for future agility. The calibration you’re doing is this: If I fix one item of toil and ship just one feature (instead of two) this release cycle, then in the next release cycle I can ship four features. Humans are not great at that kind of bird-in-the-hand versus birds-in-the-bush thinking, so you’ll need to figure out what works in your organization.

Through our experience as consultants working with product teams in different industries, we’ve used Developer Toil Audits to focus and motivate product managers and developers to fix toil and speed up their release cycles and, thus, improve how their organizations build software. This directly improves the business by adding more capabilities and capacity to deliver more features, even in the short term. Each time you fix any technical debt, you gain more capacity and capabilities to create new features and improve your software. This is how you use a modern software culture to increase business agility. Or, put more simply: Less developer toil leads to better software, and better software leads to better business.

August 8, 2022
Technical Debt: When It Works and When It Doesn’t
Every software team has probably faced technical debt—the accumulation of little development deficiencies that will require rework down the line. Just like financial debt, they accrue interest as they build up, in the form of difficult software modifications or challenges in adding new features. Time is the usual culprit; your development team has one timeframe for creation but your leadership team wants to bring the software to market faster. And when it comes to choosing between building a clean and polished piece of software or hitting that delivery date, you usually need to bow to pressures from on high. It’s at that point that your collaboration might go off track, with little pieces of incomplete code jumping into the mix.

Without careful monitoring, technical debt will eventually slow down your development and deployment processes, degrade your product quality and limit your ability to innovate in a changing market. But is technical debt unilaterally bad? Not always. Sometimes it’s in service to a greater strategy. Let’s look at the differences.

When is Debt Part of a Good Strategy?

Pankaj Gupta, senior director, product marketing at Citrix, said when technical debt impedes an organization’s ability to function, that’s a clear sign they need to wipe out the debt and start with a clean slate.

“But there are a few scenarios under which it makes sense to let debt accumulate,” he said. “For example, if the cost of addressing a technical debt is significantly higher now than in the future; if the debt is not impacting your immediate and short-term business needs or if you have an emergency release like a major security vulnerability fix.”

Daniel Viveiros, chief technology officer at CI&T, recommended that teams keep the big picture in mind when making the right tradeoffs. “For me, it’s about putting the business first and understanding that constantly activating value is imperative for success,” he said. “Well-managed technical debt can be an effective tool to shorten the lead time. Deferring refactorings, modernization efforts and other common debts in order to prioritize deployments is a constant tradeoff the engineering teams deal with on a daily basis.”

What are the Signs of Too Much Technical Debt?

Gupta provided some warning signs for technical debt overload. “You know you’re accumulating too much technical debt when the cost and/or time to fix the technical debt is higher than starting fresh,” he said. “And the time taken for each release and deployment is consistently increasing.”

A less obvious result: a DevOps exodus. The development team will be the first to feel the impact of too much technical debt, such as having a harder time completing tasks and working long hours to fix frequent breakages. “Employees are leaving the company out of frustration from working on legacy systems, spending most of their time fixing ad-hoc issues, routinely missing and being on-call all the time,” Gupta said. What debt reaches that level, resignations begin to mount.

Viveiros pointed out that compiled debt will impact system resilience: “Bugs are originated more often and the change failure rate goes up. Speed is perceived as risky instead of something beneficial to the digital delivery process.”

The ultimate sign of too much debt, he said, is poor business performance and lost revenue. “If downtime equals lost opportunities, the technical debt will become a financial debt,” Viveiros said. “Too much technical debt may impact the perceived quality of the product. As a result, the product and brand’s image will suffer.”

What’s the Best Way to Manage Debt?

Failure to deal with technical debt means kicking it down the road to another developer to tackle—and no one wants to be that person. So what’s the best way to take control and pay it off? To start, approach it as a collaborative effort and implement these four practices.
1. Identify the types of debt. Not all technical debt is created equal. Distinguish between the debt you accept in the moment as something to fix later on and the inadvertent debt you discover—and categorize it into code debt, design debt, build debt and other forms to identify origin patterns.
2. Analyze and automate. Analyze the origins of your debt and look for ways to tighten workflows or automate certain tests and processes. By reducing common errors and hidden bugs, you can stop them from snowballing into technical debt.
3. Develop new policies and standards. These should clarify when debt makes sense and when it causes unacceptable damage. For instance, an immediate release of a security patch would be deemed acceptable while errors that will eventually cause considerable downtime would not.
4. Communicate the cost to IT and business leaders. When another hyper-speed deadline comes your way, make sure your leadership understands the risks to product quality and developer retention. With the price tag made clear, they’re more likely to change delivery dates or provide funding for additional developers.
Like most out-of-control spending, harmful technical debt begins with poor decisions and a certain level of denial. By approaching decisions with greater clarity and foresight on both the Dev and Ops sides of the house, teams can give themselves more breathing room and the space to produce exceptional software.
June 9, 2022
Retaining Talent as the Developer Role Evolves
Traditional development roles are rapidly evolving due to increased remote work and the Great Resignation. Company leadership must also adapt to retain that talent and ensure the success of their dev teams.

The Bureau of Labor Statistics reported that a record 4.5 million Americans voluntarily left their jobs in November 2021—and in December 2021, CompTIA’s Tech Jobs Report showed that open job postings for tech and IT roles increased 53% year-over-year. With the need for tech and IT talent persisting in 2022, a combination of widespread burnout and technical skills gaps have made it even more difficult to find and retain top IT talent.

But that’s not the only challenge. The impact of remote work has triggered fundamental changes in the developer role, further compounded by tech companies’ increased migration to cloud infrastructure. And within organizations, a lack of foundational knowledge and formal company training is creating discrepancies in how development methodologies are being applied.

With developer expectations and skills evolving, Agile development methodology is more relevant than ever thanks to its adaptive nature and flexible, iterative processes. Company leadership must internalize this fact and put formal practices in place to standardize methodologies within their organizations. By doing so, leaders can arm developers with the tools they need to embrace rapidly changing development standards and methods—and retain top IT talent in a high-turnover marketplace.

Changing Developer Roles Bring new Possibilities and Challenges

The pandemic profoundly accelerated the need for remote-based work structures, and with employees physically out of the office, many organizations began placing a much higher priority on their migrations to cloud-based infrastructure—both physical and organizational. This push placed a huge burden on development teams to move organizational and project data to Cloud infrastructure at faster-than-normal rates. For IT professionals, the pandemic-triggered disaster response has accelerated a permanent migration to the cloud as a way to safeguard access to digital infrastructure and processes. For developers specifically, this migration triggered many teams to embrace infrastructure-as-code (IaC).

IaC is a core DevOps practice that allows for infrastructure (like networks, cloud services and virtual machines) to be constructed and managed through code instead of manual processes. This means that developers are increasingly writing the actual infrastructure their software runs on, which has game-changing advantages for automation, traceability and reliability—and impacts virtually all DevOps practices. With no one in the office to perform administrative and operational IT work, IaC is particularly useful. But since developers must have the necessary technical skills to write this infrastructure in addition to their traditional code, IaC has created a gray area where developers are also becoming IT operators and systems designers.

The shift to IaC has evolved standards for developers, and it requires a higher level of flexibility within development teams and companies as a whole. Adopting IaC goes hand-in-hand with strengthening and refining Agile processes because creating standardized Agile practices enables not just greater agility, but also more accurate visibility—both necessary capabilities as staffing models evolve and the roles of developers and IT operators become more intertwined. But company leaders can position their development teams to succeed in this changing tech landscape and reap the benefits of increased collaboration and efficiency that IaC and cloud-based technologies provide.

3 Key Ways Leadership Can Support Development Teams

The role of development teams is changing fast, and leaders must keep pace to retain IT talent and safeguard the health of their organizations. With that in mind, here are three ways you can support your development teams as they navigate changing development standards.
1. Standardize best practices. Even within a single organization or a single development methodology, go-to practices can vary among IT and development professionals. To successfully execute Agile, teams must have standardized best practices to ensure consistency in foundational knowledge. Agile’s iterative processes can lead to useful shorthand and improvisation, but only if developers are working from the same set of methodology standards and solid foundational knowledge.
2. Increase commitments to formalized training—especially for new hires. To fully benefit from Agile’s capabilities, all employees must have foundational knowledge of its processes—especially new hires. Provide formal education and training on industry-accepted Agile standards to eliminate skills gaps and cultivate institutional knowledge within your organization. Institutional knowledge ensures that quality standards are well-defined and achievable, ultimately strengthening the collaborative capabilities of your development teams.
3. Prioritize measures to decrease burnout. The pandemic shed light on burnout across industries, and the IT/tech world was no exception. It’s up to leadership to implement tangible measures aimed at reducing employee burnout—like managing realistic workloads, allowing for flexible working schedules and implementing policies that encourage work-life balance. Consistent standards lead to meaningful reporting, which ensures accurate measurement of team performance and workload.
Preparing for Future Evolutions

Remote work isn’t going anywhere, and companies that fail to embrace remote-based technologies risk employee flight and obsolescence. As development methodologies evolve to accommodate pandemic-triggered challenges and IT hiring gaps, the developer role will continue to change. In five to 10 years, it will look much different than how it looks today and companies must be willing to embrace these changes to ensure the success of their IT and development teams.

Agile development methodology was designed to adapt to rapid technological innovation and organizational change. The shift to remote work and the IaC revolution is the future—and it’s here. These forces are driving hybridized development methodologies that serve the new standards they impose. Embracing these changes is an organizational imperative, and by committing to standardized knowledge within your development teams, you can retain your top IT talent—and ensure your teams are poised and ready to meet these new challenges.
April 11, 2022
Tackling CostOps When You’re a Developer

The implementation and successful execution of IT initiatives often comes down to developers. From those working with commercial vendors to those working with open source projects, cloud-based software-as-a-service (SaaS) solutions, managed service providers (MSPs) and internal teams, they all play an important role in the successful running of IT and operations teams.

One challenge developer and operations teams face is the huge time barrier between development and follow-on operations. What a developer builds today for an organization may continue to be used decades from now, and will likely be maintained, patched, updated and modified by other developers. Though it’s a point of pride for the developer—who will have likely moved on in their career—it can be challenging for the operations team who has to continue to deploy it year after year.

With more applications than ever before, the traditional role of the developer is changing. Today, developers need to start factoring operations cost containment into their responsibilities.

Opportunity for the Developer

Developers are rarely responsible for long-term business decisions. Once code is out of their hands, they often have little to no stewardship of their work. The business will make critical IT cost optimization and containment decisions about its code, language and toolchain epochs later. Developers focus on the now and the near future. Spending hours envisioning the one winning outcome out of 10,000 possibilities years later won’t get code out the door or achieve the delivery goals of their managers.

However, developers have an often-overlooked opportunity to include the gears to activate cost containment support during initial development. From there, if their work is good, it’ll endure; extended support will eventually be necessary. The first step is to ensure new and modernized applications include solid cloud-native monitoring from the outset. This not only helps DevOps and SRE teams but benefits the entire business, who’ll own those applications for the long term.

Integrating operations-focused monitoring is a top priority for developers. In some cases, an application using platform-managed serverless or orchestrated containerization has no shortage of opportunities to provide developers and operations with detailed performance data.

A developer’s expertise with dependency analysis, code maturity and ease of integration is a great measurement tool to evaluate tool durability. A solution capable of easily combining cloud-native application performance details with infrastructure performance, user experience and operational expense data is even more likely to help developers and businesses manage costs in the long run.

A New Dimension

As cloud-native becomes a mainstream enterprise strategy, cost containment is a key opportunity. To meet the future challenges of application management, leadership is rapidly expanding their definition of application suitability and quality to incorporate all aspects of application ownership. For some teams, cost containment is a new dimension of senior developer maturity.

March 14, 2022
Speed and Security: How to Find a Balance in Development
Time management can help developers realize the necessary combination of security and speed in application development

As software continues to create competitive advantages and market differentiation for organizations focused on successful digital transformation, developers will continue to be measured by how fast they can develop and release code. Business leaders encourage DevOps teams to push the pace of innovation to deliver features quickly while leveraging the newest technologies to gain any available advantage in the market. As a result, development and operations teams work at breakneck speeds to meet the deadlines of short and frequent development life cycles.

Security teams, however, are not measured by speed but rather by certain metrics of success, such as whether they achieve a certain security rating or reduce the number of incidents. Traditionally, security sets corporate standards and best practices centrally and tries to enforce them across the enterprise. With DevOps teams becoming increasingly distributed and decentralized, these conventional security methods no longer work and have actually slowed down security and/or blocked the software development process. Security tries to keep pace, but with numerous disparate reports to review and too many results to manage, they inevitably fall behind. In the rush to catch up, either critical vulnerabilities are overlooked or frustration sets in when releases are delayed.

Developer and security teams both know the importance of delivering secure software but struggle to find the right balance. Let’s take a look at some strategies that can help ease the tension between DevOps and security teams and create a happier work environment, where safer code is delivered to the market.

Incorporating Time Management for Speed and Security

Consider implementing the following time management strategies to make life easier for developers:

Integrate AppSec in the DevOps Pipeline: Application security scanning should be incorporated early and often within the DevOps pipeline. While this statement may sound like a broken record, the days of running scanning tools independent of your pipelines are gone. Studies have shown the earlier you detect an issue, the easier it is for developers to fix—with as much as 50% time reclaimed over late-stage remediation. By including AppSec in the pipeline, you ensure all issues are identified as quickly as possible. This effort will not only improve the quality of your software but also free up more time overall for developers to write code and create cool features. Sounds like a win-win for both teams.

Incorporate automation as much as possible: DevOps teams should look to leverage automation when implementing application security wherever possible. Many of the steps for securing your software development process are repetitive. For AppSec to be successful, the steps must work within the developer’s world and along with the other tools developers love. Developers look for tools that help them move faster.

Here are some areas where application security automation can unburden developers from repetitive tasks:
- Use AppSec tools that can be easily tied into your DevOps toolchains. As a matter of fact, a developer may not even need to know what security tools are being run, only that they are executed on every run of the pipeline.
- Automate the ingestion and correlation of the security findings. There will often be a boatload of redundancies between the findings from different security tools (i.e. SAST versus DAST versus SCA) and the potential compression ratios can be in the order of 10-50:1. There are tools to automate this process and dramatically simplify making sense of all the data. The onus shouldn’t be on the developer.
- Once a security issue has been identified, it should automatically create a ticket in the ticketing system that developers use today. We all know developers don’t like being forced to use a new tool. Give them the findings in the system they are already familiar with.
Tailor security training for your teams: Security training is a great way to reduce the number of security findings within an organization, which in turn frees up developers to work on other pressing issues. However, most security training is ad hoc and not tailored to meet the needs of the development teams. Even the most basic security testing can provide baseline metrics that enable security teams to recommend tailored training for development teams. These sessions can focus on areas that teams struggle with or areas known to be susceptible. Training is also a great way to develop security champions within the development teams, dedicated professionals who have shown to improve the overall security posture and awareness of organizations. Security champions—and the programs they advocate—have the power to improve both AppSec and relationships among people, a sentiment endorsed by 84% of industry professionals.

Summary

Developers know the importance of security and truly want to deliver safe code in the timeframes they are being judged on. By incorporating the strategies outlined above, they will have a fighting chance to reach the lofty goals set by the business, while releasing higher quality and more secure code. Security teams need to meet the developers in their world and help enable application security to be automated and work at the speed of DevOps.
December 11, 2020
3 Observability Strategies to Reduce Debugging Cycles

These observability tips can help developers uncover issues that impede performance and derail customer experience

Modern technologies and methodologies such as cloud services, containers, DevOps, microservices and serverless have made it easier for organizations to deploy application code to production. However, the additional layers within the modern tech stack have increased complexity, making it more difficult to identify performance issues and trace them back to the root cause before customers are impacted.

While all these layers can be instrumented for observability, each of them spits out different information and for different roles. If something goes wrong, where does a developer begin to look? Maybe they get mentioned on a support ticket. Or they end up staring at high-level visualizations of systems from the various performance monitoring and logging tools. But there are graphs everywhere. There’s no surefire way for the developer to hone in on what they’re supposed to look at quickly. By standing back and eyeballing the graphs, they’ll obviously see the spike. But with all this information and so many variables, it’s not easy to pinpoint the reasons why.

When incidents impact large numbers of customers, they could spend many days trying to find the answer. Without context into what’s causing the issue, developers spend most of this time blindly following different trails and doing things like arbitrary code rollbacks. It’s costly. Studies show organizations spend about $4.6 million annually on incident management.

Observability: Strategies to Consider

Development teams can save time and reduce costs by implementing three overlooked observability strategies that offer a new perspective and carry learnings forward.

Compare Changes Over Time

To help reduce the time it takes to trace the issue back to a specific reason, developers should look at metrics over time. Find the closest representation of an identical window of activity and compare it to a previous point in time to help spot an outlier. For example, if a development team is alerted to an incident in which customers are reporting slow response times with an application, they can compare transaction data from similar time periods to look for correlations. Data may reveal the issue is only with API calls, although individual API call times were not significant enough to meet thresholds and trigger alarms. However, comparing those API calls to those from a week ago under the same parameters—same time, same server, etc.—they discover the aggregate of the API calls is performing 20% slower.

Looking at performance comparisons over time gives developers visibility into performance degradation before it starts to hurt a large number of users.

Analyze User Behavior Patterns

A development team should also evaluate the behavior patterns of individuals and IP addresses that are interacting with an application. This can provide insights that help to differentiate between external (user, browser) causes and internal causes (problems with code or a company’s systems).

For example, consider an e-commerce customer trying to modify items in their shopping cart before checking out. The application is responding slowly to their click (it’s taking 2 seconds), so they click repeatedly. A single occurrence might be overlooked but repeated 2-second delays will add up to a significant impact. It may even trigger alerts that the database is overloaded or there’s a code problem when in reality user behaviors are compounding an underlying problem. Some log activity may look like an attack, but investigating the users impacted may reveal the root cause is actually a browser behaving unusually or a misconfigured proxy that is sending anomalous traffic.

In either case, developers end up wasting time by implementing corrective actions that may not solve the problem. But considering patterns—viewing events that fired before and after the problem and behavior from individual users along with IP addresses—will enable developers to quickly trace performance issues back to the root cause.

Properly Assess Impact

Properly assessing the impact of an issue is another useful strategy that can help to determine response to an alert. For instance, if a subset of users < 0.5% are repeatedly encountering an error with an API, are unable to access the functionality (as in the shopping cart example) and are persistently re-attempting the API, you will see a spike in error rates that implies a bigger problem.

Another example might be new functionality that has been rolled out that is for some reason incompatible with components of older accounts, such as incompatible profile information or data thresholds. As customers explore, this small subset of users will encounter the bug. It may appear as though the feature is broken when, in reality, it is a problem with some legacy data.

When problems affect only a few customers, it is beneficial for developers to lean on monitoring tools for alerts and context necessary to resolve an issue without needing to rely on support or support tickets.

Apply Observability Learnings Forward in Post Discovery

For issues that impacted many users or were difficult to diagnose, development teams should consider adding a custom metric or monitor when they do their post-discovery analysis. Whatever the metric is—an extra line in a log, a StatsD metric or putting a catch around it to log the trace—it is important to capture the lesson learned and make it actionable for the future, which will save time during discovery when it happens again. Without this step, the mean time to resolution for future issues may continue to worsen as software complexity increases.

Lack of actionable data is one of the reasons why companies spend millions of dollars annually on managing and resolving incidents. As development teams look to expand observability practices, they should first consider what puts the developer in the best position to resolve an error or performance issue quickly and ultimately deliver exceptional customer experiences.

July 30, 2020