devopsy.kinsta.cloud

Tag: technical debt

Why Nano Updates Only Work if You Begin with the Latest and Greatest Software
Is there a silent threat of technical debt looming in your organization? You probably should take a deeper look.

Modern software systems are composed of hundreds of interdependent components. How are you updating those components – system libraries, base containers, and app packages? Do it poorly, and you will trigger unintended side effects, like failing CI pipelines, hours of debugging, or even outages.

Ironically, these risks cause teams to delay updates, even when they fix critical vulnerabilities. According to recent research, 60% of known vulnerabilities remain unaddressed in the average organization today.

The solution to reducing technical debt is simple: consistent nano updates – precise, minimal changes that reduce risk while maintaining agility. However, this strategy works best when you’re starting with the latest and greatest software from the beginning. Let’s explore how implementing continuous nano updates and starting with the latest software can reduce technical debt before it starts to accumulate.

What are Continuous Nano Updates – and Why are They Different?

Nano updates are surgical upgrades that continuously apply the smallest possible change needed to update systems. This eliminates security vulnerabilities and reduces the risk of unintended side effects while improving control and traceability. Developers replace a single vulnerable dependency in a container without altering the rest of the image, or update a package version while preserving all surrounding libraries and configurations.

Nano updates allow for continuous delivery, improved collaboration and faster triage and recovery when something does break. Knowing these distinctions can make or break your approach to updates and ensure they roll out successfully.

Starting with the Latest is a Prerequisite

Many organizations assume they’re “up to date” simply because they use mainstream Linux distributions like Red Hat or Ubuntu. In reality, these traditional Linux distributions ship a snapshot in time, typically from two to five years ago. These are outdated base images, with lagging dependencies and contribute to accumulating technical debt throughout their entire organizations. Using a one-year-old image doesn’t qualify as the latest version because it’s often hundreds of upstream releases behind, leaving hundreds of CVEs unpatched.

Nano updates only truly work when you’re already on the current software stacks. Otherwise, small updates are impractical and will have outsized effects due to compounding code drift from legacy package versions. For example, updating a base container image might pull in multiple outdated packages with incompatible and conflicting versions, yielding issues and breaking integrations. If you haven’t modernized your foundation, you can’t build a nano update strategy on top of it. It’s critical to think about this early on and ensure the foundation is there.

Ending the Tug-of-War Between Security and Engineering Teams

Once the latest and greatest versions of all tools are in place, engineering teams often operate in a steady state of “maintaining and monitoring.” At the same time, security teams are focused on making upgrades – and understanding the risks they pose – to protect the organization from further risks. By making regular nano updates, together engineering and security teams, can prevent longer-term risks and issues from the start.

Some healthcare organizations, for example, may have long periods of software freezes like during open enrollment. Broad updates are risky during this time, but nano updates enable teams to patch vulnerabilities and avoid the most destabilizing environments. The result is that patient-critical systems remain both secure and operational. Taking this approach helps bridge the gap between security and engineering teams and ensures that technical debt isn’t looming throughout the organization.

Helping Developers get a Leg Up by Starting Clean and Staying Clean

Best-in-class developers are adopting nano updates as part of modern CI/CD workflows and seeing benefits from continuous delivery as smaller incremental changes facilitate more frequent, reliable deployments. They’re also improving collaboration across their development and operations teams and seeing faster recovery as pinpointing and reverting specific changes to issues becomes more straightforward.

However, developers in legacy organizations often struggle to adopt this mindset. To bridge that gap, there are four steps they can take to set themselves up for success:
1. Rebase first: Start by modernizing base containers and dependencies. Without this, truly nano updates aren’t really possible.
2. Automate dependency management: Implement monitoring to detect and assess the impact of changes in real-time.
3. Monitor changes closely: Track the effect of each nano update using observability and alerting, rolling back quickly and triaging regressions easily.
4. Educate the team: Foster a culture that values incrementalism and sees continuous nano updates as a way to move faster with less risk.
Engineering teams can’t afford to delay updates, but they also can’t risk instability. Nano updates give them the best of both worlds – but only after they’ve adopted the latest software versions as a baseline.

The real challenge isn’t just applying updates – it’s building a culture of discipline around starting clean and maintaining great hygiene. With this approach, organizations avoid unnecessary spending, reduce CVE exposure, empower developers with the latest software features, and move confidently in a complex, fast-moving world free of technical debt.
October 23, 2025
Technical Debt: Make Developers Happier Now or Pay More Later

In a survey of over a thousand CIOs, CTOs, CISOs and other tech execs conducted by Protiviti, organizations spend an average of 30% of their total IT budgets on managing technical debt. Not only is it costly, but nearly 70% felt that technical debt highly impacted their ability to innovate.

Technical debt happens to organizations of all stripes and sizes, and while some may take solace in knowing they’re not alone, it’s no less devastating. Many point the finger at their engineering teams because it’s easy to do, but they’re not the real culprit. You’ll often find it results from poor developer experience (DX) due to years of non-technical leadership prioritizing the product roadmap over anything else. This can cause productivity to suffer and culture to turn toxic. Focusing on developer work environments offers a solution for organizations struggling with technical debt. An important part is providing advanced AI tools that reward the developer and company with growth.

An Undisputed Champion

Software development teams need a DX champion. This senior engineer can track the time a new developer may need to create their own environment. If it takes too long, the DX champ will dig in and work alongside the developer to tighten that window.

Consider the onboarding of a new developer, which is historically not always the best process. It could take weeks for a new developer to offer a simple patch or minor feature, only to render a continuous integration (CI) service unresponsive. The DX leader then needs to step back and evaluate because this could result from a less-than-well-written test suite, which the new developer did not cause.

Existing teams may be okay working with aged technology, but a step back like this can disappoint someone accustomed to modern tools. Remember that tools like CircleCI can help determine if a test is flaky. A DX champion will address this need and halt activities following each sprint to ensure code is clean and reusable moving forward.

Measure Often

A DX champion should be in tune with a company’s morale. Regular surveys can reveal if a developer is finding gratification in their work. There’s also employee turnover and specific projects that cause issues to consider. And if you want developers to stay, you have to show their concerns are being heard.

There are trackable metrics to understand technical debt and its impact, one being “time to fix.” Let’s say there’s a fixable bug: Tools do exist to measure how long resolution takes from code through completion. This could show that a minor patch took two days to ship, which should be done in hours. This insight will enable the DX champion to spearhead corrections and ensure their teams have the right technology.

Not So Secret Agents

For developers, it’s all about coding. Tools should help push projects into production, not create bottlenecks. While AI tooling can cut tremendous amounts of time, teams must set standards regarding the acceptable level of tech debt. So, a DX champion should candidly discuss acceptable debt and code quality to get their team’s buy-in.

Increasingly, organizations are deploying AI agents supported by engineer orchestration. The World Economic Forum recently projected that by 2028, a third of all enterprise software apps will include AI agents. Additionally, up to 90% of businesses now view agentic AI as a possible competitive advantage that bolsters efficiency, scale and decision making. If a report shows a small bug that AI agents can correct, this can free up developers to tackle more rewarding, higher-value tasks.

Still, it’s not something you can set and forget—human oversight is needed to monitor things and make decisions that AI cannot be trusted to make.

Make Them Happy

It’s no secret that there will be times when your organization must immediately get a product out the door. It might have weak scalability and degrade over time, but try as they may, developers will never fully correct this situation. And when this becomes a way of doing business, technical debt spirals out of control, so make them happy now.

Getting your culture right is critical, or you’ll lose a lot of money. Every company can allocate a few hours each week to strengthen the DX. And, they should be able to name a DX champion that’ll keep things on track. You can invest a little now to make developers happy, or pay more later in stalled innovation, slow performance, security headaches and limited ability to compete in the future.

October 10, 2025
Cultural Debt vs. Technical Debt in Infrastructure Automation
Infrastructure is humming, deployments are fast and dashboards are green. But something still feels off. Despite the technical efficiency, teams might be slowed down by recurring issues, misalignments, or resistance to change. These aren’t always rooted in code.

While technical debt, the accumulated cost of quick fixes and legacy code, is a familiar concept, cultural debt often goes unnoticed. It surfaces through unclear communication, misaligned incentives, or outdated norms that no longer support evolving goals.

Both forms of debt can quietly undermine DevOps practices, as they limit the very automation and agility they’re meant to enable.

In this article, we’ll explore the differences between technical and cultural debt, how each impacts DevOps and infrastructure automation and practical approaches to identifying and addressing them.

Defining the Terms

Technical debt means the trade-offs made during development to prioritize speed over long-term quality. These shortcuts might be intentional or the result of time pressure, resource constraints, or evolving requirements. While they can accelerate delivery in the short term, they often lead to brittle systems, increased maintenance burden and operational inefficiencies down the line.

Technical debt can manifest in many ways: hardcoded configurations that don’t scale across environments, inadequate or missing monitoring, outdated provisioning scripts, or reliance on manual processes that should have been automated. Over time, these issues can drag on velocity and complicate troubleshooting.

On the other hand, cultural debt is less tangible but equally impactful. It represents the accumulation of unresolved issues in team interactions, which can mean mismatched expectations, ineffective communication and outdated cultural norms that no longer support the team’s goals.

Examples include resistance to adopting new tools, siloed teams with poor cross-functional collaboration and unclear ownership of infrastructure components. Like technical debt, cultural debt builds silently and can impede progress if left unaddressed.

Why Cultural Debt is the Silent Killer in DevOps

DevOps is more than pipelines and automation. At its core, it’s a cultural transformation.

It’s about breaking down silos, fostering collaboration, building fast feedback loops and promoting shared responsibility across development and operations. Even the most advanced tools and workflows struggle to deliver their full value without the right cultural foundation.

This is where cultural debt becomes particularly insidious. Unlike technical debt, which is often visible in logs or performance metrics, cultural debt hides in team dynamics and day-to-day interactions.

It shows up when teams resist adopting new tools, cling to “the way we’ve always done it,” or hesitate to take ownership of shared responsibilities. These patterns create friction, slow onboarding and make scaling CI/CD efforts far more complex than necessary.

Moreover, cultural debt tends to compound quietly. A single unclear decision-making process or a lack of psychological safety can ripple outward, leading to missed handoffs, redundant work, or breakdowns in trust. Over time, this erodes the very principles DevOps relies on: agility, transparency and continuous improvement.

Addressing cultural debt means creating an environment where teams can adapt together. Without this cultural alignment, the technical side of DevOps is left propping up an unstable foundation.

How Technical and Cultural Debt Feed Each Other

A dysfunctional team culture, marked by poor communication, unclear ownership, or fear of failure, often leads to rushed decisions and technical shortcuts. These quick fixes accumulate into brittle systems and fragile automation.

As technical debt grows, so does the hesitation to make changes. Teams avoid touching fragile pipelines, unsure of the impact or who’s responsible. This reluctance reinforces siloed behavior and deepens mistrust, further degrading the culture.

It’s a vicious cycle: bad culture leads to poor code, and poor code makes the culture worse.

Breaking the loop requires tackling both, cleaning up systems while fostering a culture that supports shared responsibility and continuous improvement.

Diagnosing and Measuring the Debts

Recognizing technical and cultural debt early is key to preventing long-term drag on DevOps performance. While technical debt often leaves a visible trail, cultural debt tends to require a more observational, qualitative approach.

Signs of Technical Debt

These indicators often show up in system performance and team workflows:
- Frequent outages or instability after changes
- Long release cycles or painful deployments
- Manual tasks where automation should exist
- High effort to onboard new infrastructure or services
Signs of Cultural Debt

Cultural issues can be more subtle, but they have a tangible impact on collaboration and growth:
- A culture of blame instead of shared accountability
- Knowledge hoarding or unclear documentation
- Avoidance of retrospectives, feedback, or experimentation
- Siloed teams with limited cross-functional interaction
While tools like DORA metrics can help measure delivery performance (e.g., deployment frequency, lead time for changes), cultural debt often requires listening to team dynamics, observing behaviors and surfacing what’s not being said.

Technical problems may flag a deeper cultural issue, and addressing both starts with awareness.

Strategies to Address Both Debts Together

Addressing technical and cultural debt in isolation only offers a temporary fix. Both must be tackled in tandem to build sustainable DevOps practices because culture drives how technology is built, maintained and evolved.

Start with the code, but don’t stop there.

Make space for regular refactoring and cleanup cycles, as they are excellent opportunities to reassert quality standards. Strengthen test coverage and invest in observability, not just for performance but also to reduce fear when deploying changes. And when automating, be deliberate. Automation should solve real problems, not just shift complexity around.

But even the cleanest codebase won’t thrive in a broken culture. Creating psychological safety is foundational: people need to feel safe speaking up, admitting mistakes and asking questions.

Leaders should actively model DevOps values like transparency, shared responsibility and continuous improvement. Practices like cross-functional pairing and meaningful retrospectives help reinforce collaboration across silos.

Tools can support this work, platforms like Terraform Cloud, Spacelift, GitHub Actions, or Jenkins enable better infrastructure management and automation. But tools don’t solve culture; they amplify it. A strong foundation of trust and clear communication is what allows these platforms to truly accelerate delivery.

Wrapping up

Technical and cultural debt both undermine DevOps success, but cultural debt often lays the groundwork for technical shortcuts and long-term inefficiencies. While it’s easier to spot failing pipelines than failing team dynamics, it’s usually the culture that shapes how systems evolve.

Investing in tooling and automation is important, but without a healthy culture to support it, those investments won’t scale. Treat culture as a core part of your infrastructure strategy, as it’s how you build and grow.
June 9, 2025
How Microservices Guardrails Help Teams Move Faster
The evolution of enterprise software architecture has brought microservices to the forefront of modern development practices. Although a microservices architecture offers agility and scalability, it often introduces new complexity that can become overwhelming if left unchecked.

In speaking with customers about their distributed architectures, I’ve seen complexity emerge in various ways as their distributed systems have evolved over the years. This complexity naturally builds up in homegrown applications – even those with only 20-30 services. Another common challenge I’ve encountered is when organizations acquire systems with microservices-based architectures. Teams often lack clarity on service interactions and dependencies, especially when key employees have departed and technical documentation is missing. This leads to slow deployments and significant troubleshooting difficulties across hundreds of microservices.

The Mounting Challenges of Unmanaged Microservices

Although there’s a clear trend toward microservices adoption, merely implementing microservices doesn’t guarantee speed and agility. Without proper management, microservices can proliferate rapidly, creating numerous challenges for development teams. Individual teams may create unexpected cross-dependencies and redundant services, gradually drifting from the original architectural vision. The rise of AI-powered development tools and the corresponding proliferation of code compounds this challenge.

The Impact of Insufficient Governance

When microservices grow without oversight and proper documentation, often resulting in unneeded dependencies and service duplication, organizations face diminished system performance and reduced development velocity, which can result in increased operational complexity and costs. Teams spend more time debugging complex interactions between services and less time delivering new features. They face challenges such as identifying unnecessary circular dependencies, hard-to-trace multi-hop flows and difficulty visualizing and documenting microservices in real time. For large teams, onboarding new team members becomes increasingly cumbersome as the system’s complexity grows.

The long-term impact of this complexity on not only system health but on engineering velocity and team morale puts organizations at risk of lagging innovation. Without intervention, teams must either increase their time spent on maintenance and fixes or face mounting technical debt.

Creating Effective Guardrails

Well-implemented governance enables organizations to support the architectural lifecycle – a continuous process that ensures a system’s architecture remains aligned with its design and evolving requirements. It allows teams to move quickly while maintaining the overall system’s integrity and resilience. These guardrails should define clear boundaries for service interaction, data access patterns and technological choices while still allowing teams to innovate within their domains. Clear guidance helps teams make more consistent decisions, which leads to better system reliability, easier maintenance and faster development cycles.

By enforcing best practices, organizations can avoid suboptimal design decisions that can impact system efficiency and reliability. Moreover, a well-governed architecture typically requires less manual intervention and troubleshooting, reducing the operational burden on IT teams. Governance also permits systems to more efficiently handle increased loads, allowing businesses to grow without the constant need for modernization.

Best Practices for Implementing Effective Microservices Governance

Successful microservices governance requires a balance between control and flexibility. Organizations should establish clear architectural principles that guide team decisions without overly restricting their ability to solve problems. These principles should include:
- Automated checks for architectural violations.
- Clear guidelines for application boundaries.
- Established patterns for common integration scenarios.
- Defined workflows for cross-team collaboration and service changes.
Fostering a culture of architectural responsibility across teams is also important. Organizations must:
- Establish clear guidelines and standards for architectural decisions.
- Integrate architectural considerations into the development process from the beginning.
- Promote the benefits of continuous architectural improvement.
- Enable teams to automatically visualize architecture in real time as it evolves.
Additionally, adopting smaller, autonomous teams can significantly improve collaboration and accountability in microservices development. Following Amazon’s “two-pizza team” principle, the idea is to keep teams small enough that they can be fed two pizzas. This helps ensure streamlined communication and efficient coordination.

Modern Tools For Architectural Governance

Although enterprise architecture has long been a core practice in most organizations, modern architectural governance relies increasingly on automated tools that can detect and prevent architectural drift before it becomes problematic. When architectural observability—or the ability to analyze software architecture, observe drift and identify issues—is combined with AI and automation capabilities, teams can implement and act on the software architecture life cycle in real time. When complemented with application performance management (APM) tools, architectural observability accelerates incident response and reduces mean time to repair (MTTR) through enhanced system visibility and a contextual view of system flows.

Taming Complexity: Examples From the Field

With the right tools, organizations can work their way back to a stable microservices architecture. For example, a global travel services provider acquired a microservices-based system from a smaller organization. At that point, key employees had left the company and minimal documentation made it difficult to see and understand the interactions across hundreds of microservices in production. In this scenario, architectural mapping and sequence diagrams of production flows would provide the necessary visibility into their architecture to regain clarity and control of the distributed system.

In the same context, ensuring adherence to existing architectural rules of the parent company is important for alignment. By implementing architectural observability and governance, the organization can align its microservices-based system with broader portfolio protocols, improving production efficiency, accelerating deployments and simplifying troubleshooting.

This challenge isn’t unique to a single organization. Even industry leaders face governance obstacles when managing complex microservices architectures at scale. Netflix’s transition from a monolithic architecture to microservices has been widely recognized as a success in enabling scalability and innovation. However, as its system grew, the company faced governance challenges in managing service complexity, ensuring stability and maintaining rapid feature development. A prime example is Netflix’s video processing pipeline, which initially ran on a monolithic system called Reloaded but became difficult to scale and maintain as video formats and streaming demands evolved.

To address these challenges, Netflix rebuilt its video pipeline on a microservices-based platform called Cosmos, introducing clear service boundaries and workflow orchestration. This shift to microservices improved observability, accelerated feature development and increased system flexibility. By implementing structured governance and automation, Netflix transformed its video processing pipeline into a more scalable, efficient and adaptable system, ensuring continued innovation while minimizing operational complexity.

The Path To Sustainable Microservices

Microservices architecture offers benefits in terms of scalability and agility. However, its success is contingent on establishing flexible guardrails through effective governance and documentation. By implementing clear principles, automated checks, proper tooling and accurate documentation, organizations can enable their teams to move quickly while maintaining a cohesive, maintainable system architecture. This balanced approach helps ensure that teams can rapidly develop and deploy applications while maintaining the microservices promise of agility and speed.
May 22, 2025
Does Using AI Assistants Lead to Lower Code Quality?

The emergence of generative AI has permanently altered how code gets written — and due to the massive productivity boost, there’s no going back. According to GitHub, 92% of developers said they use AI tools, and developers complete tasks 55% quicker using Copilot, the popular LLM-based code generator.

Yet, new research has potentially upset the idea that AI assistants like Copilot are positively advancing software development without consequences. The report, conducted by GitClear, analyzed over 150 million lines of code over the last four years and found an upsetting uptick in code churn and a decrease in code reuse since the dawn of LLM-driven software development.

So, this begs the question: Does using AI assistants actually lead to lower code quality? Well, the answer is complicated. “I wouldn’t say that the report proves that AI assistants are reducing code quality since our data is correlational,” said Bill Harding, programmer and CEO at Amplenote & GitClear, the organization behind the aforementioned research. “But I would say that there is a confluence of indicators that show code quality decline at scale, starting around 2022 and accelerating in 2023.”

AI-Generated Code Increases Overall Volume

One potential contributor to lower code quality is the sheer amount of code produced. Since LLMs automate code generation, production is accelerated across the board. Without guardrails, generative AI will likely contribute to more technical debt.

“AI-enabled code development is going to have a major effect on coding, exponentially increasing the volume and velocity of code delivery, much of which we anticipate will be of lower quality and more bloated,” said Mehran Farimani, CEO, RapidFort.

In addition to quality concerns, there are also governance and security concerns to consider from the use of AI and surges of new code production. Some anticipate this will overwhelm security teams, which have already been struggling to keep pace. “Increased developer velocity is going to put security teams under even more pressure,” said Farimani. “The volume of scans will increase, the range of vulnerabilities that need to be addressed will increase and the patches that need to be applied will increase.”

Blindly Relying on AI Outputs

According to GitHub, developers accept code completions from Copilot 30% of the time. This percentage is likely to keep increasing as the LLM is trained on internal style and comfort with the tool increases. However, there is still the risk of developers blindly relying upon generative AI outputs without giving them the proper oversight, which could also affect code quality.

“I think there is a risk for lower quality code if a developer blindly relies on what an AI assistant generates,” said Rob Whiteley, CEO at Coder. “Generative AI tools have flaws, and if outputs are not verified, they can insert code that compiles but does not behave as a developer may have intended. It can also insert errors or bugs if not checked.”

How to Retain Code Quality in the AI Age

“I would not say that results of [the GitClear] study are conclusive,” said Dr. Eirini Kalliamvakou, staff researcher at GitHub. She believes the results are misleading since there is no way to know if an AI or a human authored the code that was analyzed for the study. That said, she acknowledged that “if we are helping people code faster, we should ask the question about code quality.” So, how do we retain the quality of all code, regardless of whether it’s human- or machine-generated?

Use Copilot Thoughtfully

The first recommendation is to use AI assistants thoughtfully. If you’re exhausted at the end of a long day, it’s a lot easier just to “hit tab” and accept an AI-generated auto-complete. But this might not be the best move. “When used thoughtfully and with oversight from the developer, I do think AI can improve the quality of a developer’s code by making suggestions and preventing human errors,” said Whiteley.

Apply the Same Code Reviews to AI-Generated Code

Additionally, development teams should be applying the same degree of rigor to AI-generated code, taking it through the same design review processes and security reviews as you would for any new application. “Just because you’re writing with Copilot doesn’t mean you should give up on code reviews, your linters, looking for security vulnerabilities and all that,” said Kalliamvakou.

Avoid Redundancy When Possible

Leaner code means efficiency boosts and cost-savings. “Much of code is redundant and AI can streamline that redundancy and offer interesting insights, but not without developers ensuring the suggestions the AI tool makes align with the code they are intending to write,” said Whiteley. “Similar to Gmail’s autocomplete sentence feature, AI can complete lines of code with accuracy, but only as long as it accurately guesses where you’re going with your sentence.”

Keep Tabs on the Impact on Quality

Multiple sources cited that around 30% of code is now AI-authored. Harding calls for more efforts to analyze the impact of AI generation on code quality. “It would benefit the greater good to get a clear and data-backed assessment of how quality fluctuates when developers hit ‘tab’ to accept an AI suggestion,” he said. This is a new field of study, and will require new metrics to fully monitor and appreciate the impact AI has on code quality.

Determine How To Measure Code Quality

Similar to the dearth of consensus on what exactly constitutes developer productivity, defining and measuring code quality is just as challenging. Therefore, Kalliamvakou recommends first defining what code quality means to your organization and what the metrics are. For instance, this could be the “number of defects.” Then, the next step would be to set up methods to alert developers about potential quality issues, she said.

The Bottom Line: AI Enhances Productivity…

AI assistants are changing the way code gets made, and much of this is for the better.

Most notably, AI can have a significant impact on enhancing productivity for up-and-coming or inexperienced developers. “If an enterprise makes technology like generative AI broadly available, then it can supplement the work and boost the productivity of the base of the talent pyramid,” said Whiteley. “Put simply, that investment massively impacts the broader organization, not just the top 1% of developers.” Even if such tools make a developer 20% more productive, they can have a truly transformative impact.

While the GitClear study doesn’t prove the culpability of AI assistants, it raises some interesting questions worth exploring, said Harding. Notably, is code becoming more repetitive? If so, that could have some negative outcomes for overall efficiency. “Code generated during 2023 more closely resembles an itinerant contributor, prone to violate the DRY-ness of the repos visited,” the GitClear report said.

… So, Assess Code Quality

“If leveraged correctly, generative AI will not lower the quality of code,” said Trisha Gee, lead developer advocate at Gradle, Inc. “However, it’s important that developers using this technology are critically evaluating AI-generated code rather than taking what AI suggests at face value. Code from generative AI may not always be ‘correct,’ be the best solution to a problem or match the code style of the codebase.”

My takeaway is that, yes, you can retain code quality while using AI assistants. However, this assessment comes with some strings attached. It will require an ongoing effort to sanitize the outputs of AI and instill the same checks we typically apply to manually-generated code.

Image source: https://vecteezy_futuristic-technology-concept-mixed-media-innovations-data_7019122_167-3.jpg

February 6, 2024
Will the Rise of Generative AI Increase Technical Debt?

In the last year and a half, we’ve seen a glut of new generative AI-related tools reach the market. Various AI pair programmers powered by large language models (LLMs) are becoming integrated into software development environments. The number of new generative AI startups is staggering, and generative AI seems to be the new “catch-all” to accelerate software development and operations.

In this new paradigm, AI tooling accumulation seems inevitable. Yet, the explosion of new AI APIs, libraries and frameworks has the very real potential to increase technical debt and, in the process, overwhelm DevOps teams. “These new applications and capabilities will require many new MLOps processes and tools that could overwhelm any existing, already overloaded DevOps team,” said Bob Quillin, vFunction chief ecosystem officer.

Below, we’ll consider if and how new generative AI tooling has the potential to exacerbate technical debt. Notably, the rush to adopt AI, shifts in the AI tooling landscape and accelerated code output could all be producers of technical debt. Choosing your tools correctly, iterating thoughtfully and upskilling teams will be important guidelines to mitigate negative outcomes. That, and creating unique solutions to ‘flip the coin’ — using AI to consolidate code and reduce existing technical debt.

How AI Will Contribute to Technical Debt

The rise of generative AI-related tools will likely increase technical debt, both due to the rush to hastily adopt new capabilities and the need to mold AI models to suit specific requirements. “New LLMs and generative AI applications will undoubtedly increase technical debt in the future, or at a minimum, greatly increase the need to manage that debt proactively,” said Quillin. “It starts with new requirements to continually manage, maintain, and nurture these models from a broad range of new KPIs from bias, concept drift, and shifting business, consumer, and environmental inputs and goals,” he said.

Incorporating AI may require a significant upfront commitment, leading to additional technical debt. “It won’t be just a build-and-maintain scenario, but rather, the first of many steps on a long road ahead,” said Prince Kohli, CTO of Automation Anywhere. Product companies with a generative AI focus must invest in creating a data and model strategy, a data architecture to work with AI, controls for the AI and more. “Technology disruptions and pivots such as this always lead to this kind of technical debt that must be continually paid down, but it’s the price of admittance,” he said.

AI is a tumultuous space, and some degree of technical debt will likely be produced by future organizational shifts. “The pace of change and innovation in the AI field itself is exceedingly rapid,” said Juan Orlandini, CTO of Insight North America, citing the staggering emergence of new AI startups and tools. “There will be consolidation, and also, unfortunately, a number of the startups will cease to exist.” When this occurs, teams must grapple with unsupported packages or tools.

Another type of debt is the nature of AI and how it accelerates development across the board. “One of the great values of these tools is the acceleration they bring to development,” said Orlandini. “Simply put, we can put out more features faster.” The other side of this double-edged sword is that AI accelerates the creation of all programming endeavors, even those that become technical debt.

Generative AI-related tools are most likely to accelerate the pace of change and deliver new code development outcomes more quickly. “In the process of consuming their output, there is a possibility of accumulating technical debt if the usage is broad, rapid, and pursued indiscriminately,” said Ajay Sabhlok, CIO and CDO, Rubrik. “Developers, in this case, may be too focused on program execution and forget to pay attention to code complexity, repetition or use of non-standard logic.”

Ways to Mitigate AI Technical Debt

Thankfully, there are specific methods organizations can adopt to manage new technical debt. For starters, you should focus on creating smaller proof of concepts and then iterating as you learn, said Kohli. “When leaders like the head of technology or IT are deploying these processes, they should first create small agile teams that learn fast and disseminate those learnings efficiently,” he said.

Also, technical debt is more likely to accrue when the wrong platform or tool is chosen. Therefore, “organizations should be very deliberate in choosing what products to leverage,” said Orlandini. He encourages IT leaders to avoid relying on companies or products with flashy presentations and poor tech or great tech with poor business acumen. “Our industry is rife with stories of great products with terrible business plans — which ultimately leads to their demise,” he said.

Other experts similarly foresee upcoming shifts that may affect internal operations. “Companies should keep their options open when considering AI model vendors or tool providers,” said Kohli. “They must keep their ‘hands on the wheel’ and be prepared for turns.”

Just like any new technological adoption, generative AI requires forethought. Therefore, putting the same amount of planning and preparation into deploying generative AI as we do for developing coding standards around manual coding could help minimize technical debt, said Sabhlok. “It will help to ease into areas of usage by adopting a tiered approach to expansion and, at each step of the way, review the results against standards, compare notes across the development organization, and course correct where necessary. This can avoid the accumulation of technical debt in the long run.”

“IT leaders should first ensure that additional resourcing and training are in place to both upskill existing teams and hire new talent as required,” said Quillin. “This includes adding observability tools that can identify new and existing forms of architectural and LLM drift. Finally, to always be ready for any new, rapidly emerging future requirements and opportunities, IT leaders should invest in best practices and tools that will proactively keep their app architectures clean and ready for the ‘next big thing.’”

How AI Could Decrease Technical Debt

On the other hand, some believe that if used properly, the contrary is true — that new AI technologies will actually reduce technical debt. “The rise of LLMs, frameworks, and APIs, when applied properly together, will actually reduce technical debt,” said Ashwin Rajeeva, CTO and co-founder of Acceldata. “Code discovery and reuse is more easily accomplished with LLMs.” He points to the ability of this technology to identify duplicative code and consolidate it into a more streamlined interface.

Automated documentation generation is another area where LLMs could help improve code discovery and reduce duplicative efforts. In addition, LLMs could also help reduce the prevalence of code that doesn’t follow company standards, said Rajeeva. “It is much easier for LLMs to understand the company standards by learning from good coding practices and applying it to newly written code.”

Others also see the power of using generative AI to enhance automation across the board. “AI tooling is probably going to help the most in automation, velocity to transform old codebases, and empowering non-coders to accelerate the adoption of new business processes into their workflows,” said Orlandini.

Final Thoughts

AI is a multifaceted subject, and its use could go one of two ways. On one hand, if not used correctly, generative AI could only add to the surmounting technical debt pile. To avoid this outcome, organizations will likely need guardrails on when and how to use LLMs, said Rajeeva. He also advocates “making it a best practice within the corporation with sufficient training for newer developers.”

Then, on the other hand, generative AI could be utilized to respond to technical debt. It has the awesome potential to unlock automation gains and enhance employee satisfaction and retention, especially within older industries that may be behind the technological curve, said Kohli. “With AI and now generative AI, intelligent automation can deliver on all those promises such that it is fast becoming a new pillar of enterprise architecture.”

So, take the new wave of AI with a grain of salt. It will open up immense solutions but will also come with a significant degree of technical debt. Thankfully, there are ways to manage debt caused by AI in a way that can benefit operations and push development objectives forward.

January 25, 2024
New Year, New ‘Tude

There are multiple types of debt in an organization. In IT alone, you can have technical, ops and (my favorite) attitude.

We work in a high-stress field with a lot of moving parts and an enterprise depending upon us to keep things running. Even if you are in the product development org at a software company, you literally make the product; people are more dependent on you doing it right most of the time and being super proactive when things go wrong.

All of this creates stress that we tend to either internalize or leave to find a new place to get our stress.

Internalizing makes for jaded IT folks. Or worse, cynical ones. Vacations are part of the salve to relieve that stress, and we just got a couple of weeks off. Or at least most of us did, some of you all are about to, and others will in February. But it is all the same.

To continue improving, we need to leave the baggage behind. “Blameless IT” improved how others judge devs and operators that make mistakes, but we’ve done little to improve when devs and operators see problems that they cannot fix and it festers.

When you return to work, take a deep breath, consider what you can do something about versus what you cannot. Then, take action on the things you can and ignore the things you cannot. Improvement is what matters; improving a specific broken process isn’t. Of course, it will eat at you if you pay a ton of attention to something you cannot change. I have always taught those working for me (and my kids, for that matter) to change what you can and not dwell on what you can’t—precisely because you can’t.

If the thing you can’t change is a show-stopper and makes your life miserable, then the thing you can change is your job. Start looking.

Between Agile, DevOps and new deployment methods, we have come a long way in technology, but we also have created a ton of work for ourselves. This stuff doesn’t implement itself. Focus on that work and on how you can make things better – for the org, for IT, for fellow developers or operators. And do that. Always improving and not dwelling will make life happier for you and generally more productive for your team.

I’m excited for a ton of things that 2024 is bringing, and a couple of things that aren’t necessarily ‘2024 new,’ but that I’m still figuring out. Let’s do new stuff with a smile, because we know we’re making things better/faster/more reliable. And implement that feature your users all want and your team hasn’t done yet, while you’re at it. Users are the reason we do this, in the end, be they internal or external.

We made it through 2023 with some great improvements—no doubt you did, too. Now, it’s off to 2024 to do more. Keep rolling out features, protecting the systems, deploying to new targets, etc. And forget about that stupid annoyance. (Until you can change it—then crush it like a bug). Meanwhile, thank you from all those that don’t realize they add to your stress every day.

January 10, 2024
But Are You a Platform?

Funny how our obsession with buzzwords sometimes shoots us in the foot. While I’ve discussed this for specific tech spaces, this time, I’m going to point out a general trend that we’re currently experiencing. I will note up front that neither you nor I can stop it—I’m just here to raise awareness.

Currently, everything IT is a platform, and it smacks of resume padding. Your keycard system is probably “A platform to secure physical and virtual infrastructure across the entire organization, specializing in end-to-end security.” Even though you’re just going to buy it to secure (and, by extension, open) doors.

All joking aside, most of the time, your vendor is going to offer much more than you need, increasingly due to M&A activity, but it’s true for internally developed stuff, too. This works really well for them, because your current need is a gateway drug, and then they can cross-sell you the other 50 functionalities. It works okay for some customers who are looking to simplify their infrastructure and others who are looking for the proverbial “one throat to choke,” but honestly, for most of us, it is a waste of resources across the board.

Ask yourself, “What percentage of this product do I need?” then ask yourself, “How much of this functionality is already implemented in the 15 other platform solutions we have that now overlap with this vendor?” And finally, ask, “What is the possible unexpected overhead of implementing a platform where a tool will do?”

The goal for us is not to provide vendors with recurring revenue; it is to get the job done and keep complexity from forcing us to hire people to manage more of the menagerie.

I have mentioned the current trend of consolidation before—the more consolidation happens, the more your existing vendors will try to solve more of your problems. Sometimes, they will do an excellent job of it and shouldn’t be dismissed just because they’ve become a platform. Other times, it is purely a marketing ploy, and you should do an evaluation of both the market and competitors before jumping on board.

For the most part, I’m preaching to the choir. Most of you have the “What do we need? What do we have available but is underutilized? What do we need that’s new?” system down, but some of you don’t. Think of a toolshed where you have 15 digging tools, but you only dig post holes. You need a post-hole digger in your shed and probably a shovel (because everyone uses a shovel now and then). The rest? In the case of physical tools, you can keep them if you have spare room in the shed but don’t need them. In the case of IT infrastructure, you’re paying for them, paying to maintain them, and/or paying for resources to keep them running. So don’t. Buy what you need, and limit broad solutions to a couple of valued partners or spaces where the intersection of previously separate tech spaces provides value (some app security, for example, is super well-served by the platforming trend because all the formerly separate steps now share data for a more broad view of a given apps’ security that is deeper than current SIEM capabilities).

And keep rocking it. Imagine if you could get down to one of each tool you need and every team was an expert and each of those tools was best of breed … Okay, enough epic fantasy. Just try to minimize what you have to work on/with, and keep serving up the organization professionally to the world and employees.

October 11, 2023
Technical Debt: Don’t Buy Buzzwords

It is absolutely true that the only constant in IT is change. No sooner do a mass of us get the ‘next big new shiny thing’ implemented than people start telling us how much we need the next ‘next big new shiny thing.’ Often, that next big new shiny thing is actually helpful, and we feel (sooner or later) compelled to implement it. Sometimes, the next big new shiny thing is not very useful at all, and yet far too many of us implement it anyway.

Not naming names here, but you’ve all got technical debt that I would classify as “bling” or “keeping up with the competition” functionality. It’s there; you’re paying for it and/or to maintain it, and yet it provides little value. Sometimes, you’re even forbidden from removing it because it is some exec’s pet project or some corner department is actually using it. You’d just have to replace this version of the next big new shiny thing with a different version.

And I’m here to tell you: Stop. When considering new technology, the very first thing to do is build a list of the problems you are trying to solve and then describe scenarios where those problems crop up. Then compare the next big new shiny thing–be it features in an existing product or a new space/product completely–to how it solves real needs. People are the absolute worst about this with open source. “It’s free, so let’s try it out.” It. Is. Never. Free. And it is a rare team that tries it out and then removes it if it doesn’t work. Most of the time, they’re off to the next project too quickly to do that level of cleanup.

SBOMs and SCA are ‘next big new shiny things’ of the recent past that actually help you clean up test/pilot garbage from the latest bout of next big new shiny things. So are containers since you can spin them up, play with the toys and then take them down. I am surprised at the number of teams out there who do not do this, but that’s changing pretty quickly. Use them; it’ll save a ton of cleanup time and a lot of technical debt.

But most importantly, only buy what you absolutely need. We’ve been dropping an ever-increasing number of tools into the development environment for a decade, and those tools do not (yet) maintain themselves. There is infrastructure cost, possibly licensing cost and staff time. Don’t waste it; focus on what is literally most important.

And that might mean putting your foot down. We in IT have been loathe to do that since Agile started, but telling a C-level that they don’t need an AI to create new drink recipes (or whatever the current generative AI boardroom craze is) is in the best interests of the company. Focus on systems that help do the work or that protect what does do the work.

You’re still rocking it, so if rocking the boat a bit in the process causes problems, you might just be at the wrong company or in the wrong role. Achieve, but achieve meaningful results.

September 27, 2023
AI: A Game-Changer for SRE Work-Sharing and Technical Debt

In my recent blog, Revolutionizing the Nine Pillars of SRE with AI-Engineered Tools, I indicated that AI can assist work-sharing and incremental technical debt. For example, AI can help analyze code and infrastructure changes to assess their impact on technical debt. In this blog, I explain in more detail how AI-engineered tools can be used to improve the SRE work-sharing and technical debt pillar of practice.

AI Use Cases for SRE Work-Sharing and Technical Debt

Sharing Wisdom of Production: AI tools like Splunk and Datadog can analyze log files and telemetry data to uncover hidden patterns and trends, improving system understanding and enabling knowledge sharing. Challenges include integrating these tools into existing infrastructure and managing the vast amount of data generated. Clear data governance policies and proper tool configuration can help overcome these.

Collaboration with Development, QA and Security: AI-powered platforms like GitHub and GitLab can automate code reviews, identify security risks and even suggest code improvements, fostering collaboration. Integrating these tools into existing workflows can be challenging, but this can be mitigated by providing proper training and phased integration.

Backlog Grooming: Tools like Atlassian’s Jira, powered by machine learning algorithms, can prioritize the backlog by predicting issue impact, allowing SREs to focus on higher-priority tasks. The challenge is to integrate these tools seamlessly into the workflow, which can be achieved through comprehensive onboarding and training sessions.

Participation in Sprints: AI tools like Tara.AI can predict sprint velocity based on historical data, helping teams adjust their workload. The challenge lies in adjusting to the predictions and estimates provided by these tools, which can be managed by slowly incorporating AI guidance into sprint planning and review processes.

Participation in Testing: AI in testing tools like Testim.io and mabl can automate test cases and intelligently adapt to changes in the system, reducing the time and effort required for manual testing. The challenge lies in selecting and configuring the right AI testing tools for your specific context, which can be mitigated by conducting a thorough needs and gap analysis before tool selection.

Participation in Automation of Environments: AI tools like Hashicorp Terraform can help automate infrastructure provisioning and management. The main challenge is understanding and defining infrastructure-as-code (IaC), which can be overcome with training and leveraging IaC best practices.

Participation in Release Management: Tools like UrbanCode Velocity harness AI to provide insights into the release management process, enabling better decision-making. However, integrating these insights into existing release management workflows can be challenging. Slow, phased integration of AI tooling supported by training can help.

Sharing On-Call Duties and Incident Response Activities: AI tools like PagerDuty and xMatters use machine learning to automate incident response, helping reduce on-call burnout. The challenge is ensuring the AI doesn’t overlook critical incidents due to false positives or negatives, which can be mitigated by continuous tuning of the AI model and maintaining a human in the loop.

RoadMap for SREs to Transform Incident Response Using AI

Here is a practical roadmap an organization can follow to implement AI tools for SRE work-sharing and technical debt:

Identify the Need and Set Goals: The first step involves identifying the specific areas where AI tools could help in work-sharing and technical debt management. It’s essential to set clear, measurable goals for what you hope to achieve with AI.

Conduct a Gap Analysis: Understand your current tooling and processes and identify where the gaps are. Evaluate how AI could fill those gaps and improve your processes.

Select the Right Tools: Some tools that could be helpful include Splunk or Datadog for data analysis and sharing production wisdom, Jira for backlog grooming, Tara.AI for sprint planning, Testim.io or mabl for automated testing, Terraform for automating infrastructure and PagerDuty or xMatters for incident response. However, the exact selection will depend on the specific needs and context of your organization.

Phased Implementation: Start by introducing AI tools in the areas where they can provide the most immediate benefit. A phased approach helps manage the transition and reduces the risk of disruption.

Training and Support: Ensure your team is trained in using the new tools and understands how to interpret the insights they provide. This may involve external training or bringing in experts to provide in-house training and support.

Measure and Adjust: Continuously evaluate the effectiveness of the AI tools against your initial goals. It’s likely that you’ll need to make adjustments along the way, either to the tools themselves or to the way you’re using them.

Iterate and Expand: Once the tools have proven their effectiveness in one area, you can start to expand their use into other areas of work sharing and technical debt management.

Summary

Transforming the way we approach work-sharing and technical debt management within SRE practices through AI can propel teams toward increased efficiency and better outcomes. By leveraging intelligent tools, we can automate routine tasks, predict and manage technical debt more effectively, enhance our decision-making in sprints and deployments and create a more collaborative and insight-driven environment. AI tools such as Splunk, Datadog, Jira, Tara.AI, Testim.io, mabl, Terraform, PagerDuty and xMatters provide an arsenal of capabilities that can revolutionize how we handle everything from backlog grooming to sharing production wisdom and incident response.

Embracing this transformation requires a well-thought-out strategy. A roadmap that begins with identifying needs and goals, conducting a gap analysis, selecting the right tools, implementing in phases, training teams and continuously measuring and adjusting for effectiveness can guide the journey. It’s about ensuring that AI serves its purpose in empowering SRE teams, fostering a culture of shared responsibility and knowledge and, ultimately, improving the reliability and efficiency of our systems. This evolution holds immense potential, and we are just starting to tap into it.

August 22, 2023