devopsy.kinsta.cloud

Tag: quality

The Hidden Cost of AI Code: Keeping Quality Up With Production
AI maturity is fundamentally about expanding the delegation boundary. You start by letting AI assist with code completion, then with features. Eventually, agents write pull requests from requirements with minimal human involvement. Each step hands more responsibility to machines.

But here’s what people skip over: Tests are the primary mechanism for making that delegation safe. You can’t let agents operate autonomously if you can’t verify what they produce. Low test coverage is the single biggest barrier to advancing along the AI maturity curve and most organizations haven’t come to terms with that. They want the AI productivity gains without doing the unglamorous work of building test infrastructure to make those gains trustworthy.

Coding assistants make delivery so fast that most QA organizations are underwater. Teams face a choice nobody wants to make: Attempt ten times the testing work, or test selectively and accept less certainty about whether what’s running is safe. Neither option is sustainable.

It’s time to shift the conversation. The question isn’t how fast teams can ship. It’s whether they can prove, with evidence, that what they shipped works.

What’s Changed

AI accelerated code delivery, but the bigger change is in who’s building software, how much is being built and what it takes to trust it. 

People without training in architectures, APIs, or security can now generate functioning applications. That’s a real unlock – and a risk multiplier. A junior developer hard-coding plain-text credentials used to be an isolated incident you’d catch in code review. Now, with AI assistants putting production-grade tools in everyone’s hands, that rookie mistake can show up across dozens of repos before anyone notices.

Throwing more people at the problem is not realistic. Companies have downsized teams. Even if fully staffed, QA has been defined by code-level inspection and automation. Inspection is a tick-box verification of functions, integrations and outputs under predefined conditions. Automation mechanizes that inspection. Neither was designed to keep pace with AI-generated code arriving at this volume.

The other response is to fight AI-generated code with AI-generated tests. Sounds logical. In practice, it creates a different problem because nobody is sure which of those tests matter, which are redundant and what’s missing. Unthoughtful AI-generated tests are just noise.

So if you can’t hire your way out and you can’t generate your way out, what’s left? It’s a governance and intelligence problem. When an agent generates fifty variations of a login test, who decides whether that’s useful or busy work? When AI creates a test suite, who ensures it traces back to requirements? Enterprise teams are demanding answers. They want test cases generated from issue trackers, linked back to originating requirements, organized and traceable, all triggered from within existing workflows. 

The organizations that build this layer of trust will scale AI-driven development. The rest will find out the hard way what happens when you ship fast without proof.

From Instructions to Outcomes

QA has historically been about verifying what an application is, not what it does. That approach worked when humans wrote code and applications changed on schedule. It falls apart when AI regenerates parts of the codebase continuously. 

The gap between what someone intended the software to do and what it does is where releases go sideways. That gap is widening. Seven of 10 software executives are concerned that application quality is suffering as AI speeds code development, new research shows and just as many are concerned about the impact going forward. 

This means the focus on quality needs to catch up to the focus on AI development speed. Product managers and engineers define the requirements. QA must validate that requirements are met. But most QA practices are still oriented around inspecting code artifacts rather than confirming outcomes. The shift needs to be from “does this component work as coded?” to “does this application behave the way the business expects it to?” We call this “Application Integrity,” continuous, measurable assurance that software works as intended with the governance to operate at AI speed and scale. 

One pain point I hear from QA teams is about the manual creation of tests, especially for each variation of applications. That’s where agents should be doing the heavy lifting. But AI-driven QA will only help if applied thoughtfully. You need agents that can test based on user behavior, adapt as interfaces change, and prioritize what’s most likely to break rather than what’s easiest to test. The difference between useful AI-driven testing and expensive noise comes down to whether the tools are built with that kind of judgment baked in, or whether they’re just fast.

Maintaining quality when everything is moving this fast requires application integrity, meaning ongoing evidence that the software behaves as intended. Continuous assurance. Can you show, at any given moment, that your software is doing what you said it would? To do that, teams need to get a lot clearer about what they’re measuring. Some of the metrics teams track today are still the right ones. Others need to be added.
- Deployment frequency and lead time for changes aren’t going away. AI should make shipping faster and release cycles shorter. If your quality practice is slowing them down, fix it. Those are throughput metrics. On their own, they don’t tell you whether the things you’re shipping actually work.
- That’s where change failure rate comes in.  This is becoming the most important metric. What percentage of deployments cause incidents, rollbacks, or hotfixes? When code volume goes up but verification doesn’t keep pace, this number climbs. Driving this percentage down is the core challenge for quality teams and probably the one with the most direct financial impact.
- Mean time to resolution deserves more attention. If deployments are landing twice as often but recovery time hasn’t improved, your exposure window just doubled. 
- Invest in functional coverage: What percentage of your app’s functionality and API surface is covered by tests. And traceability coverage: what percentage of tests and API contracts link back to a requirement or spec? This separates teams who can demonstrate their software works from teams who are pretty sure it does.
A New Standard

Code is being delivered by more people, faster than ever. Now, the new standard is proving the applications work. 
June 8, 2026
Best of 2025: Survey: AI Tools are Increasing Amount of Bad Code Needing to be Fixed

A survey of 500 software engineering leaders and practitioners published this week finds that while nearly all (95%+) believe artificial intelligence tools can reduce burnout well over half (59%) report AI tools at least half the time are creating deployment errors in their code.

As a result, more than two-thirds (67%) said they spend more time debugging AI-generated code, with 68% also noting they are now spending more time resolving AI-related security vulnerabilities. A full 92% of respondents also noted that AI tools are increasing the blast radius of the amount of bad code that needs to be debugged.

Harness Field CTO Nick Durkin said that while AI tools can generate more code faster, most of the tools being used have not been specifically trained using code samples from the production environment where code is destined to run. In the absence of having that context, it should not come as much of a surprise that the code being generated by AI tools doesn’t work in a production environment, he added.

Making matters more challenging, developers then need to spend time debugging code that they were not involved in creating, noted Durkin. That’s generally more difficult than debugging code those developers might otherwise have written themselves, he said.

Many of these issues would become more manageable if organizations had a formal policy concerning which AI tools are employed, said Durkin. The survey finds less than half of developers (48%) are using AI tools that have been approved by their organization. Well over half of respondents (58%) said their organization does not provide clear guidance on which use cases represent the lowest risk for AI adoption.

A full 60% of respondents said their organization lacks any formal processes for assessing AI-generated code vulnerabilities or errors and an equal percentage said the effectiveness of AI coding tools is not being evaluated.

On the plus side, 50% of the engineering leaders surveyed said their organization plans to invest in AI for continuous integration/continuous delivery (CI/CD), with 48% planning to prioritize performance optimization. A total of 42% also plan to use AI to improve security and compliance.

The only way to ensure that organizations are realizing the benefits of AI is to adopt platforms that provide a comprehensive framework for not only creating code using AI but also providing access to AI agents to review that code before it is deployed in a production environment, said Durkin. The issue with using standalone AI coding tools comes down to the simple fact that those tools have no understanding of the production environment where that code is expected to run, he added.

Additionally, many organizations have not yet considered the total impact these tools are having on infrastructure costs or their future sustainability goals, Durkin noted.

Harness’s research notes that if 78% of developers are spending at least 30% of their time on manual, repetitive tasks such as debugging code, that represents $8 million in lost productivity annually for organizations that employ 250 application developers.

There are, as always, concerns to what degree AI might replace developers, with 88+% of respondents still trying to determine the ultimate impact. Many organizations, however, are applying AI to the wrong tasks, said Durkin, Many developers actually enjoy writing code. It’s all the other menial tasks, such as creating tests and debugging code that they would prefer to automate, he said.

Regardless of how anyone may feel about AI, the core technologies are only going to continue to become more capable. The issue now is determining how to best apply it in a way that ultimately leads to more applications being deployed faster.

December 30, 2025
Breaking Down Testing Barriers: How Observability is Bridging Dev and Ops

The traditional divide between development and operations rapidly dissolves as modern observability practices transform software testing. According to Bryan Cole, Director of Customer Engineering at Tricentis, this evolution is crucial for delivering business value.

“Ops teams are paid to keep the lights on and deliver business value,” Cole explains. “What Dynatrace is doing, moving into the dev space, is providing context to the ops teams on what exactly is happening, and more relevantly, how things are testing out.”

This context is critical. “Nothing’s worse for an IT ops team than to be given a software build that you know nothing about to deploy Friday night,” Cole notes. “Hope is a terrible strategy.”

The integration between Tricentis NeoLoad and Dynatrace exemplifies how observability is transforming testing practices. Cole highlights how Davis AI can parse through billions of data points across complex software systems, providing focused guidance to operations teams. “That insight provides focus and guidance to the ops teams and helps them manage their time, because the ops are getting so big and complex, you can’t understand everything, let alone look at everything.”

The results are significant. Cole shares examples of customers seeing dramatic improvements: “A large retail site saw an 80-90% reduction in time to analyze performance test results. A healthcare provider experienced a 90% reduction in problem identification time and a 95% reduction in the mean time to resolution.”

Preventive operations are becoming increasingly important. “If you don’t know the symptoms your application is going to exhibit when it starts to fail, then you have a hard time understanding what should trigger you to take that preventive action,” Cole explains.

The challenge lies in changing perceptions about continuous testing. “The message of implementing quality gates and additional checks doesn’t resonate well with delivery faster,” Cole notes. “But many people are guilty looking at that narrow slice of time of just the delivery of the software in a production environment, not the consequences of it arriving in production and not working properly.”

The future of testing lies in accessibility and integration. Cole emphasizes making data accessible across platforms: “We want to make your data accessible to you, to use the way you want, in the platform where you’re comfortable and familiar.”

Mitch Ashley, VP and Practice Lead, DevOps and Application Development, The Futurum Group, believes, “Software is no longer static. It is constantly changing, closer to a fluid state than a solid. We need the production telemetry pipeline to flow upstream and down between developers, QA and Ops. No one vendor can do it alone. Partnerships such as those between Tricentis NeoLoad and Dynatrace enable customers to benefit from two-way telemetry easily.”

This integration enables what Cole calls “the continuous quality practice,” where “the volume of changes goes up, but the size of the changes goes down.” Combined with AI-powered observability, this targeted approach transforms how teams identify and resolve issues before they impact production.

February 6, 2025
Survey: AI Tools are Increasing Amount of Bad Code Needing to be Fixed

A survey of 500 software engineering leaders and practitioners published this week finds that while nearly all (95%+) believe artificial intelligence tools can reduce burnout well over half (59%) report AI tools at least half the time are creating deployment errors in their code.

As a result, more than two-thirds (67%) said they spend more time debugging AI-generated code, with 68% also noting they are now spending more time resolving AI-related security vulnerabilities. A full 92% of respondents also noted that AI tools are increasing the blast radius of the amount of bad code that needs to be debugged.

Harness Field CTO Nick Durkin said that while AI tools can generate more code faster, most of the tools being used have not been specifically trained using code samples from the production environment where code is destined to run. In the absence of having that context, it should not come as much of a surprise that the code being generated by AI tools doesn’t work in a production environment, he added.

Making matters more challenging, developers then need to spend time debugging code that they were not involved in creating, noted Durkin. That’s generally more difficult than debugging code those developers might otherwise have written themselves, he said.

Many of these issues would become more manageable if organizations had a formal policy concerning which AI tools are employed, said Durkin. The survey finds less than half of developers (48%) are using AI tools that have been approved by their organization. Well over half of respondents (58%) said their organization does not provide clear guidance on which use cases represent the lowest risk for AI adoption.

A full 60% of respondents said their organization lacks any formal processes for assessing AI-generated code vulnerabilities or errors and an equal percentage said the effectiveness of AI coding tools is not being evaluated.

On the plus side, 50% of the engineering leaders surveyed said their organization plans to invest in AI for continuous integration/continuous delivery (CI/CD), with 48% planning to prioritize performance optimization. A total of 42% also plan to use AI to improve security and compliance.

The only way to ensure that organizations are realizing the benefits of AI is to adopt platforms that provide a comprehensive framework for not only creating code using AI but also providing access to AI agents to review that code before it is deployed in a production environment, said Durkin. The issue with using standalone AI coding tools comes down to the simple fact that those tools have no understanding of the production environment where that code is expected to run, he added.

Additionally, many organizations have not yet considered the total impact these tools are having on infrastructure costs or their future sustainability goals, Durkin noted.

Harness’s research notes that if 78% of developers are spending at least 30% of their time on manual, repetitive tasks such as debugging code, that represents $8 million in lost productivity annually for organizations that employ 250 application developers.

There are, as always, concerns to what degree AI might replace developers, with 88+% of respondents still trying to determine the ultimate impact. Many organizations, however, are applying AI to the wrong tasks, said Durkin, Many developers actually enjoy writing code. It’s all the other menial tasks, such as creating tests and debugging code that they would prefer to automate, he said.

Regardless of how anyone may feel about AI, the core technologies are only going to continue to become more capable. The issue now is determining how to best apply it in a way that ultimately leads to more applications being deployed faster.

January 10, 2025
Proactive Dependency Management: Reducing Risk and Improving Software Quality
In modern software development, open-source components or dependencies are fundamental to rapid innovation. Developers typically choose these components based on personal experience, research or guidance from technical leads. However, once integrated, these dependencies often recede into the background.

Over time, these dependencies can accumulate technical debt, turning into neglected assets that may introduce bugs or security vulnerabilities.

Proactive dependency management is crucial for maintaining secure, high-quality software. I recently discussed this topic at All Day DevOps (ADDO) 2024. Read further to learn more about common pitfalls, key statistics and best practices to stay ahead of potential issues.

The Hidden Cost of Neglected Dependencies

While software dependencies play a vital role in modern applications, they can also introduce significant risks if not managed proactively.

Here are some key statistics that illustrate the impact of neglecting dependency management:
- Each application has six to 10 direct dependencies, which balloon to 180 total dependencies when accounting for transitive dependencies.
- 80% of application dependencies go un-upgraded for over a year, exposing software to bugs and security vulnerabilities.
- Each project releases an average of 15 new versions per year, and developers spend about two hours on each dependency version upgrade.
- This adds up to 300 hours per application per year — a significant investment of time that could be reduced with better automation and management
Vulnerabilities Lurking in Outdated Dependencies

One of the most critical issues with unaddressed dependencies is their potential to introduce security vulnerabilities.

Even when fixes are available, many organizations fail to update their dependencies on time. A full 96% of vulnerable releases downloaded had a fixed version available.

The notorious Log4Shell vulnerability of the Log4j component is a prime example. Despite the well-publicized risk and the availability of patches, 13% of Log4j downloads in 2024 are still vulnerable versions.

This statistic underscores the importance of staying vigilant about dependency updates, especially with critical components that could expose applications to significant risks.

Automating Dependency Management for Reliability and Security

Proactive dependency management can be a daunting task, but automation offers a solution. Automated systems can monitor dependencies and flag updates, helping teams stay ahead of vulnerabilities and ensure compatibility.

However, not all automation is equal. To be effective, automated solutions must take into account API compatibility to prevent broken builds and reduce upgrade time. Without this, a typical two-hour upgrade can balloon, disrupting other priority work.

Another valuable input to automate solutions is the popularity of open-source components. Popular components exhibit 63% more identified vulnerabilities but also address 54% more and resolve them 32% faster than less-known components. While popular projects undergo more rigorous scrutiny, they typically make updates more accessible, which becomes incredibly powerful when those updates can be integrated through automation.

The Impact of Transitive Dependencies

One of the most challenging aspects of dependency management is the ripple effect that changes made to direct dependencies can have on transitive dependencies.

Managing only 10 direct dependencies can impact up to 170 transitive ones, forming a complex web of interconnected components that require vigilant monitoring and updates. Overlooking this intricacy can lead to significant issues, such as build failures, bugs and security vulnerabilities.

For this reason, automating dependency management is essential for reducing the risk of unexpected failures. Automated tools that provide visibility into the full dependency graph enable developers to assess the impact of updates and take action before problems arise.

Best Practices for Proactive Dependency Management

A key takeaway from the presentation is that proactive dependency management is not just about staying on top of updates — it is about creating a strategy that minimizes risk while improving software quality.

Here are a few best practices that can help:
- Automate dependency updates: Leverage automation tools to track and update dependencies across your applications. By automating routine updates, you can reduce the manual workload on your team and minimize the risk of vulnerabilities.
- Monitor for vulnerabilities: Regularly scan your dependencies for known vulnerabilities and ensure that you are updating to secure versions as soon as they become available. This is especially important for popular open-source components, which are frequently targeted by attackers.
- Assess API compatibility: Ensure that your automation tools account for API compatibility to avoid broken builds and wasted time. This will help keep your development process smooth and efficient.
- Focus on transitive dependencies: Don’t just manage your direct dependencies — make sure to monitor the transitive dependencies that they introduce into your These hidden dependencies can often be the source of critical issues if left unmanaged.
Staying Ahead of Technical Debt and Security Risks

Ignoring your dependencies is not an option. The statistics presented show the consequences of failing to manage dependencies proactively. By adopting best practices and embracing automation, you can reduce technical debt, improve software quality and avoid costly security breaches.

Managing dependencies isn’t just best practice, it is an essential ongoing process. Implement these strategies in your projects to stay ahead of potential issues and ensure your software remains reliable, secure and up to date.
November 13, 2024
What Does Build Quality Look Like in Software Design?
With rapid developments in artificial intelligence (AI), mobile applications and modern software in the last year, the enterprise software market is expected to grow exponentially. Market revenue is projected to reach $292 billion in 2024, and developers are constantly implementing innovative measures into the software development life cycle (SDLC) to ensure a seamless, high-quality creation process. One of the best ways to achieve this is by leveraging modern build health tools to monitor a multi-faceted project’s overall quality and performance.

Build health tools automate the creation of executable applications from source code and encompass several areas to ensure the stability and health of the software build process. From automated testing to security scanning, developers rely on these tools and processes when defining a modern approach to software development. Before diving into the intricacies of build health quality, it is necessary to grasp the complete significance of the SDLC when attaining the final product in a global marketplace.

The Significance of the SDLC

The SDLC is the process in which developers plan, write, modify and maintain software. This methodology applies to software developed for computers, mobile phone applications, video games and other systems. Adhering to this process allows developers to reach the optimal outcome for the final product. The life cycle includes everything from the planning and brainstorming stage to setting up maintenance and operations after the product has launched. It ensures success by providing a more structured approach to building and developing software. This becomes increasingly important as the software market’s value continues to rise.

Since 2020, the demand for software engineers has doubled. Because it’s essential for business leaders and developers to keep up with the latest trends and innovations, the need for software knowledge will continue to grow exponentially. Currently, software is typically delivered in one of two ways:
- On-premises. End users install and run the software themselves on the desired device. This serves as a more secure delivery method for highly-regulated industries.
- Software-as-a-service (SaaS). This cloud-based method allows end users to access the software online rather than through a traditional download/upload. This enables providers to offer a subscription-based/pay-as-you-go structure for users. The provider is responsible for the hosting and maintenance.
Developers rely heavily on build health tools to ensure a smoothly running life cycle. Build health tools play a huge role in determining a seamless development process.

The Role of Build Health Tools in Modern Software Development

Build health tools essentially fill the gaps in the development process and enhance the software build quality. These tools deliver automated testing methods by creating multiple testing scenarios to ensure full functionality. They help developers see the different data points necessary for garnering a 360-degree view of the software and its status.

The tools also impact resource utilization and response times. Using key performance indicators (KPIs), they monitor the build quality while enhancing build production to ensure it is better than what was previously developed. In that way, it is constantly growing and improving to guarantee the project reaches a smooth and seamless completion for the developer. From there, the tools can create multiple datasets in less time, which significantly reduces overall software development time.

Common Challenges of Build Health Tools in Software Design Projects

There are several challenges with using build health tools. The most common obstacle is that automated tools and testing lack the human touch that comes from developers and their manual work. Although build health tools can perform simultaneous tasks quicker than a more manual approach, they overlook the full spectrum of available resources to improve the build. These tools are so focused on functionality that they fail to incorporate the prior research and available resources that often serve as key building blocks for a successful development project. For this reason, it is important to have developers working in conjunction with build health tools to provide a balance between manual and automated software testing.

Developers can leverage build health tools to seek out the KPIs that determine the software’s quality at a given stage of the SDLC, from initial development and quality assurance (QA) to the final phases of staging and production. Some of these KPIs include:
- Exceptions. Make sure there are no new exceptions from the previous build versus the production stage.
- Test coverage. Ensure that adequate testing is performed on the current build by defining what application programming interfaces (APIs) need to be included for certification, along with the number of executions required.
- Automated data validation. Make sure data is accurate and useful while remaining enforceable by the designated standards.
- API testing. Test the overall interface for smooth performance with trustworthy data to advance the program/project. It is also important to compare performances during different SDLC stages to confirm the status of the API, dependency, database, and functionality.
- Memory/CPU usage. Measure the overall usage of the central processing unit (CPU) and its memory to ensure stability and efficiency.
- Requests per minute. Evaluate the rate at which the software can receive and execute certain requests delivered by the end user to ensure optimal efficiency.
- SQL coding standards. Enforce standards and limitations for the overall coding within the interface by measuring quality, security, efficiency, and sustainability.
While these are only a few of the KPIs detectable with the aid of build health tools, it is essential to note how these tools can benefit a successful software release. Developers can compare the differences between current and prior builds throughout the SDLC to fully gauge any changes, developments, improvements, or downgrades during production to a successful launch.

Current Events and the Future of Build Health Quality for Developers

There are many examples of current organizations effectively leveraging available build health tools for project success. Software developers will continue to adopt new and innovative methods using AI and automation capabilities to ensure more efficient testing and higher build quality.

The future is bright for contemporary software engineering. As unexpected as the AI boom of 2023 was for the tech world, the future of modern software development may hold even more surprising developments. Build health tools are expanding and perfecting the creation process for enterprises and developers worldwide. As build quality improves on the back end, the customer experience for global industries will improve on the front end. The potential for software enhancements and innovations in this way is what makes the journey exciting.
February 22, 2024
Improve Efficiency With Smaller Code Reviews
In software development, code review plays a significant role in helping detect issues in your application’s source code early and promote collaboration among your team members. Breaking down code reviews into more manageable portions can yield great benefits. Review cycles are shorter and more convenient to comprehend when they are smaller. In other words, smaller reviews are easier to comprehend and provide feedback cycles. As a result, we can reduce development, deployment and debugging times while reducing the scope of change in each iteration.

This article presents a discussion on code reviews, why they are important, their benefits and why smaller code review cycles are more effective than larger ones.

What is Code Review?

Code review is defined as a software development process which involves reviewing code written by other developers. As part of the development process, it aims to improve code quality, ensure compliance with standards and identify and fix problems early on. Code reviews play a role in software development as they help catch errors before they turn into significant issues.

Peer reviews allow developers to detect errors or defects in each other’s code. By reviewing each other’s code, developers are able to identify bugs and flaws in the code. It helps deliver code that is secure and sustainable, adheres to standards and detects and resolves problems early in the development life cycle.

This can either be a formal process, where code is systematically evaluated according to a checklist or set of guidelines, or an informal process, where code is reviewed ad hoc. Code review can be performed by the same developers working on the code or by other developers with expertise in the relevant domain or technology.

By engaging in the exchange of feedback and actively participating in code reviews, we can ensure that the code is well-organized, easy to comprehend, adapt and extend over time. To be effective, it’s essential that everyone on the team is on the same page when it comes to their expectations and goals regarding these reviews.

Here are the key benefits of code review at a glance:
- Identify bugs and issues
- Improve code quality, consistency and maintainability
- Enforce coding standards
- Foster team collaboration
- Promote knowledge sharing
Best Practices of Code Review

Here are some of the best practices you should follow to improve your code review process:
- Review small units of code
- Set clear objectives and guidelines
- Involve multiple reviewers
- Provide constructive and respectful feedback
- Use code review tools
- Prioritize high-risk areas
- Balance attention to detail with time efficiency
- Encourage collaboration and learning
- Set realistic deadlines
- Follow up and close the loop
Automating the Code Review Process

Automated code reviews can help you enhance collaboration amongst your team and reduce the time spent in each cycle. Your developers can improve code quality and maximize productivity by automating the development workflow.

Here are the key benefits of automating the code review process:
- Enhances code quality
- Improves productivity
- Enhances code quality
- Improves collaboration
- Fosters learning opportunities
The Problem With Large Code Reviews

The traditional approach to code review often involves reviewing large chunks of code in a single session. Although this approach is initially efficient, it can lead to multiple problems. You will need more effort and spend more time reviewing large chunks of code. If code reviews are extensive and take longer to complete, it can often result in decreased attention to detail. Due to this, the likelihood of missing crucial errors and reviewer fatigue will increase, resulting in delays in feedback and completing the code review process.

The Benefits of Smaller Code Reviews

When the code review cycle is small, it becomes easier to track deployment changes easily. Smaller code reviews continuously improve development, deployment and debugging times and reduce the radius of a change set. As a result, it promotes faster deployment and helps identify issues easily and early in the software development life cycle.

Once a code review cycle is complete, the code goes through many steps before it finally reaches the production phase. Monitoring the application’s performance and detecting bugs (if any) is much easier when you make only minimal changes to your code. When the changes are small, it is easier to track the application’s behavior and detect bugs.

Here are some benefits of adopting smaller code reviews:

Focused Review

Reviewing smaller chunks of code enables you to analyze the code in a more focused and in-depth manner. This helps reviewers focus on specific functionality or components, leading to more accurate feedback.

Reduced Cognitive Load

Limiting the amount of code to review reduces cognitive overload for both developers and reviewers. Smaller code review cycles can be more efficient because they break down the review process into more manageable chunks. As a result, you’ll be able to evaluate code quality in a much more effective and efficient manner and identify potential issues or improvements more easily.

Faster Feedback Loop

Smaller code reviews can significantly speed up the feedback loop in the development process. Hence, feedback cycles will be faster, i.e., you will be able to receive feedback faster, resulting in quicker iterations and faster software delivery. The smaller the review, the easier it is to digest and the faster the feedback cycle.

Enhanced Collaboration

Smaller code reviews allow for more focused discussions and collaboration between reviewers and developers. Most importantly, since you can handle concerns effectively and efficiently, you can engage in more productive discussions. The efficiency and effectiveness of code reviews improves when the expectations of all stakeholders are aligned.

Measuring Success

To measure the success of smaller code reviews, consider the metrics and approaches given below:

Defect Detection Rate

This is a measure of the number of defects or issues identified during a code review cycle relative to the number of defects found later in testing or in the production environment. Note that a higher defect detection rate indicates you’re able to detect defects effectively.

Review Efficiency

Calculate the time it takes to review a chunk of code to understand the pace and efficiency of the code review process.

Feedback Quality

Assess the quality of the feedback provided during reviews, such as checking if the feedback is constructive, clear and actionable. Note that if the feedback is of high quality, it would typically yield better code improvements and learning opportunities for your team.

Cycle Time

The duration of the code review process is an indicator of efficiency. A shorter cycle time suggests an efficient review process, but quality should not be compromised.

Automated Checks Integration

Evaluate how effectively automated tools are integrated into the code review process to detect common issues and reduce the manual effort required in reviews.

Knowledge Sharing

Assess how well the code review process facilitates knowledge sharing and cross-team learning. This can be observed through the diversity of reviewers and the nature of the discussions in the review process.

Bug Rates Post-Release

Monitor the incidence of bugs or issues in the code post-release. A decrease in this rate can indicate that the code review process is effectively catching and addressing potential problems before release.

Conclusion

Code review helps produce code that is highly performant, scalable, secure and compliant with the recommended standards. Code reviews can make the codebase easily understood, flexible and scalable. A collaborative approach to code review can yield several benefits, such as promoting a culture of continuous development and learning, maintaining an organized codebase with clean code and clear structure, helping detect and resolve issues quickly, etc.
February 13, 2024
Does Using AI Assistants Lead to Lower Code Quality?

The emergence of generative AI has permanently altered how code gets written — and due to the massive productivity boost, there’s no going back. According to GitHub, 92% of developers said they use AI tools, and developers complete tasks 55% quicker using Copilot, the popular LLM-based code generator.

Yet, new research has potentially upset the idea that AI assistants like Copilot are positively advancing software development without consequences. The report, conducted by GitClear, analyzed over 150 million lines of code over the last four years and found an upsetting uptick in code churn and a decrease in code reuse since the dawn of LLM-driven software development.

So, this begs the question: Does using AI assistants actually lead to lower code quality? Well, the answer is complicated. “I wouldn’t say that the report proves that AI assistants are reducing code quality since our data is correlational,” said Bill Harding, programmer and CEO at Amplenote & GitClear, the organization behind the aforementioned research. “But I would say that there is a confluence of indicators that show code quality decline at scale, starting around 2022 and accelerating in 2023.”

AI-Generated Code Increases Overall Volume

One potential contributor to lower code quality is the sheer amount of code produced. Since LLMs automate code generation, production is accelerated across the board. Without guardrails, generative AI will likely contribute to more technical debt.

“AI-enabled code development is going to have a major effect on coding, exponentially increasing the volume and velocity of code delivery, much of which we anticipate will be of lower quality and more bloated,” said Mehran Farimani, CEO, RapidFort.

In addition to quality concerns, there are also governance and security concerns to consider from the use of AI and surges of new code production. Some anticipate this will overwhelm security teams, which have already been struggling to keep pace. “Increased developer velocity is going to put security teams under even more pressure,” said Farimani. “The volume of scans will increase, the range of vulnerabilities that need to be addressed will increase and the patches that need to be applied will increase.”

Blindly Relying on AI Outputs

According to GitHub, developers accept code completions from Copilot 30% of the time. This percentage is likely to keep increasing as the LLM is trained on internal style and comfort with the tool increases. However, there is still the risk of developers blindly relying upon generative AI outputs without giving them the proper oversight, which could also affect code quality.

“I think there is a risk for lower quality code if a developer blindly relies on what an AI assistant generates,” said Rob Whiteley, CEO at Coder. “Generative AI tools have flaws, and if outputs are not verified, they can insert code that compiles but does not behave as a developer may have intended. It can also insert errors or bugs if not checked.”

How to Retain Code Quality in the AI Age

“I would not say that results of [the GitClear] study are conclusive,” said Dr. Eirini Kalliamvakou, staff researcher at GitHub. She believes the results are misleading since there is no way to know if an AI or a human authored the code that was analyzed for the study. That said, she acknowledged that “if we are helping people code faster, we should ask the question about code quality.” So, how do we retain the quality of all code, regardless of whether it’s human- or machine-generated?

Use Copilot Thoughtfully

The first recommendation is to use AI assistants thoughtfully. If you’re exhausted at the end of a long day, it’s a lot easier just to “hit tab” and accept an AI-generated auto-complete. But this might not be the best move. “When used thoughtfully and with oversight from the developer, I do think AI can improve the quality of a developer’s code by making suggestions and preventing human errors,” said Whiteley.

Apply the Same Code Reviews to AI-Generated Code

Additionally, development teams should be applying the same degree of rigor to AI-generated code, taking it through the same design review processes and security reviews as you would for any new application. “Just because you’re writing with Copilot doesn’t mean you should give up on code reviews, your linters, looking for security vulnerabilities and all that,” said Kalliamvakou.

Avoid Redundancy When Possible

Leaner code means efficiency boosts and cost-savings. “Much of code is redundant and AI can streamline that redundancy and offer interesting insights, but not without developers ensuring the suggestions the AI tool makes align with the code they are intending to write,” said Whiteley. “Similar to Gmail’s autocomplete sentence feature, AI can complete lines of code with accuracy, but only as long as it accurately guesses where you’re going with your sentence.”

Keep Tabs on the Impact on Quality

Multiple sources cited that around 30% of code is now AI-authored. Harding calls for more efforts to analyze the impact of AI generation on code quality. “It would benefit the greater good to get a clear and data-backed assessment of how quality fluctuates when developers hit ‘tab’ to accept an AI suggestion,” he said. This is a new field of study, and will require new metrics to fully monitor and appreciate the impact AI has on code quality.

Determine How To Measure Code Quality

Similar to the dearth of consensus on what exactly constitutes developer productivity, defining and measuring code quality is just as challenging. Therefore, Kalliamvakou recommends first defining what code quality means to your organization and what the metrics are. For instance, this could be the “number of defects.” Then, the next step would be to set up methods to alert developers about potential quality issues, she said.

The Bottom Line: AI Enhances Productivity…

AI assistants are changing the way code gets made, and much of this is for the better.

Most notably, AI can have a significant impact on enhancing productivity for up-and-coming or inexperienced developers. “If an enterprise makes technology like generative AI broadly available, then it can supplement the work and boost the productivity of the base of the talent pyramid,” said Whiteley. “Put simply, that investment massively impacts the broader organization, not just the top 1% of developers.” Even if such tools make a developer 20% more productive, they can have a truly transformative impact.

While the GitClear study doesn’t prove the culpability of AI assistants, it raises some interesting questions worth exploring, said Harding. Notably, is code becoming more repetitive? If so, that could have some negative outcomes for overall efficiency. “Code generated during 2023 more closely resembles an itinerant contributor, prone to violate the DRY-ness of the repos visited,” the GitClear report said.

… So, Assess Code Quality

“If leveraged correctly, generative AI will not lower the quality of code,” said Trisha Gee, lead developer advocate at Gradle, Inc. “However, it’s important that developers using this technology are critically evaluating AI-generated code rather than taking what AI suggests at face value. Code from generative AI may not always be ‘correct,’ be the best solution to a problem or match the code style of the codebase.”

My takeaway is that, yes, you can retain code quality while using AI assistants. However, this assessment comes with some strings attached. It will require an ongoing effort to sanitize the outputs of AI and instill the same checks we typically apply to manually-generated code.

Image source: https://vecteezy_futuristic-technology-concept-mixed-media-innovations-data_7019122_167-3.jpg

February 6, 2024
Software Quality is the Heartbeat of the Best Organizations
JPMorgan recently announced it was hiring 2,000 engineers, despite the gloom in global economic markets. Is this not an odd risk for an organization to take, given the demand for (and cost of) software developers today? What’s happened?
JPMorgan’s hiring drive is no mystery. Virtually every company in the world is leveraging software, with many transforming themselves into software organizations to drive new efficiencies, widen profit margins and create new markets. Software has been disruptive, so much so that the current era has been dubbed the “third industrial revolution.” Consumers are benefiting from switching providers at the click of a few buttons and most consume services through software. Bad software experiences today mean rapid customer churn and competitive disadvantage.

Quality Should be the Heartbeat of any Organization

In spite of quality’s importance, testing is still perceived as a secondary discipline and a check-the-box exercise. This reflects practices developed for assembly lines during the original industrial revolution. A product gets assembled and testers verify its quality as it leaves the production line. Testing’s goal in this context was to identify problems before they reach end-users. This same approach has been applied widely to software development and is still used by “waterfall” organizations.

Yet, software development has been revolutionized by new methodologies and practices. Software today is constructed at incredible speed. Notice I’m using the word “constructed” and not “built”—the best developers leverage libraries or APIs to assemble software using existing components. They then add a few pieces of custom business logic to define the intellectual property of an application. Many organizations don’t even code software, instead customizing off-the-shelf software with vast configurations to modify functionality.

The assembly-line approach to software development is no longer appropriate. When testing is left late, defects are found before a release or even in production. Release cycles are longer and products poorer quality. Yet, many organizations still stick religiously to this approach, with testing teams operating in distinct silos.

The Role of the Tester is Evolving

The best organizations in the world are changing how software is built. Testing is no longer a check-the-box exercise at the end of the assembly line. It is transforming into a discipline that revolves around evangelizing and driving quality throughout an organization.

Quality should be baked in as early as possible, touching every artifact produced. Ideally, this should occur from the start, as a requirement is formulated, all the way through to the software being shipped. The idea of siloed teams working on different pieces is outdated. For quality to thrive, teams must collaborate.

Modeling offers one approach to driving quality and collaboration throughout an organization. Its journey starts with a requirement. These are typically formulated by business analysts or product owners. Having been a product owner, I found that this usually involves lots of scribbles on paper before communicating to stakeholders on a whiteboard. The preliminary requirement then gets reviewed and refined.

At some point, the requirement will be converted into walls of text in a project management tool. Modeling intervenes at this stage, adding visualizations that make requirements more consumable, less ambiguous and easier to understand.

The word “modeling” may seem a scary concept, but it need be no more than a flowchart that encapsulates a requirement. The benefit of modeling is not only that it provides a digital collaboration asset; models can also generate coverage-focused tests before a line of code is written. Modeling creates requirements that are quality-driven and quality-driving.

The Journey of Quality-Driven Requirements
- Models Provide Data Upfront
Developers today require rich data to develop, while testing is only as good as the available test data. Yet, many organizations still feed slices of production data into developers and test cases.
This approach is problematic for several reasons. First, it often risks legislative non-compliance. Second, production data features large volumes of low-variant data. Data may not then be available when developing unusual-but-important scenarios. Imagine an overdrawn account with several beneficiaries: What should happen when a transfer is attempted?

Modeling defines data explicitly in requirements. It allows engineers to understand a requirement explicitly, including the relevant data. From the modeled definitions, correct data can then be masked and subset, or generated synthetically. Data can further be delivered on-demand and at scale using database virtualization.
- Models Provide Understanding
Once the requirement and necessary data are available, code can be developed. Using text-based requirements, misinterpretations often lead to incorrect functionality. Visual models instead align closely with business logic, and we often see developers enriching models with additional context so that models align with the actual software implementation.
- Models Generate In-Sprint Tests
Once the code is developed, the same tests which were created earlier from the requirements can be executed against new functionality. This facilitates reuse and means the test cases are ready to be executed immediately.

Some tests might further be selected for automation. In this instance, automation can be overlaid onto the requirements model, reusing code to generate scripts. These scripts then become part of the standard regression pack as new features are added to the product.

A Change Request Comes In

With iterative development, change requests are common. This typically creates havoc for quality, since it is difficult to understand the impact of a change, and time-consuming to update artifacts like tests and data.

Traceability is key to overcoming this challenge. Models can connect to external DevOps tools to provide end-to-end traceability, from a requirement, all the way to code and tests. The impact of a change can then be identified directly. Models can be updated, automatically synchronizing test data, test cases, and automation to bring everything up to date.

Testing for the Fourth Industrial Revolution

Software is playing a vital role in virtually every organization, as already recognized by organizations thriving today. Baking quality in early is the only way to ensure quality software. The role of testing must change on a large scale. Models offer a way to achieve a “shift left” and collaborative approach to quality while testing earlier and more rigorously by automating and optimizing test and data creation.
October 27, 2022
Continuous Testing: The Quest for Quality at Speed

The DevOps goal to reduce lead times by accelerating software deliveries presents a serious challenge for assuring the quality of the software that is delivered. As I indicated In my SlideShare post, “The Quest for Quality at Speed,” testing activities span end-to-end software value streams and often account for more than 50% of bottlenecks.

The answer to these DevOps testing challenges is referred to as continuous testing (CT), which is more than simply automating tests. My blueprint for continuous testing identifies eight pillars of practices, each with strategies to reduce bottlenecks. In this article, I enumerate the strategies for each of the eight pillars of continuous testing and explain how to approach implementation for greater adoption of CT across an organization.

Continuous Testing Pillars

Continuous Testing Tenets

There are five tenets which are guiding principles that enable continuous testing:
• Shift Left: Conduct each test as early in the pipeline as possible.
• Fail Early: Arrange the tests so the most likely problems are found during the earliest possible stage in the DevOps pipeline.
• Fail Often: Run tests frequently and with many different conditions.
• Test Fast: Arrange tests to run in quick cycles.
• Be Relevant: Focus on the most important tests and results.

Leadership and Culture

Leadership and culture practices have a major impact on influencing behaviors and habits of everyone involved in testing across the DevOps value stream. Bottlenecks to CT include inadequate funding of testing due to lack of quality vision and goals at the executive level, and bottlenecks caused by distrust and lack of cooperation between Dev, QA and Ops. Executive sponsors need to clarify the definition of quality for the organization, and emphasize quality’s importance. Executives can communicate the vision and goals for quality and establish incentives that foster a culture of collaborative behavior.

Test Strategies and Plans

Test strategies and plans are needed to guide testing approaches for each product/service pipeline. Bottlenecks to CT include confusion caused by lack of product/service test strategies, and unplanned testing activities and resources are not included sufficiently in planning process. It is important to establish test coverage standards, test case standards, gate criterion and planning process that include testing requirements. CT strategies include making sure test coverage requirements are defined, all types of tests required are identified, allocate tests that are required at input gates, indicate test results are required at output gates and enforce that test tasks are included in planning backlogs and sprints.

Test Management

Test management is needed to coordinate testing resources and test schedules. Bottlenecks to CT include poor utilization of test resources due to lack of visibility and schedule bottlenecks due to lack of coordination of test schedules. It is important to have a database of tests and test resources, and test scheduling tools. CT strategies include version management of tests and resources, an inventory of test resources and dynamic test scheduling.

Test Automation

Designing, creating, selecting and executing tests in an automated fashion to get to a verdict as fast as possible is the primary strategy to accelerate testing. Bottlenecks to CT include slow test creation, slow test execution and slow time to realize verdicts. It is important to have effective automation strategies for test design, test creation, test selection, and test execution. CT strategies include test design templates, advanced test creation tools (E.g., TDD, keyword, behavior, model-based), dynamic test selection, tests designed for fast verdicts and horizontal and vertical scaling tests.

Test Tools

Test tools are designed to automate specific types of tests and integrate with DevOps platforms through APIs for orchestration, execution and reporting. Bottlenecks to CT include test tools that are not designed to scale, limited test configurations and lack of complete APIs, which make integration with DevOps platforms inefficient and error-prone. It is important to have scalable test tools with good APIs. CT strategies include limiting the selection of test tools to ones that are DevOps-ready or DevOps-native and that have well-defined, stable APIs.

Test Environment Management

Scheduling and setting up resources into the needed topologies for running different types of tests for developers and testers can be a major bottleneck if not managed efficiently. Bottlenecks to CT include lack of test resource scheduling, which causes conflicts for resources, manual test setups, which are time consuming, causing long wait times for testing and delayed retesting. It is important to have test resources scheduling, test environment orchestration and resource utilization management. CT strategies include test environment inventory tools, test environment sharing tools, test environment orchestration tools and monitoring test environment utilization.

Test Results Analysis

Reporting and analyzing test results can be a major bottleneck after test execution is completed. Bottlenecks to CT include poorly written tests which can report confusing results and that take a lot of time to analyze. It is important to have design standards for test results and test results reporting standards. CT strategies include test verdict design in test cases, common test reporting tools and test analysis tools.

Steps To Implement Continuous Testing

Start with one or two services and a small set of critical CT practices, then expand to more services and more advanced CT practices.

A gap assessment is a great way to determine the current state of CT practices, and will help focus the implementation plan. Given a list of practices from each of the pillars in the continuous test blueprint, a cross-functional team of developers, testers and ops people can assign scores to measure importance and current level of practice to generate a gap score. The combined set of gap scores will give guide and direct current priorities for the CT plan.

For example, beginning CT priorities could be to establish a strategy sponsor, define quality goals, obtain an inventory of test resources and requirements, put in place version management for tests and test standards, tool selection and test refactoring to match the standards.

An intermediate CT goal could be to complete tool integrations with DevOps platforms, automate test scheduling, orchestrate test environments and optimize test results reporting.

With maturity, advanced CT goals could be to put in place advanced test creation strategies, accelerated test execution, automated test environment management and optimize test results analysis.

What This Means

Testing activities are typically a major source of bottlenecks that slow the quest for quality at speed. Continuous test strategies that accelerate all aspects of testing can have a major impact on speeding up lead time for continuous delivery of quality software. Implementation strategies that start by applying specific continuous test strategies to a targeted small number of applications, and then expand to more advanced strategies applied to more applications is an effective approach to CT adoption across the organization.

April 9, 2021