devopsy.kinsta.cloud

Tag: AI testing

Stop Chasing 100% Test Coverage: Why DevOps Teams Need to Test Smarter, Not More
Every software team eventually runs into the same question: How much testing is enough?

For years, the default answer has been “more.” More automated tests. More regression suites. More environments. More devices. But as applications become more complex and release cycles continue to accelerate, testing everything simply isn’t realistic, even with the support of AI.

Modern software is built on microservices, third-party APIs, cloud infrastructure and code that is constantly changing. Teams are expected to release faster while maintaining quality, security and reliability. The challenge isn’t a lack of commitment to testing; it’s that time and resources are finite.

Instead of asking, “How much testing is enough?” we need to determine “What should we test first?”

The answer starts with understanding risk.

Not Every Bug Carries the Same Risk

One of the biggest mistakes organizations make is treating every bug as equally important. A typo on an internal admin page is inconvenient. A failed login, a broken payment process, or an outage affecting a core customer workflow can quickly become a serious business problem.

Risk-based testing recognizes that reality. Rather than spreading testing effort evenly across an application, teams prioritize the areas where failures would have the greatest business, operational, or customer impact.

That often includes:
- Authentication and access control
- Payment and transaction processing
- Security-sensitive functionality
- Critical APIs and third-party integrations
- High-traffic customer workflows
Lower-risk functionality still deserves testing, but it doesn’t always require the same level of scrutiny.

DevOps is Already Built Around Risk

This way of thinking isn’t new to DevOps teams. Security teams prioritize vulnerabilities based on exploitability and business impact. Site reliability engineers focus on reducing the failures most likely to cause outages. Infrastructure teams evaluate the blast radius before making production changes.

Testing should follow the same principle. Instead of asking, “What haven’t we tested?” teams should ask:
- Which failures would have the biggest impact on customers?
- Which services change most frequently?
- Which integrations have historically been the least reliable?
- What issues would be the most expensive to recover from?
Those answers provide a much stronger foundation for testing priorities than simply aiming for maximum coverage.

A Structured Approach Makes Prioritization Easier

Risk-based testing doesn’t have to be an ad hoc exercise. Several established frameworks help organizations evaluate risk and translate it into testing priorities.

For example, Product Risk Management (PRisMA) identifies risks throughout the software development lifecycle, while Rapid Risk Assessment (RRA) helps teams quickly prioritize testing based on business and technical impact. Quality Functional Deployment (QFD) ensures customer requirements influence testing decisions, and Cost of Exposure estimates the financial impact of defects to help focus testing on the failures that would be most costly.

The specific framework matters less than consistency — having a coherent and steady process for evaluating risk before every release.

Putting Risk-Based Testing Into Practice

Adopting a risk-based approach doesn’t require a complete overhaul of your testing strategy. It starts with better planning.

Bring together stakeholders from development, QA, product, operations and customer support to identify the parts of the application that would cause the greatest disruption if they failed. Consider business-critical functionality, security, data integrity, third-party integrations and customer-facing workflows.

Once those risks have been identified, classify features by priority. High-risk functionality should receive the deepest level of testing, while lower-risk features can often rely more heavily on automation or periodic regression testing. If defects are found in lower-risk areas, teams can evaluate whether they truly warrant delaying a release.

AI can also help streamline this process by identifying code changes, recommending additional test coverage and highlighting components that may require closer inspection. But AI shouldn’t make prioritization decisions on its own. Understanding customer expectations, business priorities and the real-world impact of failures still requires human judgment.

Focus on Confidence, Not Coverage

Code coverage and test execution metrics still have value, but they don’t tell the whole story.

A team can achieve impressive coverage numbers while missing the handful of scenarios most likely to cause production incidents. Conversely, another team may run fewer tests overall, but consistently deliver more reliable software by focusing on the areas that matter most.

The goal isn’t to test everything. It’s to reduce the likelihood that a critical defect reaches production.

As software systems continue to grow in complexity, exhaustive testing becomes increasingly unrealistic. The organizations that consistently deliver high-quality releases won’t be the ones running the most tests; they’ll be the ones making the smartest decisions about where testing delivers the greatest value. For DevOps teams under constant pressure to ship faster, success isn’t measured by the number of tests executed, but by how effectively those tests reduce risk before software reaches production.
September 15, 2026
Tricentis Preps Wave of Additional AI Testing Capabilities

Tricentis is providing early access to multiple artificial intelligence (AI) capabilities that it is gearing up to roll out later this year via a Tricentis Transform initiative, including an autonomous AI agent, dubbed Aida, that explores web and Windows desktop applications to surface defects, weaknesses and other potential gaps without requiring DevSecOps teams to create scripts.

Additionally, Tricentis is developing a Release Risk Intelligence capability that surfaces release-scoped coverage gaps, prioritizes risks by severity, and launches AI tasks to resolve issues that are informed by context provided by the automated Tricentis testing platform.

Finally, Tricentis is also readying AgentScore, a tool that enables organizations to evaluate AI agents by observing how agents behave in real-world workflows, recommending what should be measured, and generating composite quality scores that include review, block, or ship recommendations.

David Colwell, vice president of AI and machine learning for Tricentis, said the overall goal of the Tricentis Transform initiative is to ultimately make AI application testing a more integrated element of building and deploying any application rather than being an entirely separate function. The end result should be higher quality applications that are inherently more secure, he added.

The launch of the Tricentis Transform initiative comes on the heels of the company’s acquisition of Tabnine, which developed a knowledge graph that AI agents can invoke to discover the relationships between various application components. Armed with insights that are uniquely provided by a testing platform, AI agents are provided with the context needed to automate tasks much more reliably, noted Colwell.

It’s still early days so far as embedding AI agents into DevOps workflows is concerned, but as more code is generated using AI tools, there is a corresponding increase in the amount of pressure on existing pipelines. The first place that pressure is manifesting itself is in the need to review and test code at machine speed. In the absence of that capability, more AI-generated code is being deployed in production environments with predictable results. The number of incidents in production environments that have been traced back to AI code has already begun to exponentially increase, noted Colwell.

Exactly how long it might take to re-engineer DevOps workflows in the age of AI remains to be seen, but it’s apparent that testing will become more integral. The difference is that much of that testing is likely to be conducted by AI agents that are reviewing and analyzing code created by other AI agents. The challenge, in the meantime, is a coverage gap that is widening as the volume of code being generated overwhelms the ability of DevOps teams to test it.

Hopefully, there will come a day when the distinction between creating and testing code becomes so blurry as to be all but meaningless. That doesn’t mean the need for specialists to test applications will completely disappear, but it does mean that any issues that do require their expertise will be worth the time and effort for them, rather than an AI agent, to personally resolve.

August 26, 2026
Production-Grade AI Eval Systems. What I Learned Putting LLMs on Call
It was around 11 p.m. on a Thursday. Our AI support agent had been live for three weeks. Latency: Green. Error rate: Green. Then a customer DM landed in Slack with a screenshot — the bot had cheerfully invented a refund policy for a product we had never sold. Made up the SKU. Made up the rules. Returned a confident answer in 1.2 seconds.

Every SRE metric said the system was healthy. The system was lying to customers at scale, and we had no signal. That night I started building what I now call a production-grade eval system. This is the version I wish someone had written for me before that Thursday.

The Honest Problem

For 15 years, the SRE playbook worked because systems were deterministic — same input and output. LLMs break that contract. Your vendor can silently push a new model checkpoint on a Wednesday, and your agent develops a new personality. A re-indexed retrieval store can send ‘what’s your return policy?’ to a marketing blog post instead of the actual policy doc. None of those registers as a 4xx.

The result: Dashboards all green; product quietly degrading. Everything below exists to close that gap.

The Three Places You Must Evaluate

Most teams vibes-check their AI feature: A PM tries six prompts, says “feels good” and ships. There’s no regression suite, no quality baseline and no rubric. The right fix is to evaluate in three explicit places:

Most teams have Phase 1, sort of. Almost nobody has Phase 2. Phase 3 is where the screenshots-in-Slack live. If I could pick only one, I’d start with Phase 3 — it catches failures you didn’t predict. You need all three to call yourself production-grade.

The Four-Layer Evaluator Stack

There is no single quality metric. What works is layers — cheap checks at the bottom, expensive judges at the top — with sampling upward.

Layer 1 (deterministic code) runs on every trace — free. Layers 2–3 run on a sampled subset; LLM judges cost real money and will eat your budget if you let them run inline. Layer 4 (domain checks) runs with humans in the loop — slowest, most valuable.

The Flywheel: Where Good Evals Come From

Nobody hands you a good eval set. Vendors sell generic benchmarks that score 95% on your system because they test nothing customers care about. Real evals come from your own production failures.
- Pull production traces — bias toward anomalies, low-confidence scores and customer complaints.
- One person labels them — consistency beats coverage. Two labelers create noise that looks like signal.
- Cluster failure modes — hallucination? tone? wrong tool call? Each mode needs its own judge.
- Promote failures to regression tests — the failure can never happen silently again.
When you lack data, generate only inputs synthetically, then run your actual application on them. Never let an LLM generate both sides — you’ll get a data set that scores 99% and means nothing.

RAG: the Three Metrics That Matter

If you’re running retrieval-augmented generation — and most of you are — every interesting metric is a relationship between the question, the retrieved context and the generated answer.
- Retrieval Tier: Context precision and recall. If this is broken, nothing downstream matters. Start here.
- Core RAG Tier: Faithfulness (model stays grounded in context) and answer relevance (model answers the actual question).
- Diagnostic Tier: Citation accuracy, noise sensitivity, per-chunk hallucination rate. Run these on investigations, not alerts.
Guardrails Vs. Evaluators

This confusion causes more bad architecture than anything else I see in production AI today.

Keep them in separate code paths. Draw them in separate boxes on your architecture diagram. A guardrail that takes three seconds is a broken product. An evaluator that takes three seconds on 5% of traffic is healthy.

The Production-Readiness Checklist

If someone asks me whether their AI product is ready for production, I walk through this list:
- Golden regression data set — in the repo, version-controlled, owned by a named human.
- CI eval gate — blocks merges when quality drops on any PR touching prompts, retrievers or model config.
- Online evals on sampled production traces — at least Layers 1–2 on every span.
- LLM judges validated against humans (Cohen’s kappa >0.7), re-validated on every model upgrade.
- Weekly annotation session with a written agenda — not random sampling.
- Pinned model versions for production model and judge model. Vendor upgrades are deploys.
- Guardrails and evaluators in separate code paths.
- Someone whose job is to turn findings into roadmap items.
The Mindset Shift

Reliability for AI is not measured in uptime, it’s measured in quality of output over time. The dashboards your SRE team already built are necessary but no longer sufficient.

The new layer is the eval system — multi-tier, fed by an error analysis loop, wired into CI and live traffic, with calibrated human-in-the-loop judges. Build it, and AI stops being the system your team is scared to put on the on-call rotation.

That Thursday night was almost a year ago. We haven’t had another one like it. The dashboards are still green — and now I know what green actually means.
August 21, 2026
From AI Hype to AI Assurance: How Engineering Teams Can Safely Ship AI-Enabled Software

AI has moved very quickly from experimentation to production. A few years ago, many organizations were still asking whether AI could improve their products or internal workflows. Today, the question is different: how can teams ship AI-enabled software safely, reliably, and responsibly?

That shift matters because AI is no longer just a research project or a boardroom talking point. It is being added to customer support platforms, fraud detection systems, developer tools, compliance workflows, cloud operations, marketing engines, and enterprise applications. The opportunity is real, but so is the risk.

Traditional software usually behaves in predictable ways. If the logic is written correctly, the same input should produce the same output. AI systems are different. A generative AI feature may produce useful answers one moment and questionable answers the next. A model can hallucinate, misread context, expose sensitive information, or make recommendations that sound confident but are wrong.

This is why engineering teams need to move from AI hype to AI assurance.

AI assurance is the practice of making sure AI systems are tested, monitored, governed, and improved throughout their lifecycle. It is not about slowing innovation. It is about giving teams the confidence to scale AI without losing control of quality, safety, or trust.

One of the biggest mistakes organizations make is treating AI like a normal feature release. They build a demo, test a few examples, get a good response, and move toward launch. But AI systems need a deeper level of review. A prompt change, model update, data change, or new integration can affect the behavior of the system. What worked yesterday may not work the same way tomorrow.

That means AI testing must become continuous.

Engineering teams should test AI systems for accuracy, consistency, bias, security, hallucination, and failure handling. They should also test edge cases, not just ideal user scenarios. For example, if an AI system summarizes customer complaints, the team must know whether it preserves important details, avoids inventing information, and handles unclear messages properly. If the system is used in finance, healthcare, identity verification, or compliance, the bar must be even higher.

The first step is defining what “good” means. This sounds basic, but many teams skip it. A chatbot, fraud detection tool, code assistant, and recommendation engine should not share the same success criteria. Each AI feature needs clear standards for acceptable behavior, unacceptable behavior, and situations where the system should refuse, escalate, or ask for human review.

The second step is building evaluation datasets. These are collections of real or realistic examples used to test the AI system before release and after major changes. A good evaluation set should include common requests, difficult edge cases, sensitive scenarios, adversarial prompts, and examples where the AI should not answer directly. This gives teams a repeatable way to measure performance instead of relying on a few manual checks.

The third step is adding AI evaluations into CI/CD pipelines. Just as engineering teams run unit tests, integration tests, and security checks before deployment, they should also run AI-specific evaluations before releasing prompt updates, model changes, or workflow modifications. These checks will not make AI perfect, but they can catch many problems before users do.

The fourth step is observability. AI assurance does not stop when the feature goes live. Teams need to monitor how the AI behaves in production. They should track failed responses, user corrections, unusual outputs, latency, cost, model drift, and repeated complaint patterns. When something goes wrong, teams should be able to trace what happened and respond quickly.

Human oversight is also important. Not every AI workflow should be fully automated. In high-risk areas, the better approach is often human-in-the-loop automation. AI can draft, classify, summarize, or recommend, while a person makes the final decision. This keeps speed and judgment working together.

Leadership has a major role to play as well. AI assurance cannot be left only to individual developers. Organizations need clear ownership for model selection, data handling, prompt changes, testing standards, compliance review, and incident response. Without ownership, AI systems can quietly spread across an organization without enough accountability.

The real competitive advantage in AI will not belong only to the companies that ship fastest. It will belong to the companies that can prove their AI systems are reliable, secure, and trustworthy enough to use in real business environments.

For engineering teams, the goal is not perfection. AI will always carry some level of uncertainty. The goal is to reduce avoidable risk, catch failures earlier, and create systems that can be monitored and improved over time.

AI hype may open the door, but AI assurance is what keeps organizations from walking blindly through it.

As AI becomes part of everyday software, assurance will become a core engineering discipline. The teams that build this discipline now will be better prepared to deliver AI products that are not only innovative, but also safe, useful, and trusted.

July 15, 2026
Why AI Testing Must Live Inside Your CI/CD Pipeline

Here is a situation most engineering leaders recognize. You roll out AI coding tools. Features ship faster. Developers are more productive. Then, a few months in, you realize something unexpected: the engineers you most wanted to free up are busier than ever. They are reviewing PRs, firefighting regressions, juggling a dozen half-shipped features and the edge cases that came with each one.

AI made them more efficient. It also made them ten times more busy.

The bottleneck did not disappear. It moved. And it moved to exactly the place most teams are least equipped to handle quickly: verification.

This is the problem that CI/CD, in its current form, is not set up to solve on its own. CI/CD is a delivery mechanism. It runs what you give it. If you give it a pipeline that still depends on humans to write tests, review logic, and triage failures, adding AI on the generation side just means the human verification step gets hit harder. You are filling a faster funnel into the same narrow drain.

The fix is not to slow down the generation side. It is to close the loop on the verification side, and to do it inside the pipeline where it actually belongs.

Testing That Waits Is Testing That Fails

The traditional model treats testing as a phase: you write the feature, you test it, then you ship it. That sequencing made sense when features shipped slowly enough for each step to happen on a human timeline. It does not hold when your coding agent can produce a hundred PRs in the time it used to take a developer to write one.

What you get instead is a backlog. Tests that nobody has time to write. Suites that nobody has time to maintain. A regression that surfaces in production because the team was moving too fast to catch it earlier. Or, on the other side, a deliberate slowdown, where teams cap how fast they ship because quality cannot keep up with output.

Neither outcome is acceptable. And yet most teams have been forced to choose between the two.

The alternative is to make testing continuous rather than sequential: test generation, execution, and repair happening automatically at every stage of the pipeline, triggered by the same commits and PRs that already drive your delivery workflow. Not a separate tool someone opens when they have time. Something that runs whether or not anyone asks it to.

The Gap Inside the Pipeline

There is a deeper reason why testing needs to live inside CI/CD, and it has to do with what software actually is in production.

Code is a small piece of it. The real system is code running against database states, third-party API behavior, configuration values, permission layers, cached data, and the unpredictable patterns of real users doing things the developers did not plan for. That environment is what determines whether software works. And tests that only look at code in isolation, without modeling that environment, will miss exactly the bugs that cost the most.

The CrowdStrike outage in July 2024 is the clearest example of what this looks like at scale. A sensor configuration template and the sensor code were each individually valid. Their interaction, a mismatch in field counts that only existed at runtime, crashed 8.5 million Windows machines. Every individual component passed its tests. The failure lived in the space between them.

That space is what verification needs to reach. It requires testing against a simulation of the real environment, not just a reading of the code. And that simulation has to be continuous, running on every commit, not waiting for a scheduled test run or a manual trigger.

What This Looks Like in Practice

When testing is embedded in CI/CD, every PR gets executed before anyone reviews it. Not code-reviewed, which tells you whether the logic looks right. Executed, which tells you whether it actually works. Targeted tests are generated against exactly the code that changed, infrastructure is set up automatically, and the results come back before the PR reaches a human reviewer. The prompt-test-prompt cycle that eats engineering time either goes away entirely or runs between machines.

Coverage evolves as the product does. The agent maps your application, generates production-ready tests, and updates them when things change. Teams that previously spent weeks building and maintaining test suites get that time back.

API changes, which tend to ripple in unexpected ways, get verified against real endpoint behavior, not just schema validation.

The result is that quality becomes a property of the pipeline rather than a checkpoint at the end of it. By the time a PR reaches production, it has already been tested in conditions close to what it will actually face.

The Real Unlock

AI coding tools deliver their full value when the loop is closed. Writing code faster only compounds if shipping code is also faster, and shipping code faster only compounds if you have confidence in what you are shipping.

Closing that gap does not require changing your stack. It requires adding a layer to the pipeline you already have, one that runs continuously, covers every layer of testing, and makes verification as fast as generation.

That is the pipeline that unlocks what AI coding was actually supposed to deliver.

June 17, 2026
From Code to Confidence: Building AI Apps That Earn User Trust
As a developer, you’ve probably felt the excitement of deploying your first AI-powered feature. The model performs exceptionally well in testing with solid accuracy metrics, and stakeholders are delighted. Then, reality hits: Users start complaining about biased recommendations, confusing outputs or worse — publicly calling out your application for generating harmful content.

A recent Applause survey found that 65% of users experienced issues with AI applications between January and March 2025, including bias, hallucinations and incorrect responses. Trust has officially become the new battleground for user experience, and traditional testing approaches are no longer sufficient for AI applications.

The good news? By shifting from assumption-driven development to deliberate, user-centered testing processes, you can build AI applications that don’t just work — they earn genuine user confidence.

Most developers approach AI testing in the same way they handle traditional software development: They write unit tests, check edge cases, validate outputs and ship the product. However, AI applications are fundamentally different. They’re probabilistic, not deterministic. They evolve with new data. Most critically, they interact with humans who judge them not just on functionality but on fairness, transparency and trustworthiness.

Many teams attempt to fill resource gaps with automated testing alone. Your recommendation engine might show 95% accuracy in your automated test suite, but real users from different demographic groups may experience vastly different outcomes. Your automated tests missed what matters most — how your AI behaves across the full spectrum of human diversity. This gap between lab performance and real-world trust is a business-critical issue that can sink your application before it gains traction.

Implement Human-Centered Testing From Day One

The solution isn’t more automated tests — it’s involving humans into your testing process from the beginning. Your AI is only as trustworthy as the data and people involved in its training. If your training data comes from a narrow slice of users, your AI will reflect those limitations.

Build testing communities that accurately reflect your actual user base, encompassing diverse ages, backgrounds, languages and accessibility needs. Don’t rely solely on your internal team or contractors who share similar perspectives. Establish regular testing cycles in which human evaluators from diverse demographic groups assess your AI’s outputs for fairness. Track metrics like response quality across user segments, not just overall accuracy.

Bias isn’t a one-time check — it’s an ongoing concern requiring continuous monitoring. As your model learns from new data, new biases can emerge or existing ones can amplify. Users don’t just want your AI to be right — they want to understand why it makes specific decisions. This is especially crucial for AI applications that affect essential life decisions, such as healthcare, finance or hiring.

Build explanation capabilities into your AI architecture from the outset. Test these explanations with real users to ensure they’re understandable, not just technically accurate.

Create Inclusive Feedback Loops

Traditional feedback loops in software development often miss the nuanced ways AI impacts different users. Creating inclusive feedback mechanisms requires intentional design.

Don’t just rely on star ratings or binary feedback. Collect qualitative insights through conversational interviews, demographic-specific focus groups and longitudinal studies that track how trust evolves. Create feedback channels that accommodate different communication preferences — some users prefer quick surveys, others seek detailed conversations and some express concerns through community forums.

Lab testing can’t replicate how users interact with AI in their daily lives. Background noise affects voice AI performance differently in a coffee shop than in a quiet office. Cultural context also influences how users interpret AI-generated content. Test your AI applications in the environments where they’ll be used and partner with diverse user communities for ongoing feedback.

Establish Continuous Trust Monitoring

Building trustworthy AI isn’t a sprint — it’s an ongoing commitment to improvement. Traditional software metrics such as uptime and response time are insufficient for AI applications. You need metrics that specifically track user trust and confidence.

Develop trust-specific KPIs such as user confidence scores, bias detection rates and explanation clarity ratings. Track these alongside your technical metrics. Your AI model should evolve based on real user feedback, not just algorithmic optimization. Create feedback loops where user insights about bias, fairness and trust directly inform your model training and fine-tuning processes.

Users are more forgiving of AI limitations when they understand them. Be upfront about what your AI can and can’t do, and how you’re working to improve it. The temptation is to iterate quickly and fix problems later, but trust issues compound rapidly — once users lose confidence in your AI, regaining it is difficult than building it right the first time.

Building trustworthy AI applications requires a fundamental shift in how you approach development and testing. Begin with these concrete actions:
- Audit your current testing approach. How diverse are your testers? Are you testing for bias and fairness or only for functionality?
- Establish human evaluation processes. Regularly identify ways to incorporate diverse human perspectives into your testing pipeline.
- Create multiple feedback channels. Develop various ways for users to share concerns about bias, fairness and trust.
- Define trust metrics. Establish KPIs that measure user confidence alongside technical performance.
The future belongs to AI applications that not only work well but work well for everyone. By embracing user-centered testing processes and building inclusive feedback loops, you’re not just shipping better software — you’re laying the foundation for AI applications that users will adopt and advocate for.

In AI development, confidence truly comes from quality, and quality comes from deliberately designing for trust from the outset.
November 6, 2025
Accelerating DevOps Pipelines With AI-Native Testing

Achieving speed in DevOps and moving fast with CI/CD pipelines means getting builds done quicker, deploying more often and making changes faster to keep up with user needs.

However, keeping that pace while also testing thoroughly and ensuring high quality & security is exceedingly hard, as the older ways of testing — involving manual work or automation that breaks easily — slow things down, acting like a bottleneck that stops the whole pipeline from running smoothly. This also often leaves teams wondering how to actually speed up their DevOps without cutting corners on important testing.

Artificial intelligence (AI)-native testing changes this significantly. According to Shahid Ali Khan, Principal Engineer at LambdaTest, a cross-browser testing platform, “It’s not just adding some AI bits onto existing testing tools; it’s about using generative AI and predictive AI together to completely change how we think about, build and run tests through the whole DevOps process.” Using AI in testing lets teams automate things and add intelligence in ways they couldn’t before. Enabling continuous testing that actually matches the speed of continuous delivery is clearly the way forward.

It’s more than just a small step forward; it’s like hitting the reset button on how we approach ‘software quality at speed’. Michael Kwok, Ph.D., VP at IBM Watsonx Code Assistant and Canada Lab Director at IBM, sees AI-native testing “revolutionizing software development,” promising to bring speed and quality together, not as things that work against each other, but as built-in parts of the DevOps pipeline.

I spoke to industry leaders to explore how AI-native testing is making a real impact. Drawing insights from these conversations, we’ll look at the main AI features driving acceleration, their tangible impact on DevOps metrics and developer productivity and what to consider when bringing AI-native testing into your own process for building fast, secure software in the future.

Putting AI to Work in Your Test Pipeline

Getting AI into testing isn’t just one small task; it involves building several smart features that work together to speed up the whole testing part of the pipeline, getting software out faster with fewer hassles.

Intelligent test creation is a big one, because building good test cases takes time and a lot of thought to cover everything comprehensively. Generative AI changes this process fundamentally. Gary Arora, Chief Architect of Cloud and AI Solutions at Deloitte Consulting, stated using a platform powered by LLMs helps break down bigger project goals into user stories, and then GenAI takes over to “generate comprehensive test scenarios — positive, negative and edge cases — which are then converted into executable test scripts and integrated directly into our SDLC.” He called it “intelligent, end-to-end test creation at scale.”

Jon Matthews, VP, Engineering at Functionize, also talked about generative AI doing this autonomously, saying it helps get “full coverage achievable in days versus months,” which really removes technical hurdles in building tests quickly using things they call AI agents.

Predictive AI, on the other hand, helps teams decide which tests matter most in a given situation. After all the tests are created, or even if you have tons of older ones, you don’t always have to run every single test every time, especially if you need feedback fast within the pipeline. Khan of LambdaTest talked about predictive test selection, looking at things such as code coverage and how often tests have run before to “optimize the most relevant test suites.” He sees it helping “to identify out of all features which are the most important ones that I should run first in my pipeline based on a priority.”

Arora affirms this, saying predictive AI adds foresight by “prioritizing tests based on code impact and historical failures to help ensure we test what truly matters.” This focus means that teams can be confident about the most critical tests much faster, speeding up CI/CD pipeline runtimes, which currently can feel really unpredictable.

And when tests do fail, figuring out why is a real time sink for developers. However, AI helps here too with rapid root cause analysis. Karan Ratra, Senior Engineering Leader at Walmart, explains that when there are failures, generative and predictive AI can help “segregate the errors based on issues rather than just printing them in the logs, and help us categorize them.” He sees this saving up to 80% of developers’ time in finding the actual root cause.

Matthews stated that AI-driven root cause analysis analyzes defects “to uncover their origins,” generating reports automatically within “seconds” instead of the “15–30-minute manual fix, often across hundreds of tests per release” it might take otherwise. Arora also mentioned predictive AI analyzing logs and tracing errors straight back to the code change, saying this “turns hours of debugging into minutes of insight,” and when problems pop up, the system can even suggest the root cause based on past issues it has learned from.

AI also helps with auto-healing and test adaptation, keeping tests useful over time, especially with frequent code changes — as tests written today might not work exactly right tomorrow if parts of the application are updated or refactored. Matthews suggests that incorporating auto-healing capabilities lets tests “adapt dynamically to changes in the application without requiring manual updates.”

This combination of AI powers — intelligently creating tests, picking the right ones to run first, figuring out quickly why something broke and even helping tests fix or adapt themselves — is what makes AI-native testing capable of pushing things faster through the DevOps pipeline.

But Does AI-Native Testing Actually Move the Needle?

Understanding what AI-native testing can do in terms of capabilities is one thing but seeing how it actually changes the numbers and helps people doing the work is where the real story is told. Putting these AI capabilities into practice has a clear impact on the metrics that matter most in DevOps and on how much work developers and QA can actually get done daily.

Considering key DevOps metrics, such as how often teams can deploy or how quickly they can fix things that break, AI-native testing shows some serious promise and is already delivering results. Experts are anticipating and observing substantial improvements across the board. Deployment frequency could increase by 40% to 60%, lead time for changes could improve by 40% to 50% and mean time to recovery (MTTR) could decrease by 40% to 60%. Arora of Deloitte Consulting is seeing this happen already in early projects, where implementations “are already showing meaningful reductions in MTTR and notable drops in change failure rates,” meaning that there are “fewer surprises in production and faster recoveries when they do happen.”

Matthews stated that AI-native testing speeds up the whole software development cycle by “removing the primary bottleneck in QA and release processes,” which directly “reduces failure rates, shortens remediation time and thereby lowers mean time to recovery (MTTR) & change failure rates.” These significant benefits result from capabilities such as predictive selection, which allows teams to deploy faster by not having to wait for every single test to run before deploying, especially if the most important ones pass quickly.

This leads right into how AI-native testing makes developers and QA teams much more productive in their daily work. Not having to spend so much time on the boring or frustrating parts of testing and debugging means that they can channel their energy into more valuable work — building features and innovating. Matthews explained, saying it “reduces the time developers spend creating and fixing tests,” and because they get feedback faster within the pipeline, they “can work more efficiently overall.” He also added that with testing cycles being shortened, “more time can be devoted to coding and innovation.”

Khan also talked about how the developers at LambdaTest are now spending “less time in writing boilerplate codes or debugging fail builds” and getting “instant feedback on the code that they’re writing,” which then means “more frequent deployments in production with more confidence and less delay.” He sees QA teams building “high-value validations” and catching “oversighted or overlooked scenarios” they might have missed before, ultimately creating “stronger, firmer validation frameworks” that demonstrably improve customer experience.

Bringing AI Testing Into Your Team

Seeing the potential and the early results of AI-native testing in DevOps is exciting; however, actually introducing it into the way teams work every day involves more than just picking the right tools or platforms. For organizations looking to accelerate their DevOps processes using AI, thinking carefully about how to adopt it and how well it suits developers and QA is absolutely key to success.

Kwok of IBM sees this big picture, stating clearly that AI-native testing “will revolutionize software development.” It’s not just about making existing steps faster; it’s about reshaping the entire process and how teams operate together. This kind of fundamental change needs careful thought about how AI integrates into the existing culture and workflows without disruption.

And when it comes to bringing AI frameworks into teams effectively, Khan mentioned some really important points in our interview based on his engineering purview at LambdaTest. He talked about the importance of “how we start adopting this and how we start onboarding the teams into adopting AI frameworks” — and specifically — “how do we use that for cross functional collaborations” effectively. He also stressed that people in the team shouldn’t feel like “they are getting oversighted, or they are getting ignored because of this technology. It’s not replacing anybody.” So, making sure that everyone understands clearly that AI is a tool designed to augment and help, not replace jobs, is particularly important for trust and adoption.

Khan strongly believes this needs to go “hand in hand with transparency, ethics and enabling personnel in the teams” so they understand the technology and its benefits. Making this shift smoothly can’t happen with just a top-down mandate; it requires deep collaboration across developers, QAs, DevOps and even product managers and leadership, as pointed out earlier by Khan. It’s about building a “very symphonic framework or environment” that fosters cooperation between all the different groups involved in the software life cycle, and getting that adoption and onboarding right, built on clear ethics and thoughtful enablement, is what will really make AI-native testing a success in the long run, helping everyone from the developers to the senior leadership understand its strategic benefits.

AI-Native Testing as the Future of DevOps Acceleration

So, looking at how much ground AI-native testing covers — from making smart test cases and picking the most important ones to run, to figuring out why things broke in seconds and even helping tests fix themselves or adapt — it’s clear that AI-native testing isn’t just another tool added to the stack. It’s fundamentally the next big step in actually accelerating DevOps pipelines sustainably.

By bringing generative AI and predictive AI directly into testing workflows, teams can finally move at the speed DevOps promises without having to worry about letting crucial issues slip through quality gates. It’s definitely about getting software delivered faster for sure, but it is also about making sure that the software is reliable and that the developers building it aren’t getting burned out by frustrating manual work.

Making this shift means factoring in the platforms designed from scratch to handle this kind of intelligent, automated quality and security throughout the entire CI/CD pipeline. For making secure, high-velocity software delivery a consistent reality, platforms that bring together different security and testing capabilities and make them work smoothly within the existing DevOps ecosystem are becoming absolutely essential.

June 11, 2025
Perforce AI Testing Tool Eliminates Need of Scripts

Perforce Software this week added an ability to use artificial intelligence (AI) in place of scripts to test mobile and web applications using its Perfecto platform.

Stephen Feloney, vice president of product management at Perforce, said the AI Validation capability visually inspects applications rather than simply creating an AI copilot to generate scripts. That approach enables the Perfecto platform to continuously adapt to changes to objects in applications without either an AI tool or a human tester having to create additional scripts, he added.

The AI Validation tool made available by Perforce employs multiple large language models (LLMs) to inspect applications, which are accessed via a natural language interface. Those LLMs are even able to verify the meaning behind dynamic elements, such as charts, dashboards, or graphics, to ensure user experiences reflect the actual intent of the content being surfaced, noted Feloney.

The overall goal is to make it simpler for application developers to launch tests as they are writing code rather than having to wait for a separate testing team to generate scripts and then run a test, said Feloney.

That’s critical because manually updating scripts and object locators as applications are updated frequently results in the test as originally constructed failing to run at all, he added.

DevOps teams will also no longer need to create and maintain a library of testing scripts, most of which are never used by developers simply because either they don’t know they exist or they lack the skills and expertise to invoke them, said Feloney.

Perforce in the months ahead intends to extend the reach and scope of the AI capabilities of its testing platform to advance autonomous testing, make it simpler to create tests using low-code tools, provide suggestions for tests, pinpoint issues using a simple dashboard and even build suites of regression tests that validate the relationship between objects in an application.

Many of the regression tests that DevOps teams run today generally fail as applications are continuously updated, noted Feloney.

It’s not clear what impact AI will ultimately have on the need for a separate team to test applications but it’s not likely that AI will eliminate the need for those roles anytime soon, said Feloney. However, AI will ultimately improve the quality of applications as it becomes simpler for application developers to not only automatically run most tests themselves, but also remediate issues using advice surfaced by an AI tool, he added.

At this point, the AI genie is not going back into the bottle. The challenge now is determining how best to imply these capabilities. There’s little doubt that AI can be used to write more code faster but if the quality of that code isn’t good enough to run in a production environment there’s not enough value to justify the investment. In fact, instead of first investing in AI to create more code, DevOps teams would be better served by prioritizing investments in AI tools that improve the quality of the code being created.

January 30, 2025
Faster and Better Testing?

The whole concept of Agile and DevOps was to iterate development faster and deliver results in a more timely manner. As we learned more about both methodologies, processes and policies were put into place that improved the quality of what was created. Early ideas of quality like, “We can just do another iteration,” still exist, but are largely not used. Rollbacks are a thing in most orgs, and though the environment is more complex, rollbacks are easier to manage in the smaller development changes that iterations generally represent.

And once we got past the whole, “Anything that slows the process down is bad!” mentality that permeated early implementations of both DevOps and Agile development, we started seeing amazing things. But including things like security and functional testing were not viable in the fast-iteration world that Agile and DevOps created. And they really weren’t viable at most orgs in the slow-iteration world of waterfall, either. Staffing and time constraints made them fall far short of optimal in the vast majority of organizations. But putting them into the DevOps process challenged vendors to find a way to do things faster and challenged organizations to look at quality and security a little differently.

And now, we have tools that can test—be it functional, integration or security—at the speed of DevOps. Tools that enable Agile development with an eye toward security. A far larger number of developers now acknowledge their part in application security and are creating more secure code.

Keep an eye on how AI/ML is being used in testing—all forms of testing—and how it evolves in the next year or two. The same with full-on automation. I’ve made no secret that I view AI/ML in the testing space as advanced automation because of what it enables, so maybe “Watch what AI/ML-enabled automation is available,” is more accurate. The ability to create, run, process results and even modify code to resolve issues are all out there today, and getting better. Faster, more accurate, more automated.

Much like back when intrusion detection became intrusion prevention, I would not just turn it on and walk away, but think about the ability to upskill developers; “Here is a list of things the automated system found, and a list of fixes it proposes.” They are literally being shown how to do their craft better.

I’ve been a developer, so I’m not being offensive, here. If you get something that tells you, “This is an issue; here is why it happened and here is what to do to fix it,”take the opportunity to get better at coding. Look closely at the what, the why and how to fix it and incorporate that into your work.

And keep rocking it. We’ll be making more stable and secure code, but you’ve kept systems running without that feedback and fixed the errors along the way. This just makes it easier to find the errors early—before they crash production and make a fire drill.

November 30, 2022
ServiceNow Acquires DotWalk to Gain AI for App Testing

ServiceNow today revealed it has acquired DotWalk to gain an artificial intelligence (AI)-infused application testing capability that runs natively on its software-as-a-service (SaaS) platform.

Joe Davis, senior vice president for platform engineering at ServiceNow, said that testing capability will be made available free of charge to all organizations that build custom applications on the company’s platform.

The number of custom applications being built on the ServiceNow platform has risen sharply since the start of the pandemic. Many organizations have taken advantage of the low-code tools provided by ServiceNow to build a range of applications that use the company’s SaaS applications as a foundation. That approach enables them to more quickly build applications that drive a wide range of digital business transformation initiatives.

Davis said ServiceNow is committed to enabling organizations to improve the quality of those applications using a DotWalk testing capability that eliminates the need for developers to build and run tests themselves or for organizations to hire dedicated application testing teams.

ServiceNow decided to acquire DotWalk because the company built a testing platform using machine learning algorithms specifically for the ServiceNow platform, noted Davis. The DotWalk tools also analyze logs in the background to identify and validate error sources that can arise during any application upgrade process.

While there are a large number of professional developers that build applications on the ServiceNow platform, the company is also trying to encourage more so-called citizen developers to build applications. The challenge is many of the end users building applications lack an appreciation for design, security and what it takes to make sure an application scales as more users employ it. Testing tools infused with AI should make it easier for developers, regardless of experience, to build enterprise-class applications.

Of course, professional developers will also benefit from AI-infused testing tools. Many developers have a tendency to cut short the application development process as they rush to meet application delivery deadlines. The end result, however, is that an application must be updated almost immediately after it has been deployed in a production environment. Testing tools that make use of AI will enable professional developers to more easily run tests they don’t have to build and then maintain, said Davis.

One way or another, the number of custom applications built on SaaS platforms will increase as organizations digitize more processes. In the last year, more organizations have become adept at extending these platforms using a wide range of low-code tools that automate many aspects of application development. A significant percentage of those organizations are now also adopting DevOps best practices to better manage the pace at which those applications are developed and deployed.

In the meantime, it’s too early to say precisely how AI will transform application testing. However, one thing that is certain is that even as the number of custom apps grows, applications will be more thoroughly tested before being deployed in a production environment.

November 23, 2021