Much has been said about the adoption of coding copilots for developers. What is lesser known is that copilots can also write test scripts and translate tests from one script language to another. Copilots boost productivity for both use cases but are generally passive tools that must be fed precise requirements to get the desired results.
A new level of assistant is emerging — which takes this capability to a more active state — as AI technology is embedded directly into development and testing tools.
A QA manager from a leading tech firm put it best, “The ability of AI to not just run tests, but to learn from them and adapt, is transforming how we approach quality assurance. It’s like having an ever-evolving team member who ensures our releases are robust and reliable.”
These testing agents are beginning to appear in the wild and are bound to evolve rapidly over the next few months and years. They will address some of the major challenges enterprises face in business application development.
Crafting a Test Plan
The first challenge is that the people who understand what the app is supposed to do are generally not skilled in developing tests. The new testing agents will address this challenge by working with subject matter experts (SMEs) to craft a test plan based on the SMEs’ exploration of new features.
The second challenge is the fact that tests are not one-and-done. The first version of a test will only last so long before it will require updating. To paraphrase von Moltke the Elder, ‘Functional test plans will not survive first contact with a platform upgrade’, especially for enterprise SaaS. For traditional test development teams that means there is a constant backlog of rework. But testing agents can regenerate a new version of the test as easily as they wrote the first one.
The third challenge is running the right tests at the right time. The best companies create regression suites that can test most of the important features of their business applications before releasing them.
Everything is well and good until you realize 90% of those tests could not possibly fail based on the scope of changes that were made. This overkill costs both time and money. Wouldn’t it be great if your testing system was smart enough to know which tests should be run and which could be skipped? Testing agents that are properly integrated into the CI/CD system would be able to make that judgment.
Minimizing the Risks
By addressing these challenges, testing agents effectively alter the ROI calculation for test automation. Tests created by the agents are less expensive to maintain, thus reducing investment. They only run when needed, further reducing cost. Since the SME is involved in the creation process, tests are more likely to catch the most critical problems, thereby minimizing the risk of app downtime. Win, win and win.
All this sounds great, but there must be a downside. What are the risks?
The short-term risk is in thinking that a testing agent is fully automatic. We may get there soon, but in the near term it is best to think of these agents as a “team member who ensures our releases are robust and reliable.” The humans in the loop have a responsibility to manage the process and ensure the right outcome. Even the best LLMs still hallucinate on occasion, so until these agents establish a track record, trust but verify.