AI-powered patching and coding is putting us on a track towards a deepening dependency on future frontier models.
The modern tools we use today to fix yesterday’s coding problems are quietly creating tomorrow’s quagmire. Companies are using AI to pay down 20 years of tech debt, but in doing so we’re generating a future problem at a scale that humans alone will be unequipped to handle.
Let me explain how this could play out. Powerful new frontier LLMs like Mythos and Trusted Access are discovering bugs and misconfigurations that have been hiding in plain sight for years. Suddenly, your network appears a lot more exposed than you realized. Your to-do list just got a lot longer. And everything on it is marked “urgent.” No more summer Fridays.
To plug the leaks, many enterprises are turning to the same AI models that discovered them in the first place. AI works fast and tirelessly. It doesn’t take lunch breaks and doesn’t know about summer Fridays. It’s the quickest way to fix an urgent problem.
But here’s the rub. AI isn’t an elite software developer or engineer. In fact, it’s pretty average. Why? Because it was trained on the totality of public data, some of which is good and some of which is not. At the midpoint of all that is AI, a massively productive average programmer without the self-awareness to know it’s an average programmer.
That means AI makes all the same mistakes as any other average programmer, but it’s also prone to hallucinations, has the confidence of a robot, and the prolificacy of a rabbit. That’s a dangerous combination. AI is essentially introducing sneaky new bugs at a scale we’ve never experienced before — and without any clue that there could be any flaws in its work.
A Black Box Full of Tomorrow’s Problems
Experts claim AI tools are enabling developers to output 10-30% more code. Others put the estimate much higher. But it’s quantity over quality. Research published last year by Veracode, a cloud-based application risk management and security testing platform, tested 80 coding tasks across 100 LLM models and found that roughly 45% of AI-generated outputs introduced at least one known security flaw, even when the code behaved as expected.
The bug-injection rate has come down slightly since then, but you can see the problem.
The sheer volume of new AI-powered code means human engineers can no longer realistically review it line by line. Instead, the industry is increasingly validating AI-generated code through testing rather than understanding it fully — treating AI code as a black box and simply checking if the inputs match the outputs.
That’s a subtle but significant shift. For decades, software development operated on the assumption that there was a human architect who had built and tested the system. Engineers could show their math and explain why code worked. But in a world that moves at the speed of AI-generated code, that’s increasingly no longer the case. We’re now in a position where software is accepted because it “appears” to work, without being fully tested or vetted and oftentimes without much human oversight.
That distinction matters. Most technical debt isn’t caused by software that fails immediately. It’s caused by software that’s misconfigured or designed with hidden vulnerabilities that makes it difficult to secure or update later.
Engineers know this. Developers’ trust in AI-generated output dropped to 29% last year, down 11 points from 2024, according to Loopstudio’s 2026 State of AI in Software Development Report. The same report found that 46% of developers actively distrust the accuracy of AI tool output, up from 31% in 2024.
At the same time, demand for new AI technology keeps growing — even by those who mistrust it. The report found that 84% of developers use or plan to use AI models in their workflows this year, despite their deepening mistrust for tools they increasingly rely on.
Developers know that if we look inside the black box of AI coding, what we’ll likely find is tomorrow’s problems — many of which won’t be discovered until the next generation of frontier models can find the problems its former self created.
There’s another wrinkle to this scenario. Due to the massive volume of AI-generated code that is being produced today, the AI models of the future will likely be trained on the data that AI is generating today. The beast is starting to learn from its own work, and how that plays out is still unclear.
What is clear is that AI-coding is the camel’s nose under the tent. Today’s convenience — this marvel of modern technology — is working its way into the system. And once it’s fully inside, we may find ourselves dependent on the next generation of AI to understand, secure and maintain the software the current generation helped create.
The Prisoner’s Dilemma
The challenge going forward is to keep human engineers in the mix. It’s a daunting proposition in a world where Microsoft’s CTO has boldly predicted that 95% of code will be AI-generated by 2030. But if every company replaces junior engineers with AI frontier models today, who will be left to oversee the robots’ work in the future?
Still, many enterprises are playing the short game. Companies look at the bottom line and stop hiring junior developers. Why pay for entry-level talent when AI can handle the average grunt work faster? Consequently, veteran engineers who can spot AI hallucinations in real time are suddenly worth their weight in gold.
But what happens when the old-timers age out and retire? Who is left to supervise the robots 20 years from now if we break the global talent pipeline today?
We need to agree on a new business model that keeps human engineers in the mix — even if there is a faster and less expensive way of doing things today. Hiring and training junior developers should be viewed as the cost of doing business. It’s an insurance policy against future robot rule.
But in a market economy, the challenge is one of short-term incentives.
Every company benefits from a future workforce capable of understanding and securing increasingly AI-generated systems. But each individual company has an incentive to let someone else pay for developing that workforce, like a minor league farm system from which to poach talent in the future.
That’s the prisoner’s dilemma.
The companies racing fastest toward full AI automation may be the ones most dependent on their competitors to preserve the human expertise the whole global industry will eventually need.
