Tag: AI coding

The cost curve for generating code with AI has moved in one direction, and it has moved fast.

What used to require a senior engineer’s full attention for an afternoon can now be scaffolded in minutes, and the price per token for the models doing it keeps falling. By the metric that dominated every early AI productivity argument, the cost of generation, the economics are undeniably favorable and getting more so.

But production deployments have surfaced a different cost that the token price does not capture and that the demo environment was never designed to reveal. The cost of giving the model the right context to generate something that is actually correct, maintainable, and safe to ship is not falling. In many organizations it is rising, and the engineering infrastructure required to manage it reliably does not exist yet in most teams that need it.

Context engineering has become one of the more discussed terms in AI infrastructure circles, and the discussion has accumulated enough hype that it is worth being precise about what the problem is before examining how practitioners are solving it. The model is not the bottleneck in most underperforming production AI systems. But the retrieval system feeding it context is.

And the retrieval system is underperforming not because the technology is immature but because most teams treated it as infrastructure to configure once rather than a system to engineer and maintain with the same discipline they apply to the application layer it is supporting.

The Gap Between What the Model Can Do and What It Actually Gets

Itamar Friedman, CEO and co-founder of Qodo, makes a distinction that cuts to the heart of why context engineering has become a discipline in its own right rather than a feature of retrieval tooling. A context engine that merely stores and fetches information is a memory system. What production AI applications actually need is something closer to a brain, a system that does not just retrieve what was asked for but understands the relationships between pieces of information, recognizes when something retrieved is about to become outdated, and can surface what is relevant without being told exactly what to look for.

The practical difference between those two models shows up most clearly in how the system handles evolving information. “Maybe the memory remembers that these are all the APIs that were used in the last ten years,” Friedman explains. “But maybe it also remembers that this API is about to be deprecated and should be changed to another API. So the difference between a context engine memory and a brain is that when you ask a question about an API it is also able to infer that you are asking about this, and by the way, this is about to be deprecated and you should actually ask about this other one.”

The specificity argument that Friedman makes from this distinction is one of the more important counterintuitive positions in the current context engineering conversation. Companies claiming to have built a generic context engine that can answer any question about any domain have, if the claim is taken seriously, built something close to artificial general intelligence, because that is what it would actually require to retrieve the right context reliably across all possible domains without domain-specific structure.

The companies getting real value from context engineering are the ones building context engines that are deeply specific to a single use case, with data structures, tool architectures, and retrieval logic designed around the particular kinds of questions that use case generates. Friedman acknowledges that Qodo has made exactly this trade-off. “If you try to use the Qodo context engine for health or customer support, that is what I call a problem,” he says plainly. Specificity is not a limitation of a well-built context engine. It is the engineering choice that makes it actually work.

The Two Problems Teams Are Conflating

Carlos Juzarte Rolo, Open Source Contributions Manager at NetApp, approaches the context problem from the data infrastructure side and arrives at a diagnosis that practitioners who have built production RAG systems will recognize immediately. The reason context quality degrades in production is almost never that the generation model is wrong. It is that the retrieval step upstream of the model is failing to surface the right content, and teams are debugging the model rather than the retrieval because the model’s output is what they can see.

“RAG is a two-step problem,” Rolo explains. “You have to fix the search and then you have to give the context in the correct order to your LLM.” The two steps have different failure modes that require different solutions, and treating them as a single pipeline to tune holistically means the interventions that would fix the retrieval problem are not being tried because the symptoms look like a generation problem.

Rolo’s own experience is illustrative. Returning to keyword search alongside vector search and applying it to precise term retrieval improved his retrieval quality significantly. Not because keyword search is superior to vector search, but because they solve different problems and the retrieval layer needs both rather than choosing one as the universal answer.

The context assembly step has its own failure mode that is distinct from retrieval quality and equally important. Models given too much context, even when that context contains the right information somewhere inside it, lose coherence in ways that are hard to distinguish from model limitations rather than context management failures.

“Context rot” describes the degradation in model reasoning quality that occurs when the context window is filled with content that is marginally related rather than precisely relevant, and Rolo has observed it across multiple production systems. “Having a big context window with low quality, even if somewhere within the low-quality context there is the context you need, is not a good technique,” he argues. “The content that you need and much more that is not relevant is actually maybe as bad as having less content than you need but less garbage.”

The counterintuitive implication is that a smaller, more precise context often produces better model output than a larger, more comprehensive one, and the retrieval architecture should be optimizing for precision rather than recall at the context assembly stage.

Where Enterprise Adoption Actually Stalls

That retrieval quality problem is exactly what surfaces at scale. Mudit Singh, Co-Founder and Head of Growth at TestMu AI, observes the context engineering problem from the vantage point of enterprise adoption, and what he sees is a pattern that the tooling vendors are not positioned to acknowledge. Teams invest heavily in model selection and almost nothing in retrieval quality, and then spend far more time debugging the resulting failures than they saved by moving fast on the model choice.

The model is the visible, comparable, marketable component of the AI stack. The retrieval infrastructure is the invisible component that determines whether the model can deliver on what the selection process promised.

“Context engineering is where the difference between AI that impresses in a demo and AI that is reliable in production gets decided,” Singh explains, “and most enterprises are still treating it as an afterthought rather than a core engineering discipline.”

The consequences arrive months into deployment when edge cases that the evaluation benchmarks never covered start appearing in production and the retrieval system has no mechanism to handle them because it was built to perform well against the test set rather than to maintain quality against a shifting distribution. At TestMu AI, this pattern shows up consistently in enterprise teams that have adopted AI tools aggressively and then found themselves spending more time debugging than they saved by moving fast on the initial implementation.

The ongoing maintenance cost of a retrieval system is the part of the context engineering investment that organizations systematically underestimate because it does not appear in the build budget. The underlying data keeps changing. The distribution of questions the system receives evolves as users discover what the system can and cannot handle. The model itself gets updated in ways that change how it responds to the same context.

Each of these changes can degrade retrieval quality without triggering any alert that the engineering team has instrumented for, because the system is still functioning and the degradation is gradual rather than sudden.

“The teams getting durable value from AI pipelines are the ones that treat context engineering as an ongoing discipline with dedicated ownership, regular retrieval quality benchmarking, and a clear process for updating the pipeline when the data distribution shifts,” Singh argues.

That commitment does not fit into a sprint or a launch milestone. It is an operational posture that has to be established and maintained over the lifetime of the system.

The Infrastructure Decision Underneath It All

The database architecture supporting a context engine is where the abstraction of context engineering meets the concrete reality of what engineering teams can actually build and maintain. Rolo’s guidance here is grounded in operational reality rather than architectural idealism.

“The best advice in the database world is the database you already have,” he argues. “If you already have a database and that database supports vectors, start there.”

The organizational cost of adopting new infrastructure while simultaneously building a production AI system is real and frequently underestimated, and the performance advantage of a purpose-built vector database over a mature general-purpose database with vector support is often smaller than the switching cost that choosing the purpose-built option would require.

Friedman’s approach at Qodo uses graph databases as the primary substrate for the context engine because the relationships between code artifacts, pull requests, review decisions, and development history are fundamentally relational rather than purely semantic.

A graph structure can represent the dependency between an API and the PR that introduced it, the connection between a review comment and the code change it was responding to, and the deprecation status of a function relative to the version of the codebase being queried, in ways that a vector store alone cannot.

Multiple specialized tools sit on top of that graph structure, each one designed to answer a different category of question, and a routing agent directs queries to the tool best suited to answer them rather than running every query through a single retrieval pipeline.

“We have an ingestion mechanism where we take metadata and turn it into multiple databases that are mostly graphs and vector DBs,” Friedman explains, “and then we have a query layer which takes a query and has a first routing agent that uses multiple tools that each one of them might be agentic.”

The architecture Friedman describes is substantially more complex than a standard RAG implementation, and that complexity is justified by the specificity of what it is trying to accomplish. A context engine built for code quality and review operates in a domain where the relationships between artifacts matter as much as the content of the artifacts themselves, and a retrieval architecture that cannot represent those relationships cannot serve the use case reliably regardless of how good the underlying model is.

The engineering investment required to build it is significant. But it is the investment that closes the gap between a demo that impresses and a system that is still performing reliably in production a year after launch, and organizations that treat context engineering as a configuration step rather than an engineering discipline will keep rediscovering that gap long after their initial deployment.