JetBrains IDEs are where a lot of enterprise backend work happens — Java, Kotlin, Spring, big monorepos with a lot to break. So when an AI coding agent starts running shell commands inside that environment, the question isn’t whether it’s useful. It’s what happens if it touches the wrong file, the wrong network path, or a credential it shouldn’t have.
GitHub’s answer, rolled out this week, gives administrators direct control over that boundary. Enterprise-managed sandbox policies for Copilot in JetBrains are now in public preview, and they let IT teams centrally set what a sandboxed Copilot session can and can’t do: whether sandboxing runs at all, what filesystem and network paths it can reach, which proxy it routes through, whether it can touch developer tools, and — on Mac — whether it gets access to the Keychain. Once an admin locks a setting, it shows up in the JetBrains plugin marked “(managed),” and developers can’t override it locally.
This isn’t a new idea for GitHub. Local and cloud sandboxes for Copilot CLI went into public preview back in June, giving developers a way to restrict what an agent could reach on their machine (/sandbox enable) or offload heavier tasks to an ephemeral cloud Linux box instead. Enterprises could already enforce some of that through Intune or other device management platforms. What’s changed is where the control now lives. Instead of routing sandbox policy through general-purpose MDM tools, GitHub is building it directly into the Copilot plugin’s managed settings — the same mechanism it introduced in August for governing plugins, MCP server allowlists, and telemetry routing in JetBrains. Sandbox behavior joins that list as another lever administrators can pull without touching the operating system layer.
That’s a meaningful distinction for platform and security teams. MDM policy is broad and often slow to change. A setting baked into the tool itself, distributed through the same channel as everything else Copilot-related, lets a platform team adjust it in an afternoon instead of routing through a device management change request.
The release includes a few other pieces worth flagging alongside the sandbox controls. A policy diagnostics tool lets admins verify their managed settings are being detected and enforced on developer machines — useful, since a policy nobody can confirm is running is close to no policy at all. Copilot’s next-edit suggestions can now jump across files instead of stopping at the one you’re in, which matters for changes that touch multiple layers of a codebase. Chat can pull in project-wide context instead of requiring developers to attach files one at a time. And the Copilot CLI’s /ide command now connects a terminal session directly to JetBrains context, so a command-line agent session can see the same selections and diagnostics the IDE sees. OpenTelemetry settings for Copilot Chat, previously managed-only, are now available to everyone.
Mitch Ashley, vice president and practice lead for CIO & Technology Buyers and Software Lifecycle Engineering at The Futurum Group, sees the sandbox policy as part of a bigger shift in how enterprises need to think about AI coding tools.
“Once an agent can execute commands, edit across files, and reach the network on its own, the IDE isn’t just an editor anymore — it’s a runtime that needs the same access controls as any other system touching production code,” Ashley said. “Building sandbox policy into the tool itself, rather than leaving it entirely to device management, is what lets security teams actually verify what’s enforced instead of assuming it. That verification piece is the part most organizations still skip.”
That verification point lines up with what GitHub shipped alongside the sandbox controls. A diagnostics tool that confirms policy enforcement is really an admission that trust-but-don’t-verify doesn’t hold up once an agent is running shell commands on its own. Teams that have spent the last few months turning on Copilot’s agentic features — autopilot, bypass-approval modes, cloud sandboxes — now have a way to check that the boundaries they set are actually in place, not just configured.
None of this is dramatic if you’ve been watching GitHub’s Copilot governance work over the summer. Managed plugin lists in July, MCP server allowlists and telemetry routing in August, and now sandbox policy in September — it’s the same pattern applied to a new surface each time. The through-line is that agentic AI features keep shipping faster than the controls around them, and GitHub keeps circling back to close that gap one setting at a time.
For engineering and platform teams running JetBrains at scale, the practical move is straightforward: don’t treat this as a JetBrains plugin update to skim past. Pull up the policy diagnostics tool before turning on any of Copilot’s more autonomous modes, decide who owns the sandbox filesystem and network settings, and confirm those decisions are actually landing on developer machines rather than sitting in a config file nobody checked. The tools to do that now exist. Whether organizations use them before or after something goes wrong is the part still up to them.
Enterprise AI agents need more than stronger models. They need durable execution environments that can coordinate multi-step workflows, survive failures, pause for human review and resume reliably after disconnects or delays.
AI Agents Have Moved Beyond Chat Demos
AI agents work beautifully in demos. A user asks a question, a large language model generates a response and everyone sees possibility. Production systems are different.
Consider what these systems are now asked to do. An agent assessing a change request may pull deployment history, evaluate blast radius, check freeze-window policy, wait for a release manager’s sign-off and then schedule the rollout. An agent adjudicating an insurance claim may extract fields from submitted documents, cross-check them against prior claims, apply underwriting rules and route anything unusual to a human adjuster. An agent triaging a security alert may enrich indicators, correlate against past incidents, assess asset criticality and hold containment until an analyst approves it.
Those three workflows have almost nothing in common at the domain level. They share a shape: Multiple steps, several systems, independent failure modes and at least one point where a person decides. They take minutes or hours, not milliseconds.
The challenge is no longer only intelligence; it is execution. Many enterprise AI conversations focus on model selection, prompt engineering, retrieval-augmented generation and tool calling. Those are important, but they do not answer what happens after the agent begins executing long-running work across multiple systems.
The Problem: Most Agents Are Still Designed Like Request-Response Applications
Traditional AI applications assume a simple request-response model: User input goes in; a model response comes out. That model works well for Q&A, summarization, content generation and search-like experiences.
Enterprise workflows are different. A procurement review agent might parse a contract, compare its terms against a standard template, check vendor risk ratings, invoke a sanctions screening service and produce a recommendation. These steps may execute independently, fail independently and complete at different times.
Once an agent crosses the boundary from answering a question to executing a process, it starts to look less like a chatbot and more like a distributed system. It needs state, retries, coordination, observability and recovery.
Why Stateless APIs Break Down
Teams often start by implementing an agent as a synchronous API. That approach works until the first real workflow appears. What happens if the HTTP request times out? What if the user closes the browser? What if an external system is unavailable? What if a human approval arrives six hours later?
A stateless API can initiate work, but it is a poor place to manage long-running state. Developers quickly end up building custom queues, status tables, retry logic, compensation logic and ad hoc workflow tracking. The hidden cost is not the first prototype; it is operating the system after it begins handling real workloads.
The missing layer is a durable runtime: A workflow engine that can preserve progress, coordinate parallel work, wait for external events and resume execution reliably.
Figure 1: The Same Multi-Step Workflow Under Two Runtimes
A stateless service holds progress in process memory, so a host failure discards completed work and the retry begins again from the first step. A durable runtime checkpoints each completed activity, so a replacement worker replays the history and continues from where execution stopped.
The Key Design Shift: the Workflow is the Application
The most important design shift is this: The model is not the application. The workflow is the application. The model is one activity inside a larger execution graph.
In production-grade agentic systems, the orchestrator coordinates work while specialized agents perform specific tasks. The specialization is domain-specific, but the division of labor recurs: One agent gathers and enriches raw signal, another retrieves reference material, another looks for precedent among prior cases and another evaluates policy or risk conditions.
This design makes the system easier to reason about. Each agent has a focused responsibility, and the orchestration layer owns execution flow, state, retries, progress and human interaction.
Using Durable Orchestration as the Agent Runtime
A practical architecture places a durable orchestrator between the calling application and the AI execution layer. The caller starts the workflow and immediately receives an instance identifier. The orchestrator then coordinates specialized agents and records progress as the workflow advances.
Azure Durable Functions is a strong fit for this pattern because it supports stateful workflows, activity functions, external events, checkpointing and long-running orchestration. Azure AI Foundry can provide the model and agent execution layer, while Durable Functions handles the operational execution semantics around the agent.
The result is a cleaner separation: AI services reason; activity functions perform work; the durable orchestrator governs the process.
Figure 2: The Orchestrator Between the Calling Application and the Reasoning Layer
It owns sequencing, retries, timers and external events. Specialized agents run on the model layer with narrowly scoped access and every completed activity is checkpointed so the workflow can be rebuilt after a failure or a process restart.
Fan-Out/Fan-In Maps Naturally to Multi-Agent Systems
These investigations are rarely one sequential task. Many subtasks can run independently. Enrichment, reference retrieval, precedent search and policy evaluation usually have no dependency on one another, so they can execute in parallel. This is where fan-out/fan-in becomes a natural orchestration pattern.
The orchestrator fans out work to specialized agents, waits for them to complete, and then fans in the results to generate a synthesized recommendation. This pattern improves latency while keeping the workflow easier to understand than a large monolithic agent prompt.
The fan-in step is also where engineering judgment matters. The system should handle partial results, conflicting findings and low-confidence recommendations rather than assuming every agent will succeed perfectly every time.
Example Orchestration Sketch: The sketch below triages a security alert; the domain is incidental. Rename the activities and the same orchestration serves claims adjudication or change-risk review.
Three details in that sketch are worth calling out. The external event is raced against a durable timer, so an approval that never arrives escalates instead of leaving the instance parked indefinitely. The timer is then cancelled on the success path, because a durable timer that is neither cancelled nor allowed to expire keeps the instance in the Running state long after the orchestrator function has returned. Since Task.WhenAll surfaces the first exception it encounters, genuine partial-success handling means having each activity return a typed failure result rather than throwing it once its retries are exhausted.
Human-in-the-Loop is Not an Edge Case
In regulated and high-impact workflows, human review is part of the process by design. A payment exception above a threshold, a production change during a freeze window, a claim falling outside underwriting rules: None of these should be finalized automatically when confidence is low or the consequences of being wrong are large. The important thing is to make human review part of the workflow rather than a side channel.
A durable orchestration can pause while waiting for a review event, then resume when the approver submits a decision. The system does not need to keep an HTTP request open or hold compute resources while waiting. The workflow state remains available, and the final recommendation can include the reviewer’s context.
This capability is especially important because the slowest step in a production workflow is often not the model call. It is the human decision, the external dependency or the operational handoff.
Figure 3: One Instance Over Time
Parallel agents make wall-clock cost the slowest agent rather than the sum of all of them; a failed agent degrades the result instead of failing the run, and the review window consumes no compute because the orchestration is parked rather than blocked.
Operational Considerations Matter as Much as Prompts
A durable runtime does not remove the need for engineering discipline. It makes that discipline easier to apply. Production-grade agents still need observability, correlation identifiers, retry policies, cost controls, security boundaries and careful versioning.
Workflow telemetry should answer questions such as: Which stage is running? Which activity failed? How many retries occurred? How long did human review take? What evidence supported the final recommendation? These are operational questions, not prompt engineering questions.
Retries should be designed around external dependencies such as AI model calls, search systems, databases and APIs. At the same time, not every failure should fail the entire workflow. Multi-agent systems should be designed for partial success where appropriate. If the precedent agent fails but enrichment and retrieval succeed, the system may still produce a useful recommendation with clear caveats.
Three constraints tend to surface only once a workflow reaches production. Orchestrator code has to stay deterministic, because recovery works by replaying it: Wall-clock reads, random values, fresh identifiers and direct I/O belong in activities, not in the orchestrator. Every wait needs a deadline, else an approval that never arrives becomes an instance nobody ever looks at again. Versioning deserves a decision before the first production deploy because long-running instances guarantee that new orchestrator code will ship while old instances are still in flight.
Security and Governance Cannot Be Bolted on Later
AI agents often sit at the crossroads of sensitive enterprise data, operational tooling and user-facing experiences. That makes security and governance foundational design concerns.
Each specialized agent should receive only the access necessary for its responsibility. A retrieval agent may need the document corpus and nothing else. An enrichment agent may need read access to telemetry. The agent that composes the final decision may need only summarized evidence, never the raw sources behind it. Avoid creating one over-permissioned super-agent that can reach everything.
Prompts and outputs should also be auditable. Store workflow version, prompt version, model deployment, agent version, evidence references and correlation identifiers. As agents become part of operational decision-making, teams need to explain not only what the system recommended but why it recommended it.
Figure 4: Scoping Access per Agent Contains the Blast Radius
A single agent holding every credential turns one poisoned input into estate-wide exposure, while narrowly scoped agents limit any single compromise to one agent’s data, with the synthesis step working from summarized evidence.
What This Means for DevOps and Platform Engineering Teams
The future of enterprise AI is not just better prompts. It is reliable execution. As agents become responsible for operational workflows, DevOps and platform engineering teams will need to provide runtime capabilities similar to those used in distributed systems: Orchestration, observability, resiliency, governance and deployment discipline.
This resembles the evolution of microservices. Early conversations focused on APIs. Later, the industry learned that production systems also require service discovery, telemetry, deployment strategies, circuit breakers and operational practices. AI agents are entering a similar phase.
Organizations that treat agents as production workflows rather than chat interfaces will be better positioned to build systems that users trust.
Conclusion
Most enterprises already have access to powerful foundation models. The harder challenge is building systems that can execute reliably when work spans minutes, hours, tools, services and human decisions.
Long-running agents of any kind need a durable execution layer. They need to coordinate multiple specialized agents, survive transient failures, pause for human review, provide progress visibility and generate auditable outcomes.
Azure Durable Functions and Azure AI Foundry offer one practical way to approach this architecture: Durable Functions provides orchestration and stateful execution, while Azure AI Foundry provides the intelligence layer. The broader lesson applies beyond any specific platform: Production AI agents need durable runtimes. Intelligence is only half of the system. Execution is the other half.
The gap between how fast AI writes code and how reliably teams verify it is quietly becoming a balance-sheet risk. Enterprise leaders are learning to measure it before it forces repayment.
Every engineering organization that adopted AI coding assistants over the past two years has quietly taken on a liability that never appears on the dashboards leadership actually reviews. Developers now generate more code than their teams can meaningfully verify, and the gap between the two compounds on every passing build until an incident forces the account to settle.
The cost tends to land late and all at once, usually as an outage or a failed audit that traces back to a change nobody fully checked. For the executives who answer for delivery risk, the trouble is not the tooling their engineers reach for but the plain fact that verification capacity never scaled to match the new pace of output.
The Liability That Never Reaches the Dashboard
Leadership reporting evolved to track the things that have always mattered, delivery speed, defect counts, and uptime, and none of it shows how much unverified code a team ships in a given week. That blind spot is where validation debt takes hold, because a risk that never surfaces in the numbers is a risk leadership never thinks to question.
MuditSingh, Co-Founder and Head of Growth at TestMUAI(formerlyLambdaTest), sees this pattern repeat across the enterprise engineering teams he meets on the company’s customer events. He observes that output jumps the moment a team adopts AI assistance while verification stays flat, and leaders rarely register the widening gap until something breaks, because nothing in their regular reporting makes the debt visible. What the team learned building KaneAI, the company’s testing agent, is that the shortfall is almost never about talent and almost always about visibility, since engineers can only guard against risks they can actually see.
The broader argument Mudit presses is organizational rather than technical, because someone has to own validation coverage as a named responsibility instead of letting it spread thinly across a busy team. His recommendation for leaders is to track test escape rate as a reporting number they watch directly and to treat AI-generated changes as their own visible category, since once a leader assigns that ownership the conversation moves from whether the debt exists to how fast the organization intends to bring it down.
Evidence Becomes the Standard in Regulated Work
In regulated industries the cost of an unverified change is not measured in a quick hotfix but in an audit finding that can surface months after the code shipped. A passing test suite offers little protection when the real question is whether anyone can prove, long after the fact, that the team ever checked the behavior that mattered.
KarthikKarunanithi, a Solution Architect at IBM and a Senior Member of IEEE with two granted United States patents in secure health data, works in exactly those settings, where a passing build has never counted as proof of much. An AI-assisted change to an authorization path once shipped with tests that mocked the very layer they should have tested, and the gap surfaced only in the audit logs, when a query ran that policy should have stopped cold. Karunanithi responded by requiring every AI-assisted change to leave behind enough evidence for another engineer to see what the tests exercised and reproduce the result later.
The principle Karunanithi keeps returning to is that a green build tells him the pipeline ran and nothing more, so evidence rather than appearance has to become the standard the moment a missed check carries real cost. Teams that adopt his posture stop asking whether the build passed and start asking what the passing build genuinely proves, which quietly changes how they treat any code they did not write by hand.
Human Review Cannot Scale on a Curve
The instinct when pull request volume explodes is to ask reviewers to work harder, yet no amount of effort closes a gap that is structural rather than personal. A person can read only so much code in a day, and that ceiling does not lift just because a machine started writing several times as much.
AbhilashRaoMesala, a Senior Data Engineer at Meta with fourteen years across healthcare and enterprise systems, watched that ceiling turn into a quality problem in plain sight. On his teams the volume of AI-assisted pull requests climbed until human review could no longer keep pace, and reviewers started approving changes more shallowly even though everyone understood the depth had slipped while the builds stayed green. His group answered by routing low-risk changes to an automated reviewer inside a fixed threshold and reserving scarce human judgment for the changes that carry real consequence.
The framing Mesala offers leaders is deliberately structural, because review capacity grows in a straight line while AI output grows on a curve, and no hiring plan closes that shape of gap. He measures post-merge defects by approval type so leadership gets a clean read on whether the balance between speed and safety is holding, and the same threshold quietly discourages the sprawling changes that tend to hide the most risk.
Agentic Systems Raise the Interest Rate
Everything gets harder the moment the software stops waiting for a human to press each key and starts planning its own steps. An agent that chooses a different path on every run breaks the last comfortable assumption traditional testing rested on, which is that the same input reliably produces the same output.
AhmedZaidi, CEO and Managing Partner at Accelirate, sees the stakes climbing as enterprises move from AI-assisted code into fully agentic systems that act without a human keying each step. Zaidi points out that validation debt compounds faster once behavior stops being deterministic, because a single passing test proves even less than it did before when the system can behave differently the next time it runs.
The recommendation Zaidi lands on for leaders deploying agents is to treat validation as a continuous practice rather than a one-time gate, re-checking behavior on every meaningful change to the prompt, the model, or the data the agent leans on. No one can check a
shifting system a single time and then trust it to behave forever, so validation has to run as often as the system itself changes.
Making the Debt Visible and Payable
Validation debt behaves like any other liability a business carries, which means it stays manageable only when someone measures it, assigns it, and decides how fast to bring it down. The organizations getting ahead of it are not slowing their engineers or backing away from AI assistance in the slightest.
Mudit Singh of TestMU returns to the discipline he opened with, urging leaders to hold every change to the same gates no matter who or what wrote it and to watch defect escape into production as the figure that tells the truth about their exposure. His closing point for executives is that generation will only accelerate from here, so the teams that fund verification as a first-class capability are the ones that keep shipping fast without quietly borrowing against their own reliability.
OpenAI quietly trimmed the default input context window for GPT-5.6 inside its Codex CLI, dropping it from 372,000 tokens to 272,000 tokens. That’s a 27% cut, and developers noticed fast.
The change surfaced in aGitHub pull request to the Codex CLI, and within days, developers on Reddit and X were asking why OpenAI shrunk the window without much explanation. Some argued the smaller context would force Codex into compaction mode sooner during long coding sessions, meaning the agent would summarize or drop earlier conversation history to make room for new work. Others weren’t sure the change would matter much, since the real-world impact depends heavily on project size and how a team structures its workflow.
OpenAI hasn’t published a rationale for the reduction. But the debate it kicked off says less about one vendor’s configuration choice and more about a problem the industry has been circling for a while: context windows look big on paper, and they behave very differently in practice.
Why Bigger Windows Don’t Guarantee Better Results
There’s a growing body of research behind that gap. Studies on long-context model behavior have repeatedly found what researchers call a “lost in the middle” effect, where information placed in the middle of a long input gets less reliable attention than content near the beginning or end. One widely cited study of 18 frontier models found accuracy drops of 30% or more for details buried mid-conversation, and that degradation can start well before a model hits its advertised limit.
For coding agents specifically, that shows up as what practitioners have started calling context rot: the agent doesn’t error out, it just gets quietly worse. It re-introduces bugs it already fixed. It contradicts decisions it made earlier in the session. It sounds confident while working from a picture of the codebase that’s gone stale. None of that requires a smaller context window to happen. But a smaller window means it happens sooner.
That’s the core worry driving developer pushback on the Codex change. A tighter budget means Codex will compact and summarize more often, and every compaction is a chance to lose a decision, a constraint, or a piece of earlier reasoning that mattered.
“Managing the context window budget is a quality, reliability, and productivity reality for developers,” said Mitch Ashley, vice president and practice lead for software lifecycle engineering and AI-native software engineering at The Futurum Group. “Teams running Codex against large repositories will hit compaction sooner, and every compaction risks dropping a constraint the agent already resolved.”
The Real Fix Isn’t a Bigger Number
Analysts tracking the reaction point to a familiar lesson: teams that build workflows assuming a fixed context budget are setting themselves up to get burned every time a vendor adjusts the dial. And vendors will keep adjusting it. Context limits, pricing, and runtime behavior for managed AI coding platforms aren’t stable ground to build on.
The more durable approach is treating context as something to manage actively rather than something to max out. That means breaking large tasks into smaller, bounded units of work. It means leaning on retrieval, so the agent pulls in relevant code and documentation on demand instead of trying to hold an entire repository in memory. It means instrumenting context consumption the same way teams already instrument latency or error rates, so nobody finds out they’re near the ceiling by watching the agent start making mistakes.
Ashley sees the Codex change as part of that broader shift already underway. “Cutting back the context window is a recognition that developers are moving to multi-agent workflows, each operating with a narrow, tighter context,” he said.
That points to a different way of thinking about the problem entirely. Instead of one agent trying to hold an entire session in its head, the work gets split across multiple agents, each with a bounded job and a bounded context. Less to forget, less to lose track of, and less riding on any single vendor’s token count.
What This Means for DevOps Teams
None of this makes Codex, or GPT-5.6, unusable. For routine work, bug fixes, small features, single-file changes, the context cut likely won’t register. The friction shows up in exactly the use case AI coding agents were supposed to be best at: long-running, autonomous work across a large codebase.
That’s worth sitting with for a minute. The pitch behind agentic coding tools has always been that they’d take on the tedious, sprawling work humans don’t want to do manually. If a vendor-side configuration change can meaningfully degrade performance on that exact use case, it’s a signal that the tooling built around these agents needs more resilience than teams have been giving it credit for.
Enterprises don’t need to panic over a token count. But they should treat this as a prompt to check their assumptions, and start planning around multi-agent, narrower-context designs rather than betting everything on one large window that a vendor can shrink at any time.
The cost curve for generating code with AI has moved in one direction, and it has moved fast.
What used to require a senior engineer’s full attention for an afternoon can now be scaffolded in minutes, and the price per token for the models doing it keeps falling. By the metric that dominated every early AI productivity argument, the cost of generation, the economics are undeniably favorable and getting more so.
But production deployments have surfaced a different cost that the token price does not capture and that the demo environment was never designed to reveal. The cost of giving the model the right context to generate something that is actually correct, maintainable, and safe to ship is not falling. In many organizations it is rising, and the engineering infrastructure required to manage it reliably does not exist yet in most teams that need it.
Context engineering has become one of the more discussed terms in AI infrastructure circles, and the discussion has accumulated enough hype that it is worth being precise about what the problem is before examining how practitioners are solving it. The model is not the bottleneck in most underperforming production AI systems. But the retrieval system feeding it context is.
And the retrieval system is underperforming not because the technology is immature but because most teams treated it as infrastructure to configure once rather than a system to engineer and maintain with the same discipline they apply to the application layer it is supporting.
The Gap Between What the Model Can Do and What It Actually Gets
Itamar Friedman, CEO and co-founder of Qodo, makes a distinction that cuts to the heart of why context engineering has become a discipline in its own right rather than a feature of retrieval tooling. A context engine that merely stores and fetches information is a memory system. What production AI applications actually need is something closer to a brain, a system that does not just retrieve what was asked for but understands the relationships between pieces of information, recognizes when something retrieved is about to become outdated, and can surface what is relevant without being told exactly what to look for.
The practical difference between those two models shows up most clearly in how the system handles evolving information. “Maybe the memory remembers that these are all the APIs that were used in the last ten years,” Friedman explains. “But maybe it also remembers that this API is about to be deprecated and should be changed to another API. So the difference between a context engine memory and a brain is that when you ask a question about an API it is also able to infer that you are asking about this, and by the way, this is about to be deprecated and you should actually ask about this other one.”
The specificity argument that Friedman makes from this distinction is one of the more important counterintuitive positions in the current context engineering conversation. Companies claiming to have built a generic context engine that can answer any question about any domain have, if the claim is taken seriously, built something close to artificial general intelligence, because that is what it would actually require to retrieve the right context reliably across all possible domains without domain-specific structure.
The companies getting real value from context engineering are the ones building context engines that are deeply specific to a single use case, with data structures, tool architectures, and retrieval logic designed around the particular kinds of questions that use case generates. Friedman acknowledges that Qodo has made exactly this trade-off. “If you try to use the Qodo context engine for health or customer support, that is what I call a problem,” he says plainly. Specificity is not a limitation of a well-built context engine. It is the engineering choice that makes it actually work.
The Two Problems Teams Are Conflating
Carlos Juzarte Rolo, Open Source Contributions Manager at NetApp, approaches the context problem from the data infrastructure side and arrives at a diagnosis that practitioners who have built production RAG systems will recognize immediately. The reason context quality degrades in production is almost never that the generation model is wrong. It is that the retrieval step upstream of the model is failing to surface the right content, and teams are debugging the model rather than the retrieval because the model’s output is what they can see.
“RAG is a two-step problem,” Rolo explains. “You have to fix the search and then you have to give the context in the correct order to your LLM.” The two steps have different failure modes that require different solutions, and treating them as a single pipeline to tune holistically means the interventions that would fix the retrieval problem are not being tried because the symptoms look like a generation problem.
Rolo’s own experience is illustrative. Returning to keyword search alongside vector search and applying it to precise term retrieval improved his retrieval quality significantly. Not because keyword search is superior to vector search, but because they solve different problems and the retrieval layer needs both rather than choosing one as the universal answer.
The context assembly step has its own failure mode that is distinct from retrieval quality and equally important. Models given too much context, even when that context contains the right information somewhere inside it, lose coherence in ways that are hard to distinguish from model limitations rather than context management failures.
“Context rot” describes the degradation in model reasoning quality that occurs when the context window is filled with content that is marginally related rather than precisely relevant, and Rolo has observed it across multiple production systems. “Having a big context window with low quality, even if somewhere within the low-quality context there is the context you need, is not a good technique,” he argues. “The content that you need and much more that is not relevant is actually maybe as bad as having less content than you need but less garbage.”
The counterintuitive implication is that a smaller, more precise context often produces better model output than a larger, more comprehensive one, and the retrieval architecture should be optimizing for precision rather than recall at the context assembly stage.
Where Enterprise Adoption Actually Stalls
That retrieval quality problem is exactly what surfaces at scale. Mudit Singh, Co-Founder and Head of Growth at TestMu AI, observes the context engineering problem from the vantage point of enterprise adoption, and what he sees is a pattern that the tooling vendors are not positioned to acknowledge. Teams invest heavily in model selection and almost nothing in retrieval quality, and then spend far more time debugging the resulting failures than they saved by moving fast on the model choice.
The model is the visible, comparable, marketable component of the AI stack. The retrieval infrastructure is the invisible component that determines whether the model can deliver on what the selection process promised.
“Context engineering is where the difference between AI that impresses in a demo and AI that is reliable in production gets decided,” Singh explains, “and most enterprises are still treating it as an afterthought rather than a core engineering discipline.”
The consequences arrive months into deployment when edge cases that the evaluation benchmarks never covered start appearing in production and the retrieval system has no mechanism to handle them because it was built to perform well against the test set rather than to maintain quality against a shifting distribution. At TestMu AI, this pattern shows up consistently in enterprise teams that have adopted AI tools aggressively and then found themselves spending more time debugging than they saved by moving fast on the initial implementation.
The ongoing maintenance cost of a retrieval system is the part of the context engineering investment that organizations systematically underestimate because it does not appear in the build budget. The underlying data keeps changing. The distribution of questions the system receives evolves as users discover what the system can and cannot handle. The model itself gets updated in ways that change how it responds to the same context.
Each of these changes can degrade retrieval quality without triggering any alert that the engineering team has instrumented for, because the system is still functioning and the degradation is gradual rather than sudden.
“The teams getting durable value from AI pipelines are the ones that treat context engineering as an ongoing discipline with dedicated ownership, regular retrieval quality benchmarking, and a clear process for updating the pipeline when the data distribution shifts,” Singh argues.
That commitment does not fit into a sprint or a launch milestone. It is an operational posture that has to be established and maintained over the lifetime of the system.
The Infrastructure Decision Underneath It All
The database architecture supporting a context engine is where the abstraction of context engineering meets the concrete reality of what engineering teams can actually build and maintain. Rolo’s guidance here is grounded in operational reality rather than architectural idealism.
“The best advice in the database world is the database you already have,” he argues. “If you already have a database and that database supports vectors, start there.”
The organizational cost of adopting new infrastructure while simultaneously building a production AI system is real and frequently underestimated, and the performance advantage of a purpose-built vector database over a mature general-purpose database with vector support is often smaller than the switching cost that choosing the purpose-built option would require.
Friedman’s approach at Qodo uses graph databases as the primary substrate for the context engine because the relationships between code artifacts, pull requests, review decisions, and development history are fundamentally relational rather than purely semantic.
A graph structure can represent the dependency between an API and the PR that introduced it, the connection between a review comment and the code change it was responding to, and the deprecation status of a function relative to the version of the codebase being queried, in ways that a vector store alone cannot.
Multiple specialized tools sit on top of that graph structure, each one designed to answer a different category of question, and a routing agent directs queries to the tool best suited to answer them rather than running every query through a single retrieval pipeline.
“We have an ingestion mechanism where we take metadata and turn it into multiple databases that are mostly graphs and vector DBs,” Friedman explains, “and then we have a query layer which takes a query and has a first routing agent that uses multiple tools that each one of them might be agentic.”
The architecture Friedman describes is substantially more complex than a standard RAG implementation, and that complexity is justified by the specificity of what it is trying to accomplish. A context engine built for code quality and review operates in a domain where the relationships between artifacts matter as much as the content of the artifacts themselves, and a retrieval architecture that cannot represent those relationships cannot serve the use case reliably regardless of how good the underlying model is.
The engineering investment required to build it is significant. But it is the investment that closes the gap between a demo that impresses and a system that is still performing reliably in production a year after launch, and organizations that treat context engineering as a configuration step rather than an engineering discipline will keep rediscovering that gap long after their initial deployment.
Anaconda Inc. has acquired Kilo Code, an open-source, model-agnostic platform that embeds artificial intelligence (AI) agents directly into developer workflows.
The acquisition integrates Kilo’s rapid-growth community of over three million developers into Anaconda’s expanding enterprise AI-native development ecosystem. The transaction positions Anaconda to secure the entire software development lifecycle, from the developer’s initial prompt to final enterprise deployment.
Financial terms were not disclosed.
The deal represents a major strategic step in Anaconda’s bid to address a critical industry bottleneck: The estimated 80% of enterprise AI projects that fail to reach production. By combining Kilo’s front-end AI agents with Anaconda’s secure package distribution and its recently acquired Outerbounds orchestration platform, Anaconda aims to create a highly secure, continuous pipeline for AI-native software development.
Kilo’s platform is built for agentic engineering, where AI systems perform complex, multi-step development tasks such as planning features, modifying files, and debugging codebases rather than just offering simple line suggestions.
Operating seamlessly across popular tools like VS Code and JetBrains, Kilo’s model-agnostic gateway routes tasks across more than 500 different AI models. This routing flexibility helps enterprises manage spiraling token costs, an increasingly vital capability as Kilo currently orchestrates nearly 10 trillion tokens monthly.
The enterprise appeal of the deal lies in marrying this developer flexibility with robust corporate oversight.
“Every enterprise is asking the same question: how do we let our builders move as fast as AI now allows, without losing control of what ships or failing a compliance audit?” Anaconda CEO David DeSanto said in a statement.
By connecting Anaconda’s security scanners directly to Kilo’s developer interface, companies can enforce model and package policies from the very first keystroke.
The transaction closely follows Anaconda’s April acquisition of Outerbounds, the company behind the open-source Metaflow framework. Together, these moves establish unified, three-tiered architecture for modern development.
Kilo Code provides an agentic workspace for initial coding, debugging, and model selection. Anaconda Core governs package security, vulnerabilities, and reproducible environments. Outerbounds orchestrates production-grade workflows and scalable cloud execution.
Sid Sijbrandij, co-founder of Kilo Code and executive chair at GitLab, praised the synergy. “Kilo has many enthusiastic users doing agentic engineering with the freedom to use any model,” Sijbrandij said. “Anaconda spent over a decade building enterprise trust from packages to secure environments. The combination complements each other very well.”
For existing Kilo users, the platform will continue operating with no immediate changes to its products, pricing, or customer support. Kilo CEO Scott Breitenother noted that joining Anaconda, whose ecosystem reaches over 52 million users, will help Kilo expand into large enterprises that require strict governance. Anaconda plans to share deeper product integration details in the coming months as they align their technologies.
The Kilo Code acquisition complements Anaconda’s Python roots, but it also reveals a larger ambition: Moving up the AI stack before the value—and the developer relationship—moves beyond it.
Anaconda announced this week that it is acquiring Kilo Code, and the announcement arrived wrapped in enough AI marketing language to fill several context windows.
There is a “tokenpocalypse.” Enterprises are “token-maxxing.” CIOs are being asked whether they know where their data is. Anaconda and Kilo, meanwhile, are promising “AI on your own terms.”
Let’s strip all of that away. The deal is important enough without the promotional wrapping.
Kilo Code is an open-source, model-agnostic coding agent used by more than 3 million developers. According to the companies, it orchestrates nearly 10 trillion tokens per month and can route work across more than 500 models. It operates inside VS Code, JetBrains and the command line, placing it directly where developers and AI agents increasingly perform their work.
That makes Kilo a significant addition to Anaconda. But the more interesting question is what the acquisition says about Anaconda itself.
Most of us still think of Anaconda as the company behind the Anaconda Python distribution. That identity has served it well. Python became the lingua franca of data science, machine learning and much of artificial intelligence, while Anaconda made the language and its sprawling package ecosystem manageable for millions of developers and thousands of enterprises.
Now, Anaconda appears determined not to remain merely the Python company.
More Than a Python Distribution
The easy explanation is that Kilo complements Anaconda’s existing business. That is true, but it does not fully capture what is happening.
Anaconda’s value was never limited to packaging Python and making it easier to install. It provided environments, dependency management, curated packages, reproducibility and a level of security and governance around an open source ecosystem that could otherwise become difficult for enterprises to manage.
Anaconda did not create Python, nor did it own all the software its customers used. It made that software consumable and trustworthy. It became an important layer between a sprawling open source ecosystem and organizations that needed some assurance about what their developers and data scientists were actually running.
That same problem is reappearing at a much larger scale with AI.
Developers are no longer choosing only packages and libraries. They are selecting models, connecting agents to internal systems, passing organizational data through external services and allowing software to take actions on their behalf. One developer may use a frontier model for a complex reasoning task, an open-weight model for routine coding and a locally hosted model for work involving sensitive data. Agents may install packages, call MCP servers, access databases and generate code that eventually reaches production.
The Python environment remains important, but it is no longer the entire environment that needs to be managed.
Kilo gives Anaconda a position in this expanded development surface. It moves the company from managing what sits underneath the developer to participating in the interface directly in front of the developer.
That is complementary, but it is also transformational.
The Risk of Remaining Indispensable
Anaconda could have chosen to remain focused on its traditional position. Python is not disappearing. Neither is the need for secure packages, reproducible environments and dependency management. If anything, AI-generated software makes those capabilities more important.
But importance does not necessarily translate into control of the customer relationship or capture of the economic value.
If developers increasingly begin their work inside an AI coding agent, the agent provider can become the primary interface. That provider may decide which models are presented, which tools are connected, how context is assembled and where workloads run. The package and environment layer underneath it can remain essential while becoming less visible and more interchangeable.
This is a version of what I call The Indispensability Trap: Foundational technology can remain necessary even as value moves farther up the stack. The companies providing the foundation do the hard work of keeping everything running, while someone operating closer to the user captures the relationship, differentiation and margin.
Anaconda appears to understand the danger.
Python gave it an extraordinary position at the beginning of the data science and AI development process. Kilo is an attempt to follow that process upward as the starting point moves from opening a Python environment to instructing an agent.
The Pieces of a Larger Platform
Kilo makes even more sense when viewed alongside Anaconda’s acquisition of Outerbounds in April.
Outerbounds is the company behind Metaflow, the open-source framework originally developed at Netflix for building and operating data science and machine learning workflows. The acquisition gave Anaconda additional capabilities around compute, workflow orchestration and moving AI applications from development into production.
The pieces now form a recognizable architecture.
Traditional Anaconda supplies trusted packages, environments, models, dependency management and software supply chain governance. Outerbounds and Metaflow extend that foundation into workflow orchestration, compute and production. Kilo adds the agentic engineering layer where developers choose models, direct coding agents, connect tools and produce software.
Anaconda is assembling the ingredients for an end-to-end AI development platform.
That does not mean the company has already created one. Acquiring a collection of complementary technologies is easier than integrating them into a coherent experience. Packages, environments, model catalogs, production workflows, coding agents, model routing, policy enforcement and usage analytics may line up nicely on a presentation slide. Making them operate as one platform without adding friction for developers is considerably harder.
Still, the direction is becoming difficult to miss.
More recently, we spoke about OpenAI’s acquisition of Astral, the company behind the increasingly popular Python tools uv, Ruff and Pyright. We discussed what that acquisition could mean for Python, open source and trust in AI development.
Viewed together, the Astral and Kilo acquisitions create an interesting strategic mirror.
OpenAI, a frontier model company, moved down the stack into Python tooling. Anaconda, a trusted Python and package company, is moving up the stack into coding agents, model routing and the developer interface.
The model providers are moving toward Anaconda’s traditional territory. Anaconda is moving toward theirs without attempting to build a frontier model of its own.
The GitLab Connection
There is also a human connection behind this deal that should not be overlooked.
Kilo co-founder Sid Sijbrandij is best known as the co-founder and executive chair of GitLab. DeSanto spent six years at GitLab and became its longest-serving chief product officer, leading the product organization as GitLab expanded into an integrated, AI-enabled DevSecOps platform.
That history matters.
DeSanto and Sijbrandij did not need to begin this conversation by explaining their respective views of developers, open source or integrated platforms. They had already worked together while GitLab expanded from a company known primarily for source-code management into a much broader platform spanning the software development lifecycle. DeSanto played an important role in that transformation, including the addition of security and compliance capabilities.
We do not know who placed the first call or exactly how the Kilo transaction came together. Unless one of the participants tells us, it would be irresponsible to claim that the prior relationship produced the deal.
It would be equally naïve to pretend the relationship was irrelevant.
Acquisitions involve more than comparing product features. The buyer is betting on the people, culture, community and technology it is absorbing. The seller is deciding whether the buyer can be trusted with what it built. A long working relationship between Anaconda’s CEO and one of Kilo’s founders would give both sides knowledge that does not appear in a standard due-diligence report.
Sijbrandij’s prominent role in the acquisition announcement reinforces the connection. He described Kilo and Anaconda as a rare fit with “almost no overlap,” saying that what one lacked, the other already possessed. That is more than the customary congratulatory quote from a financial backer. It is the assessment of a Kilo co-founder who already knows DeSanto and has previously worked with him while building a developer platform.
The relationship also provides another clue about DeSanto’s ambitions for Anaconda.
At GitLab, DeSanto participated in the company’s evolution from a product associated with a particular part of the development process into a platform intended to manage much more of the software lifecycle. At Anaconda, he appears to be applying a related playbook: Start with a powerful open source foundation and a large developer community, then add adjacent capabilities until the original product becomes the center of a broader platform.
The comparison should not be pushed too far. Anaconda is not GitLab, and AI development is not simply the next version of DevSecOps. But DeSanto has seen this type of platform expansion before. More importantly, he has worked with Sijbrandij while doing it.
Kilo may therefore represent more than a complementary acquisition. It may also be a reunion of people who share a view of how developer platforms are built.
Custody of the Developer—and the Data
The strategic prize is not simply ownership of another AI coding assistant.
Kilo supports hundreds of models rather than requiring developers to commit to one provider. That potentially gives Anaconda a place between enterprises and the model companies competing for their workloads. Anaconda can offer access to frontier models for complex work, open-weight models for lower-cost tasks, local models for sensitive use cases and air-gapped deployments where data cannot leave the organization.
The value lies in governing those choices.
Which models are approved? What data is being sent to them? Which agents can access a source-code repository, database or production system? How much is the company spending across different tools and accounts? Which packages did an agent introduce? Can an organization reconstruct how a piece of AI-generated software was produced?
These are not theoretical questions. They are becoming part of everyday software development, often before enterprise governance systems are ready to answer them.
“The contest in enterprise AI development is quickly shifting to the interface where developers direct agents, choose models, and route organizational data. Anaconda’s Kilo acquisition positions it with a trust layer to govern that surface. Buyers are locking in agentic architecture before governance can answer which models agents use, what data they send, and which systems they touch. Coding agent selection is now a control-plane decision that cannot be deferred until adoption hardens”, Mitch Ashley, VP and practice lead, CIO tech buyer and software lifecycle engineering at The Futurum Group.
Model providers want to own that relationship. So do cloud providers, IDE vendors and agent platforms. Each wants to become the place where enterprises choose models, route workloads and apply policy.
Anaconda enters that contest with two advantages: trust and distribution. It already claims more than 52 million users and a presence inside 95% of the Fortune 500. Kilo gives it a way to extend those relationships into the agentic development experience.
The company is trying to move from being the environment developers rely upon to becoming the control plane enterprises use to manage AI development.
Developer Freedom Meets Enterprise Control
There is an obvious tension in this strategy.
Kilo grew because developers found it useful, open and flexible. It is model-agnostic and available across the tools developers already use. Its open source foundation gives users a degree of visibility and control that proprietary coding assistants do not always provide.
Anaconda is buying Kilo partly because of those qualities. It could also damage them.
If Kilo becomes a funnel into a heavy enterprise platform, developers may look elsewhere. If governance translates into restrictions, delays and corporate approval gates, the product will lose some of the freedom that drove its adoption. If model neutrality quietly becomes a preferred set of commercial relationships, the anti-lock-in position will be harder to defend.
Anaconda’s opportunity is to make enterprise control operate around the developer rather than against the developer. Policies should determine which models and resources are available without requiring a developer to submit a ticket each time an agent needs to perform useful work. Security should be embedded in the environment. Governance should provide visibility without turning AI development into a checkpoint marathon.
That is the promise. The acquisition does not prove that Anaconda can deliver it.
Kilo May Not Be the Last Deal
DeSanto and Anaconda have shown they are not shy about using acquisitions to accelerate the company’s evolution.
Outerbounds and Kilo arrived within a few months of each other. That looks less like opportunistic dealmaking than a deliberate effort to assemble a platform while the AI development market is still taking shape.
There are pieces Anaconda could still add.
AI evaluation and observability would give enterprises better ways to measure the quality, cost and reliability of agent behavior. Model and data lineage could provide a verifiable record of the prompts, packages, datasets and models involved in producing an application or decision. Agent identity and authorization would help control which resources autonomous software can access and which actions it may perform.
Security is another logical area. Prompt-injection defenses, model scanning, secrets management for agents and runtime protection could all complement Anaconda’s software supply chain position. Agent deployment, monitoring and rollback remain immature enough that a focused company in production agent management could fill a gap between Kilo’s development experience and Outerbounds’ orchestration capabilities.
This does not mean Anaconda will buy a company in each category. Nor should we turn the exercise into a fantasy shopping list filled with multibillion-dollar companies Anaconda is unlikely to acquire. The more plausible targets would be focused developer-tool startups and open source projects that provide important technology, community adoption and talent.
The larger point is that Anaconda may not believe its platform is finished.
DeSanto is not learning the platform-consolidation playbook on the job. He helped execute one at GitLab. Now he has acquired a company co-founded by the person with whom he helped build that platform.
Is DeSanto attempting to build the GitLab of enterprise AI development, with Python rather than source-code management as the original beachhead?
It is too early to make that declaration. But it is no longer too early to ask the question.
Anaconda is not abandoning Python. It is using Python as its beachhead, its source of credibility and the foundation beneath a larger platform. Outerbounds extended that foundation toward production. Kilo brings Anaconda into the agentic workspace. Future acquisitions may add more of the control, security and visibility required to connect the two.
Python gave Anaconda its position in the AI stack. DeSanto is now trying to climb that stack before the value and the developer relationship climbs away without it.
The question is no longer whether Anaconda wants to be more than the Python company.
It is how much of the AI development stack DeSanto ultimately intends to own.
One of the most common questions executives ask right now sounds straightforward: is the agent reliable enough yet?
It feels like the right place to start, but the framing quietly points people in the wrong direction because it assumes something about reliability that has rarely been true in complex systems.
When people ask whether an agent is reliable, they are treating the agent itself as the unit of reliability, something you either trust or do not trust in the same way you would evaluate a database or an API.
That mindset comes directly from how software has traditionally been built over the last few decades. Teams evaluate components in isolation, stack them together, and expect the overall system to inherit the guarantees of the underlying parts.
High-stakes human work has never really operated that way, and agentic systems probably will not either.
Even today, most production agents are already layered systems in practice. One model plans, another executes, and a third reviews. Each layer brings different strengths and different failure modes. By the time somebody interacts with the “agent,” the underlying system already contains multiple components coordinating together behind the scenes.
How Mature Systems Handle Reliability
Industries that genuinely need reliability rarely try to make individuals perfect. They design systems that absorb imperfection.
Airlines do not assume a pilot will never make a mistake. They assume the opposite: even highly trained professionals will occasionally miss things, become distracted, or make poor decisions under pressure. The surrounding system exists to reduce the likelihood that a single mistake turns into a catastrophe.
Checklists, redundant instrumentation, standardized communication protocols, pilot and co-pilot cross-checks, air traffic control, post-flight reviews, and formal investigations all work together to create reliability at the system level rather than the individual level.
Hospitals operate similarly. Surgeons may be highly skilled, but surgical systems still include checklists, structured pauses before procedures, repeated instrument counts, second opinions, escalation paths, and post-incident reviews. In high-risk environments, review mechanisms are not signs of distrust. They are part of the operating model.
Sales organizations behave the same way. They do not depend on perfectly consistent sales reps. They build pipelines with stage gates, forecast reviews, approval flows, and coaching loops that reduce the impact of variation at the individual level.
Engineering organizations do this too through code review, testing pipelines, deployment safeguards, observability layers, and rollback procedures designed to catch failures before they affect production systems.
Across all of these examples, reliability is not a property of the individual actor. Reliability emerges from the surrounding system.
What Changes With Agents
Once you apply that framing to AI systems, the conversation changes substantially.
The goal is not to make a model perfectly deterministic. Traditional software already handles deterministic execution extremely well. The goal is to design workflows that produce reliable outcomes even when individual steps occasionally fail.
That pushes organizations toward a familiar set of operational mechanisms.
Approval gates ensure that important outputs receive the appropriate level of scrutiny before they move forward. Sometimes that means a human reviewing a customer-facing message. Sometimes it means a second model reviewing the output of the first. Either way, the workflow introduces a checkpoint before the consequence becomes real.
Feedback loops ensure mistakes are not merely corrected once and forgotten. Corrections get captured, reused, and fed back into the system so future iterations improve instead of repeating the same failure pattern.
Review cadences align attention with risk. High-impact outputs receive deeper review while lower-risk operational work moves more quickly without unnecessary friction.
Postmortems matter too, especially when they focus on improving the surrounding system instead of assigning blame to a single actor. The important question is rarely “who made the mistake.” The important question is what allowed the mistake to pass through the workflow unchecked.
None of these mechanisms are especially new. Organizations have spent decades developing them because they are how complex systems become dependable despite imperfect participants.
Why Many Teams Get Stuck
A pattern appears repeatedly when organizations begin experimenting with agents.
Teams test a workflow, notice that the system succeeds most of the time but not all of the time, and conclude that the technology itself is not ready yet. At that point, many organizations pause and wait for the next model release, assuming reliability will eventually arrive as a property of the underlying model.
That is a passive way to think about the problem.
A systems-oriented approach starts from the same observation but asks a different question: what additional structure would make the overall workflow reliable enough for production use?
Sometimes the answer is surprisingly lightweight. A well-placed approval step catches the minority of cases where the model fails before the output reaches customers. A review loop prevents recurring mistakes from reappearing indefinitely. A clear escalation path routes ambiguous cases toward human judgment rather than forcing the system to improvise.
Meanwhile, the models continue improving underneath the workflow. Over time, both layers compound simultaneously: the models improve and the surrounding operational system improves alongside them.
Designing the Workflow Around the Model
A more useful question than “can we trust the agent?” is “what does the workflow around the agent look like?”
Start with one workflow that actually matters and map it concretely.
Where does approval happen, and who owns it? How are corrections captured and reused? Which outputs receive regular review, and how does the level of scrutiny change based on risk? When something fails, how does that information flow back into the system so the process itself improves over time?
Many organizations discover that these controls were already inconsistent even before AI entered the picture. Reviews happened irregularly. Feedback loops were informal. Approval steps existed on paper but not consistently in practice.
Agents often expose those weaknesses rather than create them.
That exposure is useful because it forces organizations to operationalize workflows that previously depended too heavily on individual heroics and tribal knowledge.
Reliability is an Organizational Property
Reliability has rarely come from any single component in isolation. It comes from how systems handle failure.
A pilot is not perfectly reliable, but commercial aviation systems are designed to tolerate human imperfection. A surgeon is not perfectly reliable, but surgical procedures are structured to catch mistakes before they escalate. A sales rep is not perfectly reliable, but a strong sales organization still produces predictable outcomes. Agents work the same way.
The competitive advantage in the next phase of AI adoption probably will not come from selecting the single best model. Models will continue improving and gradually converging across many capabilities.
The advantage will come from designing workflows that continue producing reliable outcomes even when individual steps remain imperfect.
That is ultimately less a tooling problem than an organizational design problem.
Anthropic introduced a self-hosted gateway this week that lets enterprises run Claude Code on Amazon Bedrock and Google Cloud without the credential sprawl and manual setup that have typically come with deploying AI coding tools at scale.
The Claude apps gateway is a single, stateless container that organizations deploy on their own infrastructure and back with a PostgreSQL database. It centralizes identity, policy enforcement, usage tracking, and spend management for Claude Code, addressing a problem that will sound familiar to anyone who has tried to roll out a developer tool across a large engineering org: Every new hire needs a cloud credential, every laptop needs the right settings pushed to it, and finance needs a way to see who’s spending what.
Before the gateway, none of that was centralized. IT teams provisioned a credential per developer, manually distributed configuration, and stitched together separate tooling just to get visibility into spend. That’s a lot of overhead for something that should be straightforward.
What the Gateway Actually Does
The gateway sits between developers and the underlying model infrastructure, handling five things:
Identity. The gateway acts as an OpenID Connect relying party, working with Google Workspace, Microsoft Entra ID, Okta, or any standards-compliant OIDC provider. It issues short-lived sessions instead of long-lived secrets stored on developer machines — a meaningful security improvement in its own right. Onboarding a new developer means adding them to the identity provider. Offboarding means removing them. No credential cleanup, no orphaned API keys floating around.
Policy. Admins define managed settings once, on the server. Clients pick up that policy automatically at sign-in, and the gateway enforces it on every request. Allowed models and default settings can be adjusted centrally, rather than chasing down individual machine configurations.
Telemetry. Every request is stamped with a usage metric, which the gateway relays via OTLP to a collector controlled by the organization. That data remains on the company’s own infrastructure and is subject to its retention schedule.
Routing. The gateway holds the upstream credential and routes inference traffic to the Claude API, Amazon Bedrock, or Google Cloud, with optional failover between providers.
Spend caps. Organizations can set daily, weekly, and monthly limits that apply at the org, group, or individual level.
Anthropic built the gateway into the same Claude binary developers already installed, so the /login flow is gateway-aware out of the box. Settings apply automatically at sign-in, and policy gets enforced consistently without extra configuration on the client side.
Worth noting what the gateway doesn’t do: it doesn’t send inference traffic or usage data to Anthropic unless an organization specifically configures it to use the Claude API. For Bedrock or Google Cloud deployments, that data stays put. Anthropic is also publishing the protocol that the gateway uses, enabling other gateway developers to build compatible implementations.
Why This Matters for Engineering Teams
For platform and DevOps teams, the appeal here is less about any single feature and more about consolidation. Identity, policy, telemetry, routing, and cost control have typically required separate tooling, separate dashboards, and separate processes to keep in sync. Folding all of that into a single container that an organization already controls significantly reduces the operational surface area.
It also addresses a real friction point in AI tool rollouts: the gap between what security and finance teams need (visibility, control, audit trails) and what developers want (a tool that just works without jumping through credential hoops). The gateway’s SSO-based onboarding and offboarding model handles both at once.
Mitch Ashley, VP and practice lead for software lifecycle engineering and AI-native software engineering at The Futurum Group, sees this as a significant shift in who controls the control layer. “Enterprise identity, policy, cost attribution, and spend caps now ship as first-party infrastructure for Claude Code,” he said. “The model provider is claiming the access and cost layer that third-party gateways and in-house tooling used to hold.”
Ashley also flagged the bigger question platform teams need to ask. “For platform teams, the real question is whether per-vendor gateways or a neutral control point govern a multi-model estate,” he said. “This makes one coding tool manageable at scale. It does not govern what agents do, and that is the control problem that still decides enterprise autonomy.”
Getting Started
Deployment involves downloading the Claude Code CLI binary and pointing a gateway.yaml configuration file at the organization’s OIDC issuer and upstream credential, and registering one OIDC app with the identity provider. Rolling it out to client machines means setting the forceLoginMethod and forceLoginGatewayUrl parameters in managed-settings.json. Clients connect to the gateway automatically on first boot.
The gateway is available now, with full documentation on Anthropic’s developer site.
For organizations already running Claude Code through Bedrock or Google Cloud — or considering it — this is the kind of infrastructure update that doesn’t change what the tool does, but changes how manageable it is to run at scale. That’s often the difference between a pilot program and something engineering leadership is willing to stand behind.
In complex software systems, our traditional definition of operational health has always been comfortably binary. For over a decade, site reliability engineering (SRE) teams have relied on the industry-standard ‘Four Golden Signals’ — latency, traffic, errors and saturation — as the ultimate truth of platform stability. If our API-response times are hovering at sub-100 ms, network throughput is steady, CPU cores aren’t pegged and the HTTP 500 error rate is flatlined at zero, we sleep soundly. We check our Grafana dashboards, see an entirely green pasture and assume that our platform is delivering flawless value to the business.
Then came production AI.
With organizations rapidly transitioning from deterministic, code-driven microservices tonon-deterministic, LLM-powered applications, this foundational telemetry framework is facing a quiet crisis. In an AI-driven ecosystem, a system can be structurally flawless while failing functionally. An API gateway can return a crisp HTTP 200 OK in record time, yet the payload it carries could be a hallucinated financial projection, an injection exploit or a toxic output that violates compliance. The infrastructure is entirely healthy, but the system is broken. To build truly trustworthy AI at scale, platform and site reliability engineers must look beyond hardware and network states and evolve our telemetry for a non-deterministic world.
Decoding the AI Blindspot: The Green Dashboard Paradox
The core challenge of traditional telemetry in an AI world comes down to the concept of determinism. Conventional software architectures operate on absolute rules where, given Input A and Condition B, the application will always produce Output C. If it doesn’t, a distinct exception is thrown, a 5xx error code is emitted and an on-call engineer is paged.
Generative AI, retrieval-augmented generation (RAG) pipelines and autonomous agent frameworks break this paradigm completely. These systems are inherently probabilistic. Since models rely on high-dimensional semantic spaces and complex vector retrieval, the same input can yield entirely different outputs across sequential requests.
This introduces what I call the “Green Dashboard Paradox.” Consider a high-transaction financial enterprise platform processing automated customer queries. A traditional SRE dashboard monitoring the service shows a pristine state:
Latency: Minimal
Traffic: Well within standard thresholds
Errors: Zero dropped packets or server exceptions
Saturation: GPU and memory utilization perfectly optimized
However, beneath the surface, the system is failing contextually. A slight shift in the vector database’s embedding space has caused the retrieval mechanism to fetch outdated data, causing the model to hallucinate incorrect loan rates for thousands of users. Since the transport layer successfully delivered the data without crashing, traditional infrastructure monitoring remains completely blind to the failure. We are validating that the pipes aren’t leaking, but we have no idea if the water flowing through them has turned toxic.
Where the Classic SRE Model Loses Its Grip
To understand how to fix this, we have to look closely at where the classic Google SRE handbook breaks down when interacting with LLMs and inference clusters:
The Misleading Nature of Latency:In classic web applications, high latency means a degraded user experience, which is usually fixed by scaling up instances or optimizing database queries. In LLM applications, total response latency is highly variable based on prompt size and token length. A long response isn’t necessarily an unhealthy one. Conversely, a blazing-fast response could indicate that a safety filter immediately aborted the query, meaning low latency could actually signal a high user rejection rate.
The Saturation Illusion:Traditional saturation measures CPU, memory and disk I/O. In AI infrastructure, workloads live on GPUs. GPU memory (VRAM) behaves fundamentally differently because it is often aggressively pre-allocated by inference engines, such as vLLM or Hugging Face TGI, to optimize performance. A traditional monitoring agent looking at VRAM will report 95% saturation constantly, rendering it useless as a reactive alerting trigger.
The Vanification of Error Codes:The classic error signal relies heavily on protocol-level telemetry such as HTTP status codes and gRPC status tracking. In an AI pipeline, an application failure often occurs at the semantic or alignment layer. If an LLM generates a response containing proprietary system instructions because of a prompt injection attack, it is a catastrophic security failure. However, to your load balancer, it is just a highly successful text transmission.
Operationalizing Trust: The New Evolutionary Telemetry
We don’t need to throw away the Four Golden Signals entirely. Rather, we must treat them as the baseline infrastructure layer and build a new, intelligent tier of telemetry on top of them. To engineer trustworthy, resilient AI systems, the telemetry pipeline must ingest semantic, structural and alignment signals natively.
When designing a modern observability stack for non-deterministic software, teams should look to instrument four alternative ‘Golden Signals of AI Architecture’:
Classic SRE Metric
The AI Telemetry Alternative
What it Safely Measures in Production
Latency
Time to First Token (TTFT)
It measures the duration between request submission and the arrival of the initial streaming token. This isolates model inference lag from total delivery time.
Traffic
Token Velocity and Throughput
It tracks the volume of input (prompt) versus output (completion) tokens and is critical for forecasting provider cost, managing memory buffers and preventing rate-limiting.
Errors
Guardrail Intervention Rate
It tracks how frequently secondary safety layers (such as Llama Guard or NeMo) intercept, filter or rewrite inputs and outputs before they hit the user.
(Generic)
Semantic Drift and Faithfulness
It measures the statistical degradation of output vector embeddings over a rolling window compared to known-good baselines to catch silent model decay.
By feeding these primitives into open standard frameworks such as OpenTelemetry (OTel), platform engineers can track a transaction seamlessly as it moves from user interaction, travels through API routes, runs queries across a vector store such as Pinecone or Milvus and executes inference on a GPU node.
Putting it to Work: Building AI-Focused SLOs
A meaningful observability strategy doesn’t stop at gathering data. It requires defining explicit service level objectives (SLOs) that align system performance with true business utility and trust.
When working with non-deterministic systems, your service level indicators (SLIs) must shift from pure technical uptime to semantic compliance.
For instance, instead of establishing an objective stating that 99.9% of API requests must return an HTTP 200 within 200 ms, a mature platform engineering team operating an AI service should define metrics focused on output safety and retrieval accuracy:
Guardrail Health SLI: The percentage of transactions that successfully navigate the alignment pipeline without triggering a safety policy violation or malicious prompt injection filter.
Target SLO:99.5% of monthly traffic passes validation without structural or safety intervention.
RAG Context Faithfulness SLI:The mathematical cosine similarity score between the retrieved grounding documents and the generated completion, evaluated via automated, asynchronous evaluation LLMs over a rolling 5-minute window.
Target SLO: 98% of generated answers must maintain a faithfulness score above 0.85, alerting the SRE rotation if vector database drift occurs.
Conclusion: The Infrastructure of Integrity
The shift to AI does not make the discipline of SRE obsolete. Instead, it dramatically elevates it. The ultimate goal of the platform engineer is no longer to simply ensure that computing power is accessible and data packages move quickly from point A to point B. Our responsibility has expanded to guarding the functional integrity and safety of the system itself.
By implementing advanced telemetry pipelines that monitor past the boundaries of traditional transport protocols, engineering teams can safely embrace the immense power of non-deterministic software. When we proactively measure token velocity, time to first token and guardrail interventions, we bridge the gap between abstract AI safety and hard platform engineering. The result is a robust, modern observability culture where our dashboards aren’t just superficial indicators of hardware health, but are true reflections of systemic trust, accuracy and enterprise resilience.
