devopsy.kinsta.cloud

Category: Contributed Content

The Agent Failure Your Approval Gate Can’t Catch

An agent that does the wrong thing leaves evidence — a bad diff, a malformed record, a customer complaint. An agent that does nothing and reports success leaves a green checkmark, and green is the one signal your incident process is built to trust. Worry more about the second failure, because everything a DevOps team owns for catching failure — exit codes, run status, alert rules, the dashboard itself — is instrumentation built for software that fails loudly. None of it fires when the failure is an absence.

Andrew Filev made the strongest version of the reliability argument in these pages in July (https://devops.com/reliability-comes-from-the-system-not-the-agent/): reliability is a property of the system, not the agent. “Reliability has rarely come from any single component in isolation,” he wrote. “It comes from how systems handle failure.” Aviation does not assume perfect pilots and hospitals do not assume perfect surgeons; both wrap imperfect actors in approval gates, feedback loops, review cadences and postmortems, and the system is dependable even though nobody in it is. Applied to agents: stop waiting for a model reliable enough to trust naked, and design the workflow around the one you have.

He is right. I want to push on the one place that model is blind.

Every Mechanism Waits for Something to Arrive

Look closely at the load-bearing sentence in that piece: “Approval gates ensure that important outputs receive the appropriate level of scrutiny before they move forward.” The whole mechanism lives inside the words important outputs. A gate scrutinizes what arrives at it. A review cadence reviews what exists. A feedback loop needs an output to feed back. A postmortem requires that somebody first notice there was a mortem.

Now run a different failure through that machinery: the unattended agent that quietly did nothing. The overnight job that was supposed to reconcile accounts, open tickets or produce the report terminates cleanly, logs success and produces no artifact. Nothing reaches the approval gate, so the gate approves nothing and flags nothing. Nothing enters the review queue. There is no wrong output to correct, so the feedback loop has nothing to learn from. The workflow catches the agent that does the wrong thing. The agent that does nothing and calls it success never touches the workflow at all — it does not fail the review; it never reaches it.

This is not a hole in the aviation analogy. It is the point where the analogy earns its keep. A preflight checklist catches an omission because a human walks the aircraft and looks at the flaps. The flaps either moved or they did not, and the pilot is verifying the world, not a report about the world. Most agent infrastructure does the opposite: it reads the run’s account of itself and calls that verification.

“Did Something Wrong” Generates an Event. “Did Nothing” Does Not.

This site has come close before. Back in March (https://devops.com/agentic-systems-are-breaking-reliability-frameworks/), principal engineer Shahid Ali Khan described subtle agent failure as sharply as anyone in an article by Saqib Jan: “Traditional runbooks assume failures are obvious. A service crashes, latency spikes, errors propagate. Agents fail subtly. They might complete successfully while doing something completely unintended.” Notice what even that sentence assumes: the agent did something. A misdeed produces an event — an out-of-envelope tool call, a schema violation, an anomalous write — and an event can be captured, classified and routed, which is precisely what the controls in that piece do. Every one of them, like every mechanism Filev names, fires on something happening. Absence is the one input none of them take.

The reflex answer is better observability, and it is the wrong reflex here. Observability instruments the loop; this failure is only visible in the outcome. Traces, spans and token counts may faithfully record a run that did nothing as a healthy run — a perfectly instrumented trace of nothing. More telemetry about the loop is more detail about the wrong object.

Verify the World, Not the Report

The operational change is a single move: the health check for an agent cannot be “did the run error?” It has to be “does the thing the run was supposed to change actually show the change?” That takes verification off the loop’s self-report and puts it on the state of the world. Three practices follow.

Declare the expected effect as a precondition of success. A deploy has not succeeded when the script exits zero; it has succeeded when the new version is serving traffic. Write the expected effect down — reconciliation posted, tickets opened, version live — and derive the run’s status from checking it. A run with no declarable effect is worth asking hard questions about.

Treat implausible speed as a failure signal. A run that completes far faster than its work could possibly take is not a win. It is the cheapest tell available that the hard part was skipped, and it deserves the same alert an error gets.

Trigger verification from the expectation, not from the run. Filev’s own second mechanism — “a second model reviewing the output of the first” — can catch absence, but only if something independent asks it to. A reviewer invoked by the run inherits the run’s silence: no run, no invocation, no review. The checker has to fire because work was due, on a schedule or a contract the agent does not control, and go look at the world whether or not the loop reported in.

Say the uncomfortable part plainly: this costs money. Outcome verification means paying to confirm work you already paid to have done, and almost nobody budgets for it, because the economic pitch of agents is that you stop paying for supervision. Price in the trade-off now, while the failures are still cheap.

Filev’s conclusion stands: reliability comes from how systems handle failure. The system just has to handle one more kind — the failure that never announces itself. An approval gate is only as strong as what shows up to it, and the most dangerous run in your fleet is the one that shows up nowhere, carrying nothing, wearing green.

Chase W. Hughes is a three-time founder who built and sold ProAI, one of the first commercialized GPT products, and works with companies building AI products.

October 9, 2026
Attaching Evidence to Alerts: an Enrichment Sidecar for Alertmanager

Every on-call engineer knows the ritual. A page lands in Slack: the 5xx rate on a service is above the threshold. The message tells you that something is wrong and nothing about what. So you open Kibana or Loki in another tab, set the time window to the last few minutes, filter by the service, and start reading. In my team this took two or three minutes on every alert, and these are the worst minutes of the incident, because nothing is being diagnosed yet.

The obvious request is to put the last few error lines into the alert message itself. I tried to do this inside Alertmanager, and it cannot be done there. It helps to understand why before building anything around it.

Why Alertmanager Cannot Do It

Alertmanager renders notifications through Go templates, and a template can only use what is already attached to the alert: the labels and the annotations which Prometheus put there when the rule fired. There is no mechanism which would call Elasticsearch, Loki or any other HTTP API at the moment a notification is sent. Annotations are rendered from the alerting rule at evaluation time, inside Prometheus, and Prometheus does not talk to log stores either. Fetching data at send time is simply not a capability of the design.

So the logs have to come from a separate process which stands next to Alertmanager.

The Sidecar

What runs in production now is a small service next to Alertmanager, and its shape came out of two failed attempts, so I will describe it through them.

My first attempt was a bot which received alerts and posted them to Slack itself, with the logs already attached. It looked good in a demo, but it was a mistake because the bot had become the paging path. Any bug in my code, any timeout of the log store, and nobody gets paged. I threw that version away quickly. Alertmanager has to stay the thing which posts the alert, exactly as before, and whatever adds context has to be allowed to die without anyone noticing. So the enricher hangs off a second receiver: the route which sends the alert to Slack also duplicates it to a webhook, and behind the webhook stands my service. When it is down, the page arrives as it always did, just without the extra.

The second attempt posted the logs as its own message into the channel, right after the alert. Within a day the channel was hard to read: two messages per alert, and during an alert storm they interleaved, so the logs stood three messages away from their page. The fix was to reply into the thread of the message which Alertmanager had already posted. The channel stays as it was, one message per alert, and whoever opens the incident an hour later finds the logs from the exact minute of the page sitting under it.

Finding the Message to Reply To

That thread reply is the one interesting engineering problem in the whole service. The enricher receives the alert from Alertmanager, and it has to find the Slack message which corresponds to this alert, and nobody tells it which one that is. When the webhook fires, the service reads the last couple of minutes of channel history and looks for a message which carries the same alert fingerprint. Our Slack template rendered the fingerprint into the message, so the match was exact; before I added that, I matched on the label set which appears in the message text, and that worked too. Once the message is found, the reply is one API call with thread_ts.

Then production corrected the details. Alertmanager re-sends a firing alert on repeat_interval, and my version at first posted the same log dump under the same message again and again, so now the service remembers which fingerprints it has already enriched. The lookback for matching is limited to about two minutes, since replying to some old message which happens to carry the same labels is worse than staying silent. The query into the log store is built from the labels of the alert (service, namespace, severity) and takes the last few error-level lines, ten or so. I keep it that short on purpose: the reply is a hint for the person, and the full picture is in the log store anyway. One more thing I did not expect: an empty result is also worth posting. When the reply says there were no matching error logs in the last five minutes, the person on call can broaden the investigation to the infrastructure or the alerting rule itself, and this saves the same minutes.

What It Changed

The service is a few hundred lines, and in operation it has been boring, and boring is what I want from anything standing near paging. The visible change for the team: when you open Slack on a page, the first evidence is already under it, and the phase of “which dashboard, which index, which time window” is gone from incidents. Time to diagnosis dropped on every incident which had logs behind it, without any change to the services themselves.

Logs turned out to be only the first thing worth attaching. The same service can reply with anything which Alertmanager cannot know at render time: a runbook link picked by label, the marker of the latest deploy of the affected service (the first question in half of the incidents anyway), or a dashboard link already scoped to the incident window. Alertmanager alerts, and the sidecar carries what the responder reaches for in the first minute.

If you run Prometheus with Slack, the whole thing is one small service plus one extra webhook receiver in the Alertmanager config. Do the idempotency bookkeeping first; it is the first thing which bites.

Daniil Romashov is a Site Reliability / DevOps engineer working on reliability and observability for high-load systems. Open-source work: github.com/youngpabl0.

October 8, 2026
Secrets Sprawl and Rotation: A Practical Vault Management Playbook
In 2025 alone, security firm GitGuardian detected 28.65 million new hardcoded secrets exposed in public GitHub commits — a 34% increase year over year and the largest single-year jump the company has recorded in five years of publishing its annual State of Secrets Sprawl report. Since 2021, that number has grown 152%, far outpacing the 98% growth in GitHub’s active developer base over the same period. Secrets sprawl isn’t a hypothetical DevOps hygiene problem anymore. It’s accelerating faster than most security teams can respond to it, and AI-assisted development is a large part of why.

The financial stakes keep climbing alongside it. IBM’s newly released 2026 Cost of a Data Breach Report puts the global average cost of a breach at $4.99 million, a 12% increase over the prior year and a new record high. In IBM’s 2025 edition, breaches where compromised credentials were the initial access vector carried a $4.67 million average cost on their own, and organizations took an average of 246 days just to identify and contain them. A single hardcoded credential, sitting in a repository or a config file for months, is exactly the kind of quiet, slow-burning exposure that produces numbers like these.

Why Secrets Sprawl is Getting Worse, Not Better

The intuitive assumption is that better tooling and more awareness should be shrinking this problem over time. The data says the opposite is happening, and AI-assisted coding is a specific, identifiable accelerant.

GitGuardian’s 2026 report found that secrets tied to AI services grew 81.5% year over year, making them the fastest-growing category of leaked credentials in the entire dataset. Model context protocol (MCP) configuration files, which became a de facto standard for connecting LLMs to external tools and data sources in 2025, already had 24,008 unique secrets exposed in their first year of widespread use. The report links this pattern partly to official documentation that suggests passing API keys as command-line arguments or storing them directly in JSON config files. The report also found that commits produced with AI coding assistance leaked secrets at roughly twice the rate of the GitHub-wide baseline. This was not because AI tools are malicious, but because they generate working code quickly, including secrets, without the natural pause a developer might take to ask whether a credential belongs in that file.

The consequences of this pattern aren’t abstract. GitGuardian’s research points to the Smithery.ai incident as a concrete illustration: A path-traversal vulnerability in an MCP registry exposed overprivileged tokens, granting arbitrary code execution across more than 3,000 hosted servers from a single flaw. That’s what secrets sprawl actually costs when it intersects with infrastructure that’s more centralized and more automated than the credentials scattered across it were ever designed to be.

Two other findings from the same report should reshape how security teams think about the shape of this problem, not just its size. First, internal repositories are six times more likely than public ones to contain hardcoded secrets; the sprawl problem is worse exactly where fewer external eyes are watching. Second, and more troubling, 64% of secrets that leaked in 2022 are still valid and exploitable today. Detection without remediation and rotation isn’t solving anything; it’s cataloging a problem that never actually gets fixed.

Step One: Get Secrets out of Code Entirely

The starting point for any serious secrets program is removing hardcoded credentials from source control and configuration files and centralizing them in a purpose-built secrets manager — HashiCorp Vault, AWS Secrets Manager, Azure Key Vault or Google Secret Manager are the common choices, and the specific product matters less than the discipline of having exactly one control plane rather than credentials scattered across a dozen tools and files.

In practice, this means infrastructure code references a secret at runtime instead of embedding the value directly:
- data “vault_generic_secret” “db_creds” {
- path = “database/creds/readonly”
- }
- resource “aws_db_instance” “app” {
- identifier = “appdb”
- username = data.vault_generic_secret.db_creds.data[“username”]
- password = data.vault_generic_secret.db_creds.data[“password”]
- }
The credential itself never touches version control. Terraform state still needs its own protections — state files can contain resolved secret values — but that’s a narrower, more tractable problem than an organization-wide habit of pasting API keys directly into .env files and Terraform variables under deadline pressure.

Step Two: Enforce Least Privilege Inside the Vault Itself

Centralizing secrets in one place is necessary but not sufficient. Without access controls, a single compromised service account can retrieve far more than it should ever need, turning one small breach into a much larger one. Fine-grained policies scoped to exactly what each service or role requires are what actually limit the blast radius:
- path “database/creds/production” {
- capabilities = [“read”]
- allowed_parameters = {
- “role” = [“readonly”]
- }
- }
The principle here is simple and easy to state but consistently hard to enforce in practice: A service that only ever needs read access to a database should have no path, under any policy, to a credential capable of writing to it. Every broad grant made ‘just in case’ during an incident and never walked back afterward is exactly the kind of standing risk that GitGuardian’s “64% still valid” finding describes at scale.

Step Three: Make Secrets Short-Lived by Default

Static, long-lived secrets are dangerous specifically because they don’t expire on their own. A credential leaked once in 2022 that nobody rotated is, per the data above, still a live risk in 2026. Dynamic secrets flip this: Instead of a fixed, standing credential, the secrets manager generates a short-lived one on demand, tied to a specific request, that expires automatically.

HashiCorp Vault’s database and cloud secrets engines are a common way to implement this — for example, generating a temporary AWS IAM identity scoped to a single task, valid for a bounded window, rather than handing a service a permanent access key it holds indefinitely. For secrets that genuinely can’t be made dynamic (some third-party API keys, for instance), a rotation strategy modeled on blue/green deployments works well in practice: Maintain both a current and a next-version secret, validate that the new one works end-to-end, then cut over and revoke the old one rather than rotating in place and hoping nothing breaks mid-transition.

The goal in both cases is the same: Shrink the window during which a leaked credential is actually useful to an attacker, since detection and remediation alone clearly aren’t keeping pace with the volume of secrets being created.

Step Four: Assume You’ll Need to Prove What Happened

A secrets manager without audit logging tells you what secrets exist, but nothing about who accessed them, when or whether that access was expected. Enabling audit devices — piping access logs to a SIEM, Splunk instance or equivalent — turns “we think this credential might be compromised” into “we can see exactly when and how it was used,” which is the difference between a fast, contained response and a slow, uncertain one.

This matters more, as AI agents get direct access to infrastructure. An agent with a standing credential and no logged trail of what it did with that credential is a much harder incident to investigate than one whose every access is recorded against a specific task and timestamp.

A Practical Playbook

Bringing this together into a sequence a team can actually execute:

Discover What’s Already Exposed: Run a secrets-scanning tool such as GitGuardian or TruffleHog across your full repository history, not just the current branch. Leaked secrets frequently persist in old commits long after the file that contained them was ‘fixed’. Given that 64% of secrets leaked in 2022 are still valid, assume old findings are still live until you’ve confirmed otherwise.
1. Remove and Rotate, Don’t Just Delete: Purging a secret from source history without rotating the underlying credential leaves the exposure intact — the value is already out, likely already scraped and cached elsewhere. Rotate first, then clean the history
2. Stand up a Single Secrets Manager: Consolidate into one control plane rather than a patchwork of environment variables, CI/CD secret stores and cloud-provider-specific tools that nobody has a complete inventory of.
3. Apply Least-Privilege Policies From Day One: Default every new service account and role to the narrowest access that lets it function and require an explicit, logged decision to widen it — never the reverse.
4. Convert Static Secrets to Dynamic Ones Wherever the Secrets Engine Supports It: Start with the highest-value targets. Database credentials and cloud IAM roles are usually both the most common and the most damaging if leaked.
5. Automate Rotation for Everything That Can’t be Made Dynamic: Implement scheduled rotation, blue/green cutover and automatic revocation of the previous version once the new one is confirmed to be working.
6. Enable Audit Logging and Route it Somewhere Your Security Team Actually Monitors: A log nobody reviews provides the appearance of visibility without the substance of it.
7. Pay Specific Attention to AI Tooling: Treat MCP configuration files, AI agent service accounts and CLI-passed API keys as a distinct category worth auditing on their own, given how disproportionately fast that category of leak is growing.
8. Revisit the Entire Inventory Quarterly: Secrets sprawl isn’t a project with an end date. GitGuardian’s own data shows the problem accelerating year over year, which means a program that isn’t actively maintained will fall behind, not stay even.
Conclusion

Secrets sprawl persists not because teams don’t know hardcoded credentials are risky, but because detection alone doesn’t fix anything. A scanner that finds a leaked key changes nothing if nobody rotates it, and the data shows that gap is exactly where organizations are failing. Centralizing secrets in a single control plane, enforcing least privilege inside it, converting static credentials to short-lived dynamic ones and treating audit logging as non-negotiable together close that gap in a way that scanning by itself never will.

With breach costs at a record $4.99 million globally and secrets exposure still growing faster than the developer population producing it, the return on actually finishing this work — not just starting it — has never been more direct.
October 8, 2026
AI Is Taking on Entry-Level Engineering Work. Who Trains the Young Engineers?

When I started my career on a support desk, I dealt with failed logins, missing permissions and recurring system errors every day. The work was repetitive, but repetition taught me to spot patterns, test assumptions and look beyond the symptom a user reported. Over time, I was resolving around 90% of incidents at first contact. That experience still shapes how I approach infrastructure work. As AI takes on more of these routine tasks, I worry about how the next generation of engineers will develop the same judgement.

That kind of repetition is becoming harder for early-career engineers to get. AI tools are already embedded in software development, with 84% of developers surveyed by Stack Overflow saying they use or plan to use them, while 51% of professional developers use them daily. SignalFire’s 2026 State of Tech Talent report found that entry-level hiring has fallen roughly 65% at major technology companies and 76% at early-stage startups compared with 2019. AI is increasing productivity at the same time that the traditional entry point into engineering is narrowing, making the training built into junior work increasingly important.

The Apprenticeship Hidden Inside Routine Work

Some of the most valuable lessons in my early roles came through work that looked highly repetitive from the outside. I handled joiner, leaver and mover access administration, investigated recurring permission failures and worked with developers when the same issues kept returning. I encountered former employees whose access had not been fully removed, role changes where permissions had only been partially updated, and new starters who were missing access because part of the process lived in undocumented knowledge. Treating each incident as an isolated ticket only delayed the next occurrence. Root-cause analysis, process fixes, knowledge-base documentation and runbooks helped remove whole categories of failure. The ticket queue was a training ground disguised as a cost centre.

Dealing with those problems repeatedly taught me the importance of structured fault diagnosis. When I coach colleagues, I see the difference most clearly when an infrastructure issue has no obvious answer. One engineer will work through the error systematically, rule out possible causes and build an explanation from the evidence. Another may paste the error into an AI assistant, receive a plausible fix, apply it and then become stuck when the fix fails. AI can accelerate the first attempt, but recovering when that attempt is wrong still depends on having a mental model of the system. Stack Overflow’s 2025 survey found that 46% of developers distrust the accuracy of AI output, compared with 33% who trust it, while 66% say their biggest frustration is receiving AI solutions that are almost right but not quite.

When the Training Ground Disappears

Businesses need to be careful about treating every automatable junior task as work that can simply be removed. A repetitive task can still be where an engineer learns to diagnose unfamiliar problems. If AI resolves routine tickets in seconds, companies need to create other opportunities for junior engineers to investigate failures from start to finish. That could mean reviewing AI-generated fixes, explaining why a recommendation makes sense, testing alternative causes, or giving them ownership after AI completes first-pass triage. Even when AI handles the first pass, a junior engineer should still get the opportunity to take an incident from diagnosis through to resolution. DORA’s research shows that 90% of technology professionals now use AI at work and more than 80% believe it has increased their productivity, while time saved during creation is often reallocated to auditing and verification. That verification work is part of the apprenticeship too.

Over the next five or ten years, we could end up with a workforce that is extremely fast at directing AI but has had fewer opportunities to develop independent technical judgement. Companies could have engineers who are highly capable with AI systems but less comfortable when a production problem falls outside a familiar pattern. That judgement comes from exposure to real failures, including the frustrating ones where the first diagnosis is wrong. If companies remove those experiences from junior roles, they will eventually feel the gap further up the organisation.

I think back to that early support desk because the value of those tickets was easy to miss at the time, even though they taught me how to approach unfamiliar technical problems. AI should remove unnecessary friction from engineering work and give teams more leverage. Companies therefore need to think carefully about what those saved hours mean for junior engineers who would previously have spent them diagnosing routine problems. The engineers who matter most in an AI-heavy environment will use these tools quickly and still understand what to do when the tool is wrong. Building that judgement means making sure junior engineers continue to get enough real problems to investigate and solve.

October 7, 2026
Federated Query at Petabyte Scale: A Deployment Pattern for a Governed AI-Agent Data Layer
Enterprises with data spread across ERP, CRM, cloud warehouses and observability platforms face a recurring problem: As data volume grows into the tens of petabytes, traditional ETL-and-centralize approaches produce unacceptable query latency and duplicated infrastructure cost. This case study describes an automated deployment framework for a distributed SQL query engine (federated query architecture) that queries data at its source rather than moving it, built and operated as the lead infrastructure engineer at a health care technology company. It further describes how that same data layer was extended into a governed, agent-accessible interface for enterprise GenAI, using a semantic layer and tool-exposure protocol (MCP-style) to connect a conversational AI interface to real-time business data. This architecture fully preserves the role-based access control and audit requirements appropriate for regulated (health care-adjacent) data. The contribution is a reusable deployment and governance pattern, not a speciﬁc vendor product.

Problem

A business intelligence function at a health care technology company had grown to ingest data from multiple ERP/CRM systems, multiple cloud data warehouses and observability/logging platforms, converging on roughly 30 petabytes of federated data. The broader data and analytics organization supporting this initiative numbered 30–35 people; the automated deployment framework and federated query infrastructure speciﬁcally were owned and built by this author as the team’s DevOps engineer. This growth happened faster than the data architecture could absorb: Each new source system arrived with its own connector requirements, its own access model and its own idea of what ‘current’ data meant.

The default response to this kind of sprawl is to centralize — build a pipeline that copies everything into one warehouse, then query the copy. At this scale, that approach broke down in a few speciﬁc ways:
- Query turnaround had no consistent ﬂoor. Depending on which systems a business question touched, getting an answer could mean waiting on a batch ETL cycle, requesting a manual data pull from another team or stitching together exports by hand. There was no reliable answer to how long it would take. This unpredictability was often more damaging to the business than any single slow query.
- Storage and pipeline maintenance were duplicated. Centralizing meant maintaining a second (or third) copy of data that already had an owner and a home. Every schema change upstream meant a corresponding ﬁx downstream, multiplied across every consuming pipeline.
- Governance got ﬂattened. Once data left its source system and landed in a shared warehouse, the ﬁne-grained access controls that existed at the source often didn’t carry over cleanly — meaning either over-provisioning access or building a second permissions system from scratch.
The alternative — federated query, where the engine queries data in-place at the source rather than copying it — solves the duplication and staleness problems but shifts the burden onto operating the query engine itself reliably. When this work began, distributed SQL engines built for this pattern (Starburst/Trino-based architectures) were still a relatively young category, and the available documentation focused on manual, one-time cluster setup. There was very little guidance on making that deployment repeatable, automated and safe to redeploy without losing operational history, which is the gap this work addressed directly.

Approach: Automated Federated Query Deployment

The core contribution was an integrated CI/CD deployment framework for a Kubernetes-hosted, distributed SQL query engine, built from the ground up rather than adapted from an existing internal playbook, as no such playbook existed for this engine at this scale.
- Automated Cluster Provisioning and Upgrade Pipelines: Cluster setup, conﬁguration and version upgrades were previously handled manually — a process that took 3–5 days per deployment cycle when done by hand and carried real risk of conﬁguration drift between environments (a setting changed in staging but forgotten in production, for instance). Moving this into a versioned CI/CD pipeline meant every deployment used the same tested conﬁguration, and the same pipeline that deployed to a test environment could promote to production with a predictable, auditable set of steps. This took deployment time down to a few hours — roughly a 10x reduction — but the more durable beneﬁt was consistency: Every engineer on the team could trigger a deployment with the same expected outcome, rather than the process depending on one person’s institutional knowledge of the manual steps.
- Log Persistence Across Deployments: Kubernetes pods are ephemeral by design. When a pod restarts or is redeployed, anything written only to its local ﬁlesystem is gone. For a query engine handling business-critical data, losing operational logs on every redeploy meant losing the ability to diagnose what happened before an incident. The available documentation for this engine, being relatively new to the ecosystem, didn’t have a clear answer for this. The solution was to route logs to persistent, external storage independent of pod life cycle, so redeploys (routine or emergency) never cost the team its operational history.
- Connector-Based Access to Source Systems: Rather than extracting data from ERP, CRM, cloud object storage and cloud data warehouse systems into a central store, the query engine connects to each source directly and executes queries against the live data. This eliminates the staleness problem inherent to batch ETL (data is only ever as old as the source system, not as old as the last successful pipeline run) and eliminates the duplicated storage cost of keeping a second copy of everything.
- Least-Privilege IAM as Code: Access control was provisioned through Terraform rather than conﬁgured manually, so that permissions granted to the federated query layer mirrored the access controls already in place at each source system, rather than introducing a new, separately managed permission model. This mattered speciﬁcally because a portion of the underlying data is subject to health care data protection requirements (HIPAA) — access needed to remain auditable and scoped, not ﬂattened into a convenient but overly broad tier just because the data now sat behind a diﬀerent query interface.
- Outcome: Business users can now run a single SQL query joining data across previously siloed systems and get a result in 15–30 minutes — down from a process that, before, had no consistent turnaround time at all, since it depended on ad hoc manual work across teams.
Extension: From Query Layer to AI-Agent Data Layer

Once the federated query layer was stable and fast, the natural next request from the business was: Could people just ask questions in plain language instead of writing SQL? That request is where most enterprise GenAI projects get into trouble. The easy version is to point an LLM at a document store or a database and let it retrieve whatever seems relevant, and this version is exactly what breaks down when some of that data is regulated.

The pattern built here keeps the LLM several layers removed from raw data at all times:
1. Data Products Layer: Rather than exposing raw tables, the federated query engine surfaces curated, named datasets — volume, feedback/case data, subscriptions, territory data and similar business-deﬁned products — as stable, documented interfaces. This layer is the ﬁrst checkpoint: Nothing downstream ever sees a raw table, only a data product someone has deliberately deﬁned and named.
2. Semantic/Governance Layer: A knowledge graph and catalog layer sits on top of the data products, carrying context (what does this ﬁeld mean, where did it come from), lineage (what upstream sources feed it) and access governance (who is allowed to see it). This is what lets an AI agent’s access be scoped and auditable, rather than ‘the model can see whatever the underlying database permissions allow’ — which is rarely a policy anyone actually designed on purpose.
3. Tool-Exposure Layer: A protocol-based server (following the emerging MCP pattern) exposes the governed data products as discrete, callable ‘tools’ that an LLM-based orchestrator can invoke — with deﬁned inputs and outputs — rather than granting the LLM a live database connection. The distinction matters: A tool call can be logged, rate-limited and scoped to a speciﬁc data product. A database connection generally can’t be constrained the same way once the model is composing its own queries.
4. Orchestration Layer: A commercial conversational AI product consumes the tool-exposed data to answer natural-language business questions inside a collaboration platform used company-wide. SQL generated in response to a user’s question is constrained to the governed data products deﬁned in step 1, and governing policies are applied before an answer reaches the end user — so the system’s freedom to interpret a question doesn’t extend to freedom to reach data it wasn’t given access to.
5. Compliance Boundary: Since a portion of the underlying data is subject to HIPAA, encryption, role-based access and audit logging are enforced at the data-product and semantic layers speciﬁcally — meaning the AI orchestration layer never receives ungoverned access to raw regulated data, regardless of how a user phrases their question.
The result — now in use by an estimated 300–500 business users across North America, LATAM, EMEA and APAC — is a system where a regional sales lead can ask a plain-language question in a chat interface and get an answer grounded in live, governed data without that convenience requiring anyone to loosen the access controls that existed before the AI layer was added.

This is, in eﬀect, a retrieval-and-tool-use pattern for enterprise AI agents that preserves existing data governance, rather than a RAG pipeline built on an unstructured document dump.

Why This Matters Beyond one Company

The pattern addressed here — automate federated query deployment ﬁrst, then expose that same governed layer to AI agents via a semantic/tool layer rather than piping raw data into an LLM — is broadly applicable to any enterprise trying to make GenAI safe to use against regulated or sensitive data. The speciﬁc components (query engine choice, LLM vendor, protocol implementation) are replaceable; the sequencing and governance boundary are the reusable insight:
- Don’t let the AI orchestration layer talk directly to source systems or raw warehouses.
- Put a data-products layer and a semantic/governance layer between the two.
- Automate the deployment of the query layer itself, or it becomes the bottleneck that makes ‘real-time AI answers’ impossible at petabyte scale.
Results
- Deployment Time: Manual Starburst cluster deployment/upgrade previously took 3–5 days. The automated CI/CD pipeline reduced this to a few hours — a 10x reduction.
- Query Turnaround: Cross-source business queries that previously had no consistent turnaround time (dependent on ad hoc extracts and manual joins across systems) now typically return in 15–30 minutes through the federated query layer.
- Data Volume Under Management: It is approximately 30 petabytes, federated across ERP, CRM, cloud data warehouse and observability sources.
- Adoption: The natural-language AI interface built on top of this data layer is in use by an estimated 300–500 business users across North America, LATAM, EMEA and APAC.
- Delivery: The initial version of the automated deployment framework was designed and released over 2–3 months, supporting a broader data and analytics organization of 30–35 people, with the deployment automation and query-layer work itself owned by this author.
Conclusion

Enterprises adopting GenAI over their own data face a governance problem before they face a modeling problem. This case study describes a deployment and architecture pattern — automated federated query infrastructure plus a semantic/tool-exposure layer — that let one infrastructure engineer scale a petabyte-class data platform into a governed, AI-agent-accessible system without compromising regulated-data controls.

Key Takeaways
- - At petabyte scale, federated query (query-in-place) avoids the staleness, duplicated storage and ﬂattened governance that centralize-then-query approaches introduce.
  - Automating the deployment of the query engine itself, not just the data pipelines, is what makes federated query operationally viable cutting a 3–5-day manual process to a few hours.
  - Ephemeral infrastructure such as Kubernetes pods requires deliberate log persistence design, or operational history is lost on every redeploy.
  - Making enterprise data safely available to an LLM agent requires a data-products and semantic/governance layer between the model and raw sources, not direct database access.
- A tool-exposure layer (MCP-style) lets an AI agent’s access to governed data be logged, rate-limited and scoped in ways a raw database connection cannot be — preserving existing compliance controls such as HIPAA even as AI is added on top.
October 7, 2026
Security Teams Are Fixing Vulnerabilities Faster Than Ever. So Why Is Software Getting Riskier?

For years, the software industry’s answer to vulnerability overload has been straightforward: Find issues earlier, prioritize them better and fix them faster.

That strategy is working. Automated dependency updates, policy-driven workflows and CI/CD integrations have accelerated remediation. Recent longitudinal data shows the median age of unresolved Critical and High vulnerabilities fell 59% from its January 2024 peak, while more than half of resolved violations are now addressed within a single day. Yet applications continue accumulating risk as software production and vulnerability discovery accelerate. Addressing that imbalance requires teams to look further upstream at how software decisions are made in the first place.

AI is Accelerating Dependency Decisions

Developers have always made trade-offs when selecting dependencies, weighing functionality, compatibility, maintenance, security and organizational requirements. What’s changing is the speed and volume of those decisions as AI coding assistants and agents take on more development work.

Coding agents can compress hundreds of component decisions into a much shorter development cycle. That creates a significant productivity opportunity while putting pressure on security controls that evaluate those choices only after they’ve been made.

Consider what happens when a developer or coding agent selects a vulnerable dependency. A downstream scan identifies it, an alert or ticket lands in the backlog, someone evaluates the issue and the developer is eventually asked to revisit the dependency. By then, the feature may be finished and the developer may have moved on to something else. An earlier decision has now become rework.

Some of that rework is avoidable. In many cases, lower-risk versions of vulnerable components are already available when a dependency is selected. The challenge is giving developers and AI agents enough current context to understand which choice carries less risk at the moment they’re making it.

Make the Safer Choice the Easier Choice

Faster remediation will always be essential. New vulnerabilities will continue to emerge in software that was considered secure when it was built, so organizations will always need to identify and fix problems downstream.

Teams can reduce some of that work by bringing security context closer to the original decision. If a dependency has a critical vulnerability, violates policy or has a materially safer alternative, developers and agents should have that information when they’re selecting it.

Earlier context means fewer tickets pulling developers back into completed work and fewer avoidable findings entering security queues. As agents take on more development work, those same signals and guardrails need to be available to them as they make component decisions.

The goal should be to make good security decisions part of the development workflow without adding another layer of manual review. Developers shouldn’t have to research the constantly changing risk profile of every component, and agents need access to the information required to make informed choices.

What Developer Teams Should Do Differently

Moving security earlier should make developers’ jobs easier while allowing security expertise to inform decisions without adding unnecessary friction. Four changes can help:

Bring security intelligence to the point of selection. Surface current information about vulnerabilities, component health and organizational policy while developers and agents are choosing dependencies. Giving them that context upfront can reduce issues that would otherwise surface later in development.

Give agents the same guardrails as developers. Organizational policies around approved components, risk and licensing should inform the software decisions agents make, just as they inform human development. As agents gain greater autonomy, those guardrails need to travel with them.

Automate the path to a safe upgrade. Identifying an outdated or vulnerable dependency is only the beginning. Automate routine version selection, compatibility checks and testing where possible so developers receive a validated path forward rather than another alert to investigate.

Measure what enters the pipeline. Teams already track how quickly vulnerabilities are addressed. They should also measure how frequently avoidable risk is introduced in the first place. If fixes are getting faster while the backlog keeps growing, the development process itself deserves a closer look.

None of these changes require slowing development. In fact, reducing downstream security rework becomes increasingly valuable as AI helps teams ship more software faster. The more decisions that can be made the first time correctly, the fewer interruptions developers face later.

Reduce the Rework

Prevention and remediation increasingly need to work together. Every avoidable vulnerability that enters an application creates downstream work, from triage and ticketing to fixing, testing and deployment. Preventing some of that work gives developers more time to build and security teams more capacity for the risks they can’t avoid.

AI will continue accelerating software development and increasing the number of decisions made along the way. Security practices need to keep pace with that development model. After years of optimizing how quickly vulnerabilities can be fixed, teams now have an opportunity to improve the decisions that determine which risks enter software in the first place.

October 6, 2026
Availability Should Follow the Workload, Not the Infrastructure

I find that I cannot stop asking myself this question… How did we automate almost everything in software delivery, yet when production fails, we still throw humans at the problem?

Think about what modern platform engineering has accomplished. We provision infrastructure with code. CI/CD? We build pipelines that can deploy hundreds of times a day. We use GitOps to keep environments consistent. Workloads? Ours can automatically scale based on demand. Kubernetes? We can spin up an entire Kubernetes cluster in minutes. That is incredible progress.

Then a production database fails at 2:13 a.m.

The Slack channel lights up. Someone starts a bridge call. Someone else starts digging through a runbook that hasn’t been updated in a year. And everyone waits for the one engineer who “knows how this thing works.”

How is that still our operating model?

We’ve spent the last decade teaching our platforms how to deploy software. We haven’t spent nearly as much time teaching them how to respond when production breaks. That’s the next maturity curve for platform engineering – not more deployment automation, but intelligent operational automation.

One of the biggest misconceptions I see? Organizations that equate modern infrastructure with resilient infrastructure. They’re not the same thing, not at all. Kubernetes is fantastic at orchestrating workloads. Better than any platform before, it can restart a pod, replace a failed node, and reconcile desired state.

But Kubernetes doesn’t know whether your customers can still place an order, process a payment, or complete a transaction. That’s not a knock on Kubernetes. It’s simply not its job. The problem is we’ve started treating orchestration as if it were a complete reliability strategy. It isn’t.

Orchestration answers the question, “Where should this workload run?”

Reliability answers a different question: “How do I keep this workload available when something inevitably fails?”

These are not the same problem. In modern environments, availability shouldn’t be tied to a server, a cluster, or even a cloud. It should follow the workload.

Here’s the question every platform team should ask. When I speak with engineering teams, I tell them to forget buzzwords for a minute. Forget “cloud native.” Forget “self-healing.” Forget “high availability.” Instead, ask one simple question, “If something breaks in production, what should happen next – and how much of it still depends on people?”

If the honest answer is, “Well…someone gets paged, we jump on a call, and then we figure it out,” your platform isn’t as automated as you think it is. It’s not a criticism. It’s an opportunity.

The goal isn’t to remove engineers from the process. It’s to eliminate the routine operational decisions they’ve already made hundreds of times before. Engineers should spend their time solving new problems – not repeating the same recovery steps every time a server, node, or workload fails.

So…What Should You Do Differently on Monday?

Here’s where I’d start.

Find your 2 a.m. processes.

Make a list of everything your team does manually during a production incident. Not deployments. Not upgrades. Failures. If only one person knows how to recover a critical workload, that’s not expertise – it’s technical debt. Recovery shouldn’t live inside someone’s head. It should be built into the platform.

Measure application recovery – not infrastructure recovery.

Most teams know exactly how long it takes to deploy a release. Far fewer know how long it takes for a critical application to recover from a real failure. Your users don’t care how quickly a pod restarted. They care how quickly they can get back to work. Measure what they experience. Better yet, start measuring how many recovery decisions still require human intervention. That may be the best indicator of platform maturity you have.

Test recovery as often as you test deployments.

Most organizations exercise their CI/CD pipelines constantly. In a controlled way, how often do you deliberately break production to see what happens? Is the answer, “Almost never?” – then I think you have found your next engineering project.

Stop assuming Kubernetes solved everything.

Kubernetes is one of the best orchestration platforms ever built. It’s also just one layer of your platform. Stateful applications, databases, storage, networking, and application dependencies all have their own failure modes. Make sure your operational strategy accounts for them. Modern platforms aren’t resilient simply because they run on Kubernetes. They’re resilient because every layer – from infrastructure to the application itself – is designed to recover intelligently when something breaks.

Eliminate decisions – not just steps.

The goal isn’t to make your outage runbook shorter. The goal is to make it unnecessary. The best platforms don’t just automate tasks – they automate decisions that have already been defined through policy and testing. That’s how you reduce downtime and reduce stress on your engineers at the same time.Here’s a simple test. If your most experienced engineer took two weeks off tomorrow, would your recovery process work exactly the same way? If the answer is “probably,” your platform still depends on tribal knowledge. Mature platforms don’t just automate tasks – they automate operational decisions.

The Next Frontier Isn’t Deployment. It’s Intelligent Operations

For years, we’ve measured platform engineering by one question: “How fast can we deploy?” I think there’s a better question for the decade ahead: “How intelligently can we recover?”

The answer has less to do with where an application runs than with how easily it can keep running when conditions change. Modern enterprises aren’t choosing one environment over another. They’re operating across on-premises infrastructure, virtual machines, Kubernetes clusters, public clouds, and edge environments – all at the same time. Reliability can no longer be tied to a particular server, cluster, or cloud. It has to follow the workload.

The best platform teams won’t be the ones that deploy the fastest. They’ll be the ones whose applications remain available no matter where those workloads are running – or where they need to move next.

That’s the next evolution of platform engineering: Building platforms where operational intelligence, not infrastructure, determines availability.

October 6, 2026
Taste, Discernment, and Judgment Have Become Must-Have Engineering Skills

Writing code stopped being the constrained step. Agents take a task, read the repository, change files across a codebase, and iterate against the build and test feedback that already exists. What they cannot do is decide whether the result belongs in the system. That decision is where engineering expertise now sits, and it did not get easier when the typing did.

I spent years at Uber on the parts of the system that decide whether anyone ships, the monorepo and the build and the CI queue, and later on training models against that same codebase. Making it cheaper to produce a change moves the pressure downstream onto everything that has to certify the change is safe. That is most of what happened to our industry in the last two years, and it arrived faster than anyone had planned for.

Taste is Your Standards, Written Down

Taste in engineering is not preference. It is knowing what makes a change fit the system it lands in, which is a different question from whether it satisfies the request. An agent will satisfy the request, but it will not know about the architectural constraint someone argued about for three weeks, two years ago, or the convention the team enforces in review and never wrote down, or the dependency direction that is not supposed to reverse. Nobody put any of that in a prompt. It is in people’s heads.

An agent cannot read a person’s mind. So, the practical form of taste is writing the standard down where the code lives, as rules and review guidance under version control, sitting next to the service they govern. Kept there, the guidance gets reviewed like code, since it shows up in the same diff as the change it governs, in front of the person best placed to notice it has gone stale.

Volume works against you. Hand a model forty pages of standards and it applies them about as well as a new hire would. The guidance has to be cut down to what applies to the change in front of it. Standards you cannot pick from are standards the agent averages out.

Discernment is the Line Between Proposing and Deciding

The failure mode is code that compiles, reads fine, and is wrong for this particular system, which is not a thing a diff shows you. Sonar’s research found 61% of developers agree AI often produces code that looks correct but is not reliable, and 53% said it has already hurt their technical debt. Those numbers describe a review problem rather than a generation problem, and they are not going to be fixed by a better model.

The obvious failures are cheap. It is the expensive ones that get through, the change that solves the immediate problem and quietly leaves the system harder to test or operate, where every individual line reads fine and the damage only shows up if you already know how the pieces fit together. Catching that is discernment, and it comes from having run the system in production. Knowing the call paths and what the tests actually cover, which is never what the coverage number says. It’s knowing how the service behaves at 3 am.

We ended up building the same boundary into our own product, on the theory that a model is a bad place to keep a decision. The model proposes findings, the verdict is computed in code from the state of those findings rather than declared by the reviewer, and that one change removed an entire class of bugs where the reviewer would request changes on a pull request whose findings had all already been resolved.

Resolution works the same way, as a deterministic check against the diff, and the model does not get to un-resolve what the diff settled. When several reviewers run in parallel, the arbiter can only drop candidates, never add one. An arbiter having a bad day costs us a miss rather than an invention. The rule underneath all of it is to give the probabilistic layer the jobs where being wrong is survivable and keep the state machine deterministic.

Judgment Decides What Ships, and What is Allowed to Ship Itself

Almost nothing at this level is right against wrong. You are picking which failure you would rather have. Take the migration now and eat a week of dual writes, or defer it and let the schema rot another quarter, and the engineer who has carried the pager for that service knows things about how it fails that never make it into a prompt.

The judgment itself has not changed. Where it gets applied has. It used to be spent one pull request at a time, in review. Now most of it is spent once, in writing, on the conditions under which a machine may act without asking anybody.

Teams get there in a predictable order. Detection first, then remediation, then approval under criteria they wrote themselves, then merge. Nobody jumps to the last step, and what moves them along is evidence out of their own codebase rather than a benchmark.

Constrain by scope, not by instruction. Our agent cannot push a branch or merge a pull request on its own initiative, and that is blocked at the tool level rather than asked for in a prompt, since an instruction is a preference and a missing capability is a guarantee. Automatic approval can only approve, never request changes, and it fails open, so a broken judge costs a convenience instead of a merge. Scoping the objective matters as much as scoping the permissions, which is why a CI failure unrelated to the change goes down the retry path, instead of the fix path. “Make the test stop failing” is the objective you least want a capable agent pursuing.

The Bar Does Not Have to Drop

This part we can measure. Auto-approve and auto-merge leave us a cohort where our agent was the only reviewer on the pull request, and that share keeps climbing. Those merge much faster, which surprises nobody, and the changes in them run larger on average, which I did not expect and still cannot fully explain.

The part worth reporting is what did not move. Changes get requested at about the same rate whether the reviewer was a person or our agent. The agent blocks slightly more often than people do, not less. And developers dismiss its findings less often than they dismiss findings from each other. Anyone would have predicted the speed. What I did not expect was that taking the second reviewer out would leave the remaining one behaving much the way it had before, and that is the part I would want another team to try to reproduce.

It is correlational, and teams that switch autonomous merge “on” are not a random sample of teams. These are also in-review signals rather than post-merge ones, since reverts and incidents take longer to attribute. It is still the only comparison group I know of.

Make the Expertise Repeatable

Nobody reviews their way out of this by hand once generated code is most of the diff. Enforcement belongs on the layer that behaves the same way every time. Whether a merge gets blocked should not depend on which model happened to be cheap that week. Quality gates, tests, security scanning and policy checks decide. Contextual review takes the judgment calls, whether a change does what it claims and whether it fits the codebase, and hands them over as evidence for a person to weigh.

Auditability splits the same way. Identical reruns of a model are not a promise worth making, and an audit does not need them. What it needs is a record of what was reviewed, what was raised, and who decided. Every finding we produce is persisted and never deleted, only marked resolved or dismissed, so a pull request can be reconstructed a year later by someone who was not there.

Businesses own the software they release regardless of who typed it. Accepting generated code without reading it buys delivery now and pays for it later, with interest. The teams that get the most out of coding agents will be the ones that scale their standards as fast as they scale generation, and keep the decisions that need context with the people who are accountable for them.

October 6, 2026
QA Automation Frameworks for Enterprise Applications
Modern enterprise applications demand testing strategies that can keep pace with continuous delivery, distributed architectures, and increasingly complex deployment pipelines. Traditional manual testing approaches often become a bottleneck as engineering teams scale their applications and release software more frequently. This article explores the essential building blocks of an enterprise-grade QA automation framework, focusing on maintainability, scalability, reliability, and integration with modern DevOps workflows.

Why Enterprise QA Requires a Different Approach

Unlike small applications, enterprise systems typically consist of multiple services, APIs, web applications, background workers, and third-party integrations. A successful automation strategy must support:
- Repeatable and deterministic test execution
- Parallel execution across multiple environments
- Continuous validation within CI/CD pipelines
- Cross-browser and cross-platform compatibility
- Reliable API and end-to-end testing
- Actionable reporting and failure diagnostics
Core Components of an Enterprise Automation Framework

A mature automation framework generally includes several layers working together:

Test Architecture

A modular test architecture separates test logic, reusable components, configuration, and reporting. Common patterns such as the Page Object Model (POM) or Screenplay Pattern help improve maintainability as test suites grow.

API Testing

API validation provides rapid feedback and verifies business logic before UI testing begins. Automated API tests are typically integrated into every deployment pipeline to detect regressions early.

UI Automation

User interface automation validates complete user workflows while remaining independent of implementation details whenever possible. Stable element identification and synchronization strategies help reduce flaky tests.

Test Data Management

Reliable automation depends on predictable test data. Isolated environments, seeded databases, and automated cleanup procedures help ensure repeatable execution.

Continuous Integration

Automation delivers the greatest value when executed automatically as part of every build. CI pipelines should provide immediate feedback while generating reports that allow teams to identify failures quickly.

Best Practices
- Successful enterprise QA automation initiatives commonly follow these principles:
- Prioritize API tests before UI tests whenever possible.
- Keep automated tests independent and deterministic.
- Avoid duplicated test logic through reusable components.
- Execute tests in parallel to reduce pipeline duration.
- Monitor flaky tests and continuously improve stability.
- Treat test code with the same engineering standards as production code.
Common Challenges

Organizations implementing large-scale automation frameworks often encounter:
- Long execution times
- Flaky end-to-end tests
- Poor test data management
- Environment inconsistencies
- Difficult framework maintenance
Addressing these challenges requires disciplined engineering practices, continuous refactoring, and close collaboration between development, QA, and DevOps teams.

Conclusion

Enterprise QA automation is not simply about increasing the number of automated tests. Effective frameworks provide fast feedback, improve software quality, reduce deployment risk, and support continuous delivery at scale. By combining sound architecture, reliable automation patterns, and CI/CD integration, engineering teams can build testing platforms that remain maintainable as applications continue to evolve.
October 6, 2026
AI Is Breaking Developers’ Traditional Game Testing Playbook

Game testing has always been an unpredictability problem for developers. Players move in the wrong direction, trigger events out of order, or combine mechanics no designer anticipated. Even so, developers could usually define the rules governing the world and check whether the game followed them.

AI is making that harder. NPCs now react to changing conditions, and animation systems select behaviors based on surroundings rather than a fixed sequence. Take-Two Interactive, Rockstar Games’ parent company, holds a patent for a virtual character locomotion system that uses modular animation building blocks and selection criteria to control how characters move through a 3D environment, one sign of character behavior becoming more responsive to context instead of scripted.

That can make for a more convincing game, but it leaves developers wondering: How do you test software when you can’t list every way it might behave?

The Test Matrix is Getting Harder to Define

Games already have a larger testing surface than most applications, including a physics engine, real-time movement, character states, inventory systems, dialogue trees, and multiple paths through the same scene. Each of these combinations multiplies the number of states to test.

Consider an NPC encountering a player. In a scripted system, entering a certain area causes the NPC to walk to a fixed spot and start a line of dialogue, a known input with an expected result. Now let the NPC’s response depend on the player’s location, earlier interactions, nearby objects, or what another character is doing. Several responses may all be valid, and there’s no longer a single sequence to verify. Multiply that across hundreds of characters and a large player base, and exhaustive testing stops being realistic.

Scripted testing still matters for known functionality, regression testing, and core mechanics; it just can’t catch a failure that comes from a combination nobody thought to write into the test plan.

Manual QA Can’t Try Every Combination

Human testers are good at judgments that resist a pass/fail test, including whether movement feels wrong, a mechanic is poorly balanced, or an interaction technically works but makes the game less fun. What they can’t do is exhaust every combination of actions and states, and that limit was already showing before AI entered the picture. LiveOps games, like Roblox, Grand Theft Auto Online, and Skate, ship frequent updates and seasonal content, which means repeated regression testing across devices, operating systems, and hardware configurations. This is a burden that grows with every release.

AI-driven gameplay adds behavior on top of that matrix. A QA team can run the same quest dozens of times without issue under numerous circumstances, like a player who carries a different item or approaches from another direction. However, all it takes is one tiny circumstance, such as picking a rarely used dialogue option, to expose a bug immediately. The team didn’t miss an obvious test. The combination simply never came up.

The industry’s response has been to lean harder on automation rather than headcount. By 2026, about 47% of gaming studios reported using AI for QA and playtesting, and those studios say it has cut testing costs by 30% to 40%. Square Enix has said publicly it wants 70% of its QA pipeline automated by 2027. The drive for AI-driven QA processes is generating new ways, to quickly and securely test code, like deploying synthetic players.

Synthetic Players Can Cover More Ground

Instead of following the same fixed path on every run, automated players can move through a game, interact with mechanics, and try different combinations of actions continuously. When my team was at GDC 2025, we witnessed AI-driven synthetic-player systems navigating game worlds: jumping, colliding with objects, moving through menus, and interacting with mechanics.

And speed isn’t the only value of this. These players can spend time on interactions a human tester would rarely repeat, like trying every path through a quest, carrying different inventory combinations, approaching a character from several angles, and running it all again. A synthetic player might find errors like a camera angle that hides an important UI element, or the one combination of inventory and dialogue choices that causes a crash after hours of trying.

None of that removes the need for human testers. Instead, it changes where their time goes. Machines take the repetitive exploration while developers spend more time on game feel, balance, pacing, and whether an interaction makes sense from the player’s perspective.

Production Tells You What the Test Suite Missed

Even an extensive pre-release test suite won’t reproduce every combination that real players eventually create, which makes production data useful for more than incident response. The traditional loop is reactive: crash, report, investigation, and fix. But a recurring production failure also tells developers what to test in the next build.

Roblox is a useful example of how this looks at scale. As its community grew past 10 million players, the team had to work through millions of crash reports across Windows, Linux, Mac, mobile, and consoles, and needed to identify the crashes hurting players most without drowning developers in an undifferentiated stream of reports.

Using advanced error reporting capabilities, Roblox aggregated incoming crashes and exceptions, grouped related errors, and used crash classifications and other diagnostic information to narrow down root causes. Roblox senior engineer Christopher Swiedler said the system cut the time needed to generate crash reports while improving their accuracy, and the case study also reports a 50% increase in monthly active users after implementation.

AI-Driven Games are an Early Test of a Broader Software Problem

Gaming makes this challenge especially visible because variation is the goal. Players are supposed to explore, and characters are supposed to react. But the problem isn’t unique to games. Developers are integrating AI into applications that make decisions, respond to context, and can produce more than one acceptable result.

Conventional regression tests still matter, and so does manual testing. But teams will also need to explore combinations they didn’t explicitly script, define the limits of acceptable behavior, and keep enough information to reproduce unusual failures when they occur. Developers are never going to test every move a player can make or every action an AI-driven character can take, and now, with the help of AI tools like synthetic players and advanced error reporting, they don’t need to.

October 6, 2026