When we started using large language models for API test generation at KushoAI, the results were impressive on the surface. Tests appeared in seconds. Coverage breadth went up. The team was excited.
Then we looked more carefully at what was being produced.
On one endpoint, the model generated 26 tests where a senior QA engineer would have written 9. On another, it completely missed the boundary cases that mattered most. On a third, it generated tests that checked whether a response arrived but said nothing meaningful about what the response contained. The tests were valid. The test suite was not useful.
What we had was a fast generator with no taste, and in testing, taste is almost everything.
This article is about what we learned while building toward something better: How we diagnosed what was actually wrong, why common fixes such as better prompting fall short, and what the architecture looks like when you treat test generation as a judgment problem rather than a code generation problem.
The Six Ways Fast Test Generation Fails
After running our pipeline across thousands of real APIs, six failure modes consistently emerged. Understanding them precisely is what pointed us toward the right solution.
Over-generation is the most visible problem. The model produces 20 to 30 test cases where 8 would be sufficient. The excess cases are often plausible but redundant, and they make suites harder to review and trust. A QA engineer looking at a 30-test suite for a single endpoint will struggle to identify what is actually being covered versus what is noise.
Under-generation is the opposite and arguably more dangerous. The model skips obvious scenarios — required fields left out, invalid enum values, boundary conditions, malformed formats, missing authentication cases and expected error responses. These omissions are hard to see precisely because they are absent.
Redundancy is subtle. Three generated tests labeled ‘invalid value’, ‘incorrect value’ and ‘unsupported value’ often map to a single check. The model treats them as distinct because the phrasing differs. A human reviewer sees immediately that they are the same assertion stated in three ways.
Inconsistency is what makes a test suite hard to maintain. Two structurally similar endpoints receive vastly different treatment on each run. One gets thorough positive, negative and boundary coverage. The other gets five tests, mostly a happy path. There is no predictable standard.
Weak assertions are perhaps the subtlest failure. A generated test checks if a response arrives and matches a broad schema shape. What it does not check is whether the transaction ID is absent when a payment is rejected, whether the error code is machine-readable rather than a human string or whether the rejected state is reflected consistently across nested fields. The test passes. The bug ships.
Cross-field relationship failures are the hardest to generate and the most expensive to miss in production. Individual fields are tested in isolation: Is the amount missing, is the currency wrong, is the payment method invalid. What gets skipped is the interaction: What happens when the amount is valid, the currency is valid, the payment method is valid and the combination creates an invalid business state. These failures do not trigger a 500 error. They return a 200 with corrupt data somewhere three levels deep in a nested response.
All six of these are failures of judgment, not knowledge. The model knows what an HTTP endpoint is. It knows what schema validation means. What it lacks is the calibration to decide what deserves to be tested, at what depth and with what assertions.
Why Prompting Helps but Plateaus
The natural first response to these problems is better prompting. We did a lot of it: Structured prompts, multistep chains, context injection, explicit output schemas, few-shot examples and detailed instructions about redundancy, coverage breadth and assertion specificity.
Prompting helped. It improved structure and reduced the most obvious redundancy. Few-shot examples produced the largest single improvement, pulling outputs closer to what a reviewer would keep.
That result was informative. If a few examples in the prompt improve behavior, the missing ingredient is broader exposure to examples of preferred QA decisions. The model already has the general capability. It needs calibration toward specific judgment patterns.
But prompting plateaus. It mostly improves field-level exhaustiveness: More missing-field tests, more wrong-type tests and more boundary tests. These are useful, but they remain independent mutations. Prompting does not reliably produce reasoning about field relationships, business states and workflow behavior. The hardest failure mode — cross-field relationship testing — is the one that prompting improves least.
This is exactly what we see in our public benchmark, APIEval-20. Schema-level bugs are easier to generate tests for. Complex bugs that only surface when multiple valid fields combine into an invalid state are systematically harder. Prompting narrows the gap but does not close it.
Separating Judgment From Mechanics
The architectural shift that actually helped was treating test generation as two distinct problems requiring different models.
The first problem is judgment: Which scenarios should exist, which edge cases matter, what should be asserted, how much is appropriate and what is redundant. This is a calibration problem. It requires exposure to many examples of reviewed QA decisions across many API types.
The second problem is mechanics: Constructing valid payloads, generating executable tests, writing framework-specific output and handling authentication context. This is a code generation problem. General-purpose frontier models already handle it well.
We fine-tune narrow models for the judgment layer. Frontier models handle the mechanics layer. The boundary between them is the most consequential design decision in the architecture.
The test intent model takes API metadata and a target field, operation or group of fields and then outputs a structured list of scenarios in plain language. It is responsible for coverage breadth, relevance, redundancy reduction and cross-field scenario selection. For a payments endpoint, it should not only produce tests for a missing amount or a wrong currency type but also identify cases where amount, currency, payment method, refund status and idempotency key interact to create invalid business states.
The assertion intent model takes an expected response schema and an observed response and outputs validation statements in plain language. The difference between a weak assertion and a useful one is specificity. A weak assertion says the response should be valid. A useful set says the status should be 400, the error response should include a machine-readable error code, no transaction ID should be issued and the rejected state should be reflected consistently in the response body.
Both models output intent, not executable code. The frontier model receives that intent and turns it into payloads and tests. This keeps the fine-tuned surface small and easy to evaluate. When the frontier model changes, the judgment layer remains undisturbed.
What Gets Fine-Tuned and Why
A reasonable question is whether retrieval-augmented generation (RAG) solves this instead of fine-tuning. The distinction matters.
RAG supplies context the model does not have — internal documentation, product behavior, customer-specific policies, historical incidents and organization-specific testing standards. It is the right tool when the system needs information.
Fine-tuning calibrates behavior the model already has but applies inconsistently: Coverage calibration, scenario prioritization, assertion quality, redundancy reduction and consistency across similar APIs. These are behavior-shaping problems, not retrieval problems. No amount of supplied context teaches a model which tests a QA engineer would keep. A mature system uses both as they are solving different problems.
What we fine-tune on is reviewed QA decisions, not raw API examples. For each example, the pipeline generates tests, a reviewer then removes redundant cases, adds missing scenarios, rewrites vague assertions, normalizes phrasing and stores the corrected output as the training target paired with its input context. The correction signals include accepted tests, rejected tests, redundant cases removed, missing edge cases added, weak assertions rewritten, flaky tests flagged after execution and cross-field cases added by a reviewer.
Each of these is an explicit judgment about coverage, prioritization or assertion quality. The dataset is not a collection of API examples. It is a collection of QA decisions.
Reviewer consistency is architecture, not preprocessing. Fine-tuning amplifies whatever patterns the data contains. If reviewers handle similar APIs differently, the model learns the inconsistency. Correction guidelines, accepted and rejected examples and review standards have to be documented and applied uniformly. The goal is a clean distribution of preferred decisions, not the largest possible pile of examples.
The Full Pipeline
The judgment and mechanics layers sit inside a seven-stage pipeline: Context ingestion; behavior modeling; coverage planning; test and assertion intent; payload and code generation; execution validation; feedback and maintenance.
To make this concrete, take a payments endpoint that accepts amount, currency, payment method, refund status and idempotency key. Context ingestion pulls in the schema, sample payloads, authentication requirements and any existing tests. Behavior modeling identifies the field relationships and the constraints between them. Coverage planning maps not just the individual field tests but the cross-field scenarios — what happens when all five fields are individually valid but combine into an invalid business state. The judgment models then define the specific test intents and assertion intents for that scenario in plain language. The frontier model takes those intents and produces the executable tests. Execution validates them against the live API. Any test that passes cleanly, fails unexpectedly or behaves inconsistently across runs feeds back into the correction workflow.
Since the layers are separate, a weakness in coverage planning can be diagnosed and improved without touching payload generation. A model provider can be swapped in the generation layer without disturbing the judgment layer. The layers that differ most across systems are coverage planning and intent, where judgment lives, and the feedback loop, where corrections accumulate. These are the layers worth investing in.
Execution Closes the Loop
Generated tests are only useful if they run, and if their results can be trusted.
Tests run against live APIs. Failures are isolated rather than discarded. The system identifies which part of a suite failed and repairs that part rather than regenerating everything. A suite that passes cleanly in one run and breaks in the next is scored as unstable rather than treated as correct. For high-risk changes, a human approval step gates deployment.
Execution feeds back into corrections. Accepted tests, rejected tests, edited tests and flaky tests all become signals that refine the judgment layer. This is what turns generated tests from a one-time output into a system that improves. The execution that validates a suite today produces the corrections that improve generation tomorrow.
Evaluation: What Usefulness Actually Measures
A generated test can be valid and still find nothing. Evaluation has to measure usefulness, not just validity.
The qualitative dimensions we track are coverage across scenario types, precision against redundant or low-value tests, calibration of test volume relative to API complexity, consistency across similar endpoints and across reruns, and assertion relevance. The execution-based metrics are bug detection rate against live implementations with known bugs, execution success rate, run-to-run variance, false-positive rate, flaky-test rate and the reviewer correction effort a suite requires.
The most informative evaluation method is blind reviewer comparison: Engineers judge outputs without knowing which system produced them. This surfaces quality differences that aggregate metrics obscure.
APIEval-20 is a public anchor for this. It is intentionally narrow: Each system receives a JSON schema and one valid sample payload, then generates tests executed against live APIs with planted functional bugs across 20 scenarios and 7 domains. The benchmark isolates a specific capability — whether generated tests can move beyond simple schema mutations and expose failures that depend on field relationships or operation semantics. That is where the separation between systems appears.
What This Points Toward
Most real failures do not live inside one isolated endpoint. They appear across state transitions, permissions, create-read-update-delete sequences, payment and refund flows, user onboarding and downstream side effects.
The clearest direction for adaptive test coverage is reasoning at the workflow level. The system already generates tests across related operations. The deeper work is reasoning about a workflow as a coherent flow, understanding how one operation’s state constrains the next, rather than testing each endpoint independently. Public benchmarks scope out these multistep and stateful scenarios today, which marks them as the next capability to build and measure.
Key Takeaways
API test generation is a calibration problem before it is a code generation problem. Models already know what APIs are. What they lack is the judgment to decide what is worth testing, at what depth and with what assertions.
The six failure modes in fast test generation (over-generation, under-generation, redundancy, inconsistency, weak assertions and cross-field relationship failures) are all judgment failures, not knowledge failures. Fixing them requires changing what the model is optimized for, not just what information it receives.
Fine-tuning and RAG solve different problems. RAG supplies information the model lacks. Fine-tuning calibrates behavior the model applies inconsistently. A mature system uses both; however, conflating them leads to the wrong architectural choices.
Separating the judgment layer from the mechanics layer makes the system easier to debug, improve and govern. Fine-tuned models decide what to test and what to assert. Frontier models construct payloads and generate executable tests. The boundary between them is the most important design decision in the architecture.
Execution feedback turns test generation from a one-time output into an improving system. Accepted tests, rejected tests, edited tests and flaky tests become the corrections that make the next generation sharper. The training signal is not API examples. It is reviewed QA decisions.
