Use Agile Regression Testing for Retrieval-Augmented Generation Applications
Retrieval-augmented generation apps can fail silently when chunking, embeddings, retrievers, prompts, or models change. This guide recommends testing retrieval and generation separately: golden sets measure whether expected chunks still appear, key facts check answer quality, LLM judges score faithfulness, and Keploy replays capture regressions. Agile teams can use these checks to detect retrieval drift, answer drift, format breaks, citation errors, and latency creep before releases reach users.
Key Takeaways
- RAG applications contain multiple moving parts, so changes to chunking, embeddings, retrieval settings, prompts, or models can cause silent regressions without obvious errors.
- Test each RAG layer separately: use golden sets and recall metrics for retrieval, key facts and faithfulness checks for generation, and contract tests for output format and citations.
- LLM-as-a-judge evaluation can help assess answer quality, but scores should be treated as trends and pinned to a stable judge model to reduce flakiness.
- Record and replay real model and retrieval calls, for example with Keploy, to create fast, deterministic tests for pipeline code changes on pull requests.
- Integrate regression testing into agile sprints so teams can detect degraded or wrong RAG answers before users encounter them.
Meta description: A practical guide to regression testing retrieval-augmented generation apps in agile teams: golden sets, retrieval checks, LLM judge tests, and Keploy replays.
Tags: retrieval-augmented generation, agile regression testing, golden set, LLM evaluation, Keploy
Featured image: abstract friendly illustration of layered document chunks, retrieval arrows, and test checkmarks, no text, no people, no logos.
Retrieval-augmented generation regression testing in agile teams matters when a sprint change to document chunking makes demo answers look better, then support reports refund questions wrong a week later. The refund documents and refund prompt were untouched, but the app’s answer changed because retrieval-augmented generation has several moving parts. This guide is for teams shipping AI assistants and other retrieval-augmented generation apps: it shows how to catch those silent regressions before they reach users.
Why silent regressions appear in retrieval-augmented generation apps
A retrieval-augmented generation app is not one component. It is a chain of small decisions. Someone chose how to slice documents into chunks. Someone chose an embedding model to turn those chunks into vectors. A retriever decides which chunks look relevant to a question. Only after all that does the LLM write the answer.
In agile development, those decisions can change often. Teams may adjust chunk size and overlap, swap the embedding model, change how many chunks are retrieved, add a reranker, rewrite the prompt, move to a newer LLM version, or ingest fresh documents. Any one of those changes can improve the answers the team was testing while making other answers worse.
The dangerous part is that nothing throws an error. There is no stack trace and no 500 response. The app simply answers differently, and sometimes that difference is wrong.
Most regressions fall into a few patterns. The most common is retrieval drift: the right document used to appear in the top results and now does not, so the model answers from the wrong context or from no useful context. Another is answer drift: retrieval is still fine, but the model phrases the answer in a way that drops a key detail or hedges where it used to be clear. Faithfulness can also slip, with the model adding claims that the retrieved text does not support.
Some regressions are less subtle but still easy to miss in a demo. A prompt change may alter the output format, and the frontend can no longer parse it. Citations may point at the wrong source. The answer may become twice as long, and latency may creep up. These problems can look minor in a controlled demo and still cause real trouble for users.
Test the layers separately
Trying to test the whole chain with one kind of test is usually frustrating. LLM output can wobble between runs even with the temperature at zero, so end-to-end tests flake and teams cannot tell why. A better approach is to test each layer with a method suited to that layer.
Retrieval: does the right context still show up?
Start with retrieval. This layer is easier to test properly, and it often gives the biggest return. Build a golden set: real questions that users ask, plus notes about which documents or chunks should come back for each one. Then measure how often those expected sources appear in the top results. That is basically recall at the chosen top-k cutoff, and it is a number the team can track sprint over sprint.
Retrieval tests are fairly deterministic as long as the index and embedding model do not change, which makes them useful regression checks. When the team does swap the embedding model or re-chunk everything, comparing these scores before and after can show exactly which questions got worse.
Generation: is the answer still right?
The generation layer is fuzzier. For each golden question, it helps to write down the key facts that a good answer must include, rather than one exact expected answer. Then check whether new answers still include those facts and remain faithful to the retrieved context.
Many teams use an LLM as a judge, scoring answers against a rubric. That can work, but the judge is also an LLM, so its scores can wobble. Pin the judge model and its settings, treat its scores as a trend rather than gospel, and inspect the actual answers whenever scores drop. Open-source evaluation libraries like Ragas can help with much of this scoring.
Contract: does the output still fit the app?
This is plain software testing, and teams often skip it. If the frontend expects JSON with an answer field and a citations list, check that shape on every run. If citations should point at real document IDs, check that too. These tests are cheap and deterministic, and they catch a surprising number of “it worked in the demo” bugs.
Make code changes testable on every pull request
Running live evaluations calls real models and real vector stores. That can be slow, costly, and slightly different each time. Teams generally do not want that on every pull request.
But a lot of sprint changes are not the model itself. They are the code around it: orchestration logic, how retrieved chunks are stitched into the prompt, how the response is parsed, and how citations are rendered. That code deserves fast, deterministic tests like any other code.
One practical way is to record a real run once, including model calls and retrieval calls, then replay those recorded responses in tests. Keploy, an open-source tool, does this by capturing app traffic with eBPF at the kernel level, without code changes, and saving outbound calls as mocks. That can cover HTTP calls to LLM APIs plus queries to vector stores running on Postgres or Redis. When replayed, the pipeline gets identical model and retrieval responses every run, so a failing test points to changed pipeline code, not to a model that felt creative that day.
Fit regression testing into the sprint
A useful rhythm maps well onto how agile teams already work.
- On every pull request, run the fast deterministic checks: contract tests and replayed pipeline tests. They finish in seconds and catch code-level breakage before merge.
- Nightly or mid-sprint, run retrieval checks against the golden set, especially if anyone changed chunking, embeddings, or the index.
- Before sprint review, run the full live evaluation with real retrieval, real model calls, and scored answers. Compare everything with the previous sprint’s baseline. If any question’s score dropped, look at it before release.
- Feed the golden set. Every time a user reports a bad answer, add that question to the set, along with what a good answer needs. Over a few sprints, the set becomes a more honest picture of what users actually ask.
- Version every moving part: prompt templates, chunking settings, embedding model, index build, and LLM version. When a regression appears, a team should be able to see a diff and say, “This sprint we changed chunk overlap and bumped the model,” rather than guess. This also makes rollback easier. If a change made refund answers worse, the team can revert just that part.
Common mistakes
The first mistake is testing only the whole chain. End-to-end tests can be useful, but if they fail, the team may not know whether retrieval, generation, formatting, or the LLM itself changed.
Another mistake is treating LLM judge scores as exact truth. The judge can drift. Keep its model and settings fixed, use scores as trends, and read the actual answers when quality drops.
A third mistake is skipping output contract tests. A response can be perfectly correct in prose but still break the app if the frontend cannot parse the format or the citations no longer point to valid document IDs.
The fourth mistake is not adding real user failures to the golden set. A question that used to work and now fails should not disappear into a support ticket. It should become a test.
The fifth mistake is leaving prompts, chunking settings, embeddings, index builds, and model versions unversioned. Without that history, a regression becomes a guessing game instead of a diff review.
Start with the golden set
Retrieval-augmented generation apps may need regression testing in agile even more than ordinary software. They can change in a dozen small ways every sprint, and hardly any of those changes throw an error. The safer habit is to layer the work: quick deterministic checks on every pull request, retrieval checks against the golden set during the sprint, and full live evaluation before each review.
If a team is building on retrieval-augmented generation now, start small. Pick twenty real questions users ask, write down which documents should answer each one, and check retrieval against that list before the next sprint review. That small effort will likely surface at least one question that the latest change made worse.
Comments (0)