RAG in Production: What Breaks and How to Fix It

Retrieval-augmented generation is the most useful pattern in applied AI right now, and the easiest to build badly. A demo comes together in an afternoon: chunk the documents, embed them, search on a question, hand the results to a model.

Then it goes in front of real users and the answers are subtly wrong, or confidently wrong, or right about a policy that changed last March.

Nearly every one of these failures is in the retrieval half. The model is usually fine. It is answering faithfully from material that should never have been given to it.

Failure one: chunking that severs meaning

The default approach — split every 500 characters, overlap by 50 — cuts through the middle of tables, separates headings from what they introduce, and splits a procedure across two chunks so that neither contains the whole answer.

Retrieval then returns a fragment that looks relevant and is missing the part that mattered.

What to do instead: chunk on document structure. Split on headings and sections, keep tables whole, keep list items with their introduction. Attach the document title and section heading to every chunk as a prefix, so a fragment about "the fee" still carries the context that it is the late-payment fee in the enterprise contract.

Chunk size should follow the shape of your content, not a round number someone picked in a tutorial.

Failure two: pure vector search

Embeddings capture meaning and are poor at exact tokens. A part number, an error code, a person's surname, a specific clause reference — these are precisely the queries where a user expects an exact match, and where vector similarity quietly returns something thematically adjacent.

What to do instead: run both. Keyword search alongside vector search, merged. Hybrid retrieval is the single largest quality improvement available in most systems, and it is not difficult — Postgres will do full-text and vector search in the same query.

Then re-rank. Retrieve twenty candidates cheaply, score them against the question with a re-ranking model, keep the best five. Cheap, and it consistently outperforms retrieving five directly.

Failure three: no metadata filtering

A question about the current refund policy retrieves the 2023 version because it is textually similar. A question from a UK customer retrieves the US terms.

What to do instead: store metadata with every chunk — source, date, version, region, audience, access level — and filter before ranking. Most "the AI gave the wrong answer" reports are actually "the AI gave a correct answer from the wrong document".

Access control belongs here too, and it must be enforced in the query rather than after retrieval. Filtering results after the model has seen them is not access control; the content has already reached the context.

Failure four: a stale index

The documents change. The index does not. The system answers confidently from a policy that was replaced in spring.

This is the failure that damages trust most, because the answer is well-formed and plausible and there is no signal that it is out of date.

What to do instead: re-index on change, driven by an event from wherever the content lives. Store the source's last-modified date in the metadata and surface it in the answer. Monitor the age of the oldest document in the index — if that number is climbing, your pipeline is broken and nothing else will tell you.

Failure five: no way to know if it is working

Teams ship RAG systems with no evaluation beyond people trying it and saying it seems good. Then a prompt is changed, or a model version updates underneath them, and nobody can tell whether things got better or worse.

What to do instead: build an evaluation set early. Fifty to a hundred real questions with the answer and the document that contains it. Then measure the two halves separately:

Retrieval: was the correct document in the retrieved set at all? This is the number that predicts everything else, and it is where the problems almost always are.

Generation: given the correct context, was the answer right and did it stay within the source?

Separating them tells you where to work. If retrieval accuracy is poor, no model change will help.

Failure six: costs and latency discovered in production

Every question means an embedding call, a search, often a re-rank, then a generation with a large context. It is fine at ten questions a day and a different conversation at ten thousand.

What to do instead: log token counts and latency per request from day one. Cache embeddings for repeated queries. Keep contexts tight — more retrieved text is not better, and beyond a point it measurably degrades answers as well as costing more.

Set a budget alert before launch rather than after the first surprising invoice.

What good looks like

Structure-aware chunking with contextual prefixes. Hybrid retrieval with re-ranking. Metadata filtering including access control at query time. Event-driven re-indexing with freshness monitoring. Source citations on every answer. An evaluation set that runs in CI so a prompt change cannot silently regress quality. Logging of retrieved chunks, so any bad answer can be traced to what caused it.

None of that is exotic. All of it is the difference between a demo and a system.

Before you build one at all

Ask whether search would do, and whether the use case is one of the AI features actually worth building. If your users are looking for documents rather than answers, better search is cheaper, faster and easier to trust — and it does not fabricate.

RAG earns its place when the answer must be synthesised from several sources, or when the question is asked in language that does not match the document's vocabulary. That is a real and common need — support answering is the most frequent version of it. It is just narrower than the enthusiasm suggests.

If you have a retrieval system that was impressive in the demo and disappointing since, tell us what it is getting wrong — the diagnosis is usually in the retrieved chunks, and it is usually fixable.