Why most RAG demos die in production, and the five things that fix it

Most retrieval-augmented generation projects follow the same arc. A demo impresses leadership in a week, a pilot is approved, and three months later the assistant is quietly retired because nobody trusts its answers. The failure is rarely the model. It is everything around it.

1. Chunking that ignores the document

Fixed-size chunks cut tables in half and separate headings from the paragraphs they explain. Chunk by structure instead: sections, tables, list items, with the heading path attached as metadata. Retrieval quality on technical corpora improves more from this single change than from swapping embedding models.

2. Vectors alone miss the exact match

Dense retrieval is good at meaning and bad at identifiers, error codes and product names. Keyword search is the reverse. Fuse both with Reciprocal Rank Fusion, then rerank the top candidates. On one enterprise log corpus this hybrid approach lifted relevance on technical queries to 89%, where vectors alone plateaued far lower.

3. Permissions enforced at retrieval time

If the assistant can quote a document the user cannot open, you have built a data leak. Carry the source system's permissions into the index and filter before ranking, not after.

4. Citations that link to the exact passage

Users forgive a wrong answer they can check. They do not forgive a confident one they cannot. Every answer should point at the passages it used, and the interface should make checking a one-click action.

5. A golden set, run on every change

Collect the real questions people ask, label the expected sources and answers, and run that set whenever a prompt, model, chunker or index changes. Without this, quality drifts silently and the first person to notice is a customer.

None of this is exotic. It is the difference between a demo and a system, and it is where most of the effort in a successful RAG project goes.