RAG that actually works in production

Almost every team that builds a retrieval-augmented generation system overweights the model and underweights the retrieval. The model is the part that's fun. The retrieval is the part that determines whether the system answers correctly.

This piece is the eight-stage pipeline we use, in order. It is boring and it is reliable. If your RAG system is not doing all eight, your answer quality is a function of luck.

Stage 1: Parse

Most documents are not text. They are PDFs with multi-column layouts, tables that span pages, footnotes, and watermarks. Naive parsing — strip the text, split on newlines — destroys signal that retrieval needs. Use a layout-aware parser. Preserve table structure. Preserve heading hierarchy. Throw away the watermarks.

The investment in good parsing pays for itself within a quarter. It is the single highest-leverage thing you will do in your retrieval pipeline, and it is the thing that almost no team gets right on the first attempt.

Stage 2: Chunk

Chunking is not just splitting on a token count. Splitting on a token count is what you do when you've given up. Chunk on semantic boundaries: sections, subsections, paragraphs. Keep tables intact. Keep code blocks intact. Use overlap between chunks so that a query that straddles a boundary still hits something.

Concrete defaults: 400–800 token chunks, 80 token overlap, never split mid-table. If your documents are longer-form, layer in a hierarchical chunking scheme — section-level chunks for routing, paragraph-level chunks for retrieval.

Stage 3: Enrich

Before you embed, attach metadata. Document title, document type, last-modified date, authority level, applicable jurisdictions. This metadata becomes filter predicates at query time. A good chunk should answer: where did I come from, when was I written, who is allowed to see me?

Stage 4: Index

Choose your index for your access pattern, not for your résumé. A pgvector index is fine for tens of millions of chunks. A purpose-built vector database is right when you need filtering at scale, multi-tenancy, or replication guarantees that Postgres can't give you. The vendor matters much less than the discipline of measuring recall on a held-out set.

Stage 5: Retrieve

Hybrid retrieval (dense + lexical) is the new baseline. BM25 is forty years old and still finds keyword-heavy queries that embeddings miss. Run both, fuse the results with reciprocal rank fusion, and you'll outperform either alone by a meaningful margin in your evals.

Stage 6: Rerank

After retrieval, rerank. A cross-encoder rerank over the top 30 candidates is one of the highest-ROI additions you can make. It costs ~50ms and it routinely lifts answer quality scores by 10–20 percentage points on our benchmark sets.

Stage 7: Answer

Now you generate. Critical: the generation prompt should require the model to answer only from the provided sources, and to abstain if the sources are insufficient. Abstention is not failure — abstention is the system telling you it does not know. That is the behavior you want in any high-stakes domain. Anthropic's guidance on system prompts is a good starting point for prompt structure.

Stage 8: Cite

Citations are not garnish. They are the contract between your system and your users. Every claim should map to a source. The UI should make the citation hoverable — a click goes to the original document, with the relevant span highlighted. Without citations, your RAG system is a confident guesser. With citations, it is an assistant that does its homework.

What teams skip — and why it costs them

• Skipping reranking. Saves 50ms, costs 15 points of accuracy.
• Skipping abstention. Looks decisive in demos, hallucinates in production.
• Skipping evaluation against a golden set. Means you cannot tell if a change is a regression.
• Skipping metadata. Means the agent quotes last year's policy.
• Skipping citation UI. Means users can't trust the system, and stop using it within a quarter.

Build evaluations before you build retrieval

This is the rule we hold most strictly. Before you choose a vector database or write a single chunker, build a 200-question evaluation set with known correct answers. Run every change through it. If you cannot articulate "this change improved retrieval recall by 8% on the eval set," you are not improving anything — you are cargo-culting.

A note on long context

Yes, models now accept million-token contexts. No, this does not eliminate the need for retrieval. Putting your entire corpus in the prompt is expensive, slow, and — counterintuitively — often less accurate than retrieving 10 well-chosen chunks. The frontier labs themselves recommend retrieval over stuffing for production workloads.

Closing

RAG is not magic. It is a pipeline. Build the pipeline, evaluate it stage by stage, and your answer quality will be a knob you can turn — not a property of the model you chose.