Jul 20269 min readRAGretrieval

RAG that survives production

Chunking, hybrid retrieval, reranking and honest evals: the parts of a retrieval pipeline that decide whether it works outside a notebook demo.

The demo is easy, the maintenance is not

A retrieval pipeline that answers questions about ten PDFs takes an afternoon. The same pipeline over a corpus that changes weekly, with users who phrase things badly, is an engineering project. The difference is almost never the model.

Chunking is a product decision

Fixed-size chunks are the default and the worst option. What you actually want is chunks that match the unit a human would cite: a section, a function, a policy clause, a support answer. Keep the heading path with every chunk so the model can see where it came from, and keep a pointer back to the source so the UI can show it.

If a chunk makes no sense when read alone, the model cannot use it, and neither can a human reviewing a bad answer.

Hybrid beats pure vectors

Embeddings are good at meaning and bad at exact strings: ticket IDs, error codes, product names, numbers. Run keyword search alongside vector search and merge the results, then rerank the top candidates with a cross-encoder.

The order I reach for:

Keyword plus vector retrieval, each with its own candidate list.
Fusion to one ranked list.
Reranking down to the handful of chunks that go into the prompt.

Evals, or you are guessing

Build a question set with known good sources: 60 to 200 real questions is enough to start. Then measure retrieval separately from generation.

Retrieval: did the right chunk make the candidate list? At what rank?
Generation: given the right chunks, was the answer correct, complete, and grounded?

If recall is the problem, no prompt will fix it. If grounding is the problem, no reranker will fix it either. Splitting the metrics is what makes the failures actionable.

The boring operational parts

Answer only from retrieved context, and return "I don't know" when nothing relevant comes back. Cache embeddings. Version the index so you can roll back the day a bad reindex ships. Watch latency per stage, because reranking is where the milliseconds go.