RAG Implementation Patterns

Naive RAG is four lines of code: embed the query, top-k nearest neighbours, stuff them in the prompt, generate. It usually scores around 60–75% on a task-specific eval — good enough for a demo, nowhere near good enough for production.

Getting from 75 to 95 is the whole discipline. This page is that delta, in concrete steps, ranked by how much return each move produces.

@startuml
skinparam BoxPadding 10

|User / Client Interface|
start
:Submit Domain Query;

|Query Pipeline [[/wiki/AiPoweredSearch]]|
:Analyze Intent & Preprocess;
if (Requires Query Transformation?) then (yes)
  :Generate HyDE or Decompose into Sub-Queries;
else (no)
  :Normalize Query String;
endif

|Hybrid Retrieval Engine [[/wiki/HybridRetrieval]]|
fork
  :Generate Dense Embedding Vector;
  :Execute HNSW Approximate Nearest Neighbors [[/wiki/VectorDatabases]];
  :Surface Top-K Dense Semantic Candidates;
fork again
  :Tokenize & Stem Query Terms;
  :Execute BM25 Inverted Index Keyword Scan;
  :Surface Top-K Sparse Lexical Candidates;
fork again
  :Evaluate Metadata & Security Constraints;
  :Construct Tenant & Permission Filter Bitmask;
end fork

:Apply Permission Bitmask Filtering;
:Execute Reciprocal Rank Fusion (RRF);
note right
  Combines dense & sparse ranks:
  score = sum(1 / (k + rank_i))
end note

|Precision Scoring & Reranking|
:Pass Top-N Candidates to Cross-Encoder Model;
:Compute Deep Query-Document Attention Scores;
:Sort Candidates by Relevance Probability;
:Apply Context Compression & Token Budgeting;

|Synthesis & Verification [[/wiki/AgentPlanning]]|
:Inject Grounded Context Chunks into Prompt Template;
:Execute LLM Inference for Response Generation;

if (Grounding Verification Passed?) then (yes)
  :Bind Inline Citations to Chunk Provenance;
  |User / Client Interface|
  :Deliver Verified Response with Citations;
  stop
else (no)
  |Synthesis & Verification [[/wiki/AgentPlanning]]|
  :Flag Low Grounding / Uncertainty;
  :Trigger Query Refinement or Fallback;
  |User / Client Interface|
  :Deliver Guarded Response with Warning;
  stop
endif
@enduml

The biggest single lever: hybrid retrieval

If your RAG is dense-only (pure vector search), adding BM25 keyword search and fusing with reciprocal rank fusion typically adds 5–15 points of recall at near-zero latency cost. Dense embeddings miss exact-string queries ("error 451", "SKU ABC-1234"); BM25 misses on semantic paraphrase. The fusion gets both.

Minimum RRF:

def rrf(dense_hits, bm25_hits, k=60):
    scores = {}
    for rank, d in enumerate(dense_hits):
        scores[d.id] = scores.get(d.id, 0) + 1 / (k + rank + 1)
    for rank, d in enumerate(bm25_hits):
        scores[d.id] = scores.get(d.id, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: -x[1])

See HybridRetrieval for the full ranking pipeline this wiki uses.

Do this first. Nothing else in this list beats the effort-to-payoff ratio of adding BM25.

The second biggest: chunking strategy

"How do I chunk my documents" is the second question every RAG system gets wrong. The trap is either (a) fixed-size 500-token chunks that cut sentences in half, or (b) a section-header splitter that produces one 8000-token chunk and ten 50-token ones.

Working strategy:

Split on semantic boundaries first — headings, paragraph breaks, list items. Never mid-sentence.
Size target: 256–512 tokens per chunk. Smaller hurts context; larger dilutes embeddings.
Overlap adjacent chunks by 10–20%. One-sentence overlap usually suffices; prevents information loss at boundaries.
Attach context metadata. Every chunk carries {document_id, section_path, preceding_heading}. Your retriever uses the heading path for filtering and the context in the prompt.

For structured content (code, tables, JSON) stop before applying text chunking. Treat code blocks and tables as atomic units. A chunk that contains half a table is actively harmful — the model will hallucinate the missing rows.

Rerank: the pattern most people skip

A cross-encoder reranker over the top-20 candidates typically adds another 3–8 points of nDCG. Cohere Rerank, BGE-reranker-large, or a locally hosted ms-marco-MiniLM-L-12-v2 all work.

Shape:

query → retrieve top 50 (hybrid) → cross-encode rerank → top 5-10 to prompt

Cross-encoders are slow compared to ANN (50ms vs 2ms for 50 candidates on GPU) but the recall gain outweighs it for anything with a human in the loop. Skip reranking only for real-time < 100ms budgets where you'd rather take the recall hit.

Query transformation

User queries are often bad retrieval queries. Three moves that help:

HyDE (Hypothetical Document Embeddings) — LLM writes a hypothetical answer, you embed that and retrieve. Works well when the user's query is short or ambiguous; works badly when the LLM hallucinates technical details.
Query decomposition — split "compare X and Y" into "tell me about X" + "tell me about Y" + final synthesis. Essential for multi-entity queries.
Multi-query expansion — generate 3–5 paraphrases, retrieve for each, union the results. Adds recall at the cost of one small LLM call.

Use one of these, not all three. Stacking transformations adds latency faster than quality.

Metadata filtering done right

Most RAG failures in production aren't retrieval failures, they're context failures — the right information is retrieved but it's for the wrong tenant, wrong time period, or wrong document version. Metadata filtering is the defence.

Pre-filter (narrow candidates by metadata, then ANN) when filters are selective. "Documents for tenant X" cuts 99% of candidates.
Post-filter (ANN first, then filter) when filters are broad. "Documents from last year" might match most of the corpus; pre-filter costs more than post.
Your vector DB should support both modes and let you pick per-query. Qdrant does this cleanly; pgvector is weaker but good enough for most workloads.

The prompt matters more than people admit

Once you have the right chunks, the prompt determines whether the model uses them. Three rules:

Label the chunks explicitly. [Source: doc-423, section "Returns Policy"] beats dumping raw text. The model cites correctly only if you tell it how.
Instruct citation. "Answer using only the provided sources. Cite each claim with the source label." Without this, the model will cheerfully answer from pretraining and give you a confidently wrong answer.
Handle empty retrievals. "If the sources do not contain the answer, reply 'I don't know.'" This one line eliminates a specific, high-confidence hallucination class.

Evaluation: the thing most teams skip

You cannot improve what you don't measure. Minimum RAG eval:

Metric	What it catches	Cost
Retrieval recall@k	Did the right chunk surface?	Free once you have labelled query→doc pairs
nDCG@10	Is the ranking sensible?	Same labels as recall
Answer faithfulness (RAGAS)	Does the answer stick to the retrieved context?	One LLM call per eval row
Answer correctness	Is the answer actually right?	Human label or LLM-as-judge
Latency p95	Does the pipeline fit your budget?	Free

Build a frozen eval set of 100–500 queries with gold-labelled relevant chunks and expected answers. Run it on every change. This wiki's own retrieval is evaluated this way — see HybridRetrieval and RetrievalExperimentHarness.

RAG vs long-context models

"Just put the whole corpus in a 1M-token context" sounds like it replaces RAG. It doesn't, for three reasons:

Cost. 1M-token calls are expensive. RAG with 4k of retrieved context costs 0.4% of the same query against a full-corpus long-context call.
Attention degrades with length. Models retrieve from long contexts noticeably worse than from short, focused ones (the "lost in the middle" effect). Recall drops from 98% at 4k to 60–75% at 1M depending on the model.
You can't update. A long-context call re-ingests the whole corpus every time. RAG's retrieval index lets you update incrementally.

Long context is the right answer for some tasks (e.g. summarising a single 500-page contract). RAG is the right answer for retrieval over a knowledge base. They compose — use RAG to narrow to the relevant subset, then long-context for the deep analysis.

Architectural Latency Budgets and Production Trade-Offs

Deploying the complete multi-stage RAG pipeline requires balancing accuracy against inference latency. In interactive search or agentic workflows, end-to-end latency should typically remain under 1,500ms.

Pipeline Stage	Subsystem	Typical Latency Budget	Fallback Strategy on Timeout
Query Transformation	Small Speculative LLM (HyDE / Multi-Query)	150–250ms	Bypass HyDE; use raw normalized query string
Sparse Lexical Search	BM25 / Inverted Lucene Index	10–25ms	Proceed with dense-only candidates if Lucene unresponsive
Dense Semantic Search	HNSW Index on Vector Store	20–60ms	Proceed with sparse-only candidates if vector index lags
Rank Fusion (RRF)	In-Memory Score Combiner	2–5ms	None (deterministic CPU operation)
Cross-Encoder Rerank	BGE-Reranker / Cohere Rerank API	80–180ms	Skip reranking; take top- $K$ directly from RRF scoring
Context Compression	Token Slicer / Selective Context Filter	15–30ms	Truncate context to hard token ceiling
LLM Generation	Frontier Model Streaming Inference	600–1,200ms	Stream tokens immediately with progressive citation markers

By decoupling each stage with explicit fallback thresholds, the system degrades gracefully under heavy load without degrading user-facing uptime.

Production debugging

When RAG quality regresses in prod, look in this order:

Check retrieval first. 90% of RAG regressions are retrieval regressions. Log top-k chunk IDs per query; diff against a known-good baseline.
Check the index. HNSW tombstone ratio high? Recall drops silently.
Check embedding model version. Silent model swaps on managed APIs change the embedding space — vectors indexed against v2 are noise for v3. Pin the model version in deployment config.
Check chunking. If you reindexed and recall dropped, the new chunking is probably worse than the old. A/B test chunking strategies — don't assume the new one is better.

Instrument each layer with a trace span so a single query shows retrieval, rerank, and generation cost/latency. Without this you'll spend hours reproducing intermittent issues by hand.