The anatomy of a RAG pipeline

Retrieval-augmented generation (RAG) is now a standard pattern for building LLM applications on private knowledge. But “RAG” covers a wide range of implementation decisions, each with significant impact on quality, cost, and latency. This post breaks down every component in a production RAG pipeline.

The pipeline overview

A complete RAG system has two phases:

Offline (ingestion):

1. Document loading
2. Chunking
3. Embedding
4. Index storage

Online (query):

1. Query embedding
2. Retrieval (vector search, BM25, or hybrid)
3. Re-ranking (optional)
4. Context assembly
5. LLM generation

Stage 1: Document loading

Before embedding, documents must be loaded and cleaned. Common sources:

• PDF files (extract text, handle tables and layouts)
• Web pages (strip HTML, handle pagination)
• Databases (SQL queries returning text fields)
• APIs (Notion, Confluence, Slack, GitHub)

The quality of your ingestion pipeline directly affects retrieval quality. A PDF parser that garbles table data will produce useless embeddings from that table. Invest in a good document parser.

# Example: PDF loading with PyMuPDF
import fitz  # pymupdf

def load_pdf(path: str) -> str:
    doc = fitz.open(path)
    pages = [page.get_text() for page in doc]
    return "\n\n".join(pages)

Stage 2: Chunking

Chunking splits documents into smaller pieces for embedding. The optimal chunk size depends on:

• Your embedding model’s context window (most handle 256-512 tokens well)
• Your LLM’s context window (how much retrieved text can you inject?)
• The nature of your corpus (dense technical text vs. conversational text)

Strategies:

Fixed-size with overlap — Simple, predictable, good default:

def chunk_fixed(text: str, size: int = 400, overlap: int = 80) -> list[str]:
    words = text.split()
    return [
        " ".join(words[i : i + size])
        for i in range(0, len(words), size - overlap)
    ]

Sentence-level — Chunks align with sentence boundaries, preserving meaning better than mid-sentence cuts.

Semantic chunking — Group sentences by embedding similarity. More expensive but produces coherent chunks. Libraries like chonkie implement this.

Recursive character splitting — Split on paragraphs first, then sentences, then words. Popular in LangChain’s RecursiveCharacterTextSplitter.

Practical advice: Start with fixed-size chunks at 400-600 tokens with 10-20% overlap. Tune chunk size using your retrieval eval set.

Stage 3: Embedding

Each chunk is converted to a dense vector. Key decisions:

Model selection. Balance quality vs. cost vs. latency:

Model	Dimensions	Cost	Quality
all-MiniLM-L6-v2	384	Free (self-hosted)	Good
text-embedding-3-small	1536	$0.02/1M tokens	Very good
text-embedding-3-large	3072	$0.13/1M tokens	Best

Batching. Embedding APIs are slow for single-item calls. Always batch:

def embed_batch(texts: list[str], model="text-embedding-3-small") -> list[list[float]]:
    response = client.embeddings.create(input=texts, model=model)
    return [item.embedding for item in response.data]

# Embed in batches of 100
batches = [chunks[i:i+100] for i in range(0, len(chunks), 100)]
embeddings = [embed for batch in batches for embed in embed_batch(batch)]

Consistency. Use the same model for indexing and querying. Mixing models corrupts your index.

Stage 4: Index storage

Embeddings are stored in a vector index for fast retrieval. The index structure determines query latency and recall:

HNSW (Hierarchical Navigable Small World) — the dominant algorithm for approximate nearest-neighbour search. Excellent recall@10 (90%+) at sub-10ms latency. ManyVector stores HNSW graphs in your object store.

Flat index — exhaustive search, perfect recall, slow at scale. Only practical for small corpora (<100K vectors).

IVF (Inverted File Index) — clusters vectors into partitions. Faster than flat, lower recall than HNSW. Less common in modern production systems.

For most production RAG, HNSW is the right choice.

Stage 5: Query embedding and retrieval

At query time, embed the user’s question and retrieve top-k candidates:

def retrieve(query: str, top_k: int = 10) -> list[dict]:
    # Embed query with the same model used for indexing
    q_embedding = embed_batch([query])[0]

    # Hybrid retrieval for best recall
    results = ns.query(
        vector=q_embedding,
        query_text=query,       # BM25 signal
        search_mode="hybrid",
        alpha=0.7,              # 70% vector, 30% BM25
        top_k=top_k,
    )
    return [{"text": r.attributes["text"], "id": r.id, "score": r.score} for r in results]

How many to retrieve? Top-k is a balance: too few misses relevant context; too many dilutes the LLM context with irrelevant text. Common defaults: top_k=5 for GPT-3.5, top_k=10-15 for GPT-4 or Claude (larger context windows).

Stage 6: Re-ranking (optional)

First-stage retrieval (ANN search) maximizes recall. Re-ranking maximizes precision by scoring all retrieved candidates with a more expensive model:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve_and_rerank(query: str, top_k: int = 5, oversample: int = 4):
    # Retrieve with high recall
    candidates = retrieve(query, top_k=top_k * oversample)

    # Re-rank with cross-encoder
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs)

    ranked = sorted(zip(scores, candidates), reverse=True)
    return [c for _, c in ranked[:top_k]]

Re-ranking adds 50-200ms latency but often improves NDCG@5 by 5-15%. Worth it for user-facing search; consider skipping for high-volume background pipelines.

Stage 7: Context assembly

Assemble retrieved chunks into a prompt context:

def build_context(chunks: list[dict], max_tokens: int = 2000) -> str:
    context_parts = []
    total = 0

    for chunk in chunks:
        tokens = len(chunk["text"].split()) * 1.3  # rough token estimate
        if total + tokens > max_tokens:
            break
        context_parts.append(f"[Source: {chunk['id']}]\n{chunk['text']}")
        total += tokens

    return "\n\n---\n\n".join(context_parts)

Context window management. LLMs have fixed context windows. Reserve tokens for: system prompt (~200), user query (~100), and generated response (~500). The rest can go to context. For GPT-4o (128K context), you have substantial headroom.

Stage 8: LLM generation

Generate the final answer conditioned on the retrieved context:

def generate(question: str, context: str, model="gpt-4o") -> str:
    return openai_client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful assistant. Answer based only on the provided context. "
                    "If the context is insufficient, say so explicitly. "
                    "Do not use prior knowledge beyond what is in the context.\n\n"
                    f"Context:\n{context}"
                ),
            },
            {"role": "user", "content": question},
        ],
        temperature=0,  # deterministic for factual retrieval
    ).choices[0].message.content

End-to-end pipeline

def rag(question: str) -> str:
    chunks = retrieve_and_rerank(question, top_k=5)
    context = build_context(chunks)
    return generate(question, context)

Where RAG breaks down

Poor chunking. Cutting a sentence mid-way loses context. An answer that spans two chunks may be irretrieved.

Wrong embedding model. A model trained on Wikipedia may perform poorly on legal or medical text without domain-specific fine-tuning.

Low retrieval recall. If the right chunks aren’t retrieved, the LLM cannot generate a correct answer regardless of its own capability. Measure recall@k on an eval set before obsessing over prompt engineering.

Context poisoning. Retrieved chunks that are partially relevant may mislead the LLM. Better retrieval quality reduces this. Adding a “cite your sources” instruction helps the LLM clarify what it’s drawing from.

Freshness. If your corpus updates frequently, your index must be kept in sync. ManyVector upserts are idempotent — upsert with the same ID to update a chunk in place.

Measuring RAG quality

The right eval metrics depend on what you’re measuring:

• Retrieval recall@k — are the relevant chunks in the top-k? Measure this first.
• Context precision — what fraction of retrieved chunks are actually relevant?
• Answer faithfulness — does the LLM answer stay within the provided context? (use LLM-as-judge)
• Answer relevance — does the answer actually address the question?

Libraries like RAGAS provide automated evaluation of these metrics using LLM-as-judge.

Improving retrieval quality (recall@k, context precision) is usually higher leverage than prompt engineering. Start there.