Large Language Models (LLMs) are powerful reasoning engines, but they have a fundamental limitation: they can only draw on the parametric knowledge acquired during training. This makes them prone to hallucinations, blind to private or proprietary data, and limited by their static training data.

Retrieval-Augmented Generation (RAG) is the architectural pattern designed to solve this.

At its core, RAG is not a specific model or framework, it is a pipeline. RAG connects an LLM's reasoning to the external, dynamic data sources at query time. Instead of forcing the model to guess, RAG retrieves relevant facts first and generates a response based on those facts.

Here is a breakdown of how the pipeline works, where basic implementations fail, and how to upgrade it for production.

How the RAG Pipeline Actually Works

A standard RAG system operates in two main phases: Indexing and Runtime.

rag-general-pipeline.png
RAG General Pipeline

Indexing (Preparation)

Before a user queries the system, data must be prepared:

  1. Extraction: Text is extracted from source documents (PDFs, databases, internal wikis).

  2. Chunking: The text is split into smaller segments. The industry often converges on 500-1000 tokens per chunk, this is a deliberate trade-off. Chunks that are too small lose surrounding context, while chunks that are too large mix multiple topics into one vector, making the embedding less specific to any single idea.

  3. Embedding: Each chunk is passed through an embedding model * (e.g., text-embedding-3-small or bge-large) to convert it into a high-dimensional vector.

  4. Storage: These vectors, along with the original text and any metadata, are stored in a vector database (e.g., Qdrant, Chroma, or pgvector).

* The machine learning model that acts as a translator, converting raw text into a mathematical representation.

Runtime (Query Time)

When a user submits a query, the following steps occur:

  1. Embed the Query: The user's question is converted into a vector using the same embedding model.

  2. Retrieve: The database performs a similarity search * to find the top K most relevant document chunks.

  3. Augment: The retrieved chunks are injected into a prompt template alongside the user's original query.

  4. Generate: The LLM processes the augmented prompt and generates a response using only the retrieved context.

* The mathematical operation (often Cosine Similarity) used by the Vector DB to find the vectors that are "closest" to the user's query vector. It measures semantic relatedness, not keyword overlap.

Why Naive RAG Fails in Production

A baseline RAG implementation (often called Naive RAG) typically consists of a simple retrieve-then-read loop. While it works for simple demos, it frequently breaks in production due to several issues:

  • Semantic Mismatch: Vector search excels at conceptual similarity but struggles with exact keyword matches. If a user searches for "Error 503", but the document says "Service Unavailable", pure vector search might rank it lower than a conceptually similar but factually wrong document.

  • The "Lost in the Middle" Phenomenon: Research suggests LLMs often exhibit a U-shaped attention pattern. They focus heavily on the beginning and end of a prompt, frequently ignoring critical information buried in the middle of a large retrieved context.

  • Unfiltered Search Space: Searching across all documents at once increases noise. Production systems rarely rely on raw vector search alone. They use metadata filtering (e.g., filtering by document date, department, or access level) before the vector search to shrink the search space.

Two-Stage Retrieval Pipeline

Building a robust RAG system does not require an overcomplicated architecture. The architectural change that yields the most significant improvement for the least added complexity is moving from a single vector search to a two-stage retrieval process.

rag-two-stage-retrieval.png
Two-Stage Retrieval

Instead of relying solely on dense vector embeddings, systems can combine them with sparse search (like BM25 *). Sparse search excels at exact keyword matching, while dense search handles semantic meaning. Modern vector databases support hybrid search natively, combining both approaches in a single query.

* Traditional keyword-based search (like Elasticsearch) that scores exact word matches rather than semantic meaning.

Stage 2: Cross-Encoder Re-ranking

Vector search is fast, but it is an approximation. It compares two independent vectors, which can lead to noisy top-K results.

A Cross-Encoder Re-ranker solves this. Instead of comparing independent vectors, a re-ranking model (like bge-reranker or ms-marco-MiniLM) reads the query and the document together as a single input sequence. It outputs a highly accurate relevance score.

A fast vector search retrieves a broad pool of candidates (e.g., top 20), and the re-ranker to precisely sort them, passing only the true top 3-5 to the LLM. This improves accuracy while keeping token costs low.

Here is what that looks like in a self-contained Python example:

from sentence_transformers import CrossEncoder

# 1. Mock data representing the broad pool of candidates retrieved 
# from the vector database (e.g., via hybrid search)
candidates = [
    "To reset your password, go to settings and click 'Credential Recovery'.",
    "Our servers experienced a Service Unavailable (Error 503) outage yesterday.",
    "The company was founded in 2010 and is headquartered in Seattle.",
    "For password resets, navigate to the account settings page."
]

query = "How do I reset my password?"

# 2. Initialize a lightweight cross-encoder re-ranker model
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

# 3. Format pairs of [query, document_text] for the model
pairs = [[query, doc] for doc in candidates]

# 4. Get relevance scores for each pair
scores = reranker.predict(pairs)

# 5. Sort by score in descending order and keep the top 2
scored_docs = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
top_context = [doc for doc, score in scored_docs[:2]]

print("Top Retrieved Context:")
for doc in top_context:
    print(f"- {doc}")
🧪 Interactive Sandbox: Test candidate lists, inspect score distributions, or run the complete pipeline in Google Colab. (Select File → Save a copy in Drive to unlock an editable session).

Benchmarking the Pipeline

In traditional software, we write unit tests. In RAG, manual testing isn't enough. Pipelines require programmatic evaluation.

Frameworks like RAGAS or TruLens allow you to measure three core metrics:

  • Context Relevance: Did the retriever pull back useful information, or just noise?

  • Faithfulness: Did the LLM's answer strictly derive from the retrieved context? (For example, a low faithfulness score flags when an LLM confidently states "The server runs on AWS" even though the retrieved context only mentioned generic "cloud infrastructure".)

  • Answer Relevance: Does the final output actually address the user's original prompt?

rag-triad.png
Image taken from trulens.org

Establishing a small evaluation dataset early on is a good way to confidently iterate on different chunking strategies, embedding models, and re-rankers in order to catch hallucinations and retriever failures.

Conclusion

RAG is a powerful pattern, but its effectiveness is entirely dependent on the quality of the retrieval pipeline. Naive vector search is a great starting point, but production systems require hybrid retrieval, metadata filtering, and re-ranking to handle the ambiguity and complexity of real-world queries.

Better RAG performance comes from focusing less on the final LLM and more on the quality, structure, and ranking of the data fed into it.