How to Build Your Retrieval-Augmented Generation (RAG)

01 January 2025 - 6 mins read time
Tags: RAG LLM NLP

This note is adapted from teaching experience in 1DT104 Project in Computer Systems at Uppsala University. Instructor: course teaching staff.

Central Question

Retrieval-Augmented Generation asks: how can a language model answer with evidence from an external, updateable knowledge source instead of relying only on its parameters?

The basic architecture is simple:

\[\text{RAG}=\text{retrieve relevant context}+\text{generate an answer from that context}.\]

The engineering challenge is not the slogan. It is making ingestion, retrieval, prompting, generation, evaluation, and monitoring work together reliably.

RAG system boundary showing offline ingestion, online retrieval, prompt assembly, generation, citation, and evaluation.
RAG is a system architecture: offline indexing and online answering must share metadata, chunking assumptions, and evaluation feedback.

1. Why Use RAG?

RAG is useful when knowledge is private, fast-changing, too large for a prompt, or too specific to expect from the base model. It can reduce hallucination risk by grounding responses in retrieved evidence, but it does not eliminate hallucination by itself.

RAG is usually preferable to fine-tuning when the main need is factual access to changing documents. Fine-tuning changes model behavior; retrieval changes the evidence available at inference time. Many production systems use both, but they solve different problems.

The most common failure pattern is retrieval mismatch: the answer is poor because the right evidence never reached the model. That means RAG quality starts before generation.

2. Offline Ingestion: Documents to Searchable Units

The offline stage converts raw data into indexed evidence:

  1. collect documents,
  2. parse text and structure,
  3. attach metadata,
  4. chunk into retrievable units,
  5. embed or index chunks,
  6. store vectors, sparse terms, and provenance.

Chunking is a modeling choice. A chunk should be small enough to retrieve precisely but large enough to preserve meaning. A document can be partitioned as

\[D=\bigcup_{i=1}^{m}C_i, \qquad |C_i|\le L.\]

Useful metadata includes source, section title, timestamp, access policy, document version, page number, and parent-child relationships. Without metadata, retrieval cannot filter, cite, deduplicate, or respect permissions.

Ingestion pipeline from raw documents to parsed text, chunks, embeddings, metadata, and index.
Good RAG starts at ingestion: chunk text with provenance, metadata, and update strategy before building the index.

3. Retrieval: Dense, Sparse, Hybrid, and Reranked

Dense retrieval maps queries and chunks into vectors:

\[\operatorname{sim}(q,c)=\frac{f(q)\cdot f(c)}{\|f(q)\|\|f(c)\|}.\]

Sparse retrieval uses keyword or term-frequency signals such as BM25. Dense search handles semantic similarity; sparse search often handles exact names, codes, abbreviations, and rare terms better.

Hybrid retrieval combines both. A common fusion idea is Reciprocal Rank Fusion:

\[\operatorname{score}(d)=\sum_i\frac{1}{k+\operatorname{rank}_i(d)}.\]

Reranking then scores the top candidates with a stronger but more expensive model. The retrieval stack is therefore usually staged:

Retrieval pipeline combining dense search, sparse search, metadata filtering, fusion, reranking, and selected context.
A robust retriever separates recall from precision: broad first-stage search is refined by filtering, fusion, and reranking.

4. Prompt Assembly and Generation

The prompt should tell the model how to use retrieved evidence. A minimal template is:

Answer the question using the provided context.
If the context is insufficient, say what is missing.
Quote or cite the relevant source ids when possible.

Question:
{user_question}

Context:
{retrieved_chunks}

The model should not be asked to blindly trust context. Retrieved chunks may be outdated, duplicated, irrelevant, or mutually inconsistent. Strong prompts make uncertainty and source conflict explicit.

Context packing is also an optimization problem. Given a context budget, choose passages that maximize coverage and diversity while avoiding repetition. Parent-document expansion, sentence-window retrieval, and section-level summaries can help when the first retrieved chunk is too narrow.

5. Advanced Patterns

Advanced RAG techniques mostly repair one of four weaknesses: query mismatch, corpus mismatch, context overload, or multi-step reasoning.

Useful patterns include:

Advanced RAG patterns including query rewrite, decomposition, routing, graph retrieval, reranking, and response synthesis.
Advanced RAG is usually targeted repair: rewrite queries, route to better stores, expand context, or decompose reasoning only when the baseline pipeline fails.

The caution is latency and complexity. Every extra model call can improve quality or simply add cost, variance, and failure modes. Baseline before adding orchestration.

6. Evaluation and Operations

RAG should be evaluated as a pipeline, not only by final answer preference.

Retrieval metrics include:

Generation metrics include:

Operational checks include latency, cost, index freshness, access control, prompt injection resistance, logging, and drift monitoring.

RAG evaluation loop connecting retrieval metrics, answer faithfulness, latency, cost, feedback, and index updates.
RAG quality improves through measurement loops: evaluate retrieval, generation, citations, latency, and index freshness separately.

What This Framework Lets Us Do

RAG lets teams build domain-specific assistants, search interfaces, support bots, documentation copilots, and analysis tools without putting all knowledge into model weights. Its strength is modularity: documents can be updated, retrievers swapped, and evaluation targeted to failure points.

Where the Framework Stops Being Reliable

RAG fails when the corpus lacks the answer, permissions are wrong, retrieval misses evidence, chunks remove necessary context, sources conflict, or the model ignores evidence. It is not a truth machine; it is an evidence-routing architecture.

Where the Subject Leads Next

The next steps are information retrieval, evaluation science, knowledge graphs, agentic tool use, secure document processing, vector indexing, and human-in-the-loop feedback systems.

Technical and Editorial Audit

Main Sources Used in This Note