Retrieval-Augmented Generation: Giving a Model a Memory It Doesn’t Have to Memorize
Our Large Language Models post ended on a problem: a model trained only to predict the next token knows only what was baked into its weights during pretraining, frozen at whatever point that data was collected. Retrieval-Augmented Generation (RAG) is the architectural answer to that problem.
This post focuses on how RAG works as a model architecture. If you want the practical, tooling-focused side — vector databases, embeddings, and building a RAG pipeline end to end — see Vector Databases & RAG in our Machine Learning section. The two are complementary: this one explains the “why,” that one explains the “how to build it.”
Two Kinds of Memory
The original RAG paper framed it as combining two kinds of memory:
- Parametric memory — knowledge compressed into the model’s weights during training. Fast to use, but frozen and unauditable — you can’t point to which fact in the weights produced a given answer.
- Non-parametric memory — an external, searchable knowledge source (a document collection) that the model can query at inference time. Slower per-query, but always up to date and directly inspectable.
RAG systems use a small model to search the non-parametric memory, then hand what it finds to a large model to reason over and write the final answer.
The Architecture: Retriever + Generator
A RAG system is two neural networks working together:
1. The Retriever
Typically a bi-encoder (also called a dense retriever): one encoder turns your query into a vector, a second (often the same) encoder turns every candidate document into a vector ahead of time, and retrieval becomes a nearest-neighbor search in that vector space. This is where tokenization and a transformer encoder — much smaller than the generator — do the work of turning text into something searchable.
query_vector = encoder.encode("What causes vanishing gradients?")
# nearest neighbor search over a pre-computed document index
top_docs = vector_index.search(query_vector, k=5)
2. The Generator
The retrieved documents get inserted into the LLM’s context window alongside the original question, and the LLM generates its answer conditioned on both:
Context: [Document 1] [Document 2] ... [Document 5]
Question: What causes vanishing gradients?
Answer: <generated by the LLM, grounded in the retrieved context>
This is the simplest and most common pattern in production today — often called “retrieve-then-read.” The original RAG paper also described tighter integrations (RAG-Sequence and RAG-Token), where the retrieved documents influence the probability of each generated token directly rather than just being prepended as text — more powerful in principle, but far less common in practice than the simpler concatenation approach, because it’s harder to build and serve.
Why This Actually Reduces Hallucination
An LLM’s causal self-attention (see Large Language Models) treats retrieved context the same as any other tokens in its input — it attends over it just like the rest of the prompt. When the retrieved documents actually contain the answer, the model has a much easier time “copying and rephrasing” grounded information than generating a fact purely from its parametric memory, where errors compound with nothing to check them against. RAG doesn’t make hallucination impossible, but it gives the model something concrete to point back to — and lets you show users the source, which pure generation never can.
The Failure Modes Worth Knowing
RAG is not magic, and its failures are usually retrieval failures, not generation failures:
- Bad retrieval, confident generation. If the retriever pulls irrelevant documents, the generator will often still produce a fluent, confident-sounding answer built on the wrong context.
- Context window limits. Stuffing in too many retrieved documents competes for the same limited context window covered in the tokenization post — more isn’t always better.
- Stale or duplicate indexes. The generator is only as good as the freshness of the non-parametric memory behind it.
Where to Go From Here
If you want to actually build one of these — choosing an embedding model, picking a vector database, chunking documents sensibly — that’s exactly what Vector Databases & RAG walks through.