Vector Databases & RAG: Giving LLMs a Memory They Can Trust
Large Language Models are excellent at fluent text generation but they only “know” what was baked into their training data, and they will confidently generate plausible-sounding but false statements — a problem known as hallucination. Retrieval-Augmented Generation (RAG), built on top of vector databases, is the most widely used technique to fix this: instead of relying purely on the model’s memorized knowledge, you retrieve relevant facts from your own data at query time and hand them to the LLM as context.
Embeddings: Turning Meaning Into Numbers
An embedding is a numerical vector (typically a few hundred to a few thousand floating-point numbers) that represents the meaning of a piece of text, image, or other data. Embedding models are trained so that semantically similar inputs end up close together in this vector space, and dissimilar inputs end up far apart.
For example, the sentences “How do I reset my password?” and “I forgot my login credentials” would produce embedding vectors that are close together, even though they don’t share many words — because an embedding model captures meaning, not just keywords.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
vec1 = model.encode("How do I reset my password?")
vec2 = model.encode("I forgot my login credentials")
# cosine similarity between vec1 and vec2 will be high
Vector Search: Finding “Nearby” Meaning
Once text is converted to vectors, finding relevant information becomes a geometry problem: given a query vector, find the stored vectors closest to it, usually measured with cosine similarity or Euclidean distance. This is conceptually related to the clustering ideas from our Essentials of Machine Learning post — both rely on distance in a feature space — except here the goal is nearest-neighbor retrieval rather than grouping.
Doing this search naively (comparing a query against every stored vector) doesn’t scale past a few thousand items, so vector databases use approximate nearest neighbor (ANN) indexing algorithms — such as HNSW (Hierarchical Navigable Small World graphs) or IVF (Inverted File indexes) — to search millions or billions of vectors in milliseconds, trading a small amount of accuracy for a large amount of speed.
Popular Vector Databases
| Tool | Notes |
|---|---|
| Pinecone | Fully managed, cloud-native vector database; popular for production RAG systems that don’t want to run their own infrastructure. |
| Weaviate | Open-source, supports hybrid search (combining vector similarity with traditional keyword/BM25 search) and can be self-hosted or used as a managed cloud service. |
| Chroma | Lightweight, open-source, developer-friendly — often used for prototyping RAG applications locally before scaling up. |
| FAISS (Facebook AI Similarity Search) | A library (not a full database) for efficient similarity search; widely used as the underlying ANN engine inside other tools and for in-memory research/prototyping. |
| Milvus / Qdrant / pgvector | Other common choices — Milvus for large-scale open-source deployments, Qdrant for a Rust-based performance focus, and pgvector for teams that want vector search as an extension inside PostgreSQL rather than a separate system. |
How a RAG Pipeline Works
A typical RAG system has two phases:
Indexing (done ahead of time):
- Split your source documents into chunks (e.g., 300–500 word passages).
- Convert each chunk into an embedding vector.
- Store the vector, plus the original text and metadata, in a vector database.
Querying (done at request time):
- Convert the user’s question into an embedding using the same embedding model.
- Search the vector database for the top-k most similar chunks.
- Insert those retrieved chunks into the LLM’s prompt as context.
- Ask the LLM to answer the question using only the provided context.
# Simplified RAG query using Chroma + an embedding model + an LLM call
query = "What is our refund policy for annual subscriptions?"
query_vector = embed_model.encode(query)
results = collection.query(
query_embeddings=[query_vector],
n_results=3
)
context = "\n\n".join(results["documents"][0])
prompt = f"""Answer the question using only the context below.
If the answer isn't in the context, say you don't know.
Context:
{context}
Question: {query}
"""
answer = llm.generate(prompt)
Why This Reduces Hallucination
The LLM is no longer answering purely from its training-time memory — it is answering from text you handed it moments ago, retrieved specifically because it’s relevant to the question. This has two big benefits:
- Grounding: the answer can cite or closely track real source material, rather than being invented.
- Freshness: you can update the vector database with new documents at any time without retraining the LLM itself — the model’s “knowledge” of your data is only as stale as your last indexing run.
RAG doesn’t eliminate hallucination entirely (the LLM can still misread or ignore the retrieved context), but it dramatically narrows the gap between “the model’s best guess” and “what our own documents actually say.”
Where This Fits in a Real ML Workflow
RAG sits downstream of the same data understanding work described in our Exploratory Data Analysis post — you still need to know your data, clean it, and decide how to chunk it sensibly. The pipeline typically looks like:
- Collect and clean source documents (support docs, wikis, transcripts, PDFs).
- Chunk and embed them into a vector database.
- Build a retrieval step that turns a user query into relevant chunks.
- Feed retrieved chunks + query into an LLM prompt.
- Evaluate answer quality — precision of retrieval (did we fetch the right chunks?) and faithfulness of generation (did the LLM stick to the provided context?).
- Re-index continuously as source documents change.
RAG is now the default architecture for building chatbots, internal knowledge assistants, and customer support tools on top of proprietary data — it lets you combine the fluency of a general-purpose LLM with the accuracy of your own, constantly updated knowledge base.