Is RAG better than fine-tuning?

For factual knowledge, usually yes — RAG updates in minutes, cites sources and respects access control. Fine-tuning is better for changing behaviour, tone or output format. They solve different problems and are often combined.

Why does my RAG system still hallucinate?

Most often retrieval returned the wrong chunks, so the model had nothing correct to ground on. Check what was actually retrieved before blaming the model — a large share of apparent hallucinations are retrieval failures.

How big should RAG chunks be?

Large enough to be self-contained, small enough to be specific — commonly a few hundred to a thousand tokens. Splitting on document structure matters more than the exact size.

Do I need a vector database for RAG?

Not always. At small scale, in-memory search or a Postgres extension is enough. A dedicated vector database earns its place at larger scale or when you need filtering and hybrid search.

What Is RAG? Retrieval-Augmented Generation Explained

What is RAG (retrieval-augmented generation)?

Retrieval-augmented generation is the pattern of fetching relevant documents at query time and including them in the prompt, so the model answers from supplied sources rather than from training memory alone. It is the standard way to make a general model answer accurately about private, current or domain-specific information.

The pipeline

RAG has two phases, and most failures come from the first one being treated as an afterthought.

Indexing (once, ahead of time)

Split documents into chunks
Convert each chunk into an embedding
Store embeddings in a vector database alongside the original text

Retrieval (per query)

Embed the user's question
Find the nearest chunks
Insert them into the prompt
Generate an answer grounded in those chunks

Why RAG rather than fine-tuning

For factual knowledge, RAG beats fine-tuning on nearly every practical axis:

	RAG	Fine-tuning
Update a fact	Re-index one document	Retrain
Cite a source	Yes, you have the chunk	No
Cost to change	Minutes	Hours to days
Access control	Filter at retrieval	Baked into weights

Fine-tuning is the right tool for changing behaviour, format or style. RAG is the right tool for changing what the model knows. Choosing fine-tuning to teach facts is the most common expensive mistake in this area.

Where RAG breaks

Chunking destroys context. Splitting on a fixed character count cuts tables in half and separates a heading from its content. Chunk on structure — sections, paragraphs — not on length.

Retrieval finds the wrong thing. Vector search matches meaning, so it can miss exact identifiers like error codes and SKUs. Hybrid search — keyword plus vector — fixes most of this, and reranking fixes much of the rest.

Multi-hop questions fail. A question requiring two facts from two documents often retrieves one and confidently answers half. This is the failure mode that most looks like a hallucination but is really a retrieval problem.

The model ignores the context. Supplying documents does not force their use. Groundedness has to be measured, not assumed — and the fix is usually an instruction to answer only from context plus an evaluation that catches when it doesn't.

Making it measurably better

Improvements in rough order of value per unit of effort:

Reranking — a second pass reordering candidates with a stronger model
Hybrid search — keyword plus vector
Better chunking — split on meaning, keep headings with content
Query rewriting — expand or clarify before retrieving
An eval set — fixed questions with expected answers, run on every change

The last one is not an improvement so much as the prerequisite for knowing whether any of the others worked.

Compare vector databases, embedding models and RAG frameworks in the tools directory.

Explore AI tools

Frequently asked questions

Is RAG better than fine-tuning?: For factual knowledge, usually yes — RAG updates in minutes, cites sources and respects access control. Fine-tuning is better for changing behaviour, tone or output format. They solve different problems and are often combined.
Why does my RAG system still hallucinate?: Most often retrieval returned the wrong chunks, so the model had nothing correct to ground on. Check what was actually retrieved before blaming the model — a large share of apparent hallucinations are retrieval failures.
How big should RAG chunks be?: Large enough to be self-contained, small enough to be specific — commonly a few hundred to a thousand tokens. Splitting on document structure matters more than the exact size.
Do I need a vector database for RAG?: Not always. At small scale, in-memory search or a Postgres extension is enough. A dedicated vector database earns its place at larger scale or when you need filtering and hybrid search.

Embedding: A numeric vector representing the meaning of text, so similar ideas land near each other and can be searched by similarity rather than keywords.
Vector database: A database (Pinecone, Qdrant, pgvector) that stores embeddings and finds semantically similar content — the storage layer behind RAG.
Context window: How much text (measured in tokens) a model can consider at once. Bigger windows fit more documents but cost more per call.
Hallucination: When an AI model states something false with confidence. Mitigated with retrieval (RAG), citations, and verification steps in your workflow.
Reranking: Applying a stronger relevance model to reorder an initial set of retrieved candidates. Reranking improves quality while avoiding the cost of scoring the entire collection with the stronger model.