[[Retrieval Augmented Generation|RAG]]: Trade-offs & Evaluation Strategy

---
created: 2026-02-05
tags:
  - main
---
---
# [[Retrieval Augmented Generation|RAG]]: Trade-offs & Evaluation Strategy

## 1. Pros & Cons of RAG

### ✅ Pros
* **Rapid Time to Market:** Easier to implement than fine-tuning a model from scratch.
* **Vast Expertise:** Allows the LLM to access massive external datasets without retraining.
* **Context Optimization:**
    * **Saves Context Length:** You don't need to stuff the entire knowledge base into the prompt.
    * **Focus:** Avoids distracting the model with irrelevant context (reduces noise).

### ❌ Cons
* **Trial & Error:** Requires significant tuning of chunk sizes, overlap, and retrieval parameters (k).
* **Reliability:** Can lead to unexpected "I don't know" responses if the retrieval step fails to find the right chunk, even if the answer exists in the database.

---

## 2. Evaluation Framework

To move beyond "trial and error," a formal evaluation pipeline is required.

### A. The Test Set (Ground Truth)
You cannot evaluate without a curated "Golden Dataset." This must include:
1.  **Example Questions:** Real queries users might ask.
2.  **Identified Context:** The specific document chunk IDs that *should* be retrieved.
3.  **Reference Answers:** The ideal human-written response.

### B. Measuring Retrieval (The "R")
How good is the system at finding the right document?

* **Metrics:**
    * **MRR (Mean Reciprocal Rank):** How high up the list is the first relevant result?
    * **nDCG (Normalized Discounted Cumulative Gain):** Measures ranking quality.
    * **Precision@K:** What percentage of the top K results are relevant?
    * **Recall@K:** (Crucial) Did we retrieve the relevant chunk *at all* within the top K results?

> [!TIP] The Priority Metric
> **Recall@K** is often the most important metric here. If the relevant document isn't in the context window (Recall failure), the LLM has zero chance of answering correctly.

* **Keyword Coverage:** analyzing if specific domain keywords from the query appear in the retrieved chunks.

### C. Measuring Generation (The "G")
How good is the system at synthesizing the answer?

* **LLM-as-a-Judge:** Use a stronger model (e.g., GPT-4 or Gemini 1.5 Pro) to score the RAG system's output against the Reference Answer.
* **Scoring Criteria:**
    * **Accuracy:** Is the information factually correct?
    * **Completeness:** Did it miss any key details?
    * **Relevance:** Did it actually answer the user's question?

---