Loading...
Loading...
Loading...
root((Day 20: Evaluation & Benchmarks š))
# Day 20: Evaluation & Benchmarks š
### Week 3 ā RAG, Fine-Tuning & Vector Databases
---
## š§ Concept Map
```mermaid
mindmap
root((Day 20: Evaluation & Benchmarks š))
š Section 1: Why Evaluation Is Hard for GenAI
1.1 The Evaluation Problem
1.2 Evaluation Taxonomy
š Section 2: Text Similarity Metrics
2.1 BLEU ā Bilingual Evaluation Understudy
2.2 ROUGE ā Recall-Oriented Understudy
2.3 BERTScore ā Semantic Similarity
2.4 Choosing the Right Metric
š Section 3: LLM-as-Judge
3.1 Why LLM Judges Work
š Section 4: RAGAS for RAG Evaluation
š» Full Lab: Evaluation Pipeline
š Section 5: LLM Benchmarks Overview
š§ Quiz: Day 20
š Key Takeaways
Section 5: LLM-as-Judge Evaluation
5.1 GPT-4 as a Judge
5.2 Pairwise Comparison A/B Testing
Section 6: GovEval and RAGAS
6.1 RAGAS Metrics Deep Dive
6.2 Building a Custom Evaluation Framework
Section 7: Cost-Aware Evaluation
7.1 Benchmarking with Cost Tracking
Extended Lab: Build an Automated Eval Pipeline
Extended Academic Appendix: Generative AI Complete Glossary
Extended Academic Appendix: Generative AI Complete Glossary
Extended Academic Appendix: Generative AI Complete Glossary
```
---
## šÆ Learning Objectives
By the end of today, you will:
- Compute BLEU, ROUGE, and BERTScore metrics
- Use LLM-as-Judge evaluation patterns
- Evaluate RAG systems with RAGAS
- Build a complete evaluation pipeline
- Understand major LLM benchmarks (MMLU, HellaSwag, MT-Bench)
**Estimated Time:** 3.5ā4 hours
**Difficulty:** āāā Intermediate
**Prerequisites:** Days 16ā19
---
## š Section 1: Why Evaluation Is Hard for GenAI
### 1.1 The Evaluation Problem
Traditional ML: accuracy is straightforward.
GenAI: "How good is a summary?" is inherently subjective.
```
Question: "Summarize the history of the Roman Empire."
Response A: "Rome dominated the ancient world for ~1000 years,
characterized by its military, law, and architecture."
Response B: "The Romans existed a long time ago and were important."
Both are "correct" ā but A is vastly better.
How do we measure this automatically?
```
### 1.2 Evaluation Taxonomy
```
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
ā GENAI EVALUATION METHODS ā
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤
ā REFERENCE-BASED ā REFERENCE-FREE ā
ā (compare to gold answer) ā (judge on own merit) ā
ā ā ā
ā ā¢ BLEU (precision) ā ā¢ LLM-as-Judge ā
ā ā¢ ROUGE (recall) ā ā¢ G-Eval ā
ā ā¢ BERTScore (semantic) ā ā¢ Perplexity ā
ā ā¢ Exact Match ā ā¢ MT-Bench ā
ā ā ā¢ Human evaluation ā
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
```
---
## š Section 2: Text Similarity Metrics
### 2.1 BLEU ā Bilingual Evaluation Understudy
Measures n-gram precision between generated and reference text. Originally for translation.
```python
from nltk.translate.bleu_score import sentence_bleu, corpus_bleu, SmoothingFunction
from nltk.tokenize import word_tokenize
import nltk
nltk.download("punkt", quiet=True)
def compute_bleu(reference: str, hypothesis: str) -> dict:
"""Compute BLEU-1 through BLEU-4 scores"""
ref_tokens = [word_tokenize(reference.lower())]
hyp_tokens = word_tokenize(hypothesis.lower())
smoothing = SmoothingFunction().method1
scores = {}
for n in range(1, 5):
weights = tuple([1/n] * n + [0] * (4-n))
score = sentence_bleu(ref_tokens, hyp_tokens, weights=weights, smoothing_function=smoothing)
scores[f"BLEU-{n}"] = round(score, 4)
return scores
# Examples
pairs = [
{
"reference": "The quick brown fox jumps over the lazy dog",
"hypothesis": "A fast brown fox leaps over a lazy dog"
},
{
"reference": "Machine learning algorithms learn from data automatically",
"hypothesis": "ML algorithms automatically learn patterns from data"
},
{
"reference": "The Eiffel Tower is located in Paris France",
"hypothesis": "The Eiffel Tower can be found in Madrid Spain" # Wrong answer
}
]
print("BLEU SCORES COMPARISON:")
for pair in pairs:
scores = compute_bleu(pair["reference"], pair["hypothesis"])
print(f"\nRef: {pair['reference'][:50]}")
print(f"Hyp: {pair['hypothesis'][:50]}")
print(f" {scores}")
```
### 2.2 ROUGE ā Recall-Oriented Understudy
Measures recall of n-grams (and longest common subsequence). Better for summarization.
```python
from rouge_score import rouge_scorer
def compute_rouge(reference: str, hypothesis: str) -> dict:
"""Compute ROUGE-1, ROUGE-2, and ROUGE-L"""
scorer = rouge_scorer.RougeScorer(
["rouge1", "rouge2", "rougeL"],
use_stemmer=True
)
scores = scorer.score(reference, hypothesis)
return {
"ROUGE-1": round(scores["rouge1"].fmeasure, 4),
"ROUGE-2": round(scores["rouge2"].fmeasure, 4),
"ROUGE-L": round(scores["rougeL"].fmeasure, 4),
"Precision-1": round(scores["rouge1"].precision, 4),
"Recall-1": round(scores["rouge1"].recall, 4),
}
reference_summary = """
The transformer architecture introduced in 2017 uses self-attention mechanisms
to process sequences in parallel. It consists of encoder and decoder blocks,
each with multi-head attention and feed-forward layers. Transformers have
replaced recurrent networks in most NLP tasks.
"""
generated_summaries = [
# Good summary ā covers main points
"Transformers, introduced in 2017, use self-attention for parallel sequence processing. "
"They have encoder-decoder blocks and dominated NLP, replacing RNNs.",
# Mediocre ā misses key facts
"Transformers are neural networks used for language processing tasks.",
# Bad ā barely relevant
"Deep learning has many applications in modern technology.",
]
print("ROUGE EVALUATION:")
for i, summary in enumerate(generated_summaries, 1):
scores = compute_rouge(reference_summary, summary)
print(f"\n[Summary {i}]: {summary[:70]}...")
print(f" ROUGE-1: {scores['ROUGE-1']} | ROUGE-2: {scores['ROUGE-2']} | ROUGE-L: {scores['ROUGE-L']}")
```
### 2.3 BERTScore ā Semantic Similarity
Goes beyond n-gram matching to measure semantic similarity using BERT embeddings:
```python
from bert_score import score as bert_score
# Install: pip install bert-score
def compute_bertscore(references: list[str], hypotheses: list[str]) -> dict:
"""
Compute BERTScore for a batch of reference/hypothesis pairs.
Higher scores indicate better semantic similarity.
"""
precision, recall, f1 = bert_score(
hypotheses,
references,
lang="en",
model_type="distilbert-base-uncased",
verbose=False
)
return {
"Precision": round(float(precision.mean()), 4),
"Recall": round(float(recall.mean()), 4),
"F1": round(float(f1.mean()), 4)
}
# BERTScore handles paraphrases better than BLEU/ROUGE
refs = ["The cat sat on the mat."] * 3
hyps = [
"The cat sat on the mat.", # Exact match
"A feline was resting on a rug.", # Paraphrase
"Dogs bark loudly at night." # Unrelated
]
print("BERTSCORE COMPARISON:")
for ref, hyp in zip(refs, hyps):
scores = compute_bertscore([ref], [hyp])
print(f"\nRef: {ref}")
print(f"Hyp: {hyp}")
print(f" F1: {scores['F1']}")
```
### 2.4 Choosing the Right Metric
| Task | Recommended Metrics |
|------|-------------------|
| Machine Translation | BLEU-4, BERTScore |
| Summarization | ROUGE-1/2/L, BERTScore |
| RAG Accuracy | Faithfulness, Answer Relevancy (RAGAS) |
| Open-ended Generation | LLM-as-Judge, Human Eval |
| Code Generation | Exact Match, Pass@k |
| Classification | Accuracy, F1 |
---
## š Section 3: LLM-as-Judge
### 3.1 Why LLM Judges Work
Automated metrics like BLEU miss nuance. GPT-4 evaluation correlates strongly with human judgments (85-90% agreement on many tasks).
```python
from openai import OpenAI
import os
from dotenv import load_dotenv
load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
def evaluate_with_llm(
question: str,
response: str,
reference: str = None,
criteria: list[str] = None
) -> dict:
"""
Use GPT-4o-mini as a judge to evaluate a model response.
Returns scores and reasoning for each criterion.
"""
default_criteria = [
"accuracy: Does the response contain correct information?",
"completeness: Does the response fully address the question?",
"clarity: Is the response clearly written and easy to understand?",
"conciseness: Is the response appropriately concise without unnecessary padding?"
]
criteria = criteria or default_criteria
reference_text = f"\nReference Answer: {reference}\n" if reference else ""
eval_prompt = f"""You are an expert evaluator of AI responses.
Evaluate the provided response on these criteria (score 1-5, where 5=excellent):
Question: {question}
{reference_text}
Response to Evaluate: {response}
For each criterion, provide a score and brief explanation:
{chr(10).join(f"- {c}" for c in criteria)}
Respond in this JSON format:
{{
"scores": {{
"accuracy": {{"score": 1-5, "reasoning": "explanation"}},
"completeness": {{"score": 1-5, "reasoning": "explanation"}},
"clarity": {{"score": 1-5, "reasoning": "explanation"}},
"conciseness": {{"score": 1-5, "reasoning": "explanation"}}
}},
"overall_score": (average of all scores),
"verdict": "EXCELLENT" | "GOOD" | "ADEQUATE" | "POOR"
}}"""
result = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": eval_prompt}],
response_format={"type": "json_object"},
temperature=0
)
import json
return json.loads(result.choices[0].message.content)
# Test evaluation
question = "What is the transformer architecture and why is it important?"
responses = [
# Good response
"Transformers are deep learning architectures introduced in 2017 that use self-attention mechanisms to process sequences in parallel. They're important because they replaced slower RNNs and LSTMs, enabled training on massive datasets, and became the foundation for all modern LLMs like GPT and BERT.",
# Poor response
"Transformers are a type of AI. They are very important and useful for many things.",
]
print("LLM-AS-JUDGE EVALUATION:")
for i, response in enumerate(responses, 1):
print(f"\n[Response {i}]: {response[:100]}...")
eval_result = evaluate_with_llm(question, response)
print(f" Overall Score: {eval_result['overall_score']}/5")
print(f" Verdict: {eval_result['verdict']}")
for criterion, data in eval_result['scores'].items():
print(f" {criterion}: {data['score']}/5 ā {data['reasoning'][:60]}...")
```
---
## š Section 4: RAGAS for RAG Evaluation
```python
# Full RAGAS evaluation setup
# Install: pip install ragas datasets
from datasets import Dataset
# Prepare test set: questions, expected answers, and what your RAG returns
eval_samples = [
{
"question": "What is the capital of France?",
"answer": "The capital of France is Paris, which is also its largest city.",
"contexts": ["Paris is the capital and most populous city of France..."],
"ground_truth": "Paris"
},
{
"question": "What are the key components of a transformer?",
"answer": "A transformer has encoder blocks, decoder blocks, multi-head self-attention, and feed-forward layers.",
"contexts": ["Transformers consist of encoder-decoder architecture with self-attention mechanisms and feed-forward networks..."],
"ground_truth": "Encoder, decoder, multi-head attention, feed-forward layers"
}
]
dataset = Dataset.from_list(eval_samples)
# Evaluate
# from ragas import evaluate
# from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
# results = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision, context_recall])
# print(results.to_pandas())
```
---
## š» Full Lab: Evaluation Pipeline
```python
# lab_day20_eval_pipeline.py
"""Build a complete LLM evaluation pipeline"""
import json, csv
from datetime import datetime
from dataclasses import dataclass, field, asdict
from typing import Optional
from openai import OpenAI
from rouge_score import rouge_scorer
from dotenv import load_dotenv
load_dotenv()
@dataclass
class EvalResult:
question: str
reference: str
generated: str
rouge_1: float = 0.0
rouge_l: float = 0.0
llm_score: float = 0.0
llm_verdict: str = ""
model: str = ""
timestamp: str = field(default_factory=lambda: datetime.now().isoformat())
class EvalPipeline:
"""Complete evaluation pipeline for LLM responses"""
def __init__(self, judge_model: str = "gpt-4o-mini"):
self.client = OpenAI()
self.judge_model = judge_model
self.rouge = rouge_scorer.RougeScorer(["rouge1", "rougeL"], use_stemmer=True)
self.results: list[EvalResult] = []
def generate_response(self, question: str, model: str = "gpt-4o-mini") -> str:
response = self.client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": question}],
temperature=0
)
return response.choices[0].message.content
def evaluate_rouge(self, reference: str, hypothesis: str) -> dict:
scores = self.rouge.score(reference, hypothesis)
return {
"rouge_1": round(scores["rouge1"].fmeasure, 4),
"rouge_l": round(scores["rougeL"].fmeasure, 4),
}
def evaluate_with_judge(self, question: str, response: str, reference: str) -> dict:
prompt = f"""Evaluate this response (1-5 scale). Return JSON.
Question: {question}
Reference: {reference}
Response: {response}
{{
"accuracy": <1-5>,
"completeness": <1-5>,
"clarity": <1-5>,
"overall": <average>,
"verdict": "EXCELLENT|GOOD|ADEQUATE|POOR"
}}"""
result = self.client.chat.completions.create(
model=self.judge_model,
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
temperature=0
)
return json.loads(result.choices[0].message.content)
def run(self, test_cases: list[dict], model: str = "gpt-4o-mini") -> list[EvalResult]:
"""Run evaluation on a list of {question, reference} pairs"""
print(f"Running {len(test_cases)} test cases with model: {model}")
for i, case in enumerate(test_cases, 1):
q, ref = case["question"], case["reference"]
print(f" [{i}/{len(test_cases)}] {q[:50]}...")
generated = self.generate_response(q, model=model)
rouge_scores = self.evaluate_rouge(ref, generated)
llm_eval = self.evaluate_with_judge(q, generated, ref)
result = EvalResult(
question=q,
reference=ref,
generated=generated,
rouge_1=rouge_scores["rouge_1"],
rouge_l=rouge_scores["rouge_l"],
llm_score=llm_eval.get("overall", 0),
llm_verdict=llm_eval.get("verdict", ""),
model=model
)
self.results.append(result)
return self.results
def compare_models(self, test_cases: list[dict], models: list[str]) -> None:
"""Compare multiple models on the same test set"""
print(f"\nš Comparing {len(models)} models on {len(test_cases)} questions")
model_results = {}
for model in models:
results = self.run(test_cases, model=model)
avg_rouge = sum(r.rouge_1 for r in results) / len(results)
avg_llm = sum(r.llm_score for r in results) / len(results)
model_results[model] = {"rouge_1": avg_rouge, "llm_score": avg_llm}
self.results = [] # Reset for next model
print("\n[MODEL COMPARISON]")
print(f"{'Model':25} {'ROUGE-1':10} {'LLM Score':10}")
print("-" * 50)
for model, scores in sorted(model_results.items(), key=lambda x: x[1]["llm_score"], reverse=True):
print(f"{model:25} {scores['rouge_1']:.4f} {scores['llm_score']:.2f}/5")
def save_report(self, path: str = "eval_report.csv") -> None:
with open(path, "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=list(asdict(self.results[0]).keys()))
writer.writeheader()
writer.writerows([asdict(r) for r in self.results])
print(f"š Report saved to {path}")
# āā Demo āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
test_cases = [
{
"question": "What is retrieval-augmented generation (RAG)?",
"reference": "RAG combines a retrieval system with a language model. The retrieval system finds relevant documents, which are then provided as context to the LLM for answer generation."
},
{
"question": "What is the difference between supervised and unsupervised learning?",
"reference": "Supervised learning uses labeled data to train models that predict outputs. Unsupervised learning finds patterns in unlabeled data without predefined outputs."
},
]
pipeline = EvalPipeline()
results = pipeline.run(test_cases, model="gpt-4o-mini")
print("\nš EVALUATION RESULTS:")
for r in results:
print(f"\nā {r.question[:60]}")
print(f" ROUGE-1: {r.rouge_1:.4f} | LLM Score: {r.llm_score:.2f}/5 | Verdict: {r.llm_verdict}")
pipeline.save_report("eval_report.csv")
print("\nā
Day 20 Lab Complete!")
```
---
## š Section 5: LLM Benchmarks Overview
| Benchmark | Tests | Key Metric |
|-----------|-------|-----------|
| **MMLU** | 57 subject knowledge areas | 5-shot accuracy |
| **HellaSwag** | Commonsense reasoning | Accuracy |
| **MT-Bench** | Instruction following (multi-turn) | GPT-4 score 1-10 |
| **HumanEval** | Python code generation | Pass@1 |
| **GSM8K** | Grade school math | Accuracy |
| **TruthfulQA** | Avoidance of misconceptions | % truthful |
| **BigBench Hard** | Hard reasoning tasks | Accuracy |
| **LMSYS Chatbot Arena** | Human preference (head-to-head) | ELO rating |
---
## š§ Quiz: Day 20
**Q1:** BLEU primarily measures:
- A) Recall of n-grams
- B) **Precision of n-grams ā
**
- C) Semantic similarity
- D) LLM preference
**Q2:** BERTScore's advantage over BLEU/ROUGE is:
- A) It's faster to compute
- B) It works without reference texts
- C) **It captures semantic similarity, not just exact n-gram matches ā
**
- D) It handles longer documents better
**Q3:** LLM-as-Judge is appropriate when:
- A) You have clear reference answers
- B) **You need nuanced evaluation of open-ended generation ā
**
- C) You want the fastest evaluation
- D) You have no budget for LLM API calls
**Q4:** RAGAS `faithfulness` measures:
- A) How fast the RAG system responds
- B) Whether the retrieved documents are relevant
- C) **Whether the generated answer is grounded in the retrieved context ā
**
- D) Whether context precision is high
**Q5:** You should use ROUGE over BLEU for:
- A) Machine translation evaluation
- B) Code generation assessment
- C) **Summarization evaluation (recall-focused) ā
**
- D) Chatbot response rating
---
## š Key Takeaways
| Metric | Type | Best For |
|--------|------|---------|
| **BLEU** | Reference-based, n-gram precision | Translation |
| **ROUGE-L** | Reference-based, LCS recall | Summarization |
| **BERTScore** | Reference-based, semantic | General text quality |
| **LLM-as-Judge** | Reference-free | Open-ended generation |
| **RAGAS** | Multi-dimensional RAG eval | RAG systems |
| **MMLU/MT-Bench** | Standardized benchmarks | Comparing models |
---
*Day 20 Complete ā
| GenAI Course ā Week 3 | Next: Day 21 ā Week 3 Project*
---
## Section 5: LLM-as-Judge Evaluation
### 5.1 GPT-4 as a Judge
```python
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import JsonOutputParser
from pydantic import BaseModel
class EvalScore(BaseModel):
score: float # 0.0 to 10.0
reasoning: str
verdict: str # "pass" or "fail"
def gpt4_judge(question: str, reference_answer: str, model_answer: str) -> EvalScore:
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
prompt = ChatPromptTemplate.from_template("""
You are an expert evaluator. Score the model's answer vs the reference.
Question: {question}
Reference Answer: {reference}
Model Answer: {model_answer}
Score 0-10 based on: accuracy (40%), completeness (30%), conciseness (30%).
Return JSON: {{"score": 0-10, "reasoning": "...", "verdict": "pass/fail"}}
Pass threshold: score >= 7
""")
result = (prompt | llm | JsonOutputParser()).invoke({
"question": question,
"reference": reference_answer,
"model_answer": model_answer
})
return EvalScore(**result)
# Batch evaluation
questions = [
("What is RAG?", "RAG is Retrieval-Augmented Generation, which combines vector search with LLM generation.", None),
("How does attention work?", "Attention computes weighted sums of values based on query-key similarity scores.", None),
]
scores = []
for q, ref, _ in questions:
model_ans = "RAG helps AI systems access external knowledge" if "RAG" in q else "Transformers use self-attention to process all tokens simultaneously"
result = gpt4_judge(q, ref, model_ans)
scores.append(result.score)
print(f"Q: {q[:50]}")
print(f"Score: {result.score}/10 ({result.verdict}) - {result.reasoning[:80]}")
avg = sum(scores) / len(scores)
print(f"\nAverage score: {avg:.1f}/10")
```
### 5.2 Pairwise Comparison (A/B Testing)
```python
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import JsonOutputParser
import random
def pairwise_judge(question: str, response_a: str, response_b: str) -> dict:
"""Compare two model responses ā LLM picks the better one"""
llm = ChatOpenAI(model="gpt-4o", temperature=0)
# Randomly swap to avoid position bias
swapped = random.random() > 0.5
if swapped:
response_a, response_b = response_b, response_a
result = (
ChatPromptTemplate.from_template("""
Which response better answers the question? Be fair and objective.
Question: {q}
Response A: {a}
Response B: {b}
Return JSON: {{"winner": "A" or "B", "reason": "brief explanation"}}
""") | llm | JsonOutputParser()
).invoke({"q": question, "a": response_a, "b": response_b})
winner = result["winner"]
if swapped:
winner = "B" if winner == "A" else "A"
return {"winner": winner, "reason": result["reason"]}
```
---
## Section 6: GovEval and RAGAS
### 6.1 RAGAS Metrics Deep Dive
```python
# pip install ragas
from ragas import evaluate
from ragas.metrics import (
answer_relevancy,
faithfulness,
context_recall,
context_precision,
answer_correctness
)
from datasets import Dataset
# Prepare evaluation dataset (requires ground truth)
eval_data = {
"question": [
"What is the capital of France?",
"How does BERT differ from GPT?",
],
"answer": [
"The capital of France is Paris.",
"BERT is bidirectional and uses masked language modeling, while GPT is autoregressive.",
],
"contexts": [
["France is a country in Western Europe. Its capital city is Paris, known for the Eiffel Tower."],
["BERT uses a bidirectional encoder. GPT uses a decoder-only architecture for generation."],
],
"ground_truth": [
"Paris is the capital of France.",
"BERT is bidirectional (encoder-only), GPT is unidirectional (decoder-only, autoregressive).",
],
}
dataset = Dataset.from_dict(eval_data)
# Run RAGAS evaluation
result = evaluate(
dataset=dataset,
metrics=[
faithfulness, # Is answer supported by context?
answer_relevancy, # Is answer relevant to question?
context_precision, # Are retrieved contexts relevant?
context_recall, # Were all necessary contexts retrieved?
answer_correctness # Is answer correct vs ground truth?
]
)
print("RAGAS Evaluation Results:")
print(result.to_pandas())
# Interpret scores (all 0-1):
# faithfulness: > 0.8 is good (low hallucination)
# answer_relevancy: > 0.8 is good (on-topic)
# context_precision: > 0.7 is good (retrieval is precise)
# context_recall: > 0.8 is good (retrieved enough context)
# answer_correctness: > 0.7 is good (factually correct)
```
### 6.2 Building a Custom Evaluation Framework
```python
from dataclasses import dataclass, field
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import JsonOutputParser
from typing import Callable
import time, json
@dataclass
class EvalCase:
question: str
reference: str
context: list[str] = field(default_factory=list)
tags: list[str] = field(default_factory=list)
@dataclass
class EvalResult:
question: str
model_answer: str
scores: dict[str, float]
latency_ms: float
token_count: int
passed: bool
class LLMEvaluator:
"""Production-grade LLM evaluation framework"""
def __init__(self, judge_llm: str = "gpt-4o-mini", pass_threshold: float = 0.7):
self.llm = ChatOpenAI(model=judge_llm, temperature=0)
self.threshold = pass_threshold
self.results: list[EvalResult] = []
def _score_response(self, case: EvalCase, answer: str) -> dict[str, float]:
"""Get LLM scores on multiple dimensions"""
result = (
ChatPromptTemplate.from_template("""
Evaluate this AI response. Score each dimension 0.0-1.0.
Question: {q}
Reference: {ref}
AI Answer: {answer}
JSON: {{"accuracy": 0-1, "relevance": 0-1, "completeness": 0-1, "conciseness": 0-1}}
""") | self.llm | JsonOutputParser()
).invoke({"q": case.question, "ref": case.reference, "answer": answer})
return result
def add_result(self, case: EvalCase, answer: str, latency_ms: float, tokens: int):
scores = self._score_response(case, answer)
avg = sum(scores.values()) / len(scores)
self.results.append(EvalResult(
question=case.question,
model_answer=answer,
scores=scores,
latency_ms=latency_ms,
token_count=tokens,
passed=avg >= self.threshold
))
def summary(self) -> dict:
if not self.results:
return {}
pass_rate = sum(1 for r in self.results if r.passed) / len(self.results)
avg_latency = sum(r.latency_ms for r in self.results) / len(self.results)
dim_avgs = {}
for dim in self.results[0].scores:
dim_avgs[dim] = sum(r.scores.get(dim, 0) for r in self.results) / len(self.results)
return {
"total_cases": len(self.results),
"pass_rate": pass_rate,
"avg_latency_ms": avg_latency,
"dimension_averages": dim_avgs
}
def export_report(self, output_path: str = "eval_report.json"):
report = {
"summary": self.summary(),
"cases": [
{
"question": r.question,
"answer": r.model_answer[:100],
"scores": r.scores,
"passed": r.passed,
"latency_ms": r.latency_ms
}
for r in self.results
]
}
with open(output_path, "w") as f:
json.dump(report, f, indent=2)
print(f"Report saved: {output_path}")
```
---
## Section 7: Cost-Aware Evaluation
### 7.1 Benchmarking with Cost Tracking
```python
from openai import OpenAI
import time
client = OpenAI()
PRICE_PER_1K_TOKENS = {
"gpt-4o": {"input": 0.005, "output": 0.015},
"gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
"gpt-3.5-turbo": {"input": 0.0015, "output": 0.002},
}
def benchmark_model(model: str, prompts: list[str]) -> dict:
"""Benchmark a model for latency, throughput, and cost"""
results = []
for prompt in prompts:
start = time.time()
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_tokens=200
)
latency = (time.time() - start) * 1000
usage = response.usage
prices = PRICE_PER_1K_TOKENS.get(model, {"input": 0, "output": 0})
cost = (usage.prompt_tokens / 1000 * prices["input"] +
usage.completion_tokens / 1000 * prices["output"])
results.append({
"latency_ms": latency,
"input_tokens": usage.prompt_tokens,
"output_tokens": usage.completion_tokens,
"cost_usd": cost
})
n = len(results)
return {
"model": model,
"cases_tested": n,
"avg_latency_ms": sum(r["latency_ms"] for r in results) / n,
"total_cost_usd": sum(r["cost_usd"] for r in results),
"cost_per_query": sum(r["cost_usd"] for r in results) / n,
"monthly_cost_at_10k_queries": sum(r["cost_usd"] for r in results) / n * 10000,
}
# Compare models cost-effectively
# test_prompts = ["Summarize RAG in 3 bullets.", "What is LoRA?", "Explain embeddings."]
# for model in ["gpt-4o-mini", "gpt-4o"]:
# result = benchmark_model(model, test_prompts)
# print(f"\n{result['model']}:")
# print(f" Avg latency: {result['avg_latency_ms']:.0f}ms")
# print(f" Cost/query: ${result['cost_per_query']:.5f}")
# print(f" Monthly (10K queries): ${result['monthly_cost_at_10k_queries']:.2f}")
```
---
## Extended Lab: Build an Automated Eval Pipeline
**Step 1 ā Create a golden dataset:**
- 30-50 questions with reference answers for your use case
- Include easy, medium, and hard examples
**Step 2 ā Instrument your chain:**
- Capture (question, retrieved_docs, generated_answer, latency, tokens) for each call
**Step 3 ā Run RAGAS:**
- faithfulness, answer_relevancy, context_precision for each case
- Aggregate and identify weak spots
**Step 4 ā Iterate:**
- Use failing cases to guide prompt refinement or retrieval improvement
- Re-run evaluation to confirm improvement
```python
print("=== Day 20 Extended Lab ===")
print("Goal: Build an automated eval pipeline for your RAG system")
print("Metrics: RAGAS (faithfulness, relevancy, precision, recall)")
print("Output: eval_report.json with per-case scores and summary")
print("See day-20-eval-pipeline.py for the complete implementation")
```
---
*Day 20 Extended Complete - Evaluation deep dive with LLM-as-Judge and RAGAS*
\n\n---\n\n## Expanded Expert Knowledge Base & Reference Guides\n\n
### Extended Academic Appendix: Generative AI Complete Glossary
* **Activation Function**: A mathematical equation attached to each neuron in a network that determines whether it should be activated or not. Examples include ReLU, GELU, and SwiGLU.
* **Adam Optimizer**: Adaptive Moment Estimation. An algorithm for optimization technique for gradient descent. The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients.
* **Alignment**: The process of ensuring AI systems act exactly in accordance with human intentions and values, preventing toxic, harmful, or legally dangerous logic generation paths.
* **API (Application Programming Interface)**: A software intermediary that allows two applications to talk to each other. In GenAI, it's how your software securely requests generations from massive cloud GPUs.
* **Auto-regressive**: A model that generates the future sequences step-by-step, conditioning the next prediction exclusively on the previous predictions it just generated.
* **Backpropagation**: The core algorithm behind learning in neural networks. It calculates the mathematical gradient of the loss function with respect to the weights by utilizing the chain rule, moving backwards from output to input.
* **Batch Size**: The number of training examples utilized in one single iteration of gradient descent before the network's internal mathematical parameters are updated.
* **Bias (Mathematical)**: A constant value added to the linear projection in neural layers ($y = mx + b$). It allows the activation function to shift to the left or right, increasing the flexibility of the network to fit complex data boundaries.
* **BPE (Byte Pair Encoding)**: A specific mathematical data compression technique adapted for NLP tokenization. It recursively merges the most frequently occurring pair of adjacent characters into a single new sub-word token.
* **Cache (KV Cache)**: In LLMs, the Key-Value matrices of previously generated tokens are stored in GPU VRAM so the transformer doesn't have to re-compute the entire 10,000-word essay every single time it tries to generate word 10,001.
* **Chain of Thought (CoT)**: A prompting strategy that forces the LLM to output a series of intermediate mathematical or logical reasoning steps before outputting the final answer, drastically improving accuracy on complex logic tasks.
* **Chinchilla Laws**: DeepMind's 2022 paper proving that to train compute-optimally, the dataset token count must scale perfectly linearly with the parameter count (a 20:1 ratio is strictly advised).
* **Constitutional AI**: Anthropic's proprietary alignment pipeline. A model is given a strict set of rules ('The Constitution') and autonomously critiques and revises its own responses, generating a vast RL dataset without expensive human labeling.
* **Context Window**: The maximum number of tokens (words/sub-words) a model can ingest and process mathematically in a single forward pass operation. GPT-4 handles 128k; Gemini handles 2 Million.
* **Cross-Entropy Loss**: The standard mathematical loss function used in classification tasks and language modeling. It calculates the delta between the model's predicted probability distribution and the actual rigid truth of the training data.
* **Decoder-Only Architecture**: A Transformer that abandons the bi-directional Encoder entirely (like GPT). It utilizes strictly causal, masked self-attention to generate sequential texts autoregressively.
* **Dense Model**: A standard neural network architecture where every single parameter in the computational block is activated and multiplied during every single forward pass (Contrast with MoE).
* **Discriminative Model**: Machine learning models designed fundamentally to draw mathematical boundaries between classes (e.g., Is this photo a hot dog or not a hot dog?) Contrast with Generative Models.
* **Dropout**: A cruel but effective regularization technique where a percentage of neurons in a layer are randomly completely deactivated during a training pass. This forces the network to stop relying on individual 'memorized' paths and build robust distributed representations.
* **Embedding**: The mathematical projection mapping a discrete token (like the word 'Apple') into a continuous dense continuous vector space where physical geometry and distance represent semantic linguistic meaning.
* **Epoch**: One full operational pass of the training pipeline sequentially interacting with the entire dataset. Foundation models are often trained for only 1 Epoch to prevent catastrophic overfitting logic traps.
* **Feed-Forward Network (FFN)**: The dense, localized multi-layer perceptron block inside a Transformer layer operating independently on each token vector specifically acting as the model's 'Key-Value fact database'.
* **Fine-Tuning**: Taking a massive, previously trained generic foundation model and training it further on a tiny, specific domain dataset (like medical journals) using a low learning rate to alter its core behavior.
* **FP16 (Half Precision)**: A computer number format occupying 16 bits. HuggingFace models default to this. Represents a compromise utilizing half the GPU VRAM of 32-bit floats with mathematically negligible degradation in AI loss metrics.
* **Foundation Model**: A gargantuan neural network trained utilizing massive unsupervised learning pipelines across the entire internet, serving as the base layer for countless downstream specific tasks.
* **Generative Model**: AI architecture designed to map and understand the fundamental underlying distribution of data specifically to generate completely novel, statistically adjacent synthetic data. (e.g. LLMs, Diffusion Models).
* **GQA (Grouped-Query Attention)**: An architectural optimization. Instead of calculating a massive individual Key and Value matrix for every single Query Head in Multi-Head Attention, multiple Query heads mathematically share the same Key/Value arrays, vastly saving VRAM.
* **GPU (Graphics Processing Unit)**: The physical silicon hardware engines powering AI. Thousands of cores designed specifically to execute massive parallel floating point Matrix Multiplication extremely efficiently. NVIDIA dominates this landscape.
* **Gradient Descent**: The mathematical optimization algorithm locating the minimum of a neural loss curve by taking scaled sequential steps in the exact opposite operational direction of the calculated tensor gradient.
* **Hallucination**: When a generative AI model outputs convincing, confident logic strings that are completely factually incorrect due primarily to token probability space interpolation artifacts.
* **Hugging Face**: The central structural GitHub for Machine Learning. A gigantic repository hosting open-source model weights, extensive NLP datasets, and the most heavily utilized `transformers` Python inference library globally.
* **Hyperparameters**: The architectural variables of a neural network that are set manually by the human engineer *before* training begins (e.g. Learning Rate, Batch Size, Dropout Rate) and are never altered by backpropagation.
* **In-Context Learning**: The mysterious emergent capability of massive LLMs to learn completely new tasks instantly utilizing purely the context text inside the given prompt, requiring zero permanent weight tensor alterations.
* **Instruction Tuning**: Fine-tuning base models specifically to respond obediently to 'User Prompts'. A base model will complete a sentence; an instruction-tuned model will act like a dialogue assistant.
* **INT4 (4-Bit Quantization)**: An extreme computational compression technique squashing a 16-bit weight parameter mathematically down to just 4 bits. Allows executing an 8 Billion parameter model on a standard 6GB laptop GPU.
* **Knowledge Distillation**: A training pipeline where a gargantuan 'Teacher' model generates massive amounts of high-quality synthetic data to train a tiny 'Student' model structurally imitating its superior behaviors.
* **Llama**: Meta's flagship family of Open-Weight Large Language Models. Responsible singularly for unleashing the massive democratization of the enterprise Local-LLM hosting revolution.
* **LLM (Large Language Model)**: A deep learning neural network, generally executing a Transformer architecture, possessing billions of parameters, specifically designed to process, map, and generate natural human linguistics.
* **Logits**: The raw, unnormalized massive mathematical scores output directly by the final linear projection layer of the neural network natively *before* entering the Softmax bounding probability function.
* **LoRA (Low-Rank Adaptation)**: A parameter-efficient Fine-Tuning miracle. Instead of freezing 8 billion parameters, LoRA injects two tiny matrices side-by-side, dramatically slashing the required training VRAM computational burden by 98%.
* **Masked Attention**: A mandatory structural configuration inside a Decoder enforcing causality. It mathematically blocks the model from executing any attention logic targeting tokens located positioned *after* the current token.
* **Mixture of Experts (MoE)**: A neural block containing several 'expert' sub-networks (like 8 separate FFNs). A routing gate decides exactly which two experts activate for each specific input vector, preserving massive inference speed.
* **Multi-Head Attention**: Processing multiple self-attention operations concurrently in physically separate lower-dimensional sub-spaces immediately before concatenating them. Allows parsing extreme grammatical complexity without blurring the semantic signals.
* **Next-Token Prediction**: The incredibly simple underlying foundational logic objective utilized to pre-train almost all massive modern generative language models across Trillions of raw internet text documents.
* **NLP (Natural Language Processing)**: The massive overarching subfield of AI concerned entirely with programming algorithms to process, understand, analyze, and generate human linguistics and conversational grammar.
* **Overfitting**: A critical mathematical failure where a network memorizes the exact specific noise artifacts in the training dataset perfectly, completely destroying its capacity to generalize against unseen validation data.
* **Parameter**: The actual structural weights and biases contained natively deeply inside a neural network structure. An 8B parameter model literally contains 8,000,000,000 decimal numbers in 3D multi-dimensional arrays.
* **Positional Encoding**: The critical vector matrix added to the foundational input embeddings explicitly providing the Attention mathematical permutation logic with structural information regarding the exact sequence order of the tokens.
* **Pre-training**: The primary initial phase of creating a Foundation model. Feeding Trillions of text tokens through thousands of GPUs for weeks consuming megawatts of power strictly performing unsupervised next-token prediction.
* **Prompt Engineering**: The process of empirically designing, testing, and optimizing the structural format of linguistic inputs injected into LLMs to extract specifically desired, accurate, and robust structural logic outputs.
* **Python**: The undisputed dominant syntactic language dominating the Machine Learning backend architecture globally. Used to interface natively with the massively optimized C++ PyTorch/TensorFlow backend structures.
* **Quantization**: The process of fundamentally converting the continuous mathematical precision of model weight sets from floating point 32/16-bit resolutions down to block-level 8-bit or 4-bit, sacrificing microscopic accuracy for massive VRAM deployment efficiency.
* **RAG (Retrieval-Augmented Generation)**: The absolute standard deployed architecture for enterprise applications. Preventing LLM hallucinations by intercepting the user query, searching an external enterprise vector database for facts, and injecting those raw facts into the LLM context prior to generation.
* **Recurrent Neural Network (RNN)**: The legacy sequential architecture predating Transformers. Structurally processes tokens linearly one-by-one utilizing a continuous expanding hidden state. Massively vulnerable to extreme 'vanishing gradient' failure over large context windows.
* **ReLU (Rectified Linear Unit)**: A critically vital, simple activation formula: $f(x) = max(0, x)$. It structurally forces any massive negative outputs directly to 0.0, injecting mandatory non-linearity into massive chained matrix multiplications.
* **Representation Learning**: A set of structural ML techniques allowing underlying systems to automatically discover the raw distinct structural features or representations natively required for complex classification natively from raw data blocks.
* **Residual Connection (Skip Connection)**: Crucial architectural bypass lines transmitting original input vectors structurally directly deeply around massive network layers, instantly adding them to the final outputs. These bypass physical lines solve the vanishing gradient apocalypse inside massively deep networks.
* **RLHF (Reinforcement Learning from Human Feedback)**: The specific complex post-training fine-tuning pipeline rendering GPT-4 capable of human dialogue. Involves executing a massive secondary reward-model mathematically scoring generating responses purely based on expensive curated human feedback rankings.
* **RoPE (Rotary Position Embedding)**: A 2021 revolutionary positional encoding standard deployed across Llama and Mistral sequences. It natively rotates the grammatical Query/Key vectors dimensionally in a complex physical subspace mapping exact relative distance rather than simple sequence addition.
* **Self-Attention**: The singular foundational mechanism anchoring the Transformer legacy. A mathematical mechanism physically relating massive disparate elements uniquely across entirely distinct sequences against one another directly to aggregate rich integrated semantic grammatical meaning.
* **Softmax Function**: A vital structural normalization formula algorithm physically scaling an array composed of massive unnormalized numerical sequences entirely to fit bounded logically explicitly between exactly 0.0 and 1.0, rendering them perfectly functional as a standard statistical probability distribution structure.
* **SwiGLU**: A highly advanced 2025 non-linear dynamic activation block structurally deploying Swish gating pipelines heavily substituting standard ReLU gates extensively inside modern elite model matrices like Meta's Llama sequences, extracting elite processing dynamics.
* **Temperature (T)**: The primary physical decoding mathematical scalar parameter bounding generative randomness outputs. Approaching $T=0.0$ forces rigid, repetitive deterministic absolute certainty matrices, whereas raising $T=1.0+$ enforces wild erratic linguistic distributional variance sequences.
* **Tensor**: The massive core architectural fundamental building block powering Deep Learning ecosystems mapping identical arrays strictly scaling multi-dimensional numeric matrices structurally generalizing basic matrix algebra natively processing GPU matrix logic flows.
* **Token**: The fundamental structural sub-unit blocks consumed iteratively deeply inside LLM architectures. Generally representing fractional linguistic characters parsing out uniquely exactly approximately matching mathematically structural word matrices equivalent effectively equating 4 sequential raw alphabetical English letters.
* **Transformer**: The 2017 undisputed architectural miracle sequence dominating the fundamental ecosystem of Artificial Intelligence generation natively entirely substituting standard Recurrent pipelines comprehensively utilizing massive Parallel Sparse Attention structural distributions.
* **Underfitting**: The diametric inverse computational failure dynamically destroying machine sequence accuracy natively emerging when mathematical network parameters severely lack complex deep architectural capacity parameters distinctly mapping massive underlying geometric functional relationship flows.
* **Vector Database**: An explicitly specialized indexing enterprise deployment architectural database standard structurally deployed comprehensively globally storing extreme dense multi-dimensional numeric vectors natively supporting ultra-rapid K-Nearest-Neighbor cosine search queries anchoring core RAG sequence systems.
* **Weights**: The incredibly precise decimal numbers stored iteratively physically comprising neural networks natively adjusting mathematically via backpropagation gradients structurally driving complex non-linear sequence generation logic models perfectly.
* **Zero-Shot Learning**: The massive foundational AI ecosystem paradigm natively enabling explicit model completion capabilities operating distinctly successfully parsing tasks functionally completely utterly unseen across the generative pre-training massive pipeline structure.
### Extended Academic Appendix: Generative AI Complete Glossary
* **Activation Function**: A mathematical equation attached to each neuron in a network that determines whether it should be activated or not. Examples include ReLU, GELU, and SwiGLU.
* **Adam Optimizer**: Adaptive Moment Estimation. An algorithm for optimization technique for gradient descent. The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients.
* **Alignment**: The process of ensuring AI systems act exactly in accordance with human intentions and values, preventing toxic, harmful, or legally dangerous logic generation paths.
* **API (Application Programming Interface)**: A software intermediary that allows two applications to talk to each other. In GenAI, it's how your software securely requests generations from massive cloud GPUs.
* **Auto-regressive**: A model that generates the future sequences step-by-step, conditioning the next prediction exclusively on the previous predictions it just generated.
* **Backpropagation**: The core algorithm behind learning in neural networks. It calculates the mathematical gradient of the loss function with respect to the weights by utilizing the chain rule, moving backwards from output to input.
* **Batch Size**: The number of training examples utilized in one single iteration of gradient descent before the network's internal mathematical parameters are updated.
* **Bias (Mathematical)**: A constant value added to the linear projection in neural layers ($y = mx + b$). It allows the activation function to shift to the left or right, increasing the flexibility of the network to fit complex data boundaries.
* **BPE (Byte Pair Encoding)**: A specific mathematical data compression technique adapted for NLP tokenization. It recursively merges the most frequently occurring pair of adjacent characters into a single new sub-word token.
* **Cache (KV Cache)**: In LLMs, the Key-Value matrices of previously generated tokens are stored in GPU VRAM so the transformer doesn't have to re-compute the entire 10,000-word essay every single time it tries to generate word 10,001.
* **Chain of Thought (CoT)**: A prompting strategy that forces the LLM to output a series of intermediate mathematical or logical reasoning steps before outputting the final answer, drastically improving accuracy on complex logic tasks.
* **Chinchilla Laws**: DeepMind's 2022 paper proving that to train compute-optimally, the dataset token count must scale perfectly linearly with the parameter count (a 20:1 ratio is strictly advised).
* **Constitutional AI**: Anthropic's proprietary alignment pipeline. A model is given a strict set of rules ('The Constitution') and autonomously critiques and revises its own responses, generating a vast RL dataset without expensive human labeling.
* **Context Window**: The maximum number of tokens (words/sub-words) a model can ingest and process mathematically in a single forward pass operation. GPT-4 handles 128k; Gemini handles 2 Million.
* **Cross-Entropy Loss**: The standard mathematical loss function used in classification tasks and language modeling. It calculates the delta between the model's predicted probability distribution and the actual rigid truth of the training data.
* **Decoder-Only Architecture**: A Transformer that abandons the bi-directional Encoder entirely (like GPT). It utilizes strictly causal, masked self-attention to generate sequential texts autoregressively.
* **Dense Model**: A standard neural network architecture where every single parameter in the computational block is activated and multiplied during every single forward pass (Contrast with MoE).
* **Discriminative Model**: Machine learning models designed fundamentally to draw mathematical boundaries between classes (e.g., Is this photo a hot dog or not a hot dog?) Contrast with Generative Models.
* **Dropout**: A cruel but effective regularization technique where a percentage of neurons in a layer are randomly completely deactivated during a training pass. This forces the network to stop relying on individual 'memorized' paths and build robust distributed representations.
* **Embedding**: The mathematical projection mapping a discrete token (like the word 'Apple') into a continuous dense continuous vector space where physical geometry and distance represent semantic linguistic meaning.
* **Epoch**: One full operational pass of the training pipeline sequentially interacting with the entire dataset. Foundation models are often trained for only 1 Epoch to prevent catastrophic overfitting logic traps.
* **Feed-Forward Network (FFN)**: The dense, localized multi-layer perceptron block inside a Transformer layer operating independently on each token vector specifically acting as the model's 'Key-Value fact database'.
* **Fine-Tuning**: Taking a massive, previously trained generic foundation model and training it further on a tiny, specific domain dataset (like medical journals) using a low learning rate to alter its core behavior.
* **FP16 (Half Precision)**: A computer number format occupying 16 bits. HuggingFace models default to this. Represents a compromise utilizing half the GPU VRAM of 32-bit floats with mathematically negligible degradation in AI loss metrics.
* **Foundation Model**: A gargantuan neural network trained utilizing massive unsupervised learning pipelines across the entire internet, serving as the base layer for countless downstream specific tasks.
* **Generative Model**: AI architecture designed to map and understand the fundamental underlying distribution of data specifically to generate completely novel, statistically adjacent synthetic data. (e.g. LLMs, Diffusion Models).
* **GQA (Grouped-Query Attention)**: An architectural optimization. Instead of calculating a massive individual Key and Value matrix for every single Query Head in Multi-Head Attention, multiple Query heads mathematically share the same Key/Value arrays, vastly saving VRAM.
* **GPU (Graphics Processing Unit)**: The physical silicon hardware engines powering AI. Thousands of cores designed specifically to execute massive parallel floating point Matrix Multiplication extremely efficiently. NVIDIA dominates this landscape.
* **Gradient Descent**: The mathematical optimization algorithm locating the minimum of a neural loss curve by taking scaled sequential steps in the exact opposite operational direction of the calculated tensor gradient.
* **Hallucination**: When a generative AI model outputs convincing, confident logic strings that are completely factually incorrect due primarily to token probability space interpolation artifacts.
* **Hugging Face**: The central structural GitHub for Machine Learning. A gigantic repository hosting open-source model weights, extensive NLP datasets, and the most heavily utilized `transformers` Python inference library globally.
* **Hyperparameters**: The architectural variables of a neural network that are set manually by the human engineer *before* training begins (e.g. Learning Rate, Batch Size, Dropout Rate) and are never altered by backpropagation.
* **In-Context Learning**: The mysterious emergent capability of massive LLMs to learn completely new tasks instantly utilizing purely the context text inside the given prompt, requiring zero permanent weight tensor alterations.
* **Instruction Tuning**: Fine-tuning base models specifically to respond obediently to 'User Prompts'. A base model will complete a sentence; an instruction-tuned model will act like a dialogue assistant.
* **INT4 (4-Bit Quantization)**: An extreme computational compression technique squashing a 16-bit weight parameter mathematically down to just 4 bits. Allows executing an 8 Billion parameter model on a standard 6GB laptop GPU.
* **Knowledge Distillation**: A training pipeline where a gargantuan 'Teacher' model generates massive amounts of high-quality synthetic data to train a tiny 'Student' model structurally imitating its superior behaviors.
* **Llama**: Meta's flagship family of Open-Weight Large Language Models. Responsible singularly for unleashing the massive democratization of the enterprise Local-LLM hosting revolution.
* **LLM (Large Language Model)**: A deep learning neural network, generally executing a Transformer architecture, possessing billions of parameters, specifically designed to process, map, and generate natural human linguistics.
* **Logits**: The raw, unnormalized massive mathematical scores output directly by the final linear projection layer of the neural network natively *before* entering the Softmax bounding probability function.
* **LoRA (Low-Rank Adaptation)**: A parameter-efficient Fine-Tuning miracle. Instead of freezing 8 billion parameters, LoRA injects two tiny matrices side-by-side, dramatically slashing the required training VRAM computational burden by 98%.
* **Masked Attention**: A mandatory structural configuration inside a Decoder enforcing causality. It mathematically blocks the model from executing any attention logic targeting tokens located positioned *after* the current token.
* **Mixture of Experts (MoE)**: A neural block containing several 'expert' sub-networks (like 8 separate FFNs). A routing gate decides exactly which two experts activate for each specific input vector, preserving massive inference speed.
* **Multi-Head Attention**: Processing multiple self-attention operations concurrently in physically separate lower-dimensional sub-spaces immediately before concatenating them. Allows parsing extreme grammatical complexity without blurring the semantic signals.
* **Next-Token Prediction**: The incredibly simple underlying foundational logic objective utilized to pre-train almost all massive modern generative language models across Trillions of raw internet text documents.
* **NLP (Natural Language Processing)**: The massive overarching subfield of AI concerned entirely with programming algorithms to process, understand, analyze, and generate human linguistics and conversational grammar.
* **Overfitting**: A critical mathematical failure where a network memorizes the exact specific noise artifacts in the training dataset perfectly, completely destroying its capacity to generalize against unseen validation data.
* **Parameter**: The actual structural weights and biases contained natively deeply inside a neural network structure. An 8B parameter model literally contains 8,000,000,000 decimal numbers in 3D multi-dimensional arrays.
* **Positional Encoding**: The critical vector matrix added to the foundational input embeddings explicitly providing the Attention mathematical permutation logic with structural information regarding the exact sequence order of the tokens.
* **Pre-training**: The primary initial phase of creating a Foundation model. Feeding Trillions of text tokens through thousands of GPUs for weeks consuming megawatts of power strictly performing unsupervised next-token prediction.
* **Prompt Engineering**: The process of empirically designing, testing, and optimizing the structural format of linguistic inputs injected into LLMs to extract specifically desired, accurate, and robust structural logic outputs.
* **Python**: The undisputed dominant syntactic language dominating the Machine Learning backend architecture globally. Used to interface natively with the massively optimized C++ PyTorch/TensorFlow backend structures.
* **Quantization**: The process of fundamentally converting the continuous mathematical precision of model weight sets from floating point 32/16-bit resolutions down to block-level 8-bit or 4-bit, sacrificing microscopic accuracy for massive VRAM deployment efficiency.
* **RAG (Retrieval-Augmented Generation)**: The absolute standard deployed architecture for enterprise applications. Preventing LLM hallucinations by intercepting the user query, searching an external enterprise vector database for facts, and injecting those raw facts into the LLM context prior to generation.
* **Recurrent Neural Network (RNN)**: The legacy sequential architecture predating Transformers. Structurally processes tokens linearly one-by-one utilizing a continuous expanding hidden state. Massively vulnerable to extreme 'vanishing gradient' failure over large context windows.
* **ReLU (Rectified Linear Unit)**: A critically vital, simple activation formula: $f(x) = max(0, x)$. It structurally forces any massive negative outputs directly to 0.0, injecting mandatory non-linearity into massive chained matrix multiplications.
* **Representation Learning**: A set of structural ML techniques allowing underlying systems to automatically discover the raw distinct structural features or representations natively required for complex classification natively from raw data blocks.
* **Residual Connection (Skip Connection)**: Crucial architectural bypass lines transmitting original input vectors structurally directly deeply around massive network layers, instantly adding them to the final outputs. These bypass physical lines solve the vanishing gradient apocalypse inside massively deep networks.
* **RLHF (Reinforcement Learning from Human Feedback)**: The specific complex post-training fine-tuning pipeline rendering GPT-4 capable of human dialogue. Involves executing a massive secondary reward-model mathematically scoring generating responses purely based on expensive curated human feedback rankings.
* **RoPE (Rotary Position Embedding)**: A 2021 revolutionary positional encoding standard deployed across Llama and Mistral sequences. It natively rotates the grammatical Query/Key vectors dimensionally in a complex physical subspace mapping exact relative distance rather than simple sequence addition.
* **Self-Attention**: The singular foundational mechanism anchoring the Transformer legacy. A mathematical mechanism physically relating massive disparate elements uniquely across entirely distinct sequences against one another directly to aggregate rich integrated semantic grammatical meaning.
* **Softmax Function**: A vital structural normalization formula algorithm physically scaling an array composed of massive unnormalized numerical sequences entirely to fit bounded logically explicitly between exactly 0.0 and 1.0, rendering them perfectly functional as a standard statistical probability distribution structure.
* **SwiGLU**: A highly advanced 2025 non-linear dynamic activation block structurally deploying Swish gating pipelines heavily substituting standard ReLU gates extensively inside modern elite model matrices like Meta's Llama sequences, extracting elite processing dynamics.
* **Temperature (T)**: The primary physical decoding mathematical scalar parameter bounding generative randomness outputs. Approaching $T=0.0$ forces rigid, repetitive deterministic absolute certainty matrices, whereas raising $T=1.0+$ enforces wild erratic linguistic distributional variance sequences.
* **Tensor**: The massive core architectural fundamental building block powering Deep Learning ecosystems mapping identical arrays strictly scaling multi-dimensional numeric matrices structurally generalizing basic matrix algebra natively processing GPU matrix logic flows.
* **Token**: The fundamental structural sub-unit blocks consumed iteratively deeply inside LLM architectures. Generally representing fractional linguistic characters parsing out uniquely exactly approximately matching mathematically structural word matrices equivalent effectively equating 4 sequential raw alphabetical English letters.
* **Transformer**: The 2017 undisputed architectural miracle sequence dominating the fundamental ecosystem of Artificial Intelligence generation natively entirely substituting standard Recurrent pipelines comprehensively utilizing massive Parallel Sparse Attention structural distributions.
* **Underfitting**: The diametric inverse computational failure dynamically destroying machine sequence accuracy natively emerging when mathematical network parameters severely lack complex deep architectural capacity parameters distinctly mapping massive underlying geometric functional relationship flows.
* **Vector Database**: An explicitly specialized indexing enterprise deployment architectural database standard structurally deployed comprehensively globally storing extreme dense multi-dimensional numeric vectors natively supporting ultra-rapid K-Nearest-Neighbor cosine search queries anchoring core RAG sequence systems.
* **Weights**: The incredibly precise decimal numbers stored iteratively physically comprising neural networks natively adjusting mathematically via backpropagation gradients structurally driving complex non-linear sequence generation logic models perfectly.
* **Zero-Shot Learning**: The massive foundational AI ecosystem paradigm natively enabling explicit model completion capabilities operating distinctly successfully parsing tasks functionally completely utterly unseen across the generative pre-training massive pipeline structure.
### Extended Academic Appendix: Generative AI Complete Glossary
* **Activation Function**: A mathematical equation attached to each neuron in a network that determines whether it should be activated or not. Examples include ReLU, GELU, and SwiGLU.
* **Adam Optimizer**: Adaptive Moment Estimation. An algorithm for optimization technique for gradient descent. The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients.
* **Alignment**: The process of ensuring AI systems act exactly in accordance with human intentions and values, preventing toxic, harmful, or legally dangerous logic generation paths.
* **API (Application Programming Interface)**: A software intermediary that allows two applications to talk to each other. In GenAI, it's how your software securely requests generations from massive cloud GPUs.
* **Auto-regressive**: A model that generates the future sequences step-by-step, conditioning the next prediction exclusively on the previous predictions it just generated.
* **Backpropagation**: The core algorithm behind learning in neural networks. It calculates the mathematical gradient of the loss function with respect to the weights by utilizing the chain rule, moving backwards from output to input.
* **Batch Size**: The number of training examples utilized in one single iteration of gradient descent before the network's internal mathematical parameters are updated.
* **Bias (Mathematical)**: A constant value added to the linear projection in neural layers ($y = mx + b$). It allows the activation function to shift to the left or right, increasing the flexibility of the network to fit complex data boundaries.
* **BPE (Byte Pair Encoding)**: A specific mathematical data compression technique adapted for NLP tokenization. It recursively merges the most frequently occurring pair of adjacent characters into a single new sub-word token.
* **Cache (KV Cache)**: In LLMs, the Key-Value matrices of previously generated tokens are stored in GPU VRAM so the transformer doesn't have to re-compute the entire 10,000-word essay every single time it tries to generate word 10,001.
* **Chain of Thought (CoT)**: A prompting strategy that forces the LLM to output a series of intermediate mathematical or logical reasoning steps before outputting the final answer, drastically improving accuracy on complex logic tasks.
* **Chinchilla Laws**: DeepMind's 2022 paper proving that to train compute-optimally, the dataset token count must scale perfectly linearly with the parameter count (a 20:1 ratio is strictly advised).
* **Constitutional AI**: Anthropic's proprietary alignment pipeline. A model is given a strict set of rules ('The Constitution') and autonomously critiques and revises its own responses, generating a vast RL dataset without expensive human labeling.
* **Context Window**: The maximum number of tokens (words/sub-words) a model can ingest and process mathematically in a single forward pass operation. GPT-4 handles 128k; Gemini handles 2 Million.
* **Cross-Entropy Loss**: The standard mathematical loss function used in classification tasks and language modeling. It calculates the delta between the model's predicted probability distribution and the actual rigid truth of the training data.
* **Decoder-Only Architecture**: A Transformer that abandons the bi-directional Encoder entirely (like GPT). It utilizes strictly causal, masked self-attention to generate sequential texts autoregressively.
* **Dense Model**: A standard neural network architecture where every single parameter in the computational block is activated and multiplied during every single forward pass (Contrast with MoE).
* **Discriminative Model**: Machine learning models designed fundamentally to draw mathematical boundaries between classes (e.g., Is this photo a hot dog or not a hot dog?) Contrast with Generative Models.
* **Dropout**: A cruel but effective regularization technique where a percentage of neurons in a layer are randomly completely deactivated during a training pass. This forces the network to stop relying on individual 'memorized' paths and build robust distributed representations.
* **Embedding**: The mathematical projection mapping a discrete token (like the word 'Apple') into a continuous dense continuous vector space where physical geometry and distance represent semantic linguistic meaning.
* **Epoch**: One full operational pass of the training pipeline sequentially interacting with the entire dataset. Foundation models are often trained for only 1 Epoch to prevent catastrophic overfitting logic traps.
* **Feed-Forward Network (FFN)**: The dense, localized multi-layer perceptron block inside a Transformer layer operating independently on each token vector specifically acting as the model's 'Key-Value fact database'.
* **Fine-Tuning**: Taking a massive, previously trained generic foundation model and training it further on a tiny, specific domain dataset (like medical journals) using a low learning rate to alter its core behavior.
* **FP16 (Half Precision)**: A computer number format occupying 16 bits. HuggingFace models default to this. Represents a compromise utilizing half the GPU VRAM of 32-bit floats with mathematically negligible degradation in AI loss metrics.
* **Foundation Model**: A gargantuan neural network trained utilizing massive unsupervised learning pipelines across the entire internet, serving as the base layer for countless downstream specific tasks.
* **Generative Model**: AI architecture designed to map and understand the fundamental underlying distribution of data specifically to generate completely novel, statistically adjacent synthetic data. (e.g. LLMs, Diffusion Models).
* **GQA (Grouped-Query Attention)**: An architectural optimization. Instead of calculating a massive individual Key and Value matrix for every single Query Head in Multi-Head Attention, multiple Query heads mathematically share the same Key/Value arrays, vastly saving VRAM.
* **GPU (Graphics Processing Unit)**: The physical silicon hardware engines powering AI. Thousands of cores designed specifically to execute massive parallel floating point Matrix Multiplication extremely efficiently. NVIDIA dominates this landscape.
* **Gradient Descent**: The mathematical optimization algorithm locating the minimum of a neural loss curve by taking scaled sequential steps in the exact opposite operational direction of the calculated tensor gradient.
* **Hallucination**: When a generative AI model outputs convincing, confident logic strings that are completely factually incorrect due primarily to token probability space interpolation artifacts.
* **Hugging Face**: The central structural GitHub for Machine Learning. A gigantic repository hosting open-source model weights, extensive NLP datasets, and the most heavily utilized `transformers` Python inference library globally.
* **Hyperparameters**: The architectural variables of a neural network that are set manually by the human engineer *before* training begins (e.g. Learning Rate, Batch Size, Dropout Rate) and are never altered by backpropagation.
* **In-Context Learning**: The mysterious emergent capability of massive LLMs to learn completely new tasks instantly utilizing purely the context text inside the given prompt, requiring zero permanent weight tensor alterations.
* **Instruction Tuning**: Fine-tuning base models specifically to respond obediently to 'User Prompts'. A base model will complete a sentence; an instruction-tuned model will act like a dialogue assistant.
* **INT4 (4-Bit Quantization)**: An extreme computational compression technique squashing a 16-bit weight parameter mathematically down to just 4 bits. Allows executing an 8 Billion parameter model on a standard 6GB laptop GPU.
* **Knowledge Distillation**: A training pipeline where a gargantuan 'Teacher' model generates massive amounts of high-quality synthetic data to train a tiny 'Student' model structurally imitating its superior behaviors.
* **Llama**: Meta's flagship family of Open-Weight Large Language Models. Responsible singularly for unleashing the massive democratization of the enterprise Local-LLM hosting revolution.
* **LLM (Large Language Model)**: A deep learning neural network, generally executing a Transformer architecture, possessing billions of parameters, specifically designed to process, map, and generate natural human linguistics.
* **Logits**: The raw, unnormalized massive mathematical scores output directly by the final linear projection layer of the neural network natively *before* entering the Softmax bounding probability function.
* **LoRA (Low-Rank Adaptation)**: A parameter-efficient Fine-Tuning miracle. Instead of freezing 8 billion parameters, LoRA injects two tiny matrices side-by-side, dramatically slashing the required training VRAM computational burden by 98%.
* **Masked Attention**: A mandatory structural configuration inside a Decoder enforcing causality. It mathematically blocks the model from executing any attention logic targeting tokens located positioned *after* the current token.
* **Mixture of Experts (MoE)**: A neural block containing several 'expert' sub-networks (like 8 separate FFNs). A routing gate decides exactly which two experts activate for each specific input vector, preserving massive inference speed.
* **Multi-Head Attention**: Processing multiple self-attention operations concurrently in physically separate lower-dimensional sub-spaces immediately before concatenating them. Allows parsing extreme grammatical complexity without blurring the semantic signals.
* **Next-Token Prediction**: The incredibly simple underlying foundational logic objective utilized to pre-train almost all massive modern generative language models across Trillions of raw internet text documents.
* **NLP (Natural Language Processing)**: The massive overarching subfield of AI concerned entirely with programming algorithms to process, understand, analyze, and generate human linguistics and conversational grammar.
* **Overfitting**: A critical mathematical failure where a network memorizes the exact specific noise artifacts in the training dataset perfectly, completely destroying its capacity to generalize against unseen validation data.
* **Parameter**: The actual structural weights and biases contained natively deeply inside a neural network structure. An 8B parameter model literally contains 8,000,000,000 decimal numbers in 3D multi-dimensional arrays.
* **Positional Encoding**: The critical vector matrix added to the foundational input embeddings explicitly providing the Attention mathematical permutation logic with structural information regarding the exact sequence order of the tokens.
* **Pre-training**: The primary initial phase of creating a Foundation model. Feeding Trillions of text tokens through thousands of GPUs for weeks consuming megawatts of power strictly performing unsupervised next-token prediction.
* **Prompt Engineering**: The process of empirically designing, testing, and optimizing the structural format of linguistic inputs injected into LLMs to extract specifically desired, accurate, and robust structural logic outputs.
* **Python**: The undisputed dominant syntactic language dominating the Machine Learning backend architecture globally. Used to interface natively with the massively optimized C++ PyTorch/TensorFlow backend structures.
* **Quantization**: The process of fundamentally converting the continuous mathematical precision of model weight sets from floating point 32/16-bit resolutions down to block-level 8-bit or 4-bit, sacrificing microscopic accuracy for massive VRAM deployment efficiency.
* **RAG (Retrieval-Augmented Generation)**: The absolute standard deployed architecture for enterprise applications. Preventing LLM hallucinations by intercepting the user query, searching an external enterprise vector database for facts, and injecting those raw facts into the LLM context prior to generation.
* **Recurrent Neural Network (RNN)**: The legacy sequential architecture predating Transformers. Structurally processes tokens linearly one-by-one utilizing a continuous expanding hidden state. Massively vulnerable to extreme 'vanishing gradient' failure over large context windows.
* **ReLU (Rectified Linear Unit)**: A critically vital, simple activation formula: $f(x) = max(0, x)$. It structurally forces any massive negative outputs directly to 0.0, injecting mandatory non-linearity into massive chained matrix multiplications.
* **Representation Learning**: A set of structural ML techniques allowing underlying systems to automatically discover the raw distinct structural features or representations natively required for complex classification natively from raw data blocks.
* **Residual Connection (Skip Connection)**: Crucial architectural bypass lines transmitting original input vectors structurally directly deeply around massive network layers, instantly adding them to the final outputs. These bypass physical lines solve the vanishing gradient apocalypse inside massively deep networks.
* **RLHF (Reinforcement Learning from Human Feedback)**: The specific complex post-training fine-tuning pipeline rendering GPT-4 capable of human dialogue. Involves executing a massive secondary reward-model mathematically scoring generating responses purely based on expensive curated human feedback rankings.
* **RoPE (Rotary Position Embedding)**: A 2021 revolutionary positional encoding standard deployed across Llama and Mistral sequences. It natively rotates the grammatical Query/Key vectors dimensionally in a complex physical subspace mapping exact relative distance rather than simple sequence addition.
* **Self-Attention**: The singular foundational mechanism anchoring the Transformer legacy. A mathematical mechanism physically relating massive disparate elements uniquely across entirely distinct sequences against one another directly to aggregate rich integrated semantic grammatical meaning.
* **Softmax Function**: A vital structural normalization formula algorithm physically scaling an array composed of massive unnormalized numerical sequences entirely to fit bounded logically explicitly between exactly 0.0 and 1.0, rendering them perfectly functional as a standard statistical probability distribution structure.
* **SwiGLU**: A highly advanced 2025 non-linear dynamic activation block structurally deploying Swish gating pipelines heavily substituting standard ReLU gates extensively inside modern elite model matrices like Meta's Llama sequences, extracting elite processing dynamics.
* **Temperature (T)**: The primary physical decoding mathematical scalar parameter bounding generative randomness outputs. Approaching $T=0.0$ forces rigid, repetitive deterministic absolute certainty matrices, whereas raising $T=1.0+$ enforces wild erratic linguistic distributional variance sequences.
* **Tensor**: The massive core architectural fundamental building block powering Deep Learning ecosystems mapping identical arrays strictly scaling multi-dimensional numeric matrices structurally generalizing basic matrix algebra natively processing GPU matrix logic flows.
* **Token**: The fundamental structural sub-unit blocks consumed iteratively deeply inside LLM architectures. Generally representing fractional linguistic characters parsing out uniquely exactly approximately matching mathematically structural word matrices equivalent effectively equating 4 sequential raw alphabetical English letters.
* **Transformer**: The 2017 undisputed architectural miracle sequence dominating the fundamental ecosystem of Artificial Intelligence generation natively entirely substituting standard Recurrent pipelines comprehensively utilizing massive Parallel Sparse Attention structural distributions.
* **Underfitting**: The diametric inverse computational failure dynamically destroying machine sequence accuracy natively emerging when mathematical network parameters severely lack complex deep architectural capacity parameters distinctly mapping massive underlying geometric functional relationship flows.
* **Vector Database**: An explicitly specialized indexing enterprise deployment architectural database standard structurally deployed comprehensively globally storing extreme dense multi-dimensional numeric vectors natively supporting ultra-rapid K-Nearest-Neighbor cosine search queries anchoring core RAG sequence systems.
* **Weights**: The incredibly precise decimal numbers stored iteratively physically comprising neural networks natively adjusting mathematically via backpropagation gradients structurally driving complex non-linear sequence generation logic models perfectly.
* **Zero-Shot Learning**: The massive foundational AI ecosystem paradigm natively enabling explicit model completion capabilities operating distinctly successfully parsing tasks functionally completely utterly unseen across the generative pre-training massive pipeline structure.
- Without a harness, you **can't compare** prompts, models, retrieval configs, or costs.
Evaluate, benchmark, and regression-test AI/LLM systems. Covers evaluation framework design, benchmark creation, human evaluation protocols, automated evaluation (LLM-as-judge), regression testing, statistical significance, and continuous evaluation pipelines.
<img width="1388" height="298" alt="full_diagram" src="https://github.com/user-attachments/assets/12a2371b-8be2-4219-9b48-90503eb43c69" />
A list of all public EEG-datasets. This list of EEG-resources is not exhaustive. If you find something new, or have explored any unfiltered link in depth, please update the repository.