Text Retrieval Benchmark
Runs a text retrieval benchmark using BEIR datasets, measuring recall, precision, NDCG, and MRR without LLM generation.
What this file does
Runs a text retrieval benchmark using BEIR datasets, measuring recall, precision, NDCG, and MRR without LLM generation.
When to use it
- Evaluate retrieval-only pipeline performance
- Compare BM25 against dense retrievers
- Validate retrieval metrics before adding generation
- Quickly test BEIR SciFact or NFCorpus datasets
Assumes this stack
Text Retrieval Benchmark
Run a text retrieval benchmark without generation (no LLM required).
Download Dataset
autorag-research data restore beir scifact_openai-small
Downloads BEIR SciFact (300 queries, 5,183 documents).
Create Experiment Config
# configs/experiment.yaml
db_name: beir_scifact_test_openai_small
pipelines:
retrieval:
- bm25
generation: []
metrics:
retrieval:
- recall
- precision
- ndcg
- mrr
generation: []
Run
autorag-research run --config-name=experiment
Expected Output
Pipeline: bm25
Recall@10: 0.847
Precision@10: 0.085
NDCG@10: 0.712
MRR@10: 0.634
Recommended Datasets
| Dataset | Queries | Documents | Best For |
|---|---|---|---|
| BEIR SciFact | 300 | 5,183 | Scientific claims |
| BEIR NFCorpus | 323 | 3,633 | Biomedical |
| MTEB | varies | varies | General text |
See Text Datasets for all options.
Recommended Metrics
| Metric | Measures |
|---|---|
| Recall@k | Coverage of ground truth |
| NDCG@k | Ranking quality |
| MRR | First relevant position |
See Retrieval Metrics for details.
Next
- Text RAG - Add generation
- Custom Pipeline - Implement your algorithm
What's inside
5 sections: dataset download, config YAML, run command, expected output, recommended datasets and metrics
Change this for your project
- Replace
beir scifact_openai-smallwith your chosen BEIR dataset - Replace
db_name: beir_scifact_test_openai_smallwith your own database name - Replace
bm25under pipelines.retrieval with your retriever name
Where it goes
Keep alongside your test suite. Used to define and score model evaluations.
Related Documents
IATA BCBP Standard Compliance
Documents which IATA BCBP fields and barcode formats a Swift library implements, including Version 8 gender code support.
Evaluating AI Agent Systems: Metrics, Benchmarks, and Quality Assurance (2024-2026)
Surveys 2024-2026 metrics, benchmarks, and monitoring tools for evaluating AI agent systems, with recommendations for a self-improving coding agent.
Voice AI Leaderboards, Benchmarks, and Evaluation Gaps (Jan 2025 -- Feb 2026)
Surveys 20+ voice AI benchmarks from Jan 2025, Feb 2026, identifies evaluation gaps, and provides leaderboard data for STT, TTS, and end-to-end voice agents.
LLM Evaluation — Deep Dive
Covers why LLM evaluation is hard, benchmark taxonomy, LLM-as-judge methodology, contamination, robustness, and designing an eval suite for a product.