The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
FreeDecanting the Web for the Finest Text Data at Scale
FreeFree tier
About The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
FineWeb is a 15-trillion token pretraining dataset derived from 96 Common Crawl snapshots. It is designed to produce better-performing large language models (LLMs) compared to other open pretraining datasets. The dataset is accompanied by FineWeb-Edu, a 1.3-trillion token subset of educational text filtered from FineWeb, which significantly improves performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC. The paper provides detailed documentation and ablation studies of the design choices for data curation, including deduplication and filtering strategies. The data curation codebase and all models trained during ablation experiments are publicly released.
Key Features
15-trillion token dataset derived from 96 Common Crawl snapshots
FineWeb-Edu: 1.3-trillion token educational text subset
Detailed documentation and ablation of design choices
Public release of data curation codebase and pretrained models
Outperforms other open pretraining datasets for LLMs
Enhances performance on knowledge and reasoning benchmarks (MMLU, ARC)
Pros & Cons
Pros
- Large scale with 15 trillion tokens from diverse web sources
- Educational subset dramatically improves benchmark results
- Transparent documentation of all curation steps
- Openly released code and models for reproducibility
- Derived from freely available Common Crawl data
Cons
- Dataset is derived from Common Crawl, which may contain noisy or low-quality text
- Requires significant computational resources to process and use effectively
Best For
Pretraining large language modelsEducational text filtering and analysisResearch on data curation and deduplication strategiesReproducible LLM training experiments
FAQ
What is FineWeb?
FineWeb is a 15-trillion token pretraining dataset derived from 96 Common Crawl snapshots, designed to produce better-performing LLMs than other open pretraining datasets.
What is FineWeb-Edu?
FineWeb-Edu is a 1.3-trillion token collection of educational text filtered from FineWeb, which yields dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC.
Is the dataset and code publicly available?
Yes, the data curation codebase and all models trained during ablation experiments are publicly released alongside the dataset.