Datatrove

Datatrove is an open-source library from Hugging Face designed for large-scale text data processing, filtering, and deduplication. It provides a set of prebuilt commonly used processing blocks and a framework for easily adding custom functionality. Pipelines are platform-agnostic, running locally, on Slurm clusters, or with Ray. Datatrove features low memory usage and a multiple-step design, making it ideal for workloads such as preparing training data for large language models (LLMs). It supports multiple file systems via fsspec and can read formats like WARC, ARC, WET, Parquet, and Arrow. The library includes executors for local, Slurm, Ray, and job-based execution. Key capabilities include text extraction, filtering, tokenization, minhash deduplication, sentence deduplication, synthetic data generation, progress monitoring, and benchmarking.