NeMo Curator: GPU-Accelerated LLM Training Data Curation

{ "title": "NeMo Curator Reference Guide", "body": "# NeMo Curator Reference Guide\n\n## Overview\n\nNeMo Curator is NVIDIA's GPU-accelerated toolkit for curating high-quality LLM training data. It provides fast deduplication, heuristic and classifier-based filtering, PII redaction, and multi-modal curation for text, images, video, and audio. The toolkit is designed to scale across multi-GPU clusters using Dask CUDA and Ray.\n\n### When to Use NeMo Curator\n\n- Preparing LLM training data from web scrapes (e.g., Common Crawl).\n- Needing fast deduplication (16x faster than CPU).\n- Curating multi-modal datasets (text, images, video, audio).\n- Filtering low-quality or toxic content.\n- Scaling data processing across a GPU cluster.\n\nUse alternatives such as datatrove, dolma, or Ray Data for CPU-based or general ML data processing without a curation focus.\n\n### Key Capabilities\n\n- 16x faster fuzzy deduplication (8TB RedPajama v2 benchmark).\n- 40% lower TCO vs CPU alternatives.\n- Near-linear scaling across GPU nodes.\n- 30+ heuristic quality filters (word count, repeated lines, URL ratio, non-alphanumeric, etc.).\n- Exact deduplication.\n- Fuzzy deduplication via MinHash + LSH.\n- Semantic deduplication via embedding-based cosine similarity.\n- PII redaction (EMAIL_ADDRESS, PHONE_NUMBER, PERSON, LOCATION) with replace or redact action.\n- Classifier filtering (e.g., quality classifier using DeBERTa).\n- Multi-modal curation: image (aesthetic scoring, NSFW detection, CLIP embeddings), video (scene detection, clip extraction, InternVideo2 embeddings), audio (ASR transcription, WER filtering, duration filtering).\n- Language identification filtering.\n- NSFW classification.\n- Distributed processing with multi-GPU clusters (Dask CUDA).\n- Input formats: Parquet, JSONL, CSV; Output: Parquet (recommended), JSONL; WebDataset TAR archives for multi-modal.\n\n## Installation\n\nNeMo Curator 1.x installs with uv. Extras use hyphens (PyPI-normalized): text-cuda12 / text-cpu (and image/video/audio/math variants), or all.\n\n\n# NeMo Curator 1.x installs with uv. Extras use hyphens (PyPI-normalized):\n# text-cuda12 / text-cpu (and image/video/audio/math variants), or `all`.\n\n# Text curation (CUDA 12)\nuv pip install \"nemo-curator[text-cuda12]\"\n\n# All modalities\nuv pip install \"nemo-curator[all]\"\n\n# CPU-only text (slower)\nuv pip install \"nemo-curator[text-cpu]\"\n\n\n### Prerequisites\n\n- Python environment with uv installed.\n- CUDA 12 (for GPU-accelerated extras) or CPU-only setup.\n- Sufficient GPU memory (e.g., A100) for large-scale deduplication.\n- Dask and Ray installed (for distributed processing).\n- Access to HuggingFace models (for semantic deduplication, classifiers).\n- AWS credentials (if using S3 for data).\n- For multi-modal: appropriate libraries for image/video/audio processing (installed via extras).\n- For video: scene detection requires threshold tuning.\n- For audio: ASR model requires GPU for inference.\n\n## Architecture\n\nNeMo Curator 1.x uses a Ray-based pipeline and stage architecture. Compose ProcessingStage objects into a Pipeline and run with an executor. The old 0.x API (DocumentDataset + modules) is deprecated.\n\n### Basic Text Curation Pipeline (1.x)\n\npython\nfrom nemo_curator.pipeline import Pipeline\nfrom nemo_curator.stages.base import ProcessingStage\nfrom nemo_curator.stages.resources import Resources\nfrom nemo_curator.backends.xenna import XennaExecutor\nfrom nemo_curator.core.client import RayClient\n\n# 1. Define/compose stages (load -> filter -> dedupe -> classify -> write).\n# Each stage declares its own Resources (CPU cores, GPU memory, replicas).\npipeline = Pipeline(name=\"curation\", stages=[...])\n\n# 2. Run it with an executor (Ray-backed).\nclient = RayClient()\nclient.start()\npipeline.run(XennaExecutor())\nclient.stop()\n\n\n## Text Curation\n\n### Quality Filtering\n\nApply 30+ heuristic filters using ScoreFilter and Modify.\n\npython\nfrom nemo_curator.filters import (\n WordCountFilter,\n RepeatedLinesFilter,\n UrlRatioFilter,\n NonAlphaNumericFilter\n)\n\n# Apply 30+ heuristic filters\nfrom nemo_curator import ScoreFilter\n\n# Word count filter\ndataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))\n\n# Remove repetitive content\ndataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))\n\n# URL ratio filter\ndataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))\n\n\n### Deduplication\n\n#### Exact Deduplication\n\npython\nfrom nemo_curator.modules import ExactDuplicates\n\n# Remove exact duplicates\ndeduped = ExactDuplicates(id_field=\"id\", text_field=\"text\")(dataset)\n\n\n#### Fuzzy Deduplication (MinHash + LSH)\n\npython\nfrom nemo_curator.modules import FuzzyDuplicates\n\n# MinHash + LSH deduplication\nfuzzy_dedup = FuzzyDuplicates(\n id_field=\"id\",\n text_field=\"text\",\n num_hashes=260, # MinHash parameters\n num_buckets=20,\n hash_method=\"md5\"\n)\n\ndeduped = fuzzy_dedup(dataset)\n\n\n#### Semantic Deduplication\n\npython\nfrom nemo_curator.modules import SemanticDuplicates\n\n# Embedding-based deduplication\nsemantic_dedup = SemanticDuplicates(\n id_field=\"id\",\n text_field=\"text\",\n embedding_model=\"sentence-transformers/all-MiniLM-L6-v2\",\n threshold=0.8 # Cosine similarity threshold\n)\n\ndeduped = semantic_dedup(dataset)\n\n\n### PII Redaction\n\npython\nfrom nemo_curator.modules import Modify\nfrom nemo_curator.modifiers import PIIRedactor\n\n# Redact personally identifiable information\npii_redactor = PIIRedactor(\n supported_entities=[\"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"PERSON\", \"LOCATION\"],\n anonymize_action=\"replace\" # or \"redact\"\n)\n\nredacted = Modify(pii_redactor)(dataset)\n\n\n### Classifier Filtering\n\npython\nfrom nemo_curator.classifiers import QualityClassifier\n\n# Quality classification\nquality_clf = QualityClassifier(\n model_path=\"nvidia/quality-classifier-deberta\",\n batch_size=256,\n device=\"cuda\"\n)\n\n# Filter low-quality documents\nhigh_quality = dataset.filter(lambda doc: quality_clf(doc[\"text\"]) > 0.5)\n\n\n### Web Scrape Curation (Common Crawl)\n\nA complete pipeline for Common Crawl data using nemo_curator.modules.*.\n\npython\nfrom nemo_curator import ScoreFilter, Modify\nfrom nemo_curator.filters import *\nfrom nemo_curator.modules import *\nfrom nemo_curator.datasets import DocumentDataset\n\n# Load Common Crawl data\ndataset = DocumentDataset.read_parquet(\"common_crawl/*.parquet\")\n\n# Pipeline\npipeline = [\n # 1. Quality filtering\n WordCountFilter(min_words=100, max_words=50000),\n RepeatedLinesFilter(max_repeated_line_fraction=0.2),\n SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),\n UrlRatioFilter(max_url_ratio=0.3),\n\n # 2. Language filtering\n LanguageIdentificationFilter(target_languages=[\"en\"]),\n\n # 3. Deduplication\n ExactDuplicates(id_field=\"id\", text_field=\"text\"),\n FuzzyDuplicates(id_field=\"id\", text_field=\"text\", num_hashes=260),\n\n # 4. PII redaction\n PIIRedactor(),\n\n # 5. NSFW filtering\n NSFWClassifier(threshold=0.8)\n]\n\n# Execute\nfor stage in pipeline:\n dataset = stage(dataset)\n\n# Save\ndataset.to_parquet(\"curated_common_crawl/\")\n\n\n## Multi-GPU and Distributed Processing\n\n### Multi-GPU Scaling\n\npython\nfrom nemo_curator import get_client\nimport dask_cuda\n\n# Initialize GPU cluster\nclient = get_client(cluster_type=\"gpu\", n_workers=8)\n\n# Process with 8 GPUs\ndeduped = FuzzyDuplicates(...)(dataset)\n\n\n### Distributed Processing with LocalCUDACluster\n\npython\nfrom nemo_curator import get_client\nfrom dask_cuda import LocalCUDACluster\n\n# Multi-GPU cluster\ncluster = LocalCUDACluster(n_workers=8)\nclient = get_client(cluster=cluster)\n\n# Process large dataset\ndataset = DocumentDataset.read_parquet(\"s3://large_dataset/*.parquet\")\ndeduped = FuzzyDuplicates(...)(dataset)\n\n# Cleanup\nclient.close()\ncluster.close()\n\n\n## Multi-Modal Curation\n\n### Image Curation\n\npython\nfrom nemo_curator.image import (\n AestheticFilter,\n NSFWFilter,\n CLIPEmbedder\n)\n\n# Aesthetic scoring\naesthetic_filter = AestheticFilter(threshold=5.0)\nfiltered_images = aesthetic_filter(image_dataset)\n\n# NSFW detection\nnsfw_filter = NSFWFilter(threshold=0.9)\nsafe_images = nsfw_filter(filtered_images)\n\n# Generate CLIP embeddings\nclip_embedder = CLIPEmbedder(model=\"openai/clip-vit-base-patch32\")\nimage_embeddings = clip_embedder(safe_images)\n\n\n### Video Curation\n\npython\nfrom nemo_curator.video import (\n SceneDetector,\n ClipExtractor,\n InternVideo2Embedder\n)\n\n# Detect scenes\nscene_detector = SceneDetector(threshold=27.0)\nscenes = scene_detector(video_dataset)\n\n# Extract clips\nclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)\nclips = clip_extractor(scenes)\n\n# Generate embeddings\nvideo_embedder = InternVideo2Embedder()\nvideo_embeddings = video_embedder(clips)\n\n\n### Audio Curation\n\npython\nfrom nemo_curator.audio import (\n ASRInference,\n WERFilter,\n DurationFilter\n)\n\n# ASR transcription\nasr = ASRInference(model=\"nvidia/stt_en_fastconformer_hybrid_large_pc\")\ntranscribed = asr(audio_dataset)\n\n# Filter by WER (word error rate)\nwer_filter = WERFilter(max_wer=0.3)\nhigh_quality_audio = wer_filter(transcribed)\n\n# Duration filtering\nduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)\nfiltered_audio = duration_filter(high_quality_audio)\n\n\n## Parameters Reference\n\n| Parameter | Meaning | Required |\n|---|---|---|\n| WordCountFilter min_words | Minimum word count for a document to pass filter | Yes |\n| WordCountFilter max_words | Maximum word count for a document to pass filter | Yes |\n| RepeatedLinesFilter max_repeated_line_fraction | Maximum fraction of repeated lines allowed | Yes |\n| UrlRatioFilter max_url_ratio | Maximum ratio of URLs to total content | Yes |\n| FuzzyDuplicates num_hashes | Number of MinHash hashes (default 260) | Yes |\n| FuzzyDuplicates num_buckets | Number of LSH buckets (default 20) | Yes |\n| FuzzyDuplicates hash_method | Hash method for MinHash (e.g., 'md5') | Yes |\n| SemanticDuplicates embedding_model | HuggingFace model name for embeddings | Yes |\n| SemanticDuplicates threshold | Cosine similarity threshold for duplicate detection (default 0.8) | Yes |\n| PIIRedactor supported_entities | List of PII entity types to redact | Yes |\n| PIIRedactor anonymize_action | Action: 'replace' or 'redact' | Yes |\n| QualityClassifier model_path | Path or HuggingFace model ID for quality classifier | Yes |\n| QualityClassifier batch_size | Batch size for inference (default 256) | Yes |\n| QualityClassifier device | Device for inference ('cuda' or 'cpu') | Yes |\n| AestheticFilter threshold | Minimum aesthetic score to keep image (default 5.0) | Yes |\n| NSFWFilter threshold | Maximum NSFW score allowed (default 0.9) | Yes |\n| CLIPEmbedder model | CLIP model name for embeddings | Yes |\n| SceneDetector threshold | Threshold for scene detection (default 27.0) | Yes |\n| ClipExtractor min_duration | Minimum clip duration in seconds (default 2.0) | Yes |\n| ClipExtractor max_duration | Maximum clip duration in seconds (default 10.0) | Yes |\n| ASRInference model | NVIDIA STT model name for transcription | Yes |\n| WERFilter max_wer | Maximum word error rate allowed (default 0.3) | Yes |\n| DurationFilter min_duration | Minimum audio duration in seconds (default 1.0) | Yes |\n| DurationFilter max_duration | Maximum audio duration in seconds (default 30.0) | Yes |\n| LanguageIdentificationFilter target_languages | List of target language codes (e.g., ['en']) | Yes |\n| NSFWClassifier threshold | Threshold for NSFW classification (default 0.8) | Yes |\n| SymbolToWordRatioFilter max_symbol_to_word_ratio | Maximum ratio of symbols to words | Yes |\n| get_client cluster_type | Type of cluster ('gpu') | Yes |\n| get_client n_workers | Number of GPU workers | Yes |\n| LocalCUDACluster n_workers | Number of workers for local multi-GPU cluster | Yes |\n\n## Performance and Cost\n\nNeMo Curator achieves a 40% lower TCO vs CPU alternatives. For an 8TB dataset, GPU-based curation saves 89% ($3,828) vs CPU. The cost breakdown shows GPU costs of $32.77 per hour versus CPU costs of $65.54 per hour, with total GPU costs of $491.55 and total CPU costs of $4320, resulting in a saving of $3828. The fuzzy deduplication benchmark on 8TB RedPajama v2 shows a 16x speedup using 8x A100 GPUs compared to 256 CPU cores. Exact deduplication on 1TB achieves a 16x speedup with 4x A100 vs 64 CPU cores. Quality filtering on 100GB achieves a 10x speedup with 2x A100 vs 32 CPU cores. The cost comparison assumes AWS pricing; actual costs may differ. Performance benchmarks are for specific hardware (A100 GPUs) and may vary.\n\n## Constraints and Caveats\n\n- NeMo Curator 1.x is a major rewrite; the old 0.x API (DocumentDataset + modules) is deprecated.\n- 1.x uses a Ray-based pipeline/stage architecture; compose ProcessingStage objects into a Pipeline and run with an executor.\n- The examples in this document are 0.x-style conceptual snippets; for runnable 1.x code, follow the current quickstart and modality guide.\n- Installation uses uv; extras use hyphens (e.g., text-cuda12, text-cpu, all).\n- CPU-only text curation is slower than GPU-accelerated.\n- Fuzzy deduplication parameters (num_hashes, num_buckets, hash_method) affect accuracy and performance.\n- Semantic deduplication requires an embedding model and threshold tuning.\n- PII redaction may not catch all entity types; supported entities are limited to those specified.\n- Classifier filtering requires a trained model (e.g., DeBERTa).\n- Multi-modal curation modules (image, video, audio) are separate extras.\n- Distributed processing requires a Dask CUDA cluster and appropriate GPU resources.\n- Input data must be in supported formats (Parquet, JSONL, CSV).\n- Output is recommended as Parquet.\n- WebDataset TAR archives are supported for multi-modal data.\n- The document's examples are illustrative; exact imports and APIs may differ in 1.x.\n\n## Failure Modes\n\n- Using 0.x API calls in 1.x will fail due to API changes.\n- Incorrect installation (e.g., missing extras) leads to import errors.\n- Insufficient GPU memory or workers for large datasets.\n- Fuzzy deduplication with wrong hash parameters may miss duplicates or produce false positives.\n- Semantic deduplication threshold too low may remove non-duplicates; too high may miss duplicates.\n- PII redactor may not detect all PII instances, leading to data leakage.\n- Classifier model may not generalize to all data domains.\n- Multi-modal curation may fail if data format is unsupported.\n- Distributed processing may hang or crash if cluster resources are misconfigured.\n- Reading from S3 without proper credentials or permissions will fail.\n- Writing output to a path without write permissions will fail.\n- Language identification filter may misclassify documents with mixed languages.\n- NSFW classifier may have false positives/negatives depending on threshold." }