nemo-curator

name nemo-curator description Curate LLM training data: dedupe, filter, PII redaction. version 1.0.1 author Orchestra Research license MIT dependencies ["nemo-curator","cudf","dask","rapids"] platforms ["linux","macos"] metadata {"hermes":{"tags":["Data Processing","NeMo Curator","Data Curation","GPU Acceleration","Deduplication","Quality Filtering","NVIDIA","RAPIDS","PII Redaction","Multimodal","LLM Training Data"]}} NeMo Curator - GPU-Accelerated Data Curation NVIDIA's toolkit for preparing high-quality training data for LLMs. When to use NeMo Curator Use NeMo Curator when: Preparing LLM training data from web scrapes (Common Crawl) Need fast deduplication (16× faster than CPU) Curating multi-modal datasets (text, images, video, audio) Filtering low-quality or toxic content Scaling data processing across GPU cluster Performance : 16× faster fuzzy deduplication (8TB RedPajama v2) 40% lower TCO vs CPU alternatives Near-linear scaling across GPU nodes Use alternatives instead : datatrove : CPU-based, open-source data processing dolma : Allen AI's data toolkit Ray Data : General ML data processing (no curation focus) Quick start Installation # NeMo Curator 1.x installs with uv. Extras use hyphens (PyPI-normalized): # text-cuda12 / text-cpu (and image/video/audio/math variants), or `all`. # Text curation (CUDA 12) uv pip install "nemo-curator[text-cuda12]" # All modalities uv pip install "nemo-curator[all]" # CPU-only text (slower) uv pip install "nemo-curator[text-cpu]" Basic text curation pipeline Major version rewrite (1.x): NeMo Curator was rewritten around a Ray-based pipeline/stage architecture . The old DocumentDataset + nemo_curator.modules.* / ScoreFilter / Modify call-the-object-on-a-dataset API from 0.x is gone. In 1.x you compose ProcessingStage s into a Pipeline and run it with an executor. The exact stage/import surface differs per modality — treat the examples in this skill below as conceptual (0.x-style) and follow the current quickstart and text guide for the exact 1.x APIs rather than copying imports verbatim. Shape of a 1.x pipeline (from the upstream quickstart): from nemo_curator.pipeline import Pipeline from nemo_curator.stages.base import ProcessingStage from nemo_curator.stages.resources import Resources from nemo_curator.backends.xenna import XennaExecutor from nemo_curator.core.client import RayClient # 1. Define/compose stages (load -> filter -> dedupe -> classify -> write). # Each stage declares its own Resources (CPU cores, GPU memory, replicas). pipeline = Pipeline(name= "curation" , stages=[...]) # 2. Run it with an executor (Ray-backed). client = RayClient() client.start() pipeline.run(XennaExecutor()) client.stop() The 0.x-style snippets in the sections that follow illustrate the concepts (quality filtering, exact/fuzzy/semantic dedup, PII redaction, classifier filtering). For runnable 1.x code, map each concept onto the corresponding stage from the modality guide. Data curation pipeline Stage 1: Quality filtering from nemo_curator.filters import ( WordCountFilter, RepeatedLinesFilter, UrlRatioFilter, NonAlphaNumericFilter ) # Apply 30+ heuristic filters from nemo_curator import ScoreFilter # Word count filter dataset = dataset. filter (WordCountFilter(min_words= 50 , max_words= 100000 )) # Remove repetitive content dataset = dataset. filter (RepeatedLinesFilter(max_repeated_line_fraction= 0.3 )) # URL ratio filter dataset = dataset. filter (UrlRatioFilter(max_url_ratio= 0.2 )) Stage 2: Deduplication Exact deduplication : from nemo_curator.modules import ExactDuplicates # Remove exact duplicates deduped = ExactDuplicates(id_field= "id" , text_field= "text" )(dataset) Fuzzy deduplication (16× faster on GPU): from nemo_curator.modules import FuzzyDuplicates # MinHash + LSH deduplication fuzzy_dedup = FuzzyDuplicates( id_field= "id" , text_field= "text" , num_hashes= 260 , # MinHash parameters num_buckets= 20 , hash_method= "md5" ) deduped = fuzzy_dedup(dataset) Semantic deduplication : from nemo_curator.modules import SemanticDuplicates # Embedding-based deduplication semantic_dedup = SemanticDuplicates( id_field= "id" , text_field= "text" , embedding_model= "sentence-transformers/all-MiniLM-L6-v2" , threshold= 0.8 # Cosine similarity threshold ) deduped = semantic_dedup(dataset) Stage 3: PII redaction from nemo_curator.modules import Modify from nemo_curator.modifiers import PIIRedactor # Redact personally identifiable information pii_redactor = PIIRedactor( supported_entities=[ "EMAIL_ADDRESS" , "PHONE_NUMBER" , "PERSON" , "LOCATION" ], anonymize_action= "replace" # or "redact" ) redacted = Modify(pii_redactor)(dataset) Stage 4: Classifier filtering from nemo_curator.classifiers import QualityClassifier # Quality classification quality_clf = QualityClassifier( model_path= "nvidia/quality-classifier-deberta" , batch_size= 256 , device= "cuda" ) # Filter low-quality documents high_quality = dataset. filter ( lambda doc: quality_clf(doc[ "text" ]) > 0.5 ) GPU acceleration GPU vs CPU performance Operation CPU (16 cores) GPU (A100) Speedup Fuzzy dedup (8TB) 120 hours 7.5 hours 16× Exact dedup (1TB) 8 hours 0.5 hours 16× Quality filtering 2 hours 0.2 hours 10× Multi-GPU scaling from nemo_curator import get_client import dask_cuda # Initialize GPU cluster client = get_client(cluster_type= "gpu" , n_workers= 8 ) # Process with 8 GPUs deduped = FuzzyDuplicates(...)(dataset) Multi-modal curation Image curation from nemo_curator.image import ( AestheticFilter, NSFWFilter, CLIPEmbedder ) # Aesthetic scoring aesthetic_filter = AestheticFilter(threshold= 5.0 ) filtered_images = aesthetic_filter(image_dataset) # NSFW detection nsfw_filter = NSFWFilter(threshold= 0.9 ) safe_images = nsfw_filter(filtered_images) # Generate CLIP embeddings clip_embedder = CLIPEmbedder(model= "openai/clip-vit-base-patch32" ) image_embeddings = clip_embedder(safe_images) Video curation from nemo_curator.video import ( SceneDetector, ClipExtractor, InternVideo2Embedder ) # Detect scenes scene_detector = SceneDetector(threshold= 27.0 ) scenes = scene_detector(video_dataset) # Extract clips clip_extractor = ClipExtractor(min_duration= 2.0 , max_duration= 10.0 ) clips = clip_extractor(scenes) # Generate embeddings video_embedder = InternVideo2Embedder() video_embeddings = video_embedder(clips) Audio curation from nemo_curator.audio import ( ASRInference, WERFilter, DurationFilter ) # ASR transcription asr = ASRInference(model= "nvidia/stt_en_fastconformer_hybrid_large_pc" ) transcribed = asr(audio_dataset) # Filter by WER (word error rate) wer_filter = WERFilter(max_wer= 0.3 ) high_quality_audio = wer_filter(transcribed) # Duration filtering duration_filter = DurationFilter(min_duration= 1.0 , max_duration= 30.0 ) filtered_audio = duration_filter(high_quality_audio) Common patterns Web scrape curation (Common Crawl) from nemo_curator import ScoreFilter, Modify from nemo_curator.filters import * from nemo_curator.modules import * from nemo_curator.datasets import DocumentDataset # Load Common Crawl data dataset = DocumentDataset.read_parquet( "common_crawl/*.parquet" ) # Pipeline pipeline = [ # 1. Quality filtering WordCountFilter(min_words= 100 , max_words= 50000 ), RepeatedLinesFilter(max_repeated_line_fraction= 0.2 ), SymbolToWordRatioFilter(max_symbol_to_word_ratio= 0.3 ), UrlRatioFilter(max_url_ratio= 0.3 ), # 2. Language filtering LanguageIdentificationFilter(target_languages=[ "en" ]), # 3. Deduplication ExactDuplicates(id_field= "id" , text_field= "text" ), FuzzyDuplicates(id_field= "id" , text_field= "text" , num_hashes= 260 ), # 4. PII redaction PIIRedactor(), # 5. NSFW filtering NSFWClassifier(threshold= 0.8 ) ] # Execute for stage in pipeline: dataset = stage(dataset) # Save dataset.to_parquet( "curated_common_crawl/" ) Distributed processing from nemo_curator import get_client from dask_cuda import LocalCUDACluster # Multi-GPU cluster cluster = LocalCUDACluster(n_workers= 8 ) client = get_client(cluster=cluster) # Process large dataset dataset = DocumentDataset.read_parquet( "s3://large_dataset/*.parquet" ) deduped = FuzzyDuplicates(...)(dataset) # Cleanup client.close() cluster.close() Performance benchmarks Fuzzy deduplication (8TB RedPajama v2) CPU (256 cores) : 120 hours GPU (8× A100) : 7.5 hours Speedup : 16× Exact deduplication (1TB) CPU (64 cores) : 8 hours GPU (4× A100) : 0.5 hours Speedup : 16× Quality filtering (100GB) CPU (32 cores) : 2 hours GPU (2× A100) : 0.2 hours Speedup : 10× Cost comparison CPU-based curation (AWS c5.18xlarge × 10): Cost: $3.60/hour × 10 = $36/hour Time for 8TB: 120 hours Total : $4,320 GPU-based curation (AWS p4d.24xlarge × 2): Cost: $32.77/hour × 2 = $65.54/hour Time for 8TB: 7.5 hours Total : $491.55 Savings : 89% reduction ($3,828 saved) Supported data formats Input : Parquet, JSONL, CSV Output : Parquet (recommended), JSONL WebDataset : TAR archives for multi-modal Use cases Production deployments : NVIDIA used NeMo Curator to prepare Nemotron-4 training data Open-source datasets curated: RedPajama v2, The Pile References Filtering Guide - 30+ quality filters, heuristics Deduplication Guide - Exact, fuzzy, semantic methods Resources GitHub : https://github.com/NVIDIA-NeMo/Curator Docs : https://docs.nvidia.com/nemo/curator/latest/ Version : 1.2.0 (1.x is a Ray-based pipeline rewrite — see the quickstart before copying 0.x snippets) License : Apache 2.0