better-dsh 0.2.5 → 0.3.0-c

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/mlops/mlops-nemo-curator.md

@@ -1,415 +0,0 @@
----
-title: "Nemo Curator — Curate LLM training data: dedupe, filter, PII redaction"
-sidebar_label: "Nemo Curator"
-description: "Curate LLM training data: dedupe, filter, PII redaction"
----
-
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-
-# Nemo Curator
-
-Curate LLM training data: dedupe, filter, PII redaction.
-
-## Skill metadata
-
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/mlops/nemo-curator` |
-| Path | `optional-skills/mlops\nemo-curator` |
-| Version | `1.0.1` |
-| Author | Orchestra Research |
-| License | MIT |
-| Dependencies | `nemo-curator`, `cudf`, `dask`, `rapids` |
-| Platforms | linux, macos |
-| Tags | `Data Processing`, `NeMo Curator`, `Data Curation`, `GPU Acceleration`, `Deduplication`, `Quality Filtering`, `NVIDIA`, `RAPIDS`, `PII Redaction`, `Multimodal`, `LLM Training Data` |
-
-## Reference: full SKILL.md
-
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-
-# NeMo Curator - GPU-Accelerated Data Curation
-
-NVIDIA's toolkit for preparing high-quality training data for LLMs.
-
-## When to use NeMo Curator
-
-**Use NeMo Curator when:**
-- Preparing LLM training data from web scrapes (Common Crawl)
-- Need fast deduplication (16× faster than CPU)
-- Curating multi-modal datasets (text, images, video, audio)
-- Filtering low-quality or toxic content
-- Scaling data processing across GPU cluster
-
-**Performance**:
-- **16× faster** fuzzy deduplication (8TB RedPajama v2)
-- **40% lower TCO** vs CPU alternatives
-- **Near-linear scaling** across GPU nodes
-
-**Use alternatives instead**:
-- **datatrove**: CPU-based, open-source data processing
-- **dolma**: Allen AI's data toolkit
-- **Ray Data**: General ML data processing (no curation focus)
-
-## Quick start
-
-### Installation
-
-```bash
-# NeMo Curator 1.x installs with uv. Extras use hyphens (PyPI-normalized):
-#   text-cuda12 / text-cpu (and image/video/audio/math variants), or `all`.
-
-# Text curation (CUDA 12)
-uv pip install "nemo-curator[text-cuda12]"
-
-# All modalities
-uv pip install "nemo-curator[all]"
-
-# CPU-only text (slower)
-uv pip install "nemo-curator[text-cpu]"
-```
-
-### Basic text curation pipeline
-
-> **Major version rewrite (1.x):** NeMo Curator was rewritten around a **Ray-based
-> pipeline/stage architecture**. The old `DocumentDataset` + `nemo_curator.modules.*` /
-> `ScoreFilter` / `Modify` call-the-object-on-a-dataset API from 0.x is gone. In 1.x you
-> compose `ProcessingStage`s into a `Pipeline` and run it with an executor. The exact
-> stage/import surface differs per modality — treat the examples in this skill below as
-> **conceptual** (0.x-style) and follow the current
-> [quickstart](https://github.com/NVIDIA-NeMo/Curator/blob/main/tutorials/quickstart.py)
-> and [text guide](https://docs.nvidia.com/nemo/curator/latest/get-started/text) for the
-> exact 1.x APIs rather than copying imports verbatim.
-
-Shape of a 1.x pipeline (from the upstream quickstart):
-
-```python
-from nemo_curator.pipeline import Pipeline
-from nemo_curator.stages.base import ProcessingStage
-from nemo_curator.stages.resources import Resources
-from nemo_curator.backends.xenna import XennaExecutor
-from nemo_curator.core.client import RayClient
-
-# 1. Define/compose stages (load -> filter -> dedupe -> classify -> write).
-#    Each stage declares its own Resources (CPU cores, GPU memory, replicas).
-pipeline = Pipeline(name="curation", stages=[...])
-
-# 2. Run it with an executor (Ray-backed).
-client = RayClient()
-client.start()
-pipeline.run(XennaExecutor())
-client.stop()
-```
-
-The 0.x-style snippets in the sections that follow illustrate the *concepts* (quality
-filtering, exact/fuzzy/semantic dedup, PII redaction, classifier filtering). For runnable
-1.x code, map each concept onto the corresponding stage from the modality guide.
-
-## Data curation pipeline
-
-### Stage 1: Quality filtering
-
-```python
-from nemo_curator.filters import (
-    WordCountFilter,
-    RepeatedLinesFilter,
-    UrlRatioFilter,
-    NonAlphaNumericFilter
-)
-
-# Apply 30+ heuristic filters
-from nemo_curator import ScoreFilter
-
-# Word count filter
-dataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))
-
-# Remove repetitive content
-dataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))
-
-# URL ratio filter
-dataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))
-```
-
-### Stage 2: Deduplication
-
-**Exact deduplication**:
-```python
-from nemo_curator.modules import ExactDuplicates
-
-# Remove exact duplicates
-deduped = ExactDuplicates(id_field="id", text_field="text")(dataset)
-```
-
-**Fuzzy deduplication** (16× faster on GPU):
-```python
-from nemo_curator.modules import FuzzyDuplicates
-
-# MinHash + LSH deduplication
-fuzzy_dedup = FuzzyDuplicates(
-    id_field="id",
-    text_field="text",
-    num_hashes=260,      # MinHash parameters
-    num_buckets=20,
-    hash_method="md5"
-)
-
-deduped = fuzzy_dedup(dataset)
-```
-
-**Semantic deduplication**:
-```python
-from nemo_curator.modules import SemanticDuplicates
-
-# Embedding-based deduplication
-semantic_dedup = SemanticDuplicates(
-    id_field="id",
-    text_field="text",
-    embedding_model="sentence-transformers/all-MiniLM-L6-v2",
-    threshold=0.8  # Cosine similarity threshold
-)
-
-deduped = semantic_dedup(dataset)
-```
-
-### Stage 3: PII redaction
-
-```python
-from nemo_curator.modules import Modify
-from nemo_curator.modifiers import PIIRedactor
-
-# Redact personally identifiable information
-pii_redactor = PIIRedactor(
-    supported_entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "PERSON", "LOCATION"],
-    anonymize_action="replace"  # or "redact"
-)
-
-redacted = Modify(pii_redactor)(dataset)
-```
-
-### Stage 4: Classifier filtering
-
-```python
-from nemo_curator.classifiers import QualityClassifier
-
-# Quality classification
-quality_clf = QualityClassifier(
-    model_path="nvidia/quality-classifier-deberta",
-    batch_size=256,
-    device="cuda"
-)
-
-# Filter low-quality documents
-high_quality = dataset.filter(lambda doc: quality_clf(doc["text"]) > 0.5)
-```
-
-## GPU acceleration
-
-### GPU vs CPU performance
-
-| Operation | CPU (16 cores) | GPU (A100) | Speedup |
-|-----------|----------------|------------|---------|
-| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× |
-| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× |
-| Quality filtering | 2 hours | 0.2 hours | 10× |
-
-### Multi-GPU scaling
-
-```python
-from nemo_curator import get_client
-import dask_cuda
-
-# Initialize GPU cluster
-client = get_client(cluster_type="gpu", n_workers=8)
-
-# Process with 8 GPUs
-deduped = FuzzyDuplicates(...)(dataset)
-```
-
-## Multi-modal curation
-
-### Image curation
-
-```python
-from nemo_curator.image import (
-    AestheticFilter,
-    NSFWFilter,
-    CLIPEmbedder
-)
-
-# Aesthetic scoring
-aesthetic_filter = AestheticFilter(threshold=5.0)
-filtered_images = aesthetic_filter(image_dataset)
-
-# NSFW detection
-nsfw_filter = NSFWFilter(threshold=0.9)
-safe_images = nsfw_filter(filtered_images)
-
-# Generate CLIP embeddings
-clip_embedder = CLIPEmbedder(model="openai/clip-vit-base-patch32")
-image_embeddings = clip_embedder(safe_images)
-```
-
-### Video curation
-
-```python
-from nemo_curator.video import (
-    SceneDetector,
-    ClipExtractor,
-    InternVideo2Embedder
-)
-
-# Detect scenes
-scene_detector = SceneDetector(threshold=27.0)
-scenes = scene_detector(video_dataset)
-
-# Extract clips
-clip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)
-clips = clip_extractor(scenes)
-
-# Generate embeddings
-video_embedder = InternVideo2Embedder()
-video_embeddings = video_embedder(clips)
-```
-
-### Audio curation
-
-```python
-from nemo_curator.audio import (
-    ASRInference,
-    WERFilter,
-    DurationFilter
-)
-
-# ASR transcription
-asr = ASRInference(model="nvidia/stt_en_fastconformer_hybrid_large_pc")
-transcribed = asr(audio_dataset)
-
-# Filter by WER (word error rate)
-wer_filter = WERFilter(max_wer=0.3)
-high_quality_audio = wer_filter(transcribed)
-
-# Duration filtering
-duration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)
-filtered_audio = duration_filter(high_quality_audio)
-```
-
-## Common patterns
-
-### Web scrape curation (Common Crawl)
-
-```python
-from nemo_curator import ScoreFilter, Modify
-from nemo_curator.filters import *
-from nemo_curator.modules import *
-from nemo_curator.datasets import DocumentDataset
-
-# Load Common Crawl data
-dataset = DocumentDataset.read_parquet("common_crawl/*.parquet")
-
-# Pipeline
-pipeline = [
-    # 1. Quality filtering
-    WordCountFilter(min_words=100, max_words=50000),
-    RepeatedLinesFilter(max_repeated_line_fraction=0.2),
-    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),
-    UrlRatioFilter(max_url_ratio=0.3),
-
-    # 2. Language filtering
-    LanguageIdentificationFilter(target_languages=["en"]),
-
-    # 3. Deduplication
-    ExactDuplicates(id_field="id", text_field="text"),
-    FuzzyDuplicates(id_field="id", text_field="text", num_hashes=260),
-
-    # 4. PII redaction
-    PIIRedactor(),
-
-    # 5. NSFW filtering
-    NSFWClassifier(threshold=0.8)
-]
-
-# Execute
-for stage in pipeline:
-    dataset = stage(dataset)
-
-# Save
-dataset.to_parquet("curated_common_crawl/")
-```
-
-### Distributed processing
-
-```python
-from nemo_curator import get_client
-from dask_cuda import LocalCUDACluster
-
-# Multi-GPU cluster
-cluster = LocalCUDACluster(n_workers=8)
-client = get_client(cluster=cluster)
-
-# Process large dataset
-dataset = DocumentDataset.read_parquet("s3://large_dataset/*.parquet")
-deduped = FuzzyDuplicates(...)(dataset)
-
-# Cleanup
-client.close()
-cluster.close()
-```
-
-## Performance benchmarks
-
-### Fuzzy deduplication (8TB RedPajama v2)
-
-- **CPU (256 cores)**: 120 hours
-- **GPU (8× A100)**: 7.5 hours
-- **Speedup**: 16×
-
-### Exact deduplication (1TB)
-
-- **CPU (64 cores)**: 8 hours
-- **GPU (4× A100)**: 0.5 hours
-- **Speedup**: 16×
-
-### Quality filtering (100GB)
-
-- **CPU (32 cores)**: 2 hours
-- **GPU (2× A100)**: 0.2 hours
-- **Speedup**: 10×
-
-## Cost comparison
-
-**CPU-based curation** (AWS c5.18xlarge × 10):
-- Cost: $3.60/hour × 10 = $36/hour
-- Time for 8TB: 120 hours
-- **Total**: $4,320
-
-**GPU-based curation** (AWS p4d.24xlarge × 2):
-- Cost: $32.77/hour × 2 = $65.54/hour
-- Time for 8TB: 7.5 hours
-- **Total**: $491.55
-
-**Savings**: 89% reduction ($3,828 saved)
-
-## Supported data formats
-
-- **Input**: Parquet, JSONL, CSV
-- **Output**: Parquet (recommended), JSONL
-- **WebDataset**: TAR archives for multi-modal
-
-## Use cases
-
-**Production deployments**:
-- NVIDIA used NeMo Curator to prepare Nemotron-4 training data
-- Open-source datasets curated: RedPajama v2, The Pile
-
-## References
-
-- **[Filtering Guide](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\nemo-curator/references/filtering.md)** - 30+ quality filters, heuristics
-- **[Deduplication Guide](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\nemo-curator/references/deduplication.md)** - Exact, fuzzy, semantic methods
-
-## Resources
-
-- **GitHub**: https://github.com/NVIDIA-NeMo/Curator
-- **Docs**: https://docs.nvidia.com/nemo/curator/latest/
-- **Version**: 1.2.0 (1.x is a Ray-based pipeline rewrite — see the quickstart before copying 0.x snippets)
-- **License**: Apache 2.0