@r1z4x/tezgah 1.0.0

package/skills/ai-research/08-distributed-training/megatron-core/references/production-examples.md

@@ -0,0 +1,474 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 08-distributed-training/megatron-core/references/production-examples.md; MIT (c) Orchestra Research -->
+# Production Examples
+
+Real-world production deployments using Megatron-Core and Megatron-LM.
+
+## Meta LLaMA 3
+
+### Overview
+- **Models**: 8B, 70B, 405B parameters
+- **Training Infrastructure**: Two 24,000 H100 GPU clusters
+- **Total Investment**: Massive scale, months of training
+- **Training Data**: 15 trillion tokens for 405B model
+- **Status**: Production deployed (llama.meta.com)
+
+### LLaMA 3.1 405B - Largest Open Model
+
+**Architecture**:
+```yaml
+Parameters: 405 billion
+Layers: 126
+Hidden size: 16384
+Attention heads: 128
+Query groups: 8 (GQA)
+FFN size: 53248
+Vocabulary: 128,256 tokens
+Max context: 128K tokens (supports up to)
+Position encoding: RoPE
+Activation: SwiGLU
+Normalization: RMSNorm
+```
+
+**Training Configuration**:
+```bash
+# 1024 H100 GPUs (128 nodes × 8 GPUs)
+Tensor Parallel (TP): 8     # Within node
+Pipeline Parallel (PP): 8    # Across nodes
+Context Parallel (CP): 2     # For long sequences
+Data Parallel (DP): 8        # Remaining dimension
+
+Total GPUs: 8 × 8 × 2 × 8 = 1024
+Effective batch size: 2048
+Micro-batch per GPU: 1
+Sequence length: 4096 tokens
+```
+
+**Performance Metrics**:
+- **Sustained throughput**: 400 TFlops/GPU
+- **MFU**: ~46% on H100
+- **Uptime**: 95%+ over months
+- **Efficiency improvement**: 3× vs LLaMA 2 training
+
+**Training Duration**:
+- 15 trillion tokens total
+- ~54 days on 16,384 H100 GPUs
+- Or ~6 months on 1,024 H100 GPUs
+
+**Key Optimizations Used**:
+```bash
+--use-mcore-models \
+--transformer-impl transformer_engine \
+--sequence-parallel \
+--context-parallel-size 2 \
+--use-distributed-optimizer \
+--overlap-grad-reduce \
+--overlap-param-gather \
+--use-flash-attn-v2 \
+--bf16
+```
+
+**Production Serving**:
+- Deployed on llama.meta.com
+- Available via API and download
+- Used in Meta products (Instagram, Facebook, WhatsApp)
+
+### LLaMA 3 70B
+
+**Training Configuration**:
+```bash
+# 64 H100 GPUs (8 nodes × 8 GPUs)
+TP=4, PP=4, CP=2, DP=2
+
+torchrun --nproc_per_node=8 --nnodes=8 pretrain_gpt.py \
+  --num-layers 80 \
+  --hidden-size 8192 \
+  --num-attention-heads 64 \
+  --num-query-groups 8 \
+  --seq-length 4096 \
+  --micro-batch-size 1 \
+  --global-batch-size 1024 \
+  --tensor-model-parallel-size 4 \
+  --pipeline-model-parallel-size 4 \
+  --context-parallel-size 2 \
+  --bf16 \
+  --use-mcore-models
+```
+
+**Memory per GPU**:
+- Model parameters: 140GB / 4 (TP) / 4 (PP) = 8.75GB
+- Optimizer states: ~17.5GB
+- Activations: ~3GB
+- **Total**: ~30GB per H100 (fits in 80GB)
+
+## NVIDIA Nemotron-4 340B
+
+### Overview
+- **Organization**: NVIDIA
+- **Parameters**: 340 billion
+- **Framework**: NeMo (built on Megatron-Core)
+- **Purpose**: Enterprise AI foundation model
+- **Status**: Commercial deployment
+
+**Key Features**:
+- Mixture of Experts architecture
+- Optimized for enterprise use cases
+- NeMo framework integration
+- Production-ready deployment
+
+**Architecture**:
+```yaml
+Type: Mixture of Experts (MoE)
+Total parameters: 340B
+Active parameters per token: ~40B
+Experts: 8
+Router: Top-2
+Context length: 4096
+```
+
+**Training Infrastructure**:
+- NVIDIA DGX H100 systems
+- Megatron-Core + NeMo
+- Multi-node training
+- Enterprise-grade fault tolerance
+
+**Production Features**:
+- NeMo Guardrails integration
+- Enterprise support
+- Customization options
+- On-premise deployment available
+
+## Microsoft & NVIDIA Megatron-Turing NLG 530B
+
+### Overview
+- **Organization**: Microsoft + NVIDIA collaboration
+- **Parameters**: 530 billion (largest dense model when released)
+- **Year**: 2021
+- **Framework**: DeepSpeed ZeRO-3 + Megatron tensor/pipeline parallelism
+- **Hardware**: 560 NVIDIA A100 80GB GPUs
+
+**Architecture**:
+```yaml
+Parameters: 530 billion
+Layers: 105
+Hidden size: 20480
+Attention heads: 128
+Vocabulary: 51,200 tokens
+Sequence length: 2048
+```
+
+**Training Configuration**:
+```bash
+# 560 A100 80GB GPUs
+Tensor Parallel: 8
+Pipeline Parallel: 35
+Data Parallel: 2
+Total: 8 × 35 × 2 = 560
+
+DeepSpeed ZeRO Stage 3:
+- Full parameter sharding
+- Gradient sharding
+- Optimizer state sharding
+```
+
+**Innovations**:
+- First to combine DeepSpeed ZeRO-3 with Megatron parallelism
+- Demonstrated training at 500B+ scale
+- Proved viability of extreme parallelism
+
+**Performance**:
+- Trained on 339 billion tokens
+- Multiple months of training
+- Achieved state-of-the-art results in 2021
+
+## BigScience BLOOM 176B
+
+### Overview
+- **Organization**: BigScience (1000+ researchers)
+- **Parameters**: 176 billion
+- **Year**: 2022
+- **Framework**: Megatron-DeepSpeed
+- **Hardware**: 384 NVIDIA A100 80GB GPUs
+- **Training Duration**: 46 days
+
+**Architecture**:
+```yaml
+Parameters: 176 billion
+Layers: 70
+Hidden size: 14336
+Attention heads: 112
+Vocabulary: 250,680 tokens (multilingual)
+Sequence length: 2048
+Languages: 46 natural languages + 13 programming languages
+```
+
+**Training Configuration**:
+```bash
+# 384 A100 80GB GPUs on Jean Zay supercomputer
+Tensor Parallel: 4
+Pipeline Parallel: 12
+Data Parallel: 8
+Total: 4 × 12 × 8 = 384
+
+Global batch size: 2048
+Micro-batch size: 4
+Learning rate: 6e-5
+Optimizer: Adam (β1=0.9, β2=0.95)
+```
+
+**Training Data**:
+- 366 billion tokens (1.6TB)
+- ROOTS corpus (custom multilingual dataset)
+- 46 natural languages
+- 13 programming languages
+
+**Key Achievements**:
+- Largest multilingual open-source model at release
+- Trained on public supercomputer (Jean Zay)
+- Fully documented training process
+- Open-source model and training code
+
+**Public Impact**:
+- Downloaded 100,000+ times
+- Used in hundreds of research papers
+- Enabled multilingual AI research
+- Demonstrated open science at scale
+
+## DeepSeek-V3
+
+### Overview
+- **Organization**: DeepSeek
+- **Parameters**: 671 billion total, 37B active per token
+- **Type**: Mixture of Experts (MoE)
+- **Year**: 2024-2025
+- **Framework**: Megatron-Core
+
+**Architecture**:
+```yaml
+Type: Mixture of Experts
+Total parameters: 671B
+Active parameters per token: 37B
+Layers: 61
+Hidden size: 7168
+Attention heads: 128
+Query groups: 16
+Experts: 256 (massive MoE)
+Router top-k: 8 (Multi-head Latent Attention)
+Shared expert size: 18432
+```
+
+**Training Configuration**:
+```bash
+# 1024 H100 GPUs
+Tensor Parallel (TP): 2
+Pipeline Parallel (PP): 16
+Expert Parallel (EP): 64
+Context Parallel (CP): 1
+
+Total: 2 × 16 × 64 = 2048 slots
+# Uses overlapping parallelism
+
+Global batch size: 4096
+Sequence length: 4096
+Training tokens: 14.8 trillion
+```
+
+**Innovations**:
+- Multi-head Latent Attention (MLA) router
+- Shared experts + routed experts
+- Ultra-large expert count (256)
+- Advanced load balancing
+
+**Performance**:
+- Competitive with GPT-4
+- 37B active params rivals 70B+ dense models
+- Efficient inference (only 37B active)
+
+## OpenAI GPT-3 175B (2020)
+
+### Overview
+- **Organization**: OpenAI
+- **Parameters**: 175 billion
+- **Year**: 2020
+- **Framework**: Megatron-inspired custom implementation
+- **Hardware**: Thousands of NVIDIA V100 GPUs
+
+**Architecture**:
+```yaml
+Parameters: 175 billion
+Layers: 96
+Hidden size: 12288
+Attention heads: 96
+FFN size: 49152
+Vocabulary: 50,257 tokens (GPT-2 BPE)
+Sequence length: 2048
+Context window: 2048 tokens
+```
+
+**Training Configuration**:
+```bash
+# Estimated configuration
+Tensor Parallel: 4-8
+Pipeline Parallel: 8-16
+Data Parallel: Remaining GPUs
+
+Global batch size: 1536
+Learning rate: 6e-5
+Training tokens: 300 billion
+```
+
+**Training Compute**:
+- 3.14 × 10^23 FLOPs
+- Equivalent to ~355 GPU-years on V100
+- Estimated cost: $4-12 million
+
+**Impact**:
+- Launched modern era of large language models
+- Demonstrated few-shot learning
+- Foundation for ChatGPT
+
+## Stability AI StableLM
+
+### Overview
+- **Organization**: Stability AI
+- **Framework**: GPT-NeoX (Megatron + DeepSpeed)
+- **Hardware**: Training on supercomputers
+- **Status**: Open-source
+
+**Models**:
+- StableLM-Base-Alpha: 3B, 7B
+- StableLM-Tuned-Alpha: Fine-tuned versions
+- StableCode: Code-specialized
+
+**Training Configuration**:
+```yaml
+Framework: GPT-NeoX
+Parallelism: Megatron TP/PP + DeepSpeed ZeRO
+GPUs: A100 clusters
+Training data: 1.5 trillion tokens (The Pile)
+```
+
+**Key Features**:
+- Fully open-source (Apache 2.0)
+- GPT-NeoX framework
+- Trained on The Pile dataset
+- Multiple model sizes
+
+## Common Production Patterns
+
+### Fault Tolerance
+
+**Checkpoint Strategy**:
+```bash
+--save-interval 500              # Save every 500 iterations
+--save /checkpoints/model_name  # Checkpoint directory
+--load /checkpoints/model_name  # Auto-resume from latest
+```
+
+**Monitoring**:
+```python
+# Check in progress.txt
+Job throughput: 45.2 TFLOPs/GPU
+Cumulative throughput: 44.8 TFLOPs/GPU
+Memory usage: 68.2 GB / 80 GB
+Loss: 2.143
+```
+
+### Data Pipeline
+
+**Preprocessing**:
+```bash
+python tools/preprocess_data.py \
+  --input data.jsonl \
+  --output-prefix /data/processed \
+  --vocab-file vocab.json \
+  --merge-file merges.txt \
+  --tokenizer-type GPT2BPETokenizer \
+  --append-eod \
+  --workers 64
+```
+
+**Training with Preprocessed Data**:
+```bash
+--data-path /data/processed_text_document \
+--split 969,30,1  # Train/valid/test split
+```
+
+### Monitoring & Logging
+
+**Key Metrics to Track**:
+```bash
+# Training metrics
+- Loss (should steadily decrease)
+- Learning rate (follows schedule)
+- Gradient norm (watch for spikes)
+- Throughput (TFlops/GPU)
+- MFU percentage
+
+# System metrics
+- GPU utilization (>90%)
+- Memory usage (<95% of capacity)
+- Network bandwidth (saturated for TP)
+- Data loading time (should be minimal)
+```
+
+**Production Monitoring Tools**:
+- TensorBoard for loss curves
+- Weights & Biases for experiment tracking
+- Prometheus + Grafana for system metrics
+- Custom scripts for MFU calculation
+
+### Multi-Datacenter Training
+
+**Challenges**:
+- Higher latency between datacenters
+- Network bandwidth limitations
+- Fault isolation
+
+**Solutions**:
+```bash
+# Keep TP within datacenter
+--tensor-model-parallel-size 8  # Single node only
+
+# Use PP across datacenters
+--pipeline-model-parallel-size 16  # Across sites
+
+# Data parallel across everything
+# Automatic from remaining GPUs
+```
+
+## Lessons from Production
+
+1. **Fault Tolerance is Critical**
+   - Save checkpoints frequently (every 500-1000 steps)
+   - Test checkpoint recovery regularly
+   - Monitor for GPU failures
+
+2. **Data Quality Matters More Than Quantity**
+   - LLaMA 3: Carefully curated 15T tokens
+   - Better than naive web scraping
+   - Investment in data preprocessing pays off
+
+3. **Parallelism Strategy Evolves with Scale**
+   - <70B: TP + DP sufficient
+   - 70-175B: Add PP
+   - 175B+: 3D or 4D parallelism required
+   - MoE: Add EP dimension
+
+4. **Hardware Matters**
+   - H100 vs A100: 2× speedup from better hardware
+   - NVLink topology affects TP efficiency
+   - InfiniBand essential for multi-node
+
+5. **Monitoring is Essential**
+   - Track MFU to catch performance issues
+   - Monitor loss for training health
+   - Watch memory usage to avoid OOM
+   - Log everything for debugging
+
+## References
+
+- Meta LLaMA 3 technical report
+- NVIDIA Nemotron blog posts
+- Microsoft Megatron-Turing NLG paper
+- BigScience BLOOM documentation
+- DeepSeek-V3 technical report