@r1z4x/tezgah 1.0.0

package/skills/ai-research/08-distributed-training/megatron-core/references/parallelism-guide.md

@@ -0,0 +1,405 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 08-distributed-training/megatron-core/references/parallelism-guide.md; MIT (c) Orchestra Research -->
+# Parallelism Strategies Guide
+
+Complete guide to parallelism in Megatron-Core: when to use each strategy, performance characteristics, and configuration examples.
+
+## Parallelism Types
+
+### 1. Data Parallel (DP)
+
+**How it works**: Replicate full model on each GPU, split data batches, synchronize gradients.
+
+**Memory**: Low efficiency - full model replication
+**Communication**: Low - only gradients (all-reduce)
+**Scalability**: Good for small models
+
+**When to use**:
+- Models <1B parameters that fit on single GPU
+- Plenty of GPUs available
+- Limited model complexity
+
+**Configuration**:
+```bash
+# 8 GPUs, no model parallelism
+torchrun --nproc_per_node=8 train.py \
+  --tensor-model-parallel-size 1 \
+  --pipeline-model-parallel-size 1
+# Implicit DP = 8
+```
+
+**Performance**:
+- Near-linear scaling for small models
+- 7B model on 8×A100: ~90% efficiency
+
+### 2. Tensor Parallel (TP)
+
+**How it works**: Split individual layers/tensors across GPUs (column/row partitioning of weight matrices).
+
+**Memory**: Excellent - 1/N reduction per GPU
+**Communication**: Very high - all-reduce after every layer
+**Scalability**: Best ≤8 GPUs within single node (needs NVLink)
+
+**When to use**:
+- Models >10B parameters
+- Have NVLink-connected GPUs
+- Within single node (network latency kills performance across nodes)
+
+**Configuration**:
+```bash
+# Split model across 4 GPUs with TP
+torchrun --nproc_per_node=4 train.py \
+  --tensor-model-parallel-size 4
+```
+
+**Performance**:
+- **1 node (8 GPUs, NVLink)**: 85-95% efficiency
+- **Across nodes**: <50% efficiency (avoid)
+
+**Memory savings**:
+```
+LLaMA 70B without TP: 140GB (won't fit on 80GB GPU)
+LLaMA 70B with TP=4: 35GB per GPU (fits easily)
+```
+
+**Communication volume** (70B model):
+- Per layer: ~20GB all-reduce
+- 80 layers × 20GB = 1.6TB total traffic
+- With NVLink (600GB/s): Manageable
+- With Ethernet (100Gb/s = 12.5GB/s): Too slow
+
+### 3. Pipeline Parallel (PP)
+
+**How it works**: Divide model layers into stages, assign stages to different GPUs, process microbatches in pipeline.
+
+**Memory**: Very high - divide layers evenly
+**Communication**: Low-medium - only activations between stages
+**Scalability**: Good across nodes
+
+**Pipeline Schedules**:
+
+**GPipe** (simple but inefficient):
+```
+GPU0: F F F F ........ B B B B
+GPU1: .... F F F F .... B B B B
+GPU2: ........ F F F F B B B B
+```
+Bubble: 50% idle time
+
+**1F1B** (one-forward-one-backward):
+```
+GPU0: F F F F B B B B B B B B
+GPU1: .. F F F F B B B B B B B B
+GPU2: .... F F F F B B B B B B B B
+```
+Bubble: ~25% idle time
+
+**Interleaved 1F1B** (best):
+```
+GPU0: F1 F2 F3 F4 B1 B2 B3 B4 ...
+GPU1: F1 F2 F3 F4 B1 B2 B3 B4 ...
+```
+Bubble: 5-10% idle time
+
+**When to use**:
+- Models >70B parameters
+- Multi-node training
+- Limited intra-node bandwidth
+
+**Configuration**:
+```bash
+# 4-stage pipeline
+torchrun --nproc_per_node=8 --nnodes=4 train.py \
+  --pipeline-model-parallel-size 4 \
+  --num-layers 80 \
+  --num-layers-per-virtual-pipeline-stage 2  # Interleaved
+```
+
+**Performance**:
+- Interleaved schedule: 90-95% efficiency
+- Standard 1F1B: 75-85% efficiency
+
+### 4. Sequence Parallel (SP)
+
+**How it works**: Split sequence dimension across tensor-parallel GPUs, reduce activation memory.
+
+**Memory**: Reduces activations by TP factor
+**Communication**: Same as TP (already using all-reduce)
+**Scalability**: Tied to TP
+
+**When to use**:
+- Long sequences (>4K tokens)
+- Using TP already
+- Activation memory is bottleneck
+
+**Configuration**:
+```bash
+torchrun --nproc_per_node=8 train.py \
+  --tensor-model-parallel-size 4 \
+  --sequence-parallel  # Requires TP > 1
+```
+
+**Memory savings**:
+```
+70B model, 4K sequence, TP=4:
+Without SP: 48GB activations per GPU
+With SP: 12GB activations per GPU
+Savings: 75%
+```
+
+### 5. Context Parallel (CP)
+
+**How it works**: Split very long sequences across GPUs using Ring Attention.
+
+**Memory**: Reduces KV cache and activations
+**Communication**: Medium - ring communication pattern
+**Scalability**: Good for >8K sequences
+
+**When to use**:
+- Sequences >8K tokens
+- Long-context models (>32K)
+- KV cache memory bottleneck
+
+**Configuration**:
+```bash
+torchrun --nproc_per_node=8 train.py \
+  --context-parallel-size 2 \
+  --seq-length 32768  # 32K tokens
+```
+
+**Memory savings** (32K sequence):
+```
+Without CP: 64GB KV cache
+With CP=4: 16GB KV cache per GPU
+```
+
+### 6. Expert Parallel (EP)
+
+**How it works**: For MoE models, distribute different experts across GPUs.
+
+**Memory**: Excellent - only store 1/N experts per GPU
+**Communication**: Low - only route tokens to experts
+**Scalability**: Matches number of experts
+
+**When to use**:
+- Mixture of Experts models
+- Want model capacity without memory cost
+- Have ≥8 GPUs
+
+**Configuration**:
+```bash
+# Mixtral 8x7B: 8 experts
+torchrun --nproc_per_node=8 train.py \
+  --expert-model-parallel-size 4 \
+  --num-experts 8 \
+  --tensor-model-parallel-size 2
+```
+
+**Memory** (Mixtral 8×7B):
+```
+Without EP: 8 experts × 7B = 56GB
+With EP=4: 2 experts × 7B = 14GB
+Savings: 75%
+```
+
+## Combining Parallelism Strategies
+
+### 3D Parallelism (TP + PP + DP)
+
+Standard for large models.
+
+**LLaMA 3 70B on 64 GPUs**:
+```bash
+TP=4  # Within each node
+PP=4  # Across nodes
+DP=4  # Remaining dimension
+Total = 4 × 4 × 4 = 64 GPUs
+```
+
+**Memory per GPU**: 70B / 4 (TP) / 4 (PP) = 4.4B params ≈ 20GB
+
+**Configuration**:
+```bash
+torchrun --nproc_per_node=8 --nnodes=8 train.py \
+  --tensor-model-parallel-size 4 \
+  --pipeline-model-parallel-size 4
+  # DP is implicit: 64 / (4*4) = 4
+```
+
+### 4D Parallelism (TP + PP + DP + CP)
+
+For very large models or long context.
+
+**LLaMA 3 405B on 256 GPUs**:
+```bash
+TP=8   # Max NVLink
+PP=8   # Across nodes
+CP=2   # Long sequences
+DP=2   # Remaining
+Total = 8 × 8 × 2 × 2 = 256 GPUs
+```
+
+**Configuration**:
+```bash
+torchrun --nproc_per_node=8 --nnodes=32 train.py \
+  --tensor-model-parallel-size 8 \
+  --pipeline-model-parallel-size 8 \
+  --context-parallel-size 2
+```
+
+### 4D + EP (5D Parallelism)
+
+For sparse MoE models.
+
+**DeepSeek-V3 671B (37B active) on 1024 GPUs**:
+```bash
+TP=2   # Limited by active params
+PP=16  # Many stages
+EP=64  # 256 experts / 4 experts per GPU
+DP=2   # Small data parallel
+Total = 2 × 16 × 64 × 2 = 4096 (uses 1024 in practice)
+```
+
+## Decision Guide
+
+### By Model Size
+
+| Model Size | GPUs | Recommended Strategy |
+|------------|------|---------------------|
+| <1B | 1-8 | DP only |
+| 1-10B | 8-16 | TP=2-4 + DP |
+| 10-70B | 16-64 | TP=4 + PP=2-4 + DP |
+| 70-175B | 64-256 | TP=8 + PP=4-8 + DP |
+| 175-500B | 256-1024 | TP=8 + PP=8-16 + CP=2 + DP |
+| 500B+ | 1024+ | 4D or 5D (with EP) |
+
+### By Hardware Topology
+
+**Single node (8 GPUs with NVLink)**:
+```bash
+# Up to 70B
+TP=8  # Use all NVLink bandwidth
+```
+
+**Multiple nodes (InfiniBand)**:
+```bash
+# Minimize cross-node communication
+TP=8      # Within node only
+PP=N      # Across nodes
+DP=remaining
+```
+
+**Limited network (Ethernet)**:
+```bash
+# Avoid TP across nodes
+TP=1-4    # Within node
+PP=many   # PP has low communication
+```
+
+### By Sequence Length
+
+| Sequence | Parallelism |
+|----------|------------|
+| <2K | Standard (TP + PP + DP) |
+| 2K-8K | + SP (sequence parallel) |
+| 8K-32K | + CP=2 (context parallel) |
+| 32K+ | + CP=4-8 |
+
+## Performance Characteristics
+
+### Communication Volume (per iteration)
+
+**Data Parallel**: O(model_size) - all-reduce gradients
+**Tensor Parallel**: O(model_size × layers) - all-reduce per layer
+**Pipeline Parallel**: O(batch × hidden × layers/stages) - activations only
+**Context Parallel**: O(sequence × hidden) - ring communication
+
+### Memory Breakdown (70B model example)
+
+Without parallelism:
+```
+Model parameters: 140GB (FP16)
+Gradients: 140GB
+Optimizer states: 280GB (Adam)
+Activations: 48GB (batch=1, seq=4K)
+Total: 608GB (won't fit!)
+```
+
+With TP=4, PP=4, DP=4 (64 GPUs):
+```
+Parameters: 140GB / 4 / 4 = 8.75GB per GPU
+Gradients: 8.75GB per GPU
+Optimizer: 17.5GB per GPU
+Activations: 48GB / 4 / 4 = 3GB per GPU
+Total: ~38GB per GPU (fits on A100 80GB)
+```
+
+## Best Practices
+
+1. **Start with TP within single node**
+   ```bash
+   --tensor-model-parallel-size 8  # Use all NVLink
+   ```
+
+2. **Add PP for cross-node scaling**
+   ```bash
+   --pipeline-model-parallel-size 4
+   --num-layers-per-virtual-pipeline-stage 2  # Interleaved
+   ```
+
+3. **Enable SP when using TP**
+   ```bash
+   --sequence-parallel  # Free activation savings
+   ```
+
+4. **Use CP for long sequences**
+   ```bash
+   --context-parallel-size 2  # If seq_len > 8K
+   ```
+
+5. **Avoid TP across nodes** (network latency kills performance)
+
+6. **Match TP to GPU topology** (TP=8 for 8-GPU nodes)
+
+7. **Profile first iteration** to check memory and communication:
+   ```bash
+   --profile  # Enable profiling
+   --profile-ranks 0  # Profile first rank only
+   ```
+
+## Troubleshooting
+
+**High communication overhead (low MFU)**:
+- Reduce TP degree (especially across nodes)
+- Increase PP degree instead
+- Enable interleaved pipeline schedule
+
+**Out of memory**:
+- Increase TP/PP (split model more)
+- Enable gradient checkpointing:
+  ```bash
+  --recompute-granularity full
+  --recompute-method block
+  ```
+- Reduce micro-batch size
+
+**Pipeline bubbles (low GPU util)**:
+- Use interleaved schedule:
+  ```bash
+  --num-layers-per-virtual-pipeline-stage 2
+  ```
+- Increase number of microbatches:
+  ```bash
+  --global-batch-size 1024
+  --micro-batch-size 1  # More microbatches = smaller bubbles
+  ```
+
+**Load imbalance in MoE**:
+- Tune load balancing:
+  ```bash
+  --moe-router-load-balancing-type aux_loss
+  --moe-aux-loss-coeff 0.01
+  ```
+- Increase expert parallel degree:
+  ```bash
+  --expert-model-parallel-size 8  # More experts per GPU
+  ```