@synsci/cli-darwin-x64 1.1.49

package/bin/skills/moe-training/references/training.md

@@ -0,0 +1,425 @@
+# MoE Training Guide
+
+Complete training guide based on DeepSpeed official documentation and production practices.
+
+## Table of Contents
+- DeepSpeed MoE Setup
+- Training Configuration
+- PR-MoE (Pyramid-Residual-MoE)
+- Mixture-of-Students (MoS)
+- Hyperparameter Tuning
+- Production Training
+
+## DeepSpeed MoE Setup
+
+**Source**: DeepSpeed MoE Tutorial (https://www.deepspeed.ai/tutorials/mixture-of-experts-nlg/)
+
+### Requirements
+
+```bash
+# Install DeepSpeed v0.6.0 or higher
+pip install deepspeed>=0.6.0
+
+# Clone Megatron-DeepSpeed
+git clone https://github.com/microsoft/Megatron-DeepSpeed
+cd Megatron-DeepSpeed
+pip install -r requirements.txt
+```
+
+### Basic MoE Configuration
+
+```json
+{
+  "train_batch_size": 256,
+  "gradient_accumulation_steps": 1,
+  "fp16": {
+    "enabled": true,
+    "loss_scale": 0,
+    "initial_scale_power": 16
+  },
+  "moe": {
+    "enabled": true,
+    "num_experts": 128,
+    "expert_parallel_size": 8,
+    "moe_loss_coeff": 0.01,
+    "train_capacity_factor": 1.25,
+    "eval_capacity_factor": 2.0,
+    "min_capacity": 4,
+    "drop_tokens": true
+  },
+  "zero_optimization": {
+    "stage": 1
+  }
+}
+```
+
+## Training Parameters
+
+### Core MoE Parameters
+
+**From DeepSpeed documentation:**
+
+1. **`--num-experts`**
+   - Number of experts per MoE layer
+   - Recommended: 128 experts
+   - Range: 8-256 depending on scale
+
+2. **`--moe-expert-parallel-size`**
+   - Degree of expert parallelism
+   - Distributes experts across GPUs
+   - Example: 128 experts / 8 GPUs = 16 experts per GPU
+
+3. **`--moe-loss-coeff`**
+   - MoE auxiliary loss coefficient
+   - Recommended: 0.01
+   - Controls load balancing strength
+
+4. **`--moe-train-capacity-factor`**
+   - Training capacity multiplier
+   - Default: 1.25
+   - Formula: capacity = (tokens/num_experts) × capacity_factor
+
+5. **`--moe-eval-capacity-factor`**
+   - Evaluation capacity multiplier
+   - Default: 2.0 (no token dropping during eval)
+
+6. **`--moe-min-capacity`**
+   - Minimum expert capacity
+   - Default: 4
+   - Ensures each expert processes minimum tokens
+
+7. **`--disable-moe-token-dropping`**
+   - Remove expert capacity limits
+   - All tokens processed (no dropping)
+   - May increase memory usage
+
+### Example Training Script
+
+```bash
+#!/bin/bash
+
+deepspeed --num_gpus 8 pretrain_gpt_moe.py \
+  --tensor-model-parallel-size 1 \
+  --pipeline-model-parallel-size 1 \
+  --num-layers 24 \
+  --hidden-size 1024 \
+  --num-attention-heads 16 \
+  --seq-length 2048 \
+  --max-position-embeddings 2048 \
+  --micro-batch-size 4 \
+  --global-batch-size 256 \
+  --train-iters 500000 \
+  --lr 0.0001 \
+  --min-lr 0.00001 \
+  --lr-decay-style cosine \
+  --lr-warmup-iters 2000 \
+  --clip-grad 1.0 \
+  --weight-decay 0.1 \
+  --num-experts 128 \
+  --moe-expert-parallel-size 8 \
+  --moe-loss-coeff 0.01 \
+  --moe-train-capacity-factor 1.25 \
+  --moe-eval-capacity-factor 2.0 \
+  --moe-min-capacity 4 \
+  --fp16 \
+  --deepspeed \
+  --deepspeed_config ds_config_moe.json \
+  --data-path /path/to/data \
+  --vocab-file /path/to/vocab.json \
+  --merge-file /path/to/merges.txt \
+  --save-interval 5000 \
+  --eval-interval 1000 \
+  --eval-iters 100
+```
+
+## PR-MoE: Pyramid-Residual-MoE
+
+**Source**: DeepSpeed documentation - improves parameter efficiency 3× over standard MoE
+
+### Architecture
+
+PR-MoE uses:
+- Varying number of experts per layer (pyramid structure)
+- Residual connections between expert layers
+- Better parameter efficiency
+
+### Configuration
+
+```bash
+# PR-MoE specific parameters
+--num-experts "[128, 64, 32, 16]" \  # Pyramid: different experts per layer
+--mlp-type residual \                # Use residual connections
+--moe-expert-parallel-size 4 \
+--moe-loss-coeff 0.01
+```
+
+### Full PR-MoE Training
+
+```bash
+deepspeed --num_gpus 8 pretrain_gpt_moe.py \
+  --num-layers 24 \
+  --hidden-size 1024 \
+  --num-attention-heads 16 \
+  --seq-length 2048 \
+  --max-position-embeddings 2048 \
+  --micro-batch-size 4 \
+  --global-batch-size 256 \
+  --num-experts "[128, 64, 32, 16]" \  # Pyramid structure
+  --mlp-type residual \                # Residual MoE
+  --moe-expert-parallel-size 4 \
+  --moe-loss-coeff 0.01 \
+  --moe-train-capacity-factor 1.25 \
+  --fp16 \
+  --deepspeed \
+  --deepspeed_config ds_config_moe.json \
+  --data-path /path/to/data \
+  --save-interval 5000
+```
+
+**Benefits**:
+- 3× better parameter efficiency vs standard MoE
+- Fewer total parameters for same performance
+- Better gradient flow with residual connections
+
+## Mixture-of-Students (MoS)
+
+**Source**: DeepSpeed documentation - knowledge distillation for MoE
+
+### Overview
+
+MoS = MoE + Knowledge Distillation
+- Student: MoE model (being trained)
+- Teacher: Dense model (pre-trained)
+- Transfers knowledge from dense teacher to sparse MoE student
+
+### Configuration
+
+```bash
+# MoS parameters
+--mos \                              # Enable MoS distillation
+--load-teacher /path/to/teacher \    # Teacher model checkpoint
+--teacher-forward \                  # Enable teacher forward pass
+--teacher-model-parallel-size 1
+```
+
+### Full MoS Training
+
+```bash
+deepspeed --num_gpus 8 pretrain_gpt_moe.py \
+  --num-layers 24 \
+  --hidden-size 1024 \
+  --num-attention-heads 16 \
+  --num-experts 128 \
+  --moe-expert-parallel-size 8 \
+  --moe-loss-coeff 0.01 \
+  --mos \                                    # Enable MoS
+  --load-teacher /path/to/dense/teacher \    # Teacher checkpoint
+  --teacher-forward \
+  --teacher-model-parallel-size 1 \
+  --fp16 \
+  --deepspeed \
+  --deepspeed_config ds_config_moe.json \
+  --data-path /path/to/data
+```
+
+### Staged Distillation
+
+**Recommended**: Stop distillation early
+
+```python
+# In training loop
+if iteration < 400000:
+    # Use MoS (distillation)
+    loss = moe_loss + distillation_loss
+else:
+    # Stop distillation, train MoE only
+    loss = moe_loss
+```
+
+**Benefits**:
+- Faster convergence
+- Better final performance
+- Preserves teacher knowledge while allowing MoE specialization
+
+## Hyperparameter Tuning
+
+### Learning Rate
+
+**Key insight**: MoE needs lower LR than dense models
+
+```bash
+# Dense model
+--lr 0.0006 \
+--min-lr 0.00006
+
+# MoE model (3-6× lower)
+--lr 0.0001 \        # Lower!
+--min-lr 0.00001
+```
+
+### LR Decay
+
+**Extend decay schedule** for MoE:
+
+```bash
+# Dense model
+--lr-decay-iters 300000 \
+--lr-warmup-iters 2000
+
+# MoE model (1.5-2× longer)
+--lr-decay-iters 500000 \   # Extended!
+--lr-warmup-iters 2000
+```
+
+### Capacity Factor
+
+**Tune based on memory/speed tradeoff**:
+
+```json
+{
+  "moe": {
+    // Training: Lower capacity (faster, drops tokens)
+    "train_capacity_factor": 1.0,   // Aggressive
+    "train_capacity_factor": 1.25,  // Balanced (recommended)
+    "train_capacity_factor": 1.5,   // Conservative
+
+    // Evaluation: Higher capacity (no dropping)
+    "eval_capacity_factor": 2.0     // Standard
+  }
+}
+```
+
+### Load Balancing Coefficient
+
+```json
+{
+  "moe": {
+    "moe_loss_coeff": 0.001,  // Weak balancing
+    "moe_loss_coeff": 0.01,   // Standard (recommended)
+    "moe_loss_coeff": 0.1     // Strong balancing
+  }
+}
+```
+
+**Rule**: If load imbalance persists, increase coefficient
+
+## Production Training
+
+### Performance Benchmarks
+
+**From DeepSpeed documentation:**
+
+Standard MoE:
+- **5× training cost reduction** vs dense model
+- **3× model size reduction** with PR-MoE
+
+Example:
+- Dense 13B model: 100% cost
+- MoE 13B (128 experts): 20% cost (5× faster)
+- PR-MoE 13B: 15% cost + 3× fewer params
+
+### Recommended Dataset
+
+**The Pile** - publicly available training dataset
+- 800GB of diverse text
+- Standard benchmark for MoE training
+- Used in DeepSpeed examples
+
+### Example Configs
+
+**Small MoE (8 experts)**:
+
+```bash
+deepspeed --num_gpus 4 pretrain_gpt_moe.py \
+  --num-layers 12 \
+  --hidden-size 768 \
+  --num-attention-heads 12 \
+  --num-experts 8 \
+  --moe-expert-parallel-size 2 \
+  --global-batch-size 128 \
+  --fp16
+```
+
+**Medium MoE (64 experts)**:
+
+```bash
+deepspeed --num_gpus 16 pretrain_gpt_moe.py \
+  --num-layers 24 \
+  --hidden-size 1024 \
+  --num-attention-heads 16 \
+  --num-experts 64 \
+  --moe-expert-parallel-size 8 \
+  --global-batch-size 256 \
+  --fp16
+```
+
+**Large MoE (128 experts)**:
+
+```bash
+deepspeed --num_gpus 32 pretrain_gpt_moe.py \
+  --num-layers 32 \
+  --hidden-size 2048 \
+  --num-attention-heads 32 \
+  --num-experts 128 \
+  --moe-expert-parallel-size 16 \
+  --global-batch-size 512 \
+  --fp16
+```
+
+### Monitoring
+
+Key metrics to track:
+
+```python
+# Expert load balance
+expert_counts = [expert.token_count for expert in experts]
+load_imbalance = max(expert_counts) / min(expert_counts)
+
+# Should be close to 1.0 (perfectly balanced)
+# If > 2.0, increase moe_loss_coeff
+
+# Expert utilization
+utilized_experts = sum(count > 0 for count in expert_counts)
+utilization_rate = utilized_experts / num_experts
+
+# Should be close to 1.0 (all experts used)
+
+# Token dropping rate
+dropped_tokens = total_tokens - processed_tokens
+drop_rate = dropped_tokens / total_tokens
+
+# Should be low (<5%) during training
+```
+
+## Troubleshooting
+
+### Issue: Load Imbalance
+
+**Symptoms**: Some experts get most tokens
+
+**Solutions**:
+1. Increase `moe_loss_coeff` (0.01 → 0.1)
+2. Reduce `train_capacity_factor` (forces redistribution)
+3. Add noise to router logits (gating network)
+
+### Issue: High Memory Usage
+
+**Solutions**:
+1. Enable ZeRO Stage 1 or 2
+2. Reduce `train_capacity_factor`
+3. Enable `drop_tokens`
+4. Increase `moe_expert_parallel_size`
+
+### Issue: Unstable Training
+
+**Solutions**:
+1. Lower learning rate
+2. Increase warmup steps
+3. Use gradient clipping (`--clip-grad 1.0`)
+4. Reduce router z-loss coefficient
+
+## Resources
+
+- **DeepSpeed MoE Tutorial**: https://www.deepspeed.ai/tutorials/mixture-of-experts-nlg/
+- **Megatron-DeepSpeed**: https://github.com/microsoft/Megatron-DeepSpeed
+- **Example Scripts**: `examples_deepspeed/MoE/`

package/bin/skills/nanogpt/SKILL.md

@@ -0,0 +1,290 @@
+---
+name: nanogpt
+description: Educational GPT implementation in ~300 lines. Reproduces GPT-2 (124M) on OpenWebText. Clean, hackable code for learning transformers. By Andrej Karpathy. Perfect for understanding GPT architecture from scratch. Train on Shakespeare (CPU) or OpenWebText (multi-GPU).
+version: 1.0.0
+author: Synthetic Sciences
+license: MIT
+tags: [Model Architecture, NanoGPT, GPT-2, Educational, Andrej Karpathy, Transformer, Minimalist, From Scratch, Training]
+dependencies: [torch, transformers, datasets, tiktoken, wandb]
+---
+
+# nanoGPT - Minimalist GPT Training
+
+## Quick start
+
+nanoGPT is a simplified GPT implementation designed for learning and experimentation.
+
+**Installation**:
+```bash
+pip install torch numpy transformers datasets tiktoken wandb tqdm
+```
+
+**Train on Shakespeare** (CPU-friendly):
+```bash
+# Prepare data
+python data/shakespeare_char/prepare.py
+
+# Train (5 minutes on CPU)
+python train.py config/train_shakespeare_char.py
+
+# Generate text
+python sample.py --out_dir=out-shakespeare-char
+```
+
+**Output**:
+```
+ROMEO:
+What say'st thou? Shall I speak, and be a man?
+
+JULIET:
+I am afeard, and yet I'll speak; for thou art
+One that hath been a man, and yet I know not
+What thou art.
+```
+
+## Common workflows
+
+### Workflow 1: Character-level Shakespeare
+
+**Complete training pipeline**:
+```bash
+# Step 1: Prepare data (creates train.bin, val.bin)
+python data/shakespeare_char/prepare.py
+
+# Step 2: Train small model
+python train.py config/train_shakespeare_char.py
+
+# Step 3: Generate text
+python sample.py --out_dir=out-shakespeare-char
+```
+
+**Config** (`config/train_shakespeare_char.py`):
+```python
+# Model config
+n_layer = 6          # 6 transformer layers
+n_head = 6           # 6 attention heads
+n_embd = 384         # 384-dim embeddings
+block_size = 256     # 256 char context
+
+# Training config
+batch_size = 64
+learning_rate = 1e-3
+max_iters = 5000
+eval_interval = 500
+
+# Hardware
+device = 'cpu'  # Or 'cuda'
+compile = False # Set True for PyTorch 2.0
+```
+
+**Training time**: ~5 minutes (CPU), ~1 minute (GPU)
+
+### Workflow 2: Reproduce GPT-2 (124M)
+
+**Multi-GPU training on OpenWebText**:
+```bash
+# Step 1: Prepare OpenWebText (takes ~1 hour)
+python data/openwebtext/prepare.py
+
+# Step 2: Train GPT-2 124M with DDP (8 GPUs)
+torchrun --standalone --nproc_per_node=8 \
+  train.py config/train_gpt2.py
+
+# Step 3: Sample from trained model
+python sample.py --out_dir=out
+```
+
+**Config** (`config/train_gpt2.py`):
+```python
+# GPT-2 (124M) architecture
+n_layer = 12
+n_head = 12
+n_embd = 768
+block_size = 1024
+dropout = 0.0
+
+# Training
+batch_size = 12
+gradient_accumulation_steps = 5 * 8  # Total batch ~0.5M tokens
+learning_rate = 6e-4
+max_iters = 600000
+lr_decay_iters = 600000
+
+# System
+compile = True  # PyTorch 2.0
+```
+
+**Training time**: ~4 days (8× A100)
+
+### Workflow 3: Fine-tune pretrained GPT-2
+
+**Start from OpenAI checkpoint**:
+```python
+# In train.py or config
+init_from = 'gpt2'  # Options: gpt2, gpt2-medium, gpt2-large, gpt2-xl
+
+# Model loads OpenAI weights automatically
+python train.py config/finetune_shakespeare.py
+```
+
+**Example config** (`config/finetune_shakespeare.py`):
+```python
+# Start from GPT-2
+init_from = 'gpt2'
+
+# Dataset
+dataset = 'shakespeare_char'
+batch_size = 1
+block_size = 1024
+
+# Fine-tuning
+learning_rate = 3e-5  # Lower LR for fine-tuning
+max_iters = 2000
+warmup_iters = 100
+
+# Regularization
+weight_decay = 1e-1
+```
+
+### Workflow 4: Custom dataset
+
+**Train on your own text**:
+```python
+# data/custom/prepare.py
+import numpy as np
+
+# Load your data
+with open('my_data.txt', 'r') as f:
+    text = f.read()
+
+# Create character mappings
+chars = sorted(list(set(text)))
+stoi = {ch: i for i, ch in enumerate(chars)}
+itos = {i: ch for i, ch in enumerate(chars)}
+
+# Tokenize
+data = np.array([stoi[ch] for ch in text], dtype=np.uint16)
+
+# Split train/val
+n = len(data)
+train_data = data[:int(n*0.9)]
+val_data = data[int(n*0.9):]
+
+# Save
+train_data.tofile('data/custom/train.bin')
+val_data.tofile('data/custom/val.bin')
+```
+
+**Train**:
+```bash
+python data/custom/prepare.py
+python train.py --dataset=custom
+```
+
+## When to use vs alternatives
+
+**Use nanoGPT when**:
+- Learning how GPT works
+- Experimenting with transformer variants
+- Teaching/education purposes
+- Quick prototyping
+- Limited compute (can run on CPU)
+
+**Simplicity advantages**:
+- **~300 lines**: Entire model in `model.py`
+- **~300 lines**: Training loop in `train.py`
+- **Hackable**: Easy to modify
+- **No abstractions**: Pure PyTorch
+
+**Use alternatives instead**:
+- **HuggingFace Transformers**: Production use, many models
+- **Megatron-LM**: Large-scale distributed training
+- **LitGPT**: More architectures, production-ready
+- **PyTorch Lightning**: Need high-level framework
+
+## Common issues
+
+**Issue: CUDA out of memory**
+
+Reduce batch size or context length:
+```python
+batch_size = 1  # Reduce from 12
+block_size = 512  # Reduce from 1024
+gradient_accumulation_steps = 40  # Increase to maintain effective batch
+```
+
+**Issue: Training too slow**
+
+Enable compilation (PyTorch 2.0+):
+```python
+compile = True  # 2× speedup
+```
+
+Use mixed precision:
+```python
+dtype = 'bfloat16'  # Or 'float16'
+```
+
+**Issue: Poor generation quality**
+
+Train longer:
+```python
+max_iters = 10000  # Increase from 5000
+```
+
+Lower temperature:
+```python
+# In sample.py
+temperature = 0.7  # Lower from 1.0
+top_k = 200       # Add top-k sampling
+```
+
+**Issue: Can't load GPT-2 weights**
+
+Install transformers:
+```bash
+pip install transformers
+```
+
+Check model name:
+```python
+init_from = 'gpt2'  # Valid: gpt2, gpt2-medium, gpt2-large, gpt2-xl
+```
+
+## Advanced topics
+
+**Model architecture**: See [references/architecture.md](references/architecture.md) for GPT block structure, multi-head attention, and MLP layers explained simply.
+
+**Training loop**: See [references/training.md](references/training.md) for learning rate schedule, gradient accumulation, and distributed data parallel setup.
+
+**Data preparation**: See [references/data.md](references/data.md) for tokenization strategies (character-level vs BPE) and binary format details.
+
+## Hardware requirements
+
+- **Shakespeare (char-level)**:
+  - CPU: 5 minutes
+  - GPU (T4): 1 minute
+  - VRAM: <1GB
+
+- **GPT-2 (124M)**:
+  - 1× A100: ~1 week
+  - 8× A100: ~4 days
+  - VRAM: ~16GB per GPU
+
+- **GPT-2 Medium (350M)**:
+  - 8× A100: ~2 weeks
+  - VRAM: ~40GB per GPU
+
+**Performance**:
+- With `compile=True`: 2× speedup
+- With `dtype=bfloat16`: 50% memory reduction
+
+## Resources
+
+- GitHub: https://github.com/karpathy/nanoGPT ⭐ 48,000+
+- Video: "Let's build GPT" by Andrej Karpathy
+- Paper: "Attention is All You Need" (Vaswani et al.)
+- OpenWebText: https://huggingface.co/datasets/Skylion007/openwebtext
+- Educational: Best for understanding transformers from scratch
+
+