@r1z4x/tezgah 1.0.0

package/skills/ai-research/06-post-training/miles/references/troubleshooting.md

@@ -0,0 +1,353 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 06-post-training/miles/references/troubleshooting.md; MIT (c) Orchestra Research -->
+# miles Troubleshooting Guide
+
+## FP8 Training Issues
+
+### Issue: FP8 Training Collapse
+
+**Symptoms**: Loss explodes, NaN values, reward collapses
+
+**Solutions**:
+
+1. **Use block scaling**:
+```bash
+--fp8-recipe blockwise
+export NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1
+```
+
+2. **Enable R3 for MoE models**:
+```bash
+--use-r3
+```
+
+3. **Reduce learning rate**:
+```bash
+--lr 5e-7  # Reduce from 1e-6
+```
+
+4. **Warm up from BF16**:
+```bash
+--warmup-steps 100
+--warmup-precision bf16
+```
+
+### Issue: FP8 vs BF16 Accuracy Gap
+
+**Symptoms**: FP8 model underperforms BF16 baseline
+
+**Solutions**:
+
+1. **Use E4M3 format for activations**:
+```bash
+--fp8-format e4m3
+```
+
+2. **Enable dynamic scaling**:
+```bash
+--fp8-dynamic-scaling
+```
+
+3. **Skip sensitive layers**:
+```bash
+--fp8-skip-layers "lm_head,embed"
+```
+
+## Train-Inference Mismatch Issues
+
+### Issue: Policy Divergence
+
+**Symptoms**: Model behavior differs between training and inference
+
+**Solutions**:
+
+1. **Enable Rollout Routing Replay**:
+```bash
+--use-r3
+```
+
+2. **Use importance sampling correction**:
+```bash
+--use-tis --tis-threshold 0.9
+```
+
+3. **Verify log probs match**:
+```bash
+--verify-logprobs
+```
+
+### Issue: Expert Routing Mismatch (MoE)
+
+**Symptoms**: Different experts activated during train vs inference
+
+**Solutions**:
+
+1. **Enable R3**:
+```bash
+--use-r3
+--r3-buffer-size 1000
+```
+
+2. **Use deterministic routing**:
+```bash
+--deterministic-expert-routing
+```
+
+## INT4 Training Issues
+
+### Issue: INT4 Accuracy Degradation
+
+**Symptoms**: Worse performance than BF16 or FP8
+
+**Solutions**:
+
+1. **Increase group size**:
+```bash
+--int4-group-size 256  # Increase from 128
+```
+
+2. **Use mixed precision for sensitive layers**:
+```bash
+--int4-skip-layers "lm_head,embed,layer_norm"
+```
+
+3. **Warm start from BF16**:
+```bash
+--warmup-steps 100
+--warmup-precision bf16
+```
+
+4. **Increase learning rate** (INT4 often needs higher LR):
+```bash
+--lr 2e-6  # Increase from 1e-6
+```
+
+### Issue: INT4 OOM Despite Expected Savings
+
+**Symptoms**: Still running out of memory with INT4
+
+**Solutions**:
+
+1. **Verify environment variable**:
+```bash
+export OPEN_TRAINING_INT4_FAKE_QAT_FLAG=1
+```
+
+2. **Check group size alignment**:
+```bash
+# Group size must divide hidden dimension evenly
+--int4-group-size 128  # Must divide hidden_size
+```
+
+## Speculative RL Issues
+
+### Issue: Low Acceptance Rate
+
+**Symptoms**: Draft model tokens frequently rejected
+
+**Solutions**:
+
+1. **Reduce lookahead**:
+```bash
+--spec-lookahead 3  # Reduce from 5
+```
+
+2. **Update draft more frequently**:
+```bash
+--online-sft-interval 5  # Reduce from 10
+```
+
+3. **Increase draft learning rate**:
+```bash
+--draft-lr 1e-5  # Increase
+```
+
+### Issue: Draft Model Drift
+
+**Symptoms**: Acceptance rate drops over time
+
+**Solutions**:
+
+1. **Enable online SFT**:
+```bash
+--online-sft-interval 5
+```
+
+2. **Use EMA for draft updates**:
+```bash
+--draft-ema-decay 0.99
+```
+
+3. **Reinitialize draft periodically**:
+```bash
+--reinit-draft-interval 1000
+```
+
+### Issue: Speculative Training Slower Than Expected
+
+**Symptoms**: Not achieving expected 25%+ speedup
+
+**Solutions**:
+
+1. **Verify draft model is small enough**:
+```bash
+# Draft should be 1/4 to 1/10 size of target
+```
+
+2. **Check lookahead is optimal**:
+```bash
+--spec-lookahead 5  # Sweet spot for most models
+```
+
+3. **Profile to find bottleneck**:
+```bash
+--profile-speculative
+```
+
+## Weight Synchronization Issues
+
+### Issue: Zero-Copy Sync Failures
+
+**Symptoms**: Errors with CUDA IPC, weight corruption
+
+**Solutions**:
+
+1. **Verify CUDA IPC support**:
+```bash
+nvidia-smi topo -m  # Check GPU topology
+```
+
+2. **Fall back to standard sync**:
+```bash
+# Remove --use-zero-copy-sync
+```
+
+3. **Increase bucket size**:
+```bash
+--sync-bucket-size 2147483648  # 2GB
+```
+
+### Issue: Slow Weight Sync Despite Zero-Copy
+
+**Symptoms**: Weight sync still slow
+
+**Solutions**:
+
+1. **Use colocated mode**:
+```bash
+--colocate
+```
+
+2. **Enable async weight transfer**:
+```bash
+--async-weight-sync
+```
+
+## MoE-Specific Issues
+
+### Issue: Expert Load Imbalance
+
+**Symptoms**: Some experts heavily loaded, others unused
+
+**Solutions**:
+
+1. **Enable load balancing loss**:
+```bash
+--aux-loss-coef 0.01
+```
+
+2. **Use capacity factor**:
+```bash
+--moe-capacity-factor 1.25
+```
+
+### Issue: Expert Parallelism OOM
+
+**Symptoms**: OOM with large MoE models
+
+**Solutions**:
+
+1. **Increase expert parallelism**:
+```bash
+--expert-model-parallel-size 8  # Increase from 4
+```
+
+2. **Reduce batch size per GPU**:
+```bash
+--micro-batch-size 1
+```
+
+3. **Enable expert offloading**:
+```bash
+--offload-experts
+```
+
+## Multi-Agent Issues
+
+### Issue: Co-Evolution Instability
+
+**Symptoms**: Agents oscillate or one dominates
+
+**Solutions**:
+
+1. **Use alternating updates**:
+```yaml
+co_evolution:
+  strategy: alternating
+```
+
+2. **Reduce co-evolution frequency**:
+```bash
+--co-evolution-interval 20  # Increase from 10
+```
+
+3. **Add population diversity**:
+```yaml
+co_evolution:
+  population_size: 4
+```
+
+## Debugging Tips
+
+### Enable Verbose Logging
+
+```bash
+--log-level DEBUG
+export MILES_DEBUG=1
+```
+
+### Check FP8 Tensors
+
+```python
+# Verify FP8 is active
+for name, param in model.named_parameters():
+    print(f"{name}: {param.dtype}")
+```
+
+### Profile Training
+
+```bash
+--profile
+--profile-dir /path/to/profile
+```
+
+### Verify R3 Is Working
+
+```python
+# Check routing is being recorded
+sample = samples[0]
+assert sample.rollout_routed_experts is not None
+assert len(sample.rollout_routed_experts) > 0
+```
+
+### Monitor GPU Memory
+
+```bash
+watch -n 1 nvidia-smi
+```
+
+## Resources
+
+- GitHub Issues: https://github.com/radixark/miles/issues
+- Unified FP8 Blog: https://lmsys.org/blog/2025-11-25-fp8-rl/
+- Train-Inference Mismatch Tutorial: https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/slime/mismatch/blog-en.md
+- SGLang Discord: Community support

package/skills/ai-research/06-post-training/openrlhf/SKILL.md

@@ -0,0 +1,249 @@
+---
+name: openrlhf-training
+description: High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.
+version: 1.0.0
+author: Orchestra Research
+license: MIT
+tags: [Post-Training, OpenRLHF, RLHF, PPO, GRPO, RLOO, DPO, Ray, vLLM, Distributed Training, Large Models, ZeRO-3]
+dependencies: [openrlhf, ray, vllm, torch, transformers, deepspeed]
+---
+
+# OpenRLHF - High-Performance RLHF Training
+
+## Quick start
+
+OpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration.
+
+**Installation**:
+```bash
+# Launch Docker container
+docker run --runtime=nvidia -it --rm --shm-size="10g" --cap-add=SYS_ADMIN \
+  -v $PWD:/openrlhf nvcr.io/nvidia/pytorch:25.02-py3 bash
+
+# Uninstall conflicts
+sudo pip uninstall xgboost transformer_engine flash_attn pynvml -y
+
+# Install OpenRLHF with vLLM
+pip install openrlhf[vllm]
+```
+
+**PPO Training** (Hybrid Engine):
+```bash
+ray start --head --node-ip-address 0.0.0.0 --num-gpus 8
+
+ray job submit --address="http://127.0.0.1:8265" \
+  --runtime-env-json='{"working_dir": "/openrlhf"}' \
+  -- python3 -m openrlhf.cli.train_ppo_ray \
+  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \
+  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \
+  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \
+  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \
+  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \
+  --colocate_all_models \
+  --vllm_gpu_memory_utilization 0.5 \
+  --pretrain OpenRLHF/Llama-3-8b-sft-mixture \
+  --reward_pretrain OpenRLHF/Llama-3-8b-rm-700k \
+  --save_path ./output/llama3-8b-rlhf \
+  --micro_train_batch_size 8 --train_batch_size 128 \
+  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \
+  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \
+  --zero_stage 3 --bf16 \
+  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \
+  --init_kl_coef 0.01 --normalize_reward \
+  --gradient_checkpointing --packing_samples \
+  --vllm_enable_sleep --deepspeed_enable_sleep
+```
+
+**GRPO Training** (Group Normalized Policy Optimization):
+```bash
+# Same command as PPO, but add:
+--advantage_estimator group_norm
+```
+
+## Common workflows
+
+### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)
+
+**Step 1: Train reward model** (DPO):
+```bash
+deepspeed --module openrlhf.cli.train_rm \
+  --save_path ./output/llama3-8b-rm \
+  --save_steps -1 --logging_steps 1 \
+  --eval_steps -1 --train_batch_size 256 \
+  --micro_train_batch_size 1 --pretrain meta-llama/Meta-Llama-3-8B \
+  --bf16 --max_epochs 1 --max_len 8192 \
+  --zero_stage 3 --learning_rate 9e-6 \
+  --dataset OpenRLHF/preference_dataset_mixture2_and_safe_pku \
+  --apply_chat_template --chosen_key chosen \
+  --rejected_key rejected --flash_attn --gradient_checkpointing
+```
+
+**Step 2: PPO training**:
+```bash
+ray start --head --node-ip-address 0.0.0.0 --num-gpus 8
+
+ray job submit --address="http://127.0.0.1:8265" \
+  -- python3 -m openrlhf.cli.train_ppo_ray \
+  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \
+  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \
+  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \
+  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \
+  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \
+  --colocate_all_models \
+  --pretrain OpenRLHF/Llama-3-8b-sft-mixture \
+  --reward_pretrain ./output/llama3-8b-rm \
+  --save_path ./output/llama3-8b-ppo \
+  --micro_train_batch_size 8 --train_batch_size 128 \
+  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \
+  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \
+  --zero_stage 3 --bf16 \
+  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \
+  --init_kl_coef 0.01 --normalize_reward \
+  --vllm_enable_sleep --deepspeed_enable_sleep
+```
+
+### Workflow 2: GRPO training (no critic model needed)
+
+Memory-efficient alternative to PPO:
+
+```bash
+ray job submit --address="http://127.0.0.1:8265" \
+  -- python3 -m openrlhf.cli.train_ppo_ray \
+  --advantage_estimator group_norm \
+  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \
+  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \
+  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \
+  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \
+  --colocate_all_models \
+  --pretrain OpenRLHF/Llama-3-8b-sft-mixture \
+  --reward_pretrain OpenRLHF/Llama-3-8b-rm-700k \
+  --save_path ./output/llama3-8b-grpo \
+  --micro_train_batch_size 8 --train_batch_size 128 \
+  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \
+  --max_epochs 1 --bf16 \
+  --actor_learning_rate 5e-7 \
+  --init_kl_coef 0.01 --use_kl_loss --kl_estimator k3 \
+  --normalize_reward --no_advantage_std_norm
+```
+
+**Key GRPO parameters**:
+- `--advantage_estimator group_norm` - Enables GRPO
+- `--use_kl_loss` - KL loss from GRPO paper
+- `--kl_estimator k3` - Loss function (k2 ≈ k1)
+- `--no_advantage_std_norm` - Disables std normalization
+
+### Workflow 3: DPO training (preference optimization)
+
+Simpler alternative without reward model:
+
+```bash
+deepspeed --module openrlhf.cli.train_dpo \
+  --save_path ./output/llama3-8b-dpo \
+  --save_steps -1 --logging_steps 1 \
+  --eval_steps -1 --train_batch_size 256 \
+  --micro_train_batch_size 2 --pretrain meta-llama/Meta-Llama-3-8B \
+  --bf16 --max_epochs 1 --max_len 8192 \
+  --zero_stage 3 --learning_rate 5e-7 --beta 0.1 \
+  --dataset OpenRLHF/preference_dataset_mixture2_and_safe_pku \
+  --apply_chat_template --chosen_key chosen \
+  --rejected_key rejected --flash_attn --gradient_checkpointing
+```
+
+## When to use vs alternatives
+
+**Use OpenRLHF when**:
+- Training large models (7B-70B+) with RL
+- Need vLLM inference acceleration
+- Want distributed architecture with Ray
+- Have multi-node GPU cluster
+- Need PPO/GRPO/RLOO/DPO in one framework
+
+**Algorithm selection**:
+- **PPO**: Maximum control, best for complex rewards
+- **GRPO**: Memory-efficient, no critic needed
+- **RLOO**: Modified PPO with per-token KL
+- **REINFORCE++**: More stable than GRPO, faster than PPO
+- **DPO**: Simplest, no reward model needed
+
+**Use alternatives instead**:
+- **TRL**: Single-node training, simpler API
+- **veRL**: ByteDance's framework for 671B models
+- **DeepSpeedChat**: Integrated with DeepSpeed ecosystem
+
+## Common issues
+
+**Issue: GPU OOM with large models**
+
+Disable model colocation:
+```bash
+# Remove --colocate_all_models flag
+# Allocate separate GPUs for each model
+--actor_num_gpus_per_node 8 \
+--critic_num_gpus_per_node 8 \
+--reward_num_gpus_per_node 8 \
+--ref_num_gpus_per_node 8
+```
+
+**Issue: DeepSpeed GPU index out of range**
+
+Set environment variable:
+```bash
+export RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1
+```
+
+**Issue: Training instability**
+
+Use Hybrid Engine instead of async:
+```bash
+--colocate_all_models \
+--vllm_enable_sleep \
+--deepspeed_enable_sleep
+```
+
+Adjust KL coefficient:
+```bash
+--init_kl_coef 0.05  # Increase from 0.01
+```
+
+**Issue: Slow generation during PPO**
+
+Enable vLLM acceleration:
+```bash
+--vllm_num_engines 4 \
+--vllm_tensor_parallel_size 2 \
+--vllm_gpu_memory_utilization 0.5
+```
+
+## Advanced topics
+
+**Hybrid Engine GPU sharing**: See [references/hybrid-engine.md](references/hybrid-engine.md) for vLLM sleep mode, DeepSpeed sleep mode, and optimal node allocation.
+
+**Algorithm comparison**: See [references/algorithm-comparison.md](references/algorithm-comparison.md) for PPO vs GRPO vs RLOO vs REINFORCE++ benchmarks and hyperparameters.
+
+**Multi-node setup**: See [references/multi-node-training.md](references/multi-node-training.md) for Ray cluster configuration and fault tolerance.
+
+**Custom reward functions**: See [references/custom-rewards.md](references/custom-rewards.md) for reinforced fine-tuning and agent RLHF.
+
+## Hardware requirements
+
+- **GPU**: NVIDIA A100/H100 recommended
+- **VRAM**:
+  - 7B model: 8× A100 40GB (Hybrid Engine)
+  - 70B model: 48× A100 80GB (vLLM:Actor:Critic = 1:1:1)
+- **Multi-node**: Ray cluster with InfiniBand recommended
+- **Docker**: NVIDIA PyTorch container 25.02+
+
+**Performance**:
+- 2× faster than DeepSpeedChat
+- vLLM inference acceleration
+- Hybrid Engine minimizes GPU idle time
+
+## Resources
+
+- Docs: https://github.com/OpenRLHF/OpenRLHF
+- Paper: https://arxiv.org/abs/2405.11143
+- Examples: https://github.com/OpenRLHF/OpenRLHF/tree/main/examples
+- Discord: Community support
+
+
+