@synsci/cli-darwin-x64 1.1.49

package/bin/skills/openrlhf/SKILL.md

@@ -0,0 +1,249 @@
+---
+name: openrlhf-training
+description: High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.
+version: 1.0.0
+author: Synthetic Sciences
+license: MIT
+tags: [Post-Training, OpenRLHF, RLHF, PPO, GRPO, RLOO, DPO, Ray, vLLM, Distributed Training, Large Models, ZeRO-3]
+dependencies: [openrlhf, ray, vllm, torch, transformers, deepspeed]
+---
+
+# OpenRLHF - High-Performance RLHF Training
+
+## Quick start
+
+OpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration.
+
+**Installation**:
+```bash
+# Launch Docker container
+docker run --runtime=nvidia -it --rm --shm-size="10g" --cap-add=SYS_ADMIN \
+  -v $PWD:/openrlhf nvcr.io/nvidia/pytorch:25.02-py3 bash
+
+# Uninstall conflicts
+sudo pip uninstall xgboost transformer_engine flash_attn pynvml -y
+
+# Install OpenRLHF with vLLM
+pip install openrlhf[vllm]
+```
+
+**PPO Training** (Hybrid Engine):
+```bash
+ray start --head --node-ip-address 0.0.0.0 --num-gpus 8
+
+ray job submit --address="http://127.0.0.1:8265" \
+  --runtime-env-json='{"working_dir": "/openrlhf"}' \
+  -- python3 -m openrlhf.cli.train_ppo_ray \
+  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \
+  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \
+  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \
+  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \
+  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \
+  --colocate_all_models \
+  --vllm_gpu_memory_utilization 0.5 \
+  --pretrain OpenRLHF/Llama-3-8b-sft-mixture \
+  --reward_pretrain OpenRLHF/Llama-3-8b-rm-700k \
+  --save_path ./output/llama3-8b-rlhf \
+  --micro_train_batch_size 8 --train_batch_size 128 \
+  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \
+  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \
+  --zero_stage 3 --bf16 \
+  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \
+  --init_kl_coef 0.01 --normalize_reward \
+  --gradient_checkpointing --packing_samples \
+  --vllm_enable_sleep --deepspeed_enable_sleep
+```
+
+**GRPO Training** (Group Normalized Policy Optimization):
+```bash
+# Same command as PPO, but add:
+--advantage_estimator group_norm
+```
+
+## Common workflows
+
+### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)
+
+**Step 1: Train reward model** (DPO):
+```bash
+deepspeed --module openrlhf.cli.train_rm \
+  --save_path ./output/llama3-8b-rm \
+  --save_steps -1 --logging_steps 1 \
+  --eval_steps -1 --train_batch_size 256 \
+  --micro_train_batch_size 1 --pretrain meta-llama/Meta-Llama-3-8B \
+  --bf16 --max_epochs 1 --max_len 8192 \
+  --zero_stage 3 --learning_rate 9e-6 \
+  --dataset OpenRLHF/preference_dataset_mixture2_and_safe_pku \
+  --apply_chat_template --chosen_key chosen \
+  --rejected_key rejected --flash_attn --gradient_checkpointing
+```
+
+**Step 2: PPO training**:
+```bash
+ray start --head --node-ip-address 0.0.0.0 --num-gpus 8
+
+ray job submit --address="http://127.0.0.1:8265" \
+  -- python3 -m openrlhf.cli.train_ppo_ray \
+  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \
+  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \
+  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \
+  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \
+  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \
+  --colocate_all_models \
+  --pretrain OpenRLHF/Llama-3-8b-sft-mixture \
+  --reward_pretrain ./output/llama3-8b-rm \
+  --save_path ./output/llama3-8b-ppo \
+  --micro_train_batch_size 8 --train_batch_size 128 \
+  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \
+  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \
+  --zero_stage 3 --bf16 \
+  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \
+  --init_kl_coef 0.01 --normalize_reward \
+  --vllm_enable_sleep --deepspeed_enable_sleep
+```
+
+### Workflow 2: GRPO training (no critic model needed)
+
+Memory-efficient alternative to PPO:
+
+```bash
+ray job submit --address="http://127.0.0.1:8265" \
+  -- python3 -m openrlhf.cli.train_ppo_ray \
+  --advantage_estimator group_norm \
+  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \
+  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \
+  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \
+  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \
+  --colocate_all_models \
+  --pretrain OpenRLHF/Llama-3-8b-sft-mixture \
+  --reward_pretrain OpenRLHF/Llama-3-8b-rm-700k \
+  --save_path ./output/llama3-8b-grpo \
+  --micro_train_batch_size 8 --train_batch_size 128 \
+  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \
+  --max_epochs 1 --bf16 \
+  --actor_learning_rate 5e-7 \
+  --init_kl_coef 0.01 --use_kl_loss --kl_estimator k3 \
+  --normalize_reward --no_advantage_std_norm
+```
+
+**Key GRPO parameters**:
+- `--advantage_estimator group_norm` - Enables GRPO
+- `--use_kl_loss` - KL loss from GRPO paper
+- `--kl_estimator k3` - Loss function (k2 ≈ k1)
+- `--no_advantage_std_norm` - Disables std normalization
+
+### Workflow 3: DPO training (preference optimization)
+
+Simpler alternative without reward model:
+
+```bash
+deepspeed --module openrlhf.cli.train_dpo \
+  --save_path ./output/llama3-8b-dpo \
+  --save_steps -1 --logging_steps 1 \
+  --eval_steps -1 --train_batch_size 256 \
+  --micro_train_batch_size 2 --pretrain meta-llama/Meta-Llama-3-8B \
+  --bf16 --max_epochs 1 --max_len 8192 \
+  --zero_stage 3 --learning_rate 5e-7 --beta 0.1 \
+  --dataset OpenRLHF/preference_dataset_mixture2_and_safe_pku \
+  --apply_chat_template --chosen_key chosen \
+  --rejected_key rejected --flash_attn --gradient_checkpointing
+```
+
+## When to use vs alternatives
+
+**Use OpenRLHF when**:
+- Training large models (7B-70B+) with RL
+- Need vLLM inference acceleration
+- Want distributed architecture with Ray
+- Have multi-node GPU cluster
+- Need PPO/GRPO/RLOO/DPO in one framework
+
+**Algorithm selection**:
+- **PPO**: Maximum control, best for complex rewards
+- **GRPO**: Memory-efficient, no critic needed
+- **RLOO**: Modified PPO with per-token KL
+- **REINFORCE++**: More stable than GRPO, faster than PPO
+- **DPO**: Simplest, no reward model needed
+
+**Use alternatives instead**:
+- **TRL**: Single-node training, simpler API
+- **veRL**: ByteDance's framework for 671B models
+- **DeepSpeedChat**: Integrated with DeepSpeed ecosystem
+
+## Common issues
+
+**Issue: GPU OOM with large models**
+
+Disable model colocation:
+```bash
+# Remove --colocate_all_models flag
+# Allocate separate GPUs for each model
+--actor_num_gpus_per_node 8 \
+--critic_num_gpus_per_node 8 \
+--reward_num_gpus_per_node 8 \
+--ref_num_gpus_per_node 8
+```
+
+**Issue: DeepSpeed GPU index out of range**
+
+Set environment variable:
+```bash
+export RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1
+```
+
+**Issue: Training instability**
+
+Use Hybrid Engine instead of async:
+```bash
+--colocate_all_models \
+--vllm_enable_sleep \
+--deepspeed_enable_sleep
+```
+
+Adjust KL coefficient:
+```bash
+--init_kl_coef 0.05  # Increase from 0.01
+```
+
+**Issue: Slow generation during PPO**
+
+Enable vLLM acceleration:
+```bash
+--vllm_num_engines 4 \
+--vllm_tensor_parallel_size 2 \
+--vllm_gpu_memory_utilization 0.5
+```
+
+## Advanced topics
+
+**Hybrid Engine GPU sharing**: See [references/hybrid-engine.md](references/hybrid-engine.md) for vLLM sleep mode, DeepSpeed sleep mode, and optimal node allocation.
+
+**Algorithm comparison**: See [references/algorithm-comparison.md](references/algorithm-comparison.md) for PPO vs GRPO vs RLOO vs REINFORCE++ benchmarks and hyperparameters.
+
+**Multi-node setup**: See [references/multi-node-training.md](references/multi-node-training.md) for Ray cluster configuration and fault tolerance.
+
+**Custom reward functions**: See [references/custom-rewards.md](references/custom-rewards.md) for reinforced fine-tuning and agent RLHF.
+
+## Hardware requirements
+
+- **GPU**: NVIDIA A100/H100 recommended
+- **VRAM**:
+  - 7B model: 8× A100 40GB (Hybrid Engine)
+  - 70B model: 48× A100 80GB (vLLM:Actor:Critic = 1:1:1)
+- **Multi-node**: Ray cluster with InfiniBand recommended
+- **Docker**: NVIDIA PyTorch container 25.02+
+
+**Performance**:
+- 2× faster than DeepSpeedChat
+- vLLM inference acceleration
+- Hybrid Engine minimizes GPU idle time
+
+## Resources
+
+- Docs: https://github.com/OpenRLHF/OpenRLHF
+- Paper: https://arxiv.org/abs/2405.11143
+- Examples: https://github.com/OpenRLHF/OpenRLHF/tree/main/examples
+- Discord: Community support
+
+
+

package/bin/skills/openrlhf/references/algorithm-comparison.md

@@ -0,0 +1,404 @@
+# Algorithm Comparison
+
+Complete guide to RL algorithms in OpenRLHF: PPO, REINFORCE++, GRPO, RLOO, and their variants.
+
+## Overview
+
+OpenRLHF supports 6 RL algorithms selectable via `--advantage_estimator`:
+- **gae** - PPO with Generalized Advantage Estimation
+- **reinforce** - REINFORCE++ (PPO optimizations without critic)
+- **reinforce_baseline** - REINFORCE++ with baseline
+- **group_norm** - GRPO (Group Normalized Policy Optimization)
+- **dr_grpo** - Dr. GRPO (GRPO without std normalization)
+- **rloo** - Reinforcement Learning with Online Off-policy Correction
+
+## Algorithm Details
+
+### PPO (Proximal Policy Optimization)
+
+**Formula**:
+```
+loss = -min(ratio * advantages, clip(ratio, 1-ε, 1+ε) * advantages)
+ratio = π_new(a|s) / π_old(a|s)
+```
+
+**Characteristics**:
+- **Stability**: High (clipped objective prevents large updates)
+- **Memory**: High (stores actor + critic experiences)
+- **Speed**: Medium (critic training overhead)
+- **Requires**: Critic network for value estimation
+
+**Implementation**:
+```python
+surr1 = ratio * advantages
+surr2 = ratio.clamp(1 - clip_eps_low, 1 + clip_eps_high) * advantages
+loss = -torch.min(surr1, surr2)
+```
+
+**When to use**:
+- General-purpose RLHF
+- Complex reward functions
+- Need stable training
+
+**Hyperparameters**:
+```bash
+--advantage_estimator gae  # Enable PPO
+--clip_eps_low 0.2         # Clipping lower bound
+--clip_eps_high 0.2        # Clipping upper bound
+--actor_learning_rate 1e-6
+--critic_learning_rate 9e-6
+--init_kl_coef 0.01
+```
+
+### REINFORCE++
+
+**Formula**:
+```
+loss = -ratio * advantages  (with PPO-clip)
+advantages = cumulative_returns - baseline
+```
+
+**Characteristics**:
+- **Stability**: Higher than GRPO
+- **Memory**: Lower (no critic network)
+- **Speed**: Faster than PPO
+- **Requires**: No critic network
+
+**Key innovation**: Integrates PPO optimizations (advantage normalization, PPO-clip loss) into REINFORCE while eliminating critic network overhead.
+
+**When to use**:
+- Want PPO stability without critic
+- Limited memory budget
+- Fast training priority
+
+**Hyperparameters**:
+```bash
+--advantage_estimator reinforce
+--critic_pretrain None  # No critic needed
+--init_kl_coef 0.01
+--actor_learning_rate 1e-6
+```
+
+### REINFORCE++-baseline
+
+**Formula**:
+```
+rewards = rewards - mean(rewards_same_prompt)
+```
+
+**Characteristics**:
+- **Stability**: Very high
+- **Memory**: Lower (no critic)
+- **Speed**: Faster than PPO
+- **Requires**: Multiple samples per prompt
+
+**Key innovation**: Uses mean reward of multiple samples from same prompt as baseline to reshape rewards.
+
+**When to use**:
+- RLVR (Reinforcement Learning via Verifier Rewards) settings
+- Reward patterns vary (0/1/-0.5)
+- Multiple samples per prompt available
+
+**Hyperparameters**:
+```bash
+--advantage_estimator reinforce_baseline
+--n_samples_per_prompt 4  # Must be > 1
+--init_kl_coef 0.01
+```
+
+### GRPO (Group Normalized Policy Optimization)
+
+**Formula**:
+```
+rewards = (rewards - mean(rewards)) / (std(rewards) + 1e-9)
+loss = -ratio * normalized_advantages
+KL loss (optional): k1, k2, or k3 estimator
+```
+
+**Characteristics**:
+- **Stability**: Lower than REINFORCE++
+- **Memory**: Lower (no critic)
+- **Speed**: Fast
+- **Requires**: Group reward normalization
+
+**Key innovation**: Group-based advantage normalization with optional KL loss.
+
+**When to use**:
+- Exploring policy optimization variants
+- Need reward normalization
+- Memory-constrained
+
+**Hyperparameters**:
+```bash
+--advantage_estimator group_norm
+--use_kl_loss                # Enable KL loss
+--kl_estimator k3            # k3 for loss, k2 ≈ k1
+--init_kl_coef 0.01
+--no_advantage_std_norm      # Optional: disable std norm
+```
+
+**KL estimator variance**:
+- **k3**: Larger variance under categorical distribution
+- **k1, k2**: Similar variance, k2 ≈ k1 for loss
+
+### Dr. GRPO
+
+**Formula**:
+```
+rewards = rewards - mean(rewards)  # No std normalization
+```
+
+**Characteristics**:
+- **Stability**: Similar to GRPO
+- **Memory**: Lower (no critic)
+- **Speed**: Fast
+- **Requires**: Group mean normalization only
+
+**Key innovation**: Removes local group normalization `/std` from GRPO (not needed in RL variance reduction theory).
+
+**When to use**:
+- GRPO variant experimentation
+- Avoid std normalization issues
+
+**Hyperparameters**:
+```bash
+--advantage_estimator dr_grpo
+--init_kl_coef 0.01
+```
+
+### RLOO (RL with Online Off-policy Correction)
+
+**Formula**:
+```
+baseline = (sum(rewards) - rewards) / (n_samples - 1)
+rewards = rewards - baseline
+loss = -ratio * advantages  (with PPO-clip)
+```
+
+**Characteristics**:
+- **Stability**: High (PPO-clip)
+- **Memory**: Lower (no critic)
+- **Speed**: Fast
+- **Requires**: Multiple samples per prompt, per-token KL
+
+**Key innovation**: Incorporates per-token KL reward and PPO-clip loss.
+
+**When to use**:
+- Need per-token KL rewards
+- Want PPO stability without critic
+- Multiple samples per prompt
+
+**Hyperparameters**:
+```bash
+--advantage_estimator rloo
+--n_samples_per_prompt 4  # Must be > 1
+--init_kl_coef 0.01
+```
+
+## Comparison Table
+
+| Algorithm | Critic | Stability | Memory | Speed | Best For |
+|-----------|--------|-----------|--------|-------|----------|
+| PPO | ✅ Yes | ⭐⭐⭐⭐⭐ | High | Medium | General purpose |
+| REINFORCE++ | ❌ No | ⭐⭐⭐⭐ | Low | **Fast** | Critic-free PPO |
+| REINFORCE++-baseline | ❌ No | ⭐⭐⭐⭐⭐ | Low | **Fast** | RLVR settings |
+| GRPO | ❌ No | ⭐⭐⭐ | Low | Fast | Reward normalization |
+| Dr. GRPO | ❌ No | ⭐⭐⭐ | Low | Fast | GRPO variant |
+| RLOO | ❌ No | ⭐⭐⭐⭐ | Low | Fast | Per-token KL |
+
+## Experience Data Structure
+
+**PPO (with critic)**:
+```python
+@dataclass
+class Experience:
+    sequences: torch.Tensor       # Token sequences
+    attention_mask: torch.Tensor  # Attention masks
+    action_mask: torch.Tensor     # Action masks
+    action_log_probs: torch.Tensor # Log π(a|s)
+    values: torch.Tensor          # Critic value estimates
+    returns: torch.Tensor         # Cumulative returns
+    advantages: torch.Tensor      # GAE advantages
+    reward: float                 # Total reward
+    kl: torch.Tensor             # KL divergence
+```
+
+**REINFORCE++ (no critic)**:
+```python
+# No values, returns, or advantages stored
+# Only sequences, log_probs, and rewards
+```
+
+## Memory Comparison (7B Model)
+
+| Algorithm | Components | Memory (8× A100) |
+|-----------|-----------|------------------|
+| PPO | Actor + Critic + Reward + Ref | ~40GB |
+| REINFORCE++ | Actor + Reward + Ref | ~28GB |
+| GRPO | Actor + Reward + Ref | ~28GB |
+| RLOO | Actor + Reward + Ref | ~28GB |
+
+**Savings**: ~30% memory reduction without critic
+
+## Speed Comparison
+
+**Relative training time** (7B model, 1000 steps):
+- PPO: 1.0× baseline
+- REINFORCE++: **0.75×** (25% faster)
+- GRPO: 0.80×
+- RLOO: 0.80×
+
+**Why REINFORCE++ is faster**:
+- No critic training
+- No value function updates
+- Fewer backward passes
+
+## Choosing an Algorithm
+
+### Decision Tree
+
+```
+Need maximum stability?
+  ├─ Yes → PPO (with critic)
+  └─ No ↓
+
+Have multiple samples per prompt?
+  ├─ Yes ↓
+  │   └─ RLVR setting with varying rewards?
+  │       ├─ Yes → REINFORCE++-baseline
+  │       └─ No → RLOO (if need per-token KL)
+  └─ No ↓
+
+Want faster than PPO?
+  └─ Yes → REINFORCE++ (most stable critic-free)
+
+Experimenting with normalization?
+  └─ Yes → GRPO or Dr. GRPO
+```
+
+### By Use Case
+
+**Production deployment**:
+```bash
+# Maximum stability
+--advantage_estimator gae  # PPO
+--clip_eps_low 0.2
+--init_kl_coef 0.01
+```
+
+**Memory-constrained**:
+```bash
+# No critic, stable
+--advantage_estimator reinforce  # REINFORCE++
+--critic_pretrain None
+```
+
+**RLVR / Verification rewards**:
+```bash
+# Baseline reward shaping
+--advantage_estimator reinforce_baseline
+--n_samples_per_prompt 4
+```
+
+**Research / Experimentation**:
+```bash
+# Explore GRPO variants
+--advantage_estimator group_norm
+--use_kl_loss --kl_estimator k3
+```
+
+## Advanced Configuration
+
+### Reward Normalization
+
+**PPO (no manual normalization)**:
+```bash
+--advantage_estimator gae
+# GAE handles advantage normalization
+```
+
+**GRPO (group normalization)**:
+```bash
+--advantage_estimator group_norm
+--normalize_reward  # Optional additional normalization
+```
+
+**Disable std normalization**:
+```bash
+--no_advantage_std_norm  # Keep mean norm only
+```
+
+### KL Penalty Configuration
+
+**All algorithms support**:
+```bash
+--init_kl_coef 0.01    # Initial KL coefficient
+--kl_target 0.1        # Target KL divergence
+--kl_horizon 10000     # Steps to reach target
+```
+
+**GRPO-specific**:
+```bash
+--use_kl_loss          # Enable KL loss term
+--kl_estimator k3      # Loss function choice
+```
+
+### Clipping Configuration
+
+**PPO clipping**:
+```bash
+--clip_eps_low 0.2     # Lower bound
+--clip_eps_high 0.2    # Upper bound
+```
+
+**Reward clipping**:
+```bash
+--reward_clip_range 10.0  # Clip rewards to [-10, 10]
+```
+
+## Common Issues
+
+### PPO Instability
+
+**Symptom**: Large policy updates, divergence
+
+**Solution**: Reduce clipping range
+```bash
+--clip_eps_low 0.1     # Reduce from 0.2
+--clip_eps_high 0.1
+```
+
+### GRPO High Variance
+
+**Symptom**: Unstable training with GRPO
+
+**Solution**: Switch to REINFORCE++
+```bash
+--advantage_estimator reinforce  # More stable
+```
+
+### Memory OOM with PPO
+
+**Symptom**: OOM during critic training
+
+**Solution**: Switch to critic-free
+```bash
+--advantage_estimator reinforce  # No critic
+--critic_pretrain None
+```
+
+### RLOO/Baseline Requires Multiple Samples
+
+**Symptom**: `AssertionError: n_samples_per_prompt must be > 1`
+
+**Solution**:
+```bash
+--n_samples_per_prompt 4  # Minimum 2, recommended 4-8
+```
+
+## References
+
+- PPO paper: https://arxiv.org/abs/1707.06347
+- GRPO paper: https://arxiv.org/abs/2402.03300
+- OpenRLHF: https://github.com/OpenRLHF/OpenRLHF
+- OpenRLHF paper: https://arxiv.org/abs/2405.11143