@synsci/cli-darwin-x64 1.1.97 → 1.1.99

package/bin/skills/verl/SKILL.md

@@ -1,391 +0,0 @@
----
-name: verl-rl-training
-description: Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.
-version: 1.0.0
-author: Synthetic Sciences
-license: MIT
-tags: [Reinforcement Learning, RLHF, GRPO, PPO, Post-Training, Distributed Training]
-dependencies: [verl>=0.3.0, torch>=2.0.0, ray>=2.41.0, vllm>=0.8.2, transformers>=4.40.0]
----
-
-# verl: Volcano Engine Reinforcement Learning for LLMs
-
-verl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.
-
-## When to Use verl
-
-**Choose verl when you need:**
-- Production-ready RL training at scale (tested up to 671B parameters)
-- Flexibility to swap backends (FSDP ↔ Megatron-LM ↔ vLLM ↔ SGLang)
-- Support for multiple RL algorithms (PPO, GRPO, RLOO, REINFORCE++, DAPO)
-- Multi-turn rollout with tool calling for agentic workflows
-- Vision-language model RL training
-
-**Consider alternatives when:**
-- You need Megatron-native training → use **slime** or **miles**
-- You want PyTorch-native abstractions with Monarch → use **torchforge**
-- You only need simple SFT/DPO → use **TRL** or **Axolotl**
-
-## Key Features
-
-- **Training backends**: FSDP, FSDP2, Megatron-LM
-- **Rollout engines**: vLLM, SGLang, HuggingFace Transformers
-- **Algorithms**: PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, SPPO
-- **Models**: Qwen-3, Llama-3.1, DeepSeek, Gemma-2 (0.5B to 671B)
-- **Advanced**: LoRA RL, sequence parallelism, expert parallelism, multi-turn tools
-
-## Installation
-
-```bash
-# Option 1: pip install
-pip install verl[vllm]  # or verl[sglang] for SGLang backend
-
-# Option 2: Docker (recommended for production)
-docker pull verlai/verl:vllm011.latest
-
-# Option 3: From source
-git clone https://github.com/volcengine/verl.git
-cd verl && pip install -e .[vllm,math]
-```
-
-## Quick Start: GRPO Training
-
-```bash
-python3 -m verl.trainer.main_ppo \
-    algorithm.adv_estimator=grpo \
-    data.train_files=~/data/gsm8k/train.parquet \
-    actor_rollout_ref.model.path=Qwen/Qwen2.5-7B \
-    actor_rollout_ref.rollout.n=8 \
-    actor_rollout_ref.actor.use_kl_loss=True \
-    trainer.n_gpus_per_node=8
-```
-
-## Core Architecture
-
-verl uses a **HybridFlow** programming model separating control flow from computation:
-
-```
-┌─────────────────────────────────────────────────────────┐
-│ Single-Process Controller (Ray)                         │
-│ - Synthetic Sciencestes: rollout → reward → train → sync        │
-└─────────────────────┬───────────────────────────────────┘
-                      │
-┌─────────────────────▼───────────────────────────────────┐
-│ Multi-Process Workers                                   │
-│ ├── ActorRolloutRefWorker (policy + generation)        │
-│ ├── CriticWorker (value estimation, PPO only)          │
-│ └── RewardManager (model-based or rule-based rewards)  │
-└─────────────────────────────────────────────────────────┘
-```
-
----
-
-## Workflow 1: Math Reasoning with GRPO
-
-Use this workflow for training reasoning models on math tasks like GSM8K or MATH.
-
-### Prerequisites Checklist
-- [ ] GPU cluster with 8+ GPUs (H100 recommended)
-- [ ] Dataset in parquet format with `prompt` and `reward_model` columns
-- [ ] Base model from HuggingFace Hub
-
-### Step 1: Prepare Dataset
-
-```python
-import pandas as pd
-
-data = [
-    {
-        "prompt": [{"role": "user", "content": "What is 15 + 27?"}],
-        "reward_model": {"ground_truth": "42"}
-    },
-    # ... more examples
-]
-df = pd.DataFrame(data)
-df.to_parquet("train.parquet")
-```
-
-### Step 2: Define Reward Function
-
-```python
-# reward_function.py
-import re
-
-def compute_reward(responses, ground_truths):
-    rewards = []
-    for response, gt in zip(responses, ground_truths):
-        # Extract answer from response
-        match = re.search(r'\\boxed{([^}]+)}', response)
-        if match and match.group(1).strip() == gt.strip():
-            rewards.append(1.0)
-        else:
-            rewards.append(0.0)
-    return rewards
-```
-
-### Step 3: Create Training Config
-
-```yaml
-# config/grpo_math.yaml
-algorithm:
-  adv_estimator: grpo
-  gamma: 1.0
-  lam: 1.0
-
-data:
-  train_files: /path/to/train.parquet
-  val_files: /path/to/val.parquet
-  train_batch_size: 256
-  max_prompt_length: 512
-  max_response_length: 2048
-
-actor_rollout_ref:
-  model:
-    path: Qwen/Qwen2.5-7B-Instruct
-  actor:
-    use_kl_loss: true
-    kl_loss_coef: 0.001
-    ppo_mini_batch_size: 64
-  rollout:
-    name: vllm
-    n: 8  # samples per prompt
-    temperature: 0.7
-    top_p: 0.95
-
-trainer:
-  total_epochs: 3
-  n_gpus_per_node: 8
-  save_freq: 100
-```
-
-### Step 4: Launch Training
-
-```bash
-python3 -m verl.trainer.main_ppo \
-    --config-path config \
-    --config-name grpo_math \
-    trainer.experiment_name=grpo_math_qwen7b
-```
-
-### Step 5: Monitor and Validate
-- [ ] Check WandB/TensorBoard for loss curves
-- [ ] Verify reward is increasing over steps
-- [ ] Run evaluation on held-out test set
-
----
-
-## Workflow 2: PPO with Critic Model
-
-Use this workflow when you need value-based advantage estimation (GAE).
-
-### Key Differences from GRPO
-- Requires separate critic model
-- Uses Generalized Advantage Estimation (GAE)
-- Better for tasks with dense rewards
-
-### Configuration
-
-```yaml
-algorithm:
-  adv_estimator: gae  # Use GAE instead of GRPO
-  gamma: 0.99
-  lam: 0.95
-
-critic:
-  model:
-    path: Qwen/Qwen2.5-7B-Instruct  # Can be same or different from actor
-  ppo_mini_batch_size: 64
-
-actor_rollout_ref:
-  actor:
-    use_kl_loss: true
-    kl_loss_coef: 0.02
-    clip_ratio: 0.2  # PPO clipping
-```
-
-### Launch with Critic
-
-```bash
-python3 -m verl.trainer.main_ppo \
-    algorithm.adv_estimator=gae \
-    critic.model.path=Qwen/Qwen2.5-7B-Instruct \
-    trainer.n_gpus_per_node=8
-```
-
----
-
-## Workflow 3: Large-Scale Training with Megatron
-
-Use this workflow for models >70B parameters or when you need expert parallelism.
-
-### Prerequisites
-- [ ] Install Megatron-LM bridge: `pip install mbridge`
-- [ ] Convert model to Megatron format
-- [ ] Multi-node cluster with NVLink/InfiniBand
-
-### Configuration for 70B+ Models
-
-```yaml
-actor_rollout_ref:
-  model:
-    path: /path/to/megatron/checkpoint
-    backend: megatron
-  actor:
-    strategy: megatron
-    tensor_model_parallel_size: 8
-    pipeline_model_parallel_size: 2
-  rollout:
-    name: vllm
-    tensor_parallel_size: 8
-```
-
-### Launch Multi-Node
-
-```bash
-# On head node
-ray start --head --port=6379
-
-# On worker nodes
-ray start --address='head_ip:6379'
-
-# Launch training
-python3 -m verl.trainer.main_ppo \
-    trainer.nnodes=4 \
-    trainer.n_gpus_per_node=8
-```
-
----
-
-## Configuration Reference
-
-### Algorithm Selection
-
-| Algorithm | `adv_estimator` | Use Case |
-|-----------|-----------------|----------|
-| GRPO | `grpo` | Critic-free, math/reasoning |
-| PPO/GAE | `gae` | Dense rewards, value estimation |
-| REINFORCE++ | `reinforce_plus_plus` | Variance reduction |
-| RLOO | `rloo` | Leave-one-out baseline |
-| ReMax | `remax` | Maximum reward baseline |
-| OPO | `opo` | Optimal policy optimization |
-
-### Key Parameters
-
-```yaml
-# Rollout parameters
-actor_rollout_ref.rollout.n: 8              # Samples per prompt
-actor_rollout_ref.rollout.temperature: 0.7  # Sampling temperature
-actor_rollout_ref.rollout.top_p: 0.95       # Nucleus sampling
-
-# Training parameters
-actor_rollout_ref.actor.lr: 1e-6            # Learning rate
-actor_rollout_ref.actor.ppo_mini_batch_size: 64
-actor_rollout_ref.actor.clip_ratio: 0.2     # PPO clip range
-
-# KL control
-actor_rollout_ref.actor.use_kl_loss: true
-actor_rollout_ref.actor.kl_loss_coef: 0.001
-algorithm.kl_ctrl.target_kl: 0.1            # For adaptive KL control
-```
-
----
-
-## Common Issues and Solutions
-
-### Issue: OOM During Rollout
-
-**Symptoms**: CUDA out of memory during generation phase
-
-**Solutions**:
-```yaml
-# Reduce batch size
-actor_rollout_ref.rollout.log_prob_micro_batch_size: 4
-
-# Enable gradient checkpointing
-actor_rollout_ref.model.enable_gradient_checkpointing: true
-
-# Use FSDP2 with CPU offloading
-actor_rollout_ref.actor.strategy: fsdp2
-actor_rollout_ref.actor.fsdp_config.offload_policy: true
-```
-
-### Issue: Training Instability
-
-**Symptoms**: Loss spikes, reward collapse
-
-**Solutions**:
-```yaml
-# Reduce learning rate
-actor_rollout_ref.actor.lr: 5e-7
-
-# Increase KL penalty
-actor_rollout_ref.actor.kl_loss_coef: 0.01
-
-# Enable gradient clipping
-actor_rollout_ref.actor.max_grad_norm: 1.0
-```
-
-### Issue: Slow Weight Sync
-
-**Symptoms**: Long pauses between rollout and training
-
-**Solutions**:
-```bash
-# Use FSDP2 for faster resharding
-actor_rollout_ref.actor.strategy=fsdp2
-
-# Enable async weight transfer
-trainer.async_weight_update=true
-```
-
-### Issue: vLLM Version Mismatch
-
-**Symptoms**: Import errors or generation failures
-
-**Solution**: Use compatible versions:
-```bash
-pip install vllm>=0.8.5,<=0.12.0
-# Avoid vLLM 0.7.x (known bugs)
-```
-
----
-
-## Advanced Topics
-
-### Multi-Turn Tool Calling
-
-See [references/multi-turn.md](references/multi-turn.md) for agentic workflows with tool use.
-
-### Vision-Language Models
-
-```yaml
-actor_rollout_ref:
-  model:
-    path: Qwen/Qwen2.5-VL-7B-Instruct
-  rollout:
-    name: vllm
-    enable_vision: true
-```
-
-### LoRA Training
-
-```yaml
-actor_rollout_ref:
-  actor:
-    lora:
-      enabled: true
-      r: 16
-      alpha: 32
-      target_modules: ["q_proj", "v_proj"]
-```
-
----
-
-## Resources
-
-- **Documentation**: https://verl.readthedocs.io/
-- **Paper**: https://arxiv.org/abs/2409.19256
-- **GitHub**: https://github.com/volcengine/verl
-- **Recipes**: https://github.com/verl-project/verl-recipe (DAPO, GSPO, etc.)
-- **Community**: Slack at verl-project
-

package/bin/skills/verl/references/api-reference.md

@@ -1,301 +0,0 @@
-# verl API Reference
-
-## Core Classes
-
-### RayPPOTrainer
-
-The central controller for the training loop. Manages resource allocation and coordinates worker groups.
-
-```python
-from verl import RayPPOTrainer
-
-trainer = RayPPOTrainer(
-    config=config,
-    resource_pool_manager=resource_manager,
-    ray_worker_group_cls=RayWorkerGroup,
-)
-trainer.init_workers()
-trainer.fit()
-```
-
-### ResourcePoolManager
-
-Manages GPU allocation across different worker groups using Ray PlacementGroups.
-
-```python
-from verl.trainer.ppo.resource_pool import ResourcePoolManager
-
-manager = ResourcePoolManager(
-    resource_pool_spec={
-        "actor_rollout_ref": {"gpu": 4},
-        "critic": {"gpu": 2},
-    }
-)
-```
-
-### RayWorkerGroup
-
-Abstraction for distributed method execution. Spawns Ray actors and dispatches method calls.
-
-```python
-from verl.trainer.ppo.ray_worker_group import RayWorkerGroup
-
-worker_group = RayWorkerGroup(
-    num_workers=8,
-    worker_cls=ActorRolloutRefWorker,
-    resource_pool=pool,
-)
-```
-
-### ActorRolloutRefWorker
-
-Worker class implementing policy training, generation, and reference model computations. Manages hybrid engine mode switching.
-
-```python
-# Typically configured via YAML, not instantiated directly
-# See configuration section below
-```
-
-### RolloutReplica
-
-Interface for inference backends with implementations for vLLM, SGLang, TensorRT-LLM, and HuggingFace.
-
-```python
-from verl.workers.rollout import RolloutReplica
-
-# Backend selection via config
-rollout:
-  name: vllm  # or: sglang, hf, tensorrt-llm
-```
-
-## Configuration Schema
-
-### PPO Configuration (`verl/trainer/config/ppo_trainer.yaml`)
-
-```yaml
-# Data configuration
-data:
-  train_files: /path/to/train.parquet
-  val_files: /path/to/val.parquet
-  train_batch_size: 256        # Global batch size of prompts
-  max_prompt_length: 512
-  max_response_length: 2048
-
-# Algorithm configuration
-algorithm:
-  adv_estimator: gae           # gae, grpo, rloo, reinforce_plus_plus
-  gamma: 0.99                  # Discount factor
-  lam: 0.95                    # GAE lambda
-  use_kl_in_reward: false      # Add KL term to reward
-
-# Actor configuration
-actor_rollout_ref:
-  model:
-    path: Qwen/Qwen2.5-7B-Instruct
-    backend: fsdp              # fsdp, fsdp2, megatron
-  actor:
-    ppo_mini_batch_size: 64    # Mini-batch for actor updates
-    ppo_epochs: 1              # Number of actor update epochs
-    clip_ratio: 0.2            # PPO clip range
-    use_kl_loss: true          # Use KL loss in actor
-    kl_loss_coef: 0.001        # KL loss coefficient
-    kl_loss_type: low_var      # KL divergence calculation method
-    loss_agg_mode: token-mean  # token-mean or sequence-mean
-    gradient_checkpointing: true
-    max_grad_norm: 1.0         # Gradient clipping
-    lr: 1e-6                   # Learning rate
-  rollout:
-    name: vllm                 # vllm, sglang, hf
-    n: 8                       # Samples per prompt
-    temperature: 0.7
-    top_p: 0.95
-    log_prob_micro_batch_size: 8
-
-# Critic configuration (PPO only)
-critic:
-  model:
-    path: Qwen/Qwen2.5-7B-Instruct
-  ppo_mini_batch_size: 64
-  ppo_epochs: 1                # Defaults to actor epochs
-
-# Trainer configuration
-trainer:
-  total_epochs: 3
-  n_gpus_per_node: 8
-  nnodes: 1
-  save_freq: 100
-  experiment_name: my_experiment
-  async_weight_update: false
-```
-
-### GRPO Configuration (`docs/algo/grpo.md`)
-
-```yaml
-algorithm:
-  adv_estimator: grpo          # Enable GRPO
-  gamma: 1.0
-  lam: 1.0
-
-actor_rollout_ref:
-  rollout:
-    n: 8                       # Must be > 1 for GRPO
-  actor:
-    use_kl_loss: true          # Required for GRPO
-    kl_loss_coef: 0.001
-    kl_loss_type: low_var      # or: k1, k2, k3
-    loss_agg_mode: token-mean
-```
-
-### Multi-Turn Configuration (`verl/trainer/config/rollout/rollout.yaml`)
-
-```yaml
-actor_rollout_ref:
-  rollout:
-    name: sglang               # Required for multi-turn
-    multi_turn:
-      enable: true
-      tool_config_path: /path/to/tools.yaml
-      interaction_config_path: /path/to/interaction.yaml
-```
-
-## Reward Functions
-
-### Built-in Reward Types
-
-```yaml
-# Model-based reward
-reward_model:
-  path: OpenRLHF/Llama-3-8b-rm-700k
-
-# Custom function-based reward
-custom_reward_function:
-  path: /path/to/reward.py
-  name: compute_score          # Function name, default: compute_score
-```
-
-### Custom Reward Function Signature
-
-```python
-# reward.py
-def compute_score(responses: list[str], ground_truths: list[str], **kwargs) -> list[float]:
-    """
-    Compute rewards for a batch of responses.
-
-    Args:
-        responses: Generated completions
-        ground_truths: Expected answers from data
-        **kwargs: Additional metadata
-
-    Returns:
-        List of reward scores (floats)
-    """
-    rewards = []
-    for response, gt in zip(responses, ground_truths):
-        # Your reward logic
-        score = 1.0 if correct(response, gt) else 0.0
-        rewards.append(score)
-    return rewards
-```
-
-## Backend-Specific Configuration
-
-### FSDP Configuration
-
-```yaml
-actor_rollout_ref:
-  actor:
-    strategy: fsdp
-    fsdp_config:
-      mixed_precision: bf16
-      sharding_strategy: FULL_SHARD
-      offload_policy: false
-```
-
-### FSDP2 Configuration
-
-```yaml
-actor_rollout_ref:
-  actor:
-    strategy: fsdp2
-    fsdp_config:
-      offload_policy: true     # CPU offloading
-      reshard_after_forward: true
-```
-
-### Megatron Configuration
-
-```yaml
-actor_rollout_ref:
-  model:
-    backend: megatron
-  actor:
-    strategy: megatron
-    tensor_model_parallel_size: 8
-    pipeline_model_parallel_size: 2
-    megatron:
-      use_mbridge: true        # Required for format conversion
-```
-
-### vLLM Rollout Configuration
-
-```yaml
-actor_rollout_ref:
-  rollout:
-    name: vllm
-    tensor_parallel_size: 2
-    gpu_memory_utilization: 0.9
-    max_num_seqs: 256
-    enforce_eager: false
-```
-
-### SGLang Rollout Configuration
-
-```yaml
-actor_rollout_ref:
-  rollout:
-    name: sglang
-    tp_size: 2
-    mem_fraction_static: 0.8
-    context_length: 8192
-```
-
-## Algorithm Reference
-
-| Algorithm | `adv_estimator` | Requires Critic | Best For |
-|-----------|-----------------|-----------------|----------|
-| PPO | `gae` | Yes | Dense rewards, value estimation |
-| GRPO | `grpo` | No | Sparse rewards, math/reasoning |
-| RLOO | `rloo` | No | Leave-one-out baseline |
-| REINFORCE++ | `reinforce_plus_plus` | No | Variance reduction |
-| DAPO | `dapo` | No | Doubly-adaptive optimization |
-
-## Vision-Language Model Support
-
-```yaml
-actor_rollout_ref:
-  model:
-    path: Qwen/Qwen2.5-VL-7B-Instruct
-  rollout:
-    name: vllm
-    enable_vision: true
-    max_model_len: 32768
-```
-
-## LoRA Configuration
-
-```yaml
-actor_rollout_ref:
-  actor:
-    lora:
-      enabled: true
-      r: 16
-      alpha: 32
-      target_modules: ["q_proj", "v_proj", "k_proj", "o_proj"]
-      dropout: 0.05
-```
-
-## Resources
-
-- Documentation: https://verl.readthedocs.io/
-- GitHub: https://github.com/volcengine/verl
-- Paper: https://arxiv.org/abs/2409.19256 (HybridFlow)