@synsci/cli-darwin-x64 1.1.49

package/bin/skills/unsloth/references/llms.md

@@ -0,0 +1,82 @@
+# Unsloth Documentation
+
+## Unsloth Documentation
+
+- [Unsloth Docs](/get-started/unsloth-docs.md): Train your own model with Unsloth, an open-source framework for LLM fine-tuning and reinforcement learning.
+- [Beginner? Start here!](/get-started/beginner-start-here.md)
+- [Unsloth Requirements](/get-started/beginner-start-here/unsloth-requirements.md): Here are Unsloth's requirements including system and GPU VRAM requirements.
+- [FAQ + Is Fine-tuning Right For Me?](/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me.md): If you're stuck on if fine-tuning is right for you, see here! Learn about fine-tuning misconceptions, how it compared to RAG and more:
+- [Unsloth Notebooks](/get-started/unsloth-notebooks.md): Explore our catalog of Unsloth notebooks:
+- [All Our Models](/get-started/all-our-models.md)
+- [Install & Update](/get-started/install-and-update.md): Learn to install Unsloth locally or online.
+- [Updating](/get-started/install-and-update/updating.md): To update or use an old version of Unsloth, follow the steps below:
+- [Pip Install](/get-started/install-and-update/pip-install.md): To install Unsloth locally via Pip, follow the steps below:
+- [Docker](/get-started/install-and-update/docker.md): Install Unsloth using our official Docker container
+- [Windows Installation](/get-started/install-and-update/windows-installation.md): See how to install Unsloth on Windows with or without WSL.
+- [AMD](/get-started/install-and-update/amd.md): Fine-tune with Unsloth on AMD GPUs.
+- [Conda Install](/get-started/install-and-update/conda-install.md): To install Unsloth locally on Conda, follow the steps below:
+- [Google Colab](/get-started/install-and-update/google-colab.md): To install and run Unsloth on Google Colab, follow the steps below:
+- [Fine-tuning LLMs Guide](/get-started/fine-tuning-llms-guide.md): Learn all the basics and best practices of fine-tuning. Beginner-friendly.
+- [What Model Should I Use?](/get-started/fine-tuning-llms-guide/what-model-should-i-use.md)
+- [Datasets Guide](/get-started/fine-tuning-llms-guide/datasets-guide.md): Learn how to create & prepare a dataset for fine-tuning.
+- [LoRA Hyperparameters Guide](/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide.md): Optimal lora rank. alpha, number of epochs, batch size & gradient accumulation, QLoRA vs LoRA, target modules and more!
+- [Tutorial: How to Finetune Llama-3 and Use In Ollama](/get-started/fine-tuning-llms-guide/tutorial-how-to-finetune-llama-3-and-use-in-ollama.md): Beginner's Guide for creating a customized personal assistant (like ChatGPT) to run locally on Ollama
+- [Reinforcement Learning (RL) Guide](/get-started/reinforcement-learning-rl-guide.md): Learn all about Reinforcement Learning (RL) and how to train your own DeepSeek-R1 reasoning model with Unsloth using GRPO. A complete guide from beginner to advanced.
+- [Tutorial: Train your own Reasoning model with GRPO](/get-started/reinforcement-learning-rl-guide/tutorial-train-your-own-reasoning-model-with-grpo.md): Beginner's Guide to transforming a model like Llama 3.1 (8B) into a reasoning model by using Unsloth and GRPO.
+- [Advanced RL Documentation](/get-started/reinforcement-learning-rl-guide/advanced-rl-documentation.md): Advanced documentation settings when using Unsloth with GRPO.
+- [Memory Efficient RL](/get-started/reinforcement-learning-rl-guide/memory-efficient-rl.md)
+- [RL Reward Hacking](/get-started/reinforcement-learning-rl-guide/rl-reward-hacking.md): Learn what is Reward Hacking in Reinforcement Learning and how to counter it.
+- [GSPO Reinforcement Learning](/get-started/reinforcement-learning-rl-guide/gspo-reinforcement-learning.md): Train with GSPO (Group Sequence Policy Optimization) RL in Unsloth.
+- [Reinforcement Learning - DPO, ORPO & KTO](/get-started/reinforcement-learning-rl-guide/reinforcement-learning-dpo-orpo-and-kto.md): To use the reward modelling functions for DPO, GRPO, ORPO or KTO with Unsloth, follow the steps below:
+- [DeepSeek-OCR: How to Run & Fine-tune](/new/deepseek-ocr-how-to-run-and-fine-tune.md): Guide on how to run and fine-tune DeepSeek-OCR locally.
+- [How to Fine-tune LLMs with Unsloth & Docker](/new/how-to-fine-tune-llms-with-unsloth-and-docker.md): Learn how to fine-tune LLMs or do Reinforcement Learning (RL) with Unsloth's Docker image.
+- [Vision Reinforcement Learning (VLM RL)](/new/vision-reinforcement-learning-vlm-rl.md): Train Vision/multimodal models via GRPO and RL with Unsloth!
+- [gpt-oss Reinforcement Learning](/new/gpt-oss-reinforcement-learning.md)
+- [Tutorial: How to Train gpt-oss with RL](/new/gpt-oss-reinforcement-learning/tutorial-how-to-train-gpt-oss-with-rl.md): Learn to train OpenAI gpt-oss with GRPO to autonomously beat 2048 locally or on Colab.
+- [Unsloth Dynamic GGUFs on Aider Polyglot](/new/unsloth-dynamic-ggufs-on-aider-polyglot.md): Performance of Unsloth Dynamic GGUFs on Aider Polyglot Benchmarks
+- [Qwen3-VL: How to Run & Fine-tune](/models/qwen3-vl-how-to-run-and-fine-tune.md): Learn to fine-tune and run Qwen3-VL locally with Unsloth.
+- [gpt-oss: How to Run & Fine-tune](/models/gpt-oss-how-to-run-and-fine-tune.md): Run & fine-tune OpenAI's new open-source models!
+- [Tutorial: How to Fine-tune gpt-oss](/models/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss.md): Learn step-by-step how to train OpenAI gpt-oss locally with Unsloth.
+- [Long Context gpt-oss Training](/models/gpt-oss-how-to-run-and-fine-tune/long-context-gpt-oss-training.md)
+- [GLM-4.6: How to Run Locally](/models/glm-4.6-how-to-run-locally.md): A guide on how to run Z.ai's new GLM-4.6 model on your own local device!
+- [IBM Granite 4.0](/models/ibm-granite-4.0.md): How to run IBM Granite-4.0 with Unsloth GGUFs on llama.cpp, Ollama and how to fine-tune!
+- [DeepSeek-V3.1: How to Run Locally](/models/deepseek-v3.1-how-to-run-locally.md): A guide on how to run DeepSeek-V3.1 and Terminus on your own local device!
+- [Qwen3-Coder: How to Run Locally](/models/qwen3-coder-how-to-run-locally.md): Run Qwen3-Coder-30B-A3B-Instruct and 480B-A35B locally with Unsloth Dynamic quants.
+- [Gemma 3: How to Run & Fine-tune](/models/gemma-3-how-to-run-and-fine-tune.md): How to run Gemma 3 effectively with our GGUFs on llama.cpp, Ollama, Open WebUI and how to fine-tune with Unsloth!
+- [Gemma 3n: How to Run & Fine-tune](/models/gemma-3-how-to-run-and-fine-tune/gemma-3n-how-to-run-and-fine-tune.md): Run Google's new Gemma 3n locally with Dynamic GGUFs on llama.cpp, Ollama, Open WebUI and fine-tune with Unsloth!
+- [Qwen3: How to Run & Fine-tune](/models/qwen3-how-to-run-and-fine-tune.md): Learn to run & fine-tune Qwen3 locally with Unsloth + our Dynamic 2.0 quants
+- [Qwen3-2507](/models/qwen3-how-to-run-and-fine-tune/qwen3-2507.md): Run Qwen3-30B-A3B-2507 and 235B-A22B Thinking and Instruct versions locally on your device!
+- [Tutorials: How To Fine-tune & Run LLMs](/models/tutorials-how-to-fine-tune-and-run-llms.md): Learn how to run and fine-tune models for optimal performance 100% locally with Unsloth.
+- [DeepSeek-R1-0528: How to Run Locally](/models/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-0528-how-to-run-locally.md): A guide on how to run DeepSeek-R1-0528 including Qwen3 on your own local device!
+- [Magistral: How to Run & Fine-tune](/models/tutorials-how-to-fine-tune-and-run-llms/magistral-how-to-run-and-fine-tune.md): Meet Magistral - Mistral's new reasoning models.
+- [Llama 4: How to Run & Fine-tune](/models/tutorials-how-to-fine-tune-and-run-llms/llama-4-how-to-run-and-fine-tune.md): How to run Llama 4 locally using our dynamic GGUFs which recovers accuracy compared to standard quantization.
+- [Kimi K2: How to Run Locally](/models/tutorials-how-to-fine-tune-and-run-llms/kimi-k2-how-to-run-locally.md): Guide on running Kimi K2 and Kimi-K2-Instruct-0905 on your own local device!
+- [Grok 2](/models/tutorials-how-to-fine-tune-and-run-llms/grok-2.md): Run xAI's Grok 2 model locally!
+- [Devstral: How to Run & Fine-tune](/models/tutorials-how-to-fine-tune-and-run-llms/devstral-how-to-run-and-fine-tune.md): Run and fine-tune Mistral Devstral 1.1, including Small-2507 and 2505.
+- [DeepSeek-V3-0324: How to Run Locally](/models/tutorials-how-to-fine-tune-and-run-llms/deepseek-v3-0324-how-to-run-locally.md): How to run DeepSeek-V3-0324 locally using our dynamic quants which recovers accuracy
+- [DeepSeek-R1: How to Run Locally](/models/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally.md): A guide on how you can run our 1.58-bit Dynamic Quants for DeepSeek-R1 using llama.cpp.
+- [DeepSeek-R1 Dynamic 1.58-bit](/models/tutorials-how-to-fine-tune-and-run-llms/deepseek-r1-how-to-run-locally/deepseek-r1-dynamic-1.58-bit.md): See performance comparison tables for Unsloth's Dynamic GGUF Quants vs Standard IMatrix Quants.
+- [QwQ-32B: How to Run effectively](/models/tutorials-how-to-fine-tune-and-run-llms/qwq-32b-how-to-run-effectively.md): How to run QwQ-32B effectively with our bug fixes and without endless generations + GGUFs.
+- [Phi-4 Reasoning: How to Run & Fine-tune](/models/tutorials-how-to-fine-tune-and-run-llms/phi-4-reasoning-how-to-run-and-fine-tune.md): Learn to run & fine-tune Phi-4 reasoning models locally with Unsloth + our Dynamic 2.0 quants
+- [Running & Saving Models](/basics/running-and-saving-models.md): Learn how to save your finetuned model so you can run it in your favorite inference engine.
+- [Saving to GGUF](/basics/running-and-saving-models/saving-to-gguf.md): Saving models to 16bit for GGUF so you can use it for Ollama, Jan AI, Open WebUI and more!
+- [Saving to Ollama](/basics/running-and-saving-models/saving-to-ollama.md)
+- [Saving to vLLM for deployment](/basics/running-and-saving-models/saving-to-vllm-for-deployment.md): Saving models to 16bit for vLLM deployment and serving
+- [Saving to SGLang for deployment](/basics/running-and-saving-models/saving-to-sglang-for-deployment.md): Saving models to 16bit for SGLang for deployment and serving
+- [Unsloth Inference](/basics/running-and-saving-models/unsloth-inference.md): Learn how to run your finetuned model with Unsloth's faster inference.
+- [Troubleshooting Inference](/basics/running-and-saving-models/troubleshooting-inference.md): If you're experiencing issues when running or saving your model.
+- [vLLM Engine Arguments](/basics/running-and-saving-models/vllm-engine-arguments.md)
+- [LoRA Hot Swapping Guide](/basics/running-and-saving-models/lora-hot-swapping-guide.md)
+- [Text-to-Speech (TTS) Fine-tuning](/basics/text-to-speech-tts-fine-tuning.md): Learn how to to fine-tune TTS & STT voice models with Unsloth.
+- [Unsloth Dynamic 2.0 GGUFs](/basics/unsloth-dynamic-2.0-ggufs.md): A big new upgrade to our Dynamic Quants!
+- [Vision Fine-tuning](/basics/vision-fine-tuning.md): Learn how to fine-tune vision/multimodal LLMs with Unsloth
+- [Fine-tuning LLMs with NVIDIA DGX Spark and Unsloth](/basics/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth.md): Tutorial on how to fine-tune and do reinforcement learning (RL) with OpenAI gpt-oss on NVIDIA DGX Spark.
+- [Fine-tuning LLMs with Blackwell, RTX 50 series & Unsloth](/basics/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth.md): Learn how to fine-tune LLMs on NVIDIA's Blackwell RTX 50 series and B200 GPUs with our step-by-step guide.
+- [Multi-GPU Training with Unsloth](/basics/multi-gpu-training-with-unsloth.md): Learn how to fine-tune LLMs on multiple GPUs and parallelism with Unsloth.
+- [Finetuning from Last Checkpoint](/basics/finetuning-from-last-checkpoint.md): Checkpointing allows you to save your finetuning progress so you can pause it and then continue.
+- [Troubleshooting & FAQs](/basics/troubleshooting-and-faqs.md): Tips to solve issues, and frequently asked questions.
+- [Chat Templates](/basics/chat-templates.md): Learn the fundamentals and customization options of chat templates, including Conversational, ChatML, ShareGPT, Alpaca formats, and more!
+- [Quantization-Aware Training (QAT)](/basics/quantization-aware-training-qat.md): Quantize models to 4-bit with Unsloth and PyTorch to recover accuracy.
+- [Unsloth Environment Flags](/basics/unsloth-environment-flags.md): Advanced flags which might be useful if you see breaking finetunes, or you want to turn stuff off.
+- [Continued Pretraining](/basics/continued-pretraining.md): AKA as Continued Finetuning. Unsloth allows you to continually pretrain so a model can learn a new language.
+- [Unsloth Benchmarks](/basics/unsloth-benchmarks.md): Unsloth recorded benchmarks on NVIDIA GPUs.

package/bin/skills/verl/SKILL.md

@@ -0,0 +1,391 @@
+---
+name: verl-rl-training
+description: Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.
+version: 1.0.0
+author: Synthetic Sciences
+license: MIT
+tags: [Reinforcement Learning, RLHF, GRPO, PPO, Post-Training, Distributed Training]
+dependencies: [verl>=0.3.0, torch>=2.0.0, ray>=2.41.0, vllm>=0.8.2, transformers>=4.40.0]
+---
+
+# verl: Volcano Engine Reinforcement Learning for LLMs
+
+verl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.
+
+## When to Use verl
+
+**Choose verl when you need:**
+- Production-ready RL training at scale (tested up to 671B parameters)
+- Flexibility to swap backends (FSDP ↔ Megatron-LM ↔ vLLM ↔ SGLang)
+- Support for multiple RL algorithms (PPO, GRPO, RLOO, REINFORCE++, DAPO)
+- Multi-turn rollout with tool calling for agentic workflows
+- Vision-language model RL training
+
+**Consider alternatives when:**
+- You need Megatron-native training → use **slime** or **miles**
+- You want PyTorch-native abstractions with Monarch → use **torchforge**
+- You only need simple SFT/DPO → use **TRL** or **Axolotl**
+
+## Key Features
+
+- **Training backends**: FSDP, FSDP2, Megatron-LM
+- **Rollout engines**: vLLM, SGLang, HuggingFace Transformers
+- **Algorithms**: PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, SPPO
+- **Models**: Qwen-3, Llama-3.1, DeepSeek, Gemma-2 (0.5B to 671B)
+- **Advanced**: LoRA RL, sequence parallelism, expert parallelism, multi-turn tools
+
+## Installation
+
+```bash
+# Option 1: pip install
+pip install verl[vllm]  # or verl[sglang] for SGLang backend
+
+# Option 2: Docker (recommended for production)
+docker pull verlai/verl:vllm011.latest
+
+# Option 3: From source
+git clone https://github.com/volcengine/verl.git
+cd verl && pip install -e .[vllm,math]
+```
+
+## Quick Start: GRPO Training
+
+```bash
+python3 -m verl.trainer.main_ppo \
+    algorithm.adv_estimator=grpo \
+    data.train_files=~/data/gsm8k/train.parquet \
+    actor_rollout_ref.model.path=Qwen/Qwen2.5-7B \
+    actor_rollout_ref.rollout.n=8 \
+    actor_rollout_ref.actor.use_kl_loss=True \
+    trainer.n_gpus_per_node=8
+```
+
+## Core Architecture
+
+verl uses a **HybridFlow** programming model separating control flow from computation:
+
+```
+┌─────────────────────────────────────────────────────────┐
+│ Single-Process Controller (Ray)                         │
+│ - Synthetic Sciencestes: rollout → reward → train → sync        │
+└─────────────────────┬───────────────────────────────────┘
+                      │
+┌─────────────────────▼───────────────────────────────────┐
+│ Multi-Process Workers                                   │
+│ ├── ActorRolloutRefWorker (policy + generation)        │
+│ ├── CriticWorker (value estimation, PPO only)          │
+│ └── RewardManager (model-based or rule-based rewards)  │
+└─────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Workflow 1: Math Reasoning with GRPO
+
+Use this workflow for training reasoning models on math tasks like GSM8K or MATH.
+
+### Prerequisites Checklist
+- [ ] GPU cluster with 8+ GPUs (H100 recommended)
+- [ ] Dataset in parquet format with `prompt` and `reward_model` columns
+- [ ] Base model from HuggingFace Hub
+
+### Step 1: Prepare Dataset
+
+```python
+import pandas as pd
+
+data = [
+    {
+        "prompt": [{"role": "user", "content": "What is 15 + 27?"}],
+        "reward_model": {"ground_truth": "42"}
+    },
+    # ... more examples
+]
+df = pd.DataFrame(data)
+df.to_parquet("train.parquet")
+```
+
+### Step 2: Define Reward Function
+
+```python
+# reward_function.py
+import re
+
+def compute_reward(responses, ground_truths):
+    rewards = []
+    for response, gt in zip(responses, ground_truths):
+        # Extract answer from response
+        match = re.search(r'\\boxed{([^}]+)}', response)
+        if match and match.group(1).strip() == gt.strip():
+            rewards.append(1.0)
+        else:
+            rewards.append(0.0)
+    return rewards
+```
+
+### Step 3: Create Training Config
+
+```yaml
+# config/grpo_math.yaml
+algorithm:
+  adv_estimator: grpo
+  gamma: 1.0
+  lam: 1.0
+
+data:
+  train_files: /path/to/train.parquet
+  val_files: /path/to/val.parquet
+  train_batch_size: 256
+  max_prompt_length: 512
+  max_response_length: 2048
+
+actor_rollout_ref:
+  model:
+    path: Qwen/Qwen2.5-7B-Instruct
+  actor:
+    use_kl_loss: true
+    kl_loss_coef: 0.001
+    ppo_mini_batch_size: 64
+  rollout:
+    name: vllm
+    n: 8  # samples per prompt
+    temperature: 0.7
+    top_p: 0.95
+
+trainer:
+  total_epochs: 3
+  n_gpus_per_node: 8
+  save_freq: 100
+```
+
+### Step 4: Launch Training
+
+```bash
+python3 -m verl.trainer.main_ppo \
+    --config-path config \
+    --config-name grpo_math \
+    trainer.experiment_name=grpo_math_qwen7b
+```
+
+### Step 5: Monitor and Validate
+- [ ] Check WandB/TensorBoard for loss curves
+- [ ] Verify reward is increasing over steps
+- [ ] Run evaluation on held-out test set
+
+---
+
+## Workflow 2: PPO with Critic Model
+
+Use this workflow when you need value-based advantage estimation (GAE).
+
+### Key Differences from GRPO
+- Requires separate critic model
+- Uses Generalized Advantage Estimation (GAE)
+- Better for tasks with dense rewards
+
+### Configuration
+
+```yaml
+algorithm:
+  adv_estimator: gae  # Use GAE instead of GRPO
+  gamma: 0.99
+  lam: 0.95
+
+critic:
+  model:
+    path: Qwen/Qwen2.5-7B-Instruct  # Can be same or different from actor
+  ppo_mini_batch_size: 64
+
+actor_rollout_ref:
+  actor:
+    use_kl_loss: true
+    kl_loss_coef: 0.02
+    clip_ratio: 0.2  # PPO clipping
+```
+
+### Launch with Critic
+
+```bash
+python3 -m verl.trainer.main_ppo \
+    algorithm.adv_estimator=gae \
+    critic.model.path=Qwen/Qwen2.5-7B-Instruct \
+    trainer.n_gpus_per_node=8
+```
+
+---
+
+## Workflow 3: Large-Scale Training with Megatron
+
+Use this workflow for models >70B parameters or when you need expert parallelism.
+
+### Prerequisites
+- [ ] Install Megatron-LM bridge: `pip install mbridge`
+- [ ] Convert model to Megatron format
+- [ ] Multi-node cluster with NVLink/InfiniBand
+
+### Configuration for 70B+ Models
+
+```yaml
+actor_rollout_ref:
+  model:
+    path: /path/to/megatron/checkpoint
+    backend: megatron
+  actor:
+    strategy: megatron
+    tensor_model_parallel_size: 8
+    pipeline_model_parallel_size: 2
+  rollout:
+    name: vllm
+    tensor_parallel_size: 8
+```
+
+### Launch Multi-Node
+
+```bash
+# On head node
+ray start --head --port=6379
+
+# On worker nodes
+ray start --address='head_ip:6379'
+
+# Launch training
+python3 -m verl.trainer.main_ppo \
+    trainer.nnodes=4 \
+    trainer.n_gpus_per_node=8
+```
+
+---
+
+## Configuration Reference
+
+### Algorithm Selection
+
+| Algorithm | `adv_estimator` | Use Case |
+|-----------|-----------------|----------|
+| GRPO | `grpo` | Critic-free, math/reasoning |
+| PPO/GAE | `gae` | Dense rewards, value estimation |
+| REINFORCE++ | `reinforce_plus_plus` | Variance reduction |
+| RLOO | `rloo` | Leave-one-out baseline |
+| ReMax | `remax` | Maximum reward baseline |
+| OPO | `opo` | Optimal policy optimization |
+
+### Key Parameters
+
+```yaml
+# Rollout parameters
+actor_rollout_ref.rollout.n: 8              # Samples per prompt
+actor_rollout_ref.rollout.temperature: 0.7  # Sampling temperature
+actor_rollout_ref.rollout.top_p: 0.95       # Nucleus sampling
+
+# Training parameters
+actor_rollout_ref.actor.lr: 1e-6            # Learning rate
+actor_rollout_ref.actor.ppo_mini_batch_size: 64
+actor_rollout_ref.actor.clip_ratio: 0.2     # PPO clip range
+
+# KL control
+actor_rollout_ref.actor.use_kl_loss: true
+actor_rollout_ref.actor.kl_loss_coef: 0.001
+algorithm.kl_ctrl.target_kl: 0.1            # For adaptive KL control
+```
+
+---
+
+## Common Issues and Solutions
+
+### Issue: OOM During Rollout
+
+**Symptoms**: CUDA out of memory during generation phase
+
+**Solutions**:
+```yaml
+# Reduce batch size
+actor_rollout_ref.rollout.log_prob_micro_batch_size: 4
+
+# Enable gradient checkpointing
+actor_rollout_ref.model.enable_gradient_checkpointing: true
+
+# Use FSDP2 with CPU offloading
+actor_rollout_ref.actor.strategy: fsdp2
+actor_rollout_ref.actor.fsdp_config.offload_policy: true
+```
+
+### Issue: Training Instability
+
+**Symptoms**: Loss spikes, reward collapse
+
+**Solutions**:
+```yaml
+# Reduce learning rate
+actor_rollout_ref.actor.lr: 5e-7
+
+# Increase KL penalty
+actor_rollout_ref.actor.kl_loss_coef: 0.01
+
+# Enable gradient clipping
+actor_rollout_ref.actor.max_grad_norm: 1.0
+```
+
+### Issue: Slow Weight Sync
+
+**Symptoms**: Long pauses between rollout and training
+
+**Solutions**:
+```bash
+# Use FSDP2 for faster resharding
+actor_rollout_ref.actor.strategy=fsdp2
+
+# Enable async weight transfer
+trainer.async_weight_update=true
+```
+
+### Issue: vLLM Version Mismatch
+
+**Symptoms**: Import errors or generation failures
+
+**Solution**: Use compatible versions:
+```bash
+pip install vllm>=0.8.5,<=0.12.0
+# Avoid vLLM 0.7.x (known bugs)
+```
+
+---
+
+## Advanced Topics
+
+### Multi-Turn Tool Calling
+
+See [references/multi-turn.md](references/multi-turn.md) for agentic workflows with tool use.
+
+### Vision-Language Models
+
+```yaml
+actor_rollout_ref:
+  model:
+    path: Qwen/Qwen2.5-VL-7B-Instruct
+  rollout:
+    name: vllm
+    enable_vision: true
+```
+
+### LoRA Training
+
+```yaml
+actor_rollout_ref:
+  actor:
+    lora:
+      enabled: true
+      r: 16
+      alpha: 32
+      target_modules: ["q_proj", "v_proj"]
+```
+
+---
+
+## Resources
+
+- **Documentation**: https://verl.readthedocs.io/
+- **Paper**: https://arxiv.org/abs/2409.19256
+- **GitHub**: https://github.com/volcengine/verl
+- **Recipes**: https://github.com/verl-project/verl-recipe (DAPO, GSPO, etc.)
+- **Community**: Slack at verl-project
+