@baimingtao/lbs-agent 1.0.0

package/extensions/skills/optional/mlops/simpo/SKILL.md

@@ -0,0 +1,220 @@
+---
+name: simpo-training
+description: 用于 LLM 对齐的 Simple Preference Optimization。作为 DPO 的无参考模型替代方案，具有更好的性能（在 AlpacaEval 2.0 上高 6.4 分）。无需参考模型，比 DPO 更高效。当需要比 DPO/PPO 更简单、更快速的训练时，可用于偏好对齐。
+version: 1.0.0
+author: Orchestra Research
+license: MIT
+dependencies: [torch, transformers, datasets, trl, accelerate]
+platforms: [linux, macos, windows]
+metadata:
+  hermes:
+    tags: [Post-Training, SimPO, Preference Optimization, Alignment, DPO Alternative, Reference-Free, LLM Alignment, Efficient Training]
+
+---
+
+# SimPO - Simple Preference Optimization
+
+## 快速开始
+
+SimPO 是一种无参考模型的偏好优化方法，它优于 DPO 且不需要参考模型。
+
+**安装**：
+```bash
+# Create environment
+conda create -n simpo python=3.10 && conda activate simpo
+
+# Install PyTorch 2.2.2
+# Visit: https://pytorch.org/get-started/locally/
+
+# Install alignment-handbook
+git clone https://github.com/huggingface/alignment-handbook.git
+cd alignment-handbook
+python -m pip install .
+
+# Install Flash Attention 2
+python -m pip install flash-attn --no-build-isolation
+```
+
+**训练** (Mistral 7B)：
+```bash
+ACCELERATE_LOG_LEVEL=info accelerate launch \
+  --config_file accelerate_configs/deepspeed_zero3.yaml \
+  scripts/run_simpo.py \
+  training_configs/mistral-7b-base-simpo.yaml
+```
+
+## 常见工作流
+
+### 工作流 1：从基础模型训练 (Mistral 7B)
+
+**配置** (`mistral-7b-base-simpo.yaml`)：
+```yaml
+# Model
+model_name_or_path: mistralai/Mistral-7B-v0.1
+torch_dtype: bfloat16
+
+# Dataset
+dataset_mixer:
+  HuggingFaceH4/ultrafeedback_binarized: 1.0
+dataset_splits:
+  - train_prefs
+  - test_prefs
+
+# SimPO hyperparameters
+beta: 2.0                  # Reward scaling (2.0-10.0)
+gamma_beta_ratio: 0.5       # Target margin (0-1)
+loss_type: sigmoid          # sigmoid or hinge
+sft_weight: 0.0             # Optional SFT regularization
+
+# Training
+learning_rate: 5e-7         # Critical: 3e-7 to 1e-6
+num_train_epochs: 1
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 8
+
+# Output
+output_dir: ./outputs/mistral-7b-simpo
+```
+
+**启动训练**：
+```bash
+accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \
+  scripts/run_simpo.py training_configs/mistral-7b-base-simpo.yaml
+```
+
+### 工作流 2：微调 Instruct 模型 (Llama 3 8B)
+
+**配置** (`llama3-8b-instruct-simpo.yaml`)：
+```yaml
+model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct
+
+dataset_mixer:
+  argilla/ultrafeedback-binarized-preferences-cleaned: 1.0
+
+beta: 2.5
+gamma_beta_ratio: 0.5
+learning_rate: 5e-7
+sft_weight: 0.1             # Add SFT loss to preserve capabilities
+
+num_train_epochs: 1
+per_device_train_batch_size: 2
+gradient_accumulation_steps: 4
+output_dir: ./outputs/llama3-8b-simpo
+```
+
+**启动**：
+```bash
+accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \
+  scripts/run_simpo.py training_configs/llama3-8b-instruct-simpo.yaml
+```
+
+### 工作流 3：推理密集型任务（较低 LR）
+
+**适用于数学/代码任务**：
+```yaml
+model_name_or_path: deepseek-ai/deepseek-math-7b-base
+
+dataset_mixer:
+  argilla/distilabel-math-preference-dpo: 1.0
+
+beta: 5.0                   # Higher for stronger signal
+gamma_beta_ratio: 0.7       # Larger margin
+learning_rate: 3e-7         # Lower LR for reasoning
+sft_weight: 0.0
+
+num_train_epochs: 1
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 16
+```
+
+## 何时使用及其替代方案
+
+**在以下情况使用 SimPO**：
+- 需要比 DPO 更简单的训练（无需参考模型）
+- 拥有偏好数据（chosen/rejected 对）
+- 需要比 DPO 更好的性能
+- 计算资源有限
+- 单节点训练即可满足需求
+
+**算法选择**：
+- **SimPO**：最简单，性能最佳，无参考模型
+- **DPO**：需要参考模型基线，更保守
+- **PPO**：最大程度的控制，需要 reward model，设置复杂
+- **GRPO**：内存高效的 RL，无 critic
+
+**使用替代方案**：
+- **OpenRLHF**：多节点分布式训练，PPO/GRPO
+- **TRL**：需要在单一框架中使用多种方法
+- **DPO**：成熟的基线对比
+
+## 常见问题
+
+**问题：Loss 发散**
+
+降低学习率：
+```yaml
+learning_rate: 3e-7  # Reduce from 5e-7
+```
+
+降低 beta：
+```yaml
+beta: 1.0  # Reduce from 2.0
+```
+
+**问题：模型遗忘能力**
+
+添加 SFT 正则化：
+```yaml
+sft_weight: 0.1  # Add SFT loss component
+```
+
+**问题：偏好分离效果差**
+
+增加 beta 和 margin：
+```yaml
+beta: 5.0            # Increase from 2.0
+gamma_beta_ratio: 0.8  # Increase from 0.5
+```
+
+**问题：训练期间 OOM**
+
+减小 batch size：
+```yaml
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 16  # Maintain effective batch
+```
+
+启用 gradient checkpointing：
+```yaml
+gradient_checkpointing: true
+```
+
+## 进阶主题
+
+**Loss 函数**：有关 sigmoid 与 hinge loss、数学公式以及何时使用它们的更多详情，请参阅 [references/loss-functions.md](references/loss-functions.md)。
+
+**超参数调优**：有关 beta、gamma、学习率选择指南以及针对特定模型大小的建议，请参阅 [references/hyperparameters.md](references/hyperparameters.md)。
+
+**数据集准备**：有关偏好数据格式、质量过滤和创建自定义数据集的信息，请参阅 [references/datasets.md](references/datasets.md)。
+
+## 硬件要求
+
+- **GPU**：推荐 NVIDIA A100/H100
+- **VRAM**：
+  - 7B 模型：1× A100 40GB (DeepSpeed ZeRO-3)
+  - 8B 模型：2× A100 40GB
+  - 70B 模型：8× A100 80GB
+- **单节点**：DeepSpeed ZeRO-3 即可
+- **混合精度**：推荐 BF16
+
+**内存优化**：
+- DeepSpeed ZeRO-3（默认配置）
+- Gradient checkpointing
+- Flash Attention 2
+
+## 资源
+
+- 论文：https://arxiv.org/abs/2405.14734 (NeurIPS 2024)
+- GitHub：https://github.com/princeton-nlp/SimPO
+- 模型：https://huggingface.co/princeton-nlp
+- Alignment Handbook：https://github.com/huggingface/alignment-handbook

package/extensions/skills/optional/mlops/simpo/references/datasets.md

@@ -0,0 +1,478 @@
+# 数据集 (Datasets)
+
+SimPO 训练偏好数据集完整指南。
+
+## 数据集格式
+
+### 必需 字段
+
+偏好数据集必须包含：
+```json
+{
+  "prompt": "User question or instruction",
+  "chosen": "Better/preferred response",
+  "rejected": "Worse/rejected response"
+}
+```
+
+**备选字段名**（自动检测）：
+- `prompt` → `question`, `instruction`, `input`
+- `chosen` → `response_chosen`, `winner`, `preferred`
+- `rejected` → `response_rejected`, `loser`
+
+### 示例 Entry
+
+```json
+{
+  "prompt": "Explain quantum computing in simple terms.",
+  "chosen": "Quantum computing uses quantum bits (qubits) that can exist in multiple states simultaneously through superposition. This allows quantum computers to process many possibilities at once, making them potentially much faster than classical computers for specific tasks like cryptography and optimization.",
+  "rejected": "It's like regular computing but quantum."
+}
+```
+
+## 热门数据集
+
+### 1. UltraFeedback (推荐)
+
+**HuggingFaceH4/ultrafeedback_binarized**:
+- **大小**: 60K 偏好对
+- **质量**: 高（GPT-4 标注）
+- **领域**: 通用指令遵循
+- **格式**: 干净，开箱即用
+
+**配置**:
+```yaml
+dataset_mixer:
+  HuggingFaceH4/ultrafeedback_binarized: 1.0
+dataset_splits:
+  - train_prefs
+  - test_prefs
+```
+
+### 2. Argilla UltraFeedback (Cleaned)
+
+**argilla/ultrafeedback-binarized-preferences-cleaned**:
+- **大小**: 50K 对（已过滤）
+- **质量**: 非常高（已去重，已清理）
+- **领域**: 通用
+- **格式**: 干净
+
+**配置**:
+```yaml
+dataset_mixer:
+  argilla/ultrafeedback-binarized-preferences-cleaned: 1.0
+```
+
+### 3. Distilabel Math
+
+**argilla/distilabel-math-preference-dpo**:
+- **大小**: 30K 对
+- **质量**: 高（GSM8K, MATH）
+- **领域**: 数学推理
+- **格式**: 数学专用
+
+**配置**:
+```yaml
+dataset_mixer:
+  argilla/distilabel-math-preference-dpo: 1.0
+```
+
+### 4. HelpSteer
+
+**nvidia/HelpSteer**:
+- **大小**: 38K 样本
+- **质量**: 高（人类评分）
+- **领域**: 有用性对齐
+- **格式**: 多属性评分
+
+**配置**:
+```yaml
+dataset_mixer:
+  nvidia/HelpSteer: 1.0
+```
+
+### 5. Anthropic HH-RLHF
+
+**Anthropic/hh-rlhf**:
+- **大小**: 161K 样本
+- **质量**: 高（人类偏好）
+- **领域**: 无害 + 有用
+- **格式**: 对话式
+
+**配置**:
+```yaml
+dataset_mixer:
+  Anthropic/hh-rlhf: 1.0
+```
+
+## 数据集混合
+
+### 多个数据集
+
+**等比例混合**:
+```yaml
+dataset_mixer:
+  HuggingFaceH4/ultrafeedback_binarized: 0.5
+  Anthropic/hh-rlhf: 0.5
+```
+
+**加权混合**:
+```yaml
+dataset_mixer:
+  HuggingFaceH4/ultrafeedback_binarized: 0.7
+  argilla/distilabel-math-preference-dpo: 0.2
+  nvidia/HelpSteer: 0.1
+```
+
+**特定领域侧重**:
+```yaml
+# 80% 通用 + 20% 数学
+dataset_mixer:
+  HuggingFaceH4/ultrafeedback_binarized: 0.8
+  argilla/distilabel-math-preference-dpo: 0.2
+```
+
+## 数据质量
+
+### 质量指标
+
+**优质的偏好数据**:
+- ✅ `chosen`/`rejected` 之间有明显的质量差异
+- ✅ 多样化的 `prompts`
+- ✅ 极少的噪声/标注错误
+- ✅ 合适的难度水平
+
+**劣质的偏好数据**:
+- ❌ 偏好模糊不清
+- ❌ 重复的 `prompts`
+- ❌ 标注噪声
+- ❌ 过于简单/困难的 `prompts`
+
+### 质量过滤
+
+**按长度差异过滤**:
+```python
+def filter_by_length(example):
+    chosen_len = len(example['chosen'].split())
+    rejected_len = len(example['rejected'].split())
+    # Reject if chosen is much shorter (potential low-effort)
+    return chosen_len >= rejected_len * 0.5
+
+dataset = dataset.filter(filter_by_length)
+```
+
+**按多样性过滤**:
+```python
+seen_prompts = set()
+
+def filter_duplicates(example):
+    prompt = example['prompt']
+    if prompt in seen_prompts:
+        return False
+    seen_prompts.add(prompt)
+    return True
+
+dataset = dataset.filter(filter_duplicates)
+```
+
+## 自定义数据集创建
+
+### 格式 1: JSON Lines
+
+**文件** (`preferences.jsonl`):
+```jsonl
+{"prompt": "What is Python?", "chosen": "Python is a high-level programming language...", "rejected": "It's a snake."}
+{"prompt": "Explain AI.", "chosen": "AI refers to systems that can...", "rejected": "It's computers that think."}
+```
+
+**加载**:
+```yaml
+dataset_mixer:
+  json:
+    data_files: preferences.jsonl
+```
+
+### 格式 2: HuggingFace Dataset
+
+**从 dict 创建**:
+```python
+from datasets import Dataset
+
+data = {
+    "prompt": ["What is Python?", "Explain AI."],
+    "chosen": ["Python is...", "AI refers to..."],
+    "rejected": ["It's a snake.", "It's computers..."]
+}
+
+dataset = Dataset.from_dict(data)
+dataset.push_to_hub("username/my-preferences")
+```
+
+**在配置中使用**:
+```yaml
+dataset_mixer:
+  username/my-preferences: 1.0
+```
+
+### 格式 3: ChatML
+
+**用于对话数据**:
+```json
+{
+  "prompt": [
+    {"role": "user", "content": "What is quantum computing?"}
+  ],
+  "chosen": [
+    {"role": "assistant", "content": "Quantum computing uses qubits..."}
+  ],
+  "rejected": [
+    {"role": "assistant", "content": "It's like regular computing but quantum."}
+  ]
+}
+```
+
+**应用 chat template**:
+```yaml
+dataset_text_field: null  # Will apply chat template
+```
+
+## 合成数据生成
+
+### 使用 GPT-4
+
+**Prompt template**:
+```
+Given the following question:
+{prompt}
+
+Generate two responses:
+1. A high-quality, detailed response (chosen)
+2. A low-quality, brief response (rejected)
+
+Format as JSON with "chosen" and "rejected" fields.
+```
+
+**示例 code**:
+```python
+import openai
+
+def generate_pair(prompt):
+    response = openai.ChatCompletion.create(
+        model="gpt-4",
+        messages=[{
+            "role": "user",
+            "content": f"Given: {prompt}\n\nGenerate chosen/rejected pair in JSON."
+        }]
+    )
+    return json.loads(response.choices[0].message.content)
+
+# Generate dataset
+prompts = load_prompts()
+dataset = [generate_pair(p) for p in prompts]
+```
+
+### 使用本地模型
+
+**使用 vLLM**:
+```python
+from vllm import LLM
+
+llm = LLM(model="meta-llama/Meta-Llama-3-70B-Instruct")
+
+def generate_variations(prompt):
+    # Generate multiple completions
+    outputs = llm.generate(
+        [prompt] * 4,
+        sampling_params={
+            "temperature": 0.8,
+            "top_p": 0.9,
+            "max_tokens": 512
+        }
+    )
+
+    # Select best/worst
+    chosen = max(outputs, key=lambda x: len(x.outputs[0].text))
+    rejected = min(outputs, key=lambda x: len(x.outputs[0].text))
+
+    return {
+        "prompt": prompt,
+        "chosen": chosen.outputs[0].text,
+        "rejected": rejected.outputs[0].text
+    }
+```
+
+## 数据预处理
+
+### 截断 (Truncation)
+
+**限制序列长度**:
+```yaml
+max_prompt_length: 512
+max_completion_length: 512
+max_length: 1024  # Total
+```
+
+**实现**:
+```python
+def truncate_example(example):
+    tokenizer.truncation_side = "left"  # For prompts
+    prompt_tokens = tokenizer(
+        example['prompt'],
+        max_length=512,
+        truncation=True
+    )
+
+    tokenizer.truncation_side = "right"  # For completions
+    chosen_tokens = tokenizer(
+        example['chosen'],
+        max_length=512,
+        truncation=True
+    )
+
+    return {
+        "prompt": tokenizer.decode(prompt_tokens['input_ids']),
+        "chosen": tokenizer.decode(chosen_tokens['input_ids'])
+    }
+
+dataset = dataset.map(truncate_example)
+```
+
+### 去重
+
+**移除完全重复项**:
+```python
+dataset = dataset.unique('prompt')
+```
+
+**移除近似重复项** (MinHash):
+```python
+from datasketch import MinHash, MinHashLSH
+
+def deduplicate_lsh(dataset, threshold=0.8):
+    lsh = MinHashLSH(threshold=threshold, num_perm=128)
+    seen = []
+
+    for i, example in enumerate(dataset):
+        m = MinHash(num_perm=128)
+        for word in example['prompt'].split():
+            m.update(word.encode('utf8'))
+
+        if not lsh.query(m):
+            lsh.insert(i, m)
+            seen.append(example)
+
+    return Dataset.from_list(seen)
+
+dataset = deduplicate_lsh(dataset)
+```
+
+## 数据增强
+
+### 改写 Prompts
+
+```python
+def paraphrase_prompt(example):
+    # Use paraphrasing model
+    paraphrased = paraphrase_model(example['prompt'])
+
+    return [
+        example,  # Original
+        {
+            "prompt": paraphrased,
+            "chosen": example['chosen'],
+            "rejected": example['rejected']
+        }
+    ]
+
+dataset = dataset.map(paraphrase_prompt, batched=False, remove_columns=[])
+```
+
+### 难度平衡
+
+**混合简单/中等/困难**:
+```python
+def categorize_difficulty(example):
+    prompt_len = len(example['prompt'].split())
+    if prompt_len < 20:
+        return "easy"
+    elif prompt_len < 50:
+        return "medium"
+    else:
+        return "hard"
+
+dataset = dataset.map(lambda x: {"difficulty": categorize_difficulty(x)})
+
+# Sample balanced dataset
+easy = dataset.filter(lambda x: x['difficulty'] == 'easy').shuffle().select(range(1000))
+medium = dataset.filter(lambda x: x['difficulty'] == 'medium').shuffle().select(range(1000))
+hard = dataset.filter(lambda x: x['difficulty'] == 'hard').shuffle().select(range(1000))
+
+balanced = concatenate_datasets([easy, medium, hard]).shuffle()
+```
+
+## 数据集统计
+
+### 计算统计信息
+
+```python
+def compute_stats(dataset):
+    prompt_lens = [len(x['prompt'].split()) for x in dataset]
+    chosen_lens = [len(x['chosen'].split()) for x in dataset]
+    rejected_lens = [len(x['rejected'].split()) for x in dataset]
+
+    print(f"Dataset size: {len(dataset)}")
+    print(f"Avg prompt length: {np.mean(prompt_lens):.1f} words")
+    print(f"Avg chosen length: {np.mean(chosen_lens):.1f} words")
+    print(f"Avg rejected length: {np.mean(rejected_lens):.1f} words")
+    print(f"Chosen > Rejected: {sum(c > r for c, r in zip(chosen_lens, rejected_lens)) / len(dataset):.1%}")
+
+compute_stats(dataset)
+```
+
+**预期输出**:
+```
+Dataset size: 50000
+Avg prompt length: 45.2 words
+Avg chosen length: 180.5 words
+Avg rejected length: 120.3 words
+Chosen > Rejected: 85.2%
+```
+
+## 最佳实践
+
+### 1. 数据质量重于数量
+
+- **选择**: 1 万对高质量数据
+- **而非**: 10 万对带噪声数据
+
+### 2. 清晰的偏好信号
+
+- `chosen` 应该明显更好
+- 避免微小的差异
+- 移除模棱两可的数据对
+
+### 3. 领域匹配
+
+- 将数据集领域与目标用例相匹配
+- 混合数据集以获得更广泛的覆盖范围
+- 包含经过安全过滤的数据
+
+### 4. 训练前验证
+
+```python
+# Sample 10 random examples
+samples = dataset.shuffle().select(range(10))
+
+for ex in samples:
+    print(f"Prompt: {ex['prompt']}")
+    print(f"Chosen: {ex['chosen'][:100]}...")
+    print(f"Rejected: {ex['rejected'][:100]}...")
+    print(f"Preference clear: {'✓' if len(ex['chosen']) > len(ex['rejected']) else '?'}")
+    print()
+```
+
+## 参考资料
+
+- HuggingFace Datasets: https://huggingface.co/datasets
+- Alignment Handbook: https://github.com/huggingface/alignment-handbook
+- UltraFeedback: https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized