npm - better-dsh - Versions diffs - 0.2.5 → 0.3.0-c - Mend

better-dsh 0.2.5 → 0.3.0-c

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (1002) hide show

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/mlops/mlops-training-axolotl.md DELETED Viewed

@@ -1,181 +0,0 @@
----
-title: "Axolotl — Axolotl: YAML LLM fine-tuning (LoRA, DPO, GRPO)"
-sidebar_label: "Axolotl"
-description: "Axolotl: YAML LLM fine-tuning (LoRA, DPO, GRPO)"
----
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-# Axolotl
-Axolotl: YAML LLM fine-tuning (LoRA, DPO, GRPO).
-## Skill metadata
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/mlops/axolotl` |
-| Path | `optional-skills/mlops\training\axolotl` |
-| Version | `1.0.0` |
-| Author | Orchestra Research |
-| License | MIT |
-| Dependencies | `axolotl`, `torch`, `transformers`, `datasets`, `peft`, `accelerate`, `deepspeed` |
-| Platforms | linux, macos |
-| Tags | `Fine-Tuning`, `Axolotl`, `LLM`, `LoRA`, `QLoRA`, `DPO`, `KTO`, `ORPO`, `GRPO`, `YAML`, `HuggingFace`, `DeepSpeed`, `Multimodal` |
-## Reference: full SKILL.md
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-# Axolotl Skill
-## What's inside
-Expert guidance for fine-tuning LLMs with Axolotl — YAML configs, 100+ models, LoRA/QLoRA, DPO/KTO/ORPO/GRPO, multimodal support.
-Assistance with axolotl development, generated from official documentation.
-## When to Use This Skill
-This skill should be triggered when:
-- Working with axolotl
-- Asking about axolotl features or APIs
-- Implementing axolotl solutions
-- Debugging axolotl code
-- Learning axolotl best practices
-## Quick Reference
-### Common Patterns
-**Pattern 1:** To validate that acceptable data transfer speeds exist for your training job, running NCCL Tests can help pinpoint bottlenecks, for example:
-```
-./build/all_reduce_perf -b 8 -e 128M -f 2 -g 3
-```
-**Pattern 2:** Configure your model to use FSDP in the Axolotl yaml. For example:
-```
-fsdp_version: 2
-fsdp_config:
-  offload_params: true
-  state_dict_type: FULL_STATE_DICT
-  auto_wrap_policy: TRANSFORMER_BASED_WRAP
-  transformer_layer_cls_to_wrap: LlamaDecoderLayer
-  reshard_after_forward: true
-```
-**Pattern 3:** The context_parallel_size should be a divisor of the total number of GPUs. For example:
-```
-context_parallel_size
-```
-**Pattern 4:** For example: - With 8 GPUs and no sequence parallelism: 8 different batches processed per step - With 8 GPUs and context_parallel_size=4: Only 2 different batches processed per step (each split across 4 GPUs) - If your per-GPU micro_batch_size is 2, the global batch size decreases from 16 to 4
-```
-context_parallel_size=4
-```
-**Pattern 5:** Setting save_compressed: true in your configuration enables saving models in a compressed format, which: - Reduces disk space usage by approximately 40% - Maintains compatibility with vLLM for accelerated inference - Maintains compatibility with llmcompressor for further optimization (example: quantization)
-```
-save_compressed: true
-```
-**Pattern 6:** Note It is not necessary to place your integration in the integrations folder. It can be in any location, so long as it’s installed in a package in your python env. See this repo for an example: https://github.com/axolotl-ai-cloud/diff-transformer
-```
-integrations
-```
-**Pattern 7:** Handle both single-example and batched data. - single example: sample[‘input_ids’] is a list[int] - batched data: sample[‘input_ids’] is a list[list[int]]
-```
-utils.trainer.drop_long_seq(sample, sequence_len=2048, min_sequence_len=2)
-```
-### Example Code Patterns
-**Example 1** (python):
-```python
-cli.cloud.modal_.ModalCloud(config, app=None)
-```
-**Example 2** (python):
-```python
-cli.cloud.modal_.run_cmd(cmd, run_folder, volumes=None)
-```
-**Example 3** (python):
-```python
-core.trainers.base.AxolotlTrainer(
-    *_args,
-    bench_data_collator=None,
-    eval_data_collator=None,
-    dataset_tags=None,
-    **kwargs,
-)
-```
-**Example 4** (python):
-```python
-core.trainers.base.AxolotlTrainer.log(logs, start_time=None)
-```
-**Example 5** (python):
-```python
-prompt_strategies.input_output.RawInputOutputPrompter()
-```
-## Reference Files
-This skill includes comprehensive documentation in `references/`:
-- **api.md** - Api documentation
-- **dataset-formats.md** - Dataset-Formats documentation
-- **other.md** - Other documentation
-Use `view` to read specific reference files when detailed information is needed.
-## Working with This Skill
-### For Beginners
-Start with the getting_started or tutorials reference files for foundational concepts.
-### For Specific Features
-Use the appropriate category reference file (api, guides, etc.) for detailed information.
-### For Code Examples
-The quick reference section above contains common patterns extracted from the official docs.
-## Resources
-### references/
-Organized documentation extracted from official sources. These files contain:
-- Detailed explanations
-- Code examples with language annotations
-- Links to original documentation
-- Table of contents for quick navigation
-### scripts/
-Add helper scripts here for common automation tasks.
-### assets/
-Add templates, boilerplate, or example projects here.
-## Notes
-- This skill was automatically generated from official documentation
-- Reference files preserve the structure and examples from source docs
-- Code examples include language detection for better syntax highlighting
-- Quick reference patterns are extracted from common usage examples in the docs
-## Updating
-To refresh this skill with updated documentation:
-1. Re-run the scraper with the same configuration
-2. The skill will be rebuilt with the latest information

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/mlops/mlops-training-trl-fine-tuning.md DELETED Viewed

@@ -1,512 +0,0 @@
----
-title: "Trl Fine Tuning — TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF"
-sidebar_label: "Trl Fine Tuning"
-description: "TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF"
----
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-# Trl Fine Tuning
-TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF.
-## Skill metadata
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/mlops/trl-fine-tuning` |
-| Path | `optional-skills/mlops\training\trl-fine-tuning` |
-| Version | `1.0.1` |
-| Author | Orchestra Research |
-| License | MIT |
-| Dependencies | `trl`, `transformers`, `datasets`, `peft`, `accelerate`, `torch` |
-| Platforms | linux, macos, windows |
-| Tags | `Post-Training`, `TRL`, `Reinforcement Learning`, `Fine-Tuning`, `SFT`, `DPO`, `GRPO`, `RLOO`, `RLHF`, `Preference Alignment`, `HuggingFace` |
-## Reference: full SKILL.md
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-# TRL - Transformer Reinforcement Learning
-## Quick start
-TRL provides post-training methods for aligning language models with human preferences.
-**Installation**:
-```bash
-pip install trl transformers datasets peft accelerate
-```
-**Supervised Fine-Tuning** (instruction tuning):
-```python
-from trl import SFTTrainer
-trainer = SFTTrainer(
-    model="Qwen/Qwen2.5-0.5B",
-    train_dataset=dataset,  # Prompt-completion pairs
-)
-trainer.train()
-```
-**DPO** (align with preferences):
-```python
-from trl import DPOTrainer, DPOConfig
-config = DPOConfig(output_dir="model-dpo", beta=0.1)
-trainer = DPOTrainer(
-    model=model,
-    args=config,
-    train_dataset=preference_dataset,  # chosen/rejected pairs
-    processing_class=tokenizer
-)
-trainer.train()
-```
-## Common workflows
-### Workflow 1: Full RLHF pipeline (SFT → Reward Model → RLOO)
-Complete pipeline from base model to human-aligned model.
-> **Note (TRL 1.x):** PPO has been **removed** from TRL — `PPOTrainer`, `PPOConfig`, and
-> `python -m trl.scripts.ppo` no longer exist. Use an online-RL trainer TRL still ships:
-> **RLOO** (`RLOOTrainer` / `trl rloo`) is the closest drop-in for a reward-model-driven
-> RLHF pipeline, and **GRPO** (`GRPOTrainer` / `trl grpo`, see Workflow 3) is the
-> memory-efficient alternative. The step below uses RLOO.
-Copy this checklist:
-```
-RLHF Training:
-- [ ] Step 1: Supervised fine-tuning (SFT)
-- [ ] Step 2: Train reward model
-- [ ] Step 3: RLOO reinforcement learning
-- [ ] Step 4: Evaluate aligned model
-```
-**Step 1: Supervised fine-tuning**
-Train base model on instruction-following data:
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from trl import SFTTrainer, SFTConfig
-from datasets import load_dataset
-# Load model
-model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
-tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
-# Load instruction dataset
-dataset = load_dataset("trl-lib/Capybara", split="train")
-# Configure training
-training_args = SFTConfig(
-    output_dir="Qwen2.5-0.5B-SFT",
-    per_device_train_batch_size=4,
-    num_train_epochs=1,
-    learning_rate=2e-5,
-    logging_steps=10,
-    save_strategy="epoch"
-)
-# Train
-trainer = SFTTrainer(
-    model=model,
-    args=training_args,
-    train_dataset=dataset,
-    processing_class=tokenizer
-)
-trainer.train()
-trainer.save_model()
-```
-**Step 2: Train reward model**
-Train model to predict human preferences:
-```python
-from transformers import AutoModelForSequenceClassification
-from trl import RewardTrainer, RewardConfig
-# Load SFT model as base
-model = AutoModelForSequenceClassification.from_pretrained(
-    "Qwen2.5-0.5B-SFT",
-    num_labels=1  # Single reward score
-)
-tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-SFT")
-# Load preference data (chosen/rejected pairs)
-dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
-# Configure training
-training_args = RewardConfig(
-    output_dir="Qwen2.5-0.5B-Reward",
-    per_device_train_batch_size=2,
-    num_train_epochs=1,
-    learning_rate=1e-5
-)
-# Train reward model
-trainer = RewardTrainer(
-    model=model,
-    args=training_args,
-    processing_class=tokenizer,
-    train_dataset=dataset
-)
-trainer.train()
-trainer.save_model()
-```
-**Step 3: RLOO reinforcement learning**
-Optimize policy using the reward model. PPO was removed in TRL 1.x; use the RLOO CLI
-(`trl rloo`) with the trained reward model passed via `--reward_model_name_or_path`:
-```bash
-trl rloo \
-    --model_name_or_path Qwen2.5-0.5B-SFT \
-    --reward_model_name_or_path Qwen2.5-0.5B-Reward \
-    --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \
-    --output_dir Qwen2.5-0.5B-RLOO \
-    --learning_rate 3e-6 \
-    --per_device_train_batch_size 64 \
-    --num_generations 4
-```
-Equivalent Python (`RLOOTrainer` / `RLOOConfig`):
-```python
-from trl import RLOOTrainer, RLOOConfig
-from transformers import AutoModelForSequenceClassification, AutoTokenizer
-reward_model = AutoModelForSequenceClassification.from_pretrained(
-    "Qwen2.5-0.5B-Reward", num_labels=1
-)
-config = RLOOConfig(
-    output_dir="Qwen2.5-0.5B-RLOO",
-    per_device_train_batch_size=64,
-    learning_rate=3e-6,
-    num_generations=4,
-)
-trainer = RLOOTrainer(
-    model="Qwen2.5-0.5B-SFT",
-    reward_funcs=reward_model,   # a reward model (or a callable reward function)
-    args=config,
-    train_dataset=dataset,       # prompt-only dataset
-    processing_class=tokenizer,
-)
-trainer.train()
-```
-**Step 4: Evaluate**
-```python
-from transformers import pipeline
-# Load aligned model
-generator = pipeline("text-generation", model="Qwen2.5-0.5B-RLOO")
-# Test
-prompt = "Explain quantum computing to a 10-year-old"
-output = generator(prompt, max_length=200)[0]["generated_text"]
-print(output)
-```
-### Workflow 2: Simple preference alignment with DPO
-Align model with preferences without reward model.
-Copy this checklist:
-```
-DPO Training:
-- [ ] Step 1: Prepare preference dataset
-- [ ] Step 2: Configure DPO
-- [ ] Step 3: Train with DPOTrainer
-- [ ] Step 4: Evaluate alignment
-```
-**Step 1: Prepare preference dataset**
-Dataset format:
-```json
-{
-  "prompt": "What is the capital of France?",
-  "chosen": "The capital of France is Paris.",
-  "rejected": "I don't know."
-}
-```
-Load dataset:
-```python
-from datasets import load_dataset
-dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
-# Or load your own
-# dataset = load_dataset("json", data_files="preferences.json")
-```
-**Step 2: Configure DPO**
-```python
-from trl import DPOConfig
-config = DPOConfig(
-    output_dir="Qwen2.5-0.5B-DPO",
-    per_device_train_batch_size=4,
-    num_train_epochs=1,
-    learning_rate=5e-7,
-    beta=0.1,  # KL penalty strength
-    max_prompt_length=512,
-    max_length=1024,
-    logging_steps=10
-)
-```
-**Step 3: Train with DPOTrainer**
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from trl import DPOTrainer
-model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
-tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
-trainer = DPOTrainer(
-    model=model,
-    args=config,
-    train_dataset=dataset,
-    processing_class=tokenizer
-)
-trainer.train()
-trainer.save_model()
-```
-**CLI alternative**:
-```bash
-trl dpo \
-    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
-    --dataset_name argilla/Capybara-Preferences \
-    --output_dir Qwen2.5-0.5B-DPO \
-    --per_device_train_batch_size 4 \
-    --learning_rate 5e-7 \
-    --beta 0.1
-```
-### Workflow 3: Memory-efficient online RL with GRPO
-Train with reinforcement learning using minimal memory.
-For in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see **[references/grpo-training.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\training\trl-fine-tuning/references/grpo-training.md)**. A production-ready training script is in **[templates/basic_grpo_training.py](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\training\trl-fine-tuning/templates/basic_grpo_training.py)**.
-Copy this checklist:
-```
-GRPO Training:
-- [ ] Step 1: Define reward function
-- [ ] Step 2: Configure GRPO
-- [ ] Step 3: Train with GRPOTrainer
-```
-**Step 1: Define reward function**
-```python
-def reward_function(completions, **kwargs):
-    """
-    Compute rewards for completions.
-    Args:
-        completions: List of generated texts
-    Returns:
-        List of reward scores (floats)
-    """
-    rewards = []
-    for completion in completions:
-        # Example: reward based on length and unique words
-        score = len(completion.split())  # Favor longer responses
-        score += len(set(completion.lower().split()))  # Reward unique words
-        rewards.append(score)
-    return rewards
-```
-Or use a reward model:
-```python
-from transformers import pipeline
-reward_model = pipeline("text-classification", model="reward-model-path")
-def reward_from_model(completions, prompts, **kwargs):
-    # Combine prompt + completion
-    full_texts = [p + c for p, c in zip(prompts, completions)]
-    # Get reward scores
-    results = reward_model(full_texts)
-    return [r["score"] for r in results]
-```
-**Step 2: Configure GRPO**
-```python
-from trl import GRPOConfig
-config = GRPOConfig(
-    output_dir="Qwen2-GRPO",
-    per_device_train_batch_size=4,
-    num_train_epochs=1,
-    learning_rate=1e-5,
-    num_generations=4,  # Generate 4 completions per prompt
-    max_new_tokens=128
-)
-```
-**Step 3: Train with GRPOTrainer**
-```python
-from datasets import load_dataset
-from trl import GRPOTrainer
-# Load prompt-only dataset
-dataset = load_dataset("trl-lib/tldr", split="train")
-trainer = GRPOTrainer(
-    model="Qwen/Qwen2-0.5B-Instruct",
-    reward_funcs=reward_function,  # Your reward function
-    args=config,
-    train_dataset=dataset
-)
-trainer.train()
-```
-**CLI**:
-```bash
-trl grpo \
-    --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
-    --dataset_name trl-lib/tldr \
-    --output_dir Qwen2-GRPO \
-    --num_generations 4
-```
-## When to use vs alternatives
-**Use TRL when:**
-- Need to align model with human preferences
-- Have preference data (chosen/rejected pairs)
-- Want to use reinforcement learning (RLOO, GRPO)
-- Need reward model training
-- Doing RLHF (full pipeline)
-**Method selection**:
-- **SFT**: Have prompt-completion pairs, want basic instruction following
-- **DPO**: Have preferences, want simple alignment (no reward model needed)
-- **RLOO**: Have a reward model, want online RL (the reward-model-driven RLHF path; PPO was removed in TRL 1.x)
-- **GRPO**: Memory-constrained, want online RL with reward functions
-- **Reward Model**: Building RLHF pipeline, need to score generations
-**Use alternatives instead:**
-- **HuggingFace Trainer**: Basic fine-tuning without RL
-- **Axolotl**: YAML-based training configuration
-- **LitGPT**: Educational, minimal fine-tuning
-- **Unsloth**: Fast LoRA training
-## Common issues
-**Issue: OOM during DPO training**
-Reduce batch size and sequence length:
-```python
-config = DPOConfig(
-    per_device_train_batch_size=1,  # Reduce from 4
-    max_length=512,  # Reduce from 1024
-    gradient_accumulation_steps=8  # Maintain effective batch
-)
-```
-Or use gradient checkpointing:
-```python
-model.gradient_checkpointing_enable()
-```
-**Issue: Poor alignment quality**
-Tune beta parameter:
-```python
-# Higher beta = more conservative (stays closer to reference)
-config = DPOConfig(beta=0.5)  # Default 0.1
-# Lower beta = more aggressive alignment
-config = DPOConfig(beta=0.01)
-```
-**Issue: Reward model not learning**
-Check loss type and learning rate:
-```python
-config = RewardConfig(
-    learning_rate=1e-5,  # Try different LR
-    num_train_epochs=3  # Train longer
-)
-```
-Ensure preference dataset has clear winners:
-```python
-# Verify dataset
-print(dataset[0])
-# Should have clear chosen > rejected
-```
-**Issue: Online RL (RLOO/GRPO) training unstable**
-Adjust the KL/beta regularization toward the reference policy:
-```python
-from trl import RLOOConfig
-config = RLOOConfig(
-    beta=0.05,          # KL coefficient toward the reference model (increase for stability)
-    num_generations=4,  # more samples per prompt = lower-variance advantage estimates
-)
-```
-## Advanced topics
-**SFT training guide**: See [references/sft-training.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\training\trl-fine-tuning/references/sft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.
-**DPO variants**: See [references/dpo-variants.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\training\trl-fine-tuning/references/dpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.
-**Reward modeling**: See [references/reward-modeling.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\training\trl-fine-tuning/references/reward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.
-**Online RL methods**: See [references/online-rl.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\training\trl-fine-tuning/references/online-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.
-**GRPO deep dive**: See [references/grpo-training.md](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\training\trl-fine-tuning/references/grpo-training.md) for expert-level GRPO patterns — reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting. Production-ready template in [templates/basic_grpo_training.py](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\training\trl-fine-tuning/templates/basic_grpo_training.py).
-## Hardware requirements
-- **GPU**: NVIDIA (CUDA required)
-- **VRAM**: Depends on model and method
-  - SFT 7B: 16GB (with LoRA)
-  - DPO 7B: 24GB (stores reference model)
-  - RLOO 7B: 40GB (policy + reward model)
-  - GRPO 7B: 24GB (more memory efficient)
-- **Multi-GPU**: Supported via `accelerate`
-- **Mixed precision**: BF16 recommended (A100/H100)
-**Memory optimization**:
-- Use LoRA/QLoRA for all methods
-- Enable gradient checkpointing
-- Use smaller batch sizes with gradient accumulation
-## Resources
-- Docs: https://huggingface.co/docs/trl/
-- GitHub: https://github.com/huggingface/trl
-- Papers:
-  - "Training language models to follow instructions with human feedback" (InstructGPT, 2022)
-  - "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (DPO, 2023)
-  - "Group Relative Policy Optimization" (GRPO, 2024)
-- Examples: https://github.com/huggingface/trl/tree/main/examples/scripts