cli-jaw 0.1.6 → 0.1.8

package/skills_ref/hugging-face-model-trainer/references/reliability_principles.md

@@ -1,371 +0,0 @@
-# Reliability Principles for Training Jobs
-
-These principles are derived from real production failures and successful fixes. Following them prevents common failure modes and ensures reliable job execution.
-
-## Principle 1: Always Verify Before Use
-
-**Rule:** Never assume repos, datasets, or resources exist. Verify with tools first.
-
-### What It Prevents
-
-- **Non-existent datasets** - Jobs fail immediately when dataset doesn't exist
-- **Typos in names** - Simple mistakes like "argilla-dpo-mix-7k" vs "ultrafeedback_binarized"
-- **Incorrect paths** - Old or moved repos, renamed files
-- **Missing dependencies** - Undocumented requirements
-
-### How to Apply
-
-**Before submitting ANY job:**
-
-```python
-# Verify dataset exists
-dataset_search({"query": "dataset-name", "author": "author-name", "limit": 5})
-hub_repo_details(["author/dataset-name"], repo_type="dataset")
-
-# Verify model exists
-hub_repo_details(["org/model-name"], repo_type="model")
-
-# Check script/file paths (for URL-based scripts)
-# Verify before using: https://github.com/user/repo/blob/main/script.py
-```
-
-**Examples that would have caught errors:**
-
-```python
-# ❌ WRONG: Assumed dataset exists
-hf_jobs("uv", {
-    "script": """...""",
-    "env": {"DATASET": "trl-lib/argilla-dpo-mix-7k"}  # Doesn't exist!
-})
-
-# ✅ CORRECT: Verify first
-dataset_search({"query": "argilla dpo", "author": "trl-lib"})
-# Would show: "trl-lib/ultrafeedback_binarized" is the correct name
-
-hub_repo_details(["trl-lib/ultrafeedback_binarized"], repo_type="dataset")
-# Confirms it exists before using
-```
-
-### Implementation Checklist
-
-- [ ] Check dataset exists before training
-- [ ] Verify base model exists before fine-tuning
-- [ ] Confirm adapter model exists before GGUF conversion
-- [ ] Test script URLs are valid before submitting
-- [ ] Validate file paths in repositories
-- [ ] Check for recent updates/renames of resources
-
-**Time cost:** 5-10 seconds  
-**Time saved:** Hours of failed job time + debugging
-
----
-
-## Principle 2: Prioritize Reliability Over Performance
-
-**Rule:** Default to what is most likely to succeed, not what is theoretically fastest.
-
-### What It Prevents
-
-- **Hardware incompatibilities** - Features that fail on certain GPUs
-- **Unstable optimizations** - Speed-ups that cause crashes
-- **Complex configurations** - More failure points
-- **Build system issues** - Unreliable compilation methods
-
-### How to Apply
-
-**Choose reliability:**
-
-```python
-# ❌ RISKY: Aggressive optimization that may fail
-SFTConfig(
-    torch_compile=True,  # Can fail on T4, A10G GPUs
-    optim="adamw_bnb_8bit",  # Requires specific setup
-    fp16=False,  # May cause training instability
-    ...
-)
-
-# ✅ SAFE: Proven defaults
-SFTConfig(
-    # torch_compile=True,  # Commented with note: "Enable on H100 for 20% speedup"
-    optim="adamw_torch",  # Standard, always works
-    fp16=True,  # Stable and fast
-    ...
-)
-```
-
-**For build processes:**
-
-```python
-# ❌ UNRELIABLE: Uses make (platform-dependent)
-subprocess.run(["make", "-C", "/tmp/llama.cpp", "llama-quantize"], check=True)
-
-# ✅ RELIABLE: Uses CMake (consistent, documented)
-subprocess.run([
-    "cmake", "-B", "/tmp/llama.cpp/build", "-S", "/tmp/llama.cpp",
-    "-DGGML_CUDA=OFF"  # Disable CUDA for faster, more reliable build
-], check=True)
-
-subprocess.run([
-    "cmake", "--build", "/tmp/llama.cpp/build",
-    "--target", "llama-quantize", "-j", "4"
-], check=True)
-```
-
-### Real-World Example
-
-**The `torch.compile` failure:**
-- Added for "20% speedup" on H100
-- **Failed fatally on T4-medium** with cryptic error
-- Misdiagnosed as dataset issue (cost hours)
-- **Fix:** Disable by default, add as optional comment
-
-**Result:** Reliability > 20% performance gain
-
-### Implementation Checklist
-
-- [ ] Use proven, standard configurations by default
-- [ ] Comment out performance optimizations with hardware notes
-- [ ] Use stable build systems (CMake > make)
-- [ ] Test on target hardware before production
-- [ ] Document known incompatibilities
-- [ ] Provide "safe" and "fast" variants when needed
-
-**Performance loss:** 10-20% in best case  
-**Reliability gain:** 95%+ success rate vs 60-70%
-
----
-
-## Principle 3: Create Atomic, Self-Contained Scripts
-
-**Rule:** Scripts should work as complete, independent units. Don't remove parts to "simplify."
-
-### What It Prevents
-
-- **Missing dependencies** - Removed "unnecessary" packages that are actually required
-- **Incomplete processes** - Skipped steps that seem redundant
-- **Environment assumptions** - Scripts that need pre-setup
-- **Partial failures** - Some parts work, others fail silently
-
-### How to Apply
-
-**Complete dependency specifications:**
-
-```python
-# ❌ INCOMPLETE: "Simplified" by removing dependencies
-# /// script
-# dependencies = [
-#     "transformers",
-#     "peft",
-#     "torch",
-# ]
-# ///
-
-# ✅ COMPLETE: All dependencies explicit
-# /// script
-# dependencies = [
-#     "transformers>=4.36.0",
-#     "peft>=0.7.0",
-#     "torch>=2.0.0",
-#     "accelerate>=0.24.0",
-#     "huggingface_hub>=0.20.0",
-#     "sentencepiece>=0.1.99",  # Required for tokenizers
-#     "protobuf>=3.20.0",        # Required for tokenizers
-#     "numpy",
-#     "gguf",
-# ]
-# ///
-```
-
-**Complete build processes:**
-
-```python
-# ❌ INCOMPLETE: Assumes build tools exist
-subprocess.run(["git", "clone", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"])
-subprocess.run(["make", "-C", "/tmp/llama.cpp", "llama-quantize"])  # FAILS: no gcc/make
-
-# ✅ COMPLETE: Installs all requirements
-subprocess.run(["apt-get", "update", "-qq"], check=True)
-subprocess.run(["apt-get", "install", "-y", "-qq", "build-essential", "cmake"], check=True)
-subprocess.run(["git", "clone", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"])
-# ... then build
-```
-
-### Real-World Example
-
-**The `sentencepiece` failure:**
-- Original script had it: worked fine
-- "Simplified" version removed it: "doesn't look necessary"
-- **GGUF conversion failed silently** - tokenizer couldn't convert
-- Hard to debug: no obvious error message
-- **Fix:** Restore all original dependencies
-
-**Result:** Don't remove dependencies without thorough testing
-
-### Implementation Checklist
-
-- [ ] All dependencies in PEP 723 header with version pins
-- [ ] All system packages installed by script
-- [ ] No assumptions about pre-existing environment
-- [ ] No "optional" steps that are actually required
-- [ ] Test scripts in clean environment
-- [ ] Document why each dependency is needed
-
-**Complexity:** Slightly longer scripts  
-**Reliability:** Scripts "just work" every time
-
----
-
-## Principle 4: Provide Clear Error Context
-
-**Rule:** When things fail, make it obvious what went wrong and how to fix it.
-
-### How to Apply
-
-**Wrap subprocess calls:**
-
-```python
-# ❌ UNCLEAR: Silent failure
-subprocess.run([...], check=True, capture_output=True)
-
-# ✅ CLEAR: Shows what failed
-try:
-    result = subprocess.run(
-        [...],
-        check=True,
-        capture_output=True,
-        text=True
-    )
-    print(result.stdout)
-    if result.stderr:
-        print("Warnings:", result.stderr)
-except subprocess.CalledProcessError as e:
-    print(f"❌ Command failed!")
-    print("STDOUT:", e.stdout)
-    print("STDERR:", e.stderr)
-    raise
-```
-
-**Validate inputs:**
-
-```python
-# ❌ UNCLEAR: Fails later with cryptic error
-model = load_model(MODEL_NAME)
-
-# ✅ CLEAR: Fails fast with clear message
-if not MODEL_NAME:
-    raise ValueError("MODEL_NAME environment variable not set!")
-
-print(f"Loading model: {MODEL_NAME}")
-try:
-    model = load_model(MODEL_NAME)
-    print(f"✅ Model loaded successfully")
-except Exception as e:
-    print(f"❌ Failed to load model: {MODEL_NAME}")
-    print(f"Error: {e}")
-    print("Hint: Check that model exists on Hub")
-    raise
-```
-
-### Implementation Checklist
-
-- [ ] Wrap external calls with try/except
-- [ ] Print stdout/stderr on failure
-- [ ] Validate environment variables early
-- [ ] Add progress indicators (✅, ❌, 🔄)
-- [ ] Include hints for common failures
-- [ ] Log configuration at start
-
----
-
-## Principle 5: Test the Happy Path on Known-Good Inputs
-
-**Rule:** Before using new code in production, test with inputs you know work.
-
-### How to Apply
-
-**Known-good test inputs:**
-
-```python
-# For training
-TEST_DATASET = "trl-lib/Capybara"  # Small, well-formatted, widely used
-TEST_MODEL = "Qwen/Qwen2.5-0.5B"  # Small, fast, reliable
-
-# For GGUF conversion
-TEST_ADAPTER = "evalstate/qwen-capybara-medium"  # Known working model
-TEST_BASE = "Qwen/Qwen2.5-0.5B"  # Compatible base
-```
-
-**Testing workflow:**
-
-1. Test with known-good inputs first
-2. If that works, try production inputs
-3. If production fails, you know it's the inputs (not code)
-4. Isolate the difference
-
-### Implementation Checklist
-
-- [ ] Maintain list of known-good test models/datasets
-- [ ] Test new scripts with test inputs first
-- [ ] Document what makes inputs "good"
-- [ ] Keep test jobs cheap (small models, short timeouts)
-- [ ] Only move to production after test succeeds
-
-**Time cost:** 5-10 minutes for test run  
-**Debugging time saved:** Hours
-
----
-
-## Summary: The Reliability Checklist
-
-Before submitting ANY job:
-
-### Pre-Flight Checks
-- [ ] **Verified** all repos/datasets exist (hub_repo_details)
-- [ ] **Tested** with known-good inputs if new code
-- [ ] **Using** proven hardware/configuration
-- [ ] **Included** all dependencies in PEP 723 header
-- [ ] **Installed** system requirements (build tools, etc.)
-- [ ] **Set** appropriate timeout (not default 30m)
-- [ ] **Configured** Hub push with HF_TOKEN
-- [ ] **Added** clear error handling
-
-### Script Quality
-- [ ] Self-contained (no external setup needed)
-- [ ] Complete dependencies listed
-- [ ] Build tools installed by script
-- [ ] Progress indicators included
-- [ ] Error messages are clear
-- [ ] Configuration logged at start
-
-### Job Configuration
-- [ ] Timeout > expected runtime + 30% buffer
-- [ ] Hardware appropriate for model size
-- [ ] Secrets include HF_TOKEN
-- [ ] Environment variables set correctly
-- [ ] Cost estimated and acceptable
-
-**Following these principles transforms job success rate from ~60-70% to ~95%+**
-
----
-
-## When Principles Conflict
-
-Sometimes reliability and performance conflict. Here's how to choose:
-
-| Scenario | Choose | Rationale |
-|----------|--------|-----------|
-| Demo/test | Reliability | Fast failure is worse than slow success |
-| Production (first run) | Reliability | Prove it works before optimizing |
-| Production (proven) | Performance | Safe to optimize after validation |
-| Time-critical | Reliability | Failures cause more delay than slow runs |
-| Cost-critical | Balanced | Test with small model, then optimize |
-
-**General rule:** Reliability first, optimize second.
-
----
-
-## Further Reading
-
-- `troubleshooting.md` - Common issues and fixes
-- `training_patterns.md` - Proven training configurations
-- `gguf_conversion.md` - Production GGUF workflow

package/skills_ref/hugging-face-model-trainer/references/trackio_guide.md

@@ -1,189 +0,0 @@
-# Trackio Integration for TRL Training
-
-**Trackio** is an experiment tracking library that provides real-time metrics visualization for remote training on Hugging Face Jobs infrastructure.
-
-⚠️ **IMPORTANT**: For Jobs training (remote cloud GPUs):
-- Training happens on ephemeral cloud runners (not your local machine)
-- Trackio syncs metrics to a Hugging Face Space for real-time monitoring
-- Without a Space, metrics are lost when the job completes
-- The Space dashboard persists your training metrics permanently
-
-## Setting Up Trackio for Jobs
-
-**Step 1: Add trackio dependency**
-```python
-# /// script
-# dependencies = [
-#     "trl>=0.12.0",
-#     "trackio",  # Required!
-# ]
-# ///
-```
-
-**Step 2: Create a Trackio Space (one-time setup)**
-
-**Option A: Let Trackio auto-create (Recommended)**
-Pass a `space_id` to `trackio.init()` and Trackio will automatically create the Space if it doesn't exist.
-
-**Option B: Create manually**
-- Create Space via Hub UI at https://huggingface.co/new-space
-- Select Gradio SDK
-- OR use command: `huggingface-cli repo create my-trackio-dashboard --type space --space_sdk gradio`
-
-**Step 3: Initialize Trackio with space_id**
-```python
-import trackio
-
-trackio.init(
-    project="my-training",
-    space_id="username/trackio",  # CRITICAL for Jobs! Replace 'username' with your HF username
-    config={
-        "model": "Qwen/Qwen2.5-0.5B",
-        "dataset": "trl-lib/Capybara",
-        "learning_rate": 2e-5,
-    }
-)
-```
-
-**Step 4: Configure TRL to use Trackio**
-```python
-SFTConfig(
-    report_to="trackio",
-    # ... other config
-)
-```
-
-**Step 5: Finish tracking**
-```python
-trainer.train()
-trackio.finish()  # Ensures final metrics are synced
-```
-
-## What Trackio Tracks
-
-Trackio automatically logs:
-- ✅ Training loss
-- ✅ Learning rate
-- ✅ GPU utilization
-- ✅ Memory usage
-- ✅ Training throughput
-- ✅ Custom metrics
-
-## How It Works with Jobs
-
-1. **Training runs** → Metrics logged to local SQLite DB
-2. **Every 5 minutes** → Trackio syncs DB to HF Dataset (Parquet)
-3. **Space dashboard** → Reads from Dataset, displays metrics in real-time
-4. **Job completes** → Final sync ensures all metrics persisted
-
-## Default Configuration Pattern
-
-**Use sensible defaults for trackio configuration unless user requests otherwise.**
-
-### Recommended Defaults
-
-```python
-import trackio
-
-trackio.init(
-    project="qwen-capybara-sft",
-    name="baseline-run",             # Descriptive name user will recognize
-    space_id="username/trackio",     # Default space: {username}/trackio
-    config={
-        # Keep config minimal - hyperparameters and model/dataset info only
-        "model": "Qwen/Qwen2.5-0.5B",
-        "dataset": "trl-lib/Capybara",
-        "learning_rate": 2e-5,
-        "num_epochs": 3,
-    }
-)
-```
-
-**Key principles:**
-- **Space ID**: Use `{username}/trackio` with "trackio" as default space name
-- **Run naming**: Unless otherwise specified, name the run in a way the user will recognize
-- **Config**: Keep minimal - don't automatically capture job metadata unless requested
-- **Grouping**: Optional - only use if user requests organizing related experiments
-
-## Grouping Runs (Optional)
-
-The `group` parameter helps organize related runs together in the dashboard sidebar. This is useful when user is running multiple experiments with different configurations but wants to compare them together:
-
-```python
-# Example: Group runs by experiment type
-trackio.init(project="my-project", run_name="baseline-run-1", group="baseline")
-trackio.init(project="my-project", run_name="augmented-run-1", group="augmented")
-trackio.init(project="my-project", run_name="tuned-run-1", group="tuned")
-```
-
-Runs with the same group name can be grouped together in the sidebar, making it easier to compare related experiments. You can group by any configuration parameter:
-
-```python
-# Hyperparameter sweep - group by learning rate
-trackio.init(project="hyperparam-sweep", run_name="lr-0.001-run", group="lr_0.001")
-trackio.init(project="hyperparam-sweep", run_name="lr-0.01-run", group="lr_0.01")
-```
-
-## Environment Variables for Jobs
-
-You can configure trackio using environment variables instead of passing parameters to `trackio.init()`. This is useful for managing configuration across multiple jobs.
-
-
-
-**`HF_TOKEN`**
-Required for creating Spaces and writing to datasets (passed via `secrets`):
-```python
-hf_jobs("uv", {
-    "script": "...",
-    "secrets": {
-        "HF_TOKEN": "$HF_TOKEN"  # Enables Space creation and Hub push
-    }
-})
-```
-
-### Example with Environment Variables
-
-```python
-hf_jobs("uv", {
-    "script": """
-# Training script - trackio config from environment
-import trackio
-from datetime import datetime
-
-# Auto-generate run name
-timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M")
-run_name = f"sft_qwen25_{timestamp}"
-
-# Project and space_id can come from environment variables
-trackio.init(run_name=run_name, group="SFT")
-
-# ... training code ...
-trackio.finish()
-""",
-    "flavor": "a10g-large",
-    "timeout": "2h",
-    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
-})
-```
-
-**When to use environment variables:**
-- Managing multiple jobs with same configuration
-- Keeping training scripts portable across projects
-- Separating configuration from code
-
-**When to use direct parameters:**
-- Single job with specific configuration
-- When clarity in code is preferred
-- When each job has different project/space
-
-## Viewing the Dashboard
-
-After starting training:
-1. Navigate to the Space: `https://huggingface.co/spaces/username/trackio`
-2. The Gradio dashboard shows all tracked experiments
-3. Filter by project, compare runs, view charts with smoothing
-
-## Recommendation
-
-- **Trackio**: Best for real-time monitoring during long training runs
-- **Weights & Biases**: Best for team collaboration, requires account

package/skills_ref/hugging-face-model-trainer/references/training_methods.md

@@ -1,150 +0,0 @@
-# TRL Training Methods Overview
-
-TRL (Transformer Reinforcement Learning) provides multiple training methods for fine-tuning and aligning language models. This reference provides a brief overview of each method.
-
-## Supervised Fine-Tuning (SFT)
-
-**What it is:** Standard instruction tuning with supervised learning on demonstration data.
-
-**When to use:**
-- Initial fine-tuning of base models on task-specific data
-- Teaching new capabilities or domains
-- Most common starting point for fine-tuning
-
-**Dataset format:** Conversational format with "messages" field, OR text field, OR prompt/completion pairs
-
-**Example:**
-```python
-from trl import SFTTrainer, SFTConfig
-
-trainer = SFTTrainer(
-    model="Qwen/Qwen2.5-0.5B",
-    train_dataset=dataset,
-    args=SFTConfig(
-        output_dir="my-model",
-        push_to_hub=True,
-        hub_model_id="username/my-model",
-        eval_strategy="no",  # Disable eval for simple example
-        # max_length=1024 is the default - only set if you need different length
-    )
-)
-trainer.train()
-```
-
-**Note:** For production training with evaluation monitoring, see `scripts/train_sft_example.py`
-
-**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/sft_trainer")`
-
-## Direct Preference Optimization (DPO)
-
-**What it is:** Alignment method that trains directly on preference pairs (chosen vs rejected responses) without requiring a reward model.
-
-**When to use:**
-- Aligning models to human preferences
-- Improving response quality after SFT
-- Have paired preference data (chosen/rejected responses)
-
-**Dataset format:** Preference pairs with "chosen" and "rejected" fields
-
-**Example:**
-```python
-from trl import DPOTrainer, DPOConfig
-
-trainer = DPOTrainer(
-    model="Qwen/Qwen2.5-0.5B-Instruct",  # Use instruct model
-    train_dataset=dataset,
-    args=DPOConfig(
-        output_dir="dpo-model",
-        beta=0.1,  # KL penalty coefficient
-        eval_strategy="no",  # Disable eval for simple example
-        # max_length=1024 is the default - only set if you need different length
-    )
-)
-trainer.train()
-```
-
-**Note:** For production training with evaluation monitoring, see `scripts/train_dpo_example.py`
-
-**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/dpo_trainer")`
-
-## Group Relative Policy Optimization (GRPO)
-
-**What it is:** Online RL method that optimizes relative to group performance, useful for tasks with verifiable rewards.
-
-**When to use:**
-- Tasks with automatic reward signals (code execution, math verification)
-- Online learning scenarios
-- When DPO offline data is insufficient
-
-**Dataset format:** Prompt-only format (model generates responses, reward computed online)
-
-**Example:**
-```python
-# Use TRL maintained script
-hf_jobs("uv", {
-    "script": "https://raw.githubusercontent.com/huggingface/trl/main/examples/scripts/grpo.py",
-    "script_args": [
-        "--model_name_or_path", "Qwen/Qwen2.5-0.5B-Instruct",
-        "--dataset_name", "trl-lib/math_shepherd",
-        "--output_dir", "grpo-model"
-    ],
-    "flavor": "a10g-large",
-    "timeout": "4h",
-    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
-})
-```
-
-**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/grpo_trainer")`
-
-## Reward Modeling
-
-**What it is:** Train a reward model to score responses, used as a component in RLHF pipelines.
-
-**When to use:**
-- Building RLHF pipeline
-- Need automatic quality scoring
-- Creating reward signals for PPO training
-
-**Dataset format:** Preference pairs with "chosen" and "rejected" responses
-
-**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/reward_trainer")`
-
-## Method Selection Guide
-
-| Method | Complexity | Data Required | Use Case |
-|--------|-----------|---------------|----------|
-| **SFT** | Low | Demonstrations | Initial fine-tuning |
-| **DPO** | Medium | Paired preferences | Post-SFT alignment |
-| **GRPO** | Medium | Prompts + reward fn | Online RL with automatic rewards |
-| **Reward** | Medium | Paired preferences | Building RLHF pipeline |
-
-## Recommended Pipeline
-
-**For most use cases:**
-1. **Start with SFT** - Fine-tune base model on task data
-2. **Follow with DPO** - Align to preferences using paired data
-3. **Optional: GGUF conversion** - Deploy for local inference
-
-**For advanced RL scenarios:**
-1. **Start with SFT** - Fine-tune base model
-2. **Train reward model** - On preference data
-
-## Dataset Format Reference
-
-For complete dataset format specifications, use:
-```python
-hf_doc_fetch("https://huggingface.co/docs/trl/dataset_formats")
-```
-
-Or validate your dataset:
-```bash
-uv run https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py \
-  --dataset your/dataset --split train
-```
-
-## See Also
-
-- `references/training_patterns.md` - Common training patterns and examples
-- `scripts/train_sft_example.py` - Complete SFT template
-- `scripts/train_dpo_example.py` - Complete DPO template
-- [Dataset Inspector](https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py) - Dataset format validation tool