opencode-skills-collection 3.1.12 → 3.1.14

package/bundled-skills/hugging-face-model-trainer/SKILL.md

@@ -1,9 +1,13 @@
 ---
-source: "https://github.com/huggingface/skills/tree/main/skills/huggingface-llm-trainer"
 name: hugging-face-model-trainer
-description: Train or fine-tune TRL language models on Hugging Face Jobs, including SFT, DPO, GRPO, and GGUF export.
-license: Complete terms in LICENSE.txt
+description: Train or fine-tune language and vision models using TRL (Transformer Reinforcement Learning) or Unsloth with Hugging Face Jobs infrastructure. Covers SFT, DPO, GRPO and reward modeling training methods, plus GGUF conversion for local deployment. Includes guidance on the TRL Jobs...
 risk: unknown
+source: https://github.com/huggingface/skills/tree/main/skills/huggingface-llm-trainer
+source_repo: huggingface/skills
+source_type: official
+date_added: 2026-07-01
+license: Apache-2.0
+license_source: https://github.com/huggingface/skills/blob/main/LICENSE
 ---
 
 # TRL Training on Hugging Face Jobs
@@ -76,7 +80,7 @@ Before starting any training job, verify:
 - Hugging Face Account with [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan (Jobs require paid plan)
 - Authenticated login: Check with `hf_whoami()`
 - **HF_TOKEN for Hub Push** ⚠️ CRITICAL - Training environment is ephemeral, must push to Hub or ALL training results are lost
-- Token must have write permissions  
+- Token must have write permissions
 - **MUST pass `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config** to make token available (the `$HF_TOKEN` syntax
   references your actual token value)
 
@@ -422,6 +426,26 @@ Before submitting:
 
 **On timeout:** Job killed immediately, all unsaved progress lost, must restart from beginning
 
+## Choose a Base Model (Model Selection)
+
+**Identify models to train based on task type or benchmark results.**
+
+Use `scripts/hf_benchmarks.py` to identify top-performing models for specific tasks. This helps the user select a model as the base for training, whilst keeping size and hardware constraints in mind.
+
+```bash
+# Get help on the benchmarks command:
+uv run scripts/hf_benchmarks.py --help
+```
+
+### Example -- choosing an OCR base model
+```bash
+# Search for benchmarks containing whose name contains the text `ocr`
+uv run scripts/hf_benchmarks.py search --query ocr
+
+# Get the ranked leaderboard for the allenai/olmOCR-bench benchmark
+uv run scripts/hf_benchmarks.py leaderboard allenai/olmOCR-bench
+```
+
 ## Cost Estimation
 
 **Offer to estimate cost when planning jobs with known parameters.** Use `scripts/estimate_cost.py`:
@@ -467,7 +491,7 @@ These scripts demonstrate proper Hub saving, Trackio integration, checkpoint man
 - **Space ID**: `{username}/trackio` (use "trackio" as default space name)
 - **Run naming**: Unless otherwise specified, name the run in a way the user will recognize (e.g., descriptive of the task, model, or purpose)
 - **Config**: Keep minimal - only include hyperparameters and model/dataset info
-- **Project Name**: Use a Project Name to associate runs with a particular Project 
+- **Project Name**: Use a Project Name to associate runs with a particular Project
 
 **User overrides:** If user requests specific trackio configuration (custom space, run naming, grouping, or additional config), apply their preferences instead of defaults.
 
@@ -598,11 +622,16 @@ hf_jobs("uv", {
     "env": {
         "ADAPTER_MODEL": "username/my-finetuned-model",
         "BASE_MODEL": "Qwen/Qwen2.5-0.5B",
-        "OUTPUT_REPO": "username/my-model-gguf"
+        "OUTPUT_REPO": "username/my-model-gguf",
+        "TRUST_REMOTE_CODE": "0"
     }
 })
 ```
 
+Keep `TRUST_REMOTE_CODE=0` unless both model repositories have been reviewed and
+the architecture requires custom Python code. Setting it to `1` allows
+Transformers to import code from the model repository.
+
 ## Common Training Patterns
 
 See `references/training_patterns.md` for detailed examples including:
@@ -617,9 +646,9 @@ See `references/training_patterns.md` for detailed examples including:
 ### Out of Memory (OOM)
 
 **Fix (try in order):**
-1. Reduce batch size: `per_device_train_batch_size=1`, increase `gradient_accumulation_steps=8`. Effective batch size is `per_device_train_batch_size` x `gradient_accumulation_steps`. For best performance keep effective batch size close to 128. 
+1. Reduce batch size: `per_device_train_batch_size=1`, increase `gradient_accumulation_steps=8`. Effective batch size is `per_device_train_batch_size` x `gradient_accumulation_steps`. For best performance keep effective batch size close to 128.
 2. Enable: `gradient_checkpointing=True`
-3. Upgrade hardware: t4-small → l4x1, a10g-small → a10g-large etc. 
+3. Upgrade hardware: t4-small → l4x1, a10g-small → a10g-large etc.
 
 ### Dataset Misformatted
 
@@ -692,6 +721,7 @@ Add to PEP 723 header:
 - `scripts/unsloth_sft_example.py` - Unsloth text LLM training template (faster, less VRAM)
 - `scripts/estimate_cost.py` - Estimate time and cost (offer when appropriate)
 - `scripts/convert_to_gguf.py` - Complete GGUF conversion script
+- `scripts/hf_benchmarks.py` - Search for benchmark results and leaderboards by task, alias or free text.
 
 ### External Scripts
 - [Dataset Inspector](https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py) - Validate dataset format before training (use via `uv run` or `hf_jobs`)
@@ -719,6 +749,7 @@ Add to PEP 723 header:
 10. **Choose appropriate hardware** for model size; use LoRA for models >7B
 
 ## Limitations
-- Use this skill only when the task clearly matches the scope described above.
-- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
-- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
+
+- Use this skill only when the task clearly matches its upstream product or API scope.
+- Verify commands, API behavior, pricing, quotas, credentials, and deployment effects against current official documentation before making changes.
+- Do not treat generated examples as a substitute for environment-specific tests, security review, or user approval for destructive or costly actions.

package/bundled-skills/hugging-face-model-trainer/references/gguf_conversion.md

@@ -117,11 +117,16 @@ hf_jobs("uv", {
         "ADAPTER_MODEL": "username/my-finetuned-model",
         "BASE_MODEL": "Qwen/Qwen2.5-0.5B",
         "OUTPUT_REPO": "username/my-model-gguf",
+        "TRUST_REMOTE_CODE": "0",
         "HF_USERNAME": "username"  # Optional, for README
     }
 })
 ```
 
+Keep `TRUST_REMOTE_CODE=0` unless both model repositories have been reviewed and
+the architecture requires custom Python code. Setting it to `1` allows
+Transformers to import code from the model repository.
+
 ## Conversion Process
 
 The script performs these steps:

package/bundled-skills/hugging-face-model-trainer/scripts/convert_to_gguf.py

@@ -34,48 +34,25 @@ Usage:
     - BASE_MODEL: Base model used for fine-tuning (e.g., "Qwen/Qwen2.5-0.5B")
     - OUTPUT_REPO: Where to upload GGUF files (e.g., "username/my-model-gguf")
     - HF_USERNAME: Your Hugging Face username (optional, for README)
+    - TRUST_REMOTE_CODE: Set to "1" only for reviewed model repositories that require custom code
 
 Dependencies: All required packages are declared in PEP 723 header above.
 """
 
 import os
+import re
 import sys
 import torch
-import re
-import shutil
 from transformers import AutoModelForCausalLM, AutoTokenizer
 from peft import PeftModel
 from huggingface_hub import HfApi
 import subprocess
 
-HF_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*(/[A-Za-z0-9][A-Za-z0-9._-]*)?$")
-SAFE_FILENAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
-
-
-def require_hf_id(value, name):
-    if not HF_ID_RE.match(value or ""):
-        raise ValueError(f"{name} must be a Hugging Face model/repo id")
-    return value
-
-
-def safe_filename(value, name):
-    if not SAFE_FILENAME_RE.match(value or ""):
-        raise ValueError(f"{name} must be a safe filename segment")
-    return value
-
-
-def safe_output_file(root, filename):
-    root_path = os.path.abspath(root)
-    target = os.path.abspath(os.path.join(root_path, filename))
-    if os.path.commonpath([root_path, target]) != root_path:
-        raise ValueError(f"Output path escapes {root_path}")
-    return target
-
 
 def check_system_dependencies():
     """Check if required system packages are available."""
     print("🔍 Checking system dependencies...")
-    
+
     # Check for git
     if subprocess.run(["which", "git"], capture_output=True).returncode != 0:
         print("  ❌ git is not installed. Please install it:")
@@ -83,18 +60,18 @@ def check_system_dependencies():
         print("     RHEL/CentOS: sudo yum install git")
         print("     macOS: brew install git")
         return False
-    
+
     # Check for make or cmake
     has_make = subprocess.run(["which", "make"], capture_output=True).returncode == 0
     has_cmake = subprocess.run(["which", "cmake"], capture_output=True).returncode == 0
-    
+
     if not has_make and not has_cmake:
         print("  ❌ Neither make nor cmake found. Please install build tools:")
         print("     Ubuntu/Debian: sudo apt-get install build-essential cmake")
         print("     RHEL/CentOS: sudo yum groupinstall 'Development Tools' && sudo yum install cmake")
         print("     macOS: xcode-select --install && brew install cmake")
         return False
-    
+
     print("  ✅ System dependencies found")
     return True
 
@@ -103,22 +80,44 @@ def run_command(cmd, description):
     """Run a command with error handling."""
     print(f"   {description}...")
     try:
-        args = [str(part) for part in cmd]
-        if not args or any("\0" in part for part in args):
-            raise ValueError("Command arguments must be non-empty strings without NUL bytes")
-        executable = args[0] if os.path.isabs(args[0]) else shutil.which(args[0])
-        if not executable:
-            raise FileNotFoundError(args[0])
-        return_code = os.spawnv(os.P_WAIT, executable, args)
-        if return_code == 0:
-            return True
-        print(f"   ❌ Command failed with exit code {return_code}: {' '.join(args)}")
+        result = subprocess.run(
+            cmd,
+            check=True,
+            capture_output=True,
+            text=True
+        )
+        if result.stdout:
+            print(f"   {result.stdout[:200]}")  # Show first 200 chars
+        return True
+    except subprocess.CalledProcessError as e:
+        print(f"   ❌ Command failed: {' '.join(cmd)}")
+        if e.stdout:
+            print(f"   STDOUT: {e.stdout[:500]}")
+        if e.stderr:
+            print(f"   STDERR: {e.stderr[:500]}")
         return False
-    except (FileNotFoundError, OSError, ValueError) as e:
-        print(f"   ❌ Command failed: {e}")
+    except FileNotFoundError:
+        print(f"   ❌ Command not found: {cmd[0]}")
         return False
 
 
+HF_REPO_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,95}/[A-Za-z0-9][A-Za-z0-9._-]{0,95}$")
+
+
+def require_hf_repo_id(value, name):
+    """Reject local paths, URLs, and shell-like values before loading models."""
+    if not HF_REPO_ID_RE.fullmatch(value):
+        print(
+            f"   Invalid {name}: {value!r}. Use a Hugging Face repo id like owner/model.",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+
+def env_flag(name):
+    return os.environ.get(name, "").strip().lower() in {"1", "true", "yes", "on"}
+
+
 print("🔄 GGUF Conversion Script")
 print("=" * 60)
 
@@ -128,16 +127,21 @@ if not check_system_dependencies():
     sys.exit(1)
 
 # Configuration from environment variables
-ADAPTER_MODEL = require_hf_id(os.environ.get("ADAPTER_MODEL", "evalstate/qwen-capybara-medium"), "ADAPTER_MODEL")
-BASE_MODEL = require_hf_id(os.environ.get("BASE_MODEL", "Qwen/Qwen2.5-0.5B"), "BASE_MODEL")
-OUTPUT_REPO = require_hf_id(os.environ.get("OUTPUT_REPO", "evalstate/qwen-capybara-medium-gguf"), "OUTPUT_REPO")
-username = require_hf_id(os.environ.get("HF_USERNAME", ADAPTER_MODEL.split('/')[0]), "HF_USERNAME")
-TRUST_REMOTE_CODE = os.environ.get("TRUST_REMOTE_CODE", "").strip().lower() in {"1", "true", "yes"}
+ADAPTER_MODEL = os.environ.get("ADAPTER_MODEL", "evalstate/qwen-capybara-medium")
+BASE_MODEL = os.environ.get("BASE_MODEL", "Qwen/Qwen2.5-0.5B")
+OUTPUT_REPO = os.environ.get("OUTPUT_REPO", "evalstate/qwen-capybara-medium-gguf")
+username = os.environ.get("HF_USERNAME", ADAPTER_MODEL.split('/')[0])
+TRUST_REMOTE_CODE = env_flag("TRUST_REMOTE_CODE")
+
+require_hf_repo_id(ADAPTER_MODEL, "ADAPTER_MODEL")
+require_hf_repo_id(BASE_MODEL, "BASE_MODEL")
+require_hf_repo_id(OUTPUT_REPO, "OUTPUT_REPO")
 
 print(f"\n📦 Configuration:")
 print(f"   Base model: {BASE_MODEL}")
 print(f"   Adapter model: {ADAPTER_MODEL}")
 print(f"   Output repo: {OUTPUT_REPO}")
+print(f"   Trust remote code: {TRUST_REMOTE_CODE}")
 
 # Step 1: Load base model and adapter
 print("\n🔧 Step 1: Loading base model and LoRA adapter...")
@@ -170,7 +174,10 @@ except Exception as e:
 
 try:
     # Load tokenizer
-    tokenizer = AutoTokenizer.from_pretrained(ADAPTER_MODEL, trust_remote_code=TRUST_REMOTE_CODE)
+    tokenizer = AutoTokenizer.from_pretrained(
+        ADAPTER_MODEL,
+        trust_remote_code=TRUST_REMOTE_CODE,
+    )
     print("   ✅ Tokenizer loaded")
 except Exception as e:
     print(f"   ❌ Failed to load tokenizer: {e}")
@@ -224,8 +231,7 @@ os.makedirs(gguf_output_dir, exist_ok=True)
 
 convert_script = "/tmp/llama.cpp/convert_hf_to_gguf.py"
 model_name = ADAPTER_MODEL.split('/')[-1]
-model_name = safe_filename(model_name, "model_name")
-gguf_file = safe_output_file(gguf_output_dir, f"{model_name}-f16.gguf")
+gguf_file = f"{gguf_output_dir}/{model_name}-f16.gguf"
 
 print(f"   Running conversion...")
 if not run_command(
@@ -281,7 +287,7 @@ quant_formats = [
 quantized_files = []
 for quant_type, description in quant_formats:
     print(f"   Creating {quant_type} quantization ({description})...")
-    quant_file = safe_output_file(gguf_output_dir, f"{model_name}-{quant_type.lower()}.gguf")
+    quant_file = f"{gguf_output_dir}/{model_name}-{quant_type.lower()}.gguf"
 
     if not run_command(
         [quantize_bin, gguf_file, quant_file, quant_type],
@@ -291,7 +297,7 @@ for quant_type, description in quant_formats:
         continue
 
     quantized_files.append((quant_file, quant_type))
-    
+
     # Get file size
     size_mb = os.path.getsize(quant_file) / (1024 * 1024)
     print(f"   ✅ {quant_type}: {size_mb:.1f} MB")