opencode-skills-collection 4.0.69 → 4.0.70

package/bundled-skills/huggingface-community-evals/SKILL.md

@@ -0,0 +1,228 @@
+---
+description: Curated upstream guidance for Huggingface Community Evals; use when the workflow matches the user goal.
+name: huggingface-community-evals
+source_repo: huggingface/skills
+source_type: official
+source: huggingface
+date_added: '2026-09-21'
+risk: unknown
+---
+## When to Use
+- Use when this upstream workflow matches the user's stated goal.
+- Use when the task requires the procedures documented in this skill.
+
+# Overview
+
+This skill is for **running evaluations against models on the Hugging Face Hub on local hardware**.
+
+It covers:
+- `inspect-ai` with local inference
+- `lighteval` with local inference
+- choosing between `vllm`, Hugging Face Transformers, and `accelerate`
+- smoke tests, task selection, and backend fallback strategy
+
+It does **not** cover:
+- Hugging Face Jobs orchestration
+- model-card or `model-index` edits
+- README table extraction
+- Artificial Analysis imports
+- `.eval_results` generation or publishing
+- PR creation or community-evals automation
+
+If the user wants to **run the same eval remotely on Hugging Face Jobs**, hand off to the `hugging-face-jobs` skill and pass it one of the local scripts in this skill.
+
+If the user wants to **publish results into the community evals workflow**, stop after generating the evaluation run and hand off that publishing step to `~/code/community-evals`.
+
+> All paths below are relative to the directory containing this `SKILL.md`.
+
+# When To Use Which Script
+
+| Use case | Script |
+|---|---|
+| Local `inspect-ai` eval on a Hub model via inference providers | `scripts/inspect_eval_uv.py` |
+| Local GPU eval with `inspect-ai` using `vllm` or Transformers | `scripts/inspect_vllm_uv.py` |
+| Local GPU eval with `lighteval` using `vllm` or `accelerate` | `scripts/lighteval_vllm_uv.py` |
+| Extra command patterns | `examples/USAGE_EXAMPLES.md` |
+
+# Prerequisites
+
+- Prefer `uv run` for local execution.
+- Set `HF_TOKEN` for gated/private models.
+- For local GPU runs, verify GPU access before starting:
+
+```bash
+uv --version
+printenv HF_TOKEN >/dev/null
+nvidia-smi
+```
+
+If `nvidia-smi` is unavailable, either:
+- use `scripts/inspect_eval_uv.py` for lighter provider-backed evaluation, or
+- hand off to the `hugging-face-jobs` skill if the user wants remote compute.
+
+# Core Workflow
+
+1. Choose the evaluation framework.
+   - Use `inspect-ai` when you want explicit task control and inspect-native flows.
+   - Use `lighteval` when the benchmark is naturally expressed as a lighteval task string, especially leaderboard-style tasks.
+2. Choose the inference backend.
+   - Prefer `vllm` for throughput on supported architectures.
+   - Use Hugging Face Transformers (`--backend hf`) or `accelerate` as compatibility fallbacks.
+3. Start with a smoke test.
+   - `inspect-ai`: add `--limit 10` or similar.
+   - `lighteval`: add `--max-samples 10`.
+4. Scale up only after the smoke test passes.
+5. If the user wants remote execution, hand off to `hugging-face-jobs` with the same script + args.
+
+# Quick Start
+
+## Option A: inspect-ai with local inference providers path
+
+Best when the model is already supported by Hugging Face Inference Providers and you want the lowest local setup overhead.
+
+```bash
+uv run scripts/inspect_eval_uv.py \
+  --model meta-llama/Llama-3.2-1B \
+  --task mmlu \
+  --limit 20
+```
+
+Use this path when:
+- you want a quick local smoke test
+- you do not need direct GPU control
+- the task already exists in `inspect-evals`
+
+## Option B: inspect-ai on Local GPU
+
+Best when you need to load the Hub model directly, use `vllm`, or fall back to Transformers for unsupported architectures.
+
+Local GPU:
+
+```bash
+uv run scripts/inspect_vllm_uv.py \
+  --model meta-llama/Llama-3.2-1B \
+  --task gsm8k \
+  --limit 20
+```
+
+Transformers fallback:
+
+```bash
+uv run scripts/inspect_vllm_uv.py \
+  --model microsoft/phi-2 \
+  --task mmlu \
+  --backend hf \
+  --trust-remote-code \
+  --limit 20
+```
+
+## Option C: lighteval on Local GPU
+
+Best when the task is naturally expressed as a `lighteval` task string, especially Open LLM Leaderboard style benchmarks.
+
+Local GPU:
+
+```bash
+uv run scripts/lighteval_vllm_uv.py \
+  --model meta-llama/Llama-3.2-3B-Instruct \
+  --tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" \
+  --max-samples 20 \
+  --use-chat-template
+```
+
+`accelerate` fallback:
+
+```bash
+uv run scripts/lighteval_vllm_uv.py \
+  --model microsoft/phi-2 \
+  --tasks "leaderboard|mmlu|5" \
+  --backend accelerate \
+  --trust-remote-code \
+  --max-samples 20
+```
+
+# Remote Execution Boundary
+
+This skill intentionally stops at **local execution and backend selection**.
+
+If the user wants to:
+- run these scripts on Hugging Face Jobs
+- pick remote hardware
+- pass secrets to remote jobs
+- schedule recurring runs
+- inspect / cancel / monitor jobs
+
+then switch to the **`hugging-face-jobs`** skill and pass it one of these scripts plus the chosen arguments.
+
+# Task Selection
+
+`inspect-ai` examples:
+- `mmlu`
+- `gsm8k`
+- `hellaswag`
+- `arc_challenge`
+- `truthfulqa`
+- `winogrande`
+- `humaneval`
+
+`lighteval` task strings use `suite|task|num_fewshot`:
+- `leaderboard|mmlu|5`
+- `leaderboard|gsm8k|5`
+- `leaderboard|arc_challenge|25`
+- `lighteval|hellaswag|0`
+
+Multiple `lighteval` tasks can be comma-separated in `--tasks`.
+
+# Backend Selection
+
+- Prefer `inspect_vllm_uv.py --backend vllm` for fast GPU inference on supported architectures.
+- Use `inspect_vllm_uv.py --backend hf` when `vllm` does not support the model.
+- Prefer `lighteval_vllm_uv.py --backend vllm` for throughput on supported models.
+- Use `lighteval_vllm_uv.py --backend accelerate` as the compatibility fallback.
+- Use `inspect_eval_uv.py` when Inference Providers already cover the model and you do not need direct GPU control.
+
+# Hardware Guidance
+
+| Model size | Suggested local hardware |
+|---|---|
+| `< 3B` | consumer GPU / Apple Silicon / small dev GPU |
+| `3B - 13B` | stronger local GPU |
+| `13B+` | high-memory local GPU or hand off to `hugging-face-jobs` |
+
+For smoke tests, prefer cheaper local runs plus `--limit` or `--max-samples`.
+
+# Troubleshooting
+
+- CUDA or vLLM OOM:
+  - reduce `--batch-size`
+  - reduce `--gpu-memory-utilization`
+  - switch to a smaller model for the smoke test
+  - if necessary, hand off to `hugging-face-jobs`
+- Model unsupported by `vllm`:
+  - switch to `--backend hf` for `inspect-ai`
+  - switch to `--backend accelerate` for `lighteval`
+- Gated/private repo access fails:
+  - verify `HF_TOKEN`
+- Custom model code required:
+  - add `--trust-remote-code`
+
+# Examples
+
+See:
+- `examples/USAGE_EXAMPLES.md` for local command patterns
+- `scripts/inspect_eval_uv.py`
+- `scripts/inspect_vllm_uv.py`
+- `scripts/lighteval_vllm_uv.py`
+
+
+## Examples
+
+```text
+User: Apply this skill to my current task.
+Assistant: Follow the workflow in this skill, cite limitations, and ask before risky steps.
+```
+
+## Limitations
+
+- Imported upstream skill; verify credentials, permissions, and safety boundaries before execution.
+- Does not replace environment-specific validation, testing, or maintainer review.

package/bundled-skills/huggingface-community-evals/examples/.env.example

@@ -0,0 +1,3 @@
+# Hugging Face Token (required for gated/private models)
+# Get your token at: https://huggingface.co/settings/tokens
+HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

package/bundled-skills/huggingface-community-evals/examples/USAGE_EXAMPLES.md

@@ -0,0 +1,101 @@
+# Usage Examples
+
+This document provides practical examples for **running evaluations locally** against Hugging Face Hub models.
+
+## What this skill covers
+
+- `inspect-ai` local runs
+- `inspect-ai` with `vllm` or Transformers backends
+- `lighteval` local runs with `vllm` or `accelerate`
+- smoke tests and backend fallback patterns
+
+## What this skill does NOT cover
+
+- `model-index`
+- `.eval_results`
+- community eval publication workflows
+- model-card PR creation
+- Hugging Face Jobs orchestration
+
+If you want to run these same scripts remotely, use the `hugging-face-jobs` skill and pass one of the scripts in `scripts/`.
+
+## Setup
+
+```bash
+cd skills/hugging-face-evaluation
+export HF_TOKEN=hf_xxx
+uv --version
+```
+
+For local GPU runs:
+
+```bash
+nvidia-smi
+```
+
+## inspect-ai examples
+
+### Quick smoke test
+
+```bash
+uv run scripts/inspect_eval_uv.py \
+  --model meta-llama/Llama-3.2-1B \
+  --task mmlu \
+  --limit 10
+```
+
+### Local GPU with vLLM
+
+```bash
+uv run scripts/inspect_vllm_uv.py \
+  --model meta-llama/Llama-3.2-8B-Instruct \
+  --task gsm8k \
+  --limit 20
+```
+
+### Transformers fallback
+
+```bash
+uv run scripts/inspect_vllm_uv.py \
+  --model microsoft/phi-2 \
+  --task mmlu \
+  --backend hf \
+  --trust-remote-code \
+  --limit 20
+```
+
+## lighteval examples
+
+### Single task
+
+```bash
+uv run scripts/lighteval_vllm_uv.py \
+  --model meta-llama/Llama-3.2-3B-Instruct \
+  --tasks "leaderboard|mmlu|5" \
+  --max-samples 20
+```
+
+### Multiple tasks
+
+```bash
+uv run scripts/lighteval_vllm_uv.py \
+  --model meta-llama/Llama-3.2-3B-Instruct \
+  --tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" \
+  --max-samples 20 \
+  --use-chat-template
+```
+
+### accelerate fallback
+
+```bash
+uv run scripts/lighteval_vllm_uv.py \
+  --model microsoft/phi-2 \
+  --tasks "leaderboard|mmlu|5" \
+  --backend accelerate \
+  --trust-remote-code \
+  --max-samples 20
+```
+
+## Hand-off to Hugging Face Jobs
+
+When local hardware is not enough, switch to the `hugging-face-jobs` skill and run one of these scripts remotely. Keep the script path and args; move the orchestration there.

package/bundled-skills/huggingface-community-evals/scripts/inspect_eval_uv.py

@@ -0,0 +1,104 @@
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+#     "inspect-ai>=0.3.0",
+#     "inspect-evals",
+#     "openai",
+# ]
+# ///
+
+"""
+Entry point script for running inspect-ai evaluations against Hugging Face inference providers.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import subprocess
+import sys
+from pathlib import Path
+from typing import Optional
+
+
+def _inspect_evals_tasks_root() -> Optional[Path]:
+    """Return the installed inspect_evals package path if available."""
+    try:
+        import inspect_evals
+
+        return Path(inspect_evals.__file__).parent
+    except Exception:
+        return None
+
+
+def _normalize_task(task: str) -> str:
+    """Allow lighteval-style `suite|task|shots` strings by keeping the task name."""
+    if "|" in task:
+        parts = task.split("|")
+        if len(parts) >= 2 and parts[1]:
+            return parts[1]
+    return task
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Inspect-ai job runner")
+    parser.add_argument("--model", required=True, help="Model ID on Hugging Face Hub")
+    parser.add_argument("--task", required=True, help="inspect-ai task to execute")
+    parser.add_argument("--limit", type=int, default=None, help="Limit number of samples to evaluate")
+    parser.add_argument(
+        "--tasks-root",
+        default=None,
+        help="Optional path to inspect task files. Defaults to the installed inspect_evals package.",
+    )
+    parser.add_argument(
+        "--sandbox",
+        default="local",
+        help="Sandbox backend to use (default: local for HF jobs without Docker).",
+    )
+    args = parser.parse_args()
+
+    # Ensure downstream libraries can read the token passed as a secret
+    hf_token = os.getenv("HF_TOKEN")
+    if hf_token:
+        os.environ.setdefault("HUGGING_FACE_HUB_TOKEN", hf_token)
+        os.environ.setdefault("HF_HUB_TOKEN", hf_token)
+
+    task = _normalize_task(args.task)
+    tasks_root = Path(args.tasks_root) if args.tasks_root else _inspect_evals_tasks_root()
+    if tasks_root and not tasks_root.exists():
+        tasks_root = None
+
+    cmd = [
+        "inspect",
+        "eval",
+        task,
+        "--model",
+        f"hf-inference-providers/{args.model}",
+        "--log-level",
+        "info",
+        # Reduce batch size to avoid OOM errors (default is 32)
+        "--max-connections",
+        "1",
+        # Set a small positive temperature (HF doesn't allow temperature=0)
+        "--temperature",
+        "0.001",
+    ]
+
+    if args.sandbox:
+        cmd.extend(["--sandbox", args.sandbox])
+
+    if args.limit:
+        cmd.extend(["--limit", str(args.limit)])
+
+    try:
+        subprocess.run(cmd, check=True, cwd=tasks_root)
+        print("Evaluation complete.")
+    except subprocess.CalledProcessError as exc:
+        location = f" (cwd={tasks_root})" if tasks_root else ""
+        print(f"Evaluation failed with exit code {exc.returncode}{location}", file=sys.stderr)
+        raise
+
+
+if __name__ == "__main__":
+    main()
+