opencode-skills-collection 3.1.2 → 3.1.4

package/bundled-skills/remote-gpu-trainer/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yuyuan Han
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/bundled-skills/remote-gpu-trainer/README.md

@@ -0,0 +1,267 @@
+# remote-gpu-trainer
+
+**An Agent Skill for running long GPU jobs on machines you rent but don't own.** Deploy, train,
+monitor, and tear down safely across [AutoDL](https://www.autodl.com), RunPod, vast.ai, Lambda,
+Paperspace, the Chinese platforms (恒源云 / 矩池云 / Featurize / 揽睿星舟), bare SSH boxes, Slurm, and
+Kubernetes. One instance, or a fan-out of many.
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
+[![Agent Skills standard](https://img.shields.io/badge/Agent%20Skills-SKILL.md-blue)](https://agentskills.io)
+[![agentskills validate](https://img.shields.io/badge/agentskills%20validate-passing-brightgreen)](https://agentskills.io/specification)
+[![Platforms](https://img.shields.io/badge/platform%20profiles-7-orange)](#whats-inside)
+[![Status](https://img.shields.io/badge/status-pre--release-yellow)](#verification-status)
+
+> **Disambiguation:** "AutoDL" here is the **autodl.com** GPU-rental platform, not AutoML or NAS. And
+> this is an **Agent Skill** — a `SKILL.md` with reference docs and script templates — not a CLI or an
+> SDK. It rides *above* each platform's API and encodes the operational survival knowledge those APIs
+> leave out.
+
+The whole skill is built on one mental model: **you are a short-term tenant on someone else's machine.**
+So it teaches tenant survival — detach the job, make the result outlive the box, stop the meter without
+losing data — and treats that as a single model across every backend. Only the per-platform specifics
+(stop-vs-destroy billing, machine-locked volumes, `/root` ephemerality, acceleration proxy vs HF mirror,
+spot grace) get pushed down into one profile per platform.
+
+```mermaid
+flowchart TD
+    TASK(["Your task: deploy / train / monitor / tear down<br/>a job on a GPU box you rent, not own"])
+    TASK --> MATCH{"description keywords<br/>match the task?"}
+    MATCH -->|skill activates| HUB
+    HUB["<b>SKILL.md</b> — the always-loaded hub<br/>10 operating principles · 6-phase lifecycle · platform selector"]
+    HUB --> CORE["<b>references/</b><br/>platform-agnostic core"]
+    HUB --> PROF["<b>profiles/</b><br/>per-platform specifics"]
+    HUB --> EXEC["<b>scripts · examples · evals</b>"]
+    CORE --> CORE1["principles · gotchas U1–U39 · monitoring<br/>spot-resilience · ssh · china-network"]
+    CORE --> CORE2["training/ ×8 — the DL-debug layer<br/>OOM · NCCL-hang · NaN · throughput · ckpt · convergence · data"]
+    PROF --> PROF1["autodl (deepest) · runpod · vastai · lambda<br/>paperspace · china · generic-ssh"]
+    EXEC --> EXEC1["runnable wrappers + monitors · one worked example<br/>no-API-key retrieval drift-guard"]
+```
+
+## Contents
+
+[Why this exists](#why-this-exists) · [How it differs](#how-it-differs) ·
+[Architecture and layout](#architecture-and-layout) · [Install and deploy](#install-and-deploy) ·
+[What's inside](#whats-inside) · [Scope](#scope) · [Verification status](#verification-status) ·
+[Disclaimer](#disclaimer) · [中文简介](#中文简介) · [Contributing](#contributing) ·
+[License](#license) · [Citing](#citing)
+
+## Why this exists
+
+Renting a GPU is the easy part. The expensive surprises come from everything around the job: a stopped
+box that quietly keeps billing, a "synced" checkpoint that never actually wrote because the disk ran out
+of inodes, a download that stalls behind the wrong mirror, a `terminate` that deletes the only copy of a
+week's training. None of that is in a platform's API docs, and most of it only bites once you've already
+paid for it.
+
+This skill collects that knowledge into a form an agent can act on: ten operating principles for *why*
+each step matters, a six-phase lifecycle that ends every phase in a runnable check, and one profile per
+platform that pins the concrete commands. It is opinionated about the things that cost money or data, and
+quiet about the rest.
+
+## How it differs
+
+General orchestrators — **SkyPilot**, **dstack**, **Modal** — own or abstract the infrastructure and
+price-shop across Western clouds. They are excellent at that, and this skill does not compete with them.
+But none of them supports AutoDL or the Chinese platforms, and each assumes its own daemon or cluster
+model.
+
+`remote-gpu-trainer` meets you on the **raw rented instance you already control**, and concentrates on a
+blind spot those tools leave open: the Chinese platforms and bare-SSH cheap rentals, where disk-budget
+design, inode caps, mirror stalls, cgroup OOM, spot-grace windows, and *irreversible* teardown are the
+actual job. The two approaches compose well: let SkyPilot or dstack move the box for you, then let this
+skill make your *code* resume-correct so their recovery actually restores progress.
+
+## Architecture and layout
+
+The design follows the Agent Skills idea of **progressive disclosure**: a small always-loaded hub, and
+deeper material loaded only when a phase needs it. The split that makes it portable is
+**platform-agnostic core, platform-specific edges** — the principles and lifecycle hold everywhere, and
+every concrete path, proxy, billing verb, and spot rule lives in exactly one place, the profile.
+
+The six-phase lifecycle is the operational spine. Each phase delegates its substrate to the active
+profile and ends in a check you can run:
+
+```mermaid
+flowchart LR
+    P0["0 · audit<br/>df -i · cgroup · GPU"] --> P1["1 · ssh + creds"]
+    P1 --> P2["2 · CPU smoke<br/><i>before you rent</i>"]
+    P2 --> P3["3 · detached launch"]
+    P3 --> P4["4 · durable monitor<br/>(four-layer)"]
+    P4 --> P5["5 · verify + teardown<br/><b>Iron Law</b>"]
+```
+
+The folders map onto that architecture directly:
+
+```text
+remote-gpu-trainer/
+├── SKILL.md                     # the hub: 10 principles + 6-phase lifecycle + platform selector
+├── references/                  # platform-agnostic knowledge, loaded on demand
+│   ├── principles.md            #   the 10 invariants, expanded with cross-platform nuance
+│   ├── lifecycle_checklist.md   #   the 6 phases as a per-platform checklist
+│   ├── gotchas_universal.md     #   U1–U39, symptom → root cause → fix (U36–U38 are cross-links)
+│   ├── monitoring_patterns.md   #   four-layer durable monitoring + cross-host portability map
+│   ├── spot-resilience.md       #   preemption signals, Young/Daly cadence, atomic-write resume
+│   ├── ssh_transport.md         #   ssh config, resumable rsync/scp, secrets via stdin, CRLF
+│   ├── china-network.md         #   mirrors, HF_ENDPOINT, the no_proxy trap
+│   ├── parallel_ablation.md     #   FS-shared fan-out + the reconciliation step
+│   ├── multinode.md             #   NCCL / fabric-manager / elastic training (advanced)
+│   ├── self-improvement.md      #   how the skill captures new gotchas without corrupting itself
+│   └── training/                #   the DL-training debug layer — when the run breaks, not the box
+│       ├── oom-memory.md            #   CUDA/host OOM + the fit-it ladder
+│       ├── distributed-launch.md    #   torchrun/accelerate/deepspeed + the multi-GPU HANGS toolkit
+│       ├── precision-stability.md   #   fp16/bf16/tf32, NaN/Inf hunting, LLM loss spikes
+│       ├── throughput-profiling.md  #   GPU-bound vs data-bound vs comms-bound
+│       ├── checkpoint-resume.md     #   full-state + sharded save/resume, the resume bugs
+│       ├── by-domain.md             #   LLM / vision / diffusion / RL / multimodal gotchas
+│       ├── convergence-debugging.md #   runs but won't learn: optimizer/LR/loss-fn/freezing
+│       └── data-pipeline.md         #   dataloader & dataset correctness (not speed)
+├── profiles/                    # one file per platform — the only place concrete specifics live
+│   ├── _schema.md               #   the shared 8-field contract every profile fills
+│   ├── autodl.md                #   deepest, battle-tested
+│   ├── runpod.md  vastai.md  lambda.md  paperspace.md
+│   ├── china.md                 #   恒源云 / 矩池云 / Featurize / 揽睿星舟
+│   └── generic-ssh.md           #   bare SSH / Slurm / K8s / Colab-Kaggle
+├── scripts/                     # parameterized, runnable templates
+│   ├── run_one.sh.template  run_queue.sh.template  health_patrol.sh.template
+│   ├── mem_monitor.sh  gpu_health.sh  reap_vram_zombies.sh
+│   ├── aggregate_to_fs.sh  download_loop.sh  setup-china-mirrors.sh
+│   └── verify_local.py          #   load-and-verify each artifact before any teardown
+├── examples/autodl_sweep/       # one complete worked case, end to end
+└── evals/                       # cases.jsonl + run_evals.py (no-API-key drift guard) + RESULTS.md
+```
+
+Each profile fills the same eight fields, so a platform you've never used reads like one you have:
+launch · storage survival-matrix · network · spot/resume · teardown/billing · daemon · gotchas · script
+overrides.
+
+## Install and deploy
+
+This is a standard [Agent Skill](https://agentskills.io): one folder with a `SKILL.md` at its root.
+Installing it means cloning that folder into wherever your agent looks for skills, then restarting the
+agent. It auto-triggers on remote or rented-GPU deploy / train / monitor tasks — you don't invoke it by
+name. Keep the folder named `remote-gpu-trainer`; the standard requires the directory name to match the
+skill's `name:` field.
+
+**Claude Code**
+
+```bash
+git clone https://github.com/Hanyuyuan6/remote-gpu-trainer.git ~/.claude/skills/remote-gpu-trainer
+```
+
+**OpenAI Codex**
+
+```bash
+git clone https://github.com/Hanyuyuan6/remote-gpu-trainer.git ~/.agents/skills/remote-gpu-trainer
+```
+
+**Cursor · Trae · Gemini CLI · VS Code / Copilot · Goose · Kiro · other compatible agents**
+
+Clone the same folder into that agent's skills directory (each agent's docs, or
+[agentskills.io](https://agentskills.io), give the exact location). Because they all read the same open
+`SKILL.md` standard, the folder works unchanged across every one of them.
+
+**Verify the install (optional).** With [uv](https://github.com/astral-sh/uv):
+
+```bash
+uvx --from skills-ref agentskills validate ~/.claude/skills/remote-gpu-trainer   # → "Valid skill"
+```
+
+> **Two caveats.** The companion skills this one cross-links (`verifying-dl-experiments`,
+> `superpowers:*`, `huggingface-skills:*`) are optional separate installs; it works standalone without
+> them. And a few durable-monitoring recipes assume a host background-task runner plus a scheduler — map
+> those to your agent's equivalents, using the per-host table in `references/monitoring_patterns.md` §7.
+
+## What's inside
+
+- **`SKILL.md`** — the hub. Ten platform-agnostic operating principles, the six-phase lifecycle with a
+  runnable gate per phase, the platform selector, and the cross-links into everything below.
+- **`references/`** — the platform-agnostic knowledge: `principles.md` (the ten invariants expanded),
+  `gotchas_universal.md` (U1–U39, each a `symptom → root cause → fix`; U36–U38 are delegated cross-links), `monitoring_patterns.md`
+  (four-layer durable monitoring plus a cross-host portability map), and the focused playbooks for SSH
+  transport, China networking, spot resilience, parallel ablation, multi-node, and self-improvement.
+- **`references/training/`** — the **DL-training debug layer**, eight files for when the *run* breaks
+  rather than the platform: OOM, distributed launch and multi-GPU hangs, precision and loss spikes,
+  throughput profiling, checkpoint/resume, per-domain gotchas, convergence ("runs but won't learn"), and
+  dataloader correctness.
+- **`profiles/`** — one file per platform, the only place concrete specifics live. `autodl` is the
+  deepest; alongside it are `runpod`, `vastai`, `lambda`, `paperspace`, `china`, and `generic-ssh`
+  (covering Slurm, K8s, Colab, Kaggle). `_schema.md` defines the shared eight-field contract.
+- **`scripts/`** — parameterized wrapper templates, a memory monitor, a GPU-health probe, a VRAM-zombie
+  reaper, a read-only health-patrol tick, FS aggregation, a resumable download loop, the China-mirror
+  setup, and a load-and-verify checker.
+- **`examples/autodl_sweep/`** — one complete worked case, end to end.
+- **`evals/`** — a retrieval drift-guard: `cases.jsonl` holds realistic scenarios, `run_evals.py` checks
+  with no API key that every scenario's answer is still present at its documented location, and
+  `RESULTS.md` records fresh-agent navigation runs.
+
+## Scope
+
+- **For:** rented or remote GPU instances (Chinese and Western clouds, bare SSH, Slurm, K8s); single or
+  multi-instance; long-running jobs — training, eval, ablation sweeps, batch inference, large data
+  processing.
+- **Not for:** purely-local single-GPU training, in-instance multi-GPU DDP (use `torchrun` /
+  `accelerate`), managed multi-cloud price-shopping (use SkyPilot's skill), or zero-ops serverless (use
+  Modal).
+
+## Verification status
+
+The **AutoDL** profile reflects the author's hands-on, daily use. The other six profiles — RunPod,
+vast.ai, Lambda, Paperspace, the Chinese platforms, and the generic SSH / Slurm / K8s core — are
+researched from each platform's official documentation and community reports. Every money-affecting fact
+is cited inline and stamped `verified <month>`, but they are **not yet independently live-tested** by the
+author. Treat them as a well-sourced starting map, not a guarantee.
+
+The skill is built to **verify before any irreversible or costly action** (the Phase-0 live measurement,
+the teardown Iron Law), so a stale fact surfaces as "re-check the docs," not a silent loss. Corrections,
+and "I ran this, here's what changed" reports, are very welcome — please open an issue or PR.
+
+## Disclaimer
+
+This is an independent community resource. It is **not affiliated with, endorsed by, or sponsored by**
+AutoDL, RunPod, vast.ai, Lambda, Paperspace, DigitalOcean, or any platform named here. All product names
+and trademarks belong to their respective owners and are used **nominatively**, only to identify the
+platform a piece of guidance applies to. Platform facts are synthesized from public documentation and
+community reports (cited inline) and were accurate at the noted `verified` date. **Platforms change their
+pricing, billing verbs, and limits, so verify against current official docs before relying on a teardown
+or billing fact** (see `references/self-improvement.md` §5). Provided "as is" under the MIT License,
+without warranty.
+
+## 中文简介
+
+面向在**租来的 / 远程 GPU**(不是你自己的机器)上跑长任务的研究者与工程师,覆盖 AutoDL、RunPod、
+vast.ai、Lambda、Paperspace、国内平台(恒源云 / 矩池云 / Featurize / 揽睿星舟)、裸 SSH 机器、Slurm、
+Kubernetes,单机或多机并行。
+
+核心隐喻:**你是别人机器上的短期租客。** 所以技能教的是「让作业活过这台租来的机器」:把作业 detach、
+让结果先于实例存活、再安全地停掉计费。一套心智模型跨所有后端,只把每个平台的差异(停止 vs 销毁的计费、
+机器锁定的网盘、`/root` 是否易失、加速代理 vs HF 镜像、spot 抢占宽限)参数化下沉到各
+`profiles/<平台>.md`。
+
+它专注的,正是 SkyPilot / dstack / Modal 这类抽象层略过的盲区:**AutoDL + 国内平台 + 裸 SSH 廉价租卡**
+上的磁盘预算、inode 上限、镜像卡顿、cgroup OOM、spot 宽限窗口,以及不可逆的销毁操作。安装方式见
+[Install and deploy](#install-and-deploy):把整个文件夹克隆进对应 agent 的 skills 目录即可,重启后自动
+触发。
+
+## Contributing
+
+Issues and PRs are welcome, especially **new platform profiles** and **new gotchas** with a concrete
+`symptom → root cause → fix`. Keep every example generic: no real project names, hostnames, IPs, ports,
+or keys. The `references/self-improvement.md` protocol describes the bar a new gotcha has to clear
+(root-caused, reproduced, generalizable) before it earns a place in the catalog.
+
+## License
+
+MIT — see [LICENSE](LICENSE). Copyright (c) 2026 Yuyuan Han.
+
+## Citing
+
+A link back is plenty. If you need a formal reference:
+
+```bibtex
+@software{han_remote_gpu_trainer_2026,
+  author = {Han, Yuyuan},
+  title  = {remote-gpu-trainer: an Agent Skill for long GPU jobs on rented instances},
+  year   = {2026},
+  url    = {https://github.com/Hanyuyuan6/remote-gpu-trainer}
+}
+```

package/bundled-skills/remote-gpu-trainer/SKILL.md

@@ -0,0 +1,249 @@
+---
+name: remote-gpu-trainer
+description: "Deploy, monitor, and debug long GPU jobs on RENTED/remote instances (AutoDL, RunPod, vast.ai, Lambda, Slurm, K8s): teardown/billing safety, spot resilience, resumable checkpointing, OOM/NaN triage."
+risk: safe
+source: community
+source_type: community
+source_repo: Hanyuyuan6/remote-gpu-trainer
+date_added: "2026-06-20"
+category: ml-ops
+license: "MIT"
+license_source: "https://github.com/Hanyuyuan6/remote-gpu-trainer/blob/main/LICENSE"
+compatibility: |
+  Any Agent-Skills (SKILL.md)-compatible agent — Claude Code, Codex, Cursor, Trae, Gemini CLI, etc.
+  Needs a shell + SSH (or a platform CLI/API) to drive the remote box; scripts are bash/python. A few
+  durable-monitoring recipes assume a host background-task runner + scheduler — map to the running
+  agent's equivalents (references/monitoring_patterns.md §7). Companion skills (verifying-dl-experiments,
+  superpowers:*, huggingface-skills:*) are optional separate installs.
+---
+
+# remote-gpu-trainer — Remote GPU Job Orchestration
+
+## Overview
+
+Deploy and babysit long-running GPU jobs on **rented boxes you don't own**, across any platform, and
+get the result off the box before the meter or a preemption kills it. The core insight: **you are a
+short-term tenant on someone else's machine** — so the job is to *detach the work, make the result
+outlive the instance, and stop the meter safely*, not to provision a cluster.
+
+This skill is **platform-agnostic at the core, platform-specific at the edges**: a fixed set of
+operating principles + a 6-phase lifecycle that hold everywhere, plus one **profile per platform**
+(`profiles/<platform>.md`) that owns every concrete path, proxy, billing verb, and spot semantic. Its
+defensible value is the union the big orchestrators skip: **Chinese cgroup-isolated rentals + bare-SSH
+cheap boxes + the disk-budget / monitoring / teardown reality** that *is* the job on metered hardware.
+
+## When to Use This Skill
+
+Use whenever the user deploys, trains, monitors, or troubleshoots a long-running GPU job on a **RENTED
+or remote instance they do not own** — training, eval, ablation sweeps, batch inference, or large data
+processing — on AutoDL, RunPod, vast.ai, Lambda, Paperspace, Chinese platforms (恒源云/矩池云/Featurize/
+揽睿星舟), a bare SSH box, Slurm, or Kubernetes; single OR multi-instance. Triggers (multilingual):
+"远程 GPU 训练", "GPU 租赁", "GPU rental", "租卡", "spot 抢占", "spot preemption", "断点续训",
+"resumable training", "tmux 训练守护", "防 SSH 断线", "scp/rsync 上传", "多实例 ablation",
+"远程 GPU 监控", "省钱关机/销毁实例", "stop vs terminate billing", "checkpoint 磁盘满",
+"CUDA OOM/显存不足", "loss NaN/loss spike", "loss 不下降/不收敛", "overfit 单 batch",
+"FSDP/DeepSpeed 配置", "多卡训练 hang", "dataloader worker/数据增广 bug". **NOT** for purely local
+single-GPU training, in-instance multi-GPU DDP (use torchrun/accelerate), managed multi-cloud
+price-shopping (use SkyPilot's skill), or zero-ops serverless (use Modal).
+
+## When NOT to use — and what to use instead
+
+| Situation | Use instead |
+|---|---|
+| Local single-GPU, or multi-GPU **DDP inside one box** | `torchrun` / `accelerate` directly |
+| Managed multi-cloud price-shopping + auto spot-recovery across **Western** clouds | **SkyPilot** (has its own Agent Skill) — then come back here to make your *code* resume-correct so its recovery actually works |
+| Open BYOC dev environments | **dstack** |
+| Zero-ops serverless inference | **Modal** |
+| "Is this metric / ablation delta real?" | **REQUIRED:** `verifying-dl-experiments` (this skill owns *running* the job; that one owns *whether the number is true*) |
+
+**This skill is for the blind spot those tools leave:** AutoDL + Chinese platforms, bare SSH/Slurm/K8s
+rentals, and the operational gotchas (inode caps, mirror stalls, cgroup OOM, silent sync, spot grace
+windows, irreversible teardown) that survive whichever provisioner you use.
+
+## Operating principles (the WHY — 10 invariants)
+
+These hold on every metered, isolated, rented GPU; only the paths/CLI change. One line each; the deep
+form with cross-platform nuance is in **`references/principles.md`** (read it before Phase 0).
+
+1. **Minimize paid wall-clock.** The meter runs the whole time — smoke locally on CPU before renting, launch detached, release the instant verification passes.
+2. **Cheap checks before expensive compute.** A 1–2 batch CPU smoke (logger off) kills import/config/shape/scale bugs for ~free. (Smoke *content* → `verifying-dl-experiments`.)
+3. **Trust artifacts you loaded, not log lines that claim success.** "synced/saved/done" lies under a silently-failed write; a watcher's own state is also a claim — reconcile it against the real process/artifact.
+4. **Know what survives stop vs destroy.** Per platform, identify exactly which mount survives a *stop* and which survives a *terminate* — the data you need often lives on the volatile one. (The single biggest portability trap.)
+5. **Storage fails on the dimension you're not watching.** Disk dies on **inodes** before bytes; the real hog hides in a symlinked cache; clean by value (keep tiny evidence, drop big scratch); monitor `df -i`, not just `df -h`.
+6. **Never mutate inputs under a live run.** A running job holds its scripts in memory by byte-offset; overwriting one mid-run re-executes blocks. Version filenames.
+7. **Design for retry — failure is probabilistic, transfers are flaky, mirrors are route-specific.** Make wrappers idempotent + resumable; retry the *identical* config; wrap bulk transfers in `timeout`+resume loops; a mirror/proxy speeds ONE route — validate on the same route the real transfer uses.
+8. **Checkpoint-to-durable + idempotent resume is the universal spine.** File checkpoint to the platform's durable location + unconditional load-latest-on-startup is the *one* mechanism that survives an SSH drop, a Slurm walltime kill, a K8s reschedule, a spot preemption, and a Colab disconnect. The detach primitive (tmux/sbatch/Job/commit) is the swappable plug; this is the invariant.
+9. **Cost and destructive actions are the user's call.** Never auto-release/terminate, never delete durable files without confirmation; if cleanup can't free space, **ask to expand the disk** rather than silently shrink the experiment.
+10. **Teach the user the platform, don't just drive it.** Most users don't know a platform's non-obvious **conveniences** (one-click SSH-key registration, GPU-availability notifications, built-in panels) or its **danger clocks** (auto-release/auto-delete timers on a *stopped* box — AutoDL releases a 关机 instance after 15 days → data disk gone; a stop that keeps billing; low-balance purge). Surface them on first contact — #9 stops the agent *doing* the dangerous thing, #10 *warns the human* before the clock fires. Per-platform list → each profile's **Surface to the user** block.
+
+> **Monitoring physics (substrate for #3):** foreground Bash hard-caps at 600 s; `run_in_background` has no cap and notifies on exit; a never-exiting watcher never notifies; an unquoted `|` in a poll regex reads stdin and hangs forever. The four-layer monitoring architecture is built on these facts → `references/monitoring_patterns.md`.
+
+## Code discipline (the wrapper & training scripts you write)
+
+Two rules govern the launch/wrapper/training code this skill has you write — corollaries of #1 and #8, not new invariants:
+
+1. **Reuse before writing.** Take the lowest rung that already works before adding code: the base image's pre-installed stack + platform features → a framework/library utility (`torchrun` / `accelerate` / HF) → your existing `scripts/` templates → minimal new code. On a metered box a needless `pip install` also burns paid wall-clock and can break the image's ABI — Phase 1's rule (*the prebuilt image **is** the env; don't `conda create` on a rental*) is exactly this principle applied to dependencies.
+2. **Floor — `minimum` bounds scope, not correctness.** Shrinking code must never drop what makes an expensive run survivable: checkpoint-to-durable + idempotent resume (#8), atomic writes, the error handling that prevents losing a long run, or seed/determinism logging. Keep one minimal self-check for non-trivial logic.
+
+## Pick your platform profile FIRST
+
+Read the matching profile **before Phase 0** — it owns every path, proxy, credential location, billing
+verb, and spot rule the phases below delegate to. Each follows the same 8-field schema
+(`profiles/_schema.md`).
+
+> **New here? The path is:** (1) find your platform in the table below → (2) read that profile's **LAUNCH**
+> section (it walks rent → register SSH key → reach the box) → (3) come back and run the 6 phases from Phase 0.
+> Already have a box you can `ssh` into? Skip straight to Phase 0.
+
+| You're on… | Profile | Kind | Detach primitive | Meter-stop verb |
+|---|---|---|---|---|
+| AutoDL (deepest, battle-tested) | `profiles/autodl.md` | ssh-rental | tmux | 关机 (stops meter, **keeps disk** — the AutoDL exception) |
+| RunPod | `profiles/runpod.md` | ssh-rental | tmux | **terminate** (stop still bills 2×; destroys volume disk) |
+| vast.ai | `profiles/vastai.md` | ssh-rental (spot) | tmux | **destroy** (stop bills disk forever) |
+| Lambda | `profiles/lambda.md` | cloud-api | tmux | **terminate** (no stop state) |
+| Paperspace | `profiles/paperspace.md` | cloud-api | tmux | **destroy + release IP + delete storage** (shut-down stops compute only) |
+| 恒源云 / 矩池云 / Featurize / 揽睿星舟 | `profiles/china.md` | ssh-rental | tmux | per-platform (data disk often bills while stopped) |
+| Bare SSH box / Slurm / K8s / Colab-Kaggle | `profiles/generic-ssh.md` | ssh / slurm / k8s | tmux / sbatch / Job / commit | **manual** (a forgotten box bills 24/7) |
+
+> **Profile confidence:** AutoDL is battle-tested from the author's daily use; the other six profiles are
+> built from each platform's official docs + community reports (cited inline, `verified <month>`) and not
+> yet independently live-tested — lean on the Phase-0 live measurements and **re-verify any teardown/
+> billing fact against current docs before betting money or data** (`references/self-improvement.md` §5).
+
+**Mental verb model** (one API across all platforms; the profile binds each verb to real commands):
+`up` (rent+reach) → `push` (code/data on) → `run` (detached + checkpointing) → `watch` (durable monitor) → `pull` (results off + verify) → `down` (stop the meter).
+
+## Default workflow (6 phases)
+
+Skip phases already done. Each phase delegates substrate to the profile and **ends in a runnable check**.
+
+**Phase 0 — Environment audit.** Read the profile's STORAGE survival-matrix + region/DC-lock. Measure live:
+`df -h && df -i <data-mount>`, cgroup `memory.max`, `nvidia-smi`. Pre-compute the checkpoint disk budget
+(`ckpt_size × N + scratch`). → **verify:** `nvidia-smi` shows the expected GPU and `df -i` is not near 100%.
+
+**Phase 1 — SSH + credentials.** Set the alias/env per the profile (the prebuilt image/base IS the env —
+do not `conda create` on a rental). **Never rented before? the profile's LAUNCH section walks rent → register SSH key → connect.** Push secrets via **stdin, never onto a shared/durable FS**
+(`references/ssh_transport.md`). → **verify:** `ssh <alias> 'python -c "import torch;print(torch.cuda.is_available())"'`.
+
+**Phase 2 — Wrapper + CPU-smoke gate.** Build an idempotent `run_one`/`run_queue` from `scripts/` (parameterized
+from the profile's OVERRIDES; **size batch/workers to the box for a standalone run, but PIN them across cells for a fair comparison** — `references/training/throughput-profiling.md`). **Run the cheap CPU smoke locally BEFORE renting** — it kills the dumb,
+expensive failures (e.g. `python -m <your.train.module> --limit-batches 2 --epochs 1` — substitute your own entrypoint; this gate needs your training code plugged in). → **verify:** that smoke exits 0 on 2 batches with the logger disabled.
+
+**Phase 3 — Detached launch.** Launch via the profile's detach primitive; probe briefly (log head + alive +
+no traceback), then **hand back** — never a blocking foreground `sleep`. → **verify:** within 60 s, the detach
+session is alive and the first log line shows the expected step/epoch.
+
+**Phase 4 — Durable monitoring.** For anything over ~1–2 h, deploy the **four-layer architecture**
+(`references/monitoring_patterns.md`): on-box self-completion chain + session patrol loop + event sentinels +
+recovery handbook. **On Claude Code, fire the L2 patrol via `/loop 30m` (or `ScheduleWakeup`) running `scripts/health_patrol.sh.template`**; a host with no local recurring runner wires the on-box self-push instead (`references/monitoring_patterns.md` §7). A session-bound watcher alone dies with the session. Classify each outcome →
+fixed remediation; **never blind-retry**. → **verify:** the patrol reports even when nothing changed.
+
+**Phase 5 — Aggregate + verify + teardown.** Checked-sync to durable storage (gate the success line on the
+copy result — principle #3), then **load-and-verify each artifact** (`scripts/verify_local.py`), THEN the profile's
+meter-stopping action. → **verify:** `verify_local.py` reports 100% OK *before* any teardown.
+
+> **Iron Law — teardown gate:** NO `release` / `terminate` / `destroy` / file-delete until checkpoints are
+> **pulled to local AND verified by load**, and the user has explicitly approved the cost-affecting action.
+> "It looked done in the log" is not evidence (principle #3). On most platforms the meter-stopping action is
+> **irreversible** (deletes the disk) — confirmation matters more, not less.
+
+## Parallel ablation fan-out
+
+For N ablation cells: one job per cell, an **isolated write path per job** (no shared mutable output), launched
+across instances/queues. **REQUIRED:** `superpowers:dispatching-parallel-agents` supplies the independence
+predicate (don't fan out onto shared state) and the mandatory post-fan-out reconciliation. FS-shared deployment
+pattern → `references/parallel_ablation.md`.
+
+## Quick reference — the four facts that bite per platform
+
+Full detail in each profile; this table is the at-a-glance.
+
+| Platform | Survives **stop** | Survives **destroy** | Spot grace | China mirror needed |
+|---|---|---|---|---|
+| AutoDL | /root + data + FS | FS only | n/a | yes (`/etc/network_turbo`, hf-mirror) |
+| RunPod | volume disk (bills 2×) | Network Volume only | ~5 s SIGTERM→KILL | no (`hf_transfer`) |
+| vast.ai | disk (bills forever) | nothing | ~0 s (abrupt) | no |
+| Lambda | n/a (no stop) | nothing | n/a (on-demand) | no |
+| China (恒源云/矩池云/…) | varies; data disk bills | per-platform persistent vol | n/a | yes |
+| generic-SSH/Slurm/K8s | you own it | you own it | Slurm SIGTERM→KillWait (def 30 s) | only if in China |
+
+## Common gotchas (top 8 inline — full catalog in references/)
+
+The universal ones that cost the most GPU-hours. Symptom → fix; root cause + the rest in
+**`references/gotchas_universal.md`** (run `grep -i '<keyword>' references/gotchas_universal.md` to jump).
+
+1. **SSH drops on `pkill -9`** (exit 255 + "Connection reset") — normal; re-ssh to verify, don't panic.
+2. **tmux holds the script in memory** — editing it mid-run re-executes blocks; version the filename.
+3. **Disk-full crashes `torch.save`** (`iostream error`) — pre-budget; auto-prune `latest.pth`, keep `best`.
+4. **cgroup OOM with no traceback** (bare `Killed` / exit 137) — `num_workers × big-tensor`; size workers vs `memory.max`, not CPU count.
+5. **Silent sync failure** — `cp … 2>/dev/null; echo synced` lies on a full/inode-exhausted FS; gate the success line on the actual copy result.
+6. **Spot preemption grace is tiny (~5 s → ~0 s on the platforms profiled here; AWS-style 2-min grace only on clouds not profiled)** — a SIGTERM-flush handler is NOT a safety net; checkpoint on a timer to durable storage, load-latest unconditionally (`references/spot-resilience.md`).
+7. **"Stop" rarely stops the meter** — only `terminate`/`destroy` does, and it's irreversible (deletes the disk). Know the verb from the profile before you click, and on RunPod a stopped Pod can even restart with zero GPUs.
+8. **CRLF breaks `.sh` on Linux** — author on Windows → `.gitattributes` `*.sh text eol=lf`; on-box unblock `sed -i 's/\r$//'`.
+
+## When training itself breaks (the model, not the platform)
+
+Platform ops is only half the job — once the box is running, training breaks in its own ways. The
+`references/training/` layer is the debug knowledge for the run itself. Boundary: **this layer owns
+"make it run, fast, and not crash"; `verifying-dl-experiments` owns "is the *number* real"** —
+cross-link it for collapse / leakage / metric-validity. Every entry is symptom → root cause → fix with
+cited current docs.
+
+- `references/training/oom-memory.md` — CUDA/VRAM + host-RAM OOM and the fit-it ladder (grad-accum → bf16 → activation-checkpointing → `expandable_segments` → FSDP/ZeRO → CPU/NVMe offload → LoRA/QLoRA); OOM-at-a-specific-step (first backward / val / longest batch); the memory snapshot + visualizer.
+- `references/training/distributed-launch.md` — `torchrun`/`accelerate`/`deepspeed` launch + env contract, DDP/FSDP/ZeRO config, and the multi-GPU **HANGS** toolkit (one-rank-diverged, rank-conditional collective, dataloader-length mismatch). Multi-node wire → `references/multinode.md`.
+- `references/training/precision-stability.md` — fp16/bf16/tf32 + AMP/GradScaler, NaN/Inf hunting (`detect_anomaly`), LLM **loss spikes** + divergence (warmup, clip, init, z-loss).
+- `references/training/throughput-profiling.md` — GPU-bound vs data-bound vs comms-bound; dataloader knobs; `torch.compile` traps; flash-attention; `torch.profiler` / Nsight.
+- `references/training/checkpoint-resume.md` — full-state save/resume mechanics, sharded (FSDP/DeepSpeed) checkpoints, and the resume bugs (epoch restart, data reshuffle, scaler/EMA dropped). Spot cadence → `references/spot-resilience.md`.
+- `references/training/by-domain.md` — per-domain gotchas: LLM/transformer, vision (det/seg), diffusion, RL, multimodal/VLM.
+- `references/training/convergence-debugging.md` — the **"runs but won't learn / learns badly"** layer: the overfit-one-batch smoke, params-not-updating, optimizer/LR/weight-decay/schedule config, loss-function footguns (double-softmax, BCEWithLogits, CE-target form), fine-tuning/freezing (frozen-BN drift, discriminative LR, LoRA wiring), and the training-dynamics dashboard (update:weight ratio, dead-ReLU, GradScaler-scale).
+- `references/training/data-pipeline.md` — dataloader/dataset **correctness** (not speed): the worker-RNG augmentation-duplication bug, IterableDataset worker/rank sharding, collate/`__len__`/`pin_memory`/`spawn` contracts, and preprocessing/label/shuffle traps (RGB-vs-BGR, ToTensor ÷255, `set_epoch`).
+
+## Companion skills (separate installs; REQUIRED reading where present)
+
+These are **separate** Agent Skills, not bundled here — install them for the full experience. On an
+agent where a companion isn't installed, treat its pointer below as an optional cross-reference; this
+skill still works standalone.
+
+- **`verifying-dl-experiments`** — owns *is-the-number-real*: smoke content, retry-vs-safeguard, keepable-checkpoint, eval sizing, tracker forensics, GPU-0%-util diagnosis. This skill owns *where/when/how-much-$*.
+- **`huggingface-skills:hf-cli`** — the transport verbs (`hf download --resume`, `hf upload-large-folder`, `hf cache verify`); this skill owns the China-mirror swap + stall-retry (`references/china-network.md`).
+- **`huggingface-skills:huggingface-trackio`** — hosted tracker so metrics survive teardown (gotcha U20); poll `trackio` alerts as a structured monitor instead of brittle ssh-tail.
+- **`superpowers:verification-before-completion`** — the Iron Law's general form; gates every "training done / synced / teardown complete" claim.
+- **`superpowers:dispatching-parallel-agents`** — independence predicate + reconciliation for ablation fan-out.
+
+## Getting better over time (capture new gotchas + personalize)
+
+This skill is static, but every run can teach it something — without corrupting it.
+Protocol → **`references/self-improvement.md`**. In short: when a run surfaces a gotcha the catalog
+lacks, **only sediment a root-caused, reproduced, generalizable one** (a one-off flake is a hypothesis,
+not a gotcha — principle #3); **route it** — user/project-specific → the host's memory system,
+generalizable → propose adding to `references/gotchas_universal.md` / the profile §7 /
+`references/training/` (and offer an upstream PR); **never silently rewrite a skill file — draft the
+`symptom → root cause → fix` and let the user approve.** On first use, capture the user's platforms +
+paths + tracker entity into memory so later runs are pre-parameterized. Platform facts carry a `verified
+<month>` stamp — re-verify any teardown/billing fact against current docs before betting money or data.
+
+## Limitations
+
+- Does not replace a real cloud orchestrator or managed provisioner; use it to make rented-box work survivable, not to optimize multi-cloud procurement.
+- Platform billing, stop, destroy, and data-retention behavior can drift; re-check current provider docs before destructive or money-impacting actions.
+- Requires user-owned credentials, SSH/API access, and explicit confirmation before teardown, deletion, or other irreversible cleanup.
+- Companion skills named above are not bundled here; treat them as optional references unless installed in the current agent environment.
+
+## Bundled resources
+
+Load only what the current phase needs.
+
+- `references/principles.md` — the 10 invariants expanded, with the cross-platform nuance behind each.
+- `references/lifecycle_checklist.md` — the 6-phase runbook as a per-platform checklist.
+- `references/gotchas_universal.md` — universal + mixed gotchas (TOC + grep index at top).
+- `references/monitoring_patterns.md` — the four-layer durable-monitoring architecture + robust ssh-poll template.
+- `references/ssh_transport.md` — ssh config, rsync/scp resumable patterns, secrets-via-stdin, CRLF, two-SSH-flavor caveat.
+- `references/china-network.md` — mirrors table + HF_ENDPOINT + resumable-download ladder + the `no_proxy` trap (all CN platforms).
+- `references/spot-resilience.md` — preemption signals, Young/Daly checkpoint cadence, atomic-write resume.
+- `references/parallel_ablation.md` — FS-shared fan-out + the independence predicate + reconciliation.
+- `references/multinode.md` — (advanced) NCCL / fabric-manager / elastic-training gotchas; single-box users skip.
+- `references/training/` — the **DL-training debug layer** (8 files: oom-memory, distributed-launch, precision-stability, throughput-profiling, checkpoint-resume, by-domain, convergence-debugging, data-pipeline) — see "When training breaks" above.
+- `references/self-improvement.md` — the feedback loop: capture a new gotcha (at a bar) into memory or the catalog, personalize on first run, keep platform facts fresh.
+- `scripts/` — wrapper templates (`run_one`/`run_queue`), monitors (`mem_monitor`, `gpu_health`, `reap_vram_zombies`), the read-only patrol (`health_patrol.sh.template`), transfer/aggregation (`download_loop`, `aggregate_to_fs`, `setup-china-mirrors`), the load-and-verify checker (`verify_local.py`), and the `verified`-stamp freshness linter (`check_staleness.py`).
+- `profiles/<platform>.md` — the per-platform substrate (one per platform; `_schema.md` defines the 8 fields).
+- `examples/autodl_sweep/` — one complete, runnable worked case end to end.

package/bundled-skills/remote-gpu-trainer/evals/README.md

@@ -0,0 +1,57 @@
+# Evals — does the skill actually route to the right answer?
+
+A skill is only as good as an agent's ability to *find and apply* the right entry under a real
+problem. These evals test that, in two tiers, against a fixed set of realistic scenarios
+([`cases.jsonl`](cases.jsonl)) spanning both halves of the skill (remote-GPU operations on every
+platform family + the DL-training-debug layer, including the `convergence-debugging` and
+`data-pipeline` files).
+
+## Tier 1 — structural reachability (runnable, no API key)
+
+```bash
+python evals/run_evals.py        # exits non-zero if any case regresses
+```
+
+For each scenario it asserts the answer is **present, at the documented location, with the
+expected entry IDs / keywords intact**: every `expect_files` exists, every `expect_ids` is still a
+`### <ID>` header there, every `expect_grep` term is still in the text. This is a **drift guard** —
+it catches a renamed/removed entry, a moved section, a deleted file, or a fact rewritten away from
+its key term. Run it in CI; it needs nothing but Python 3.
+
+What it does **not** prove: that an agent actually *navigates* there (Tier 2), or that the platform
+*facts* are correct on a live box (see Verification status).
+
+## Tier 2 — agentic navigation (the gold standard)
+
+The real test: give a **fresh agent** the skill and one scenario's `prompt`, let it navigate **from
+SKILL.md only** (following the documented routing, not blind grep), and check it reaches a correct,
+specific answer covering the case's `must_cover` points within ~2 hops. Each case records its last
+such run in the `agentic` field; the collected runs are in [`RESULTS.md`](RESULTS.md).
+
+To re-run Tier 2 with any agent/harness: load the skill, paste a case `prompt`, and grade the
+answer against `expect_files` / `expect_ids` / `must_cover`. (Anthropic's skill best-practices
+recommend ≥3 evals across Haiku/Sonnet/Opus — re-running these cases per model is the way to meet
+that bar; results to date were gathered on the development model and are labelled as such.)
+
+## Adding a case
+
+Append one JSON object per line to `cases.jsonl`:
+
+```json
+{"id": "kebab-id", "prompt": "the user's situation, verbatim-ish",
+ "expect_files": ["references/training/<file>.md"], "expect_ids": ["O7"],
+ "expect_grep": ["lr finder"], "must_cover": "the key points a correct answer must hit",
+ "agentic": "PASS/FAIL (date): the navigation path observed"}
+```
+
+Use `expect_ids` for the training catalogs (they have `### O7 / DP1 / M17 …` headers) and
+`expect_grep` for platform profiles (which are section-structured). Then `python evals/run_evals.py`.
+
+## Verification status (important)
+
+These evals test **retrieval and routing inside the skill** — not the truth of the platform facts
+on a live instance. Only the AutoDL profile is battle-tested by the author; the other six platform
+profiles are researched from official docs + community reports and **not yet live-validated** (see
+the repo README's "Verification status" and `references/self-improvement.md` §5). A case passing
+here means "the skill leads an agent to *this documented answer*," not "this answer was confirmed on
+a rented box."