gpu-compute-provider 0.4.2__py3-none-any.whl

supercomputer/SKILL.md

@@ -0,0 +1,1565 @@
+---
+name: supercomputer
+description: >-
+  Use supercomputer to provision GPU or CPU hosts, run remote commands and scripts,
+  manage persistent sessions, transfer files, and monitor long-running jobs from
+  agent workflows.
+---
+
+# supercomputer — GPU compute from the command line
+
+supercomputer provisions GPU and CPU hosts, runs code on them, and transfers files — all
+from the terminal. It works with any AI coding agent (Cursor, Claude Code, Codex,
+etc.) so your agent can provision hardware, run experiments, and pull results
+autonomously.
+
+Package: `supercomputer` | CLI command: `super` | Python 3.9+
+
+---
+
+## Setup
+
+```bash
+pip install gpu-compute-provider
+super set_token <api_key>  # one-time; saves to ~/.supercomputer/token
+super health               # verify connectivity
+```
+
+(A human sets up with `super login`, which signs in through the browser and
+mints a key for the machine; agents use `set_token` with a key they were given.)
+
+Token resolution order:
+
+1. `api_key=` parameter to `Client()`
+2. `SUPERCOMPUTER_API_KEY` environment variable
+3. Saved token in `~/.supercomputer/token`
+4. `SUPERCOMPUTER_API_KEY` from the nearest `.env` file
+
+(For a human at a terminal, plain `super set_token` with no argument prompts
+with hidden input, keeping the key out of shell history.)
+
+---
+
+## Quick Start
+
+### Run a script on a GPU (throwaway session)
+
+```bash
+# Upload a script, run it on an L4, print output, tear down
+super run train.py --sku gpu_1x_l4
+
+# Upload a directory, run a specific script, pass arguments
+super run . --script train.py --sku gpu_1x_l4 -- --epochs 50 --lr 1e-4
+
+# Run a one-off command (no file upload)
+super exec --sku gpu_1x_l4 nvidia-smi -L
+```
+
+### Choosing a SKU
+
+Request hardware by passing `--sku` to `exec`/`run`/`create`. SKUs follow the
+pattern `gpu_<count>x_<model>` or `cpu_<count>x_<family>`: `gpu_1x_l4`,
+`gpu_8x_h100`, `cpu_2x_epyc`, etc. Run `super skus` (or `super skus --json`)
+for the live list with availability and the rate each SKU was last billed at.
+If a SKU has no available capacity, the session queues until a host is
+provisioned.
+
+---
+
+## CLI Reference
+
+### Global Flags
+
+
+| Flag     | Description                                                           |
+| -------- | --------------------------------------------------------------------- |
+| `--json` | Machine-readable JSON output to stdout. Status messages go to stderr. |
+
+
+Use `--json` in agent workflows so output can be parsed programmatically. Place
+`--json` before your remote command (e.g. `super exec --json -- python app.py`):
+flags after the command — or after a `--` — belong to *your* program, so
+`super exec -- python app.py --json` passes `--json` to `app.py`, not to super.
+Unknown flags are rejected (e.g. a typo'd `--limitt`) rather than silently
+ignored.
+
+### Infrastructure
+
+
+| Command                | Description                               |
+| ---------------------- | ----------------------------------------- |
+| `super health`          | Check API connectivity                    |
+| `super skus`            | List SKUs: availability, `MAX` concurrency, last-billed `$/HR` (`cost_per_hr` in `--json`; spot, so it moves; `-` = not billed in 7 days) |
+| `super set_token <key>` | Save API key                              |
+| `super credits`         | Prepaid balance, auto-refill, card         |
+| `super credits buy <usd>` | Buy credit (opens hosted checkout)      |
+
+
+### Execution
+
+
+| Command                                                                       | Description             |
+| ----------------------------------------------------------------------------- | ----------------------- |
+| `super run [SID] [--upload PATH]... [--detach] [--retries N] [--wait DUR] -- CMD` | Run a command; upload files first if needed |
+| `super run FILE\|DIR [--script NAME] [-- args...]` | Existing upload-and-run form |
+| `super exec [SID] [flags] -- CMD` | Run command spelling with a 30-second default wait |
+| `super spawn [--sku SKU \| --on SID] [flags] -- CMD` | Start a command detached |
+| `super upload [--on SID] <local> [remote]` / `super download [--on SID] <remote> [local]` | Transfer files (session via `--on` or positional) |
+
+
+Without a session ID, `run` creates a session and gives its box back when the
+command ends. Its `/workspace` remains saved and downloadable. Use `--on SID`
+or a positional SID to run in an existing session; it stays available for later
+commands. `exec` is the same command as `run`, except its wait defaults to 30s
+instead of forever. `--include` is an alias of `--upload`; old command forms
+still work.
+
+Cached images complete in ~2s. Cold builds (first run on a host) take 30–120s.
+
+### Exec Semantics
+
+`super exec` is optimized for agents and ML workloads. It starts commands as
+managed processes and waits for a foreground result for a configurable budget.
+
+**Working directory.** Commands run from `/workspace` by default — the same
+place `super upload` puts your files. So an uploaded `train.py` runs as
+`python train.py` (relative path, no `cd` and no need to spell out
+`/workspace/train.py`), and commands that write to relative paths also land in
+`/workspace` and persist. Do **not** assume a different default (e.g. `/root`)
+or hardcode `/root/...` paths — that's the single most common cwd mistake. To
+run elsewhere for one command, `cd` inside it: `sh -c 'cd /some/dir && ...'`.
+This applies uniformly to `exec`, `run`, `spawn`, and `shell`.
+
+```bash
+# Default wait budget is 30s
+super exec --on s-abc123 python3 /workspace/train.py
+
+# Wait longer before detaching
+super exec --on s-abc123 --wait 10m python3 /workspace/train.py
+
+# Positional session alias for --on
+super exec s-abc123 --wait 10m -- python3 /workspace/train.py
+
+# Alias for --wait, useful when an agent thinks in timeouts
+super exec --on s-abc123 --timeout 10m python3 /workspace/train.py
+```
+
+If the command exits within `--wait`, `exec` behaves like a normal foreground
+command: stdout/stderr are printed and the CLI exits with the remote return code.
+
+If the command is still running when the wait budget expires, `exec` does **not**
+kill it. It detaches, prints a process ID on stderr and exits **75** (exit 0 only
+for `--wait 0`, which detaches immediately):
+
+```bash
+Command is still running; use super logs s-abc123 proc-456 --follow
+```
+
+In `--json` mode, a detached command returns `status: "running"`,
+`returncode: null`, plus `session_id` and `process_id`. Agents should treat that
+(and its exit code 75) as success-with-handle, not as command failure, then block on it with
+`super wait <session_id> <process_id>` (returns the real exit code + tail) or
+tail it with `super logs --follow`.
+
+```json
+{
+  "stdout": "",
+  "stderr": "",
+  "returncode": null,
+  "status": "running",
+  "session_id": "s-abc123",
+  "process_id": "proc-456",
+  "message": "Command is still running; use super logs s-abc123 proc-456 --follow"
+}
+```
+
+`exec` returns a bounded stdout payload by default so a noisy command cannot flood
+the caller. If the JSON response has `stdout_truncated: true`, do **not** rerun the
+job just to recover logs. Use the returned `session_id` and `process_id`:
+
+```bash
+# Fetch the durable combined process log captured by the host
+super logs s-abc123 proc-456 --output all
+
+# Fetch a specific range or stream
+super logs s-abc123 proc-456 --output head --output-bytes 128k
+super logs s-abc123 proc-456 --stream stderr --output all
+```
+
+> The default stream is **combined** (stdout + stderr) — prefer it. Most ML
+> tooling (tqdm progress bars, HuggingFace/TRL logs) writes to **stderr**, so
+> `--stream stdout` on a training job often looks empty even though the job is
+> producing output. Reach for `--stream stdout`/`stderr` only when you
+> specifically need one split.
+
+**Liveness while polling a long job:** `super logs <sid> <pid> --json` includes
+`last_write_ts` (when the durable log was last written) and `session_status`.
+A `status: "running"` process whose `last_write_ts` stops advancing across
+polls is stalled or on a lost host — no need to diff `output_total_bytes`
+between polls. A `status: "lost_host"` (with `process_alive: false`) means the
+session ended without the process's exit ever being recorded: the host was
+lost, the tail is final, don't keep polling.
+
+SDK equivalent:
+
+```python
+log = session.logs("proc-456", output_range="all")
+print(log.output)
+
+stderr = session.logs("proc-456", stream="stderr", output_range="all")
+```
+
+Persistent-session commands (`--on <sid>`) print a one-line `✓ ready` to
+stderr on every attach; fuller provision/setup narration appears only when
+the session actually has setup work to do. Terminal/problem statuses also
+print to stderr. Parse stdout (or use `--json`) — never stderr.
+
+#### Shell Parsing Rules
+
+Prefer passing commands as normal argv:
+
+```bash
+super exec --on s-abc123 ls /workspace
+super exec s-abc123 -- ls /workspace
+super exec --on s-abc123 python3 /workspace/train.py --epochs 10
+```
+
+A single quoted string is treated as shell text (it runs via `bash -lc`),
+so both of these work and are equivalent:
+
+```bash
+super exec --on s-abc123 'ls /workspace'
+super exec --on s-abc123 'echo one two && echo piped | tr a-z A-Z'
+```
+
+When you need shell syntax (`;`, `&&`, pipes, redirects, variables), either pass a
+single quoted shell expression or be explicit with `sh -c`:
+
+```bash
+super exec --on s-abc123 'cd /workspace; ls'
+super exec --on s-abc123 -- sh -c 'sleep 45; echo done'
+```
+
+Remember that unquoted shell metacharacters are handled by the local shell before
+`super` sees them:
+
+```bash
+# Runs "sleep 45" remotely, then "echo done" locally
+super exec --on s-abc123 sleep 45; echo done
+```
+
+### Jobs (`super job`) — run to completion
+
+`super job run` runs like `super run`, except for what `--retries` counts. It keeps `/workspace` on the
+session, so you can inspect or download files after the command ends. A new
+session gives its box back when the command ends. `--retries N` re-runs only
+after a lost box; a command's non-zero exit is final. (`super run --retries N`
+instead re-runs a command that exits non-zero, after a short growing pause — except
+exit 126/127 and an OOM kill, which a re-run on the same box cannot change.) With mounted volumes,
+the attached command waits for their commit before returning. Use `--detach`
+to get a handle immediately, then follow its logs or wait for completion
+(`--detach` cannot be combined with `--retries`). For work that may be
+re-run, save checkpoints in `/workspace` and make the command load them on start:
+a retry runs in the same session and `/workspace`. Only when the session itself
+ended (failed or dead) does the retry start a new session with an empty
+`/workspace` (uploads are sent again; stderr says so) — checkpoints that must
+survive that belong on a volume. A job closed on request (`super job kill`,
+`super stop`) is never retried.
+
+| Command | Description |
+|---|---|
+| `super job run [flags] -- CMD` | Run on a session, keep `/workspace`, retry up to `--retries N` |
+| `super job ls [--all] [--limit N]` | List jobs (`--all` includes finished ones) |
+| `super job logs <job-id> [<pid>] [--follow]` | No pid: list the job's processes. With a pid: its durable output (`--follow` for live) |
+| `super job wait <job-id> [--timeout DUR]` | Block until the job ends; exits with its command's exit code (`69` host lost, `75` still running at `--timeout`) |
+| `super job kill <job-id>` | Stop the job's session: billing stops at once; `/workspace` is saved by the teardown that follows (`super stop <job-id> --wait` confirms) |
+
+`job run` accepts the `run` flags: `--sku SKU`, `--volume NAME[:MOUNT]`,
+`--upload PATH` (also `--include`), `--env K=V`,
+`--secret KEY[=VALUE]`, `--retries N`, `--min-disk-gb N`, and `--detach`.
+After preemption, follow the same job ID with `super job wait` or
+`super job logs`; do not submit a duplicate run. For example:
+
+```bash
+super job run --sku gpu_1x_l4 --upload train.py --retries 2 -- \
+  python train.py --out /workspace/model.pt
+super download <session-id> /workspace/model.pt ./model.pt
+```
+
+The SDK's `project.job` remains an ephemeral job: its outputs live in volumes
+and logs. The CLI `super job run` uses a saved workspace, like `super run`.
+
+SDK example:
+
+```python
+job = project.job("python train.py --out /results/model.pt",
+                  sku="gpu_1x_l4",
+                  volume_mounts=[{"name": "results", "mount_path": "/results"}],
+                  include=["./src"], retries=1)
+log = job.wait()                   # ProcessLog: exit_code, output
+
+client.list_volume_files("results")           # [{path, size, ...}]
+data = client.read_volume_file("results", "model.pt")   # bytes, no session
+client.download_volume_file("results", "model.pt", "./model.pt")
+client.delete_volume("results")               # when you're done with it
+```
+
+Use `job run` for a command with a retry budget; use a persistent session when
+you want to run several commands against the same `/workspace`.
+
+### Persistent Sessions
+
+
+| Command                       | Description                                          |
+| ----------------------------- | ---------------------------------------------------- |
+| `super create [--sku SKU]`     | Create a session (cold — $0, no host, until first used) |
+| `super sessions [--all] [--limit N] [--json]` | List sessions and spending; JSON includes `restart_check` after a restart |
+| `super usage [--since 24h]`   | What the account has been charged over a window (default 30 days), by resource and session |
+| `super stop <session-id> [--wait]` | Stop a session now: billing stops when the stop is accepted; files are saved by the final sync during teardown. `--wait` blocks until teardown settles and reports a failed final save when the server records one for this stop: it exits `1` naming `close_flush_incomplete` (the final sync did not finish) or `volume_commit_failed`. A state loss recorded before this stop is mentioned, not failed. It also exits `1` if the session ends `failed`/`dead` during the stop, and when `--timeout` expires |
+
+
+`create` registers a session and returns a session ID without allocating a host —
+it is cold and costs nothing until you use it. The first `exec`/`run`/`upload`/
+`shell`/`ssh` brings it online automatically with its files intact. Use
+`--on <session-id>` with `exec`, `run`, `upload`, or `download` to interact with it
+repeatedly.
+
+Sessions resume transparently: there is **no `resume` command**. A session that was
+stopped or reaped while idle comes back online (on a fresh host, with its workspace
+restored from cloud storage) the moment you use it again. `super stop` exists only to halt spend
+immediately — your files are preserved either way. `job run` also keeps its
+`/workspace`; see Jobs.
+
+`super stop` returns once the stop is accepted (billing has stopped), which is
+before teardown has saved the files — and teardown can be deferred and retried
+(`stop accepted … its host did not answer the teardown`). Anything that reads
+the saved manifest (`super ls`, `super mount`, `super map --from <sid>`) must
+wait for it: pass **`super stop <sid> --wait`** (`--timeout SECONDS`, default
+300, `0` waits forever) instead of polling `super sessions` for `state=cold`
+yourself. It exits 1 when the wait expires (the stop is unaffected), when the
+server records a failed final save for this stop (`close_flush_incomplete`,
+`volume_commit_failed`), or when the session ends `failed`/`dead` during the stop.
+
+**Keep a client attached until the host is up.** First use claims a host for the
+session; if no client stays attached while it comes online, the host is reclaimed
+~60s after it's claimed (the session goes back to cold and resumes on next use —
+no data loss, but you paid for a boot that did nothing). To run **N boxes at
+once**, keep one client attached per box for its whole run — e.g. N parallel
+`super exec --on <sid> --wait <long>` (or `spawn` + `super wait`) — rather than
+creating N sessions and attending to them one at a time, which lets the idle
+ones lose their hosts.
+
+### Surviving preemption
+
+Sessions run on spot capacity. **Restart is on by default**: on a spot notice,
+supercomputer flushes `/workspace`, gets a new box, restores the files and
+re-runs the command. After an unannounced host death it restores the last
+completed workspace sync and re-runs the command. Your program must save and
+load its own checkpoints to recover progress; unsynced files and in-memory
+state can be lost. See *Make your program resumable* below.
+
+For restart, the new box first goes where the session's files are stored.
+`--home-wait DUR` (default `10m`) sets how long to wait for a box there. If none
+is available, supercomputer moves the files elsewhere only when the estimated
+transfer cost fits `--egress-budget USD` (default `$5`). Both settings require
+restart and apply only when the session is created.
+
+`--no-restart` (SDK `restart=False`, project `"restart": false`,
+`SUPERCOMPUTER_RESTART=off`, or `super config set restart off`) turns all
+automatic recovery off. After a spot notice the session returns cold on any
+host with `/workspace` kept, processes lost and nothing re-run (SDK:
+`client.describe_session(sid).state_loss_reason` is
+`spot_preempt_checkpoint_disabled`). After host death it ends (`dead`). You
+must start new work yourself.
+
+`--checkpoint` is an **EXPERIMENTAL opt-in**: it first tries live migration of
+the running process, including GPU state, and falls back to restart if that
+fails. It is not available on multi-GPU SKUs and requires restart to be on.
+Live migration restores where the sandbox checkpoint is stored and can wait
+there for capacity (`provisioning.reason` `awaiting_restore_capacity`).
+`--no-checkpoint` disables live migration but leaves restart enabled.
+
+```bash
+super run --sku gpu_1x_h100 --upload train.py --upload checkpoint_progress.py --detach -- python -u train.py
+super run --sku gpu_1x_h100 --checkpoint --upload train.py --upload checkpoint_progress.py --detach -- python -u train.py
+super create --sku gpu_1x_l4 --no-restart
+super config set restart off
+super config get restart
+super config unset restart
+super run --home-wait 20m --egress-budget 8 --upload train.py --detach -- python train.py
+```
+
+`SUPERCOMPUTER_MIGRATE_RESTART=never` stops automatic command re-runs during
+restart. It is redundant with `--no-restart` for that session, but remains
+useful for restart-enabled sessions and existing callers.
+
+Precedence, strongest first:
+
+- `--checkpoint`/`--no-checkpoint` (SDK `checkpoint=True/False`) →
+  `SUPERCOMPUTER_CHECKPOINT=on|off` → supercomputer.json `"checkpoint"` →
+  `super config set checkpoint on|off` → server default (off).
+- `--restart`/`--no-restart` (SDK `restart=True/False`) →
+  `SUPERCOMPUTER_RESTART=on|off` → supercomputer.json `"restart"` →
+  `super config set restart on|off` → server default (on).
+
+Both settings are fixed when the session is created; `--on SID` uses that
+policy. There is no `auto` or `app` spelling for these settings.
+`super sessions --json` reports `checkpoint` (live migration enabled) and
+`restart` (automatic recovery enabled).
+
+Run `super sessions --json` and read each session's `restart_check`, for
+example `{"status":"pass","saved_step":100,"resumed_step":101,
+"at":"2026-09-25T12:00:00Z"}`. Status is `pass`, `fail`, or `no_signal`;
+the step fields may be `null`, and `at` is RFC3339. `pass` means the first `start` after
+the restart is the last pre-restart `save` step plus one (or step `0` or `1` when
+there was no save). `fail` means it started elsewhere. `no_signal` means the
+progress file did not survive the restart, or there was no new `start` while
+the re-run process was running.
+
+#### Make your program resumable
+
+Save under `/workspace`, not a volume. While a session runs, periodic syncs
+start no sooner than 2 minutes after the preceding sync completes; the time
+until the next sync can be longer. Volumes commit only when the session ends.
+On a preemption notice super stops the program and flushes `/workspace` once
+more before the new box starts. When a box dies with no notice, the resume
+starts from the last completed periodic sync, and
+files the program was still rewriting at that sync (the progress file, the
+newest checkpoint) can be missing from it, so the resume can start from an
+older save or from scratch.
+Run Python unbuffered (`python -u` or `PYTHONUNBUFFERED=1`) so step lines show
+up in `super logs` as they happen. Write a checkpoint
+to a temporary path, flush and `fsync` it, rename it atomically, then update
+`LATEST` and keep the newest two. Load `LATEST` when the program starts. Include
+model weights, optimizer and scheduler state, AMP gradient scaler, Python/NumPy/
+PyTorch RNG state, dataloader position, and global step. Restoring weights alone
+does not resume training from the same step.
+
+Append JSON lines to `/workspace/.supercomputer/progress.jsonl`. Write
+`{"event":"start","step":N}` once when the program starts, where `N` is the
+first step it will run. Write `{"event":"save","step":N}` only after the
+checkpoint for step `N` is durably in place, after the atomic rename. Other
+lines, such as `{"step":N,"loss":x}`, may share the file; the restart check
+reads only lines that carry `event`. Use this small helper in each recipe below:
+
+```python
+# /workspace/checkpoint_progress.py
+import json
+import os
+import shutil
+import tempfile
+from pathlib import Path
+
+ROOT = Path(os.environ.get("WORKSPACE", "/workspace")) / ".supercomputer"  # super run sets WORKSPACE; set it to a local folder to test
+CHECKPOINTS = ROOT / "checkpoints"
+LATEST = ROOT / "LATEST"
+PROGRESS = ROOT / "progress.jsonl"
+
+def progress(event, step):
+    ROOT.mkdir(parents=True, exist_ok=True)
+    with PROGRESS.open("a") as file:
+        file.write(json.dumps({"event": event, "step": int(step)}) + "\n")
+        file.flush()
+        os.fsync(file.fileno())
+
+def latest():
+    return Path(LATEST.read_text().strip()) if LATEST.exists() else None
+
+def _sync_dir(path):
+    fd = os.open(path, os.O_RDONLY)
+    try:
+        os.fsync(fd)
+    finally:
+        os.close(fd)
+
+def save(step, write_checkpoint):
+    CHECKPOINTS.mkdir(parents=True, exist_ok=True)
+    temp = Path(tempfile.mkdtemp(prefix=".pending-", dir=CHECKPOINTS))
+    write_checkpoint(temp)
+    for path in temp.rglob("*"):
+        if path.is_file():
+            with path.open("rb") as file:
+                os.fsync(file.fileno())
+    for path in sorted((p for p in temp.rglob("*") if p.is_dir()),
+                       key=lambda p: len(p.parts), reverse=True):
+        _sync_dir(path)
+    _sync_dir(temp)
+    final = CHECKPOINTS / f"step-{step}-{temp.name[len('.pending-'):]}"
+    os.replace(temp, final)
+    _sync_dir(CHECKPOINTS)
+    with tempfile.NamedTemporaryFile(mode="w", dir=ROOT, delete=False) as file:
+        file.write(str(final) + "\n")
+        file.flush()
+        os.fsync(file.fileno())
+        latest_temp = file.name
+    os.replace(latest_temp, LATEST)
+    _sync_dir(ROOT)
+    saved = sorted(CHECKPOINTS.glob("step-*"), key=lambda p: p.stat().st_mtime_ns)
+    for old in saved[:-2]:
+        shutil.rmtree(old)
+    progress("save", step)
+```
+
+**Plain PyTorch.** Put this around your existing `model`, `optimizer`,
+`scheduler`, `scaler`, and `loader`. Use a deterministic sampler; the stored
+epoch and batch index let you skip batches already consumed. Set `total_epochs`
+and `save_every` for your run.
+
+```python
+import random
+import numpy as np
+import torch
+from checkpoint_progress import latest, progress, save
+
+checkpoint = latest()
+epoch0 = batch0 = step = 0
+if checkpoint:
+    state = torch.load(checkpoint / "state.pt", map_location="cpu", weights_only=False)
+    model.load_state_dict(state["model"])
+    optimizer.load_state_dict(state["optimizer"])
+    scheduler.load_state_dict(state["scheduler"])
+    scaler.load_state_dict(state["scaler"])
+    random.setstate(state["python_rng"])
+    np.random.set_state(state["numpy_rng"])
+    torch.set_rng_state(state["torch_rng"])
+    if torch.cuda.is_available():
+        torch.cuda.set_rng_state_all(state["cuda_rng"])
+    epoch0, batch0, step = state["epoch"], state["next_batch"], state["step"]
+progress("start", step + 1)
+for epoch in range(epoch0, total_epochs):
+    if hasattr(loader.sampler, "set_epoch"):
+        loader.sampler.set_epoch(epoch)
+    for batch_index, batch in enumerate(loader):
+        if epoch == epoch0 and batch_index < batch0:
+            continue
+        # Replace this line with your forward, backward, optimizer, scaler and scheduler steps.
+        train_batch(batch, model, optimizer, scheduler, scaler)
+        step += 1
+        if step % save_every == 0:
+            next_epoch, next_batch = (epoch + 1, 0) if batch_index + 1 == len(loader) else (epoch, batch_index + 1)
+            state = dict(model=model.state_dict(), optimizer=optimizer.state_dict(),
+                         scheduler=scheduler.state_dict(), scaler=scaler.state_dict(),
+                         python_rng=random.getstate(), numpy_rng=np.random.get_state(),
+                         torch_rng=torch.get_rng_state(),
+                         cuda_rng=torch.cuda.get_rng_state_all() if torch.cuda.is_available() else [],
+                         epoch=next_epoch, next_batch=next_batch, step=step)
+            save(step, lambda path: torch.save(state, path / "state.pt"))
+    batch0 = 0
+```
+
+**Hugging Face Trainer.** Needs `transformers` and `accelerate`. Add this
+callback to your existing `model` and `train_dataset` setup. `save_steps` sets your save schedule. Trainer writes its
+full checkpoint first; the helper copies it to a temporary directory and
+publishes it after the atomic rename. `resume_from_checkpoint=True` works when
+Trainer's output directory has the latest checkpoint; passing `latest()` uses
+the published copy directly.
+
+```python
+import shutil
+from transformers import Trainer, TrainerCallback, TrainingArguments
+from checkpoint_progress import latest, progress, save
+
+class ProgressCallback(TrainerCallback):
+    def on_train_begin(self, args, state, control, **kwargs):
+        progress("start", state.global_step + 1)
+
+    def on_save(self, args, state, control, **kwargs):
+        source = f"{args.output_dir}/checkpoint-{state.global_step}"
+        save(state.global_step, lambda path: shutil.copytree(source, path, dirs_exist_ok=True))
+
+args = TrainingArguments(output_dir="/workspace/.supercomputer/trainer",
+                         save_strategy="steps", save_steps=save_steps)
+trainer = Trainer(model=model, args=args, train_dataset=train_dataset,
+                  callbacks=[ProgressCallback()])
+trainer.train(resume_from_checkpoint=str(latest()) if latest() else None)
+```
+
+**PyTorch Lightning.** Add these callbacks to your existing `model` and
+`train_loader` setup. Lightning's `ckpt_path="last"` loads the last checkpoint;
+the checkpoint callback copies each completed `last.ckpt` into the atomic store
+and then writes the progress line.
+
+```python
+import shutil
+import lightning as L
+from lightning.pytorch.callbacks import Callback, ModelCheckpoint
+from checkpoint_progress import latest, progress, save
+
+class StartProgress(Callback):
+    def on_train_start(self, trainer, pl_module):
+        progress("start", trainer.global_step + 1)
+
+class ProgressCheckpoint(ModelCheckpoint):
+    def on_train_epoch_end(self, trainer, pl_module):
+        super().on_train_epoch_end(trainer, pl_module)
+        source = self.last_model_path
+        save(trainer.global_step,
+             lambda path: shutil.copy2(source, path / "last.ckpt"))
+
+checkpoint = ProgressCheckpoint(dirpath="/workspace/.supercomputer/lightning",
+                                save_last=True, every_n_epochs=1,
+                                save_on_train_epoch_end=True)
+if latest():
+    checkpoint.last_model_path = str(latest() / "last.ckpt")
+trainer = L.Trainer(callbacks=[StartProgress(), checkpoint])
+trainer.fit(model, train_dataloaders=train_loader,
+            ckpt_path="last" if latest() else None)
+```
+
+Start long work with `super run --sku gpu_1x_l4 --detach --upload train.py
+--upload checkpoint_progress.py -- python train.py` so supercomputer can re-run the
+command after a restart. A command typed inside `super shell` is not re-run. The
+host does not keep `--env`/`--secret` values, so a command started with them is
+re-run only by a client still attached to it (not after `--detach`).
+
+### File Transfer
+
+| Command                                | Description                                 | Needs active session?           |
+| -------------------------------------- | ------------------------------------------- | ------------------------------- |
+| `super upload <sid> <local> [remote]`   | Upload files to a session                   | Yes — brings a cold session online (a host starts billing) |
+| `super download <sid> <remote> [local]` | Download a file from a session              | No — works on active **or** closed |
+| `super ls <sid> [path]`                 | List a session's workspace manifest         | No                              |
+| `super mount <sid> <mountpoint>`        | Mount a session's files (read-only)         | No                              |
+
+Only `upload` needs a live host. `download` uses the live host when the
+session is active, for the freshest bytes, and falls back to the saved cloud-storage
+manifest otherwise. `ls` and `mount` are manifest views: while the session runs
+they can lag its files by a sync or two; after `super stop --wait` they are final.
+A closed CLI job also has a saved workspace; use its session ID with `download`,
+`ls`, or `mount`.
+
+
+Uploaded files land in `/workspace/` on the host. Files **at or under 5 MiB**
+stream inline over the WebSocket; anything larger goes client → cloud storage
+→ host via a pre-signed URL minted by the supercomputer API. `super upload` prints which
+files took that second route (and names them again if it fails), so a broken
+large-file upload is attributable to the file, not to the command — one
+oversized results JSON in an otherwise tiny code tree is enough to put the
+batch on the pre-signed path. That path is a single PUT, so each file is
+capped at **5 GiB**; split larger files (e.g. `split -b 4G`) or stage them
+through a volume/import path instead. Downloads work the same way in
+reverse. Uploads keep each file's permission bits (`rwx` for user/group/other;
+setuid/setgid/sticky are dropped), so a `0755` script stays executable;
+`download` does not restore modes locally.
+
+**Client version.** Every request states the client's version, and the API
+rejects clients below the deployment's floor with HTTP 426 / `client_too_old`
+(`supercomputer.ClientTooOldError`) naming the upgrade command. If you see that, run
+`pip install -U gpu-compute-provider` — it is a client-age problem, never a credential one.
+
+### Browsing & mounting a session
+
+`super ls` and `super mount` read the session's file manifest. The host
+refreshes it after each periodic workspace sync while the session runs (so it
+can lag by a couple of minutes) and writes the final copy at stop. They are
+most useful after a `super stop --wait` (the common "grab my outputs later"
+case).
+
+```bash
+super ls s-abc123              # list files at the root
+super ls s-abc123 data        # list files under data/
+super mount s-abc123 ~/mnt     # browse in Finder/Explorer (read-only)
+```
+
+A stopped session's **durable logs** appear under a virtual `_logs/` directory
+next to its workspace files — the image build log, the session lifecycle log,
+and the per-process exec logs the host streamed to cloud storage:
+
+```bash
+super ls s-abc123 _logs                  # build.log, session.log, exec-<pid>.log, ...
+super download s-abc123 _logs/build.log  # read a log long after the session is gone
+```
+
+These are the same streams `super logs <sid> <pid>` shows live; `_logs/` just
+makes them browsable and downloadable post-hoc, and they appear under `_logs/`
+in `super mount` too.
+
+`mount` streams file bytes **directly from cloud storage** to your machine (the
+API only serves the listing + a redirect), so it stays fast for large
+files. It uses rclone's **NFS mount** (the OS NFS client), so it needs only
+rclone — **no macFUSE, no kernel extension, no recovery-mode approval**:
+
+```bash
+brew install rclone          # macOS
+# Linux: sudo apt-get install -y rclone   (or curl https://rclone.org/install.sh | sudo bash)
+```
+
+Then `super mount <sid> <dir>` and open `<dir>` in Finder/Explorer. `<dir>`
+must be a path that does **not** exist yet — super creates the mountpoint and
+deletes it on unmount, and refuses an existing folder so it can never touch
+your data. Note: on
+macOS the **Terminal** may be blocked from reading the mount (a privacy/TCC
+quirk) — browse it in **Finder**, or use `super ls` from the CLI. `super ls`
+needs no rclone at all (plain API call). Press Ctrl-C to unmount.
+
+### Volumes
+
+Volumes are named, manifest-backed folders for moving data between sessions.
+They are not attached disks and do not reserve separate block storage: file bytes
+stay in the shared content-addressed blob store, while the volume records the
+path → blob manifest.
+
+| Command                                                     | Description                                      |
+| ----------------------------------------------------------- | ------------------------------------------------ |
+| `super volume create <name>`                                 | Create an empty volume                           |
+| `super volume list`                                          | List your volumes                                |
+| `super volume files <name> [path]`                           | List files in a volume                           |
+| `super volume download <name> <vol-path> [local] [-r]`       | Fetch a file (or, with `-r`, a directory) locally |
+| `super volume cat <name> <vol-path>`                         | Print a volume file to stdout                    |
+| `super volume delete <name>`                                 | Delete a volume and its manifest                 |
+| `super volume import <name> <sid> [session-path] [vol-path]` | Copy a folder from a stopped session into volume |
+
+Reads need no session and no host — `files` / `download` / `cat` serve the
+volume's committed manifest straight from the API, so this is how you
+get a job's outputs back:
+
+```bash
+super volume files results            # what's in there
+super volume cat results slice-0.json # small file to stdout
+super volume download results slice-0.json
+super volume download results / ./out -r   # whole volume into ./out
+```
+
+`delete` returns a conflict while any session still has the volume mounted;
+stop or detach that session first.
+
+Mount a volume into a session with repeatable `--volume` flags — on `exec`,
+`run`, `shell`, `spawn`, `job run` and `map`:
+
+```bash
+super exec --volume checkpoints:/workspace/checkpoints --sku gpu_1x_l4 \
+  python train.py
+super run --volume checkpoints . --script train.py --sku gpu_1x_l4
+```
+
+`--volume NAME` mounts at `/workspace/NAME`; `--volume NAME:/absolute/path`
+mounts at that absolute path inside the session rootfs.
+
+**Volumes are not block devices and are not mounted with the unix `mount`
+command.** There is no `/dev/*` node — don't probe for one with `mount`,
+`lsblk`, or `df` inside the session, and don't try to mount/umount/format
+anything. The host composes each volume into the session's root filesystem as a
+read-only content-addressed layer (file bytes fetched lazily from the blob
+store on first read) plus a writable overlay at the mount path. Inside the
+session a volume is just an ordinary directory: use normal file I/O. Writes
+land in the overlay and are committed back to the volume as a new version on
+detach or session close.
+
+On a new session,
+`--volume` rides the create and is composed into the filesystem at setup, so a
+volume-mounted command queues and provisions a host like any other session;
+`--volume` also works against an already-running session (`--on <sid>`), where
+it is attached live. Any number of sessions may attach the
+same volume concurrently — commits merge per path, last writer wins per file
+(so key parallel writers' outputs by distinct paths). Only `delete` and
+`import` conflict while a volume is attached; browsing works any time. Append
+`:ro` (`--volume data:/vol:ro`) to attach read-only: the session sees the
+volume but its writes are discarded at close (no commit).
+
+Declare volumes once in `supercomputer.json` to skip `--volume` on every command (see
+[Project Config](#project-config-supercomputerjson)):
+
+```json
+{ "sku": "gpu_1x_l4", "volumes": [{ "name": "checkpoints", "mount": "/workspace/checkpoints" }] }
+```
+
+The effective set is the `supercomputer.json` volumes plus any `--volume` flags, keyed by
+name; a `--volume` for a name already in `supercomputer.json` re-maps its mount path for
+that invocation (just like `--sku` overrides the config `sku`). The SDK
+re-attaches this set every time it brings the session online — including after a
+stop/resume on a fresh host — and attach is idempotent, so re-passing a volume
+already mounted is a safe no-op.
+
+`volume import` is a single batch metadata operation at folder scope; it does
+not issue per-file API calls and does not copy bytes when blobs already exist.
+The source session must be stopped/closed so the import reads a finalized
+manifest. Import (like delete) is refused while the target volume is attached
+to an active session; browsing is not.
+
+```bash
+super volume create checkpoints
+super stop s-abc123
+super volume import checkpoints s-abc123 runs/run-42 run-42
+super volume files checkpoints run-42
+```
+
+#### End-to-end: persist results into a volume across sessions
+
+A volume is the way to carry data from one session to another (or to a later
+run on a fresh host). Writes land in the volume on session close; reads see them
+on the next attach — no manual sync step.
+
+```bash
+# 1. Create the volume once.
+super volume create runs
+
+# 2. Run a job that writes into the mounted volume. This queues and
+#    provisions a host even when none is idle (no --volume special-casing).
+super exec --volume runs:/workspace/runs --sku gpu_1x_l4 \
+  -- sh -lc 'python train.py && cp model.pt /workspace/runs/model.pt'
+
+# 3. After the session closes, the file is committed to the volume.
+super volume files runs            # -> model.pt
+
+# 4. A later session mounts the same volume and sees the file.
+super exec --volume runs:/workspace/runs --sku gpu_1x_l4 \
+  -- ls -l /workspace/runs/model.pt
+```
+
+Two things worth knowing:
+
+- **Mount onto an empty path.** Attaching a volume onto a directory that
+  already contains session files is rejected (`volume_mount_target_not_empty`)
+  rather than silently shadowing them. Mount at a fresh path like
+  `/workspace/runs`. The same applies to `super run <dir> --volume`: a mount
+  at the upload destination (`/workspace/<dir>`), or below it where the
+  uploaded tree has content, is rejected before anything is uploaded or run.
+- **Commit mid-run by stopping.** Closing the session commits attached volumes
+  automatically — that is the common case. To commit a volume mid-run, use
+  `super stop <sid> --wait` and use the session again — stopping commits the
+  volume, and the session comes back on next use with the mount intact.
+  The commit lands asynchronously, up to ~30s after the session reports
+  closing — a `volume files`/`cat` in that window reads the previous version.
+
+### Background Processes
+
+
+| Command                                          | Description                                     |
+| ------------------------------------------------ | ----------------------------------------------- |
+| `super spawn [--sku SKU \| --on SID] [--volume NAME[:MOUNT]] [--emit-pid FILE] -- CMD` | `super run --detach`; can create a session |
+| `super wait <sid> <pid> [--timeout DUR]`          | Block until a process exits; return its exit code + tail |
+| `super logs <sid> <pid> [--follow] [--output MODE]` | Show output from a process                 |
+| `super kill <sid> <pid>`                          | Kill a background or auto-detached exec process |
+
+`spawn --emit-pid FILE` writes `{"session_id": ..., "process_id": ...}` JSON —
+the same format `exec`/`run --emit-pid` produce — not a bare process id.
+
+Use `--follow` for live monitoring. Use `--output all` after a command has already
+run to fetch its durable log. `--stream stdout|stderr|combined` selects a stream for
+newer process logs; `combined` is the default and works with older logs.
+
+**Waiting on a job.** Prefer `super wait <sid> <pid>` over hand-rolled
+`until super logs ... | grep` loops. It blocks until the process actually
+terminates, prints its exit code and log tail, and **exits with the process's
+own exit code** (`75` if a `--timeout` elapses while it's still running, `69`
+if the host was lost with the process, `1` if it was killed with no code). It
+reads durable logs, so it also works after the host is gone. This is the
+reliable way to detect a failed background job — a grep loop can miss failures
+whose text doesn't match the pattern, and can't tell "still starting" from
+"already died". In `--json` mode `super logs <sid> <pid>` and `super wait` both
+surface terminal `status` and `exit_code`, so a poller can check
+`status != "running"` instead of grepping log text.
+
+### SSH
+
+
+| Command                              | Description                                               |
+| ------------------------------------ | --------------------------------------------------------- |
+| `super shell [--sku SKU] [--volume NAME[:MOUNT]]` | Start an interactive supercomputer-native shell in a new session |
+| `super shell [--on SID|SID] [--volume NAME[:MOUNT]] [cmd...]` | Attach an interactive PTY to an existing session |
+| `super ssh <session-id>`              | SSH into a session                                        |
+| `super tunnel <sid> [--port PORT]`    | SSH ProxyCommand tunnel                                   |
+
+
+`shell` is the supercomputer-native PTY path. `super shell` creates a temporary
+interactive session, while `super shell <sid>` or `super shell --on <sid>`
+attaches to a persistent session and leaves it alive on exit. `--volume` works
+the same as on `exec`: mounts ride the create for a new session and are
+attached before the prompt on an existing one. `ssh` uses the
+local OpenSSH client over `super tunnel`; use it when you specifically want SSH
+tooling/config behavior.
+
+`super tunnel <sid>` defaults to the session SSH server port (`2222`) so it can
+be used directly as an OpenSSH `ProxyCommand`. Pass `--port` only when tunneling
+to a different TCP service inside the session.
+
+`super ssh` requires an SSH server inside the session image. If `sshd` is not
+already present, the CLI bootstraps it with `apt-get install openssh-server` on
+the first SSH connection, which can add tens of seconds. To avoid that cost,
+include `openssh-server` in the project image when SSH access matters:
+
+```json
+{
+  "image": {
+    "apt": ["openssh-server"]
+  }
+}
+```
+
+`super ssh` stops that `sshd` when it exits (so the session can go idle); the
+next `super ssh` restarts it, which is quick once the package is installed.
+
+---
+
+## Sweeps (`super map`) — fan one command out over N bindings
+
+A sweep runs your command N times with different arguments as a **durable
+server-side fan-out**: submit, disconnect, and the service drives every
+task through its own session — queueing for capacity, retrying failures, and
+recording exit codes + logs. No changes to your code; your script's CLI *is*
+the interface.
+
+```bash
+# Hyperparameter sweep (cartesian product: 3 × 2 = 6 tasks)
+super map --name lr-sweep --sku gpu_1x_l4 \
+  -- python3 train.py --lr {lr} --bs {bs} ::: lr=1e-4,3e-4,1e-3 ::: bs=32,64
+
+# Datagen over indices (shell brace expansion feeds the group)
+super map --name gen -- python3 gen.py --shard {} --of 500 ::: {0..499}
+
+# One task per line from a pipe (no ::: → stdin binds {})
+jq -r '.items[]' work.json | super map --name batch -- python3 proc.py {}
+```
+
+**Supplying args (GNU-parallel style).** Everything after the command is
+parsed for `:::` markers: each one starts a **group** of values (space- or
+comma-separated), and each task runs the command with one combination of
+values substituted in. Four ways to feed values:
+
+| Spec | Tasks | Meaning |
+|---|---|---|
+| `cmd {} ::: a b c` | 3 | One task per value; `{}` (or `{1}`) is the value |
+| `cmd {1} {2} ::: a b ::: x y` | 2×2 = 4 | Multiple groups **cross** (cartesian product) |
+| `cmd {1} {2} ::: m1 m2 :::+ t1 t2` | 2 | `:::+` **zips** with the previous group (paired values like model+tokenizer; lengths must match) |
+| `producer \| super map -- cmd {}` | one per line | No `:::` groups → each stdin line is one task |
+
+Placeholders: `{}` = the single group's value · `{1}` `{2}` = group by
+position · `{lr}` = named group (write it `::: lr=1e-4,3e-4`) · `{i}` = task
+index (0-based) · `{n}` = task count. A command with **no** placeholders gets
+each task's values appended at the end (`cmd ::: a b` runs `cmd a`, `cmd b`).
+Every task also gets env vars `SUPERCOMPUTER_TASK_ID`, `SUPERCOMPUTER_WORLD_SIZE`, `SUPERCOMPUTER_JOB` —
+a script can read those to self-shard and skip templating entirely.
+`super help map` prints this reference.
+
+**Code binding modes.** The sweep pins an immutable snapshot at submit, so
+task 400 runs the same bytes as task 0:
+- **Default (Mode 2):** the current dir (or `--code DIR`) is hashed and
+  uploaded once; each task session starts from it. Image comes from
+  `supercomputer.json` or the SKU default.
+- **`--from <session-id>` (Mode 1):** fork the session you just verified —
+  each task starts from that session's synced workspace (deps installed,
+  weights downloaded) and inherits its image. The session must have synced a
+  manifest (a stopped session always has; a live one syncs periodically).
+
+**Outputs.** Write task-keyed files (`shard-{i}.jsonl`) into the task's own
+workspace and pull them all with **`super sweep harvest <name> <remote-glob>
+<local-dir>`** — it resolves each task's session from the sweep and writes
+`<local-dir>/task-<index>/<path>`, so there is no session-id bookkeeping and
+no cross-task filename collisions. (`super download <task-session> <path>` is
+still there for one file from one task.) Or write into a shared volume. Volume commits are per-path
+merges (last writer wins per file): parallel tasks writing DISTINCT files
+into one shared volume all land safely; concurrent writes to the SAME file
+keep only the last committer, so key every task's output by `{i}`. Commits
+ride each task session's close and can trail the task's "succeeded" state
+by ~30s — read the volume after commits settle.
+
+**Code.** With no `--from`/`--code`, map pins the current directory (upload
+skip rules apply) and prints the path/file-count/byte summary before
+uploading; it refuses to pin `$HOME` or a filesystem root implicitly.
+
+**Fan-out width.** `--max-parallel N` caps how many tasks run at once;
+without it the cap is **8**, and submitting more tasks than that runs them
+in waves. Submit says so out loud, and the cap is changeable on a live
+sweep: `super sweep set <name> --max-parallel N`.
+
+**Canary.** Task 0 runs first and gates the rest: a broken sweep costs one
+task, not N. `super map` blocks until the canary passes, then detaches (Ctrl-C
+detaches the watch — it does NOT cancel). `--no-canary` / `--no-wait` opt out.
+When one task takes minutes, `--canary-smoke 'python3 -c "import train"'`
+gates on a cheap check instead and task 0 then runs its real command with
+the rest — cheaper than `--no-canary`, which risks N bad tasks to save one.
+With `--retries` left at its default, a canary whose command exits non-zero
+fails the gate on that first attempt; pass `--retries N` and the canary gets
+that budget like every other task (the rest still wait behind it). Fix it, then
+`super sweep retry <name>` re-runs the canary and launches the rest once it passes.
+Failures don't consume the retry budget when caused by spot preemption — the
+task just requeues.
+
+**Skip the line.** When tasks are blocked on capacity, `super sweep boost
+<name>` (or pressing `s` in `sweep status --watch`, offered only while a
+line exists) moves the sweep to the fast claim lane: queued tasks skip
+ahead of free-lane work, and freed machines alternate fast/free so nobody
+is starved. SDK: `handle.boost()`. Unpriced for now.
+
+**Waiting for capacity is not failure.** A task with no host yet keeps its
+queued session, charges no retry, and is reported as blocked:
+`super sweep status` prints `blocked on capacity: N task(s), oldest 6m12s`
+followed by a `why:` line saying what the wait actually is — a line
+(`3 session(s) ahead in the gpu_1x_l4 queue (1 fast-lane)`), the platform
+(`a new machine is being provisioned`, `waiting for a busy machine to free
+up`), or, for a sweep with a `deadline` that is missed or at risk, `could
+not find capacity for this SKU in time for the deadline`. Use it to decide:
+a line or a provision resolves itself (or `boost` past the free lane); the
+deadline line means pick another SKU, move the deadline or cancel. A `state: waiting` sweep is a
+pipeline hold, and status names the upstream sweeps it waits on.
+
+**Retry backoff.** A charged failure re-queues with a growing delay (45s ×
+attempt, capped at 10m) so a deterministic crasher doesn't burn its budget
+in seconds; preemption/lost-host requeues never consume the budget (the
+first two re-run immediately, later ones back off the same way, and a task
+fails with an `infra:` error after 20).
+
+**Pipelines (`--after`).** `--after SWEEP[,SWEEP]` holds a sweep in
+`waiting` until the named sweeps finish, then launches it — a durable
+server-side barrier, so a whole generate → train → eval pipeline submits up
+front and the client can disconnect:
+
+```bash
+super map --name gen --volume run42:/vol \
+  -- python3 gen.py --out /vol/data/shard-{i}.jsonl ::: {0..63}
+super map --name train --sku gpu_8x_h100 --volume run42:/vol --after gen \
+  -- python3 train.py --data /vol/data --ckpt /vol/ckpt
+super map --name eval --volume run42:/vol --after train \
+  -- python3 eval.py --ckpt /vol/ckpt --suite {} ::: mmlu,gsm8k,mbpp
+```
+
+Data flows between stages through the shared volume: map tasks write
+disjoint task-keyed paths, the next stage mounts the same volume and sees
+the union (the gate waits for the upstream's volume commits to land before
+releasing). An upstream failure fails the downstream sweep with
+`upstream_failed:<name>` instead of running it on missing inputs; after
+fixing and re-running the upstream, `super sweep retry <name> --all`
+re-arms the downstream. A `succeeded_partial` upstream counts as finished
+unless `--after-strict`. Edges bind at submit, so upstreams must be
+submitted first (which also makes cycles impossible). SDK:
+`client.map(..., after=gen_handle)` (or a name, or a list; `after_strict=True`).
+
+**Tracking & recovery:**
+```bash
+super sweeps                          # all sweeps: state + done/run/fail counts
+super sweep status <name>             # counts + failures clustered by error, with exemplar task
+super sweep status <name> --watch     # same, repainted until terminal (--interval S)
+super sweep harvest <name> '*.json' ./out   # every task's outputs → out/task-<i>/…
+super sweep logs <name> [--task N] [--attempt K]  # durable output (defaults to the exemplar failure, falling back to earlier attempts; --attempt 1 = most recent attempt, explicitly)
+super sweep set <name> --max-parallel N  # change the fan-out cap on a live sweep
+super sweep retry <name> [--all]      # re-run failed tasks (fix code first: resubmit is idempotent); after a canary failure it re-runs the canary first and launches the rest when it passes; --all adds canceled + WEDGED in-flight tasks (lost host / dead placement) and never restarts healthy running work
+super sweep cancel <name> [--remaining]  # stop; --remaining keeps running tasks (partial success)
+super sweep wait <name> [--poll S] [--timeout S]  # block until terminal; exit 0 = succeeded (incl. succeeded_partial), 1 = failed/canceled, 124 = still running at timeout
+```
+`--webhook URL` POSTs the status document on canary pass/fail and
+completion — agents should submit with `--no-wait --json` and wake on the
+webhook instead of polling. There is no account-default webhook: without
+`--webhook`, nothing is POSTed.
+
+**The status JSON is a versioned public contract** (`schema_version: 1`),
+specified in `docs/sweep-status-schema.md`: `counts` is an open map keyed by
+task state, the task→session mapping is `tasks[].session_id`, sweep state is
+`job.state`, and per-session spend is under `cost` (requested with
+`super sweep status <name> --cost`). Additive changes keep the
+version; a removal or rename bumps it. Do not reverse-engineer it — and
+prefer `super sweep harvest` over re-deriving the task→session→download walk
+by hand.
+
+**SDK mirror** — `client.map(command, args=…)`. The *type* of `args` picks
+the combinator (no `:::` syntax in the SDK):
+
+| `args` | Tasks | CLI equivalent |
+|---|---|---|
+| `{"lr": [1e-4, 3e-4], "bs": [32, 64]}` | 4 — cartesian product, binds `{lr}`/`{bs}` | `::: lr=… ::: bs=…` |
+| `[{"model": "a", "tok": "x"}, {"model": "b", "tok": "y"}]` | 2 — one per row, pre-paired | `::: … :::+ …` |
+| `["f1.json", "f2.json"]` | 2 — one per item, binds `{}` | `::: f1.json f2.json` |
+| `500` or `range(500)` | 500 — same command N times; index via `{i}`/`SUPERCOMPUTER_TASK_ID`, nothing appended | `::: {0..499}` |
+| `None` (default) | 1 | — |
+
+```python
+sweep = client.map("python3 train.py --lr {lr} --bs {bs}",
+                   args={"lr": [1e-4, 3e-4], "bs": [32, 64]},   # 4 tasks
+                   name="lr-sweep", sku="gpu_1x_l4", code_dir=".")
+sweep.wait_canary(); sweep.wait()  # or poll sweep.status() / .retry_failed()
+# from_session="s-…" instead of code_dir for Mode 1. Returns a SweepHandle.
+```
+
+---
+
+## Agent Workflow Patterns
+
+### Pattern 1: Quick one-off command
+
+The simplest pattern. Run a command, get the result, done.
+
+```bash
+super exec --sku gpu_1x_l4 --json nvidia-smi -L
+# {"stdout": "GPU 0: NVIDIA L4 ...\n", "returncode": 0, "status": "exited", ...}
+```
+
+### Pattern 2: Run a training script
+
+Upload code, run it, get results. Session lifecycle is fully managed.
+
+```bash
+super run train.py --sku gpu_1x_l4 --json -- --epochs 50
+# {"stdout": "...", "stderr": "", "returncode": 0, "duration_ms": 34521}
+```
+
+For a project directory with dependencies:
+
+```bash
+super run . --script train.py --sku gpu_1x_l4 -- --batch-size 32
+```
+
+### Pattern 3: Persistent session (multi-step workflow)
+
+Create a session once, interact with it across multiple commands. The session's
+filesystem persists between invocations — no re-uploading. It comes online on the
+first use and resumes transparently if it was reaped while idle.
+
+```bash
+# Create (returns a session ID; cold, $0 until first used)
+super create --sku gpu_1x_l4 --json
+# {"session_id": "s-abc123", "status": "closed", "sku": "gpu_1x_l4", "checkpoint": false, "restart": true}
+
+# Install dependencies (brings the session online automatically)
+super exec --on s-abc123 pip install torch transformers
+
+# Upload training data
+super upload s-abc123 ./data
+
+# Run training
+super run train.py --on s-abc123 -- --epochs 50 --lr 1e-4
+
+# Same persistent-session target using the positional alias
+super run s-abc123 train.py -- --epochs 50 --lr 1e-4
+
+# Check results
+super exec --on s-abc123 cat /workspace/results.json
+
+# Download model weights
+super download s-abc123 /workspace/model.pt ./model.pt
+
+# Halt spend now (files kept; resumes on next use)
+super stop s-abc123
+```
+
+Each command connects in ~1s, runs, and detaches. The session's files persist
+across all invocations on the host's `/workspace/`.
+
+This is the recommended pattern for agents running multi-step experiments —
+cheaper and faster than creating a new session for every command.
+
+Use `--wait` for commands that are expected to take longer than 30s but should
+remain foreground from the agent's perspective. If the command detaches anyway,
+follow the returned `process_id` instead of retrying the same command.
+
+### Pattern 4: SDK session (programmatic multi-step)
+
+For multi-step workflows (install deps → upload data → train → download results),
+use the Python SDK directly. A session stays alive across all operations.
+
+```python
+from supercomputer import Client, Image
+
+client = Client()
+image = (Image.debian_slim()
+         .pip_install(["torch==2.8.0"], index_url="https://download.pytorch.org/whl/cu128")
+         .pip_install(["numpy"]))
+project = client.project("my-experiment", image=image)
+
+with project.session(sku="gpu_1x_l4") as s:
+    # Install and verify
+    r = s.run(["python3", "-c", "import torch; print(torch.cuda.is_available())"])
+    print(r.stdout)  # True
+
+    # Upload training data
+    s.upload("./data", "data")
+
+    # Run training
+    r = s.run(["python3", "/workspace/data/train.py"], timeout_ms=3600_000)
+    print(r.stdout)
+
+    # Download results
+    weights = s.download("model.pt")
+    open("model.pt", "wb").write(weights)
+
+    # Session closes automatically on exit
+```
+
+### Pattern 5: Long-running job with monitoring
+
+For jobs that take minutes/hours, **lead with `spawn` + `wait`, not a maxed-out
+`exec --wait`.** `spawn` returns a durable `process_id` immediately; `wait`
+blocks on that id reading the durable log, so it survives host migration and
+never depends on a foreground stream staying alive:
+
+```bash
+# Start the job — prints a durable proc-id right away, no blocking.
+super spawn s-abc123 python3 /workspace/train.py
+# Process proc-456 spawned in session s-abc123
+
+# Block until it exits (reads durable logs; survives migration). --timeout 0
+# waits indefinitely; a positive timeout exits 75 (still running) so you retry.
+# A process id the service has no record of (typo'd or stale) errors out
+# with "unknown process id" (exit 1) instead of telling you to keep waiting.
+super wait s-abc123 proc-456 --timeout 0
+
+# Or watch it live:
+super logs s-abc123 proc-456 --follow
+# Fetch the full log after it finishes (default stream is combined):
+super logs s-abc123 proc-456 --output all
+# Stop it if needed:
+super kill s-abc123 proc-456
+```
+
+**Don't max out `--wait` on a long job, and never pipe a live `exec` through
+`| tail`.** `exec --wait 60m ... | tail` buffers all output until the process
+ends (so you see nothing live) *and* swallows the detach line carrying the
+`process_id` — leaving you unable to call `super wait` / `super logs --follow` at
+all. If you must background a long `exec`, capture the id with `--emit-pid`
+(`spawn` takes the same flag and writes the same file), which writes
+`{session_id, process_id}` JSON to a file the instant the job starts —
+readable even when stdout/stderr are piped or backgrounded:
+
+```bash
+super exec --on s-abc123 --emit-pid /tmp/job.json --wait 60m python3 train.py &
+# ... the file exists as soon as the process starts:
+PID=$(python3 -c 'import json;print(json.load(open("/tmp/job.json"))["process_id"])')
+super wait s-abc123 "$PID" --timeout 0
+```
+
+> **Transient unreachability (`rc=75`).** A foreground `exec` can exit **`rc=75`**
+> (EX_TEMPFAIL) when the session is briefly unreachable — most often the first
+> exec right after a host boots, while the session is still binding to it, or a
+> host reclaimed mid-run (spot preemption) that supercomputer recovers on a
+> new host. `exec` retries the attach for you and only reports `rc=75` once it
+> still can't reach the session; the message states the real cause ("not
+> reachable yet" vs. an ended session) instead of assuming a migration. Just
+> retry — the next attempt lands on the ready/restored host. `super wait` /
+> `super logs` ride a migration for you. After one, confirm the process is still
+> alive (`super logs <sid>` lists it as `running`) rather than assuming `rc=75`
+> means your job is gone.
+>
+> **Out of credit (`rc=4`).** A 402 with `code: insufficient_credit` means the
+> prepaid balance is exhausted: sessions are closed and queued jobs canceled.
+> **This is not retryable** — retrying returns the same refusal until credit
+> lands. Distinguish it from `rc=3` (`payment_overdue`, a failed payment on a
+> bill) and from `rc=75` (transient unreachability, which *is* retryable). The
+> remedy is `super credits buy 25` or `super credits autorefill on`; the block
+> clears about a minute after the credit posts. In the SDK it raises
+> `supercomputer.InsufficientCreditError`, a sibling of `PaymentOverdueError` — code
+> that catches only `PaymentOverdueError` will not catch it.
+>
+> **Migration rides honor `--wait`.** When a foreground `exec` rides a
+> restart or experimental checkpoint/restore, the ride is bounded by your `--wait` budget — it never
+> blocks past it (`super run` with no `--wait`, or `--wait 0`, has no bound; the
+> ride then uses the full ~10-minute restore window). If the budget runs out while the restore is
+> still in flight, the result is **`status: "migrating"`** with **`rc=75`**:
+> the process is neither confirmed alive nor dead yet. JSON callers can use
+> the status to distinguish a ride-in-progress from a live process
+> (`status: "running"`, also `rc=75`, which means the command itself outlived
+> `--wait`). Either way, follow up with `super logs <sid> <pid> --follow` or
+> `super wait`.
+
+With the SDK:
+
+```python
+from supercomputer import Client
+
+client = Client()
+project = client.project("long-job")
+
+with project.session(sku="gpu_1x_l4") as s:
+    s.upload("./project")
+    proc = s.spawn(["python3", "/workspace/project/train.py"])
+
+    # Poll periodically
+    import time
+    while proc.status.value == "running":
+        proc.poll(lines=20)
+        print(proc.output[-200:])  # last 200 chars
+        time.sleep(60)
+
+    print(f"Done: exit_code={proc.exit_code}")
+    print(s.logs(proc, output_range="all").output)
+```
+
+### Pattern 6: Stop now and come back later (file persistence)
+
+Files written during a session are synced to durable cloud storage. After you
+stop it (or it is reaped while idle), the session comes back online automatically
+the next time you use it, with all files restored. There is no manual resume step.
+
+```bash
+# Create a session, do work
+super create --sku gpu_1x_l4
+# Session s-abc123 created (cold — no GPU running, $0 until used).
+
+super exec --on s-abc123 python3 train.py
+super exec --on s-abc123 ls /workspace/
+# model.pt  metrics.json  train.py
+
+# Halt spend now (files sync to cloud storage)
+super stop s-abc123
+
+# Hours later: just use it again — it comes back online automatically,
+# with files restored from cloud storage.
+super exec --on s-abc123 ls /workspace/
+# model.pt  metrics.json  train.py  ← files restored from cloud storage
+```
+
+With the Python SDK:
+
+```python
+from supercomputer import Client, Image
+
+client = Client()
+project = client.project("experiment", image=Image.debian_slim())
+
+# First run: train on L4
+with project.session(sku="gpu_1x_l4") as s:
+    s.run(["python3", "train.py"])
+    sid = s.session_id
+
+# Later: just attach and use it again — a cold session comes back
+# online automatically (rebuilt on a fresh host, files restored).
+s2 = client.attach_session(sid)
+s2.run(["ls", "/workspace/"])  # original files are here
+s2.run(["python3", "finetune.py"])
+s2.close()
+```
+
+**What survives a resume**: `/workspace` always survives. Changes *outside*
+`/workspace` (e.g. `pip install` into the image's site-packages) survive an
+ordinary same-image resume, but are **reset to the image** when the session
+resumes with a changed `supercomputer.json` image spec (a re-baseline) or when the
+previous close could not finish syncing — the session prints an
+`environment reset` banner when that happens. For packages that must survive
+unconditionally, install them into a venv under the workspace:
+
+```bash
+super exec --on s-abc123 'python3 -m venv /workspace/venv && /workspace/venv/bin/pip install torch'
+super exec --on s-abc123 /workspace/venv/bin/python train.py
+```
+
+Better yet, put packages in the image spec (`pip_install([...])`) so every
+resume rebuilds them from the content-addressed image cache. A sweep whose
+canary pip-installs packages gets them baked into its image automatically
+before the fan-out (the pass line names them; `--no-bake` disables) — but
+the pins still belong in `supercomputer.json` so the *next* sweep never pays them.
+
+---
+
+## Image Builder
+
+Images define the environment (base OS, packages, env vars). They're declarative
+and content-addressed — same layers = cache hit, instant startup.
+
+```python
+from supercomputer import Image
+
+# GPU-ready image with PyTorch
+image = (
+    Image.debian_slim()
+    .apt_install(["git", "build-essential"])
+    .pip_install(["torch==2.8.0"], index_url="https://download.pytorch.org/whl/cu128")
+    .pip_install(["transformers", "numpy"])
+    .env({"PYTHONUNBUFFERED": "1"})
+)
+
+project = client.project("my-project", image=image)
+```
+
+Use `Image.from_registry(...)` when the workload needs a specific container base:
+
+```python
+from supercomputer import Image
+
+# CUDA devel images include nvcc, ptxas, CUDA headers, and other build-time tools.
+image = (
+    Image.from_registry("nvidia/cuda:12.8.1-devel-ubuntu24.04")
+    .apt_install(["ninja-build"])
+    .pip_install(["torch==2.8.0"], index_url="https://download.pytorch.org/whl/cu128")
+    .pip_install(["triton", "numpy"])
+    .env({"PYTHONUNBUFFERED": "1"})
+)
+```
+
+Reach for a CUDA `devel` base when packages build CUDA code from source
+(`torch.utils.cpp_extension`, flash-attn, xformers, mamba, custom kernels). Runtime
+CUDA/driver availability is not enough for those workflows; they need the compiler
+toolchain inside the image.
+
+For GPU workloads, the base image must also provide a glibc new enough to load the
+host NVIDIA driver libraries injected by the runtime. This is distro-agnostic:
+Debian, Ubuntu, SUSE, Rocky, etc. can all work if their glibc satisfies the NVIDIA
+library requirement. Do not choose old GPU bases casually; failures usually show up
+as `GLIBC_2.xx not found`, CUDA initialization failures, or `nvidia-smi`/torch CUDA
+errors. Prefer a recent CUDA base, for example an NVIDIA CUDA image based on Ubuntu
+24.04, unless the user has a specific compatible base. For ordinary CPU/Python
+packages, `Image.debian_slim()` is usually faster to build and cache.
+
+On a GPU SKU, `supercomputer.json` `image.pip` installs torch packages from the
+CUDA 12.8 PyTorch index in their own layer, because an unpinned torch from PyPI
+can be a newer CUDA build the hosts' drivers cannot run. The SDK's
+`Image.pip_install` does not do this; pass `index_url` yourself, as above. Do
+not put system build tools such as `ninja` in pip requirements just because a
+package build mentions them; install `ninja-build` with apt instead.
+If a workload mixes PyTorch wheels with ordinary PyPI-only packages, split them
+into separate image layers or configure the package indexes explicitly.
+
+Pre-built named images:
+
+- `Image.from_name("agent-gpu")` — the unified default for every `gpu_*` SKU:
+  `nvidia/cuda:12.8.1-devel-ubuntu24.04` (nvcc + CUDA headers, Python 3.12)
+  with a pinned training stack (torch 2.8.0+cu128, transformers, trl, peft,
+  accelerate, flash-attn prebuilt wheel). One image covers T4 through
+  Blackwell (sm75–sm120). Note: flash-attn *kernels* need sm80+ — on T4 the
+  import works but attention stays on sdpa, so avoid TRL
+  `packing`/`padding_free` there. vLLM is part of the pinned stack
+  (`import vllm` / `vllm serve` work out of the box; no extra installs).
+- `Image.from_name("agent-cpu")` — CPU-only ML stack
+- `Image.from_name("scientific")` — numpy, scipy, matplotlib, jupyter
+
+With no `image` section in `supercomputer.json`, CLI commands (`exec`, `run`,
+`shell`, `sweep`) use the SKU's default image: `agent-gpu` on `gpu_*` SKUs
+(Python 3.12, torch 2.8.0+cu128 and the pinned stack above) and `agent-cpu`
+on `cpu_*` SKUs (Python 3.11 with numpy, pandas and scipy; no torch).
+
+---
+
+## Project Config (`supercomputer.json`)
+
+Drop a `supercomputer.json` in your project root to configure the image and default SKU.
+The CLI auto-discovers it by walking up from cwd (same as `.env`).
+
+```json
+{
+    "sku": "gpu_1x_l4",
+    "volumes": [{"name": "checkpoints", "mount": "/workspace/checkpoints"}],
+    "image": {
+        "base": "nvidia/cuda:12.8.1-devel-ubuntu24.04",
+        "apt": ["git", "build-essential", "ninja-build", "ffmpeg"],
+        "pip": ["torch", "triton", "numpy", "transformers"],
+        "env": {"PYTHONUNBUFFERED": "1"},
+        "run": ["echo setup-complete"]
+    }
+}
+```
+
+All fields are optional. Explicit `--sku` flags override the file; `--volume`
+flags merge with (and re-map by name) the `volumes` defaults.
+
+
+| Field        | Description                                |
+| ------------ | ------------------------------------------ |
+| `sku`        | Default hardware SKU for `exec` and `run`  |
+| `volumes`    | Volumes auto-attached to every `exec`/`run` session. Each entry is `"name"`, `"name:/mount"`, or `{"name": ..., "mount": ...}` |
+| `checkpoint` | `false` (default): live migration off. `true`: EXPERIMENTAL live migration, falling back to restart; single-GPU only and requires restart. See *Surviving preemption* |
+| `restart` | `true` (default): restore `/workspace` on a new box and re-run the command, which must load its own checkpoints. `false`: recovery off; cold after a spot notice with nothing re-run, ends after host death |
+| `image.base` | Base image. An `image` section without `base` starts from `python:3.11-slim`, not from the SKU default, so a GPU SKU gets no torch unless `image.pip` adds it |
+| `image.apt`  | Packages to `apt-get install`              |
+| `image.pip`  | Packages to `pip install`                  |
+| `image.env`  | Environment variables baked into the image |
+| `image.run`  | Shell commands to run during build         |
+
+
+When `supercomputer.json` is present, `super exec` and `super run` automatically build the
+specified image. The image is cached by content hash — same config = instant startup.
+
+---
+
+## Key Concepts
+
+**SKU**: Hardware type identifier. Format: `gpu_<N>x_<family>` or `cpu_<N>x_<family>`.
+Examples: `gpu_1x_l4`, `gpu_8x_h100`, `cpu_2x_epyc`. `super skus` lists what exists now.
+
+**Session**: A persistent workspace (filesystem at `/workspace/`) that runs commands
+in a sandboxed container on a host. A session is a filesystem, not a held GPU:
+
+- *Throwaway*: created and released per command (`super exec`, `super run` without
+`--on`) — the filesystem is still saved and downloadable afterwards.
+- *Persistent*: created with `super create`, interacted with via `--on <sid>`. It is
+cold ($0) until used, comes online automatically on use, and `super stop` halts spend
+without deleting it.
+
+**Job**: `super job run` runs a command; its `--retries` re-runs only after a lost box. Its `/workspace`
+persists after the command, and it follows the session's preemption policy.
+The box is released when the command ends. See the Jobs section.
+
+Sessions resume transparently — a stopped or idle-reaped session is rebuilt on a fresh
+host (files restored from cloud storage) the next time you use it. You never run a resume command.
+
+**Warmth**: `super sessions` reports a session's state — `hot` (running, spending),
+`warm` (idle host held, spending), `stopping` (finishing teardown, not billed), `cold`
+(stopped, $0, resumes on use), `preparing` (coming online), or `ended` (a
+finished SDK job, never resumes).
+The `$/HR` column is the rate billed on the session's open usage interval while a
+host is held and billed (0 when cold or stopping); `SPENT` is compute charged so far; the footer sums `$/HR`. `super usage` reports what was actually charged.
+
+**Image**: Declarative container spec. Layers are cached by content hash. First run
+on a host triggers a build (30–120s); subsequent runs with the same image are instant.
+
+**Project**: Groups sessions with a shared image definition. Created implicitly by
+CLI commands or explicitly via `client.project("name", image=...)`.
+
+---
+
+## Tips for Agents
+
+1. **Always use `--json`** when parsing output. Human-readable output may change format.
+2. **Use `--sku`** to target specific hardware. Without it (and without a `sku` in
+  `supercomputer.json`), the scheduler picks any available host and the default
+  image is `agent-cpu` (no torch), even on a GPU host.
+3. **Use `super create` for multi-step work**. One persistent session is faster and
+  cheaper than creating a new ephemeral session for every command.
+4. **Treat `returncode: null` as detached, not failed**. Read `session_id` and
+  `process_id`, then call `super logs <sid> <pid> --follow` or poll the process.
+  If stdout was truncated, call `super logs <sid> <pid> --output all`.
+5. **Use `--wait` to express foreground patience**. Increase it for commands that
+  should finish soon; let truly long jobs detach and monitor them by process ID.
+6. **Pass argv normally; quote a single string (or use `sh -c`) for shell
+  expressions**. A single quoted argument runs as shell text via `bash -lc`.
+7. **Cached images are fast** (~2s). If your workflow creates sessions repeatedly with
+  the same image, it's cheap.
+8. **Files live in `/workspace/` — and it's your default working directory.**
+  Uploaded files are accessible as relative paths (`python train.py`) or as
+  `/workspace/<filename>` / `/workspace/<dirname>/...`. Commands already run
+  from `/workspace`, so don't `cd /root` or hardcode `/root/...` paths.
+9. **Pick a SKU explicitly** with `--sku`. If capacity is unavailable, the session
+  queues until a host is provisioned rather than failing.
+10. **Use `super stop` to halt spend now**. Ephemeral sessions (`exec`, `run`
+  without `--on`) release automatically. Persistent sessions (`create`) keep their
+  files; they stop on their own when idle, so `stop` is only needed to halt spend
+  immediately.
+11. **Just use a session again to pick up where you left off**. Files persist across
+  stop/idle via cloud storage, and the session comes back online automatically (no resume
+  command). You can target different hardware with `--sku`.
+12. **Gracefully stop GPU engines when possible**. Prefer the engine's shutdown
+  method or `super kill <sid> <pid>` over in-session `pkill -9`; if GPU memory
+  remains pinned after a hard kill, recover with `super stop <sid>` and then use the
+  session again.
+
+## Autonomous research loops (autoresearch)
+
+To run an overnight keep/discard experiment loop on the user's code — after
+[karpathy/autoresearch](https://github.com/karpathy/autoresearch), with the GPU on
+supercomputer instead of under the desk:
+
+```bash
+super autoresearch init      # writes autoresearch/SETUP.md + program.template.md
+```
+
+Then read `autoresearch/SETUP.md` and follow it. It walks you through deciding the
+metric, the one mutable file vs the frozen eval, and the fixed time budget with the
+user; you write `program.md`, run the baseline on a persistent session, and hand
+off to the loop, which only ever re-uploads the mutable file and greps the metric.
+Progress is visible on the session's page in the supercomputer dashboard (Autoresearch
+tab: metric chart + results table) once the loop's `results.tsv` upload syncs;
+point the user there when the loop starts.
+
+## Anti-Patterns
+
+
+| Don't                                          | Why                                                                            |
+| ---------------------------------------------- | ------------------------------------------------------------------------------ |
+| Omit `--json` when parsing output              | Human format is for display, not parsing                                       |
+| Request unavailable SKUs                       | Session will queue until a host is provisioned; pick a SKU you expect to exist |
+| Retry after `returncode: null`                 | The original command is still running; follow its process ID                   |
+| Max out `exec --wait` on a long job            | Prefer `spawn` + `super wait`; a huge `--wait` blocks the foreground and depends on the stream staying alive |
+| Pipe a live `exec` through `\| tail`           | Buffers until EOF (no live output) and swallows the detach line with the `process_id`. Use `--emit-pid FILE`, or `spawn` + `wait` |
+| Pipe `spawn` (or any launch) through `\| head` | `head` can exit before the `process_id` line is read: the process runs, but you lose its id. Filter with `grep`, or use `--emit-pid` |
+| Treat a migration `rc=75` as "job died"        | After preemption the command is re-run on a new host (restored, with `--checkpoint`); use `super wait`/`logs`, which ride the migration |
+| Immediately retry a preempted job              | The system automatically attempts to requeue it; resubmitting races the requeue and duplicates the run |
+| Put local shell syntax outside quotes          | The local shell handles it before `super`; quote it or use `sh -c`              |
+| Use `pkill -9` as the first response to a hung GPU engine | Hard kills can bypass framework cleanup; try graceful shutdown or `super kill`, then `super stop` and reuse if needed |
+| Run commands without `--sku` in production     | May land on a CPU when you need a GPU, and the default image has no torch      |
+| Run long work inside `super shell`             | Commands typed there are not re-run after a restart. Start them with `super run --detach` |
+| Save restart checkpoints only to a volume      | Volumes commit when the session ends. Save them under `/workspace` during the run |