opencode-skills-collection 3.1.2 → 3.1.3

package/bundled-skills/remote-gpu-trainer/references/spot-resilience.md

@@ -0,0 +1,235 @@
+# Spot / Preemption Resilience
+
+Make a job survive being killed at a random instant — the price of riding the 50–90 %-cheaper
+spot/preemptible/interruptible tier. The whole layer reduces to **principle #8**
+(`references/principles.md`): checkpoint full state to durable storage on a Young/Daly timer, load-latest
+unconditionally on startup, write atomically, treat the preemption signal only as an opportunistic
+last-flush. This file is the deep form: per-platform grace windows, the cadence formula with a worked
+number, the atomic-write resume recipe, and a commented Python skeleton.
+
+To jump: `grep -in '<keyword>' references/spot-resilience.md` — keywords: `grace`, `signal`, `young`,
+`daly`, `cadence`, `atomic`, `rename`, `resume`, `skeleton`, `managed`, `skypilot`, `sagemaker`, `slurm`.
+
+## Table of contents
+
+1. [Preemption signals + grace windows (per platform)](#1-preemption-signals--grace-windows-per-platform)
+2. [Checkpoint cadence — the Young/Daly formula](#2-checkpoint-cadence--the-youngdaly-formula)
+3. [The atomic-write resume recipe](#3-the-atomic-write-resume-recipe)
+4. [Managed-spot frameworks move the box; the checkpoint-load restores the state](#4-managed-spot-frameworks-move-the-box-the-checkpoint-load-restores-the-state)
+5. [Python checkpoint/resume skeleton](#5-python-checkpointresume-skeleton)
+
+---
+
+## 1. Preemption signals + grace windows (per platform)
+
+The grace window dictates the design: it decides whether checkpoint-on-signal is even possible, or
+whether the timer is the *only* durability. **The window is NOT the safety net** — see the design-breaking
+gotcha below the table. Concrete per-platform reach/billing detail lives in each `profiles/<platform>.md`
+§4; this is the cross-platform signal map.
+
+| Platform | Detection signal | Grace window | Implication |
+|---|---|---|---|
+| **AWS EC2 Spot** | IMDS `http://169.254.169.254/latest/meta-data/spot/instance-action` (404 = none, 200 = pending); rebalance-recommendation fires ~10–20 min earlier | **~120 s** | On-signal flush of a *small* checkpoint is viable; still timer-checkpoint for the big one |
+| **GCP Spot** | metadata preemption flag + ACPI G2 Soft Off → shutdown script | **~30 s** default (configurable up to 120 s, Preview) | Timer-primary; on-signal flush only if checkpoint write < window |
+| **GCP Preemptible (legacy)** | same signal, **plus a hard 24 h cap** regardless of capacity | ~30 s **+ guillotined at 24 h** | Prefer Spot for long runs; Preemptible dies at 24 h even idle |
+| **Azure Spot** | IMDS Scheduled Events `/metadata/scheduledevents`, event type `Preempt` | **≥30 s** (Preempt is the short event; others give ≥5 min) | Timer-primary |
+| **Slurm preemption / walltime** | `SIGTERM` (then `SIGKILL`); with `#SBATCH --signal=B:SIGTERM@360` the batch step gets SIGTERM ~360 s before the kill | **SIGTERM → ~30 s** default; widen via `--signal` lead time | `--requeue` + an in-script SIGTERM trap to checkpoint, then resume on requeue |
+| **RunPod Spot** | OS **SIGTERM → SIGKILL** (also "interruptible without notice") | **~5 s** | Far too short to flush a large checkpoint — timer is the only real durability |
+| **vast.ai Interruptible** | **no signal** — bid-based; instance is *paused* (processes killed) the instant it is outbid | **~0 s (abrupt)** | Pure timer; assume cold restart + reload every time |
+
+URLs: AWS [spot-instance-termination-notices](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html),
+[rebalance-recommendations](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/rebalance-recommendations.html);
+GCP [preemptible](https://docs.cloud.google.com/compute/docs/instances/preemptible),
+[spot](https://docs.cloud.google.com/compute/docs/instances/spot);
+Azure [scheduled-events](https://learn.microsoft.com/en-us/azure/virtual-machines/windows/scheduled-events);
+Slurm [sbatch `--signal`](https://slurm.schedmd.com/sbatch.html);
+RunPod [spot-vs-on-demand](https://www.runpod.io/blog/spot-vs-on-demand-instances-runpod);
+vast.ai [Rental-Types](https://vast.ai/article/Rental-Types).
+
+**Gotcha — the design-breaking one.**
+Symptom: a "catch SIGTERM, flush the 40 GB checkpoint to durable storage" handler works in testing on AWS
+(120 s) but the job dies before the flush completes on RunPod (5 s) / vast.ai (0 s).
+Root cause: treating the grace window as the *primary* durability mechanism — it spans 2 min down to ~0
+across platforms, so any handler that needs more than a few seconds is a coin flip.
+Fix: checkpoint on a **periodic timer to durable storage** (§2); use the signal trap **only** as an
+opportunistic "save a final partial checkpoint if there is time" bonus, never as the safety net.
+
+**Gotcha — GCP Preemptible 24 h guillotine.**
+Symptom: a multi-day run on a Preemptible VM stops dead at 24 h even though nothing reclaimed it.
+Root cause: legacy Preemptible has a hard 24 h max runtime; Spot VMs have no cap.
+Fix: use **Spot, not Preemptible** for anything past a day (prefer Spot over legacy Preemptible for any run past a day).
+
+---
+
+## 2. Checkpoint cadence — the Young/Daly formula
+
+Cadence is a **formula, not a guess.** The optimal checkpoint interval that minimizes total wasted
+wall-clock (rollback re-compute after a kill **plus** checkpoint-write overhead) is the Young/Daly result:
+
+```
+W = sqrt(2 * mu * C)
+```
+
+- `mu` = mean time between preemptions (MTBF), in seconds.
+- `C`  = time to write one checkpoint to durable storage, in seconds.
+- `W`  = checkpoint interval (write a checkpoint every `W` seconds).
+
+**Worked example.** A checkpoint takes `C = 30 s` to write; the instance is preempted on average every
+`mu = 3 h = 10800 s`. Then:
+
+```
+W = sqrt(2 * 10800 * 30) = sqrt(648000) ≈ 805 s ≈ 13.4 min  →  checkpoint every ~13 min.
+```
+
+Higher preemption rate (smaller `mu`) → shorter interval. Slower checkpoint (larger `C`) → longer interval
+(each save costs more, so amortize it over more progress).
+
+**Round W DOWN to an iteration/epoch boundary.** Young/Daly assumes a checkpoint can be taken at *any*
+instant, but real iterative training can only snapshot at a step or epoch boundary. So convert `W` to an
+integer number of iterations and round *down*: at ~2 s/iteration, `805 s → 402 iters → checkpoint every
+400 iters`. Rounding down checkpoints slightly more often than optimal, which is the safe direction.
+
+**Distributed multiplier.** With `N` workers, one preemption wastes `N×` the compute (the whole group rolls
+back), so distributed jobs should checkpoint *more* frequently than the single-GPU `W` suggests.
+
+URLs: Young/Daly [robustness paper, INRIA](https://people.bordeaux.inria.fr/gaupy/ressources/pub/confs/icpp20_robustness.pdf),
+[Optimal Checkpointing Period, LAWN 281](https://www.netlib.org/lapack/lawnspdf/lawn281.pdf),
+[Optimal Checkpointing for Iterative Applications, IEEE](https://ieeexplore.ieee.org/document/9495174/).
+
+---
+
+## 3. The atomic-write resume recipe
+
+Two failure modes turn "I have checkpoints" into "my resume is broken": a **partial weight save** and a
+**corrupt-on-kill checkpoint**. The recipe fixes both.
+
+**Save FULL training state, not just model weights.** A resume that restores only weights silently
+restarts the epoch, reshuffles data, and degrades accuracy. The checkpoint must include:
+
+- model `state_dict`
+- optimizer `state_dict`
+- LR-scheduler `state_dict`
+- epoch **and** global step/iteration counter
+- RNG state (Python `random`, NumPy, `torch`, and CUDA)
+- dataloader position (sampler epoch / resumable-sampler offset)
+
+**Write atomically: tmp → fsync → os.replace.** A preemption mid-write corrupts the file, and a naive
+overwrite can leave zero good checkpoints. `os.replace` maps to the atomic POSIX `rename(2)` on the same
+filesystem (and, unlike `os.rename`, overwrites atomically on Windows too), so:
+
+1. Write the whole state to `latest.pt.tmp`.
+2. `fsync` the file (and the directory) so bytes hit disk before the rename.
+3. `os.replace("latest.pt.tmp", "latest.pt")` — the swap is all-or-nothing.
+4. Keep the previous `latest.pt` until the new one is committed; a kill at any point leaves one intact file.
+
+**Checkpoint to the platform's DURABLE location, not local scratch** (principle #4). A managed replacement
+node is *fresh* — anything not on a cloud bucket / network volume / shared FS is gone. On a marketplace box
+where local disk persists across a pause, still mirror to durable storage at intervals.
+
+**Load-latest UNCONDITIONALLY on startup.** Use the *same code path* for first launch (no checkpoint →
+start fresh) and every restart-after-preemption (checkpoint exists → resume). This is what makes the job
+idempotent: the **identical launch command** run any number of times converges to the same end state, which
+is exactly what makes principle #7's "retry the identical config" actually resume progress instead of
+restarting from zero.
+
+URLs: [Check-N-Run, arXiv](https://arxiv.org/pdf/2010.08679),
+[SkyPilot training-guide](https://docs.skypilot.co/en/latest/reference/training-guide.html),
+[SageMaker resume-from-checkpoint](https://docs.aws.amazon.com/sagemaker/latest/dg/model-checkpoints-resume.html).
+
+---
+
+## 4. Managed-spot frameworks move the box; the checkpoint-load restores the state
+
+Managed frameworks **auto-provision a replacement** on preemption — but they restart the **process from
+scratch**. The framework moves the box; the checkpoint-load written in §3/§5 is what restores progress.
+This is the single most-misunderstood point: the framework does **not** resume training on its own.
+
+- **SkyPilot Managed Jobs** — strongest cross-cloud recommendation (re-provisions in a different
+  region/cloud to chase capacity, then re-runs the task). Caveat: it auto-recovers **only**
+  preemption/hardware failures — a user-code non-zero exit is **not** auto-recovered.
+  [managed-jobs](https://docs.skypilot.co/en/latest/examples/managed-jobs.html).
+- **AWS SageMaker Managed Spot** — set `use_spot_instances=True` + `checkpoint_s3_uri`; SageMaker syncs the
+  checkpoint dir to S3 during training and copies it back on restart (up to ~90 % savings). Gotcha:
+  **`max_wait` must be greater than `max_run`** — `max_wait` covers wait-for-capacity *plus* run time
+  *plus* interruption gaps; set it too tight and the job is killed mid-resume.
+  [managed-spot docs](https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html).
+
+Universal multi-cloud auto-failover is **out of scope for this skill** — use SkyPilot/dstack for that, then
+return here to make the *code* resume-correct so their recovery actually lands on progress
+(`superpowers:verification-before-completion` gates the "it resumed" claim against a loaded checkpoint, not
+a log line). For the elastic / multi-node tier (torchrun `--max-restarts`, Elastic Horovod) see
+`references/multinode.md`; the same invariant holds — the framework restarts processes, the per-epoch
+snapshot restores state.
+
+---
+
+## 5. Python checkpoint/resume skeleton
+
+Read this for the algorithm; adapt into the training script. The shape is platform-agnostic — only
+`DURABLE_DIR` changes per profile (§8 SCRIPT OVERRIDES).
+
+```python
+import os, random, signal, time
+import numpy as np
+import torch
+
+DURABLE_DIR = os.environ["DURABLE_DIR"]   # profile-supplied bucket/FS/volume mount, NOT local scratch
+CKPT = os.path.join(DURABLE_DIR, "latest.pt")
+CKPT_EVERY_ITERS = 400                     # = round_down(Young/Daly W / sec_per_iter); see section 2
+
+def save_full_state(model, opt, sched, epoch, step):
+    """Atomic write: tmp -> fsync -> os.replace. A kill at any point leaves one intact file."""
+    state = {
+        "model": model.state_dict(),
+        "opt": opt.state_dict(),
+        "sched": sched.state_dict(),
+        "epoch": epoch, "step": step,        # resume the exact position, not the epoch start
+        "rng_python": random.getstate(),
+        "rng_numpy": np.random.get_state(),
+        "rng_torch": torch.get_rng_state(),
+        "rng_cuda": torch.cuda.get_rng_state_all(),
+    }
+    tmp = CKPT + ".tmp"
+    with open(tmp, "wb") as f:
+        torch.save(state, f)
+        f.flush()
+        os.fsync(f.fileno())                 # bytes hit disk BEFORE the rename
+    os.replace(tmp, CKPT)                     # POSIX-atomic swap; prev file valid until this returns
+
+def load_latest_if_any(model, opt, sched):
+    """Unconditional load-latest: identical command resumes OR starts fresh. Returns (epoch, step)."""
+    if not os.path.exists(CKPT):
+        return 0, 0                          # first run, no checkpoint -> start from scratch
+    s = torch.load(CKPT, map_location="cpu")
+    model.load_state_dict(s["model"])
+    opt.load_state_dict(s["opt"])
+    sched.load_state_dict(s["sched"])
+    random.setstate(s["rng_python"])
+    np.random.set_state(s["rng_numpy"])
+    torch.set_rng_state(s["rng_torch"])
+    torch.cuda.set_rng_state_all(s["rng_cuda"])
+    return s["epoch"], s["step"]             # caller skips dataloader to this position
+
+# --- opportunistic last-flush only; NOT the safety net (section 1) ---
+_preempted = {"flag": False}
+def _on_sigterm(signum, frame):
+    _preempted["flag"] = True                # set a flag; flush at the next safe boundary, do not block here
+signal.signal(signal.SIGTERM, _on_sigterm)
+
+def train(model, opt, sched, dataloader, total_epochs):
+    start_epoch, start_step = load_latest_if_any(model, opt, sched)
+    step = start_step
+    for epoch in range(start_epoch, total_epochs):
+        for batch in dataloader:             # a resumable sampler should fast-forward to start_step
+            # ... forward / backward / opt.step() / sched.step() ...
+            step += 1
+            if step % CKPT_EVERY_ITERS == 0 or _preempted["flag"]:
+                save_full_state(model, opt, sched, epoch, step)
+                if _preempted["flag"]:
+                    return                   # grace window may be ~0s; exit cleanly after the flush
+```
+
+Verify the resume path before trusting it: kill the process mid-epoch, relaunch the *identical* command,
+and confirm step/epoch/loss continue rather than reset (this is the `verifying-dl-experiments`
+reproducibility check, applied to preemption). Trust the **loaded** checkpoint, not the "resumed" log line
+(principle #3).

package/bundled-skills/remote-gpu-trainer/references/ssh_transport.md

@@ -0,0 +1,270 @@
+# SSH Transport — keys, keepalive, resumable copy, secrets-via-stdin
+
+Platform-agnostic SSH + file-transfer substrate for every `ssh-rental` profile (AutoDL, RunPod,
+vast.ai, Lambda, Paperspace, China, bare SSH). One-time config so subsequent commands are short and
+password-less, plus the copy/secret patterns that survive flaky networks and short rentals. Concrete
+hosts, ports, and credential locations are **profile facts** — this file owns the *mechanism*, the
+profile (`profiles/<platform>.md` §1/§3/§8) owns the *values*.
+
+To jump: `grep -in '<keyword>' references/ssh_transport.md` (e.g. `keepalive`, `rsync`, `stdin`, `crlf`).
+
+## Table of contents
+
+1. Key generation
+2. Push the public key to an instance
+3. `~/.ssh/config` alias + keepalive tuning
+4. Verify the alias
+5. Resumable copy — rsync vs scp, and WHY rsync
+6. Bulk per-dir download loop
+7. Move secrets via stdin — never inline a key, never on a durable FS
+8. CRLF — `.sh` authored on Windows breaks on Linux
+9. Two SSH flavors — proxied/basic SSH cannot `scp`
+10. Transport gotchas (Symptom → Root cause → Fix)
+
+---
+
+## 1. Key generation
+
+Skip if `~/.ssh/id_ed25519` already exists.
+
+```bash
+ssh-keygen -t ed25519 -C "<label>"
+# Save path: Enter for the default ~/.ssh/id_ed25519
+# Passphrase: optional (Enter for none, or set one + use ssh-agent)
+```
+
+`ed25519` is shorter and more secure than RSA; every rental platform accepts both. One local key is
+reused across all instances — generate once, push the **public** half (§2) to each box. The private
+half (`~/.ssh/id_ed25519`, no `.pub`) never leaves the local machine and **never** goes onto a rental,
+a shared FS, or a cloud agent (a cloud scheduler runs in an isolated sandbox with no access to it — and
+putting a private key there is a secret leak; see `references/monitoring_patterns.md`).
+
+## 2. Push the public key to an instance
+
+Copy the connection string from the platform's web console / API; it has the shape
+`ssh -p <PORT> root@connect.<region>.<provider>.com`. Push the public key once:
+
+```bash
+ssh-copy-id -p <PORT> root@connect.<region>.<provider>.com
+# enter the platform-provided password ONCE
+```
+
+If `ssh-copy-id` is absent (common on Windows-native shells), append the key manually:
+
+```bash
+cat ~/.ssh/id_ed25519.pub          # copy the entire line
+ssh -p <PORT> root@connect.<region>.<provider>.com
+# on the remote:
+mkdir -p ~/.ssh && chmod 700 ~/.ssh
+echo "<paste the public key line>" >> ~/.ssh/authorized_keys
+chmod 600 ~/.ssh/authorized_keys
+exit
+```
+
+Test: re-running the `ssh …` line should connect **without** a password prompt.
+
+## 3. `~/.ssh/config` alias + keepalive tuning
+
+One block per instance turns `ssh -p <PORT> root@connect.<region>.<provider>.com` into `ssh <alias>`,
+and folds in the keepalive options that keep long monitoring/transfer connections from dropping.
+
+```ssh-config
+Host proj-1
+    HostName connect.<region>.<provider>.com
+    Port <PORT>
+    User root
+    IdentityFile ~/.ssh/id_ed25519
+    ServerAliveInterval 60
+    ServerAliveCountMax 120
+    TCPKeepAlive yes
+    # LogLevel VERBOSE   # uncomment to debug a refused/hung connection
+
+Host proj-2
+    HostName connect.<region>.<provider>.com
+    Port <PORT>
+    User root
+    IdentityFile ~/.ssh/id_ed25519
+    ServerAliveInterval 60
+    ServerAliveCountMax 120
+```
+
+**Naming**: `<project>-<index>` (e.g. `proj-1`, `proj-2`) reads cleanly in a fan-out loop; avoid bare
+`gpu1`. **Why the three keepalive options**:
+
+- `ServerAliveInterval 60` — send an application-layer heartbeat every 60 s, so a NAT/idle timeout on
+  the path does not silently drop a parked connection (mid-`scp`, or an open monitor).
+- `ServerAliveCountMax 120` — tolerate up to 120 missed heartbeats before declaring the link dead (≈2 h
+  of network instability survived). Lower it (e.g. 3) for a *bounded* monitor that should self-kill on a
+  blip rather than hang — see the short-connection poll in `references/monitoring_patterns.md`.
+- `TCPKeepAlive yes` — let the OS also emit TCP-layer keepalives, catching a peer that vanishes
+  ungracefully.
+
+Ports change when a profile re-issues an instance (`ssh-rental` boxes assign a new port on
+re-creation) — update the `Port` line after each create/recreate, then re-run §4.
+
+## 4. Verify the alias
+
+```bash
+for a in proj-1 proj-2 proj-3 proj-4; do
+    echo "=== $a ==="
+    ssh -o ConnectTimeout=10 "$a" "hostname; date"
+done
+```
+
+Each should print a distinct hostname. Then the env probe (SKILL.md Phase 1):
+`ssh <alias> 'python -c "import torch;print(torch.cuda.is_available())"'`.
+
+## 5. Resumable copy — rsync vs scp, and WHY rsync
+
+`scp` opens **one** SSH stream for the whole transfer and **cannot resume**: any blip mid-copy aborts
+the entire run and a re-run starts from zero. `rsync` compares source/dest and ships only the delta, so
+a re-run after a drop **continues** instead of restarting — the single most important property on a
+metered box where a 130 GB pull can blip at minute 45.
+
+**Prefer `rsync` for anything large or multi-file:**
+
+```bash
+rsync -avz --partial --inplace --progress \
+    -e ssh \
+    <alias>:/root/autodl-tmp/checkpoints/ /path/to/local/checkpoints/
+```
+
+- `-a` archive (recurse + preserve perms/times/symlinks), `-v` verbose, `-z` compress on the wire.
+- `--partial` keeps a partially-transferred file on interruption so the next run resumes mid-file
+  (without it, rsync deletes the partial and re-sends from the start).
+- `--inplace` writes directly into the destination file (resume-friendly; avoids a full temp copy on a
+  tight local disk). Drop it if atomic-replace of an existing dest matters more than resumability.
+- Re-run the **identical** command after any failure — that *is* the resume (principle #7).
+
+Use plain `scp` only for a **single small** file (a config, one checkpoint < ~1 GB) where resume is
+moot. For a large *tree*, even `scp` users should fall back to the **per-dir loop** (§6) so one dir's
+failure doesn't lose the rest. If `rsync` is missing on the remote image, `apt-get install rsync` (when
+online) or use the §6 loop.
+
+> The bulk-download stall-retry ladder (HF/ModelScope mirror swaps, `timeout … && break` loops) is a
+> *download-from-the-internet* concern, not host↔host copy — that lives in `references/china-network.md`.
+
+## 6. Bulk per-dir download loop
+
+For a large directory tree (many run/checkpoint dirs), wrap each dir in its **own** SSH session so a
+single drop loses only that dir, and a re-run **skips already-complete dirs**:
+
+→ `scripts/download_loop.sh` (parameterize `LOCAL_TARGET`, `REMOTE_ALIAS`, `REMOTE_PATH`).
+
+Its shape, and why each piece matters:
+
+- **List once, copy per-dir** — each `scp -r <alias>:<remote>/$d ./` is an independent session; one
+  failure ≠ whole-transfer loss (the `scp` single-stream trap, §5).
+- **Size-threshold skip** — a dir already ≥ threshold counts as complete and is skipped; a partial dir
+  is removed and re-pulled. Re-running the whole script is therefore idempotent and resumable.
+- **Per-dir `ConnectTimeout` + the §3 keepalive flags** on every `scp` so a hung session self-kills
+  instead of blocking the loop.
+
+## 7. Move secrets via stdin — never inline a key, never on a durable FS
+
+Putting a credential **in a command** (`ssh host "echo 'KEY' > …"`, or `scp key.txt host:…`) leaks the
+value into shell history, agent transcripts, and hook logs. Putting it on a **shared /
+durable FS** is worse: the value persists for every co-tenant, and some platforms' upload classifiers
+*block or corrupt* a file matching a known key pattern — so a credential written to the cross-instance
+FS may silently never arrive. **Push credentials to each box's per-instance system disk, via stdin**, so
+the value flows file → pipe → file and appears in no command text or output:
+
+```bash
+# stream exactly one credential block — value never appears on a command line
+grep -A 2 "machine api.<provider>.com" ~/.netrc \
+  | ssh <alias> 'umask 077; cat > /root/.netrc && chmod 600 /root/.netrc'
+```
+
+```bash
+# or a single token, same principle (stdin in, file out, chmod 600)
+printf '%s\n' "$TOKEN_FROM_ENV" \
+  | ssh <alias> 'umask 077; cat > /root/.<service>_key && chmod 600 /root/.<service>_key'
+```
+
+Rules that make this safe:
+
+- **One block, not the whole file.** Stream a single `machine …` stanza, never the entire `~/.netrc` —
+  it carries unrelated machines' credentials, and security hooks (rightly) block copying the whole file.
+- **Reference, never echo.** Source the token from an env var (`$TOKEN_FROM_ENV`) or a keyring; never
+  paste the literal value into the command.
+- **Per-instance system disk, not the shared FS.** Write to `/root/.<service>_key` (volatile but
+  private), not the cross-instance durable mount. The wrapper reads it and exports the env var before
+  launch (e.g. `export WANDB_API_KEY=$(cat /root/.wandb_key)`).
+- **Verify by capability, not by echoing the value:**
+  `ssh <alias> 'python -c "import wandb; print(wandb.Api(timeout=20).default_entity)"'`.
+
+## 8. CRLF — `.sh` authored on Windows breaks on Linux
+
+Symptom → Root cause → Fix:
+
+- **Symptom**: a synced launcher does nothing (empty log); run by hand it errors `set: -: invalid
+  option`, `cd: /path\r: No such file or directory`, or `syntax error near unexpected token $'do\r'` —
+  every line "ends in `\r`".
+- **Root cause**: Windows `core.autocrlf=true` (or `git archive` exporting with the working-tree EOL)
+  writes `.sh` with CRLF; Linux `bash` treats the trailing `\r` as part of each token. (`.py` is
+  unaffected — Python's universal newlines tolerate CRLF; specifically `bash`/`.sh` breaks.)
+- **Fix**: add `.gitattributes` with `*.sh text eol=lf` so `git archive`/checkout always emits LF; as an
+  immediate on-box unblock, `sed -i 's/\r$//' scripts/*.sh`.
+
+Every shell script in `scripts/` ships LF and starts `#!/usr/bin/env bash` + `set -u`; keep that
+contract when authoring new ones. **Never** put an unquoted `|` inside a `grep` regex in a transport or
+poll script — the shell splits it into piped commands and the first reads stdin → hangs forever
+(`references/monitoring_patterns.md`).
+
+## 9. Two SSH flavors — proxied/basic SSH cannot `scp`
+
+Some `ssh-rental` platforms expose **two** SSH endpoints, and the difference dictates whether file
+transfer works at all:
+
+- **Direct TCP SSH** — a real TCP port to the container (the `connect.<region>.<provider>.com:<PORT>`
+  shape above). Full `scp`/`rsync`/`sftp` work. This is what every transfer in this file assumes.
+- **Proxied / "basic" SSH** — a relayed or web-terminal SSH (common on RunPod and vast.ai for the
+  default exposed endpoint). It carries an **interactive shell only**: `scp`/`rsync`/`sftp` fail (often
+  with `subsystem request failed` / a hung handshake) because the proxy doesn't forward the SFTP
+  subsystem.
+
+**Fix**: for any code/data/checkpoint transfer, use the **direct-TCP** endpoint — on RunPod expose a
+TCP port (the `ssh root@<ip> -p <PORT>` form, not the proxied `ssh <pod>@ssh.runpod.io` one); on vast.ai
+use the instance's direct SSH port. Each profile's §3 NETWORK names which endpoint is which and whether
+ports change on restart. If only proxied SSH is available, transfer out-of-band instead (push results to
+object storage / HF Hub from on-box and pull from there).
+
+## 10. Transport gotchas (Symptom → Root cause → Fix)
+
+Universal gotchas (disk-full, inode, OOM, silent sync) are **not** repeated here — see
+`references/gotchas_universal.md`. These are transport-specific.
+
+**T1 — SSH exits 255 / "Connection reset" right after a `pkill`/`kill`.**
+Symptom: `ssh <alias> 'pkill -9 -f src.train'` returns `Connection reset by peer`, exit 255. → Root
+cause: killing the process tree disrupts the PTY chain; the SSH client receives EOF and exits — and
+anything *after* the kill in that same one-liner never runs. → Fix: this is **normal**, not a failure.
+Re-ssh to verify (`ssh <alias> "pgrep -af src.train | head -1 || echo CLEAN"`). Split kill and relaunch
+into **two** ssh calls — never `pkill X; relaunch X` in one command, the relaunch is dropped with the
+session.
+
+**T2 — large `scp -r` drops with "Read from remote host … reset by peer" 30–60 min in.**
+Symptom: a 130 GB `scp -r` aborts mid-transfer; the local tree has only the first few dirs, the rest
+gone. → Root cause: one SSH stream for the whole transfer; any blip kills it and `scp` does not resume.
+→ Fix: use `rsync --partial` (§5) or the per-dir loop (§6) — each dir an independent session, re-run
+skips completed dirs.
+
+**T3 — `.sh` "ends in `\r`" after a Windows→Linux sync.**
+See §8 (`.gitattributes` `*.sh text eol=lf`; on-box `sed -i 's/\r$//'`).
+
+**T4 — a credential leaks into history / a shared FS, or its FS upload silently fails.**
+Symptom: a key pasted into an `ssh`/`scp` command lands in transcripts and hook logs; an scp of the key
+to the shared FS "succeeds" but the file is missing or corrupt. → Root cause: the value appeared in a
+command line; and some platforms' FS classifiers block/corrupt credential-shaped uploads. → Fix: §7 —
+stream one block via stdin to the per-instance disk, verify by capability not by echo.
+
+**T5 — `scp dest open "/root/x/": Failure` instantly.**
+Symptom: a (often parallel/background) `scp big.tar <alias>:/root/x/` fails at once because the
+destination dir doesn't exist — a sibling command meant to `mkdir` it ran later, or was blocked. → Root
+cause: the transfer assumed a directory a *different* command was supposed to create (a parallel-setup
+race). → Fix: make every transfer self-sufficient — create the dest in the same command:
+`ssh <alias> 'mkdir -p /root/x' && scp … || retry`. Never assume a sibling created the destination.
+
+**T6 — `Host key verification failed` after an instance is recreated.**
+Symptom: same `connect.<region>.<provider>.com` host, new host key, so SSH refuses. → Root cause: the
+recreated container presents a different host key on the reused hostname/port. → Fix:
+`ssh-keygen -R '[connect.<region>.<provider>.com]:<PORT>'`, then reconnect (re-accepts the new key).