fcloud-sdk 0.1.0__py3-none-any.whl

fcloud/SKILL.md

@@ -0,0 +1,1185 @@
+---
+name: fcloud
+description: >-
+  Use fcloud to provision GPU or CPU hosts, run remote commands and scripts,
+  manage persistent sessions, transfer files, and monitor long-running jobs from
+  agent workflows.
+---
+
+# fcloud — GPU compute from the command line
+
+fcloud provisions GPU and CPU hosts, runs code on them, and transfers files — all
+from the terminal. It works with any AI coding agent (Cursor, Claude Code, Codex,
+etc.) so your agent can provision hardware, run experiments, and pull results
+autonomously.
+
+Package: `fcloud` | CLI command: `fcloud` | Python 3.10+
+
+---
+
+## Setup
+
+```bash
+pip install fcloud-sdk
+fcloud set_token <api_key>  # one-time; saves to ~/.fcloud/token
+fcloud health               # verify connectivity
+```
+
+Token resolution order:
+
+1. `api_key=` parameter to `Client()`
+2. `FCLOUD_API_KEY` environment variable (legacy `FOOM_API_KEY` still accepted)
+3. Saved token in `~/.fcloud/token`
+4. `FCLOUD_API_KEY` from the nearest `.env` file
+
+(For a human at a terminal, plain `fcloud set_token` with no argument prompts
+with hidden input, keeping the key out of shell history.)
+
+---
+
+## Quick Start
+
+### Run a script on a GPU (throwaway session)
+
+```bash
+# Upload a script, run it on an L4, print output, tear down
+fcloud run train.py --sku gpu_1x_l4
+
+# Upload a directory, run a specific script, pass arguments
+fcloud run . --script train.py --sku gpu_1x_l4 -- --epochs 50 --lr 1e-4
+
+# Run a one-off command (no file upload)
+fcloud exec --sku gpu_1x_l4 nvidia-smi -L
+```
+
+### Choosing a SKU
+
+Request hardware by passing `--sku` to `exec`/`run`/`create`. SKUs follow the
+pattern `gpu_<count>x_<family>` or `cpu_<count>x_<family>`: `gpu_1x_l4`,
+`gpu_1x_a10`, `gpu_8x_h100`, `cpu_generic`, `cpu_2x_epyc`, etc. If a SKU has
+no available capacity, the session queues until a host is provisioned.
+
+---
+
+## CLI Reference
+
+### Global Flags
+
+
+| Flag     | Description                                                           |
+| -------- | --------------------------------------------------------------------- |
+| `--json` | Machine-readable JSON output to stdout. Status messages go to stderr. |
+
+
+Use `--json` in agent workflows so output can be parsed programmatically. Place
+`--json` before your remote command (e.g. `fcloud exec --json -- python app.py`):
+flags after the command — or after a `--` — belong to *your* program, so
+`fcloud exec -- python app.py --json` passes `--json` to `app.py`, not to fcloud.
+Unknown flags are rejected (e.g. a typo'd `--limitt`) rather than silently
+ignored.
+
+### Infrastructure
+
+
+| Command                | Description                               |
+| ---------------------- | ----------------------------------------- |
+| `fcloud health`          | Check API connectivity                    |
+| `fcloud set_token <key>` | Save API key                              |
+
+
+### Execution
+
+
+| Command                                                                       | Description             |
+| ----------------------------------------------------------------------------- | ----------------------- |
+| `fcloud exec [--volume NAME[:MOUNT]] [--sku SKU] [--on SID|SID] [--wait 30s] [--emit-pid FILE] <cmd...>` | Run a command on a host |
+| `fcloud run [--volume NAME[:MOUNT]] [--on SID|SID] <file|dir> [--sku SKU] [--script NAME] [-- args...]` | Upload + run on a host |
+| `fcloud upload [--on SID|SID] <local> [remote]` / `fcloud download [--on SID|SID] <remote> [local]` | Transfer files (session via `--on` or positional) |
+
+
+Without `--on`, `exec` and `run` create a throwaway session, do the work, and
+release it automatically — its filesystem is still saved and downloadable
+afterwards. For a true no-persistence run (cheaper; nothing syncs to
+storage), use a **job** — see the Jobs section. With `--on <session-id>` or a
+positional session ID alias, they attach to an existing session, run, and
+detach (session stays alive).
+
+Cached images complete in ~2s. Cold builds (first run on a host) take 30–120s.
+
+### Exec Semantics
+
+`fcloud exec` is optimized for agents and ML workloads. It starts commands as
+managed processes and waits for a foreground result for a configurable budget.
+
+**Working directory.** Commands run from `/workspace` by default — the same
+place `fcloud upload` puts your files. So an uploaded `train.py` runs as
+`python train.py` (relative path, no `cd` and no need to spell out
+`/workspace/train.py`), and commands that write to relative paths also land in
+`/workspace` and persist. Do **not** assume a different default (e.g. `/root`)
+or hardcode `/root/...` paths — that's the single most common cwd mistake. To
+run elsewhere for one command, `cd` inside it: `sh -c 'cd /some/dir && ...'`.
+This applies uniformly to `exec`, `run`, `spawn`, and `shell`.
+
+```bash
+# Default wait budget is 30s
+fcloud exec --on s-abc123 python3 /workspace/train.py
+
+# Wait longer before detaching
+fcloud exec --on s-abc123 --wait 10m python3 /workspace/train.py
+
+# Positional session alias for --on
+fcloud exec s-abc123 --wait 10m -- python3 /workspace/train.py
+
+# Alias for --wait, useful when an agent thinks in timeouts
+fcloud exec --on s-abc123 --timeout 10m python3 /workspace/train.py
+```
+
+If the command exits within `--wait`, `exec` behaves like a normal foreground
+command: stdout/stderr are printed and the CLI exits with the remote return code.
+
+If the command is still running when the wait budget expires, `exec` does **not**
+kill it. It detaches and prints a process ID:
+
+```bash
+Command is still running; use fcloud logs s-abc123 proc-456 --follow
+```
+
+In `--json` mode, a detached command returns `status: "running"`,
+`returncode: null`, plus `session_id` and `process_id`. Agents should treat that
+as success-with-handle, not as command failure, then block on it with
+`fcloud wait <session_id> <process_id>` (returns the real exit code + tail) or
+tail it with `fcloud logs --follow`.
+
+```json
+{
+  "stdout": "",
+  "stderr": "",
+  "returncode": null,
+  "status": "running",
+  "session_id": "s-abc123",
+  "process_id": "proc-456",
+  "message": "Command is still running; use fcloud logs s-abc123 proc-456 --follow"
+}
+```
+
+`exec` returns a bounded stdout payload by default so a noisy command cannot flood
+the caller. If the JSON response has `stdout_truncated: true`, do **not** rerun the
+job just to recover logs. Use the returned `session_id` and `process_id`:
+
+```bash
+# Fetch the durable combined process log captured by the host
+fcloud logs s-abc123 proc-456 --output all
+
+# Fetch a specific range or stream
+fcloud logs s-abc123 proc-456 --output head --output-bytes 128k
+fcloud logs s-abc123 proc-456 --stream stderr --output all
+```
+
+> The default stream is **combined** (stdout + stderr) — prefer it. Most ML
+> tooling (tqdm progress bars, HuggingFace/TRL logs) writes to **stderr**, so
+> `--stream stdout` on a training job often looks empty even though the job is
+> producing output. Reach for `--stream stdout`/`stderr` only when you
+> specifically need one split.
+
+**Liveness while polling a long job:** `fcloud logs <sid> <pid> --json` includes
+`last_write_ts` (when the durable log was last written) and `session_status`.
+A `status: "running"` process whose `last_write_ts` stops advancing across
+polls is stalled or on a lost host — no need to diff `output_total_bytes`
+between polls. A `status: "lost_host"` (with `process_alive: false`) means the
+session ended without the process's exit ever being recorded: the host was
+lost, the tail is final, don't keep polling.
+
+SDK equivalent:
+
+```python
+log = session.logs("proc-456", output_range="all")
+print(log.output)
+
+stderr = session.logs("proc-456", stream="stderr", output_range="all")
+```
+
+Persistent-session commands (`--on <sid>`) print a one-line `✓ ready` to
+stderr on every attach; fuller provision/setup narration appears only when
+the session actually has setup work to do. Terminal/problem statuses also
+print to stderr. Parse stdout (or use `--json`) — never stderr.
+
+#### Shell Parsing Rules
+
+Prefer passing commands as normal argv:
+
+```bash
+fcloud exec --on s-abc123 ls /workspace
+fcloud exec s-abc123 -- ls /workspace
+fcloud exec --on s-abc123 python3 /workspace/train.py --epochs 10
+```
+
+A single quoted string is treated as shell text (it runs via `bash -lc`),
+so both of these work and are equivalent:
+
+```bash
+fcloud exec --on s-abc123 'ls /workspace'
+fcloud exec --on s-abc123 'echo one two && echo piped | tr a-z A-Z'
+```
+
+When you need shell syntax (`;`, `&&`, pipes, redirects, variables), either pass a
+single quoted shell expression or be explicit with `sh -c`:
+
+```bash
+fcloud exec --on s-abc123 'cd /workspace; ls'
+fcloud exec --on s-abc123 -- sh -c 'sleep 45; echo done'
+```
+
+Remember that unquoted shell metacharacters are handled by the local shell before
+`fcloud` sees them:
+
+```bash
+# Runs "sleep 45" remotely, then "echo done" locally
+fcloud exec --on s-abc123 sleep 45; echo done
+```
+
+### Jobs (`fcloud job`) — ephemeral run-to-completion
+
+A job is a session **without a saved filesystem**: image + `--include` files
++ volumes in; exit code, logs, and volume writes out. The workspace is
+discarded at close — a job can never be resumed, `ls`'d, mounted, or
+downloaded after it ends (those return a clear error pointing at volumes and
+logs). In exchange, nothing syncs to storage while it runs, and close only
+has the volumes to flush.
+
+`job run` returns once the command has exited **and** its volumes are
+committed (it prints `committing volumes…` while that runs), so
+`fcloud volume files <name>` right after it shows the job's output. If the
+host is unreachable at close the commit is deferred to its teardown replay
+and the command says so — `--json` reports it as `volume_sync`.
+
+| Command | Description |
+|---|---|
+| `fcloud job run [flags] -- <cmd...>` | Run to completion on a fresh ephemeral session; exits with the command's exit code |
+| `fcloud job ls [--all] [--limit N]` | List jobs (`--all` includes finished ones) |
+| `fcloud job logs <job-id> [--follow]` | Durable output — live or after close |
+| `fcloud job wait <job-id>` | Block until the job ends |
+| `fcloud job kill <job-id>` | Stop now; workspace discarded, volume commits + logs survive |
+
+`job run` flags: `--sku SKU` · `--volume NAME[:MOUNT]` (repeatable — the
+durable output surface) · `--include PATH` (repeatable; uploaded before the
+command starts) · `--env K=V` · `--secret KEY[=VALUE]` · `--retries N` ·
+`--min-disk-gb N` · `--detach` (print the job id and return).
+
+Jobs are never checkpoint/migrated: a lost host (spot preemption) kills the
+run, and `--retries N` re-runs the command on a fresh session — so make job
+commands re-runnable. If a job is preempted, the system automatically
+attempts to requeue it — don't immediately retry/resubmit yourself; follow
+the same job id with `fcloud job wait`/`logs`. Write outputs to a volume:
+
+```bash
+fcloud volume create results
+fcloud job run --sku gpu_1x_l4 --volume results:/results -- \
+  python train.py --out /results/model.pt
+fcloud volume files results          # outputs survive the job
+fcloud volume download results model.pt   # pull them back locally
+```
+
+SDK mirror:
+
+```python
+job = project.job("python train.py --out /results/model.pt",
+                  sku="gpu_1x_l4",
+                  volume_mounts=[{"name": "results", "mount_path": "/results"}],
+                  include=["./src"], retries=1)
+log = job.wait()                   # ProcessLog: exit_code, output
+
+client.list_volume_files("results")           # [{path, size, ...}]
+data = client.read_volume_file("results", "model.pt")   # bytes, no session
+client.download_volume_file("results", "model.pt", "./model.pt")
+client.delete_volume("results")               # when you're done with it
+```
+
+Rule of thumb: a **session** when you'll iterate interactively or want the
+filesystem later; a **job** for batch/CI-style runs with declared outputs.
+
+### Persistent Sessions
+
+
+| Command                       | Description                                          |
+| ----------------------------- | ---------------------------------------------------- |
+| `fcloud create [--sku SKU]`     | Create a session (cold — $0, no host, until first used) |
+| `fcloud sessions [--all]`       | List sessions                                        |
+| `fcloud stop <session-id> [--wait]` | Stop a session now: halt GPU spend (files kept). `--wait` blocks until the workspace manifest is durable |
+
+
+`create` registers a session and returns a session ID without allocating a host —
+it is cold and costs nothing until you use it. The first `exec`/`run`/`upload`/
+`shell`/`ssh` brings it online automatically with its files intact. Use
+`--on <session-id>` with `exec`, `run`, `upload`, or `download` to interact with it
+repeatedly.
+
+Sessions resume transparently: there is **no `resume` command**. A session that was
+stopped or reaped while idle comes back online (on a fresh host, with its workspace
+restored from cloud storage) the moment you use it again. `fcloud stop` exists only to halt spend
+immediately — your files are preserved either way. (Ephemeral **job**
+sessions are the one exception: their workspace is never saved and they can
+never resume — see Jobs.)
+
+`fcloud stop` can return `stop deferred; host will retry teardown` — the stop
+stands, but the workspace manifest is not durable yet. Anything that reads
+that manifest (`fcloud ls`, `fcloud mount`, `fcloud map --from <sid>`) must wait
+for it: pass **`fcloud stop <sid> --wait`** (`--timeout SECONDS`, default 300,
+`0` waits forever) instead of polling `fcloud sessions` for `state=cold`
+yourself. It exits 1 if the wait expires; the stop is unaffected.
+
+**Keep a client attached until the host is up.** First use claims a host for the
+session; if no client stays attached while it comes online, the host is reclaimed
+~60s after it's claimed (the session goes back to cold and resumes on next use —
+no data loss, but you paid for a boot that did nothing). To run **N boxes at
+once**, keep one client attached per box for its whole run — e.g. N parallel
+`fcloud exec --on <sid> --wait <long>` (or `spawn` + `fcloud wait`) — rather than
+creating N sessions and attending to them one at a time, which lets the idle
+ones lose their hosts.
+
+### File Transfer
+
+| Command                                | Description                                 | Needs active session?           |
+| -------------------------------------- | ------------------------------------------- | ------------------------------- |
+| `fcloud upload <sid> <local> [remote]`   | Upload files to a session                   | Yes (live host receives them)   |
+| `fcloud download <sid> <remote> [local]` | Download a file from a session              | No — works on active **or** closed |
+| `fcloud ls <sid> [path]`                 | List a stopped session's workspace manifest | No                              |
+| `fcloud mount <sid> <mountpoint>`        | Mount a stopped session's files (read-only) | No                              |
+
+Only `upload` needs a live session. `download` uses the live host when the
+session is active, for the freshest bytes, and falls back to the saved cloud-storage
+manifest otherwise. `ls` and `mount` are post-hoc manifest views: use them after
+`fcloud stop`, not while the session is active. None of the post-close views
+work for ephemeral **jobs** (no manifest is ever saved) — a closed job's
+outputs live in its volumes (`fcloud volume files <name>`) and logs
+(`fcloud job logs <id>`).
+
+
+Uploaded files land in `/workspace/` on the host. Files **at or under 5 MiB**
+stream inline over the WebSocket; anything larger goes client → cloud storage
+→ host via a pre-signed URL minted by the fcloud API. `fcloud upload` prints which
+files took that second route (and names them again if it fails), so a broken
+large-file upload is attributable to the file, not to the command — one
+oversized results JSON in an otherwise tiny code tree is enough to put the
+batch on the pre-signed path. That path is a single PUT, so each file is
+capped at **5 GiB**; split larger files (e.g. `split -b 4G`) or stage them
+through a volume/import path instead. Downloads work the same way in
+reverse.
+
+**Client version.** Every request states the client's version, and the API
+rejects clients below the deployment's floor with HTTP 426 / `client_too_old`
+(`fcloud.ClientTooOldError`) naming the upgrade command. If you see that, run
+`pip install -U fcloud-sdk` — it is a client-age problem, never a credential one.
+
+### Browsing & mounting a session
+
+`fcloud ls` and `fcloud mount` read the session's file manifest, which the
+host writes **when the session is stopped**. So they show a session's files
+after a `fcloud stop` (the common "grab my outputs later" case); a session
+that has never been stopped lists empty.
+
+```bash
+fcloud ls s-abc123              # list files at the root
+fcloud ls s-abc123 data        # list files under data/
+fcloud mount s-abc123 ~/mnt     # browse in Finder/Explorer (read-only)
+```
+
+A stopped session's **durable logs** appear under a virtual `_logs/` directory
+next to its workspace files — the image build log, the session lifecycle log,
+and the per-process exec logs the host streamed to cloud storage:
+
+```bash
+fcloud ls s-abc123 _logs                  # build.log, session.log, exec-<pid>.log, ...
+fcloud download s-abc123 _logs/build.log  # read a log long after the session is gone
+```
+
+These are the same streams `fcloud logs <sid> <pid>` shows live; `_logs/` just
+makes them browsable and downloadable post-hoc, and they appear under `_logs/`
+in `fcloud mount` too.
+
+`mount` streams file bytes **directly from cloud storage** to your machine (the
+API only serves the listing + a redirect), so it stays fast for large
+files. It uses rclone's **NFS mount** (the OS NFS client), so it needs only
+rclone — **no macFUSE, no kernel extension, no recovery-mode approval**:
+
+```bash
+brew install rclone          # macOS
+# Linux: sudo apt-get install -y rclone   (or curl https://rclone.org/install.sh | sudo bash)
+```
+
+Then `fcloud mount <sid> <dir>` and open `<dir>` in Finder/Explorer. `<dir>`
+must be a path that does **not** exist yet — fcloud creates the mountpoint and
+deletes it on unmount, and refuses an existing folder so it can never touch
+your data. Note: on
+macOS the **Terminal** may be blocked from reading the mount (a privacy/TCC
+quirk) — browse it in **Finder**, or use `fcloud ls` from the CLI. `fcloud ls`
+needs no rclone at all (plain API call). Press Ctrl-C to unmount.
+
+### Volumes
+
+Volumes are named, manifest-backed folders for moving data between sessions.
+They are not EBS volumes and do not reserve separate block storage: file bytes
+stay in the shared content-addressed blob store, while the volume records the
+path → blob manifest.
+
+| Command                                                     | Description                                      |
+| ----------------------------------------------------------- | ------------------------------------------------ |
+| `fcloud volume create <name>`                                 | Create an empty volume                           |
+| `fcloud volume list`                                          | List your volumes                                |
+| `fcloud volume files <name> [path]`                           | List files in a volume                           |
+| `fcloud volume download <name> <vol-path> [local] [-r]`       | Fetch a file (or, with `-r`, a directory) locally |
+| `fcloud volume cat <name> <vol-path>`                         | Print a volume file to stdout                    |
+| `fcloud volume delete <name>`                                 | Delete a volume and its manifest                 |
+| `fcloud volume import <name> <sid> [session-path] [vol-path]` | Copy a folder from a stopped session into volume |
+
+Reads need no session and no host — `files` / `download` / `cat` serve the
+volume's committed manifest straight from the API, so this is how you
+get a job's outputs back:
+
+```bash
+fcloud volume files results            # what's in there
+fcloud volume cat results slice-0.json # small file to stdout
+fcloud volume download results slice-0.json
+fcloud volume download results / ./out -r   # whole volume into ./out
+```
+
+`delete` returns a conflict while any session still has the volume mounted;
+stop or detach that session first.
+
+Mount a volume into a session with repeatable `--volume` flags — on `exec`,
+`run`, `shell`, `spawn`, `job run` and `map`:
+
+```bash
+fcloud exec --volume checkpoints:/workspace/checkpoints --sku gpu_1x_l4 \
+  python train.py
+fcloud run --volume checkpoints . --script train.py --sku gpu_1x_l4
+```
+
+`--volume NAME` mounts at `/workspace/NAME`; `--volume NAME:/absolute/path`
+mounts at that absolute path inside the session rootfs.
+
+**Volumes are not block devices and are not mounted with the unix `mount`
+command.** There is no `/dev/*` node — don't probe for one with `mount`,
+`lsblk`, or `df` inside the session, and don't try to mount/umount/format
+anything. The host composes each volume into the session's root filesystem as a
+read-only content-addressed layer (file bytes fetched lazily from the blob
+store on first read) plus a writable overlay at the mount path. Inside the
+session a volume is just an ordinary directory: use normal file I/O. Writes
+land in the overlay and are committed back to the volume as a new version on
+detach or session close.
+
+Volumes are attached
+**after** the session is online, so a volume-mounted command queues and
+provisions a host like any other session, and `--volume` also works against an
+already-running session (`--on <sid>`). Mounted volumes are active in at most
+one session, and browsing/import/delete returns a conflict while a volume is
+attached.
+
+Declare volumes once in `fcloud.json` to skip `--volume` on every command (see
+[Project Config](#project-config-fcloudjson)):
+
+```json
+{ "sku": "gpu_1x_l4", "volumes": [{ "name": "checkpoints", "mount": "/workspace/checkpoints" }] }
+```
+
+The effective set is the `fcloud.json` volumes plus any `--volume` flags, keyed by
+name; a `--volume` for a name already in `fcloud.json` re-maps its mount path for
+that invocation (just like `--sku` overrides the config `sku`). The SDK
+re-attaches this set every time it brings the session online — including after a
+stop/resume on a fresh host — and attach is idempotent, so re-passing a volume
+already mounted is a safe no-op.
+
+`volume import` is a single batch metadata operation at folder scope; it does
+not issue per-file API calls and does not copy bytes when blobs already exist.
+The source session must be stopped/closed so the import reads a finalized
+manifest. Volume browsing/modification is blocked while that volume is attached
+to an active session.
+
+```bash
+fcloud volume create checkpoints
+fcloud stop s-abc123
+fcloud volume import checkpoints s-abc123 runs/run-42 run-42
+fcloud volume files checkpoints run-42
+```
+
+#### End-to-end: persist results into a volume across sessions
+
+A volume is the way to carry data from one session to another (or to a later
+run on a fresh host). Writes land in the volume on session close; reads see them
+on the next attach — no manual sync step.
+
+```bash
+# 1. Create the volume once.
+fcloud volume create runs
+
+# 2. Run a job that writes into the mounted volume. This queues and
+#    provisions a host even when none is idle (no --volume special-casing).
+fcloud exec --volume runs:/workspace/runs --sku gpu_1x_l4 \
+  -- sh -lc 'python train.py && cp model.pt /workspace/runs/model.pt'
+
+# 3. After the ephemeral session closes, the file is committed to the volume.
+fcloud volume files runs            # -> model.pt
+
+# 4. A later session mounts the same volume and sees the file.
+fcloud exec --volume runs:/workspace/runs --sku gpu_1x_l4 \
+  -- ls -l /workspace/runs/model.pt
+```
+
+Two things worth knowing:
+
+- **Mount onto an empty path.** Attaching a volume onto a directory that
+  already contains session files is rejected (`volume_mount_target_not_empty`)
+  rather than silently shadowing them. Mount at a fresh path like
+  `/workspace/runs`. The same applies to `fcloud run <dir> --volume`: a mount
+  at the upload destination (`/workspace/<dir>`), or below it where the
+  uploaded tree has content, is rejected before the session starts.
+- **Save without stopping.** Closing the session commits attached volumes
+  automatically — that is the common case. To checkpoint a volume mid-run while
+  the session keeps running, use `fcloud stop` and start again — stopping commits
+  the volume, and the session comes back on next use with the mount intact.
+  The commit lands asynchronously, up to ~30s after the session reports
+  closing — a `volume files`/`cat` in that window reads the previous version.
+
+### Background Processes (requires an active session)
+
+
+| Command                                          | Description                                     |
+| ------------------------------------------------ | ----------------------------------------------- |
+| `fcloud spawn [--on SID\|SID] [--volume NAME[:MOUNT]] [--emit-pid FILE] <cmd...>` | Start a background process |
+| `fcloud wait <sid> <pid> [--timeout DUR]`          | Block until a process exits; return its exit code + tail |
+| `fcloud logs <sid> <pid> [--follow] [--output MODE]` | Show output from a process                 |
+| `fcloud kill <sid> <pid>`                          | Kill a background or auto-detached exec process |
+
+`spawn --emit-pid FILE` writes `{"session_id": ..., "process_id": ...}` JSON —
+the same format `exec`/`run --emit-pid` produce — not a bare process id.
+
+Use `--follow` for live monitoring. Use `--output all` after a command has already
+run to fetch its durable log. `--stream stdout|stderr|combined` selects a stream for
+newer process logs; `combined` is the default and works with older logs.
+
+**Waiting on a job.** Prefer `fcloud wait <sid> <pid>` over hand-rolled
+`until fcloud logs ... | grep` loops. It blocks until the process actually
+terminates, prints its exit code and log tail, and **exits with the process's
+own exit code** (`75` if a `--timeout` elapses while it's still running). It
+reads durable logs, so it also works after the host is gone. This is the
+reliable way to detect a failed background job — a grep loop can miss failures
+whose text doesn't match the pattern, and can't tell "still starting" from
+"already died". In `--json` mode `fcloud logs <sid> <pid>` and `fcloud wait` both
+surface terminal `status` and `exit_code`, so a poller can check
+`status != "running"` instead of grepping log text.
+
+### SSH
+
+
+| Command                              | Description                                               |
+| ------------------------------------ | --------------------------------------------------------- |
+| `fcloud shell [--sku SKU] [--volume NAME[:MOUNT]]` | Start an interactive fcloud-native shell in a new session |
+| `fcloud shell [--on SID|SID] [--volume NAME[:MOUNT]] [cmd...]` | Attach an interactive PTY to an existing session |
+| `fcloud ssh <session-id>`              | SSH into a session                                        |
+| `fcloud tunnel <sid> [--port PORT]`    | SSH ProxyCommand tunnel                                   |
+
+
+`shell` is the fcloud-native PTY path. `fcloud shell` creates a temporary
+interactive session, while `fcloud shell <sid>` or `fcloud shell --on <sid>`
+attaches to a persistent session and leaves it alive on exit. `--volume` works
+the same as on `exec`: mounts ride the create for a new session and are
+attached before the prompt on an existing one. `ssh` uses the
+local OpenSSH client over `fcloud tunnel`; use it when you specifically want SSH
+tooling/config behavior.
+
+`fcloud tunnel <sid>` defaults to the session SSH server port (`2222`) so it can
+be used directly as an OpenSSH `ProxyCommand`. Pass `--port` only when tunneling
+to a different TCP service inside the session.
+
+`fcloud ssh` requires an SSH server inside the session image. If `sshd` is not
+already present, the CLI bootstraps it with `apt-get install openssh-server` on
+the first SSH connection, which can add tens of seconds. To avoid that cost,
+include `openssh-server` in the project image when SSH access matters:
+
+```json
+{
+  "image": {
+    "apt": ["openssh-server"]
+  }
+}
+```
+
+Repeat SSH connections to the same session reuse the already-running `sshd`.
+
+---
+
+## Sweeps (`fcloud map`) — fan one command out over N bindings
+
+A sweep runs your command N times with different arguments as a **durable
+server-side fan-out**: submit, disconnect, and the service drives every
+task through its own session — queueing for capacity, retrying failures, and
+recording exit codes + logs. No changes to your code; your script's CLI *is*
+the interface.
+
+```bash
+# Hyperparameter sweep (cartesian product: 3 × 2 = 6 tasks)
+fcloud map --name lr-sweep --sku gpu_1x_l4 \
+  -- python3 train.py --lr {lr} --bs {bs} ::: lr=1e-4,3e-4,1e-3 ::: bs=32,64
+
+# Datagen over indices (shell brace expansion feeds the group)
+fcloud map --name gen -- python3 gen.py --shard {} --of 500 ::: {0..499}
+
+# One task per line from a pipe (no ::: → stdin binds {})
+jq -r '.items[]' work.json | fcloud map --name batch -- python3 proc.py {}
+```
+
+**Supplying args (GNU-parallel style).** Everything after the command is
+parsed for `:::` markers: each one starts a **group** of values (space- or
+comma-separated), and each task runs the command with one combination of
+values substituted in. Four ways to feed values:
+
+| Spec | Tasks | Meaning |
+|---|---|---|
+| `cmd {} ::: a b c` | 3 | One task per value; `{}` (or `{1}`) is the value |
+| `cmd {1} {2} ::: a b ::: x y` | 2×2 = 4 | Multiple groups **cross** (cartesian product) |
+| `cmd {1} {2} ::: m1 m2 :::+ t1 t2` | 2 | `:::+` **zips** with the previous group (paired values like model+tokenizer; lengths must match) |
+| `producer \| fcloud map -- cmd {}` | one per line | No `:::` groups → each stdin line is one task |
+
+Placeholders: `{}` = the single group's value · `{1}` `{2}` = group by
+position · `{lr}` = named group (write it `::: lr=1e-4,3e-4`) · `{i}` = task
+index (0-based) · `{n}` = task count. A command with **no** placeholders gets
+each task's values appended at the end (`cmd ::: a b` runs `cmd a`, `cmd b`).
+Every task also gets env vars `FCLOUD_TASK_ID`, `FCLOUD_WORLD_SIZE`, `FCLOUD_JOB` —
+a script can read those to self-shard and skip templating entirely.
+`fcloud help map` prints this reference.
+
+**Code binding modes.** The sweep pins an immutable snapshot at submit, so
+task 400 runs the same bytes as task 0:
+- **Default (Mode 2):** the current dir (or `--code DIR`) is hashed and
+  uploaded once; each task session starts from it. Image comes from
+  `fcloud.json` or the SKU default.
+- **`--from <session-id>` (Mode 1):** fork the session you just verified —
+  each task starts from that session's synced workspace (deps installed,
+  weights downloaded) and inherits its image. The session must have synced a
+  manifest (a stopped session always has; a live one syncs periodically).
+
+**Outputs.** Write task-keyed files (`shard-{i}.jsonl`) into the task's own
+workspace and pull them all with **`fcloud sweep harvest <name> <remote-glob>
+<local-dir>`** — it resolves each task's session from the sweep and writes
+`<local-dir>/task-<index>/<path>`, so there is no session-id bookkeeping and
+no cross-task filename collisions. (`fcloud download <task-session> <path>` is
+still there for one file from one task.) Or write into a shared volume. Volume commits are per-path
+merges (last writer wins per file): parallel tasks writing DISTINCT files
+into one shared volume all land safely; concurrent writes to the SAME file
+keep only the last committer, so key every task's output by `{i}`. Commits
+ride each task session's close and can trail the task's "succeeded" state
+by ~30s — read the volume after commits settle.
+
+**Code.** With no `--from`/`--code`, map pins the current directory (upload
+skip rules apply) and prints the path/file-count/byte summary before
+uploading; it refuses to pin `$HOME` or a filesystem root implicitly.
+
+**Fan-out width.** `--max-parallel N` caps how many tasks run at once;
+without it the cap is **8**, and submitting more tasks than that runs them
+in waves. Submit says so out loud, and the cap is changeable on a live
+sweep: `fcloud sweep set <name> --max-parallel N`.
+
+**Canary.** Task 0 runs first and gates the rest: a broken sweep costs one
+task, not N. `fcloud map` blocks until the canary passes, then detaches (Ctrl-C
+detaches the watch — it does NOT cancel). `--no-canary` / `--no-wait` opt out.
+When one task takes minutes, `--canary-smoke 'python3 -c "import train"'`
+gates on a cheap check instead and task 0 then runs its real command with
+the rest — cheaper than `--no-canary`, which risks N bad tasks to save one.
+Failures don't consume the retry budget when caused by spot preemption — the
+task just requeues.
+
+**Waiting for capacity is not failure.** A task with no host yet keeps its
+queued session, charges no retry, and is reported as blocked:
+`fcloud sweep status` prints `blocked on capacity: N task(s), oldest 6m12s`.
+That is the fleet scaling up, not your sweep breaking — cancel if the wait
+is longer than the work is worth.
+
+**Tracking & recovery:**
+```bash
+fcloud sweeps                          # all sweeps: state + done/run/fail counts
+fcloud sweep status <name>             # counts + failures clustered by error, with exemplar task
+fcloud sweep status <name> --watch     # same, repainted until terminal (--interval S)
+fcloud sweep harvest <name> '*.json' ./out   # every task's outputs → out/task-<i>/…
+fcloud sweep logs <name> [--task N]    # durable output (defaults to the exemplar failure; falls back to earlier attempts)
+fcloud sweep set <name> --max-parallel N  # change the fan-out cap on a live sweep
+fcloud sweep retry <name> [--all]      # re-run only failed tasks (fix code first: resubmit is idempotent)
+fcloud sweep cancel <name> [--remaining]  # stop; --remaining keeps running tasks (partial success)
+fcloud sweep wait <name>               # block until terminal; exit 0 on success
+```
+`--webhook URL` (or account default) POSTs the status document on canary
+pass/fail and completion — agents should submit with `--no-wait --json` and
+wake on the webhook instead of polling.
+
+**The status JSON is a versioned public contract** (`schema_version: 1`),
+specified in `docs/sweep-status-schema.md`: `counts` is an open map keyed by
+task state, the task→session mapping is `tasks[].session_id`, sweep state is
+`job.state`, and per-session spend is under `cost`. Additive changes keep the
+version; a removal or rename bumps it. Do not reverse-engineer it — and
+prefer `fcloud sweep harvest` over re-deriving the task→session→download walk
+by hand.
+
+**SDK mirror** — `client.map(command, args=…)`. The *type* of `args` picks
+the combinator (no `:::` syntax in the SDK):
+
+| `args` | Tasks | CLI equivalent |
+|---|---|---|
+| `{"lr": [1e-4, 3e-4], "bs": [32, 64]}` | 4 — cartesian product, binds `{lr}`/`{bs}` | `::: lr=… ::: bs=…` |
+| `[{"model": "a", "tok": "x"}, {"model": "b", "tok": "y"}]` | 2 — one per row, pre-paired | `::: … :::+ …` |
+| `["f1.json", "f2.json"]` | 2 — one per item, binds `{}` | `::: f1.json f2.json` |
+| `500` or `range(500)` | 500 — same command N times; index via `{i}`/`FCLOUD_TASK_ID`, nothing appended | `::: {0..499}` |
+| `None` (default) | 1 | — |
+
+```python
+sweep = client.map("python3 train.py --lr {lr} --bs {bs}",
+                   args={"lr": [1e-4, 3e-4], "bs": [32, 64]},   # 4 tasks
+                   name="lr-sweep", sku="gpu_1x_l4", code_dir=".")
+sweep.wait_canary(); sweep.wait()  # or poll sweep.status() / .retry_failed()
+# from_session="s-…" instead of code_dir for Mode 1. Returns a SweepHandle.
+```
+
+---
+
+## Agent Workflow Patterns
+
+### Pattern 1: Quick one-off command
+
+The simplest pattern. Run a command, get the result, done.
+
+```bash
+fcloud exec --sku gpu_1x_l4 --json nvidia-smi -L
+# {"stdout": "GPU 0: NVIDIA L4 ...\n", "returncode": 0, "status": "exited", ...}
+```
+
+### Pattern 2: Run a training script
+
+Upload code, run it, get results. Session lifecycle is fully managed.
+
+```bash
+fcloud run train.py --sku gpu_1x_l4 --json -- --epochs 50
+# {"stdout": "...", "stderr": "", "returncode": 0, "duration_ms": 34521}
+```
+
+For a project directory with dependencies:
+
+```bash
+fcloud run . --script train.py --sku gpu_1x_l4 -- --batch-size 32
+```
+
+### Pattern 3: Persistent session (multi-step workflow)
+
+Create a session once, interact with it across multiple commands. The session's
+filesystem persists between invocations — no re-uploading. It comes online on the
+first use and resumes transparently if it was reaped while idle.
+
+```bash
+# Create (returns a session ID; cold, $0 until first used)
+fcloud create --sku gpu_1x_l4 --json
+# {"session_id": "s-abc123", "status": "closed", "sku": "gpu_1x_l4"}
+
+# Install dependencies (brings the session online automatically)
+fcloud exec --on s-abc123 pip install torch transformers
+
+# Upload training data
+fcloud upload s-abc123 ./data
+
+# Run training
+fcloud run train.py --on s-abc123 -- --epochs 50 --lr 1e-4
+
+# Same persistent-session target using the positional alias
+fcloud run s-abc123 train.py -- --epochs 50 --lr 1e-4
+
+# Check results
+fcloud exec --on s-abc123 cat /workspace/results.json
+
+# Download model weights
+fcloud download s-abc123 /workspace/model.pt ./model.pt
+
+# Halt spend now (files kept; resumes on next use)
+fcloud stop s-abc123
+```
+
+Each command connects in ~1s, runs, and detaches. The session's files persist
+across all invocations on the host's `/workspace/`.
+
+This is the recommended pattern for agents running multi-step experiments —
+cheaper and faster than creating a new session for every command.
+
+Use `--wait` for commands that are expected to take longer than 30s but should
+remain foreground from the agent's perspective. If the command detaches anyway,
+follow the returned `process_id` instead of retrying the same command.
+
+### Pattern 4: SDK session (programmatic multi-step)
+
+For multi-step workflows (install deps → upload data → train → download results),
+use the Python SDK directly. A session stays alive across all operations.
+
+```python
+from fcloud import Client, Image
+
+client = Client()
+image = Image.debian_slim().pip_install(["torch", "numpy"])
+project = client.project("my-experiment", image=image)
+
+with project.session(sku="gpu_1x_l4") as s:
+    # Install and verify
+    r = s.run(["python3", "-c", "import torch; print(torch.cuda.is_available())"])
+    print(r.stdout)  # True
+
+    # Upload training data
+    s.upload("./data", "data")
+
+    # Run training
+    r = s.run(["python3", "/workspace/data/train.py"], timeout_ms=3600_000)
+    print(r.stdout)
+
+    # Download results
+    weights = s.download("model.pt")
+    open("model.pt", "wb").write(weights)
+
+    # Session closes automatically on exit
+```
+
+### Pattern 5: Long-running job with monitoring
+
+For jobs that take minutes/hours, **lead with `spawn` + `wait`, not a maxed-out
+`exec --wait`.** `spawn` returns a durable `process_id` immediately; `wait`
+blocks on that id reading the durable log, so it survives host migration and
+never depends on a foreground stream staying alive:
+
+```bash
+# Start the job — prints a durable proc-id right away, no blocking.
+fcloud spawn s-abc123 python3 /workspace/train.py
+# Process proc-456 spawned in session s-abc123
+
+# Block until it exits (reads durable logs; survives migration). --timeout 0
+# waits indefinitely; a positive timeout exits 75 (still running) so you retry.
+# A process id the service has no record of (typo'd or stale) errors out
+# with "unknown process id" (exit 1) instead of telling you to keep waiting.
+fcloud wait s-abc123 proc-456 --timeout 0
+
+# Or watch it live:
+fcloud logs s-abc123 proc-456 --follow
+# Fetch the full log after it finishes (default stream is combined):
+fcloud logs s-abc123 proc-456 --output all
+# Stop it if needed:
+fcloud kill s-abc123 proc-456
+```
+
+**Don't max out `--wait` on a long job, and never pipe a live `exec` through
+`| tail`.** `exec --wait 60m ... | tail` buffers all output until the process
+ends (so you see nothing live) *and* swallows the detach line carrying the
+`process_id` — leaving you unable to call `fcloud wait` / `fcloud logs --follow` at
+all. If you must background a long `exec`, capture the id with `--emit-pid`
+(`spawn` takes the same flag and writes the same file), which writes
+`{session_id, process_id}` JSON to a file the instant the job starts —
+readable even when stdout/stderr are piped or backgrounded:
+
+```bash
+fcloud exec --on s-abc123 --emit-pid /tmp/job.json --wait 60m python3 train.py &
+# ... the file exists as soon as the process starts:
+PID=$(python3 -c 'import json;print(json.load(open("/tmp/job.json"))["process_id"])')
+fcloud wait s-abc123 "$PID" --timeout 0
+```
+
+> **Transient unreachability (`rc=75`).** A foreground `exec` can exit **`rc=75`**
+> (EX_TEMPFAIL) when the session is briefly unreachable — most often the first
+> exec right after a host boots, while the session is still binding to it, or a
+> host reclaimed mid-run (spot preemption) that fcloud checkpoint-restores onto a
+> new host. `exec` retries the attach for you and only reports `rc=75` once it
+> still can't reach the session; the message states the real cause ("not
+> reachable yet" vs. an ended session) instead of assuming a migration. Just
+> retry — the next attempt lands on the ready/restored host. `fcloud wait` /
+> `fcloud logs` ride a migration for you. After one, confirm the process is still
+> alive (`fcloud logs <sid>` lists it as `running`) rather than assuming `rc=75`
+> means your job is gone.
+>
+> **Migration rides honor `--wait`.** When a foreground `exec` rides a
+> checkpoint/restore, the ride is bounded by your `--wait` budget — it never
+> blocks past it (`--wait 0` disables the bound; the ride then uses the full
+> ~10-minute restore window). If the budget runs out while the restore is
+> still in flight, the result is **`status: "migrating"`** with **`rc=75`**:
+> the process is neither confirmed alive nor dead yet. JSON callers can use
+> the status to distinguish a ride-in-progress from a live process
+> (`status: "running"`, `rc=0`, which means the command itself outlived
+> `--wait`). Either way, follow up with `fcloud logs <sid> <pid> --follow` or
+> `fcloud wait`.
+
+With the SDK:
+
+```python
+from fcloud import Client
+
+client = Client()
+project = client.project("long-job")
+
+with project.session(sku="gpu_1x_l4") as s:
+    s.upload("./project")
+    proc = s.spawn(["python3", "/workspace/project/train.py"])
+
+    # Poll periodically
+    import time
+    while proc.status.value == "running":
+        proc.poll(lines=20)
+        print(proc.output[-200:])  # last 200 chars
+        time.sleep(60)
+
+    print(f"Done: exit_code={proc.exit_code}")
+    print(s.logs(proc, output_range="all").output)
+```
+
+### Pattern 6: Stop now and come back later (file persistence)
+
+Files written during a session are synced to durable cloud storage. After you
+stop it (or it is reaped while idle), the session comes back online automatically
+the next time you use it, with all files restored. There is no manual resume step.
+
+```bash
+# Create a session, do work
+fcloud create --sku gpu_1x_l4
+# Session s-abc123 created (cold — $0 until used)
+
+fcloud exec --on s-abc123 python3 train.py
+fcloud exec --on s-abc123 ls /workspace/
+# model.pt  metrics.json  train.py
+
+# Halt spend now (files sync to cloud storage)
+fcloud stop s-abc123
+
+# Hours later: just use it again — it comes back online automatically,
+# with files restored from cloud storage.
+fcloud exec --on s-abc123 ls /workspace/
+# model.pt  metrics.json  train.py  ← files restored from cloud storage
+```
+
+With the Python SDK:
+
+```python
+from fcloud import Client, Image
+
+client = Client()
+project = client.project("experiment", image=Image.debian_slim())
+
+# First run: train on L4
+with project.session(sku="gpu_1x_l4") as s:
+    s.run(["python3", "train.py"])
+    sid = s.session_id
+
+# Later: just attach and use it again — a cold session comes back
+# online automatically (rebuilt on a fresh host, files restored).
+s2 = client.attach_session(sid)
+s2.run(["ls", "/workspace/"])  # original files are here
+s2.run(["python3", "finetune.py"])
+s2.close()
+```
+
+**What survives a resume**: `/workspace` always survives. Changes *outside*
+`/workspace` (e.g. `pip install` into the image's site-packages) survive an
+ordinary same-image resume, but are **reset to the image** when the session
+resumes with a changed `fcloud.json` image spec (a re-baseline) or when the
+previous close could not finish syncing — the session prints an
+`environment reset` banner when that happens. For packages that must survive
+unconditionally, install them into a venv under the workspace:
+
+```bash
+fcloud exec --on s-abc123 'python3 -m venv /workspace/venv && /workspace/venv/bin/pip install torch'
+fcloud exec --on s-abc123 /workspace/venv/bin/python train.py
+```
+
+Better yet, put packages in the image spec (`pip_install([...])`) so every
+resume rebuilds them from the content-addressed image cache.
+
+---
+
+## Image Builder
+
+Images define the environment (base OS, packages, env vars). They're declarative
+and content-addressed — same layers = cache hit, instant startup.
+
+```python
+from fcloud import Image
+
+# GPU-ready image with PyTorch
+image = (
+    Image.debian_slim()
+    .apt_install(["git", "build-essential"])
+    .pip_install(["torch", "transformers", "numpy"])
+    .env({"PYTHONUNBUFFERED": "1"})
+)
+
+project = client.project("my-project", image=image)
+```
+
+Use `Image.from_registry(...)` when the workload needs a specific container base:
+
+```python
+from fcloud import Image
+
+# CUDA devel images include nvcc, ptxas, CUDA headers, and other build-time tools.
+image = (
+    Image.from_registry("nvidia/cuda:12.8.1-devel-ubuntu24.04")
+    .apt_install(["ninja-build"])
+    .pip_install(["torch", "triton", "numpy"])
+    .env({"PYTHONUNBUFFERED": "1"})
+)
+```
+
+Reach for a CUDA `devel` base when packages build CUDA code from source
+(`torch.utils.cpp_extension`, flash-attn, xformers, mamba, custom kernels). Runtime
+CUDA/driver availability is not enough for those workflows; they need the compiler
+toolchain inside the image.
+
+For GPU workloads, the base image must also provide a glibc new enough to load the
+host NVIDIA driver libraries injected by the runtime. This is distro-agnostic:
+Debian, Ubuntu, SUSE, Rocky, etc. can all work if their glibc satisfies the NVIDIA
+library requirement. Do not choose old GPU bases casually; failures usually show up
+as `GLIBC_2.xx not found`, CUDA initialization failures, or `nvidia-smi`/torch CUDA
+errors. Prefer a recent CUDA base, for example an NVIDIA CUDA image based on Ubuntu
+24.04, unless the user has a specific compatible base. For ordinary CPU/Python
+packages, `Image.debian_slim()` is usually faster to build and cache.
+
+For PyTorch GPU images, `fcloud` may route pip installs through a PyTorch wheel
+index. Do not put system build tools such as `ninja` in pip requirements just
+because a package build mentions them; install `ninja-build` with apt instead.
+If a workload mixes PyTorch wheels with ordinary PyPI-only packages, split them
+into separate image layers or configure the package indexes explicitly.
+
+Pre-built named images:
+
+- `Image.from_name("agent-gpu")` — the unified default for every `gpu_*` SKU:
+  `nvidia/cuda:12.8.1-devel-ubuntu24.04` (nvcc + CUDA headers, Python 3.12)
+  with a pinned training stack (torch 2.8.0+cu128, transformers, trl, peft,
+  accelerate, flash-attn prebuilt wheel). One image covers T4 through
+  Blackwell (sm75–sm120). Note: flash-attn *kernels* need sm80+ — on T4 the
+  import works but attention stays on sdpa, so avoid TRL
+  `packing`/`padding_free` there. vLLM is part of the pinned stack
+  (`import vllm` / `vllm serve` work out of the box; no extra installs).
+- `Image.from_name("agent-cpu")` — CPU-only ML stack
+- `Image.from_name("scientific")` — numpy, scipy, matplotlib, jupyter
+
+The default image for CLI commands (`exec`, `run`) is `debian_slim` (Python 3.11).
+
+---
+
+## Project Config (`fcloud.json`)
+
+Drop a `fcloud.json` in your project root to configure the image and default SKU.
+The CLI auto-discovers it by walking up from cwd (same as `.env`).
+
+```json
+{
+    "sku": "gpu_1x_l4",
+    "volumes": [{"name": "checkpoints", "mount": "/workspace/checkpoints"}],
+    "image": {
+        "base": "nvidia/cuda:12.8.1-devel-ubuntu24.04",
+        "apt": ["git", "build-essential", "ninja-build", "ffmpeg"],
+        "pip": ["torch", "triton", "numpy", "transformers"],
+        "env": {"PYTHONUNBUFFERED": "1"},
+        "run": ["echo setup-complete"]
+    }
+}
+```
+
+All fields are optional. Explicit `--sku` flags override the file; `--volume`
+flags merge with (and re-map by name) the `volumes` defaults.
+
+
+| Field        | Description                                |
+| ------------ | ------------------------------------------ |
+| `sku`        | Default hardware SKU for `exec` and `run`  |
+| `volumes`    | Volumes auto-attached to every `exec`/`run` session. Each entry is `"name"`, `"name:/mount"`, or `{"name": ..., "mount": ...}` |
+| `image.base` | Base image (default: `python:3.11-slim`)   |
+| `image.apt`  | Packages to `apt-get install`              |
+| `image.pip`  | Packages to `pip install`                  |
+| `image.env`  | Environment variables baked into the image |
+| `image.run`  | Shell commands to run during build         |
+
+
+When `fcloud.json` is present, `fcloud exec` and `fcloud run` automatically build the
+specified image. The image is cached by content hash — same config = instant startup.
+
+---
+
+## Key Concepts
+
+**SKU**: Hardware type identifier. Format: `gpu_<N>x_<family>` or `cpu_<N>x_<family>`.
+Examples: `gpu_1x_l4`, `gpu_8x_h100`, `cpu_2x_epyc`.
+
+**Session**: A persistent workspace (filesystem at `/workspace/`) that runs commands
+in a sandboxed container on a host. A session is a filesystem, not a held GPU:
+
+- *Throwaway*: created and released per command (`fcloud exec`, `fcloud run` without
+`--on`) — the filesystem is still saved and downloadable afterwards.
+- *Persistent*: created with `fcloud create`, interacted with via `--on <sid>`. It is
+cold ($0) until used, comes online automatically on use, and `fcloud stop` halts spend
+without deleting it.
+
+**Job**: An *ephemeral* run-to-completion session (`fcloud job run`): its workspace is
+never persisted, it cannot resume, and its post-close outputs are volume commits,
+the exit code, and logs. Cheaper (no storage sync); dies on preemption and re-runs
+(`--retries`) instead of migrating. See the Jobs section.
+
+Sessions resume transparently — a stopped or idle-reaped session is rebuilt on a fresh
+host (files restored from cloud storage) the next time you use it. You never run a resume command.
+
+**Warmth**: `fcloud sessions` reports a session's state — `hot` (running, spending),
+`warm` (idle host held, spending), `stopping` (finishing teardown), `cold`
+(stopped, $0, resumes on use), or `preparing` (coming online).
+
+**Image**: Declarative container spec. Layers are cached by content hash. First run
+on a host triggers a build (30–120s); subsequent runs with the same image are instant.
+
+**Project**: Groups sessions with a shared image definition. Created implicitly by
+CLI commands or explicitly via `client.project("name", image=...)`.
+
+---
+
+## Tips for Agents
+
+1. **Always use `--json`** when parsing output. Human-readable output may change format.
+2. **Use `--sku`** to target specific hardware. Without it, the scheduler picks any
+  available host.
+3. **Use `fcloud create` for multi-step work**. One persistent session is faster and
+  cheaper than creating a new ephemeral session for every command.
+4. **Treat `returncode: null` as detached, not failed**. Read `session_id` and
+  `process_id`, then call `fcloud logs <sid> <pid> --follow` or poll the process.
+  If stdout was truncated, call `fcloud logs <sid> <pid> --output all`.
+5. **Use `--wait` to express foreground patience**. Increase it for commands that
+  should finish soon; let truly long jobs detach and monitor them by process ID.
+6. **Pass argv normally; quote a single string (or use `sh -c`) for shell
+  expressions**. A single quoted argument runs as shell text via `bash -lc`.
+7. **Cached images are fast** (~2s). If your workflow creates sessions repeatedly with
+  the same image, it's cheap.
+8. **Files live in `/workspace/` — and it's your default working directory.**
+  Uploaded files are accessible as relative paths (`python train.py`) or as
+  `/workspace/<filename>` / `/workspace/<dirname>/...`. Commands already run
+  from `/workspace`, so don't `cd /root` or hardcode `/root/...` paths.
+9. **Pick a SKU explicitly** with `--sku`. If capacity is unavailable, the session
+  queues until a host is provisioned rather than failing.
+10. **Use `fcloud stop` to halt spend now**. Ephemeral sessions (`exec`, `run`
+  without `--on`) release automatically. Persistent sessions (`create`) keep their
+  files; they stop on their own when idle, so `stop` is only needed to halt spend
+  immediately.
+11. **Just use a session again to pick up where you left off**. Files persist across
+  stop/idle via cloud storage, and the session comes back online automatically (no resume
+  command). You can target different hardware with `--sku`.
+12. **Gracefully stop GPU engines when possible**. Prefer the engine's shutdown
+  method or `fcloud kill <sid> <pid>` over in-session `pkill -9`; if GPU memory
+  remains pinned after a hard kill, recover with `fcloud stop <sid>` and then use the
+  session again.
+
+## Anti-Patterns
+
+
+| Don't                                          | Why                                                                            |
+| ---------------------------------------------- | ------------------------------------------------------------------------------ |
+| Omit `--json` when parsing output              | Human format is for display, not parsing                                       |
+| Request unavailable SKUs                       | Session will queue until a host is provisioned; pick a SKU you expect to exist |
+| Retry after `returncode: null`                 | The original command is still running; follow its process ID                   |
+| Max out `exec --wait` on a long job            | Prefer `spawn` + `fcloud wait`; a huge `--wait` blocks the foreground and depends on the stream staying alive |
+| Pipe a live `exec` through `\| tail`           | Buffers until EOF (no live output) and swallows the detach line with the `process_id`. Use `--emit-pid FILE`, or `spawn` + `wait` |
+| Treat a migration `rc=75` as "job died"        | Spot preemption restores the process on a new host; retry the exec, or use `fcloud wait`/`logs` which ride the migration |
+| Immediately retry a preempted job              | The system automatically attempts to requeue it; resubmitting races the requeue and duplicates the run |
+| Put local shell syntax outside quotes          | The local shell handles it before `fcloud`; quote it or use `sh -c`              |
+| Use `pkill -9` as the first response to a hung GPU engine | Hard kills can bypass framework cleanup; try graceful shutdown or `fcloud kill`, then `fcloud stop` and reuse if needed |
+| Run commands without `--sku` in production     | May land on a CPU when you need a GPU                                          |
+| Write a job's outputs only to `/workspace`     | A job's workspace is discarded at close — write results to a `--volume` (or download while it runs); only volumes, logs, and the exit code survive |
+| Expect `fcloud ls`/`download`/resume on a closed job | Jobs never save a manifest; those refuse by design. Use `fcloud volume files <name>` and `fcloud job logs <id>` |