fcloud-sdk 0.1.2__tar.gz → 0.2.0__tar.gz

{fcloud_sdk-0.1.2 → fcloud_sdk-0.2.0}/CHANGELOG.md

@@ -4,6 +4,82 @@ All notable changes to the `fcloud` Python SDK and CLI are recorded here.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
 versions follow [Semantic Versioning](https://semver.org/).
 
+## [0.2.0] - 2026-09-08
+
+### Added
+- Per-session checkpoint/restore opt-out: `--no-checkpoint` on
+  `create`/`exec`/`run`/`shell` (SDK: `checkpoint=False` on
+  `Project.session`/`cold_session`, `Client.create_session`/
+  `create_cold_session`). A preempted opt-out session is rebuilt cold on any
+  available host (`/workspace` kept, running processes lost) instead of being
+  checkpointed and restored pinned to the region its checkpoint lives in.
+  Defaults resolve `--checkpoint`/`--no-checkpoint` → `FCLOUD_CHECKPOINT` →
+  `fcloud.json` `"checkpoint"` → the new `fcloud config set checkpoint on|off`
+  user default (`~/.fcloud/config.json`) → server default (on).
+  `SessionInfo.checkpoint` and `fcloud sessions --json` expose the policy; the
+  create fails loudly against a dispatcher too old to honour the opt-out.
+- `--no-resume` on `exec`/`run`/`shell`: refuse to wake a cold (stopped)
+  session instead of transparently re-provisioning its hardware. The refusal
+  exits 1 with a typed `session_cold` error naming the session and the SKU a
+  resume would have provisioned. SDK: `Client.attach_session(...,
+  resume=False)` raises the new `fcloud.SessionColdError`.
+- Pipelines: `fcloud map --after SWEEP[,SWEEP]` (SDK: `client.map(...,
+  after=handle_or_name, after_strict=…)`) holds a sweep in the new
+  `waiting` state until the named sweeps finish, then launches it — the
+  barrier is durable server-side, so a generate → train → eval chain
+  submits up front and the client can disconnect. An upstream failure
+  fails the downstream sweep (`upstream_failed:<name>`; re-arm with
+  `fcloud sweep retry <name> --all`); a partially-succeeded upstream
+  counts as finished unless `--after-strict`.
+- `--volume NAME[:MOUNT][:ro]` (also fcloud.json `"read_only": true` or
+  the `name:/mount:ro` string form) attaches a volume read-only: the
+  session sees it, but its writes are discarded at close (no commit).
+- `fcloud autoresearch init`: scaffolds a karpathy/autoresearch-style keep/discard
+  research loop (agent setup guide + `program.md` template) that runs experiments
+  on a persistent fcloud session.
+  The loop uploads `results.tsv` to the session each experiment; the dashboard's
+  session page then shows an Autoresearch tab (metric chart + results table).
+- Skip-the-line boost: `fcloud sweep boost <name>` / pressing `s` in
+  `fcloud sweep status --watch` (offered while blocked on capacity) moves a
+  sweep to the fast claim lane; queued tasks re-lane immediately
+  (`SweepHandle.boost()` in the SDK). Freed machines alternate fast/free.
+- `fcloud sweep status` explains WHY a sweep is waiting: a `why:` line under
+  `blocked on capacity` names the line (`3 session(s) ahead in the gpu_1x_l4
+  queue (1 fast-lane)`) and/or the platform (`a new machine is being
+  provisioned`, `no capacity ... (stockout)`, `cloud quota exhausted`, ...),
+  from the new `blocked.sku/queued_reason/queued_ahead/fast_ahead` status
+  fields; a `state: waiting` (pipeline) sweep names the upstream sweeps it
+  holds for. Older dispatchers omit the fields and status reads as before.
+- Async sweep scheduling: `fcloud map --not-before WHEN` (park until), `--by WHEN`
+  (deadline; opportunistic idle-fill until `deadline - --runtime - pad`, then
+  normal launch) and `--runtime DUR` (per-task estimate, default 1h).
+  `fcloud sweep status` shows the window phase; a missed deadline marks the
+  sweep `deadline_missed` (webhook `job.deadline_missed`) but never cancels it.
+- First-run login prompt: with no API key configured, the CLI offers to open
+  the dashboard login and store the key instead of failing with a bare error.
+- CLI errors carry typed error codes and the request id; the client never
+  provisions hardware for a request the server would reject.
+
+### Changed
+- Resuming a cold session now always prints one stderr line (`session <sid>
+  is cold — resuming (provisions <sku>); pass --no-resume to refuse`), even
+  when stderr is not a TTY or `FCLOUD_QUIET` is set — waking billed hardware
+  is a spend event, and non-TTY runs used to do it silently.
+- Failed map task attempts now retry with a growing delay (45s × attempt,
+  capped at 10m) instead of relaunching immediately; preemption requeues
+  remain immediate and still don't consume the retry budget.
+
+### Fixed
+- `fcloud map --volume name:/path` sent the custom mount path under a key
+  the server ignores, silently mounting the volume at its default path.
+  Custom mount paths now apply to sweep task sessions.
+
+## [0.1.3] - 2026-08-27
+
+### Changed
+- `fcloud setup` installs the agent skill for all agents by default (Enter or
+  non-interactive stdin = `all`); `--agents none` opts out.
+
 ## [0.1.2] - 2026-08-27
 
 ### Changed

{fcloud_sdk-0.1.2/src/fcloud_sdk.egg-info → fcloud_sdk-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fcloud-sdk
-Version: 0.1.2
+Version: 0.2.0
 Summary: Python SDK and CLI for the fcloud GPU compute platform
 Author: fcloud
 License-Expression: Apache-2.0
@@ -51,13 +51,15 @@ pip install fcloud-sdk
 
 Create an account on https://fcloud-home.vercel.app.
 Enter a credit card, and copy your API key.
+Or just run any `fcloud` command (or `fcloud login`): a fresh install
+walks you through sign-up and saves the key you paste.
 
 ```bash
 fcloud setup            # prompts for the key; optionally installs the agent skill
 fcloud health           # verify connectivity
 ```
 
-`fcloud setup --agents all` also installs the fcloud skill file for supported
+`fcloud setup` also installs (by default) the fcloud skill file for supported
 coding agents (Cursor, Claude Code, Codex) so an agent can drive fcloud for you.
 Use `--agents none` to skip that.
 
@@ -201,16 +203,18 @@ Other environment switches:
 | `FCLOUD_QUIET=1` | Suppress "still waiting" progress lines while a host is provisioned |
 | `FCLOUD_QUEUE_TIMEOUT=<seconds>` | How long to wait for capacity before giving up (default 1200) |
 | `FCLOUD_MIGRATE_RESTART=never` | Don't automatically re-run a command after a host rebuild (default `auto`) |
+| `FCLOUD_CHECKPOINT=off` | Default checkpoint/restore policy for new sessions. `off`: a preempted session rebuilds cold on any available host (`/workspace` kept, processes lost) instead of restoring pinned to its checkpoint's region. Per-session: `--checkpoint`/`--no-checkpoint`; per-user: `fcloud config set checkpoint off`; per-project: `fcloud.json` `"checkpoint": false` |
 | `FCLOUD_INSECURE_HTTP=1` | Allow a plaintext `http://` API URL to a non-loopback host (refused by default — the API key would travel unencrypted). Loopback URLs never need this |
 | `FCLOUD_TELEMETRY=0` | Disable all client telemetry. When enabled (the default), the client reports failures the backend cannot otherwise see — an uncaught CLI error, a queue-wait timeout, exhausted connect retries — as a fixed-allowlist payload (session id, event type, error class, truncated message, SKU/timing fields); never file contents, paths from OS errors, or credentials |
 
 A `fcloud.json` at the project root can set defaults (image build steps, default
-volumes). Note that fcloud will run the build steps it finds there, so treat a
+volumes, checkpoint policy); `fcloud config` stores per-user defaults in
+`~/.fcloud/config.json`. Note that fcloud will run the build steps it finds there, so treat a
 cloned repo's `fcloud.json` the way you would its Dockerfile.
 
 ## Agent skill
 
-`fcloud setup --agents all` installs `SKILL.md` for supported coding agents. It
+`fcloud setup` installs `SKILL.md` by default (`--agents none` to skip) for supported coding agents. It
 is the long-form, agent-oriented guide: workflow patterns, monitoring loops,
 and anti-patterns.
 

{fcloud_sdk-0.1.2 → fcloud_sdk-0.2.0}/README.md

@@ -17,13 +17,15 @@ pip install fcloud-sdk
 
 Create an account on https://fcloud-home.vercel.app.
 Enter a credit card, and copy your API key.
+Or just run any `fcloud` command (or `fcloud login`): a fresh install
+walks you through sign-up and saves the key you paste.
 
 ```bash
 fcloud setup            # prompts for the key; optionally installs the agent skill
 fcloud health           # verify connectivity
 ```
 
-`fcloud setup --agents all` also installs the fcloud skill file for supported
+`fcloud setup` also installs (by default) the fcloud skill file for supported
 coding agents (Cursor, Claude Code, Codex) so an agent can drive fcloud for you.
 Use `--agents none` to skip that.
 
@@ -167,16 +169,18 @@ Other environment switches:
 | `FCLOUD_QUIET=1` | Suppress "still waiting" progress lines while a host is provisioned |
 | `FCLOUD_QUEUE_TIMEOUT=<seconds>` | How long to wait for capacity before giving up (default 1200) |
 | `FCLOUD_MIGRATE_RESTART=never` | Don't automatically re-run a command after a host rebuild (default `auto`) |
+| `FCLOUD_CHECKPOINT=off` | Default checkpoint/restore policy for new sessions. `off`: a preempted session rebuilds cold on any available host (`/workspace` kept, processes lost) instead of restoring pinned to its checkpoint's region. Per-session: `--checkpoint`/`--no-checkpoint`; per-user: `fcloud config set checkpoint off`; per-project: `fcloud.json` `"checkpoint": false` |
 | `FCLOUD_INSECURE_HTTP=1` | Allow a plaintext `http://` API URL to a non-loopback host (refused by default — the API key would travel unencrypted). Loopback URLs never need this |
 | `FCLOUD_TELEMETRY=0` | Disable all client telemetry. When enabled (the default), the client reports failures the backend cannot otherwise see — an uncaught CLI error, a queue-wait timeout, exhausted connect retries — as a fixed-allowlist payload (session id, event type, error class, truncated message, SKU/timing fields); never file contents, paths from OS errors, or credentials |
 
 A `fcloud.json` at the project root can set defaults (image build steps, default
-volumes). Note that fcloud will run the build steps it finds there, so treat a
+volumes, checkpoint policy); `fcloud config` stores per-user defaults in
+`~/.fcloud/config.json`. Note that fcloud will run the build steps it finds there, so treat a
 cloned repo's `fcloud.json` the way you would its Dockerfile.
 
 ## Agent skill
 
-`fcloud setup --agents all` installs `SKILL.md` for supported coding agents. It
+`fcloud setup` installs `SKILL.md` by default (`--agents none` to skip) for supported coding agents. It
 is the long-form, agent-oriented guide: workflow patterns, monitoring loops,
 and anti-patterns.
 

{fcloud_sdk-0.1.2 → fcloud_sdk-0.2.0}/SKILL.md

@@ -336,6 +336,39 @@ once**, keep one client attached per box for its whole run — e.g. N parallel
 creating N sessions and attending to them one at a time, which lets the idle
 ones lose their hosts.
 
+### Checkpoint/restore on preemption (and how to turn it off)
+
+Sessions run on spot capacity. By default, when a host is reclaimed fcloud
+**checkpoints** the live sandbox (GPU state included) and **restores** it on a
+fresh box, so running processes continue. The restore is pinned to the region
+the checkpoint lives in (cross-region transfer of a multi-GB image is not free),
+so a restore can wait in `awaiting_restore_capacity` until that region has spot
+capacity again.
+
+If you would rather get **any** available host quickly and re-run your command
+yourself, turn checkpoint/restore off for the session. A preempted
+`--no-checkpoint` session is rebuilt cold on any host in any region:
+`/workspace` is preserved (synced before the box dies), running processes are
+lost, and the session carries `state_loss_reason=spot_preempt_checkpoint_disabled`
+in `fcloud sessions --json`. `fcloud run`/`exec` re-run the command after a
+rebuild (`FCLOUD_MIGRATE_RESTART`), so a resumable script (one that reloads its
+own checkpoints from `/workspace` or a volume) is the natural pairing.
+
+```bash
+fcloud run train.py --sku gpu_1x_h100 --no-checkpoint   # this session only
+fcloud create --sku gpu_1x_l4 --no-checkpoint          # for the session's life
+fcloud config set checkpoint off                       # my default, every project
+fcloud config get checkpoint                           # -> off
+fcloud config unset checkpoint                         # back to on
+```
+
+Precedence, strongest first: `--checkpoint`/`--no-checkpoint` (or SDK
+`checkpoint=True/False`) → `FCLOUD_CHECKPOINT=on|off` → `fcloud.json`
+`"checkpoint": false` → `fcloud config set checkpoint` → server default (on).
+The policy is fixed when the session is **created**; `--on <sid>` keeps the
+session's existing policy. Jobs (`fcloud job`, `fcloud map`) never checkpoint.
+`fcloud sessions` marks opted-out sessions `[no-checkpoint]`.
+
 ### File Transfer
 
 | Command                                | Description                                 | Needs active session?           |
@@ -470,9 +503,12 @@ detach or session close.
 Volumes are attached
 **after** the session is online, so a volume-mounted command queues and
 provisions a host like any other session, and `--volume` also works against an
-already-running session (`--on <sid>`). Mounted volumes are active in at most
-one session, and browsing/import/delete returns a conflict while a volume is
-attached.
+already-running session (`--on <sid>`). Any number of sessions may attach the
+same volume concurrently — commits merge per path, last writer wins per file
+(so key parallel writers' outputs by distinct paths). Only `delete` and
+`import` conflict while a volume is attached; browsing works any time. Append
+`:ro` (`--volume data:/vol:ro`) to attach read-only: the session sees the
+volume but its writes are discarded at close (no commit).
 
 Declare volumes once in `fcloud.json` to skip `--volume` on every command (see
 [Project Config](#project-config-fcloudjson)):
@@ -678,6 +714,18 @@ without it the cap is **8**, and submitting more tasks than that runs them
 in waves. Submit says so out loud, and the cap is changeable on a live
 sweep: `fcloud sweep set <name> --max-parallel N`.
 
+**Scheduling (async).** `--not-before WHEN` parks the sweep until then;
+`--by WHEN` sets a deadline: the platform runs tasks opportunistically on
+already-idle hosts (no capacity is provisioned early) and switches to
+normal launching at `deadline - --runtime - a provisioning pad`, so the
+sweep finishes by the deadline without paying for eager scale-up.
+`--runtime DUR` is the per-task runtime estimate (default 1h). WHEN is a
+duration from now (`30m`, `8h`, `2d`) or an ISO time (`2026-09-01T02:00`,
+local). A missed deadline never cancels the sweep — status shows
+`deadline_missed` and the webhook fires `job.deadline_missed`. While
+waiting, `fcloud sweep status` prints the window phase and when launches
+begin.
+
 **Canary.** Task 0 runs first and gates the rest: a broken sweep costs one
 task, not N. `fcloud map` blocks until the canary passes, then detaches (Ctrl-C
 detaches the watch — it does NOT cancel). `--no-canary` / `--no-wait` opt out.
@@ -687,11 +735,53 @@ the rest — cheaper than `--no-canary`, which risks N bad tasks to save one.
 Failures don't consume the retry budget when caused by spot preemption — the
 task just requeues.
 
+**Skip the line.** When tasks are blocked on capacity, `fcloud sweep boost
+<name>` (or pressing `s` in `sweep status --watch`, offered only while a
+line exists) moves the sweep to the fast claim lane: queued tasks skip
+ahead of free-lane work, and freed machines alternate fast/free so nobody
+is starved. SDK: `handle.boost()`. Unpriced for now.
+
 **Waiting for capacity is not failure.** A task with no host yet keeps its
 queued session, charges no retry, and is reported as blocked:
-`fcloud sweep status` prints `blocked on capacity: N task(s), oldest 6m12s`.
-That is the fleet scaling up, not your sweep breaking — cancel if the wait
-is longer than the work is worth.
+`fcloud sweep status` prints `blocked on capacity: N task(s), oldest 6m12s`
+followed by a `why:` line saying what the wait actually is — a line
+(`3 session(s) ahead in the gpu_1x_l4 queue (1 fast-lane)`), the platform
+(`a new machine is being provisioned`, `waiting for a busy machine to free
+up`), or bad news worth acting on (`no capacity ... (stockout) — consider
+another SKU`, `cloud quota exhausted`). Use it to decide: a line or a
+provision resolves itself (or `boost` past the free lane); a stockout or
+quota wall means pick another SKU or cancel. A `state: waiting` sweep is a
+pipeline hold, and status names the upstream sweeps it waits on.
+
+**Retry backoff.** A charged failure re-queues with a growing delay (45s ×
+attempt, capped at 10m) so a deterministic crasher doesn't burn its budget
+in seconds; preemption/lost-host requeues re-run immediately and never
+consume the budget.
+
+**Pipelines (`--after`).** `--after SWEEP[,SWEEP]` holds a sweep in
+`waiting` until the named sweeps finish, then launches it — a durable
+server-side barrier, so a whole generate → train → eval pipeline submits up
+front and the client can disconnect:
+
+```bash
+fcloud map --name gen --volume run42:/vol \
+  -- python3 gen.py --out /vol/data/shard-{i}.jsonl ::: {0..63}
+fcloud map --name train --sku gpu_8x_h100 --volume run42:/vol --after gen \
+  -- python3 train.py --data /vol/data --ckpt /vol/ckpt
+fcloud map --name eval --volume run42:/vol --after train \
+  -- python3 eval.py --ckpt /vol/ckpt --suite {} ::: mmlu,gsm8k,mbpp
+```
+
+Data flows between stages through the shared volume: map tasks write
+disjoint task-keyed paths, the next stage mounts the same volume and sees
+the union (the gate waits for the upstream's volume commits to land before
+releasing). An upstream failure fails the downstream sweep with
+`upstream_failed:<name>` instead of running it on missing inputs; after
+fixing and re-running the upstream, `fcloud sweep retry <name> --all`
+re-arms the downstream. A `succeeded_partial` upstream counts as finished
+unless `--after-strict`. Edges bind at submit, so upstreams must be
+submitted first (which also makes cycles impossible). SDK:
+`client.map(..., after=gen_handle)` (or a name, or a list; `after_strict=True`).
 
 **Tracking & recovery:**
 ```bash
@@ -699,15 +789,16 @@ fcloud sweeps                          # all sweeps: state + done/run/fail count
 fcloud sweep status <name>             # counts + failures clustered by error, with exemplar task
 fcloud sweep status <name> --watch     # same, repainted until terminal (--interval S)
 fcloud sweep harvest <name> '*.json' ./out   # every task's outputs → out/task-<i>/…
-fcloud sweep logs <name> [--task N]    # durable output (defaults to the exemplar failure; falls back to earlier attempts)
+fcloud sweep logs <name> [--task N] [--attempt K]  # durable output (defaults to the exemplar failure, falling back to earlier attempts; --attempt 1 = most recent attempt, explicitly)
 fcloud sweep set <name> --max-parallel N  # change the fan-out cap on a live sweep
-fcloud sweep retry <name> [--all]      # re-run only failed tasks (fix code first: resubmit is idempotent)
+fcloud sweep retry <name> [--all]      # re-run failed tasks (fix code first: resubmit is idempotent); --all adds canceled + WEDGED in-flight tasks (lost host / dead placement) and never restarts healthy running work
 fcloud sweep cancel <name> [--remaining]  # stop; --remaining keeps running tasks (partial success)
-fcloud sweep wait <name>               # block until terminal; exit 0 on success
+fcloud sweep wait <name> [--poll S] [--timeout S]  # block until terminal; exit 0 = success, 124 = still running at timeout
 ```
-`--webhook URL` (or account default) POSTs the status document on canary
-pass/fail and completion — agents should submit with `--no-wait --json` and
-wake on the webhook instead of polling.
+`--webhook URL` POSTs the status document on canary pass/fail and
+completion — agents should submit with `--no-wait --json` and wake on the
+webhook instead of polling. There is no account-default webhook: without
+`--webhook`, nothing is POSTed.
 
 **The status JSON is a versioned public contract** (`schema_version: 1`),
 specified in `docs/sweep-status-schema.md`: `counts` is an open map keyed by
@@ -1086,6 +1177,7 @@ flags merge with (and re-map by name) the `volumes` defaults.
 | ------------ | ------------------------------------------ |
 | `sku`        | Default hardware SKU for `exec` and `run`  |
 | `volumes`    | Volumes auto-attached to every `exec`/`run` session. Each entry is `"name"`, `"name:/mount"`, or `{"name": ..., "mount": ...}` |
+| `checkpoint` | `false` turns checkpoint/restore off for sessions created from this project (preemption rebuilds cold on any host). Default `true`. See *Checkpoint/restore on preemption* |
 | `image.base` | Base image (default: `python:3.11-slim`)   |
 | `image.apt`  | Packages to `apt-get install`              |
 | `image.pip`  | Packages to `pip install`                  |
@@ -1166,6 +1258,24 @@ CLI commands or explicitly via `client.project("name", image=...)`.
   remains pinned after a hard kill, recover with `fcloud stop <sid>` and then use the
   session again.
 
+## Autonomous research loops (autoresearch)
+
+To run an overnight keep/discard experiment loop on the user's code — after
+[karpathy/autoresearch](https://github.com/karpathy/autoresearch), with the GPU on
+fcloud instead of under the desk:
+
+```bash
+fcloud autoresearch init      # writes autoresearch/SETUP.md + program.template.md
+```
+
+Then read `autoresearch/SETUP.md` and follow it. It walks you through deciding the
+metric, the one mutable file vs the frozen eval, and the fixed time budget with the
+user; you write `program.md`, run the baseline on a persistent session, and hand
+off to the loop, which only ever re-uploads the mutable file and greps the metric.
+Progress is visible on the session's page in the fcloud dashboard (Autoresearch
+tab: metric chart + results table) once the loop's `results.tsv` upload syncs;
+point the user there when the loop starts.
+
 ## Anti-Patterns
 
 

{fcloud_sdk-0.1.2 → fcloud_sdk-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "fcloud-sdk"
-version = "0.1.2"
+version = "0.2.0"
 description = "Python SDK and CLI for the fcloud GPU compute platform"
 readme = "README.md"
 license = "Apache-2.0"
@@ -45,7 +45,7 @@ where = ["src"]
 # editable copy at the package root) so `fcloud setup` can find it via
 # importlib.resources under any install scheme, including `pip install --user`.
 [tool.setuptools.package-data]
-fcloud = ["py.typed", "SKILL.md"]
+fcloud = ["py.typed", "SKILL.md", "autoresearch/*.md"]
 
 [project.optional-dependencies]
 dev = [

{fcloud_sdk-0.1.2 → fcloud_sdk-0.2.0}/src/fcloud/SKILL.md

@@ -336,6 +336,39 @@ once**, keep one client attached per box for its whole run — e.g. N parallel
 creating N sessions and attending to them one at a time, which lets the idle
 ones lose their hosts.
 
+### Checkpoint/restore on preemption (and how to turn it off)
+
+Sessions run on spot capacity. By default, when a host is reclaimed fcloud
+**checkpoints** the live sandbox (GPU state included) and **restores** it on a
+fresh box, so running processes continue. The restore is pinned to the region
+the checkpoint lives in (cross-region transfer of a multi-GB image is not free),
+so a restore can wait in `awaiting_restore_capacity` until that region has spot
+capacity again.
+
+If you would rather get **any** available host quickly and re-run your command
+yourself, turn checkpoint/restore off for the session. A preempted
+`--no-checkpoint` session is rebuilt cold on any host in any region:
+`/workspace` is preserved (synced before the box dies), running processes are
+lost, and the session carries `state_loss_reason=spot_preempt_checkpoint_disabled`
+in `fcloud sessions --json`. `fcloud run`/`exec` re-run the command after a
+rebuild (`FCLOUD_MIGRATE_RESTART`), so a resumable script (one that reloads its
+own checkpoints from `/workspace` or a volume) is the natural pairing.
+
+```bash
+fcloud run train.py --sku gpu_1x_h100 --no-checkpoint   # this session only
+fcloud create --sku gpu_1x_l4 --no-checkpoint          # for the session's life
+fcloud config set checkpoint off                       # my default, every project
+fcloud config get checkpoint                           # -> off
+fcloud config unset checkpoint                         # back to on
+```
+
+Precedence, strongest first: `--checkpoint`/`--no-checkpoint` (or SDK
+`checkpoint=True/False`) → `FCLOUD_CHECKPOINT=on|off` → `fcloud.json`
+`"checkpoint": false` → `fcloud config set checkpoint` → server default (on).
+The policy is fixed when the session is **created**; `--on <sid>` keeps the
+session's existing policy. Jobs (`fcloud job`, `fcloud map`) never checkpoint.
+`fcloud sessions` marks opted-out sessions `[no-checkpoint]`.
+
 ### File Transfer
 
 | Command                                | Description                                 | Needs active session?           |
@@ -470,9 +503,12 @@ detach or session close.
 Volumes are attached
 **after** the session is online, so a volume-mounted command queues and
 provisions a host like any other session, and `--volume` also works against an
-already-running session (`--on <sid>`). Mounted volumes are active in at most
-one session, and browsing/import/delete returns a conflict while a volume is
-attached.
+already-running session (`--on <sid>`). Any number of sessions may attach the
+same volume concurrently — commits merge per path, last writer wins per file
+(so key parallel writers' outputs by distinct paths). Only `delete` and
+`import` conflict while a volume is attached; browsing works any time. Append
+`:ro` (`--volume data:/vol:ro`) to attach read-only: the session sees the
+volume but its writes are discarded at close (no commit).
 
 Declare volumes once in `fcloud.json` to skip `--volume` on every command (see
 [Project Config](#project-config-fcloudjson)):
@@ -678,6 +714,18 @@ without it the cap is **8**, and submitting more tasks than that runs them
 in waves. Submit says so out loud, and the cap is changeable on a live
 sweep: `fcloud sweep set <name> --max-parallel N`.
 
+**Scheduling (async).** `--not-before WHEN` parks the sweep until then;
+`--by WHEN` sets a deadline: the platform runs tasks opportunistically on
+already-idle hosts (no capacity is provisioned early) and switches to
+normal launching at `deadline - --runtime - a provisioning pad`, so the
+sweep finishes by the deadline without paying for eager scale-up.
+`--runtime DUR` is the per-task runtime estimate (default 1h). WHEN is a
+duration from now (`30m`, `8h`, `2d`) or an ISO time (`2026-09-01T02:00`,
+local). A missed deadline never cancels the sweep — status shows
+`deadline_missed` and the webhook fires `job.deadline_missed`. While
+waiting, `fcloud sweep status` prints the window phase and when launches
+begin.
+
 **Canary.** Task 0 runs first and gates the rest: a broken sweep costs one
 task, not N. `fcloud map` blocks until the canary passes, then detaches (Ctrl-C
 detaches the watch — it does NOT cancel). `--no-canary` / `--no-wait` opt out.
@@ -687,11 +735,53 @@ the rest — cheaper than `--no-canary`, which risks N bad tasks to save one.
 Failures don't consume the retry budget when caused by spot preemption — the
 task just requeues.
 
+**Skip the line.** When tasks are blocked on capacity, `fcloud sweep boost
+<name>` (or pressing `s` in `sweep status --watch`, offered only while a
+line exists) moves the sweep to the fast claim lane: queued tasks skip
+ahead of free-lane work, and freed machines alternate fast/free so nobody
+is starved. SDK: `handle.boost()`. Unpriced for now.
+
 **Waiting for capacity is not failure.** A task with no host yet keeps its
 queued session, charges no retry, and is reported as blocked:
-`fcloud sweep status` prints `blocked on capacity: N task(s), oldest 6m12s`.
-That is the fleet scaling up, not your sweep breaking — cancel if the wait
-is longer than the work is worth.
+`fcloud sweep status` prints `blocked on capacity: N task(s), oldest 6m12s`
+followed by a `why:` line saying what the wait actually is — a line
+(`3 session(s) ahead in the gpu_1x_l4 queue (1 fast-lane)`), the platform
+(`a new machine is being provisioned`, `waiting for a busy machine to free
+up`), or bad news worth acting on (`no capacity ... (stockout) — consider
+another SKU`, `cloud quota exhausted`). Use it to decide: a line or a
+provision resolves itself (or `boost` past the free lane); a stockout or
+quota wall means pick another SKU or cancel. A `state: waiting` sweep is a
+pipeline hold, and status names the upstream sweeps it waits on.
+
+**Retry backoff.** A charged failure re-queues with a growing delay (45s ×
+attempt, capped at 10m) so a deterministic crasher doesn't burn its budget
+in seconds; preemption/lost-host requeues re-run immediately and never
+consume the budget.
+
+**Pipelines (`--after`).** `--after SWEEP[,SWEEP]` holds a sweep in
+`waiting` until the named sweeps finish, then launches it — a durable
+server-side barrier, so a whole generate → train → eval pipeline submits up
+front and the client can disconnect:
+
+```bash
+fcloud map --name gen --volume run42:/vol \
+  -- python3 gen.py --out /vol/data/shard-{i}.jsonl ::: {0..63}
+fcloud map --name train --sku gpu_8x_h100 --volume run42:/vol --after gen \
+  -- python3 train.py --data /vol/data --ckpt /vol/ckpt
+fcloud map --name eval --volume run42:/vol --after train \
+  -- python3 eval.py --ckpt /vol/ckpt --suite {} ::: mmlu,gsm8k,mbpp
+```
+
+Data flows between stages through the shared volume: map tasks write
+disjoint task-keyed paths, the next stage mounts the same volume and sees
+the union (the gate waits for the upstream's volume commits to land before
+releasing). An upstream failure fails the downstream sweep with
+`upstream_failed:<name>` instead of running it on missing inputs; after
+fixing and re-running the upstream, `fcloud sweep retry <name> --all`
+re-arms the downstream. A `succeeded_partial` upstream counts as finished
+unless `--after-strict`. Edges bind at submit, so upstreams must be
+submitted first (which also makes cycles impossible). SDK:
+`client.map(..., after=gen_handle)` (or a name, or a list; `after_strict=True`).
 
 **Tracking & recovery:**
 ```bash
@@ -699,15 +789,16 @@ fcloud sweeps                          # all sweeps: state + done/run/fail count
 fcloud sweep status <name>             # counts + failures clustered by error, with exemplar task
 fcloud sweep status <name> --watch     # same, repainted until terminal (--interval S)
 fcloud sweep harvest <name> '*.json' ./out   # every task's outputs → out/task-<i>/…
-fcloud sweep logs <name> [--task N]    # durable output (defaults to the exemplar failure; falls back to earlier attempts)
+fcloud sweep logs <name> [--task N] [--attempt K]  # durable output (defaults to the exemplar failure, falling back to earlier attempts; --attempt 1 = most recent attempt, explicitly)
 fcloud sweep set <name> --max-parallel N  # change the fan-out cap on a live sweep
-fcloud sweep retry <name> [--all]      # re-run only failed tasks (fix code first: resubmit is idempotent)
+fcloud sweep retry <name> [--all]      # re-run failed tasks (fix code first: resubmit is idempotent); --all adds canceled + WEDGED in-flight tasks (lost host / dead placement) and never restarts healthy running work
 fcloud sweep cancel <name> [--remaining]  # stop; --remaining keeps running tasks (partial success)
-fcloud sweep wait <name>               # block until terminal; exit 0 on success
+fcloud sweep wait <name> [--poll S] [--timeout S]  # block until terminal; exit 0 = success, 124 = still running at timeout
 ```
-`--webhook URL` (or account default) POSTs the status document on canary
-pass/fail and completion — agents should submit with `--no-wait --json` and
-wake on the webhook instead of polling.
+`--webhook URL` POSTs the status document on canary pass/fail and
+completion — agents should submit with `--no-wait --json` and wake on the
+webhook instead of polling. There is no account-default webhook: without
+`--webhook`, nothing is POSTed.
 
 **The status JSON is a versioned public contract** (`schema_version: 1`),
 specified in `docs/sweep-status-schema.md`: `counts` is an open map keyed by
@@ -1086,6 +1177,7 @@ flags merge with (and re-map by name) the `volumes` defaults.
 | ------------ | ------------------------------------------ |
 | `sku`        | Default hardware SKU for `exec` and `run`  |
 | `volumes`    | Volumes auto-attached to every `exec`/`run` session. Each entry is `"name"`, `"name:/mount"`, or `{"name": ..., "mount": ...}` |
+| `checkpoint` | `false` turns checkpoint/restore off for sessions created from this project (preemption rebuilds cold on any host). Default `true`. See *Checkpoint/restore on preemption* |
 | `image.base` | Base image (default: `python:3.11-slim`)   |
 | `image.apt`  | Packages to `apt-get install`              |
 | `image.pip`  | Packages to `pip install`                  |
@@ -1166,6 +1258,24 @@ CLI commands or explicitly via `client.project("name", image=...)`.
   remains pinned after a hard kill, recover with `fcloud stop <sid>` and then use the
   session again.
 
+## Autonomous research loops (autoresearch)
+
+To run an overnight keep/discard experiment loop on the user's code — after
+[karpathy/autoresearch](https://github.com/karpathy/autoresearch), with the GPU on
+fcloud instead of under the desk:
+
+```bash
+fcloud autoresearch init      # writes autoresearch/SETUP.md + program.template.md
+```
+
+Then read `autoresearch/SETUP.md` and follow it. It walks you through deciding the
+metric, the one mutable file vs the frozen eval, and the fixed time budget with the
+user; you write `program.md`, run the baseline on a persistent session, and hand
+off to the loop, which only ever re-uploads the mutable file and greps the metric.
+Progress is visible on the session's page in the fcloud dashboard (Autoresearch
+tab: metric chart + results table) once the loop's `results.tsv` upload syncs;
+point the user there when the loop starts.
+
 ## Anti-Patterns
 
 

{fcloud_sdk-0.1.2 → fcloud_sdk-0.2.0}/src/fcloud/__init__.py

@@ -22,6 +22,7 @@ import fcloud._legacy_env  # noqa: F401 — must run before any env read below
 from fcloud.client import (
     Client,
     FcloudError,
+    NoApiKeyError,
     PaymentOverdueError,
     SessionNotFoundError,
 )
@@ -31,6 +32,7 @@ from fcloud.errors import (
     FcloudRuntimeError,
     FoomError,
     RateLimitedError,
+    SessionColdError,
     VolumeHydrationError,
 )
 from fcloud.image import Image
@@ -75,8 +77,10 @@ __all__ = [
     "FcloudError",
     "FcloudRuntimeError",
     "MapSpecError",
+    "NoApiKeyError",
     "PaymentOverdueError",
     "RateLimitedError",
+    "SessionColdError",
     "SessionNotFoundError",
     "VolumeConflictError",
     "VolumeHydrationError",

{fcloud_sdk-0.1.2 → fcloud_sdk-0.2.0}/src/fcloud/__main__.py

@@ -36,6 +36,7 @@ from fcloud.cli.common import (  # noqa: F401
     _shape_exec_command,
     _split_trailing_args,
 )
+from fcloud.cli.config_cmd import cmd_config  # noqa: F401
 from fcloud.cli.console import (  # noqa: F401
     _exec_result,
     _live_console_sink,