@microck/raft 0.2.0 → 0.2.1

package/README.md

@@ -76,6 +76,11 @@ see the task guides for [background jobs](docs/guides/background-jobs.mdx),
 [backup and recovery](docs/guides/backup-recovery.mdx)
 and [Docker](docs/guides/docker.mdx).
 
+See [project setup and templates](docs/guides/project-setup.mdx),
+[persistent services](docs/guides/long-running-services.mdx),
+[scripting and CI](docs/guides/scripting-ci.mdx), and
+[upgrades](docs/guides/upgrades.mdx) for complete operator workflows.
+
 stop the source box before taking a snapshot, restoring one or making a fork.
 stopping a box ends its jobs without saving process memory.
 

package/deploy/verify-features.py

@@ -157,6 +157,62 @@ def verify_archives():
     )
 
 
+def verify_readiness(location, expect_stale=False):
+    # Isolate the selected host without losing the controller's verified SSH config.
+    with tempfile.TemporaryDirectory(prefix="raft-doctor-test-") as work:
+        home = Path(work)
+        (home / ".ssh").symlink_to(Path.home() / ".ssh", target_is_directory=True)
+        config = home / ".config/raft/incus.json"
+        config.parent.mkdir(parents=True)
+        config.write_text(json.dumps({location: settings(location)}))
+        response = live.raft("doctor", check=False, env={**os.environ, "HOME": str(home)})
+    assert location + ": " in response.stdout, response
+    if expect_stale:
+        assert response.returncode == 1, response
+        assert (
+            f"raft: {location}: Host expiry worker differs from this CLI; "
+            "deploy the current Raft checkout first"
+        ) in response.stderr, response
+    else:
+        assert response.returncode == 0, response
+        assert "saved boxes; system containers (shared host kernel)" in response.stdout, response
+    print(f"{location}: doctor {'rejected stale worker' if expect_stale else 'passed'}", flush=True)
+
+
+def verify_transaction_stdin(location):
+    program = """import sys, tempfile
+from raft import transaction
+assert transaction(sys.argv[1], 'cat') == ''
+with tempfile.TemporaryFile() as output:
+    transaction(sys.argv[1], 'cat', stdout=output)
+    output.seek(0)
+    assert output.read() == b''
+"""
+    with subprocess.Popen(
+        [sys.executable, "-c", program, location],
+        cwd=ROOT,
+        stdin=subprocess.PIPE,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    ) as process:
+        try:
+            process.stdin.write(b"must-not-reach-the-host\n")
+            process.stdin.flush()
+            process.wait(timeout=30)
+        finally:
+            # Release even a regressed remote reader before leaving this test.
+            process.stdin.close()
+            try:
+                process.wait(timeout=30)
+            except subprocess.TimeoutExpired:
+                process.kill()
+                process.wait()
+                raise
+        assert process.returncode == 0, process.stderr.read().decode()
+    assert transaction(location, "printf lock-released") == "lock-released"
+    print(f"{location}: open stdin ignored and lifecycle lock released", flush=True)
+
+
 def verify(location, isolated_worker=False):
     if inventory(location):
         raise RuntimeError(
@@ -236,6 +292,8 @@ def verify(location, isolated_worker=False):
         return any(box["name"] == parse_handle(handle)[1] for box in inventory(location))
 
     try:
+        verify_readiness(location, expect_stale=isolated_worker)
+        verify_transaction_stdin(location)
         box = create("--memory", "1GiB")
         live.wait_running(box)
         _, name = parse_handle(box)

package/docs/contract.md

@@ -35,7 +35,7 @@ Raft manages persistent unprivileged native-architecture Incus system containers
 | Lifecycle defaults | Defaults to first configured location, 1 CPU, 2 GiB RAM, and 600-second lifetime. Supported choices are 1 or 2 CPUs, 1/2/4 GiB RAM, and 60 to 2,592,000 seconds (30 days) TTL. |
 | Admission limit | Hard ceiling of four saved boxes per host, including stopped boxes. |
 | TTL calculation | Running TTL begins after launch completes, using the host clock. Cold image preparation does not consume TTL. |
-| Host locking | Creation, lifecycle transitions, and expiry are serialized by host file lock (`/run/lock/raft-incus.lock`). Native Incus admin calls bypass CLI policy. |
+| Host locking | Creation, lifecycle transitions, and expiry are serialized by host file lock (`/run/lock/raft-incus.lock`). Noninteractive transactions close controller stdin so commands cannot wait for piped input while holding this lock. Exec, SSH, and file-transfer paths remain unchanged. Piped input to `raft exec` is unsupported. Native Incus admin calls bypass CLI policy. |
 
 ## Capacity and admission inspection
 
@@ -51,7 +51,7 @@ Raft manages persistent unprivileged native-architecture Incus system containers
 
 - **Stop and resume**: Stop terminates guest processes and preserves disk contents; stopped boxes retain disk but do not reserve guest RAM. Resume starts the container anew with an explicit TTL without restoring process memory.
 - **Lifetime extension**: `raft extend --ttl` resets a running box's deadline from current host time without restarting; stopped boxes require `raft resume`.
-- **Host worker contract**: Disposable creation/fork and retention cleanup require the current worker protocol. A mismatch fails before executing deletion policy or creating a disposable box; deploy the current checkout explicitly.
+- **Host worker contract**: `doctor`, disposable creation/fork, and retention cleanup require the current worker protocol. `doctor` checks it without running expiry or changing boxes, and reports the failing location. A mismatch fails before executing deletion policy or creating a disposable box; deploy the current checkout explicitly. Installing a new CLI does not update host workers.
 - **Disposable lifecycle**: `new --disposable` opts a box into deletion on stop or TTL expiry, including its snapshots. Clones and recovered boxes default to persistent; fork accepts an explicit `--disposable`.
 - **Retention cleanup**: `prune --older-than DAYS` previews persistent stopped boxes with a recorded stop timestamp. `--yes` deletes matching boxes under the host lifecycle lock. Running boxes and boxes without a recorded stop time are excluded. No automatic persistent-box deletion is enabled.
 - **Resizing**: Resume and fork accept optional supported CPU/RAM overrides, preserving omitted values. Resume validates stopped state under the lock before changing sizing.

package/docs/first-box.mdx

@@ -71,6 +71,9 @@ raft destroy "$box"
 
 ## Next steps
 
+- [Project setup and templates](./guides/project-setup.mdx): Prepare a project and reuse its dependencies.
+- [Lifetime and persistent services](./guides/long-running-services.mdx): Manage deadlines and restart applications after resume.
+- [Scripting and CI](./guides/scripting-ci.mdx): Parse CLI output and handle failures.
 - [Docker in workspaces](./guides/docker.mdx): Run nested containers and build images.
 - [File transfer](./guides/files.mdx): Upload and download files over SSH.
 - [Background jobs](./guides/background-jobs.mdx): Run detached commands under systemd.
@@ -78,5 +81,6 @@ raft destroy "$box"
 - [Snapshots and forks](./guides/snapshots-forks.mdx): Capture disk states and create same-host clones.
 - [Backup and recovery](./guides/backup-recovery.mdx): Export portable archives and restore across hosts.
 - [Troubleshooting](./guides/troubleshooting.mdx): Diagnose common errors and inspect host services.
+- [Upgrades and image changes](./guides/upgrades.mdx): Update the controller, hosts, and image pins.
 
 For temporary work, create with `--disposable`. Stopping or expiry then deletes the box and its snapshots. For persistent boxes, `raft prune --older-than 7` previews old stopped workspaces; add `--yes` only after reviewing the candidates.

package/docs/guides/background-jobs.mdx

@@ -90,3 +90,6 @@ raft extend "$box" --ttl 1800
 ```
 
 `raft extend` resets the container expiration deadline to `now + TTL` seconds using the host clock. It does not add time to the existing deadline.
+
+For deadline inspection and an enabled systemd service that starts again after
+resume or fork, see [Lifetime and persistent services](./long-running-services.mdx).

package/docs/guides/desktop-services.mdx

@@ -66,7 +66,10 @@ Start the desktop and establish a private tunnel:
 raft desktop "$box" --local 6080
 ```
 
-`raft desktop` starts `raft-desktop.service` inside the container if needed, waits for Xvfb and VNC readiness, and opens a tunnel to local port `6080`.
+`raft desktop` requests startup of `raft-desktop.service` inside the container if
+needed and opens a tunnel to local port `6080`. The service waits for Xvfb before
+launching its desktop clients. The CLI does not poll noVNC readiness; if the first
+browser request fails during startup, retry after the service initializes.
 
 Open the client in your browser:
 

package/docs/guides/long-running-services.mdx

@@ -0,0 +1,109 @@
+---
+title: Lifetime and persistent services
+description: Inspect deadlines, manage disposable and persistent boxes, and restart applications after resume.
+---
+
+# Lifetime and persistent services
+
+A box's lifetime controls when the host stops or deletes it. Activity, an open
+terminal, and detached jobs do not extend that deadline automatically.
+
+## Choose a lifetime and retention policy
+
+```sh title="CONTROLLER: Create a persistent box"
+box=$(raft new --location lab --ttl 3600)
+```
+
+TTL is in seconds: 60 to 2,592,000 (30 days), with a creation default of 600.
+The host sets the deadline after launch completes, using its own clock.
+Raft has no option to disable automatic expiry.
+
+| Operation | Persistent box (default) | Disposable box |
+| --- | --- | --- |
+| `stop` or TTL expiry | Stops processes; keeps disk and snapshots | Deletes the box and snapshots |
+| `resume --ttl SECONDS` | Starts the saved box with a new deadline | Only possible while the box still exists |
+| `destroy` | Deletes the box and snapshots | Deletes the box and snapshots |
+
+Use `raft new --disposable` for temporary work. A fork is persistent unless
+`--disposable` is explicitly supplied; recovered boxes are persistent.
+Back up work before stopping a disposable box.
+
+Persistent stopped boxes still occupy a saved slot and storage. Each host allows
+four saved boxes. `raft prune --older-than 7` previews old stopped boxes;
+`--yes` applies deletion. There is no automatic persistent-box retention cleanup.
+
+## Read and reset the deadline
+
+Select a running handle from `raft list`. These examples use controller `jq`:
+
+```sh title="CONTROLLER: Inspect and extend a running box"
+raft list
+box="lab:rf-a1b2c3d4e5f60718"  # Replace with your actual handle
+raft info "$box" | jq '{status, expires: .config["user.raft.expires"]}'
+raft extend "$box" --ttl 7200
+raft info "$box" | jq '{status, expires: .config["user.raft.expires"]}'
+```
+
+`user.raft.expires` is a Unix timestamp encoded as a string, not a remaining
+duration. `raft list` exposes it as `stopAt`; it may be empty on stopped boxes.
+`extend` sets `now + TTL`, rather than adding time to the old deadline.
+It requires a running box. Start a stopped box with `raft resume "$box" --ttl 7200`.
+
+The host timer checks every 15 seconds. Graceful shutdown, a host outage, or a
+long transaction holding the lifecycle lock can delay expiry. `raft gc --location lab`
+runs an immediate deadline check. A deadline is not a guarantee that a job will
+finish before shutdown; extend before starting work that needs more time.
+
+## Restart a service after resume
+
+One-off detached commands do not survive a stop. To start an application again
+after resume or fork, save a systemd unit on disk and enable it. This starts a new
+process; it does not recover process memory or an interrupted request.
+
+The following example serves `/workspace` as the guest `developer` user. Use a
+persistent running box and ensure files the application needs are readable by
+that user. For an actual application, replace `ExecStart` and the working directory.
+
+```sh title="CONTROLLER: Install and enable an example service"
+raft exec "$box" -- bash -lc 'cat > /etc/systemd/system/workspace-web.service <<"UNIT"
+[Unit]
+Description=Workspace HTTP example
+After=network.target
+
+[Service]
+User=developer
+WorkingDirectory=/workspace
+ExecStart=/usr/bin/python3 -m http.server 8080 --bind 0.0.0.0
+Restart=on-failure
+RestartSec=2
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+systemctl daemon-reload
+systemctl enable --now workspace-web.service'
+raft exec "$box" -- systemctl status workspace-web.service --no-pager
+raft exec "$box" -- journalctl -u workspace-web.service -n 30 --no-pager
+```
+
+Open `raft forward "$box" --remote 8080 --local 8080` and visit
+`http://127.0.0.1:8080`. The tunnel runs in the foreground; close it with `Ctrl+C`.
+
+```sh title="CONTROLLER: Check service startup after resume"
+raft stop "$box"
+raft resume "$box" --ttl 3600
+raft exec "$box" -- systemctl is-active workspace-web.service
+```
+
+Reopen the tunnel after resume. A fork or snapshot template copies the enabled
+unit too, so its new box starts its own service. Application secrets on disk are
+also copied; see [Project setup and templates](./project-setup.mdx).
+
+To disable the example, run:
+
+```sh
+raft exec "$box" -- systemctl disable --now workspace-web.service
+```
+
+See [Background jobs](./background-jobs.mdx) for one-off tasks and
+[Desktop and private services](./desktop-services.mdx) for tunnel behavior.

package/docs/guides/meta.json

@@ -3,12 +3,16 @@
   "collapsible": true,
   "defaultOpen": false,
   "pages": [
+    "project-setup",
     "docker",
     "files",
     "background-jobs",
+    "long-running-services",
+    "scripting-ci",
     "desktop-services",
     "snapshots-forks",
     "backup-recovery",
+    "upgrades",
     "troubleshooting"
   ]
 }

package/docs/guides/project-setup.mdx

@@ -0,0 +1,112 @@
+---
+title: Project setup and templates
+description: Upload or clone a project, run setup scripts, and create reusable same-host snapshot templates.
+---
+
+# Project setup and templates
+
+Raft starts a clean development image. Prepare a project with ordinary guest
+commands, then save a snapshot to reuse installed dependencies. There is no
+managed repository registry, environment registry, setup hook, or secret-injection
+service. Commands run as guest root in `/workspace` unless you choose another user.
+
+## Put a project in the box
+
+Create a persistent box after [host setup](../operations.mdx):
+
+```sh title="CONTROLLER: Create a project box"
+box=$(raft new --location lab --ttl 3600)
+```
+
+Upload a local project, excluding credentials and files you do not want copied.
+The example assumes `./my-project` contains your project and `setup.sh`:
+
+```sh title="CONTROLLER: Upload local project"
+raft upload "$box" ./my-project /workspace/my-project --recursive
+```
+
+The destination directory must not exist. For a public repository, an alternative
+is guest `git clone`; replace the example URL with your repository:
+
+```sh title="CONTROLLER: Clone instead of uploading"
+raft exec "$box" -- git clone https://github.com/OWNER/REPOSITORY.git /workspace/my-project
+```
+
+Choose one method. Private repositories require authentication configured inside
+the guest; Raft does not forward your controller's GitHub login or SSH agent.
+The clean image does not bundle coding agents or their credentials. Install and
+authenticate any agent inside the box using that tool's own instructions.
+
+## Run setup explicitly
+
+Use `bash -lc` for directory changes or shell syntax:
+
+```sh title="CONTROLLER: Run project setup"
+raft exec "$box" -- bash -lc 'cd /workspace/my-project && bash ./setup.sh'
+```
+
+Make setup scripts safe to rerun. Raft does not execute them automatically on
+create, resume, or fork. A failing synchronous command returns its exit code;
+see [Scripting and CI](./scripting-ci.mdx).
+
+For slow setup, use a detached job and inspect its result:
+
+```sh title="CONTROLLER: Run slow setup instead of synchronous setup"
+job=$(raft exec "$box" --detach -- bash -lc 'cd /workspace/my-project && bash ./setup.sh')
+raft status "$box" "$job"
+raft logs "$box" "$job"
+```
+
+Wait for successful completion before making a template. Extending the lifetime
+may be necessary; a detached job does not prevent expiry.
+
+## Supply application configuration
+
+For an application that reads a local `.env` file, upload it explicitly and set
+permissions for the application's user. This example assumes a controller file
+`./app.env` and an application running as `developer`:
+
+```sh title="CONTROLLER: Install a private application configuration file"
+raft upload "$box" ./app.env /workspace/my-project/.env
+raft exec "$box" -- chown developer:developer /workspace/my-project/.env
+raft exec "$box" -- chmod 600 /workspace/my-project/.env
+```
+
+The application must load that file; upload does not export environment variables.
+If running the whole project as `developer`, adjust project ownership separately.
+Do not place secrets in command arguments, committed files, or logs. Host journals
+can capture guest command lines. Raft does not rotate or scrub credentials.
+
+## Save and reuse a prepared template
+
+Finish setup, remove credentials that should not be inherited, and stop the box:
+
+```sh title="CONTROLLER: Save the prepared filesystem"
+raft stop "$box"
+raft snapshot "$box" prepared
+```
+
+Create another box from the snapshot:
+
+```sh title="CONTROLLER: Start a copy with explicit resources"
+copy=$(raft new --from "$box/prepared" --ttl 3600 --cpu 2 --memory 4GiB)
+raft exec "$copy" -- ls -la /workspace/my-project
+```
+
+The template stays on its source host. Its source box can be running when you
+copy the saved snapshot. `new --from` uses normal creation defaults when sizing
+is omitted: 1 CPU and 2 GiB RAM. The new box has its own handle, MAC address,
+deadline, and saved slot.
+
+The snapshot includes packages, project files, credentials, and enabled services.
+It does not include process memory. Changing the original after snapshot creation
+does not update the saved template. Keep its source box and snapshot until you
+no longer need to create copies; destroying the source deletes its snapshots.
+
+To copy the stopped box's current filesystem instead, use
+`raft fork "$box" --ttl 3600`. Fork inherits omitted CPU/RAM limits and does not
+copy the source snapshots. See [Snapshots and forks](./snapshots-forks.mdx).
+
+For a different host, use [Backup and recovery](./backup-recovery.mdx) with matching
+CPU architecture. For services that start on resume, see
+[Lifetime and persistent services](./long-running-services.mdx).

package/docs/guides/scripting-ci.mdx

@@ -0,0 +1,133 @@
+---
+title: Scripting and CI
+description: Parse CLI output, preserve command exit codes, wait for readiness, and handle interrupted operations.
+---
+
+# Scripting and CI
+
+Raft is a controller CLI using host SSH. A Linux CI runner needs Python 3.11.8+,
+OpenSSH, the installed CLI, host inventory, verified SSH host keys, batch
+authentication, and hosts with passwordless sudo. No Raft HTTP API, SDK, or API key
+exists. Give access only to trusted jobs: the configured SSH account has host sudo
+and controls all boxes on that host.
+
+## Output formats
+
+Raft does not have a global `--json` option. Parse each command's actual format:
+
+| Command | stdout |
+| --- | --- |
+| `new`, `fork`, `recover` | One location-qualified handle |
+| `exec --detach` | One `rfcmd-` job ID |
+| `list` | JSONL: one object per box, with `box`, `state`, `stopAt` |
+| `info` | One pretty-printed native Incus metadata JSON object |
+| `usage` | One JSON object containing running resource counters |
+| `limits --json` | One JSON object per location (JSONL across multiple hosts) |
+| `prune` | JSONL candidates or deleted records with `box`, `action` |
+| `status` | systemd `ActiveState`, `SubState`, `ExecMainStatus` property lines |
+| `logs` | Guest journal text |
+| `exec` | Payload stdout/stderr |
+| Other inspection/maintenance commands | Human-readable text or no output |
+
+`list` emits nothing when there are no boxes. `stopAt` is the host Unix deadline
+as a string, or an empty string when absent. `info` uses native Incus keys rather
+than a normalized Raft schema. `usage` includes `memoryBytes`, `memoryLimit`,
+`cpuTime`, `effectiveCpus`, `filesystemUsedBytes`, `filesystemAvailableBytes`, and
+`filesystemScope`; disk values describe the shared pool, not an individual box.
+See [Capacity limits](../reference/capacity-limits.mdx) for capacity calculations.
+
+These examples require `jq` on the controller:
+
+```sh title="CONTROLLER: Parse records without mixing stderr"
+raft list --location lab | jq -r 'select(.state == "Running") | .box'
+raft limits --location lab --json | jq '{location, savedSlots, warnings}'
+raft prune --location lab --older-than 7 | jq -r '.box'
+```
+
+Do not redirect `2>&1` into machine-readable output. Recovery reports its allocated
+handle on stderr before import finishes; only a successful completion prints the
+final handle on stdout.
+
+## Exit codes and job results
+
+| Situation | Exit code |
+| --- | --- |
+| Successful control operation | `0` |
+| Synchronous `exec` | Transport/command return code; preserves payload exit status when transport succeeds |
+| Recognized runtime/configuration failure | `1`, with `raft: ...` on stderr |
+| Argument parsing failure | `2`, with argparse diagnostics on stderr |
+| Caught keyboard interruption | `130` |
+
+SSH transport failures may return SSH's status rather than the payload's result.
+The foreground tunnel replaces the CLI process with SSH and uses SSH's exit code.
+A zero exit from `status` means the query succeeded; read `ExecMainStatus` for the
+job result and `ActiveState`/`SubState` for whether it finished. Do not interpret a
+running job's initial `ExecMainStatus=0` as completion. See
+[Background jobs](./background-jobs.mdx).
+
+## Create, wait, run, and stop
+
+`new` returns after Incus starts the box, before every guest service is necessarily
+ready. Probe a harmless command before running setup. This Bash example waits up
+to about a minute plus transport time, executes an uploaded project check, and
+attempts to stop the persistent box on exit. It assumes `./my-project/check.sh`
+exists and the configured host has a free saved slot:
+
+```bash title="CONTROLLER or trusted Linux CI runner"
+set -euo pipefail
+box=""
+cleanup() {
+  result=$?
+  trap - EXIT
+  if [ -n "$box" ]; then
+    if ! raft stop "$box"; then
+      echo "Cleanup failed; inspect $box" >&2
+      [ "$result" -ne 0 ] || result=1
+    fi
+  fi
+  exit "$result"
+}
+trap cleanup EXIT
+
+box=$(raft new --location lab --ttl 3600)
+printf 'Created %s\n' "$box" >&2
+ready=false
+for attempt in {1..30}; do
+  if raft exec "$box" -- true; then
+    ready=true
+    break
+  fi
+  sleep 2
+done
+if [ "$ready" != true ]; then
+  echo "Guest did not become command-ready: $box" >&2
+  exit 1
+fi
+
+raft upload "$box" ./my-project /workspace/my-project --recursive
+raft exec "$box" -- bash -lc 'cd /workspace/my-project && bash ./check.sh'
+```
+
+The stopped box remains available for inspection and still consumes a saved slot.
+Destroy it after collecting needed files, or preview old stopped boxes with prune.
+For disposable CI boxes, deletion on stop/expiry removes artifacts and snapshots;
+download results before cleanup. A new box or a retry still consumes a saved slot.
+
+Command readiness does not mean Docker or a web application is ready. Probe the
+specific dependency (`docker info`, an HTTP health endpoint, or systemd status)
+with its own bounded wait. Extend the deadline before work exceeding the current
+lifetime. This example's retries are only for the harmless readiness probe.
+
+## Interrupted operations and retries
+
+Raft does not automatically retry commands or provide idempotency keys. If SSH
+drops during creation, resume, import, or execution, the remote operation may have
+already happened. Inspect `raft list`, `raft info`, guest logs, and the host before
+repeating an operation that creates boxes or changes data. A failed create can
+leave a saved box even when no handle reached stdout.
+
+For an interrupted recovery, inspect the handle printed on stderr and follow
+[Host crash during recovery](./troubleshooting.mdx#host-crash-during-recovery).
+For uncertain command execution, check its effects before rerunning it; use
+idempotent setup scripts where possible. Do not treat a lost connection as proof
+of deletion or of a command not having run.

package/docs/guides/troubleshooting.mdx

@@ -19,15 +19,36 @@ raft doctor
 
 `raft doctor` connects to each host over SSH and reports:
 - Incus version running in project `raft`
+- Compatibility of the installed expiry worker with the CLI
 - Active status of host systemd services (`incus`, `raft-expire.timer`, `raft-network.service`)
 - Host root filesystem disk usage (`df -h /`)
 - Status and usage of the `raft-data` Btrfs pool
 - Count of existing saved containers
 
 <Callout type="info" title="Doctor scope">
-`raft doctor` reports runtime daemon and storage status. It does not validate image fingerprints, inspect nftables bridge rules, enforce admission caps, or emit 5 GiB low-disk warnings.
+`raft doctor` reports runtime daemon, expiry-worker compatibility, and storage status. It does not validate image fingerprints, inspect nftables bridge rules, enforce admission caps, or emit 5 GiB low-disk warnings.
 </Callout>
 
+### Expiry worker differs from the CLI
+
+An active expiration timer can still run an older worker. Installing or updating
+the controller CLI does not deploy host code. `doctor` checks the same worker
+protocol as disposable creation and retention cleanup, without running expiry or
+changing any boxes. It exits nonzero and names the location on a mismatch.
+
+Review the host deployment changes, then deploy from the checkout matching your
+CLI. Replace `lab` with the location reported by the diagnostic:
+
+```sh title="CONTROLLER: Explicitly update a reviewed host deployment"
+python3 deploy/deploy.py --location lab
+raft doctor
+```
+
+Deployment updates host infrastructure as well as the expiry worker. Do not run
+it on a shared host without authorization. Do not remove the compatibility check
+or mark an old worker as current; older workers may retain expired disposable
+boxes instead of deleting them.
+
 ### Listing active and stopped containers
 
 Inspect workspace states across locations:
@@ -74,13 +95,17 @@ Host system journals can capture command lines, arguments, and environment varia
 
 ### Manually triggering garbage collection
 
-The host timer stops expired containers every 15 seconds. Trigger an immediate expiration run from the controller:
+The host timer enforces expiry every 15 seconds. Trigger an immediate expiration run from the controller:
 
 ```sh title="CONTROLLER: Run expiration workers manually"
 raft gc
 ```
 
-Expired containers transition from `Running` to `Stopped`. Expiration never deletes container files or snapshots.
+Expired persistent containers transition from `Running` to `Stopped` and retain
+files and snapshots. Expired disposable containers are deleted, including their
+snapshots. Use `raft gc --location lab` to check only one host. The timer normally
+checks every 15 seconds; host outages, graceful shutdown and long locked
+operations can delay expiry. See [Lifetime and persistent services](./long-running-services.mdx).
 
 ## Common error scenarios
 

package/docs/guides/upgrades.mdx

@@ -0,0 +1,113 @@
+---
+title: Upgrades and image changes
+description: Update the controller CLI and host worker, replace image fingerprints, and understand existing-box behavior.
+---
+
+# Upgrades and image changes
+
+The controller CLI, host deployment scripts, and workspace image are separate
+parts of Raft. Updating one does not automatically update the others.
+
+| Component | Update method | Effect on existing boxes |
+| --- | --- | --- |
+| Controller CLI | Install the chosen npm or Python release | Changes controller commands; does not rewrite guest files |
+| Host worker and firewall | Run the chosen release's deployer per host | Replaces host policy code and reapplies filtering |
+| Pinned development image | Import an image and change the configuration fingerprint | Used for future image-based `new`; existing boxes keep their files |
+
+## Choose a release and prepare
+
+Review [release notes](https://github.com/Microck/raft/releases) and use matching
+CLI and deployment sources. The examples below pin `v0.2.0`; replace that version
+consistently when adopting another release.
+
+```sh title="CONTROLLER: Inspect the current installation"
+raft --version
+raft doctor
+raft list
+```
+
+Save a copy of your controller inventory. Export important persistent boxes using
+[Backup and recovery](./backup-recovery.mdx) before host changes. Backup requires a
+stopped box. Stopping a disposable box deletes it, so copy needed files out before
+stopping one. Schedule redeployment when interruptions are acceptable.
+
+## Update the controller
+
+For an npm installation:
+
+```sh title="CONTROLLER: Install a pinned CLI release"
+npm install -g @microck/raft@0.2.0
+raft --version
+```
+
+For a Python installation, download the wheel and `SHA256SUMS` from the same
+release, verify the checksum, then install it with `uv`:
+
+```sh title="CONTROLLER: Install a verified release wheel instead of npm"
+sha256sum --ignore-missing -c SHA256SUMS
+uv tool install --force ./raft-0.2.0-py3-none-any.whl
+raft --version
+```
+
+Proceed only if checksum verification succeeds. Choose one installation method
+and check which executable your shell uses with `command -v raft`.
+
+For an editable source installation, updating the checkout changes the installed
+tool's source. Keep uncommitted work separate and run `uv sync --locked` for the
+chosen checkout. Raft has no `self-update` command.
+
+## Redeploy each host
+
+Obtain a separate checkout of the matching release:
+
+```sh title="CONTROLLER: Obtain and deploy matching host code"
+git clone --branch v0.2.0 https://github.com/Microck/raft.git raft-v0.2.0
+cd raft-v0.2.0
+python3 deploy/deploy.py --location lab
+raft doctor
+```
+
+Repeat for each configured location. The deployer uses your existing
+`~/.config/raft/incus.json`, verifies owned project/pool/network/profile state,
+replaces host scripts and units, reapplies firewall rules, and starts the expiry
+timer. It does not import an image or migrate guest files.
+
+Do not bypass a conflicting-resource or storage-size error. Inspect the host and
+preserve backups before explicit recovery; redeployment does not repair arbitrary
+infrastructure drift. Disposable creation/fork and prune check the worker protocol
+and fail if it differs from the CLI. That check is not an automatic upgrade.
+
+## Replace the image pin
+
+Follow [Acquire a native workspace image](../operations.mdx#acquire-a-native-workspace-image)
+to download and verify or build the image for each host's architecture. Retrieve
+its full fingerprint and replace only that location's `image` field in the
+controller inventory. Keep the previous fingerprint if you may need it again.
+
+Changing the pin affects ordinary `raft new` calls. Resume retains the existing
+filesystem; forks and `new --from` use their source filesystem, not the new image.
+Install guest updates explicitly inside an existing box, or create a new box and
+transfer the project and needed data. Recovering an old backup also restores that
+backup's guest filesystem rather than rebuilding it from the new image.
+
+## Verify the update
+
+```sh title="CONTROLLER: Smoke-check a new persistent box"
+box=$(raft new --location lab --ttl 600)
+raft exec "$box" -- node --version
+raft exec "$box" -- docker info
+raft stop "$box"
+raft resume "$box" --ttl 600
+raft exec "$box" -- ls /workspace
+raft destroy "$box"
+```
+
+Creation can return before guest services are ready. Use the readiness pattern in
+[Scripting and CI](./scripting-ci.mdx) if an immediate check fails during startup.
+For dedicated test infrastructure, [Verification](../verification.mdx) describes
+the real-host suites and their effects.
+
+To revert an image choice for future boxes, restore the previous fingerprint
+while that image remains imported. This does not revert existing boxes. Raft has
+no automatic migration or rollback of host policy or guest packages; review the
+chosen release and backups before downgrading those components.

package/docs/index.mdx

@@ -27,10 +27,14 @@ Stopping a workspace preserves files and packages on disk. Resuming resets its r
 
 | Task | Guide |
 | :--- | :--- |
+| Prepare a project and reuse its installed dependencies | [Project setup and templates](./guides/project-setup.mdx) |
 | Run nested containers and build images | [Docker in workspaces](./guides/docker.mdx) |
 | Transfer files between controller and workspace | [File transfer](./guides/files.mdx) |
 | Run detached commands with systemd logs | [Background jobs](./guides/background-jobs.mdx) |
+| Manage deadlines and restart services after resume | [Lifetime and persistent services](./guides/long-running-services.mdx) |
+| Automate the CLI and parse its output | [Scripting and CI](./guides/scripting-ci.mdx) |
 | Forward guest ports or open noVNC desktop | [Desktop and private services](./guides/desktop-services.mdx) |
 | Capture disk state or clone workspaces | [Snapshots and forks](./guides/snapshots-forks.mdx) |
 | Export portable backups across hosts | [Backup and recovery](./guides/backup-recovery.mdx) |
+| Update the CLI, host worker, and workspace image | [Upgrades and image changes](./guides/upgrades.mdx) |
 | Inspect host services and diagnose errors | [Troubleshooting](./guides/troubleshooting.mdx) |

package/docs/operations.mdx

@@ -33,18 +33,31 @@ sudo -n true && echo "Passwordless sudo is functional"
 
 <Step>
 
-### Clone repository and install controller CLI
+### Install the CLI and obtain deployment scripts
 
-On your controller machine, clone the repository and install the `raft` CLI tool using `uv`:
+On your Linux controller with Python 3.11.8+, install the published CLI:
 
-```sh title="CONTROLLER: Clone repository and install CLI"
-git clone https://github.com/Microck/raft.git raft
+```sh title="CONTROLLER: Install published CLI"
+npm install -g @microck/raft@0.2.0
+raft --version
+```
+
+Installation does not configure hosts or download images. Clone the matching
+release to obtain the deployment scripts and example configuration:
+
+```sh title="CONTROLLER: Obtain deployment scripts"
+git clone --branch v0.2.0 https://github.com/Microck/raft.git raft
 cd raft
+```
+
+Alternatively, install from that checkout using `uv` instead of npm:
+
+```sh title="CONTROLLER: Clone repository and install CLI"
 uv sync --locked
 uv tool install --editable .
 ```
 
-Add `~/.local/bin` to `PATH` and verify the CLI:
+For the `uv` installation, add `~/.local/bin` to `PATH` and verify the CLI:
 
 ```sh title="CONTROLLER: Verify CLI installation"
 export PATH="$HOME/.local/bin:$PATH"
@@ -122,16 +135,16 @@ Workspaces require an unprivileged Debian 13 development image matching the host
 
 <Tab value="Download release image">
 
-Download the release archive, checksum, and manifest from [Raft releases](https://github.com/Microck/raft/releases). The `sha256sum` check gates import so corrupted archives are rejected:
+Download the release archive, checksum, and manifest from [Raft releases](https://github.com/Microck/raft/releases). These examples pin `v0.2.0`, matching the CLI and deployment checkout above. The `sha256sum` check gates import so corrupted archives are rejected:
 
 <Tabs items={["ARM64", "AMD64"]}>
 
 <Tab value="ARM64">
 
 ```sh title="HOST (ARM64): Download, verify, and import release image"
-curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/raft-dev-arm64.tar.gz
-curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/SHA256SUMS
-curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/manifest-arm64.json
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.2.0/raft-dev-arm64.tar.gz
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.2.0/SHA256SUMS
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.2.0/manifest-arm64.json
 
 sha256sum --ignore-missing -c SHA256SUMS && \
   sudo incus --project raft image import ./raft-dev-arm64.tar.gz --alias raft-dev
@@ -142,9 +155,9 @@ sha256sum --ignore-missing -c SHA256SUMS && \
 <Tab value="AMD64">
 
 ```sh title="HOST (AMD64): Download, verify, and import release image"
-curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/raft-dev-amd64.tar.gz
-curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/SHA256SUMS
-curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/manifest-amd64.json
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.2.0/raft-dev-amd64.tar.gz
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.2.0/SHA256SUMS
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.2.0/manifest-amd64.json
 
 sha256sum --ignore-missing -c SHA256SUMS && \
   sudo incus --project raft image import ./raft-dev-amd64.tar.gz --alias raft-dev
@@ -222,3 +235,5 @@ Check that Incus, the network service, and the expiration timer are active. For
 ## Next step
 
 Proceed to [First working box](./first-box.mdx) to launch, verify, and manage your first workspace.
+
+For later releases, follow [Upgrades and image changes](./guides/upgrades.mdx).

package/docs/reference/capacity-limits.mdx

@@ -50,7 +50,16 @@ Select CPU and memory limits when launching with `raft new`:
 - `--cpu`: `1` or `2` shared vCPUs managed by Incus (default: `1`).
 - `--memory`: `1GiB`, `2GiB`, or `4GiB` cgroup limit (default: `2GiB`).
 
-Resizing existing containers or modifying limits upon resume is not supported.
+Change supported CPU/RAM limits when resuming a stopped box or creating a fork:
+
+```sh
+raft resume "$box" --ttl 1800 --cpu 2 --memory 4GiB
+copy=$(raft fork "$box" --ttl 1800 --memory 2GiB)
+```
+
+Resume requires a stopped box; fork requires a stopped source. Omitted limits
+stay unchanged on resume and inherit the source limits on fork. Live resizing is
+not exposed by the CLI. See [Snapshots and forks](../guides/snapshots-forks.mdx).
 
 ## Host resource reserves and active states
 

package/docs/reference/cli.md

@@ -10,6 +10,10 @@ Command-line reference for the `raft` controller tool.
 Syntax:
 `raft <subcommand> [options] [arguments]`
 
+`raft --help` lists commands, `raft <subcommand> --help` lists its arguments,
+and `raft --version` prints the installed version without loading configuration.
+Output is command-specific; see [Scripting and CI](../guides/scripting-ci.mdx).
+
 Workspaces are referenced by location-qualified handles formatted as `<location>:<instance-name>` (for example `lab:rf-a1b2c3d4e5f60718`).
 
 ## Command index
@@ -31,7 +35,7 @@ Workspaces are referenced by location-qualified handles formatted as `<location>
 
 Create and launch a new workspace container.
 
-`raft new [--location LOCATION] [--ttl TTL] [--cpu {1,2}] [--memory {1GiB,2GiB,4GiB}]`
+`raft new [--location LOCATION] [--ttl TTL] [--cpu {1,2}] [--memory {1GiB,2GiB,4GiB}] [--disposable] [--from LOCATION:BOX/SNAPSHOT]`
 
 - `--location LOCATION`: Target host location in `~/.config/raft/incus.json`. Defaults to first configured location.
 - `--ttl TTL`: Lifetime in seconds before automatic stop (`60` to `2592000`). Default: `600`.
@@ -59,6 +63,9 @@ Start a stopped workspace with a renewed lifetime deadline.
 
 - `--ttl TTL`: Lifetime in seconds from resumption (`60` to `2592000`). Required.
 
+CPU/RAM overrides on resume apply before startup. Omitted values stay unchanged.
+A running box is rejected before changing its limits.
+
 ### `extend`
 
 Reset the expiration deadline for a running workspace without restarting it.
@@ -75,8 +82,6 @@ Permanently delete a workspace container and its snapshots.
 
 Stops the container if running, removes all associated Btrfs snapshots, and releases host storage.
 
-CPU/RAM overrides on resume apply before startup. Omitted values stay unchanged. A running box is rejected before changing its limits.
-
 ## Execution and inspection commands
 
 ### `exec`
@@ -252,7 +257,7 @@ Inspect host resource capacity, active allocations, and recommended workspace li
 
 ### `doctor`
 
-Inspect host connectivity, Incus version, active systemd units (`incus`, `raft-expire.timer`, `raft-network.service`), root filesystem usage, storage pool status, and saved box count across configured hosts.
+Inspect host connectivity, Incus version, expiry-worker compatibility, active systemd units (`incus`, `raft-expire.timer`, `raft-network.service`), root filesystem usage, storage pool status, and saved box count across configured hosts. An incompatible worker returns a nonzero exit status and names the failing location without running expiry or modifying boxes.
 
 `raft doctor`
 
@@ -262,6 +267,10 @@ List existing workspace containers across configured hosts.
 
 `raft list [--location LOCATION]`
 
+Prints one JSON object per box (JSONL), with `box`, `state`, and `stopAt` fields.
+An empty inventory prints no records. See [Scripting and CI](../guides/scripting-ci.mdx)
+for output formats, exit codes, readiness, and failure handling.
+
 ### `gc`
 
 Enforce deadlines immediately. Persistent boxes stop; disposable boxes are deleted.

package/docs/reference/configuration.mdx

@@ -39,7 +39,7 @@ The configuration file maps location names to host definition objects:
 ### Location keys
 
 - **Naming format**: Matches `^[a-zA-Z0-9][a-zA-Z0-9_-]*$`. Starts with an alphanumeric character and contains ASCII letters, digits, underscores, or hyphens.
-- **Default location**: The first location listed is the default for `raft new`. `raft list` and `raft limits` accept `--location` filters; `raft doctor` and `raft gc` check all configured hosts.
+- **Default location**: The first location listed is the default for `raft new`. `raft list`, `raft limits`, `raft gc`, and `raft prune` accept `--location` filters. Without a filter they check all configured hosts; `raft doctor` always checks all configured hosts.
 
 ### Host entry fields
 

package/docs/releases/0.2.1.md

@@ -0,0 +1,14 @@
+---
+title: Raft 0.2.1
+---
+
+# Raft 0.2.1
+
+- `raft doctor` now rejects an incompatible host expiry worker and identifies the location that needs deployment. It does not run expiry or change boxes.
+- Lifecycle transactions close controller stdin so allocation cannot wait for piped input while holding the host lock.
+
+Install on Linux with Python 3.11.8+ using `npm install -g @microck/raft@0.2.1`, or install the attached Python wheel. Updating the CLI does not update host workers. Deploy the matching checkout to hosts that fail the readiness check.
+
+Exec, SSH, and file-transfer behavior is unchanged. Piped input to `raft exec` is unsupported.
+
+ARM64 and AMD64 development images accompany this release. Verify `SHA256SUMS` before importing an image for your host architecture.

package/docs/verification.mdx

@@ -57,9 +57,15 @@ The [final runtime validation](https://github.com/Microck/raft/actions/runs/3754
 
 ## New workspace controls
 
-`python3 deploy/verify-features.py --location lab` verifies recursive trees and unsafe-archive rejection, templates from a running source, snapshot deletion, CPU/RAM overrides, disposable timer deletion, and retention preview/apply. It uses only its own fixtures. Run the archive checks without `--location` to avoid allocating boxes.
+`python3 deploy/verify-features.py --location lab` verifies `doctor` readiness against the installed worker, recursive trees and unsafe-archive rejection, templates from a running source, snapshot deletion, CPU/RAM overrides, disposable timer deletion, and retention preview/apply. It uses only its own fixtures. Run the archive checks without `--location` to avoid allocating boxes.
 
-For an older deployment whose services should stay unchanged, `--isolated-worker` stages the current worker under a private temporary path. This checks worker logic without installing it; automatic timer integration is verified on disposable CI hosts. Current deployments use the normal verification mode; the isolated mode rejects them before allocation.
+The live suite also holds a controller stdin pipe open while running real remote
+readers through both captured and streamed lifecycle transactions. Both must
+receive EOF without consuming controller input, and a subsequent transaction
+must acquire the same host lock. This covers allocation stalls caused by Incus
+waiting for stdin while the expiration worker waits for the lifecycle lock.
+
+For an older deployment whose services should stay unchanged, `--isolated-worker` first verifies that `doctor` rejects the stale installed worker with a location-specific deployment diagnostic, then uses the current worker under a private temporary path. This checks worker logic without installing it; automatic timer integration is verified on disposable CI hosts. Current deployments use the normal verification mode; the isolated mode rejects them before allocation.
 
 ## Workspace controls validation (2026-10-07)
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@microck/raft",
-  "version": "0.2.0",
+  "version": "0.2.1",
   "description": "Self-hosted Linux workspaces powered by Incus, an open-source alternative to boat.dev",
   "license": "MIT",
   "repository": {

package/raft.py

@@ -16,7 +16,7 @@ import uuid
 
 import raft_files
 
-__version__ = "0.2.0"
+__version__ = "0.2.1"
 
 CONFIG = Path.home() / ".config/raft/incus.json"
 PROJECT = "raft"
@@ -304,7 +304,9 @@ def transaction(location, script, *arguments, stdout=None):
         *arguments,
     ]
     connection = [
+        # Lifecycle transactions take arguments, never interactive or piped input.
         "ssh",
+        "-n",
         "-T",
         "-o",
         "BatchMode=yes",
@@ -710,6 +712,10 @@ def dispatch(args):
                     print(output, end="")
             elif args.action == "doctor":
                 print(location + ": " + incus(location, "version").strip())
+                try:
+                    transaction(location, CHECK_WORKER)
+                except RuntimeError as error:
+                    raise RuntimeError(f"{location}: {error}") from error
                 response = subprocess.run(
                     [
                         "ssh",