@microck/raft 0.2.0 → 0.3.0

package/docs/contract.md

@@ -23,6 +23,7 @@ Raft manages persistent unprivileged native-architecture Incus system containers
 - **Strict infrastructure validation**: Deployment refuses conflicting storage or network names and incomplete infrastructure rather than overwriting or repairing state.
 - **Storage pool sizing**: Existing owned Btrfs pools must retain the configured 60 GiB size; drift is rejected without automatic resizing.
 - **Firewall precedence**: Redeployment reapplies firewall rules. Incus startup requires successful firewall application before the daemon starts, including socket activation.
+- **Guided onboarding**: `raft host add LOCATION --ssh TARGET` probes the target read-only and refuses unsupported hosts, including macOS, before any change. It prints the exact plan and applies it only after an interactive yes or `--yes`; `--dry-run` stops after the plan. It runs the same deployer as `deploy/deploy.py`, imports an image only after its SHA-256 matches (release `SHA256SUMS` or the local archive digest) and its architecture matches the host, then registers the location with the image fingerprint. It finishes with `doctor` and a disposable create/exec/destroy smoke test; if either fails, it removes the new location from the configuration and leaves host changes for a rerun. It uses existing key-based SSH and never collects passwords or changes SSH, other firewall rules or IP configuration. It requires the deployment files from the npm package or a source checkout.
 - **CLI coverage**: The single-operator workflow spans lifecycle, execution, files, snapshots, jobs, tunnels, backup, and capacity inspection; see the [command reference](./reference/cli.md).
 
 ## Configuration and lifecycle defaults
@@ -32,13 +33,15 @@ Raft manages persistent unprivileged native-architecture Incus system containers
 | Host identification | Configuration requires named SSH targets using letters, digits, underscores, or hyphens. |
 | Qualified handles | Every handle includes location and instance name (`<location>:<name>`). |
 | Image pinning | Workspace creation requires the immutable 64-hex image fingerprint, not an updating alias. |
-| Lifecycle defaults | Defaults to first configured location, 1 CPU, 2 GiB RAM, and 600-second lifetime. Supported choices are 1 or 2 CPUs, 1/2/4 GiB RAM, and 60 to 2,592,000 seconds (30 days) TTL. |
+| Lifecycle defaults | Defaults to first configured location, 1 CPU, 2 GiB RAM, and 600-second lifetime. Supported choices are 1 or 2 CPUs, 1/2/4/8 GiB RAM, and 60 to 2,592,000 seconds (30 days) TTL. |
 | Admission limit | Hard ceiling of four saved boxes per host, including stopped boxes. |
 | TTL calculation | Running TTL begins after launch completes, using the host clock. Cold image preparation does not consume TTL. |
-| Host locking | Creation, lifecycle transitions, and expiry are serialized by host file lock (`/run/lock/raft-incus.lock`). Native Incus admin calls bypass CLI policy. |
+| Host locking | Creation, lifecycle transitions, and expiry are serialized by host file lock (`/run/lock/raft-incus.lock`). Noninteractive transactions close controller stdin so commands cannot wait for piped input while holding this lock. Exec, SSH, and file-transfer paths remain unchanged. Piped input to `raft exec` is unsupported. Native Incus admin calls bypass CLI policy. |
 
 ## Capacity and admission inspection
 
+`raft locations` lists configured locations as JSONL with `location`, `default` and `note` fields without contacting hosts. `raft note LOCATION TEXT` and `raft note LOCATION --clear` edit the optional `note` field in the controller configuration atomically, preserving location order and mode `0600`. Notes are advisory guidance; Raft shows them in `locations`, `limits` and on stderr from `new`, and never enforces them.
+
 `raft limits [--location NAME] [--json]` inspects hosts without modifying state. It reports effective CPUs, total and available RAM, saved and active box counts, active resource limits, and disk space.
 
 - **Reserve calculations**: Recommends running boxes after reserving 1 CPU and the larger of 2 GiB or 10% host RAM.
@@ -51,7 +54,7 @@ Raft manages persistent unprivileged native-architecture Incus system containers
 
 - **Stop and resume**: Stop terminates guest processes and preserves disk contents; stopped boxes retain disk but do not reserve guest RAM. Resume starts the container anew with an explicit TTL without restoring process memory.
 - **Lifetime extension**: `raft extend --ttl` resets a running box's deadline from current host time without restarting; stopped boxes require `raft resume`.
-- **Host worker contract**: Disposable creation/fork and retention cleanup require the current worker protocol. A mismatch fails before executing deletion policy or creating a disposable box; deploy the current checkout explicitly.
+- **Host worker contract**: `doctor`, disposable creation/fork, and retention cleanup require the current worker protocol. `doctor` checks it without running expiry or changing boxes, and reports the failing location. A mismatch fails before executing deletion policy or creating a disposable box; deploy the current checkout explicitly. Installing a new CLI does not update host workers.
 - **Disposable lifecycle**: `new --disposable` opts a box into deletion on stop or TTL expiry, including its snapshots. Clones and recovered boxes default to persistent; fork accepts an explicit `--disposable`.
 - **Retention cleanup**: `prune --older-than DAYS` previews persistent stopped boxes with a recorded stop timestamp. `--yes` deletes matching boxes under the host lifecycle lock. Running boxes and boxes without a recorded stop time are excluded. No automatic persistent-box deletion is enabled.
 - **Resizing**: Resume and fork accept optional supported CPU/RAM overrides, preserving omitted values. Resume validates stopped state under the lock before changing sizing.

package/docs/first-box.mdx

@@ -71,6 +71,9 @@ raft destroy "$box"
 
 ## Next steps
 
+- [Project setup and templates](./guides/project-setup.mdx): Prepare a project and reuse its dependencies.
+- [Lifetime and persistent services](./guides/long-running-services.mdx): Manage deadlines and restart applications after resume.
+- [Scripting and CI](./guides/scripting-ci.mdx): Parse CLI output and handle failures.
 - [Docker in workspaces](./guides/docker.mdx): Run nested containers and build images.
 - [File transfer](./guides/files.mdx): Upload and download files over SSH.
 - [Background jobs](./guides/background-jobs.mdx): Run detached commands under systemd.
@@ -78,5 +81,6 @@ raft destroy "$box"
 - [Snapshots and forks](./guides/snapshots-forks.mdx): Capture disk states and create same-host clones.
 - [Backup and recovery](./guides/backup-recovery.mdx): Export portable archives and restore across hosts.
 - [Troubleshooting](./guides/troubleshooting.mdx): Diagnose common errors and inspect host services.
+- [Upgrades and image changes](./guides/upgrades.mdx): Update the controller, hosts, and image pins.
 
 For temporary work, create with `--disposable`. Stopping or expiry then deletes the box and its snapshots. For persistent boxes, `raft prune --older-than 7` previews old stopped workspaces; add `--yes` only after reviewing the candidates.

package/docs/guides/background-jobs.mdx

@@ -90,3 +90,6 @@ raft extend "$box" --ttl 1800
 ```
 
 `raft extend` resets the container expiration deadline to `now + TTL` seconds using the host clock. It does not add time to the existing deadline.
+
+For deadline inspection and an enabled systemd service that starts again after
+resume or fork, see [Lifetime and persistent services](./long-running-services.mdx).

package/docs/guides/desktop-services.mdx

@@ -66,7 +66,10 @@ Start the desktop and establish a private tunnel:
 raft desktop "$box" --local 6080
 ```
 
-`raft desktop` starts `raft-desktop.service` inside the container if needed, waits for Xvfb and VNC readiness, and opens a tunnel to local port `6080`.
+`raft desktop` requests startup of `raft-desktop.service` inside the container if
+needed and opens a tunnel to local port `6080`. The service waits for Xvfb before
+launching its desktop clients. The CLI does not poll noVNC readiness; if the first
+browser request fails during startup, retry after the service initializes.
 
 Open the client in your browser:
 

package/docs/guides/long-running-services.mdx

@@ -0,0 +1,109 @@
+---
+title: Lifetime and persistent services
+description: Inspect deadlines, manage disposable and persistent boxes, and restart applications after resume.
+---
+
+# Lifetime and persistent services
+
+A box's lifetime controls when the host stops or deletes it. Activity, an open
+terminal, and detached jobs do not extend that deadline automatically.
+
+## Choose a lifetime and retention policy
+
+```sh title="CONTROLLER: Create a persistent box"
+box=$(raft new --location lab --ttl 3600)
+```
+
+TTL is in seconds: 60 to 2,592,000 (30 days), with a creation default of 600.
+The host sets the deadline after launch completes, using its own clock.
+Raft has no option to disable automatic expiry.
+
+| Operation | Persistent box (default) | Disposable box |
+| --- | --- | --- |
+| `stop` or TTL expiry | Stops processes; keeps disk and snapshots | Deletes the box and snapshots |
+| `resume --ttl SECONDS` | Starts the saved box with a new deadline | Only possible while the box still exists |
+| `destroy` | Deletes the box and snapshots | Deletes the box and snapshots |
+
+Use `raft new --disposable` for temporary work. A fork is persistent unless
+`--disposable` is explicitly supplied; recovered boxes are persistent.
+Back up work before stopping a disposable box.
+
+Persistent stopped boxes still occupy a saved slot and storage. Each host allows
+four saved boxes. `raft prune --older-than 7` previews old stopped boxes;
+`--yes` applies deletion. There is no automatic persistent-box retention cleanup.
+
+## Read and reset the deadline
+
+Select a running handle from `raft list`. These examples use controller `jq`:
+
+```sh title="CONTROLLER: Inspect and extend a running box"
+raft list
+box="lab:rf-a1b2c3d4e5f60718"  # Replace with your actual handle
+raft info "$box" | jq '{status, expires: .config["user.raft.expires"]}'
+raft extend "$box" --ttl 7200
+raft info "$box" | jq '{status, expires: .config["user.raft.expires"]}'
+```
+
+`user.raft.expires` is a Unix timestamp encoded as a string, not a remaining
+duration. `raft list` exposes it as `stopAt`; it may be empty on stopped boxes.
+`extend` sets `now + TTL`, rather than adding time to the old deadline.
+It requires a running box. Start a stopped box with `raft resume "$box" --ttl 7200`.
+
+The host timer checks every 15 seconds. Graceful shutdown, a host outage, or a
+long transaction holding the lifecycle lock can delay expiry. `raft gc --location lab`
+runs an immediate deadline check. A deadline is not a guarantee that a job will
+finish before shutdown; extend before starting work that needs more time.
+
+## Restart a service after resume
+
+One-off detached commands do not survive a stop. To start an application again
+after resume or fork, save a systemd unit on disk and enable it. This starts a new
+process; it does not recover process memory or an interrupted request.
+
+The following example serves `/workspace` as the guest `developer` user. Use a
+persistent running box and ensure files the application needs are readable by
+that user. For an actual application, replace `ExecStart` and the working directory.
+
+```sh title="CONTROLLER: Install and enable an example service"
+raft exec "$box" -- bash -lc 'cat > /etc/systemd/system/workspace-web.service <<"UNIT"
+[Unit]
+Description=Workspace HTTP example
+After=network.target
+
+[Service]
+User=developer
+WorkingDirectory=/workspace
+ExecStart=/usr/bin/python3 -m http.server 8080 --bind 0.0.0.0
+Restart=on-failure
+RestartSec=2
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+systemctl daemon-reload
+systemctl enable --now workspace-web.service'
+raft exec "$box" -- systemctl status workspace-web.service --no-pager
+raft exec "$box" -- journalctl -u workspace-web.service -n 30 --no-pager
+```
+
+Open `raft forward "$box" --remote 8080 --local 8080` and visit
+`http://127.0.0.1:8080`. The tunnel runs in the foreground; close it with `Ctrl+C`.
+
+```sh title="CONTROLLER: Check service startup after resume"
+raft stop "$box"
+raft resume "$box" --ttl 3600
+raft exec "$box" -- systemctl is-active workspace-web.service
+```
+
+Reopen the tunnel after resume. A fork or snapshot template copies the enabled
+unit too, so its new box starts its own service. Application secrets on disk are
+also copied; see [Project setup and templates](./project-setup.mdx).
+
+To disable the example, run:
+
+```sh
+raft exec "$box" -- systemctl disable --now workspace-web.service
+```
+
+See [Background jobs](./background-jobs.mdx) for one-off tasks and
+[Desktop and private services](./desktop-services.mdx) for tunnel behavior.

package/docs/guides/meta.json

@@ -3,12 +3,16 @@
   "collapsible": true,
   "defaultOpen": false,
   "pages": [
+    "project-setup",
     "docker",
     "files",
     "background-jobs",
+    "long-running-services",
+    "scripting-ci",
     "desktop-services",
     "snapshots-forks",
     "backup-recovery",
+    "upgrades",
     "troubleshooting"
   ]
 }

package/docs/guides/project-setup.mdx

@@ -0,0 +1,112 @@
+---
+title: Project setup and templates
+description: Upload or clone a project, run setup scripts, and create reusable same-host snapshot templates.
+---
+
+# Project setup and templates
+
+Raft starts a clean development image. Prepare a project with ordinary guest
+commands, then save a snapshot to reuse installed dependencies. There is no
+managed repository registry, environment registry, setup hook, or secret-injection
+service. Commands run as guest root in `/workspace` unless you choose another user.
+
+## Put a project in the box
+
+Create a persistent box after [host setup](../operations.mdx):
+
+```sh title="CONTROLLER: Create a project box"
+box=$(raft new --location lab --ttl 3600)
+```
+
+Upload a local project, excluding credentials and files you do not want copied.
+The example assumes `./my-project` contains your project and `setup.sh`:
+
+```sh title="CONTROLLER: Upload local project"
+raft upload "$box" ./my-project /workspace/my-project --recursive
+```
+
+The destination directory must not exist. For a public repository, an alternative
+is guest `git clone`; replace the example URL with your repository:
+
+```sh title="CONTROLLER: Clone instead of uploading"
+raft exec "$box" -- git clone https://github.com/OWNER/REPOSITORY.git /workspace/my-project
+```
+
+Choose one method. Private repositories require authentication configured inside
+the guest; Raft does not forward your controller's GitHub login or SSH agent.
+The clean image does not bundle coding agents or their credentials. Install and
+authenticate any agent inside the box using that tool's own instructions.
+
+## Run setup explicitly
+
+Use `bash -lc` for directory changes or shell syntax:
+
+```sh title="CONTROLLER: Run project setup"
+raft exec "$box" -- bash -lc 'cd /workspace/my-project && bash ./setup.sh'
+```
+
+Make setup scripts safe to rerun. Raft does not execute them automatically on
+create, resume, or fork. A failing synchronous command returns its exit code;
+see [Scripting and CI](./scripting-ci.mdx).
+
+For slow setup, use a detached job and inspect its result:
+
+```sh title="CONTROLLER: Run slow setup instead of synchronous setup"
+job=$(raft exec "$box" --detach -- bash -lc 'cd /workspace/my-project && bash ./setup.sh')
+raft status "$box" "$job"
+raft logs "$box" "$job"
+```
+
+Wait for successful completion before making a template. Extending the lifetime
+may be necessary; a detached job does not prevent expiry.
+
+## Supply application configuration
+
+For an application that reads a local `.env` file, upload it explicitly and set
+permissions for the application's user. This example assumes a controller file
+`./app.env` and an application running as `developer`:
+
+```sh title="CONTROLLER: Install a private application configuration file"
+raft upload "$box" ./app.env /workspace/my-project/.env
+raft exec "$box" -- chown developer:developer /workspace/my-project/.env
+raft exec "$box" -- chmod 600 /workspace/my-project/.env
+```
+
+The application must load that file; upload does not export environment variables.
+If running the whole project as `developer`, adjust project ownership separately.
+Do not place secrets in command arguments, committed files, or logs. Host journals
+can capture guest command lines. Raft does not rotate or scrub credentials.
+
+## Save and reuse a prepared template
+
+Finish setup, remove credentials that should not be inherited, and stop the box:
+
+```sh title="CONTROLLER: Save the prepared filesystem"
+raft stop "$box"
+raft snapshot "$box" prepared
+```
+
+Create another box from the snapshot:
+
+```sh title="CONTROLLER: Start a copy with explicit resources"
+copy=$(raft new --from "$box/prepared" --ttl 3600 --cpu 2 --memory 4GiB)
+raft exec "$copy" -- ls -la /workspace/my-project
+```
+
+The template stays on its source host. Its source box can be running when you
+copy the saved snapshot. `new --from` uses normal creation defaults when sizing
+is omitted: 1 CPU and 2 GiB RAM. The new box has its own handle, MAC address,
+deadline, and saved slot.
+
+The snapshot includes packages, project files, credentials, and enabled services.
+It does not include process memory. Changing the original after snapshot creation
+does not update the saved template. Keep its source box and snapshot until you
+no longer need to create copies; destroying the source deletes its snapshots.
+
+To copy the stopped box's current filesystem instead, use
+`raft fork "$box" --ttl 3600`. Fork inherits omitted CPU/RAM limits and does not
+copy the source snapshots. See [Snapshots and forks](./snapshots-forks.mdx).
+
+For a different host, use [Backup and recovery](./backup-recovery.mdx) with matching
+CPU architecture. For services that start on resume, see
+[Lifetime and persistent services](./long-running-services.mdx).

package/docs/guides/scripting-ci.mdx

@@ -0,0 +1,134 @@
+---
+title: Scripting and CI
+description: Parse CLI output, preserve command exit codes, wait for readiness, and handle interrupted operations.
+---
+
+# Scripting and CI
+
+Raft is a controller CLI using host SSH. A Linux CI runner needs Python 3.11.8+,
+OpenSSH, the installed CLI, host inventory, verified SSH host keys, batch
+authentication, and hosts with passwordless sudo. No Raft HTTP API, SDK, or API key
+exists. Give access only to trusted jobs: the configured SSH account has host sudo
+and controls all boxes on that host.
+
+## Output formats
+
+Raft does not have a global `--json` option. Parse each command's actual format:
+
+| Command | stdout |
+| --- | --- |
+| `new`, `fork`, `recover` | One location-qualified handle |
+| `exec --detach` | One `rfcmd-` job ID |
+| `list` | JSONL: one object per box, with `box`, `state`, `stopAt` |
+| `info` | One pretty-printed native Incus metadata JSON object |
+| `usage` | One JSON object containing running resource counters |
+| `limits --json` | One JSON object per location (JSONL across multiple hosts) |
+| `locations` | One JSON object per configured location (JSONL) |
+| `prune` | JSONL candidates or deleted records with `box`, `action` |
+| `status` | systemd `ActiveState`, `SubState`, `ExecMainStatus` property lines |
+| `logs` | Guest journal text |
+| `exec` | Payload stdout/stderr |
+| Other inspection/maintenance commands | Human-readable text or no output |
+
+`list` emits nothing when there are no boxes. `stopAt` is the host Unix deadline
+as a string, or an empty string when absent. `info` uses native Incus keys rather
+than a normalized Raft schema. `usage` includes `memoryBytes`, `memoryLimit`,
+`cpuTime`, `effectiveCpus`, `filesystemUsedBytes`, `filesystemAvailableBytes`, and
+`filesystemScope`; disk values describe the shared pool, not an individual box.
+See [Capacity limits](../reference/capacity-limits.mdx) for capacity calculations.
+
+These examples require `jq` on the controller:
+
+```sh title="CONTROLLER: Parse records without mixing stderr"
+raft list --location lab | jq -r 'select(.state == "Running") | .box'
+raft limits --location lab --json | jq '{location, savedSlots, warnings}'
+raft prune --location lab --older-than 7 | jq -r '.box'
+```
+
+Do not redirect `2>&1` into machine-readable output. Recovery reports its allocated
+handle on stderr before import finishes; only a successful completion prints the
+final handle on stdout.
+
+## Exit codes and job results
+
+| Situation | Exit code |
+| --- | --- |
+| Successful control operation | `0` |
+| Synchronous `exec` | Transport/command return code; preserves payload exit status when transport succeeds |
+| Recognized runtime/configuration failure | `1`, with `raft: ...` on stderr |
+| Argument parsing failure | `2`, with argparse diagnostics on stderr |
+| Caught keyboard interruption | `130` |
+
+SSH transport failures may return SSH's status rather than the payload's result.
+The foreground tunnel replaces the CLI process with SSH and uses SSH's exit code.
+A zero exit from `status` means the query succeeded; read `ExecMainStatus` for the
+job result and `ActiveState`/`SubState` for whether it finished. Do not interpret a
+running job's initial `ExecMainStatus=0` as completion. See
+[Background jobs](./background-jobs.mdx).
+
+## Create, wait, run, and stop
+
+`new` returns after Incus starts the box, before every guest service is necessarily
+ready. Probe a harmless command before running setup. This Bash example waits up
+to about a minute plus transport time, executes an uploaded project check, and
+attempts to stop the persistent box on exit. It assumes `./my-project/check.sh`
+exists and the configured host has a free saved slot:
+
+```bash title="CONTROLLER or trusted Linux CI runner"
+set -euo pipefail
+box=""
+cleanup() {
+  result=$?
+  trap - EXIT
+  if [ -n "$box" ]; then
+    if ! raft stop "$box"; then
+      echo "Cleanup failed; inspect $box" >&2
+      [ "$result" -ne 0 ] || result=1
+    fi
+  fi
+  exit "$result"
+}
+trap cleanup EXIT
+
+box=$(raft new --location lab --ttl 3600)
+printf 'Created %s\n' "$box" >&2
+ready=false
+for attempt in {1..30}; do
+  if raft exec "$box" -- true; then
+    ready=true
+    break
+  fi
+  sleep 2
+done
+if [ "$ready" != true ]; then
+  echo "Guest did not become command-ready: $box" >&2
+  exit 1
+fi
+
+raft upload "$box" ./my-project /workspace/my-project --recursive
+raft exec "$box" -- bash -lc 'cd /workspace/my-project && bash ./check.sh'
+```
+
+The stopped box remains available for inspection and still consumes a saved slot.
+Destroy it after collecting needed files, or preview old stopped boxes with prune.
+For disposable CI boxes, deletion on stop/expiry removes artifacts and snapshots;
+download results before cleanup. A new box or a retry still consumes a saved slot.
+
+Command readiness does not mean Docker or a web application is ready. Probe the
+specific dependency (`docker info`, an HTTP health endpoint, or systemd status)
+with its own bounded wait. Extend the deadline before work exceeding the current
+lifetime. This example's retries are only for the harmless readiness probe.
+
+## Interrupted operations and retries
+
+Raft does not automatically retry commands or provide idempotency keys. If SSH
+drops during creation, resume, import, or execution, the remote operation may have
+already happened. Inspect `raft list`, `raft info`, guest logs, and the host before
+repeating an operation that creates boxes or changes data. A failed create can
+leave a saved box even when no handle reached stdout.
+
+For an interrupted recovery, inspect the handle printed on stderr and follow
+[Host crash during recovery](./troubleshooting.mdx#host-crash-during-recovery).
+For uncertain command execution, check its effects before rerunning it; use
+idempotent setup scripts where possible. Do not treat a lost connection as proof
+of deletion or of a command not having run.

package/docs/guides/troubleshooting.mdx

@@ -19,15 +19,36 @@ raft doctor
 
 `raft doctor` connects to each host over SSH and reports:
 - Incus version running in project `raft`
+- Compatibility of the installed expiry worker with the CLI
 - Active status of host systemd services (`incus`, `raft-expire.timer`, `raft-network.service`)
 - Host root filesystem disk usage (`df -h /`)
 - Status and usage of the `raft-data` Btrfs pool
 - Count of existing saved containers
 
 <Callout type="info" title="Doctor scope">
-`raft doctor` reports runtime daemon and storage status. It does not validate image fingerprints, inspect nftables bridge rules, enforce admission caps, or emit 5 GiB low-disk warnings.
+`raft doctor` reports runtime daemon, expiry-worker compatibility, and storage status. It does not validate image fingerprints, inspect nftables bridge rules, enforce admission caps, or emit 5 GiB low-disk warnings.
 </Callout>
 
+### Expiry worker differs from the CLI
+
+An active expiration timer can still run an older worker. Installing or updating
+the controller CLI does not deploy host code. `doctor` checks the same worker
+protocol as disposable creation and retention cleanup, without running expiry or
+changing any boxes. It exits nonzero and names the location on a mismatch.
+
+Review the host deployment changes, then deploy from the checkout matching your
+CLI. Replace `lab` with the location reported by the diagnostic:
+
+```sh title="CONTROLLER: Explicitly update a reviewed host deployment"
+python3 deploy/deploy.py --location lab
+raft doctor
+```
+
+Deployment updates host infrastructure as well as the expiry worker. Do not run
+it on a shared host without authorization. Do not remove the compatibility check
+or mark an old worker as current; older workers may retain expired disposable
+boxes instead of deleting them.
+
 ### Listing active and stopped containers
 
 Inspect workspace states across locations:
@@ -74,13 +95,17 @@ Host system journals can capture command lines, arguments, and environment varia
 
 ### Manually triggering garbage collection
 
-The host timer stops expired containers every 15 seconds. Trigger an immediate expiration run from the controller:
+The host timer enforces expiry every 15 seconds. Trigger an immediate expiration run from the controller:
 
 ```sh title="CONTROLLER: Run expiration workers manually"
 raft gc
 ```
 
-Expired containers transition from `Running` to `Stopped`. Expiration never deletes container files or snapshots.
+Expired persistent containers transition from `Running` to `Stopped` and retain
+files and snapshots. Expired disposable containers are deleted, including their
+snapshots. Use `raft gc --location lab` to check only one host. The timer normally
+checks every 15 seconds; host outages, graceful shutdown and long locked
+operations can delay expiry. See [Lifetime and persistent services](./long-running-services.mdx).
 
 ## Common error scenarios
 

package/docs/guides/upgrades.mdx

@@ -0,0 +1,113 @@
+---
+title: Upgrades and image changes
+description: Update the controller CLI and host worker, replace image fingerprints, and understand existing-box behavior.
+---
+
+# Upgrades and image changes
+
+The controller CLI, host deployment scripts, and workspace image are separate
+parts of Raft. Updating one does not automatically update the others.
+
+| Component | Update method | Effect on existing boxes |
+| --- | --- | --- |
+| Controller CLI | Install the chosen npm or Python release | Changes controller commands; does not rewrite guest files |
+| Host worker and firewall | Run the chosen release's deployer per host | Replaces host policy code and reapplies filtering |
+| Pinned development image | Import an image and change the configuration fingerprint | Used for future image-based `new`; existing boxes keep their files |
+
+## Choose a release and prepare
+
+Review [release notes](https://github.com/Microck/raft/releases) and use matching
+CLI and deployment sources. The examples below pin `v0.2.0`; replace that version
+consistently when adopting another release.
+
+```sh title="CONTROLLER: Inspect the current installation"
+raft --version
+raft doctor
+raft list
+```
+
+Save a copy of your controller inventory. Export important persistent boxes using
+[Backup and recovery](./backup-recovery.mdx) before host changes. Backup requires a
+stopped box. Stopping a disposable box deletes it, so copy needed files out before
+stopping one. Schedule redeployment when interruptions are acceptable.
+
+## Update the controller
+
+For an npm installation:
+
+```sh title="CONTROLLER: Install a pinned CLI release"
+npm install -g @microck/raft@0.2.0
+raft --version
+```
+
+For a Python installation, download the wheel and `SHA256SUMS` from the same
+release, verify the checksum, then install it with `uv`:
+
+```sh title="CONTROLLER: Install a verified release wheel instead of npm"
+sha256sum --ignore-missing -c SHA256SUMS
+uv tool install --force ./raft-0.2.0-py3-none-any.whl
+raft --version
+```
+
+Proceed only if checksum verification succeeds. Choose one installation method
+and check which executable your shell uses with `command -v raft`.
+
+For an editable source installation, updating the checkout changes the installed
+tool's source. Keep uncommitted work separate and run `uv sync --locked` for the
+chosen checkout. Raft has no `self-update` command.
+
+## Redeploy each host
+
+Obtain a separate checkout of the matching release:
+
+```sh title="CONTROLLER: Obtain and deploy matching host code"
+git clone --branch v0.2.0 https://github.com/Microck/raft.git raft-v0.2.0
+cd raft-v0.2.0
+python3 deploy/deploy.py --location lab
+raft doctor
+```
+
+Repeat for each configured location. The deployer uses your existing
+`~/.config/raft/incus.json`, verifies owned project/pool/network/profile state,
+replaces host scripts and units, reapplies firewall rules, and starts the expiry
+timer. It does not import an image or migrate guest files.
+
+Do not bypass a conflicting-resource or storage-size error. Inspect the host and
+preserve backups before explicit recovery; redeployment does not repair arbitrary
+infrastructure drift. Disposable creation/fork and prune check the worker protocol
+and fail if it differs from the CLI. That check is not an automatic upgrade.
+
+## Replace the image pin
+
+Follow [Acquire a native workspace image](../operations.mdx#acquire-a-native-workspace-image)
+to download and verify or build the image for each host's architecture. Retrieve
+its full fingerprint and replace only that location's `image` field in the
+controller inventory. Keep the previous fingerprint if you may need it again.
+
+Changing the pin affects ordinary `raft new` calls. Resume retains the existing
+filesystem; forks and `new --from` use their source filesystem, not the new image.
+Install guest updates explicitly inside an existing box, or create a new box and
+transfer the project and needed data. Recovering an old backup also restores that
+backup's guest filesystem rather than rebuilding it from the new image.
+
+## Verify the update
+
+```sh title="CONTROLLER: Smoke-check a new persistent box"
+box=$(raft new --location lab --ttl 600)
+raft exec "$box" -- node --version
+raft exec "$box" -- docker info
+raft stop "$box"
+raft resume "$box" --ttl 600
+raft exec "$box" -- ls /workspace
+raft destroy "$box"
+```
+
+Creation can return before guest services are ready. Use the readiness pattern in
+[Scripting and CI](./scripting-ci.mdx) if an immediate check fails during startup.
+For dedicated test infrastructure, [Verification](../verification.mdx) describes
+the real-host suites and their effects.
+
+To revert an image choice for future boxes, restore the previous fingerprint
+while that image remains imported. This does not revert existing boxes. Raft has
+no automatic migration or rollback of host policy or guest packages; review the
+chosen release and backups before downgrading those components.

package/docs/index.mdx

@@ -27,10 +27,14 @@ Stopping a workspace preserves files and packages on disk. Resuming resets its r
 
 | Task | Guide |
 | :--- | :--- |
+| Prepare a project and reuse its installed dependencies | [Project setup and templates](./guides/project-setup.mdx) |
 | Run nested containers and build images | [Docker in workspaces](./guides/docker.mdx) |
 | Transfer files between controller and workspace | [File transfer](./guides/files.mdx) |
 | Run detached commands with systemd logs | [Background jobs](./guides/background-jobs.mdx) |
+| Manage deadlines and restart services after resume | [Lifetime and persistent services](./guides/long-running-services.mdx) |
+| Automate the CLI and parse its output | [Scripting and CI](./guides/scripting-ci.mdx) |
 | Forward guest ports or open noVNC desktop | [Desktop and private services](./guides/desktop-services.mdx) |
 | Capture disk state or clone workspaces | [Snapshots and forks](./guides/snapshots-forks.mdx) |
 | Export portable backups across hosts | [Backup and recovery](./guides/backup-recovery.mdx) |
+| Update the CLI, host worker, and workspace image | [Upgrades and image changes](./guides/upgrades.mdx) |
 | Inspect host services and diagnose errors | [Troubleshooting](./guides/troubleshooting.mdx) |