@microck/raft 0.0.0-stage → 0.2.1

package/docs/reference/configuration.mdx

@@ -0,0 +1,77 @@
+---
+title: Configuration reference
+description: Schema, fields, and security requirements for controller configuration.
+---
+
+# Configuration reference
+
+The Raft controller CLI reads host inventory and image mappings from `~/.config/raft/incus.json`.
+
+## File location and security
+
+```text
+~/.config/raft/incus.json
+```
+
+Ensure the configuration file is accessible only by your user account:
+
+```sh title="CONTROLLER: Set strict file permissions"
+chmod 600 ~/.config/raft/incus.json
+```
+
+## Schema format
+
+The configuration file maps location names to host definition objects:
+
+```json title="Example ~/.config/raft/incus.json"
+{
+  "lab": {
+    "ssh": "ubuntu@192.0.2.10",
+    "image": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
+  },
+  "remote-builder": {
+    "ssh": "builder.example.com",
+    "image": ""
+  }
+}
+```
+
+### Location keys
+
+- **Naming format**: Matches `^[a-zA-Z0-9][a-zA-Z0-9_-]*$`. Starts with an alphanumeric character and contains ASCII letters, digits, underscores, or hyphens.
+- **Default location**: The first location listed is the default for `raft new`. `raft list`, `raft limits`, `raft gc`, and `raft prune` accept `--location` filters. Without a filter they check all configured hosts; `raft doctor` always checks all configured hosts.
+
+### Host entry fields
+
+| Field | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| `ssh` | `string` | Yes | Non-empty SSH connection string (IP, `user@host`, or SSH host alias from `~/.ssh/config`). |
+| `image` | `string` | For `raft new` only | Full 64-character hexadecimal SHA-256 fingerprint (`^[a-f0-9]{64}$`). An empty string `""` is allowed during initial host provisioning. |
+
+<Callout type="warn" title="Replace placeholder image fingerprint">
+The `image` field requires the full 64-character hash for `raft new`; aliases such as `raft-dev` are rejected. Replace the example hash with the actual imported image fingerprint for that host.
+
+ARM64 and AMD64 images produce different fingerprints; do not reuse an ARM64 hash on an AMD64 host.
+</Callout>
+
+## SSH configuration and connection behavior
+
+Primary remote commands (`raft ssh`, `upload`, `download`, and control commands) run OpenSSH with `BatchMode=yes` and `-o ConnectTimeout=10` to fail fast on unreachable hosts. The timeout applies to initial connection establishment, not session or transfer duration.
+
+Other operations, recovery routines, and tunnels use standard SSH defaults without an explicit timeout flag.
+
+Configure host connections in `~/.ssh/config` for batch authentication:
+
+```text title="~/.ssh/config"
+Host lab-host
+    HostName 192.0.2.10
+    User ubuntu
+    IdentityFile ~/.ssh/id_ed25519
+    IdentitiesOnly yes
+```
+
+Verify passwordless batch SSH before running Raft commands:
+
+```sh title="CONTROLLER: Verify passwordless batch connection"
+ssh -o BatchMode=yes lab-host "echo SSH connection verified"
+```

package/docs/reference/meta.json

@@ -0,0 +1,10 @@
+{
+  "title": "Reference",
+  "collapsible": true,
+  "defaultOpen": false,
+  "pages": [
+    "cli",
+    "configuration",
+    "capacity-limits"
+  ]
+}

package/docs/releases/0.2.0.md

@@ -0,0 +1,22 @@
+---
+title: Raft 0.2.0
+---
+
+# Raft 0.2.0
+
+Install the CLI with `npm install -g @microck/raft` on Linux with Python 3.11.8+, or use the Python wheel/source archive attached to this release. The command remains `raft`.
+
+- Disposable boxes delete themselves on stop or expiry. Persistent boxes keep their files by default.
+- `prune --older-than DAYS` previews stopped boxes eligible for deletion. Add `--yes` to apply it.
+- Create boxes from named snapshots and delete snapshots directly.
+- Upload and download directories, with no-overwrite publication and rejection of unsafe links, special files, incomplete streams and source-replacement races.
+- Resize CPU/RAM on resume or fork. Omitted limits are preserved.
+- `raft --version` reports the installed release.
+
+Redeploy the host worker from this release before using disposable creation or retention cleanup. npm installation does not configure hosts or install Python; follow the [setup guide](https://raft.micr.dev/docs/operations).
+
+ARM64 and AMD64 development images, manifests, package inventories and startup reports accompany this release. Verify `SHA256SUMS` before importing the archive for your host architecture.
+
+Both architectures pass native lifecycle, desktop, development-tool, capacity, directory-transfer and full-image recovery checks. Fresh provisioning, image import, firewall ordering and reboot pass on a disposable AMD64 host.
+
+Raft is an independent open-source alternative to boat.dev. Containers share the host kernel. Physical storage capacity, ARM64 host reboot and cross-architecture recovery are outside the verified scope.

package/docs/releases/0.2.1.md

@@ -0,0 +1,14 @@
+---
+title: Raft 0.2.1
+---
+
+# Raft 0.2.1
+
+- `raft doctor` now rejects an incompatible host expiry worker and identifies the location that needs deployment. It does not run expiry or change boxes.
+- Lifecycle transactions close controller stdin so allocation cannot wait for piped input while holding the host lock.
+
+Install on Linux with Python 3.11.8+ using `npm install -g @microck/raft@0.2.1`, or install the attached Python wheel. Updating the CLI does not update host workers. Deploy the matching checkout to hosts that fail the readiness check.
+
+Exec, SSH, and file-transfer behavior is unchanged. Piped input to `raft exec` is unsupported.
+
+ARM64 and AMD64 development images accompany this release. Verify `SHA256SUMS` before importing an image for your host architecture.

package/docs/requirements.mdx

@@ -0,0 +1,47 @@
+---
+title: System requirements
+description: Hardware, operating system, network, and tool requirements for running Raft.
+---
+
+# System requirements
+
+Raft coordinates unprivileged Linux containers across two environments: dedicated Incus workspace hosts, and a controller running the Raft CLI.
+
+## Workspace host requirements
+
+Each workspace host must be a dedicated physical machine or virtual instance:
+
+| Component | Requirement | Notes |
+| :--- | :--- | :--- |
+| **Architecture** | ARM64 or AMD64 | Workspaces run native binaries matching the host CPU architecture. |
+| **Operating system** | Ubuntu 22.04 LTS or 24.04 LTS | The deployer installs Incus 6.0 LTS and nftables bridge isolation rules. |
+| **CPU** | At least 2 CPU cores | Capacity calculations reserve 1 CPU core for the host OS and services (an advisory estimate, not an enforced host reservation). |
+| **RAM** | At least 8 GiB | Sizing calculations reserve the larger of 2 GiB or 10% of host RAM (an advisory estimate, not an enforced host reservation). |
+| **Free disk** | At least 80 GiB on host disk | The deployer provisions a 60 GiB dedicated Btrfs pool (`raft-data`). |
+
+Host CPU and RAM reserves are advisory estimates for `raft limits`, not enforced cgroup locks.
+
+<Callout type="warn" title="Dedicated host requirement">
+Do not deploy Raft on hosts with existing Incus storage pools or shared virtualization workloads. Deployment configures dedicated nftables rules, storage pools, and system services.
+</Callout>
+
+### Host access and permissions
+
+- **SSH access**: SSH access with passwordless `sudo` (`sudo -n true`).
+- **Security model**: Single operator with root access. Deploy only on trusted hardware and networks.
+- **Network connectivity**: Outbound internet for Debian packages, Incus releases, and container images. Inbound SSH from the controller. No public Incus listener or cloud account needed.
+
+## Controller machine requirements
+
+The controller is your local workstation, developer machine, or CI runner:
+
+| Component | Requirement | Notes |
+| :--- | :--- | :--- |
+| **Operating system** | Linux | Controller CLI commands invoke standard Linux utilities. |
+| **Python** | Python 3.11.8 or newer | Required to run the `raft` CLI tool. |
+| **Toolchain** | `git`, `uv`, OpenSSH client (`ssh`), `scp` | `uv` manages the Python environment; OpenSSH provides the transport. |
+| **Network** | Network route to host SSH targets | Batch SSH commands must connect without interactive password prompts. |
+
+## Next step
+
+Proceed to [Installation and setup](./operations.mdx) to install the CLI, configure inventory, provision Incus, and acquire a workspace image.

package/docs/reviews/code-review-2026-10-06.md

@@ -0,0 +1,168 @@
+# Full-code review: 2026-10-06
+
+Reviewed snapshot: `198bb1dfad60630b47a8fe40f4cd69e7d5c60162`.
+Scope: the full CLI, deployment scripts, image build, verification tools, services,
+workflows, packaging and agent skill. Two independent read-only reviews used
+`docs/contract.md` as the specification and the applicable coding instructions as
+standards. Findings below refer to this pinned snapshot, before documentation-site changes.
+
+This is a code review, not a claim that every runtime path passed a new E2E run.
+No host configuration changed during the original review.
+The six distinct findings have since been fixed; see the follow-up verification below.
+
+## Standards
+
+### P1: stopped-state checks happen outside the lifecycle lock
+
+Locations: [raft.py](../../raft.py), snapshot line 661, restore line 669,
+fork line 675 in the reviewed snapshot.
+
+Each command checks the previously fetched box status before acquiring the host
+lifecycle lock. A concurrent resume can start the source before snapshot, restore
+or copy runs. This violates the documented stopped-source requirement and the
+correctness rule that validation and enforcement must agree.
+
+Check the current state inside the same locked transaction as the operation.
+The backup path already follows this approach. Evidence: code inspection of the
+status read, branch and lock boundary; the race was not reproduced on a live host.
+
+### P1: repeat deployment does not apply changed firewall rules
+
+Locations: [deploy/deploy.py](../../deploy/deploy.py), lines 145-146;
+[raft-network.service](../../deploy/raft-network.service), line 9.
+
+Deployment overwrites the firewall script, then calls `systemctl enable --now`.
+The oneshot service has `RemainAfterExit=yes`. Starting an already active service
+does not rerun its script, so updated rules remain unapplied while deployment
+reports that the private bridge is ready. Apply and verify the installed rules
+on redeployment. Evidence: code and unit inspection; no host service was restarted.
+
+### P1: firewall ordering does not gate guest autostart
+
+Location: [raft-network.service](../../deploy/raft-network.service), line 3.
+
+The firewall service runs after Incus. Incus can restore running guests before
+filtering is installed. No dependency gates guest startup on the firewall.
+The existing reboot checks wait for the firewall to become active before probing,
+so they cannot detect exposure during startup. Install persistent filtering before
+guest autostart or explicitly gate startup on it. This is a structural risk found
+by inspection, not a demonstrated boot-time exploit.
+
+### P2: native E2E workflow misses deployment and expiry changes
+
+Location: [images.yml](../../.github/workflows/images.yml), push path filter.
+
+Changes to `deploy/deploy.py`, `deploy/raft-expire.py` and the expiry service/timer
+do not trigger native fresh-host, reboot and TTL checks. Package checks do not
+exercise those host behaviors. Include these sources in the native workflow's
+trigger list. Evidence: comparison of source ownership with the workflow paths.
+
+No separate heuristic refactor was warranted by the review.
+
+## Spec
+
+### P1: stopped-source requirement can be bypassed by concurrent resume
+
+Locations: [raft.py](../../raft.py), snapshot, restore and fork branches.
+
+Contract: “Snapshots, restores and forks require a stopped source for consistent disk contents.”
+
+The stopped check is outside the lifecycle lock. A concurrent resume can invalidate
+it before disk operations begin. Validate stopped state inside the locked operation.
+Evidence: code inspection; no live race experiment.
+
+### P1: redeployment can leave the network contract unenforced
+
+Locations: [deploy/deploy.py](../../deploy/deploy.py) and
+[raft-network.service](../../deploy/raft-network.service).
+
+Contract: “firewall rules reject guest access to host services, cloud metadata, private address ranges.”
+
+`enable --now` does not reapply changed firewall rules when the persistent oneshot
+is already active. A redeployed host can retain obsolete filtering. Reapply and
+verify rules before reporting success. Evidence: code and service inspection.
+
+### P2: clean-image inspection misses non-root SSH credentials
+
+Location: [verify-image.py](../../deploy/verify-image.py), credential check near line 35.
+
+Contract: “The development image omits ... account credentials.”
+
+The inspector rejects `rootfs/root/.ssh/` but accepts a private-key path such as
+a private key beneath a non-root user’s `.ssh` directory. It also skips nonregular archive entries
+before inspecting credential paths. A real temporary tar fixture containing the
+non-root private-key path passed verification and printed `Clean image verified`.
+This proves a gap in the inspector, not that a published image contains credentials.
+Inspect SSH credential paths across user homes and relevant archive entry types.
+
+### P2: an oversized existing owned pool passes deployment validation
+
+Location: [deploy/deploy.py](../../deploy/deploy.py), pool validation near line 108.
+
+Contract: “Btrfs storage is bounded to a 60 GiB pool per host.”
+
+Validation checks the Btrfs driver but not the configured pool size. An existing
+owned pool configured above 60 GiB passes the shown checks. Reject contract drift
+with explicit recovery instructions rather than silently altering storage.
+Evidence: code inspection, not a destructive live pool-resize test.
+
+## Counts
+
+Standards: four findings, with P1 lifecycle consistency and firewall enforcement
+as the highest severity. Spec: four findings, with P1 lifecycle consistency and
+firewall enforcement as the highest severity. The two axes intentionally retain
+overlapping findings rather than combining their counts.
+
+## Documentation-site follow-up
+
+The new site received separate standards and spec follow-up reviews. Three site
+issues were corrected before delivery: bare Markdown links opening raw text,
+a repository workflow link resolving to an absent local route, and missing
+`Vary: Accept` on HTML responses. Canonical document links now begin with `./`,
+source-file links use GitHub, and both negotiated representations set the header.
+
+Verification: production build, strict TypeScript and Oxlint passed. A real
+production-server test checked nine pages, 16 local link targets, search results,
+Markdown exports, JSON downloads, the root redirect and unknown-page 404s.
+Browser checks covered desktop and 390-pixel mobile layouts, sidebar navigation,
+Ctrl+K search and result navigation. Computed body colors were black and white;
+each page had one H1 and browser error inspection was empty. Wide parity tables
+scroll within their container rather than widening the mobile page.
+
+The simplification pass ran in the parent context across reuse, quality and
+efficiency. No behavior-preserving refactor was warranted. No runtime review
+finding above was fixed at that stage, and site checks did not modify
+workspace hosts.
+
+## Runtime-fix follow-up
+
+All six distinct findings are addressed by the follow-up patch. Both review axes
+reported no additional actionable findings after inspecting the fixes and tests.
+
+| Finding | Fix | Verification |
+| --- | --- | --- |
+| Stopped-source race | Ownership and state are checked inside the same locked transaction as snapshot, restore, fork and backup | Real queued snapshot/restore/fork operations reject a source started first under the host lock on two ARM64 hosts |
+| Stale rules on redeploy | Reloadable firewall oneshot explicitly reapplies owned chains transactionally, preserving unrelated chains and the running Incus daemon | Isolated first/repeat application passed; fresh-host CI checks injected stale-rule removal, unrelated-chain retention and unchanged daemon start timestamp |
+| Guest autostart before filtering | Incus requires the firewall and starts after successful installation, including socket activation | Installed dependency verified on two hosts; fresh-host CI tests failed filtering and post-reboot start timestamps |
+| Missing native CI triggers | All `deploy/**` changes trigger the native workflow | Workflow path inspection; the fix push triggers both native architectures and the fresh-host job |
+| Non-root image credentials | Check credential paths and link targets before skipping nonregular entries | 15 real tar cases passed, including generated non-root private keys, symlinks, hardlinks, FIFO entries and clean controls |
+| Oversized owned storage | Reject an existing pool whose configured size differs from `60GiB`, with recovery guidance and no resize | Existing 60 GiB pools pass; fresh-host CI grows a disposable pool to 61 GiB and verifies rejection without resizing |
+
+The extended lifecycle suite passed on two configured ARM64 hosts. Each completed
+all 31 developer-tool checks, six sizing combinations, the deterministic race
+cases, Docker, files, jobs, snapshots, forks, restore, private tunnels, desktop,
+network restrictions and scheduled TTL stop/resume. Each removed its fixtures.
+A cross-host control run also passed portable backup/recovery, snapshot rollback,
+permissions, cancellation and usage checks with a 385,265,470-byte archive.
+Both hosts had zero remaining test boxes afterward. Lint, formatting, compilation
+and release archive/privacy checks passed.
+
+The [fix verification run](https://github.com/Microck/raft/actions/runs/37496577945)
+passed native ARM64 and AMD64 builds, extended lifecycle with daemon restart,
+capacity transitions, full development-image backup/recovery, startup measurements
+and clean image inspection for commit `27b08c143dc1837d9cfb95f3c3a784318879b284`.
+The disposable AMD64 KVM host passed stale-rule removal without an Incus restart,
+failed-firewall rejection for normal and socket startup, real host reboot with
+filtering before Incus, and rejection of a 61 GiB pool without resizing it.
+ARM64 host reboot remains unverified. Package and documentation-site workflows
+also passed. No published release assets or configured image fingerprints changed.

package/docs/site-development.md

@@ -0,0 +1,80 @@
+---
+title: Documentation site
+---
+
+# Documentation site
+
+The Fumadocs site in `website/` compiles Markdown and MDX files in
+`docs/` directly. Edit those files to update both GitHub documentation and the site.
+`docs/meta.json` and directory `meta.json` files control sidebar order and groups. Review reports stay outside the site navigation.
+
+## Run locally
+
+Use Node.js 24 and pnpm 11.25.0. Python 3 serves the local static preview.
+
+```sh
+cd website
+pnpm install --frozen-lockfile
+pnpm dev --hostname 127.0.0.1 --port 4300
+```
+
+Open `http://127.0.0.1:4300/docs/`. The site includes navigation, full-text search,
+code highlighting and a table of contents. Light mode is the default; the theme switch also offers dark mode.
+
+## Check and build
+
+```sh
+pnpm lint
+pnpm types:check
+pnpm build
+pnpm test
+pnpm start
+```
+
+### Build output
+
+The build copies benchmark reports and example configuration from `docs/` into the static export. Generated assets and Next.js output stay outside version control. The site is a separate package from the Python CLI.
+
+### Site tests
+
+`pnpm test` checks the exported pages, links, search, Markdown downloads, and 404s over controller loopback. It does not contact workspace hosts.
+
+Set `RAFT_DOCS_TEST_PORT` if port 4391 is occupied.
+
+## Machine-readable documentation
+
+- `/llms.txt` lists the Markdown files for every documentation page.
+- `/llms-full.txt` exports the complete documentation.
+- `/llms.mdx/docs/contract/content.md` exports the contract as Markdown.
+
+GitHub Pages serves fixed files. Use the explicit Markdown URLs; documentation
+pages return HTML regardless of the `Accept` header.
+
+## Hosting
+
+The public site is [raft.micr.dev](https://raft.micr.dev/). GitHub Pages serves
+`website/out/` at the domain root. Search runs in the browser using a build-time index.
+
+- Pull requests build and test the site. Pushes to `main` deploy the verified export.
+- Set the repository Pages publishing source to **GitHub Actions**.
+- For a fork, replace `website/public/CNAME` and the `metadataBase` URL in
+  `website/app/layout.tsx` with your domain, then configure Pages and DNS.
+
+For a local production preview, `pnpm start` serves the same export at
+`http://127.0.0.1:4300/`. Any static HTTP host can serve the exported files.
+Publishing the documentation does not provide public
+ingress to Raft workspaces.
+
+## Typography
+
+| Element | Font |
+| --- | --- |
+| Page titles | Funnel Display |
+| Body text | Inter |
+| Code blocks | Monospace |
+
+The pairing follows boat.dev. Upright Latin WOFF2 files from Fontsource live in `website/public/fonts/`; their SIL Open Font Licenses are in `boat-fonts-license.txt`.
+
+`next/font/local` serves the bundled fonts without third-party requests.
+
+Light mode uses Boat's white background, dark text, and thin borders. The theme switch also offers dark mode.

package/docs/verification.mdx

@@ -0,0 +1,237 @@
+---
+title: Verification
+---
+
+# Verification
+
+Raft is verified using real disposable Incus workspaces, guest processes, files, Docker builds, and network connections. Tests contain no mocked services and verify Raft behavior rather than conformance with the hosted Boat service.
+
+## Verification boundaries
+
+Tests do not certify arbitrary long-running workloads, host loss, multi-tenant security boundaries, or exact Boat image/version parity. ARM64 host reboot and physical storage capacity are not certified by the AMD64 VM test. Containers share the host kernel and do not offer VM isolation.
+
+Storage accounting covers shared pools rather than per-box quotas. Large backup transfers hold the host lifecycle lock, which delays expiration checks.
+
+## Coverage matrix
+
+| Implemented workflow | E2E assertion |
+| --- | --- |
+| Create/list/info/delete | Qualified handle, native state, ownership and cleanup |
+| Root terminal/exec | Real PTY, workspace directory, literal arguments and exit code |
+| Resource sizing | Effective CPU affinity and cgroup memory limit |
+| Host capacity | Mixed-size budgets, stopped/frozen boxes, low RAM/disk and four saved slots; live create/stop/resume/destroy changes in CPU/memory allocations |
+| Background jobs | Journal output, nonzero exit status, cancellation of child processes, inspection after cancellation or log rotation, and rejection of unknown job IDs |
+| TTL/extend/resume | Host-enforced stop, manual `gc`, retained files, uninterrupted lifetime extension and 30-day resume |
+| Usage | Real cgroup counters and explicitly shared-pool disk scope |
+| File transfer | Binary content, paths containing spaces, stopped-workspace transfers, preservation of the destination after a failed download, and real controller write-error rejection |
+| Docker | Real image build and container execution |
+| Snapshot/restore/fork | File rollback, independent copy, retained network identity and deterministic concurrent-resume rejection under the real host lock |
+| Backup/recovery | Stopped source, no overwrite, snapshots, content, permissions and fresh MAC |
+| Private forwarding | Guest HTTP response through controller loopback tunnel |
+| Desktop | WebSocket/RFB negotiation, framebuffer, pointer, keyboard, clipboard and reconnect |
+| Network separation | Positive service control, negative IPv4/IPv6 peer, host and metadata probes |
+| Development image | Real developer-user tool checks, immutable selection and credential-path archive rejection |
+| Deployment/firewall | Repeat deployment removes stale rules without restarting Incus; failed filtering blocks normal and socket startup; boot timestamps prove filtering precedes Incus |
+| Storage contract | Disposable oversized pool is rejected without automatic resizing |
+| CLI options and errors | Help for all 28 commands; default location, TTL and desktop port; all six sizing profiles; global and per-location reports; invalid TTLs, sizes, required arguments, handles, paths, snapshot names, jobs and ports |
+
+## Command and option audit (2026-10-06)
+
+<details>
+<summary>Coverage, regressions and remaining limits</summary>
+
+The audit checked help for all 26 commands and expanded the live suites to cover defaults, invalid arguments, manual garbage collection, maximum lifetimes and unknown job IDs. Two ARM64 hosts passed extended lifecycle tests, all six sizing profiles and all 31 development-tool checks.
+
+After the fixes, live lifecycle and same-host recovery passed again. Cross-host recovery passed with a 384,827,472-byte archive. Test boxes and archives were removed without rebooting or reconfiguring the existing hosts.
+
+The audit found and fixed two CLI bugs:
+
+- Unknown background jobs appeared to succeed. Job inspection now distinguishes unknown IDs, loaded jobs and canceled jobs with retained journal entries. Tests cover inspection after cancellation and journal rotation.
+- SSH could report success after a controller write failed. Downloads and backups now copy the stream through Python so local write errors reach the caller. Real `/dev/full` tests verify both receiving paths. The same Incus/SSH producer falsely succeeded with the previous implementation and raised `ENOSPC` with the fix.
+
+This covers implemented workflows and selected option combinations, not every possible workload or environment. A full-image export exceeded the local controller's free disk space and was interrupted. Full-sized archive validation runs on disposable CI hosts with sufficient scratch space.
+
+The [final runtime validation](https://github.com/Microck/raft/actions/runs/37542355027) tests commit `ac618e64`. ARM64 and AMD64 both passed extended lifecycle tests with daemon restarts, capacity transitions and full development-image backup/recovery. [Packaging checks](https://github.com/Microck/raft/actions/runs/37542355021) passed on Python 3.11 and 3.14, and [documentation checks](https://github.com/Microck/raft/actions/runs/37542355020) passed.
+
+</details>
+
+## New workspace controls
+
+`python3 deploy/verify-features.py --location lab` verifies `doctor` readiness against the installed worker, recursive trees and unsafe-archive rejection, templates from a running source, snapshot deletion, CPU/RAM overrides, disposable timer deletion, and retention preview/apply. It uses only its own fixtures. Run the archive checks without `--location` to avoid allocating boxes.
+
+The live suite also holds a controller stdin pipe open while running real remote
+readers through both captured and streamed lifecycle transactions. Both must
+receive EOF without consuming controller input, and a subsequent transaction
+must acquire the same host lock. This covers allocation stalls caused by Incus
+waiting for stdin while the expiration worker waits for the lifecycle lock.
+
+For an older deployment whose services should stay unchanged, `--isolated-worker` first verifies that `doctor` rejects the stale installed worker with a location-specific deployment diagnostic, then uses the current worker under a private temporary path. This checks worker logic without installing it; automatic timer integration is verified on disposable CI hosts. Current deployments use the normal verification mode; the isolated mode rejects them before allocation.
+
+## Workspace controls validation (2026-10-07)
+
+<details>
+<summary>Native feature tests and transfer regressions</summary>
+
+At runtime commit `a7fef1f5`, ARM64 and AMD64 passed the new workspace-control suite in [native CI](https://github.com/Microck/raft/actions/runs/37603576786). It verifies installed-timer deletion, retention preview and exclusions, live-source snapshot templates, snapshot deletion, and effective CPU/RAM limits after resume and fork.
+
+Directory tests cover binary files, permissions, relative symlinks, hardlinks, empty directories and existing destinations. Unix sockets and unsafe archive entries are rejected. A real Incus transfer truncated between complete archive entries fails without publishing a partial directory.
+
+Both architectures passed extended lifecycle tests, full development-image recovery, startup measurements and clean-template inspection. Fresh provisioning, image import, firewall ordering and reboot passed on a disposable AMD64 KVM host.
+
+The source-race fix also passed a real concurrent writer test that swaps files and directories for external symlinks. Accepted archives retain only the selected files. Two configured ARM64 hosts passed the final feature suite with a staged worker, without changing installed services. Local fixtures and staged files were removed.
+
+</details>
+
+## Running verification suites
+
+Test commands require configured hosts and immutable image fingerprints. Suites destroy only the fixtures they create.
+
+Allocation and archive inspection:
+- `python3 deploy/verify-limits.py`: Checks allocation policies without a host. Add `--location lab` to verify live create, stop, resume, and destroy reporting.
+- `python3 deploy/verify-image-tests.py`: Inspects archive structures, including symlinks, hardlinks, FIFOs, and credential detection.
+
+Live host suites (expand the check you need):
+
+<details>
+<summary>Lifecycle</summary>
+
+```sh
+python3 deploy/verify-live.py --location lab --extended
+```
+
+</details>
+
+<details>
+<summary>Daemon restart on a dedicated test host</summary>
+
+```sh
+python3 deploy/verify-live.py --location lab --restart-incus
+```
+
+</details>
+
+<details>
+<summary>Cross-host recovery</summary>
+
+```sh
+python3 deploy/verify-controls.py --source lab --target second-host --directory /path/to/archive-storage
+```
+
+</details>
+
+<details>
+<summary>Same-host recovery</summary>
+
+```sh
+python3 deploy/verify-controls.py --source lab --target lab --directory /path/to/archive-storage
+```
+
+</details>
+
+<details>
+<summary>Full development-image recovery</summary>
+
+```sh
+python3 deploy/verify-controls.py --source lab --target second-host --directory /path/to/archive-storage --development-image
+```
+
+</details>
+
+`verify-tools.py <qualified-box>` executes 31 tool checks as the unprivileged `developer` user inside a running workspace. It compiles and runs code across installed runtimes and package managers without external agent authentication.
+
+## Extended lifecycle results (2026-10-05)
+
+<details>
+<summary>Test evidence and run details</summary>
+
+The extended lifecycle suite and strengthened tool checks passed on two ARM64 Ubuntu hosts with Incus 6.0.6 on 2026-10-05.
+
+Both hosts executed all six sizing combinations, all 31 developer tool checks, Docker builds, file transfers, stopped snapshots, forks, restore, jobs, and private HTTP and desktop tunnels.
+
+Desktop testing validated keyboard input, exact clipboard synchronization, pointer coordinates, framebuffer updates, and secondary client connections. Headed Chromium testing passed with sandboxing enabled, confirming canvas rendering at 1280x720 over the noVNC interface.
+
+**Admission and backup runs**
+
+The four-saved-box admission test passed, including rejection of a fifth create, fork, or recovery attempt. Full development-image recovery succeeded with a 5,254,224,965-byte archive, verifying inventories, tool checks, permissions, fresh MACs, and snapshot rollback.
+
+The execution wrapper reported exit 143 after the success message; independent native inventory and archive checks confirmed that all fixtures and temporary archives were removed.
+
+Debian fixture backup and recovery passed across the following tests:
+- Same-host recovery: 385,278,177-byte archive with snapshot rollback and distinct MAC/IP identities.
+- Reverse cross-host recovery: 385,282,465-byte archive.
+- Forward cross-host admission run: 385,272,379-byte archive (exit code 0).
+
+All recovery fixtures were destroyed; existing operator hosts were not rebooted or reconfigured.
+
+</details>
+
+## Review-fix validation (2026-10-06)
+
+<details>
+<summary>Test evidence and run details</summary>
+
+On 2026-10-06, the extended suite passed on two configured ARM64 hosts after redeploying review fixes. Both ran all 31 tool checks, six sizing profiles, and deterministic race rejection when a source container started before obtaining the host lock. Cross-host recovery passed with a 385,265,470-byte archive.
+
+Both hosts had no remaining test boxes; existing hosts were not rebooted or reconfigured.
+
+The image inspector passed 15 test cases, including non-root SSH keys, symlinks, hardlinks, and clean controls. Network testing verified initial and repeated firewall application while preserving unrelated chains.
+
+The [review-fix native run](https://github.com/Microck/raft/actions/runs/37496577945) passed all native steps on ARM64 and AMD64: development-image builds, extended lifecycle with daemon restarts, capacity transitions, and clean-image inspection. Recovery archives were 3,844,136,311 bytes on ARM64 and 3,985,872,081 bytes on AMD64.
+
+**Disposable KVM host tests**
+
+Fresh provisioning on a disposable AMD64 KVM host confirmed:
+- Repeat deployment removed injected rules while leaving Incus daemon start timestamps unchanged.
+- Failed firewall installation blocked both normal and socket-activated Incus startup.
+- Host reboot verified firewall rules became active before Incus started.
+- A disposable pool resized to 61 GiB caused deployment to fail and preserved the existing size.
+
+ARM64 host reboot and physical disk capacity remain outside this verification.
+
+</details>
+
+## Native image validation (2026-10-06)
+
+<details>
+<summary>Test evidence and run details</summary>
+
+On 2026-10-06, the [native CI run](https://github.com/Microck/raft/actions/runs/37447415532) built ARM64 and AMD64 images on Ubuntu 24 and passed extended lifecycle tests on both. Workspaces executed all six sizing combinations, 31 tool checks, sandboxed Chromium, noVNC desktop sessions, and peer isolation checks. Timings are recorded in [Boot performance](./boot-performance.mdx).
+
+In the [release-validation run](https://github.com/Microck/raft/actions/runs/37469540182), native image jobs passed full development-image recovery on both architectures. Package inventories matched and all 31 tool checks passed. Archives measured 3,844,135,523 bytes on ARM64 and 3,985,894,517 bytes on AMD64.
+
+</details>
+
+## Fresh-host and reboot validation
+
+<details>
+<summary>Test evidence and run details</summary>
+
+The [release run](https://github.com/Microck/raft/actions/runs/37469540182) validated fresh-host setup on a disposable AMD64 KVM instance running Ubuntu 24. The test verified canonical deployment, repeat deployment, image import, and lifecycle execution.
+
+Following a VM reboot, changed kernel boot IDs and active services were verified alongside retained files, snapshots, container states, Docker, and bridge isolation. The host test uses a sparse 100 GiB virtual disk and accessible KVM.
+
+It preserves the production deployer's 80 GiB free-disk check but does not prove physical storage capacity. ARM64 host reboot, host loss, and cross-architecture recovery remain unverified. The VM, SSH keys, and controller configuration were disposable; existing operator hosts were not rebooted or reconfigured.
+
+</details>
+
+## Packaging and release checks
+
+<details>
+<summary>Test evidence and run details</summary>
+
+Package builds run via `uv build`. Source and wheel archives are inspected to exclude version-control files and local caches. Clean wheel installation is verified outside the worktree alongside Ruff, syntax checks, and secret scans.
+
+Packaging workflows passed on Python 3.11 and 3.14. Guidelines for visibility changes and repository hygiene are detailed in [Publishing](./publishing.md).
+
+</details>
+
+## Capacity command validation (2026-10-06)
+
+<details>
+<summary>Test evidence and run details</summary>
+
+On 2026-10-06, `raft limits` passed read-only text and JSON inspection and live lifecycle tests on two ARM64 Ubuntu hosts. Tests verified allocation adjustments during create, stop, resume, and destroy cycles.
+
+Allocation fixtures validated mixed sizing budgets, frozen containers, low RAM and disk conditions, host reserves, and admission limits. The four-stopped-box admission test verified that creation, forking, and recovery reject a fifth container when slots are exhausted. These are conservative allocation checks, not workload saturation benchmarks.
+
+</details>