@microck/raft 0.0.0-stage → 0.2.0

package/docs/guides/troubleshooting.mdx

@@ -0,0 +1,127 @@
+---
+title: Troubleshooting
+description: Diagnose configuration, host infrastructure, network filtering, and container lifecycle issues.
+---
+
+# Troubleshooting
+
+Diagnostic procedures and solutions for common deployment, configuration, and runtime issues.
+
+## Controller diagnostics
+
+### Preflight checks with `raft doctor`
+
+Run `raft doctor` on your controller to inspect host health across configured locations:
+
+```sh title="CONTROLLER: Run environment health check"
+raft doctor
+```
+
+`raft doctor` connects to each host over SSH and reports:
+- Incus version running in project `raft`
+- Active status of host systemd services (`incus`, `raft-expire.timer`, `raft-network.service`)
+- Host root filesystem disk usage (`df -h /`)
+- Status and usage of the `raft-data` Btrfs pool
+- Count of existing saved containers
+
+<Callout type="info" title="Doctor scope">
+`raft doctor` reports runtime daemon and storage status. It does not validate image fingerprints, inspect nftables bridge rules, enforce admission caps, or emit 5 GiB low-disk warnings.
+</Callout>
+
+### Listing active and stopped containers
+
+Inspect workspace states across locations:
+
+```sh title="CONTROLLER: List workspaces across all hosts"
+raft list
+
+# Filter by a single location
+raft list --location lab
+```
+
+If a host is unreachable over SSH, the CLI reports a connection failure. An unreachable host does not mean containers are absent or deleted.
+
+## Host-side diagnostics
+
+Connect to the host over SSH to inspect underlying services when commands fail.
+
+### Inspecting Incus project status
+
+Verify container states and storage pools in project `raft`:
+
+```sh title="HOST: Inspect containers and storage pools"
+sudo incus --project raft list
+sudo incus --project raft storage info raft-data
+```
+
+### Checking host systemd units
+
+Raft installs two systemd units on the host:
+- `raft-network.service`: Applies firewall and bridge isolation rules before Incus starts.
+- `raft-expire.timer` / `raft-expire.service`: Runs the expiration worker every 15 seconds to stop expired workspaces.
+
+Inspect unit logs with `journalctl`:
+
+```sh title="HOST: Check Raft systemd unit logs"
+sudo systemctl status raft-network.service
+sudo systemctl status raft-expire.timer
+sudo journalctl -u raft-expire.service -n 50 --no-pager
+```
+
+<Callout type="warn" title="Host journals and guest secrets">
+Host system journals can capture command lines, arguments, and environment variables executed in guest containers. Treat host logs as sensitive records that may contain guest credentials.
+</Callout>
+
+### Manually triggering garbage collection
+
+The host timer stops expired containers every 15 seconds. Trigger an immediate expiration run from the controller:
+
+```sh title="CONTROLLER: Run expiration workers manually"
+raft gc
+```
+
+Expired containers transition from `Running` to `Stopped`. Expiration never deletes container files or snapshots.
+
+## Common error scenarios
+
+### Configuration errors
+
+#### Missing configuration file error
+If `~/.config/raft/incus.json` does not exist, the CLI raises `FileNotFoundError`. Create the configuration directory and file. Set mode `0600` on the file only.
+
+#### "Configure at least one host in ~/.config/raft/incus.json"
+The configuration file contains valid JSON that is `{}` or is not an object (an empty file raises a JSON decoding error instead). Ensure the file defines at least one location with a non-empty `ssh` string.
+
+#### "Build an image and configure its full immutable fingerprint before creating boxes"
+The `image` field is missing or is not a 64-character hexadecimal SHA-256 fingerprint. Supply the full fingerprint from `sudo incus --project raft image list raft-dev --format json`.
+
+### Admission and capacity errors
+
+#### "Location already holds four boxes"
+Raft enforces a hard limit of four saved boxes per location. Both running and stopped containers count toward this limit. Destroy unused boxes with `raft destroy <box>` before creating, forking, or recovering a box.
+
+#### Low disk warnings
+`raft limits` warns when free space on host disk or the `raft-data` pool falls below 5 GiB. This warning is diagnostic and does not halt container operations. Clean up dangling images, old snapshots, or unused containers to free space.
+
+### Network and forwarding issues
+
+#### Port forward connects but returns connection refused
+Verify that the guest service binds to `0.0.0.0` or the container's `eth0` IP address. Services bound strictly to `127.0.0.1` inside the container cannot receive traffic routed through the bridge.
+
+#### Workspaces cannot reach other workspaces on the same host
+This is intended behavior. Raft configures nftables bridge filtering on `rfbr0` to prevent communication between workspaces on the same bridge.
+
+### Host crash during recovery
+
+If the controller crashes or SSH drops while `raft recover` is transferring an archive:
+1. Check the `Recovery handle: <location>:<name>` output printed to stderr when `raft recover` started.
+2. Inspect the reported handle with `raft list --location <location>` and confirm container status before removing staging.
+3. Identify the exact staging file (`/tmp/raft-recovery.XXXXXXXX`) on the host by owner and timestamp.
+4. After confirming container status, remove only that owned file:
+   ```sh title="HOST: Remove verified owned staging file"
+   sudo -n rm -f -- /tmp/raft-recovery.XXXXXXXX
+   ```
+
+<Callout type="warn" title="Do not delete by wildcard">
+Never run `rm /tmp/raft-recovery.*`. It can delete files from another recovery.
+</Callout>

package/docs/incus.example.json

@@ -0,0 +1,3 @@
+{
+  "lab": {"ssh": "your-ssh-host", "image": ""}
+}

package/docs/index.mdx

@@ -0,0 +1,36 @@
+---
+title: Overview
+description: Persistent Linux workspaces powered by Incus.
+---
+
+# Raft
+
+Raft is an independent, self-hosted open-source alternative to [boat.dev](https://boat.dev).
+It provides persistent Linux workspaces on ARM64 and AMD64 hardware,
+supporting command execution, Docker, file transfer, snapshots, forks, and desktop access.
+Raft is not affiliated with Boat.
+
+## Getting started
+
+Read [System requirements](./requirements.mdx), follow [Installation and host setup](./operations.mdx), and create your [First working box](./first-box.mdx).
+
+```sh title="CONTROLLER: Create and use a workspace"
+box=$(raft new --location lab --ttl 600)
+raft exec "$box" -- node --version
+raft ssh "$box"
+raft stop "$box"
+```
+
+Stopping a workspace preserves files and packages on disk. Resuming resets its running lifetime.
+
+## Task guides
+
+| Task | Guide |
+| :--- | :--- |
+| Run nested containers and build images | [Docker in workspaces](./guides/docker.mdx) |
+| Transfer files between controller and workspace | [File transfer](./guides/files.mdx) |
+| Run detached commands with systemd logs | [Background jobs](./guides/background-jobs.mdx) |
+| Forward guest ports or open noVNC desktop | [Desktop and private services](./guides/desktop-services.mdx) |
+| Capture disk state or clone workspaces | [Snapshots and forks](./guides/snapshots-forks.mdx) |
+| Export portable backups across hosts | [Backup and recovery](./guides/backup-recovery.mdx) |
+| Inspect host services and diagnose errors | [Troubleshooting](./guides/troubleshooting.mdx) |

package/docs/meta.json

@@ -0,0 +1,21 @@
+{
+  "title": "Raft",
+  "pages": [
+    "index",
+    "---Start here---",
+    "requirements",
+    "operations",
+    "first-box",
+    "guides",
+    "reference",
+    "contract",
+    "---Comparison---",
+    "boat-capabilities",
+    "boot-performance",
+    "---Contributing---",
+    "boat-parity-design",
+    "verification",
+    "publishing",
+    "site-development"
+  ]
+}

package/docs/operations.mdx

@@ -0,0 +1,224 @@
+---
+title: Installation and host setup
+description: Install the Raft controller CLI, provision a dedicated host, and acquire the native workspace image.
+---
+
+# Installation and host setup
+
+Set up a dedicated Ubuntu host, install the Raft CLI, provision Incus, acquire a workspace image, and verify readiness.
+
+Review prerequisites in [System requirements](./requirements.mdx) before starting.
+
+<Steps>
+
+<Step>
+
+### Install prerequisites on the workspace host
+
+Connect to your host over SSH, verify its host key fingerprint, and install baseline packages:
+
+```sh title="HOST: Install baseline system packages"
+sudo apt-get update
+sudo apt-get install -y --no-install-recommends \
+  ca-certificates curl gnupg python3 openssh-server iptables btrfs-progs
+```
+
+Verify passwordless `sudo` on the host:
+
+```sh title="HOST: Verify passwordless sudo"
+sudo -n true && echo "Passwordless sudo is functional"
+```
+
+</Step>
+
+<Step>
+
+### Clone repository and install controller CLI
+
+On your controller machine, clone the repository and install the `raft` CLI tool using `uv`:
+
+```sh title="CONTROLLER: Clone repository and install CLI"
+git clone https://github.com/Microck/raft.git raft
+cd raft
+uv sync --locked
+uv tool install --editable .
+```
+
+Add `~/.local/bin` to `PATH` and verify the CLI:
+
+```sh title="CONTROLLER: Verify CLI installation"
+export PATH="$HOME/.local/bin:$PATH"
+raft --help
+```
+
+</Step>
+
+<Step>
+
+### Configure host inventory on the controller
+
+The controller reads host definitions from `~/.config/raft/incus.json`. Initialize the configuration directory and template:
+
+```sh title="CONTROLLER: Initialize configuration file"
+mkdir -p ~/.config/raft
+cp docs/incus.example.json ~/.config/raft/incus.json
+chmod 600 ~/.config/raft/incus.json
+```
+
+Verify interactive SSH to establish known host keys:
+
+```sh title="CONTROLLER: Verify host SSH access"
+ssh user@host-address "echo Host SSH connected successfully"
+ssh -o BatchMode=yes user@host-address "echo Batch SSH functional"
+```
+
+Edit `~/.config/raft/incus.json`. Set the `ssh` target. Leave `image` empty during initial setup:
+
+```json title="CONTROLLER: ~/.config/raft/incus.json"
+{
+  "lab": {
+    "ssh": "user@host-address",
+    "image": ""
+  }
+}
+```
+
+Location names use letters, digits, underscores, or hyphens. The first configured location is the default for `raft new`.
+
+</Step>
+
+<Step>
+
+### Provision the dedicated host
+
+Run the automated host deployer from your controller:
+
+```sh title="CONTROLLER: Provision host infrastructure"
+python3 deploy/deploy.py --location lab
+```
+
+<details>
+<summary>Infrastructure created by the deployer</summary>
+
+The deployer runs over SSH and configures the host:
+1. Installs Incus 6.0 LTS and nftables packages.
+2. Creates the isolated `raft` Incus project.
+3. Creates a 60 GiB Btrfs storage pool (`raft-data`).
+4. Creates private bridge interface `rfbr0` (`10.232.0.1/24`) with NAT.
+5. Configures nftables bridge filters to block inter-workspace traffic and restrict host access.
+6. Installs and enables `raft-network.service` and `raft-expire.timer` units.
+
+</details>
+
+</Step>
+
+<Step>
+
+### Acquire a native workspace image
+
+Workspaces require an unprivileged Debian 13 development image matching the host architecture:
+
+<Tabs items={["Download release image", "Build from source"]}>
+
+<Tab value="Download release image">
+
+Download the release archive, checksum, and manifest from [Raft releases](https://github.com/Microck/raft/releases). The `sha256sum` check gates import so corrupted archives are rejected:
+
+<Tabs items={["ARM64", "AMD64"]}>
+
+<Tab value="ARM64">
+
+```sh title="HOST (ARM64): Download, verify, and import release image"
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/raft-dev-arm64.tar.gz
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/SHA256SUMS
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/manifest-arm64.json
+
+sha256sum --ignore-missing -c SHA256SUMS && \
+  sudo incus --project raft image import ./raft-dev-arm64.tar.gz --alias raft-dev
+```
+
+</Tab>
+
+<Tab value="AMD64">
+
+```sh title="HOST (AMD64): Download, verify, and import release image"
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/raft-dev-amd64.tar.gz
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/SHA256SUMS
+curl -fsSL -O https://github.com/Microck/raft/releases/download/v0.1.0/manifest-amd64.json
+
+sha256sum --ignore-missing -c SHA256SUMS && \
+  sudo incus --project raft image import ./raft-dev-amd64.tar.gz --alias raft-dev
+```
+
+</Tab>
+
+</Tabs>
+
+</Tab>
+
+<Tab value="Build from source">
+
+Run the build script from your controller. It pulls Debian 13 to the host, installs development runtimes and Docker, and publishes alias `raft-dev`:
+
+```sh title="CONTROLLER: Build native development image"
+python3 deploy/build-image.py --location lab
+```
+
+<Callout type="warn" title="Do not pin the base image fingerprint">
+During the build, the script downloads an upstream Debian base image and logs its intermediate fingerprint. Do not copy that initial fingerprint; retrieve and pin the final `raft-dev` image fingerprint in the next step.
+</Callout>
+
+</Tab>
+
+</Tabs>
+
+</Step>
+
+<Step>
+
+### Retrieve and pin the image fingerprint
+
+Query the published `raft-dev` image on the host:
+
+```sh title="HOST: Retrieve published image fingerprint"
+sudo incus --project raft image list raft-dev --format json | python3 -c '
+import json, sys
+data = json.load(sys.stdin)
+print(next(img["fingerprint"] for img in data if any(a.get("name") == "raft-dev" for a in img.get("aliases", []))))
+'
+```
+
+Copy the 64-character hex output into `image` in `~/.config/raft/incus.json` on the controller:
+
+```json title="CONTROLLER: ~/.config/raft/incus.json"
+{
+  "lab": {
+    "ssh": "user@host-address",
+    "image": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
+  }
+}
+```
+
+Replace the example string with your actual fingerprint. Raft requires this immutable fingerprint to prevent unexpected image mutations when creating workspaces.
+
+</Step>
+
+<Step>
+
+### Check host status with doctor
+
+Run `raft doctor` on your controller to inspect host readiness:
+
+```sh title="CONTROLLER: Run doctor diagnostics"
+raft doctor
+```
+
+Check that Incus, the network service, and the expiration timer are active. For the full report contents, see [Controller diagnostics](./guides/troubleshooting.mdx#controller-diagnostics).
+
+</Step>
+
+</Steps>
+
+## Next step
+
+Proceed to [First working box](./first-box.mdx) to launch, verify, and manage your first workspace.

package/docs/publishing.md

@@ -0,0 +1,48 @@
+---
+title: Publishing
+---
+
+# Publishing a clean source release
+
+Public releases must contain only reusable code, examples, and documentation. Keep controller configuration, SSH keys, instance archives, journals, and private reports outside the repository.
+
+## Release assets and image publishing
+
+Release `main` through the `native images` workflow with a semantic `release_tag`, such as `v0.2.0`. The tag must match both Python and npm versions. Clean upstream Debian bases provide the images.
+
+The workflow builds and inspects Python and npm packages, tests their installed executables outside the checkout, and publishes `@microck/raft` using the repository's `NPM_TOKEN` secret with npm provenance. Only then does it create the GitHub release with the wheel, source archive, npm tarball and verified image assets. Published versions are immutable; after partial publication, fix the failure and use a new patch version.
+
+```sh
+gh workflow run images.yml --ref main -f release_tag=v0.2.0
+```
+
+- **Validation gating**: Publication requires native lifecycle and full-image recovery jobs alongside fresh AMD64 KVM provisioning and reboot verification.
+- **Asset immutability**: The workflow refuses existing tags and never overwrites release assets.
+- **Provenance records**: Release assets include `SHA256SUMS`, architecture manifests, package inventories, and boot reports. Manifests identify the source commit, CI run, and Incus image fingerprint.
+
+## Repository history hygiene
+
+Scan **all Git history**, branches, and tags before changing repository visibility. A clean working tree does not remove sensitive data from commit history.
+
+- **Manual audit**: Secret scanners do not catch every identifier. Inspect old commits for host addresses, usernames, local paths, and private reports.
+- **Fresh repository snapshot**: If commit history contains sensitive data, publish a clean snapshot in a fresh public repository. History rewrites can leave orphaned commits accessible in GitHub cache (see GitHub guidance on [removing sensitive data](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository)).
+- **Keep source private**: Do not make an existing private repository public merely because its current working tree is clean.
+
+## Release verification commands
+
+Run security scanning, linting, formatting checks, and package builds before tagging:
+
+```sh
+gitleaks dir --redact --no-banner .
+gitleaks git --redact --no-banner .
+uv sync --locked
+uvx ruff check raft.py deploy/*.py
+uvx ruff format --check raft.py deploy/*.py
+uv build
+```
+
+## Image distribution rules
+
+Do not upload local workspace backups or development exports as release assets. Distributable images must be built from clean upstream bases, inspected for credentials, and accompanied by fingerprints, manifests, and licenses.
+
+Review the package contents and capability table before release. State container isolation accurately without claiming VM isolation or exact Boat compatibility. Raft is an independent alternative to boat.dev and is not affiliated with that service.

package/docs/reference/capacity-limits.mdx

@@ -0,0 +1,83 @@
+---
+title: Capacity limits and planning
+description: Host resource reserves, sizing profiles, advisory calculations, and admission limits.
+---
+
+# Capacity limits and planning
+
+Raft provides capacity inspection and allocation policies designed to keep dedicated hosts stable while running concurrent workspaces.
+
+## Inspecting capacity with `raft limits`
+
+Run `raft limits` to view host resources and workspace allocation calculations:
+
+```sh title="CONTROLLER: View host capacity report"
+raft limits
+
+# Output JSON for programmatic inspection
+raft limits --location lab --json
+```
+
+The output displays host CPU count, total RAM, configured limits, available RAM, disk headroom, and advisory recommendations across all six sizing profiles:
+
+### Illustrative host sizing
+
+Illustrative totals for 1 CPU / 2 GiB boxes on an empty host, using exact usable RAM values:
+
+| Host CPUs | Host RAM | Recommended running | Enforced saved maximum |
+| ---: | ---: | ---: | ---: |
+| 2 | 8 GiB | 1 | 4 |
+| 4 | 8 GiB | 3 | 4 |
+| 4 | 16 GiB | 3 | 4 |
+| 8 | 32 GiB | 4 | 4 |
+
+Operating systems report less usable RAM than advertised hardware size. Near sizing boundaries, `raft limits` may calculate fewer recommended boxes. Available RAM and host processes can further reduce running capacity. These values are point-in-time advisory estimates, not hardware reservations.
+
+### Column definitions and algorithms
+
+| Column | Calculation |
+| --- | --- |
+| Total running | `min(4, floor(CPU budget / box CPU), floor(Memory budget / box memory))` |
+| More running | Bounded by remaining CPU, memory, available RAM, and active slots |
+| New boxes | `min(more running, max(0, 4 - saved boxes))` |
+
+**Total running** assumes an empty host. **More running** accounts for current allocations, `MemAvailable - Memory reserve`, and `4 - active boxes`. Active limits are not subtracted twice.
+
+## Sizing profiles
+
+Select CPU and memory limits when launching with `raft new`:
+
+- `--cpu`: `1` or `2` shared vCPUs managed by Incus (default: `1`).
+- `--memory`: `1GiB`, `2GiB`, or `4GiB` cgroup limit (default: `2GiB`).
+
+Resizing existing containers or modifying limits upon resume is not supported.
+
+## Host resource reserves and active states
+
+Raft calculates advisory host safety margins:
+- **CPU reserve**: 1 CPU core reserved for the host OS.
+- **Memory reserve**: The larger of `2 GiB` or `10%` of host RAM.
+
+These reserves are advisory estimates for planning, not hard cgroup locks.
+
+All non-`Stopped` states (such as `Running` or `Frozen`) count as active allocations. Stopped containers release CPU and RAM allocations while retaining their saved slot and disk storage.
+
+## Enforced admission limits
+
+<Callout type="warn" title="Hard limit: four saved boxes per location">
+Raft enforces a hard limit of four saved boxes per location. Both running and stopped containers occupy storage and count against this limit.
+</Callout>
+
+Exceeding four saved boxes fails with an admission error:
+```text
+Location already holds four boxes
+```
+
+Destroy unused containers with `raft destroy <box>` to free slots. Advisory CPU and memory calculations do not block container creation or resumption.
+
+## Storage capacity
+
+Every Raft host provisions a dedicated 60 GiB Btrfs pool (`raft-data`):
+- **Shared pool**: All containers, snapshots, and Docker layers share this 60 GiB pool.
+- **No strict per-box quotas**: Nested subvolumes prevent strict per-box quota enforcement.
+- **Disk warnings**: `raft limits` warns when free space on host disk or `raft-data` falls below 5 GiB. Warnings are diagnostic and do not block operations. `raft doctor` reports storage usage without emitting 5 GiB warnings.