@microck/raft 0.0.0-stage → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/assets/raft-logo.png +0 -0
- package/LICENSE +21 -0
- package/README.md +167 -2
- package/deploy/benchmark-boot.py +150 -0
- package/deploy/build-image.py +95 -0
- package/deploy/deploy.py +176 -0
- package/deploy/incus-network.conf +4 -0
- package/deploy/install-incus.sh +22 -0
- package/deploy/raft-expire.py +62 -0
- package/deploy/raft-expire.service +8 -0
- package/deploy/raft-expire.timer +8 -0
- package/deploy/raft-network.service +12 -0
- package/deploy/raft-network.sh +40 -0
- package/deploy/verify-controls.py +241 -0
- package/deploy/verify-features.py +519 -0
- package/deploy/verify-fresh-host.py +301 -0
- package/deploy/verify-image-tests.py +139 -0
- package/deploy/verify-image.py +95 -0
- package/deploy/verify-limits.py +110 -0
- package/deploy/verify-live.py +903 -0
- package/deploy/verify-release.py +132 -0
- package/deploy/verify-tools.py +105 -0
- package/docs/benchmarks/amd64-native-ci.json +137 -0
- package/docs/benchmarks/arm64-baseline-a.json +108 -0
- package/docs/benchmarks/arm64-baseline-b.json +108 -0
- package/docs/benchmarks/arm64-candidate-a.json +109 -0
- package/docs/benchmarks/arm64-desktop-local.json +22 -0
- package/docs/benchmarks/arm64-native-ci.json +137 -0
- package/docs/boat-capabilities.md +64 -0
- package/docs/boat-parity-design.md +26 -0
- package/docs/boot-performance.mdx +102 -0
- package/docs/contract.md +93 -0
- package/docs/first-box.mdx +86 -0
- package/docs/guides/background-jobs.mdx +95 -0
- package/docs/guides/backup-recovery.mdx +111 -0
- package/docs/guides/desktop-services.mdx +90 -0
- package/docs/guides/docker.mdx +84 -0
- package/docs/guides/files.mdx +91 -0
- package/docs/guides/long-running-services.mdx +109 -0
- package/docs/guides/meta.json +18 -0
- package/docs/guides/project-setup.mdx +112 -0
- package/docs/guides/scripting-ci.mdx +133 -0
- package/docs/guides/snapshots-forks.mdx +107 -0
- package/docs/guides/troubleshooting.mdx +152 -0
- package/docs/guides/upgrades.mdx +113 -0
- package/docs/incus.example.json +3 -0
- package/docs/index.mdx +40 -0
- package/docs/meta.json +21 -0
- package/docs/operations.mdx +239 -0
- package/docs/publishing.md +48 -0
- package/docs/reference/capacity-limits.mdx +92 -0
- package/docs/reference/cli.md +288 -0
- package/docs/reference/configuration.mdx +77 -0
- package/docs/reference/meta.json +10 -0
- package/docs/releases/0.2.0.md +22 -0
- package/docs/releases/0.2.1.md +14 -0
- package/docs/requirements.mdx +47 -0
- package/docs/reviews/code-review-2026-10-06.md +168 -0
- package/docs/site-development.md +80 -0
- package/docs/verification.mdx +237 -0
- package/images/incus/build.sh +57 -0
- package/images/incus/desktop.sh +21 -0
- package/images/incus/raft-desktop.service +7 -0
- package/package.json +46 -4
- package/raft.py +961 -0
- package/raft_files.py +134 -0
- package/skills/raft-cli/SKILL.md +117 -0
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Project setup and templates
|
|
3
|
+
description: Upload or clone a project, run setup scripts, and create reusable same-host snapshot templates.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Project setup and templates
|
|
7
|
+
|
|
8
|
+
Raft starts a clean development image. Prepare a project with ordinary guest
|
|
9
|
+
commands, then save a snapshot to reuse installed dependencies. There is no
|
|
10
|
+
managed repository registry, environment registry, setup hook, or secret-injection
|
|
11
|
+
service. Commands run as guest root in `/workspace` unless you choose another user.
|
|
12
|
+
|
|
13
|
+
## Put a project in the box
|
|
14
|
+
|
|
15
|
+
Create a persistent box after [host setup](../operations.mdx):
|
|
16
|
+
|
|
17
|
+
```sh title="CONTROLLER: Create a project box"
|
|
18
|
+
box=$(raft new --location lab --ttl 3600)
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Upload a local project, excluding credentials and files you do not want copied.
|
|
22
|
+
The example assumes `./my-project` contains your project and `setup.sh`:
|
|
23
|
+
|
|
24
|
+
```sh title="CONTROLLER: Upload local project"
|
|
25
|
+
raft upload "$box" ./my-project /workspace/my-project --recursive
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
The destination directory must not exist. For a public repository, an alternative
|
|
29
|
+
is guest `git clone`; replace the example URL with your repository:
|
|
30
|
+
|
|
31
|
+
```sh title="CONTROLLER: Clone instead of uploading"
|
|
32
|
+
raft exec "$box" -- git clone https://github.com/OWNER/REPOSITORY.git /workspace/my-project
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Choose one method. Private repositories require authentication configured inside
|
|
36
|
+
the guest; Raft does not forward your controller's GitHub login or SSH agent.
|
|
37
|
+
The clean image does not bundle coding agents or their credentials. Install and
|
|
38
|
+
authenticate any agent inside the box using that tool's own instructions.
|
|
39
|
+
|
|
40
|
+
## Run setup explicitly
|
|
41
|
+
|
|
42
|
+
Use `bash -lc` for directory changes or shell syntax:
|
|
43
|
+
|
|
44
|
+
```sh title="CONTROLLER: Run project setup"
|
|
45
|
+
raft exec "$box" -- bash -lc 'cd /workspace/my-project && bash ./setup.sh'
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Make setup scripts safe to rerun. Raft does not execute them automatically on
|
|
49
|
+
create, resume, or fork. A failing synchronous command returns its exit code;
|
|
50
|
+
see [Scripting and CI](./scripting-ci.mdx).
|
|
51
|
+
|
|
52
|
+
For slow setup, use a detached job and inspect its result:
|
|
53
|
+
|
|
54
|
+
```sh title="CONTROLLER: Run slow setup instead of synchronous setup"
|
|
55
|
+
job=$(raft exec "$box" --detach -- bash -lc 'cd /workspace/my-project && bash ./setup.sh')
|
|
56
|
+
raft status "$box" "$job"
|
|
57
|
+
raft logs "$box" "$job"
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Wait for successful completion before making a template. Extending the lifetime
|
|
61
|
+
may be necessary; a detached job does not prevent expiry.
|
|
62
|
+
|
|
63
|
+
## Supply application configuration
|
|
64
|
+
|
|
65
|
+
For an application that reads a local `.env` file, upload it explicitly and set
|
|
66
|
+
permissions for the application's user. This example assumes a controller file
|
|
67
|
+
`./app.env` and an application running as `developer`:
|
|
68
|
+
|
|
69
|
+
```sh title="CONTROLLER: Install a private application configuration file"
|
|
70
|
+
raft upload "$box" ./app.env /workspace/my-project/.env
|
|
71
|
+
raft exec "$box" -- chown developer:developer /workspace/my-project/.env
|
|
72
|
+
raft exec "$box" -- chmod 600 /workspace/my-project/.env
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
The application must load that file; upload does not export environment variables.
|
|
76
|
+
If running the whole project as `developer`, adjust project ownership separately.
|
|
77
|
+
Do not place secrets in command arguments, committed files, or logs. Host journals
|
|
78
|
+
can capture guest command lines. Raft does not rotate or scrub credentials.
|
|
79
|
+
|
|
80
|
+
## Save and reuse a prepared template
|
|
81
|
+
|
|
82
|
+
Finish setup, remove credentials that should not be inherited, and stop the box:
|
|
83
|
+
|
|
84
|
+
```sh title="CONTROLLER: Save the prepared filesystem"
|
|
85
|
+
raft stop "$box"
|
|
86
|
+
raft snapshot "$box" prepared
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Create another box from the snapshot:
|
|
90
|
+
|
|
91
|
+
```sh title="CONTROLLER: Start a copy with explicit resources"
|
|
92
|
+
copy=$(raft new --from "$box/prepared" --ttl 3600 --cpu 2 --memory 4GiB)
|
|
93
|
+
raft exec "$copy" -- ls -la /workspace/my-project
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
The template stays on its source host. Its source box can be running when you
|
|
97
|
+
copy the saved snapshot. `new --from` uses normal creation defaults when sizing
|
|
98
|
+
is omitted: 1 CPU and 2 GiB RAM. The new box has its own handle, MAC address,
|
|
99
|
+
deadline, and saved slot.
|
|
100
|
+
|
|
101
|
+
The snapshot includes packages, project files, credentials, and enabled services.
|
|
102
|
+
It does not include process memory. Changing the original after snapshot creation
|
|
103
|
+
does not update the saved template. Keep its source box and snapshot until you
|
|
104
|
+
no longer need to create copies; destroying the source deletes its snapshots.
|
|
105
|
+
|
|
106
|
+
To copy the stopped box's current filesystem instead, use
|
|
107
|
+
`raft fork "$box" --ttl 3600`. Fork inherits omitted CPU/RAM limits and does not
|
|
108
|
+
copy the source snapshots. See [Snapshots and forks](./snapshots-forks.mdx).
|
|
109
|
+
|
|
110
|
+
For a different host, use [Backup and recovery](./backup-recovery.mdx) with matching
|
|
111
|
+
CPU architecture. For services that start on resume, see
|
|
112
|
+
[Lifetime and persistent services](./long-running-services.mdx).
|
|
@@ -0,0 +1,133 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Scripting and CI
|
|
3
|
+
description: Parse CLI output, preserve command exit codes, wait for readiness, and handle interrupted operations.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Scripting and CI
|
|
7
|
+
|
|
8
|
+
Raft is a controller CLI using host SSH. A Linux CI runner needs Python 3.11.8+,
|
|
9
|
+
OpenSSH, the installed CLI, host inventory, verified SSH host keys, batch
|
|
10
|
+
authentication, and hosts with passwordless sudo. No Raft HTTP API, SDK, or API key
|
|
11
|
+
exists. Give access only to trusted jobs: the configured SSH account has host sudo
|
|
12
|
+
and controls all boxes on that host.
|
|
13
|
+
|
|
14
|
+
## Output formats
|
|
15
|
+
|
|
16
|
+
Raft does not have a global `--json` option. Parse each command's actual format:
|
|
17
|
+
|
|
18
|
+
| Command | stdout |
|
|
19
|
+
| --- | --- |
|
|
20
|
+
| `new`, `fork`, `recover` | One location-qualified handle |
|
|
21
|
+
| `exec --detach` | One `rfcmd-` job ID |
|
|
22
|
+
| `list` | JSONL: one object per box, with `box`, `state`, `stopAt` |
|
|
23
|
+
| `info` | One pretty-printed native Incus metadata JSON object |
|
|
24
|
+
| `usage` | One JSON object containing running resource counters |
|
|
25
|
+
| `limits --json` | One JSON object per location (JSONL across multiple hosts) |
|
|
26
|
+
| `prune` | JSONL candidates or deleted records with `box`, `action` |
|
|
27
|
+
| `status` | systemd `ActiveState`, `SubState`, `ExecMainStatus` property lines |
|
|
28
|
+
| `logs` | Guest journal text |
|
|
29
|
+
| `exec` | Payload stdout/stderr |
|
|
30
|
+
| Other inspection/maintenance commands | Human-readable text or no output |
|
|
31
|
+
|
|
32
|
+
`list` emits nothing when there are no boxes. `stopAt` is the host Unix deadline
|
|
33
|
+
as a string, or an empty string when absent. `info` uses native Incus keys rather
|
|
34
|
+
than a normalized Raft schema. `usage` includes `memoryBytes`, `memoryLimit`,
|
|
35
|
+
`cpuTime`, `effectiveCpus`, `filesystemUsedBytes`, `filesystemAvailableBytes`, and
|
|
36
|
+
`filesystemScope`; disk values describe the shared pool, not an individual box.
|
|
37
|
+
See [Capacity limits](../reference/capacity-limits.mdx) for capacity calculations.
|
|
38
|
+
|
|
39
|
+
These examples require `jq` on the controller:
|
|
40
|
+
|
|
41
|
+
```sh title="CONTROLLER: Parse records without mixing stderr"
|
|
42
|
+
raft list --location lab | jq -r 'select(.state == "Running") | .box'
|
|
43
|
+
raft limits --location lab --json | jq '{location, savedSlots, warnings}'
|
|
44
|
+
raft prune --location lab --older-than 7 | jq -r '.box'
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Do not redirect `2>&1` into machine-readable output. Recovery reports its allocated
|
|
48
|
+
handle on stderr before import finishes; only a successful completion prints the
|
|
49
|
+
final handle on stdout.
|
|
50
|
+
|
|
51
|
+
## Exit codes and job results
|
|
52
|
+
|
|
53
|
+
| Situation | Exit code |
|
|
54
|
+
| --- | --- |
|
|
55
|
+
| Successful control operation | `0` |
|
|
56
|
+
| Synchronous `exec` | Transport/command return code; preserves payload exit status when transport succeeds |
|
|
57
|
+
| Recognized runtime/configuration failure | `1`, with `raft: ...` on stderr |
|
|
58
|
+
| Argument parsing failure | `2`, with argparse diagnostics on stderr |
|
|
59
|
+
| Caught keyboard interruption | `130` |
|
|
60
|
+
|
|
61
|
+
SSH transport failures may return SSH's status rather than the payload's result.
|
|
62
|
+
The foreground tunnel replaces the CLI process with SSH and uses SSH's exit code.
|
|
63
|
+
A zero exit from `status` means the query succeeded; read `ExecMainStatus` for the
|
|
64
|
+
job result and `ActiveState`/`SubState` for whether it finished. Do not interpret a
|
|
65
|
+
running job's initial `ExecMainStatus=0` as completion. See
|
|
66
|
+
[Background jobs](./background-jobs.mdx).
|
|
67
|
+
|
|
68
|
+
## Create, wait, run, and stop
|
|
69
|
+
|
|
70
|
+
`new` returns after Incus starts the box, before every guest service is necessarily
|
|
71
|
+
ready. Probe a harmless command before running setup. This Bash example waits up
|
|
72
|
+
to about a minute plus transport time, executes an uploaded project check, and
|
|
73
|
+
attempts to stop the persistent box on exit. It assumes `./my-project/check.sh`
|
|
74
|
+
exists and the configured host has a free saved slot:
|
|
75
|
+
|
|
76
|
+
```bash title="CONTROLLER or trusted Linux CI runner"
|
|
77
|
+
set -euo pipefail
|
|
78
|
+
box=""
|
|
79
|
+
cleanup() {
|
|
80
|
+
result=$?
|
|
81
|
+
trap - EXIT
|
|
82
|
+
if [ -n "$box" ]; then
|
|
83
|
+
if ! raft stop "$box"; then
|
|
84
|
+
echo "Cleanup failed; inspect $box" >&2
|
|
85
|
+
[ "$result" -ne 0 ] || result=1
|
|
86
|
+
fi
|
|
87
|
+
fi
|
|
88
|
+
exit "$result"
|
|
89
|
+
}
|
|
90
|
+
trap cleanup EXIT
|
|
91
|
+
|
|
92
|
+
box=$(raft new --location lab --ttl 3600)
|
|
93
|
+
printf 'Created %s\n' "$box" >&2
|
|
94
|
+
ready=false
|
|
95
|
+
for attempt in {1..30}; do
|
|
96
|
+
if raft exec "$box" -- true; then
|
|
97
|
+
ready=true
|
|
98
|
+
break
|
|
99
|
+
fi
|
|
100
|
+
sleep 2
|
|
101
|
+
done
|
|
102
|
+
if [ "$ready" != true ]; then
|
|
103
|
+
echo "Guest did not become command-ready: $box" >&2
|
|
104
|
+
exit 1
|
|
105
|
+
fi
|
|
106
|
+
|
|
107
|
+
raft upload "$box" ./my-project /workspace/my-project --recursive
|
|
108
|
+
raft exec "$box" -- bash -lc 'cd /workspace/my-project && bash ./check.sh'
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
The stopped box remains available for inspection and still consumes a saved slot.
|
|
112
|
+
Destroy it after collecting needed files, or preview old stopped boxes with prune.
|
|
113
|
+
For disposable CI boxes, deletion on stop/expiry removes artifacts and snapshots;
|
|
114
|
+
download results before cleanup. A new box or a retry still consumes a saved slot.
|
|
115
|
+
|
|
116
|
+
Command readiness does not mean Docker or a web application is ready. Probe the
|
|
117
|
+
specific dependency (`docker info`, an HTTP health endpoint, or systemd status)
|
|
118
|
+
with its own bounded wait. Extend the deadline before work exceeding the current
|
|
119
|
+
lifetime. This example's retries are only for the harmless readiness probe.
|
|
120
|
+
|
|
121
|
+
## Interrupted operations and retries
|
|
122
|
+
|
|
123
|
+
Raft does not automatically retry commands or provide idempotency keys. If SSH
|
|
124
|
+
drops during creation, resume, import, or execution, the remote operation may have
|
|
125
|
+
already happened. Inspect `raft list`, `raft info`, guest logs, and the host before
|
|
126
|
+
repeating an operation that creates boxes or changes data. A failed create can
|
|
127
|
+
leave a saved box even when no handle reached stdout.
|
|
128
|
+
|
|
129
|
+
For an interrupted recovery, inspect the handle printed on stderr and follow
|
|
130
|
+
[Host crash during recovery](./troubleshooting.mdx#host-crash-during-recovery).
|
|
131
|
+
For uncertain command execution, check its effects before rerunning it; use
|
|
132
|
+
idempotent setup scripts where possible. Do not treat a lost connection as proof
|
|
133
|
+
of deletion or of a command not having run.
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Snapshots and forks
|
|
3
|
+
description: Capture point-in-time filesystem snapshots and fork independent workspace clones.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Snapshots and forks
|
|
7
|
+
|
|
8
|
+
Raft captures filesystem snapshots, rolls back changes, and clones workspaces on the same host.
|
|
9
|
+
|
|
10
|
+
## Target workspace handle
|
|
11
|
+
|
|
12
|
+
Commands in this guide require a stopped workspace:
|
|
13
|
+
|
|
14
|
+
```sh title="CONTROLLER: Select and stop target workspace"
|
|
15
|
+
raft list
|
|
16
|
+
box="lab:rf-a1b2c3d4e5f60718" # Replace with your actual workspace handle
|
|
17
|
+
raft stop "$box"
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Stopped-source requirement
|
|
21
|
+
|
|
22
|
+
<Callout type="warn" title="Workspace must be stopped">
|
|
23
|
+
Creating a snapshot, restoring from a snapshot, or forking requires a stopped container. Operations run under host lock `/run/lock/raft-incus.lock` to prevent concurrent disk modification.
|
|
24
|
+
</Callout>
|
|
25
|
+
|
|
26
|
+
## Creating snapshots
|
|
27
|
+
|
|
28
|
+
Take a point-in-time snapshot of the stopped workspace filesystem:
|
|
29
|
+
|
|
30
|
+
```sh title="CONTROLLER: Create a named snapshot"
|
|
31
|
+
raft snapshot "$box" pre-upgrade
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Snapshot names must use letters, digits, underscores, or hyphens up to 64 characters.
|
|
35
|
+
|
|
36
|
+
## Listing snapshots
|
|
37
|
+
|
|
38
|
+
View existing snapshots for a workspace:
|
|
39
|
+
|
|
40
|
+
```sh title="CONTROLLER: List existing snapshots"
|
|
41
|
+
raft snapshots "$box"
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
The command prints snapshot names and creation timestamps from Incus metadata.
|
|
45
|
+
|
|
46
|
+
## Restoring from a snapshot
|
|
47
|
+
|
|
48
|
+
Roll back a stopped workspace to a previously recorded snapshot:
|
|
49
|
+
|
|
50
|
+
```sh title="CONTROLLER: Restore snapshot"
|
|
51
|
+
raft restore "$box" pre-upgrade
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
<Callout type="info" title="Configuration preservation during restore">
|
|
55
|
+
`raft restore` executes Incus snapshot restore with `--diskonly`. It rolls back filesystem blocks while preserving container configuration, CPU and memory limits, and the `eth0` network MAC address.
|
|
56
|
+
</Callout>
|
|
57
|
+
|
|
58
|
+
Resume the workspace with a renewed lifetime:
|
|
59
|
+
|
|
60
|
+
```sh title="CONTROLLER: Resume restored workspace"
|
|
61
|
+
raft resume "$box" --ttl 600
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
## Forking a workspace
|
|
65
|
+
|
|
66
|
+
Forking creates an independent clone on the same host:
|
|
67
|
+
|
|
68
|
+
```sh title="CONTROLLER: Fork workspace into a new clone"
|
|
69
|
+
copy=$(raft fork "$box" --ttl 600)
|
|
70
|
+
echo "Created fork: $copy"
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
### Characteristics of forked workspaces
|
|
74
|
+
|
|
75
|
+
- **Instance-only copy**: Uses Incus `--instance-only`. Filesystem data, installed packages, and `/workspace` files are copied; source snapshots are not inherited.
|
|
76
|
+
- **Immediate start**: `raft fork` launches the new container immediately with the specified TTL while the source remains stopped.
|
|
77
|
+
- **Fresh network identity**: The fork receives a new container name, a fresh MAC address on `rfbr0`, and an independent IP address.
|
|
78
|
+
- **Capacity accounting**: The fork counts toward the location's four-saved-box limit and uses storage from the shared 60 GiB Btrfs pool.
|
|
79
|
+
|
|
80
|
+
Destroy the clone when finished to release capacity:
|
|
81
|
+
|
|
82
|
+
```sh title="CONTROLLER: Destroy forked workspace"
|
|
83
|
+
raft destroy "$copy"
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
## Starting from a prepared snapshot
|
|
87
|
+
|
|
88
|
+
```sh
|
|
89
|
+
copy=$(raft new --from "$box/pre-upgrade" --ttl 600)
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
The source box can be running because Raft copies the saved snapshot. The new box stays on the same host, uses normal creation sizing defaults and has a fresh network identity. Filesystem secrets are copied too.
|
|
93
|
+
|
|
94
|
+
Remove a snapshot you no longer need:
|
|
95
|
+
|
|
96
|
+
```sh
|
|
97
|
+
raft snapshot-delete "$box" pre-upgrade
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
## Choosing clone resources
|
|
101
|
+
|
|
102
|
+
```sh
|
|
103
|
+
copy=$(raft fork "$box" --ttl 600 --cpu 2 --memory 4GiB)
|
|
104
|
+
raft resume "$box" --ttl 600 --memory 2GiB
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Omitted sizing values keep the source limits. Forks are persistent by default; add `--disposable` to delete a clone when it stops or expires.
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Troubleshooting
|
|
3
|
+
description: Diagnose configuration, host infrastructure, network filtering, and container lifecycle issues.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Troubleshooting
|
|
7
|
+
|
|
8
|
+
Diagnostic procedures and solutions for common deployment, configuration, and runtime issues.
|
|
9
|
+
|
|
10
|
+
## Controller diagnostics
|
|
11
|
+
|
|
12
|
+
### Preflight checks with `raft doctor`
|
|
13
|
+
|
|
14
|
+
Run `raft doctor` on your controller to inspect host health across configured locations:
|
|
15
|
+
|
|
16
|
+
```sh title="CONTROLLER: Run environment health check"
|
|
17
|
+
raft doctor
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
`raft doctor` connects to each host over SSH and reports:
|
|
21
|
+
- Incus version running in project `raft`
|
|
22
|
+
- Compatibility of the installed expiry worker with the CLI
|
|
23
|
+
- Active status of host systemd services (`incus`, `raft-expire.timer`, `raft-network.service`)
|
|
24
|
+
- Host root filesystem disk usage (`df -h /`)
|
|
25
|
+
- Status and usage of the `raft-data` Btrfs pool
|
|
26
|
+
- Count of existing saved containers
|
|
27
|
+
|
|
28
|
+
<Callout type="info" title="Doctor scope">
|
|
29
|
+
`raft doctor` reports runtime daemon, expiry-worker compatibility, and storage status. It does not validate image fingerprints, inspect nftables bridge rules, enforce admission caps, or emit 5 GiB low-disk warnings.
|
|
30
|
+
</Callout>
|
|
31
|
+
|
|
32
|
+
### Expiry worker differs from the CLI
|
|
33
|
+
|
|
34
|
+
An active expiration timer can still run an older worker. Installing or updating
|
|
35
|
+
the controller CLI does not deploy host code. `doctor` checks the same worker
|
|
36
|
+
protocol as disposable creation and retention cleanup, without running expiry or
|
|
37
|
+
changing any boxes. It exits nonzero and names the location on a mismatch.
|
|
38
|
+
|
|
39
|
+
Review the host deployment changes, then deploy from the checkout matching your
|
|
40
|
+
CLI. Replace `lab` with the location reported by the diagnostic:
|
|
41
|
+
|
|
42
|
+
```sh title="CONTROLLER: Explicitly update a reviewed host deployment"
|
|
43
|
+
python3 deploy/deploy.py --location lab
|
|
44
|
+
raft doctor
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Deployment updates host infrastructure as well as the expiry worker. Do not run
|
|
48
|
+
it on a shared host without authorization. Do not remove the compatibility check
|
|
49
|
+
or mark an old worker as current; older workers may retain expired disposable
|
|
50
|
+
boxes instead of deleting them.
|
|
51
|
+
|
|
52
|
+
### Listing active and stopped containers
|
|
53
|
+
|
|
54
|
+
Inspect workspace states across locations:
|
|
55
|
+
|
|
56
|
+
```sh title="CONTROLLER: List workspaces across all hosts"
|
|
57
|
+
raft list
|
|
58
|
+
|
|
59
|
+
# Filter by a single location
|
|
60
|
+
raft list --location lab
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
If a host is unreachable over SSH, the CLI reports a connection failure. An unreachable host does not mean containers are absent or deleted.
|
|
64
|
+
|
|
65
|
+
## Host-side diagnostics
|
|
66
|
+
|
|
67
|
+
Connect to the host over SSH to inspect underlying services when commands fail.
|
|
68
|
+
|
|
69
|
+
### Inspecting Incus project status
|
|
70
|
+
|
|
71
|
+
Verify container states and storage pools in project `raft`:
|
|
72
|
+
|
|
73
|
+
```sh title="HOST: Inspect containers and storage pools"
|
|
74
|
+
sudo incus --project raft list
|
|
75
|
+
sudo incus --project raft storage info raft-data
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
### Checking host systemd units
|
|
79
|
+
|
|
80
|
+
Raft installs two systemd units on the host:
|
|
81
|
+
- `raft-network.service`: Applies firewall and bridge isolation rules before Incus starts.
|
|
82
|
+
- `raft-expire.timer` / `raft-expire.service`: Runs the expiration worker every 15 seconds to stop expired workspaces.
|
|
83
|
+
|
|
84
|
+
Inspect unit logs with `journalctl`:
|
|
85
|
+
|
|
86
|
+
```sh title="HOST: Check Raft systemd unit logs"
|
|
87
|
+
sudo systemctl status raft-network.service
|
|
88
|
+
sudo systemctl status raft-expire.timer
|
|
89
|
+
sudo journalctl -u raft-expire.service -n 50 --no-pager
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
<Callout type="warn" title="Host journals and guest secrets">
|
|
93
|
+
Host system journals can capture command lines, arguments, and environment variables executed in guest containers. Treat host logs as sensitive records that may contain guest credentials.
|
|
94
|
+
</Callout>
|
|
95
|
+
|
|
96
|
+
### Manually triggering garbage collection
|
|
97
|
+
|
|
98
|
+
The host timer enforces expiry every 15 seconds. Trigger an immediate expiration run from the controller:
|
|
99
|
+
|
|
100
|
+
```sh title="CONTROLLER: Run expiration workers manually"
|
|
101
|
+
raft gc
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Expired persistent containers transition from `Running` to `Stopped` and retain
|
|
105
|
+
files and snapshots. Expired disposable containers are deleted, including their
|
|
106
|
+
snapshots. Use `raft gc --location lab` to check only one host. The timer normally
|
|
107
|
+
checks every 15 seconds; host outages, graceful shutdown and long locked
|
|
108
|
+
operations can delay expiry. See [Lifetime and persistent services](./long-running-services.mdx).
|
|
109
|
+
|
|
110
|
+
## Common error scenarios
|
|
111
|
+
|
|
112
|
+
### Configuration errors
|
|
113
|
+
|
|
114
|
+
#### Missing configuration file error
|
|
115
|
+
If `~/.config/raft/incus.json` does not exist, the CLI raises `FileNotFoundError`. Create the configuration directory and file. Set mode `0600` on the file only.
|
|
116
|
+
|
|
117
|
+
#### "Configure at least one host in ~/.config/raft/incus.json"
|
|
118
|
+
The configuration file contains valid JSON that is `{}` or is not an object (an empty file raises a JSON decoding error instead). Ensure the file defines at least one location with a non-empty `ssh` string.
|
|
119
|
+
|
|
120
|
+
#### "Build an image and configure its full immutable fingerprint before creating boxes"
|
|
121
|
+
The `image` field is missing or is not a 64-character hexadecimal SHA-256 fingerprint. Supply the full fingerprint from `sudo incus --project raft image list raft-dev --format json`.
|
|
122
|
+
|
|
123
|
+
### Admission and capacity errors
|
|
124
|
+
|
|
125
|
+
#### "Location already holds four boxes"
|
|
126
|
+
Raft enforces a hard limit of four saved boxes per location. Both running and stopped containers count toward this limit. Destroy unused boxes with `raft destroy <box>` before creating, forking, or recovering a box.
|
|
127
|
+
|
|
128
|
+
#### Low disk warnings
|
|
129
|
+
`raft limits` warns when free space on host disk or the `raft-data` pool falls below 5 GiB. This warning is diagnostic and does not halt container operations. Clean up dangling images, old snapshots, or unused containers to free space.
|
|
130
|
+
|
|
131
|
+
### Network and forwarding issues
|
|
132
|
+
|
|
133
|
+
#### Port forward connects but returns connection refused
|
|
134
|
+
Verify that the guest service binds to `0.0.0.0` or the container's `eth0` IP address. Services bound strictly to `127.0.0.1` inside the container cannot receive traffic routed through the bridge.
|
|
135
|
+
|
|
136
|
+
#### Workspaces cannot reach other workspaces on the same host
|
|
137
|
+
This is intended behavior. Raft configures nftables bridge filtering on `rfbr0` to prevent communication between workspaces on the same bridge.
|
|
138
|
+
|
|
139
|
+
### Host crash during recovery
|
|
140
|
+
|
|
141
|
+
If the controller crashes or SSH drops while `raft recover` is transferring an archive:
|
|
142
|
+
1. Check the `Recovery handle: <location>:<name>` output printed to stderr when `raft recover` started.
|
|
143
|
+
2. Inspect the reported handle with `raft list --location <location>` and confirm container status before removing staging.
|
|
144
|
+
3. Identify the exact staging file (`/tmp/raft-recovery.XXXXXXXX`) on the host by owner and timestamp.
|
|
145
|
+
4. After confirming container status, remove only that owned file:
|
|
146
|
+
```sh title="HOST: Remove verified owned staging file"
|
|
147
|
+
sudo -n rm -f -- /tmp/raft-recovery.XXXXXXXX
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
<Callout type="warn" title="Do not delete by wildcard">
|
|
151
|
+
Never run `rm /tmp/raft-recovery.*`. It can delete files from another recovery.
|
|
152
|
+
</Callout>
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Upgrades and image changes
|
|
3
|
+
description: Update the controller CLI and host worker, replace image fingerprints, and understand existing-box behavior.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Upgrades and image changes
|
|
7
|
+
|
|
8
|
+
The controller CLI, host deployment scripts, and workspace image are separate
|
|
9
|
+
parts of Raft. Updating one does not automatically update the others.
|
|
10
|
+
|
|
11
|
+
| Component | Update method | Effect on existing boxes |
|
|
12
|
+
| --- | --- | --- |
|
|
13
|
+
| Controller CLI | Install the chosen npm or Python release | Changes controller commands; does not rewrite guest files |
|
|
14
|
+
| Host worker and firewall | Run the chosen release's deployer per host | Replaces host policy code and reapplies filtering |
|
|
15
|
+
| Pinned development image | Import an image and change the configuration fingerprint | Used for future image-based `new`; existing boxes keep their files |
|
|
16
|
+
|
|
17
|
+
## Choose a release and prepare
|
|
18
|
+
|
|
19
|
+
Review [release notes](https://github.com/Microck/raft/releases) and use matching
|
|
20
|
+
CLI and deployment sources. The examples below pin `v0.2.0`; replace that version
|
|
21
|
+
consistently when adopting another release.
|
|
22
|
+
|
|
23
|
+
```sh title="CONTROLLER: Inspect the current installation"
|
|
24
|
+
raft --version
|
|
25
|
+
raft doctor
|
|
26
|
+
raft list
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Save a copy of your controller inventory. Export important persistent boxes using
|
|
30
|
+
[Backup and recovery](./backup-recovery.mdx) before host changes. Backup requires a
|
|
31
|
+
stopped box. Stopping a disposable box deletes it, so copy needed files out before
|
|
32
|
+
stopping one. Schedule redeployment when interruptions are acceptable.
|
|
33
|
+
|
|
34
|
+
## Update the controller
|
|
35
|
+
|
|
36
|
+
For an npm installation:
|
|
37
|
+
|
|
38
|
+
```sh title="CONTROLLER: Install a pinned CLI release"
|
|
39
|
+
npm install -g @microck/raft@0.2.0
|
|
40
|
+
raft --version
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
For a Python installation, download the wheel and `SHA256SUMS` from the same
|
|
44
|
+
release, verify the checksum, then install it with `uv`:
|
|
45
|
+
|
|
46
|
+
```sh title="CONTROLLER: Install a verified release wheel instead of npm"
|
|
47
|
+
sha256sum --ignore-missing -c SHA256SUMS
|
|
48
|
+
uv tool install --force ./raft-0.2.0-py3-none-any.whl
|
|
49
|
+
raft --version
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Proceed only if checksum verification succeeds. Choose one installation method
|
|
53
|
+
and check which executable your shell uses with `command -v raft`.
|
|
54
|
+
|
|
55
|
+
For an editable source installation, updating the checkout changes the installed
|
|
56
|
+
tool's source. Keep uncommitted work separate and run `uv sync --locked` for the
|
|
57
|
+
chosen checkout. Raft has no `self-update` command.
|
|
58
|
+
|
|
59
|
+
## Redeploy each host
|
|
60
|
+
|
|
61
|
+
Obtain a separate checkout of the matching release:
|
|
62
|
+
|
|
63
|
+
```sh title="CONTROLLER: Obtain and deploy matching host code"
|
|
64
|
+
git clone --branch v0.2.0 https://github.com/Microck/raft.git raft-v0.2.0
|
|
65
|
+
cd raft-v0.2.0
|
|
66
|
+
python3 deploy/deploy.py --location lab
|
|
67
|
+
raft doctor
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Repeat for each configured location. The deployer uses your existing
|
|
71
|
+
`~/.config/raft/incus.json`, verifies owned project/pool/network/profile state,
|
|
72
|
+
replaces host scripts and units, reapplies firewall rules, and starts the expiry
|
|
73
|
+
timer. It does not import an image or migrate guest files.
|
|
74
|
+
|
|
75
|
+
Do not bypass a conflicting-resource or storage-size error. Inspect the host and
|
|
76
|
+
preserve backups before explicit recovery; redeployment does not repair arbitrary
|
|
77
|
+
infrastructure drift. Disposable creation/fork and prune check the worker protocol
|
|
78
|
+
and fail if it differs from the CLI. That check is not an automatic upgrade.
|
|
79
|
+
|
|
80
|
+
## Replace the image pin
|
|
81
|
+
|
|
82
|
+
Follow [Acquire a native workspace image](../operations.mdx#acquire-a-native-workspace-image)
|
|
83
|
+
to download and verify or build the image for each host's architecture. Retrieve
|
|
84
|
+
its full fingerprint and replace only that location's `image` field in the
|
|
85
|
+
controller inventory. Keep the previous fingerprint if you may need it again.
|
|
86
|
+
|
|
87
|
+
Changing the pin affects ordinary `raft new` calls. Resume retains the existing
|
|
88
|
+
filesystem; forks and `new --from` use their source filesystem, not the new image.
|
|
89
|
+
Install guest updates explicitly inside an existing box, or create a new box and
|
|
90
|
+
transfer the project and needed data. Recovering an old backup also restores that
|
|
91
|
+
backup's guest filesystem rather than rebuilding it from the new image.
|
|
92
|
+
|
|
93
|
+
## Verify the update
|
|
94
|
+
|
|
95
|
+
```sh title="CONTROLLER: Smoke-check a new persistent box"
|
|
96
|
+
box=$(raft new --location lab --ttl 600)
|
|
97
|
+
raft exec "$box" -- node --version
|
|
98
|
+
raft exec "$box" -- docker info
|
|
99
|
+
raft stop "$box"
|
|
100
|
+
raft resume "$box" --ttl 600
|
|
101
|
+
raft exec "$box" -- ls /workspace
|
|
102
|
+
raft destroy "$box"
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Creation can return before guest services are ready. Use the readiness pattern in
|
|
106
|
+
[Scripting and CI](./scripting-ci.mdx) if an immediate check fails during startup.
|
|
107
|
+
For dedicated test infrastructure, [Verification](../verification.mdx) describes
|
|
108
|
+
the real-host suites and their effects.
|
|
109
|
+
|
|
110
|
+
To revert an image choice for future boxes, restore the previous fingerprint
|
|
111
|
+
while that image remains imported. This does not revert existing boxes. Raft has
|
|
112
|
+
no automatic migration or rollback of host policy or guest packages; review the
|
|
113
|
+
chosen release and backups before downgrading those components.
|
package/docs/index.mdx
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Overview
|
|
3
|
+
description: Persistent Linux workspaces powered by Incus.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Raft
|
|
7
|
+
|
|
8
|
+
Raft is an independent, self-hosted open-source alternative to [boat.dev](https://boat.dev).
|
|
9
|
+
It provides persistent Linux workspaces on ARM64 and AMD64 hardware,
|
|
10
|
+
supporting command execution, Docker, file transfer, snapshots, forks, and desktop access.
|
|
11
|
+
Raft is not affiliated with Boat.
|
|
12
|
+
|
|
13
|
+
## Getting started
|
|
14
|
+
|
|
15
|
+
Read [System requirements](./requirements.mdx), follow [Installation and host setup](./operations.mdx), and create your [First working box](./first-box.mdx).
|
|
16
|
+
|
|
17
|
+
```sh title="CONTROLLER: Create and use a workspace"
|
|
18
|
+
box=$(raft new --location lab --ttl 600)
|
|
19
|
+
raft exec "$box" -- node --version
|
|
20
|
+
raft ssh "$box"
|
|
21
|
+
raft stop "$box"
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
Stopping a workspace preserves files and packages on disk. Resuming resets its running lifetime.
|
|
25
|
+
|
|
26
|
+
## Task guides
|
|
27
|
+
|
|
28
|
+
| Task | Guide |
|
|
29
|
+
| :--- | :--- |
|
|
30
|
+
| Prepare a project and reuse its installed dependencies | [Project setup and templates](./guides/project-setup.mdx) |
|
|
31
|
+
| Run nested containers and build images | [Docker in workspaces](./guides/docker.mdx) |
|
|
32
|
+
| Transfer files between controller and workspace | [File transfer](./guides/files.mdx) |
|
|
33
|
+
| Run detached commands with systemd logs | [Background jobs](./guides/background-jobs.mdx) |
|
|
34
|
+
| Manage deadlines and restart services after resume | [Lifetime and persistent services](./guides/long-running-services.mdx) |
|
|
35
|
+
| Automate the CLI and parse its output | [Scripting and CI](./guides/scripting-ci.mdx) |
|
|
36
|
+
| Forward guest ports or open noVNC desktop | [Desktop and private services](./guides/desktop-services.mdx) |
|
|
37
|
+
| Capture disk state or clone workspaces | [Snapshots and forks](./guides/snapshots-forks.mdx) |
|
|
38
|
+
| Export portable backups across hosts | [Backup and recovery](./guides/backup-recovery.mdx) |
|
|
39
|
+
| Update the CLI, host worker, and workspace image | [Upgrades and image changes](./guides/upgrades.mdx) |
|
|
40
|
+
| Inspect host services and diagnose errors | [Troubleshooting](./guides/troubleshooting.mdx) |
|
package/docs/meta.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
{
|
|
2
|
+
"title": "Raft",
|
|
3
|
+
"pages": [
|
|
4
|
+
"index",
|
|
5
|
+
"---Start here---",
|
|
6
|
+
"requirements",
|
|
7
|
+
"operations",
|
|
8
|
+
"first-box",
|
|
9
|
+
"guides",
|
|
10
|
+
"reference",
|
|
11
|
+
"contract",
|
|
12
|
+
"---Comparison---",
|
|
13
|
+
"boat-capabilities",
|
|
14
|
+
"boot-performance",
|
|
15
|
+
"---Contributing---",
|
|
16
|
+
"boat-parity-design",
|
|
17
|
+
"verification",
|
|
18
|
+
"publishing",
|
|
19
|
+
"site-development"
|
|
20
|
+
]
|
|
21
|
+
}
|