@crvouga/dev-fleet 0.7.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +28 -3
- package/ansible/macos/manager-privilege.yml +37 -0
- package/ansible/macos/manager.yml +13 -5
- package/ansible/macos/worker-host-key.yml +33 -0
- package/bootstrap/worker-seed.yaml +61 -0
- package/docs/authentication.md +2 -0
- package/docs/consumer-setup.md +32 -31
- package/docs/declared.md +24 -24
- package/docs/manager-setup.md +27 -0
- package/docs/runbooks.md +15 -1
- package/docs/topology.md +15 -6
- package/docs/ubuntu-sync.md +21 -15
- package/package.json +2 -2
- package/schema/check-ids.json +32 -0
- package/schema/fleet.schema.json +22 -2
- package/src/check-owners.ts +1 -0
- package/src/cli/fleet-manifest.ts +102 -0
- package/src/cli/fleet-node-lifecycle.ts +455 -0
- package/src/cli/fleet-node-seed.ts +151 -0
- package/src/cli/fleet-sync-forward.ts +2 -0
- package/src/cli/fleet-sync-manager.ts +50 -5
- package/src/cli/fleet-sync-remote.ts +21 -6
- package/src/cli/fleet-sync-state.ts +12 -3
- package/src/cli/fleet-sync-vault.ts +75 -26
- package/src/cli/fleet-sync.ts +22 -2
- package/src/cli/fleet-validate.ts +239 -0
- package/src/cli/register-fleet.ts +136 -0
- package/src/cli/registry.ts +33 -0
- package/src/definition.ts +444 -237
- package/src/machine/state.ts +33 -1
- package/src/manager-sync.ts +24 -4
- package/src/secrets.ts +3 -0
- package/src/worker-credentials.ts +201 -0
- package/src/worker-manifest.ts +201 -0
- package/src/worker-policy.ts +21 -35
- package/src/worker-seed.ts +138 -0
package/README.md
CHANGED
|
@@ -128,8 +128,10 @@ npm install --save-dev --save-exact @crvouga/dev-fleet@latest
|
|
|
128
128
|
npx devfleet fleet-sync # the fleet moves to that release
|
|
129
129
|
```
|
|
130
130
|
|
|
131
|
-
Install an older version and run `fleet-sync` to go back.
|
|
132
|
-
proves a new revision before the others receive it
|
|
131
|
+
Install an older version and run `fleet-sync` to go back. One worker, the
|
|
132
|
+
canary, proves a new revision before the others receive it: the first
|
|
133
|
+
reachable worker, or the one `canary` in `fleet.json` (or `--canary`) names
|
|
134
|
+
when it is reachable. An offline worker never holds the others back.
|
|
133
135
|
|
|
134
136
|
## Your project's own tools
|
|
135
137
|
|
|
@@ -222,7 +224,26 @@ propagates them without a full sync.
|
|
|
222
224
|
Ubuntu machine, over SSH
|
|
223
225
|
([`fresh-box.feature`](features/fresh-box/fresh-box.feature)).
|
|
224
226
|
|
|
225
|
-
|
|
227
|
+
A new machine can also join by itself: `npx devfleet fleet-node-seed --id ID
|
|
228
|
+
--output seed.yaml --apply --yes` writes cloud-init user data (or an Ubuntu
|
|
229
|
+
autoinstall file with `--autoinstall`) holding the worker account with your
|
|
230
|
+
manager's key, the privilege rule, a host key the manager pins at once and the
|
|
231
|
+
pinned Tailscale with a short-lived single-use auth key from your secrets
|
|
232
|
+
provider (`tailnet-auth-key`). Boot a stock Ubuntu image with it, then add the
|
|
233
|
+
worker with `fleet-node-add` and run `fleet-sync`.
|
|
234
|
+
|
|
235
|
+
`npx devfleet fleet-manifest --id ID` measures what a worker is made of
|
|
236
|
+
(packages, active tools, always-on services) with a digest that is the same
|
|
237
|
+
for two workers built from the same revision; its check `worker:manifest`
|
|
238
|
+
names every difference from the declared pins and, with `--reference` (the
|
|
239
|
+
`--json` manifest of a fresh build), from that build.
|
|
240
|
+
|
|
241
|
+
Remove a machine with `npx devfleet fleet-node-retire --id ID --apply --yes`
|
|
242
|
+
(it revokes the worker's Vault token, forgets its host key and roster entry
|
|
243
|
+
and leaves a tombstone in the fleet state), then delete its entry and run
|
|
244
|
+
`fleet-sync`. To put a fresh machine in place of a worker under the same id,
|
|
245
|
+
run `npx devfleet fleet-node-replace --id ID --host-key "TYPE BASE64" --apply
|
|
246
|
+
--yes` and then `fleet-sync --node ID`.
|
|
226
247
|
|
|
227
248
|
## Configuration
|
|
228
249
|
|
|
@@ -231,6 +252,10 @@ Your `fleet.json` is validated by
|
|
|
231
252
|
package carries: `fleet-state --example` prints a complete example that points
|
|
232
253
|
`$schema` at it, so an editor checks the file as you type.
|
|
233
254
|
[`examples/fleet.json`](examples/fleet.json) is the same example.
|
|
255
|
+
`devfleet fleet-validate` checks the file without contacting anything and
|
|
256
|
+
names the first field that breaks a rule (`{code, path, expected, actual}`);
|
|
257
|
+
`--ansible` also renders the Ansible inventory and syntax-checks every
|
|
258
|
+
playbook with the pinned Ansible.
|
|
234
259
|
|
|
235
260
|
| Block | What it declares |
|
|
236
261
|
| ---------- | ------------------------------------------------------------------- |
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
---
|
|
2
|
+
# The one privilege grant a manager needs, given once by its administrator:
|
|
3
|
+
# ansible-playbook -K manager-privilege.yml -e user=$(id -un)
|
|
4
|
+
# It lets that user run exactly `pmset -c` (the AC power settings
|
|
5
|
+
# manager.yml declares) without a password, so every fleet-sync holds the
|
|
6
|
+
# manager's power policy with no prompt. Run again with -e grant=absent to
|
|
7
|
+
# remove it; nothing else on the manager needs root.
|
|
8
|
+
- name: Grant the fleet manager its power settings without a prompt
|
|
9
|
+
hosts: localhost
|
|
10
|
+
connection: local
|
|
11
|
+
become: true
|
|
12
|
+
gather_facts: false
|
|
13
|
+
vars:
|
|
14
|
+
grant: present
|
|
15
|
+
tasks:
|
|
16
|
+
- name: Check the request
|
|
17
|
+
ansible.builtin.assert:
|
|
18
|
+
that:
|
|
19
|
+
- "user is match('^[a-z_][a-z0-9_.-]{0,31}$')"
|
|
20
|
+
- "user != 'root'"
|
|
21
|
+
- "grant in ['present', 'absent']"
|
|
22
|
+
- name: Allow exactly the AC power settings
|
|
23
|
+
ansible.builtin.copy:
|
|
24
|
+
dest: '/etc/sudoers.d/dev-fleet-manager-{{ user }}'
|
|
25
|
+
content: |
|
|
26
|
+
# Managed by dev-fleet: manager power policy
|
|
27
|
+
{{ user }} ALL=(root) NOPASSWD: /usr/bin/pmset -c *
|
|
28
|
+
owner: root
|
|
29
|
+
group: wheel
|
|
30
|
+
mode: '0440'
|
|
31
|
+
validate: /usr/sbin/visudo -cf %s
|
|
32
|
+
when: grant == 'present'
|
|
33
|
+
- name: Remove the grant
|
|
34
|
+
ansible.builtin.file:
|
|
35
|
+
path: '/etc/sudoers.d/dev-fleet-manager-{{ user }}'
|
|
36
|
+
state: absent
|
|
37
|
+
when: grant == 'absent'
|
|
@@ -2,13 +2,13 @@
|
|
|
2
2
|
# Manager power policy. Workers depend on the manager's singleton services
|
|
3
3
|
# (CLI proxy, Beszel hub), so the manager never sleeps on AC power and starts
|
|
4
4
|
# again by itself when power returns (autorestart). Sleep on battery is left
|
|
5
|
-
# to the human. The settings are the `pmset` object in
|
|
6
|
-
#
|
|
7
|
-
#
|
|
5
|
+
# to the human. The settings are the `pmset` object in policy.json; fleet-sync
|
|
6
|
+
# measures the same object as manager:power and applies this playbook on every
|
|
7
|
+
# run, without a prompt, because `pmset -c` runs through the narrow grant
|
|
8
|
+
# manager-privilege.yml installs once.
|
|
8
9
|
- name: Keep the fleet manager awake on AC power
|
|
9
10
|
hosts: localhost
|
|
10
11
|
connection: local
|
|
11
|
-
become: true
|
|
12
12
|
gather_facts: false
|
|
13
13
|
vars_files:
|
|
14
14
|
- policy.json
|
|
@@ -20,7 +20,15 @@
|
|
|
20
20
|
register: pmset_custom
|
|
21
21
|
- name: Apply the declared AC power settings
|
|
22
22
|
ansible.builtin.command:
|
|
23
|
-
argv:
|
|
23
|
+
argv:
|
|
24
|
+
[
|
|
25
|
+
/usr/bin/sudo,
|
|
26
|
+
-n,
|
|
27
|
+
/usr/bin/pmset,
|
|
28
|
+
-c,
|
|
29
|
+
'{{ item.key }}',
|
|
30
|
+
'{{ item.value }}',
|
|
31
|
+
]
|
|
24
32
|
loop: '{{ pmset | dict2items }}'
|
|
25
33
|
loop_control:
|
|
26
34
|
label: '{{ item.key }}'
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
---
|
|
2
|
+
# A worker's pinned SSH host key in the manager's known_hosts. fleet-sync
|
|
3
|
+
# connects with StrictHostKeyChecking=yes, so this is the only way a key is
|
|
4
|
+
# trusted: devfleet fleet-node-replace pins the key of a replacement worker
|
|
5
|
+
# (given by the operator, or by the seed that created it with
|
|
6
|
+
# fleet-node-seed) and devfleet fleet-node-retire forgets it. A key is never
|
|
7
|
+
# learned on first use. Variables: host (the name ssh connects to, with
|
|
8
|
+
# [host]:port for another port), pin (present or absent), key (with present:
|
|
9
|
+
# "TYPE BASE64") and known_hosts (the file).
|
|
10
|
+
- name: Pin or forget a worker's SSH host key on the manager
|
|
11
|
+
hosts: localhost
|
|
12
|
+
connection: local
|
|
13
|
+
gather_facts: false
|
|
14
|
+
tasks:
|
|
15
|
+
- name: Check the request
|
|
16
|
+
ansible.builtin.assert:
|
|
17
|
+
that:
|
|
18
|
+
- "host is match('^\\[?[a-zA-Z0-9][a-zA-Z0-9._-]{0,252}(\\]:[0-9]{1,5})?$')"
|
|
19
|
+
- "pin in ['present', 'absent']"
|
|
20
|
+
- "pin == 'absent' or (key is match('^(ssh-ed25519|ecdsa-sha2-nistp256|ssh-rsa) [A-Za-z0-9+/=]{16,}$'))"
|
|
21
|
+
- name: Pin the worker's host key
|
|
22
|
+
ansible.builtin.known_hosts:
|
|
23
|
+
name: '{{ host }}'
|
|
24
|
+
key: '{{ host }} {{ key }}'
|
|
25
|
+
path: '{{ known_hosts }}'
|
|
26
|
+
state: present
|
|
27
|
+
when: pin == 'present'
|
|
28
|
+
- name: Forget every key of the retired worker
|
|
29
|
+
ansible.builtin.known_hosts:
|
|
30
|
+
name: '{{ host }}'
|
|
31
|
+
path: '{{ known_hosts }}'
|
|
32
|
+
state: absent
|
|
33
|
+
when: pin == 'absent'
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
# The first boot of a new worker, as cloud-init user data. devfleet
|
|
2
|
+
# fleet-node-seed fills each ${NAME} with a value of the worker it seeds:
|
|
3
|
+
# values from fleet.json, the manager's public key, a host key generated for
|
|
4
|
+
# the machine, the Tailscale version and key digest ansible/ubuntu/policy.json
|
|
5
|
+
# pins, and a short-lived single-use auth key that is deleted once the machine
|
|
6
|
+
# has joined. The sudoers fragment is the one ansible/ubuntu/authorize-sudo.sh
|
|
7
|
+
# writes: its path and marker are already on running machines.
|
|
8
|
+
hostname: '${HOSTNAME}'
|
|
9
|
+
users:
|
|
10
|
+
- name: '${USER}'
|
|
11
|
+
groups: [sudo]
|
|
12
|
+
shell: /bin/bash
|
|
13
|
+
lock_passwd: true
|
|
14
|
+
ssh_authorized_keys: ['${MANAGER_KEY}']
|
|
15
|
+
ssh_pwauth: false
|
|
16
|
+
ssh_deletekeys: true
|
|
17
|
+
ssh_genkeytypes: []
|
|
18
|
+
ssh_keys:
|
|
19
|
+
ed25519_private: '${HOST_KEY_PRIVATE}'
|
|
20
|
+
ed25519_public: '${HOST_KEY_PUBLIC}'
|
|
21
|
+
write_files:
|
|
22
|
+
- path: '/etc/sudoers.d/personal-fleet-${USER}'
|
|
23
|
+
owner: root:root
|
|
24
|
+
permissions: '0440'
|
|
25
|
+
content: "# Managed by personal Fleet: designated development user\n${USER} ALL=(ALL:ALL) NOPASSWD: ALL\n"
|
|
26
|
+
- path: /run/devfleet/tailnet-auth-key
|
|
27
|
+
owner: root:root
|
|
28
|
+
permissions: '0600'
|
|
29
|
+
content: "${AUTH_KEY}\n"
|
|
30
|
+
runcmd:
|
|
31
|
+
- [visudo, -cf, '/etc/sudoers.d/personal-fleet-${USER}']
|
|
32
|
+
- [
|
|
33
|
+
curl,
|
|
34
|
+
--fail,
|
|
35
|
+
--silent,
|
|
36
|
+
--show-error,
|
|
37
|
+
--location,
|
|
38
|
+
--output,
|
|
39
|
+
/usr/share/keyrings/tailscale-archive-keyring.gpg,
|
|
40
|
+
'https://pkgs.tailscale.com/stable/ubuntu/${CODENAME}.noarmor.gpg',
|
|
41
|
+
]
|
|
42
|
+
- [
|
|
43
|
+
sh,
|
|
44
|
+
-c,
|
|
45
|
+
"echo '${TAILSCALE_KEY_SHA256} /usr/share/keyrings/tailscale-archive-keyring.gpg' | sha256sum -c -",
|
|
46
|
+
]
|
|
47
|
+
- [
|
|
48
|
+
sh,
|
|
49
|
+
-c,
|
|
50
|
+
"echo 'deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/ubuntu ${CODENAME} main' > /etc/apt/sources.list.d/tailscale.list",
|
|
51
|
+
]
|
|
52
|
+
- [apt-get, update]
|
|
53
|
+
- [apt-get, install, --yes, 'tailscale=${TAILSCALE_VERSION}']
|
|
54
|
+
- [
|
|
55
|
+
tailscale,
|
|
56
|
+
up,
|
|
57
|
+
'--auth-key=file:/run/devfleet/tailnet-auth-key',
|
|
58
|
+
'--hostname=${HOSTNAME}',
|
|
59
|
+
'--ssh=false',
|
|
60
|
+
]
|
|
61
|
+
- [rm, -f, /run/devfleet/tailnet-auth-key]
|
package/docs/authentication.md
CHANGED
|
@@ -59,6 +59,7 @@ Every failure code is in [the runbooks](runbooks.md).
|
|
|
59
59
|
| `monitoring-backup-secret-key` | when its feature is declared | `DEVFLEET_SECRET_MONITORING_BACKUP_SECRET_KEY` | `~/.config/dev-fleet/secrets/monitoring-backup-secret-key` |
|
|
60
60
|
| `notify-webhook-url` | when its feature is declared | `DEVFLEET_SECRET_NOTIFY_WEBHOOK_URL` | `~/.config/dev-fleet/secrets/notify-webhook-url` |
|
|
61
61
|
| `notify-ntfy-url` | when its feature is declared | `DEVFLEET_SECRET_NOTIFY_NTFY_URL` | `~/.config/dev-fleet/secrets/notify-ntfy-url` |
|
|
62
|
+
| `tailnet-auth-key` | when its feature is declared | `DEVFLEET_SECRET_TAILNET_AUTH_KEY` | `~/.config/dev-fleet/secrets/tailnet-auth-key` |
|
|
62
63
|
|
|
63
64
|
<!-- end: secrets -->
|
|
64
65
|
|
|
@@ -70,6 +71,7 @@ Every failure code is in [the runbooks](runbooks.md).
|
|
|
70
71
|
| ----------------------------- | ------------------------------------------ |
|
|
71
72
|
| `auth:github` | `src/github/checks.ts` |
|
|
72
73
|
| `auth:vault` | `src/cli/fleet-sync-vault.ts` |
|
|
74
|
+
| `auth:vault-isolation` | `src/cli/fleet-sync-vault.ts` |
|
|
73
75
|
| `worker:git-identity` | `src/git-identity.ts` |
|
|
74
76
|
| `worker:git-push-auth` | `src/definition.ts`, `src/git-identity.ts` |
|
|
75
77
|
| `worker:github-sync` | `src/worker-access-checks.ts` |
|
package/docs/consumer-setup.md
CHANGED
|
@@ -153,36 +153,37 @@ npx devfleet fleet-state --example > fleet.json
|
|
|
153
153
|
Then replace every placeholder. Each field, from
|
|
154
154
|
[`schema/fleet.schema.json`](../schema/fleet.schema.json):
|
|
155
155
|
|
|
156
|
-
| Field | Value
|
|
157
|
-
| ----------------------------------------------------- |
|
|
158
|
-
| `version` | `3`
|
|
159
|
-
| `source.repository` | `https://github.com/crvouga/dev-fleet.git`, or your fork if you publish one. Machines only accept code from it.
|
|
160
|
-
| `state` | `{ "bucket", "endpoint"?, "region"?, "prefix"? }` or `{ "directory": "/absolute/path" }`. Never credentials.
|
|
161
|
-
| `secrets` | See [secrets](#secrets).
|
|
162
|
-
| `manager.id` | A short lowercase id, `^[a-z][a-z0-9-]{0,62}$`, e.g. `manager-mac`.
|
|
163
|
-
| `manager.os` | `"macOS"`
|
|
164
|
-
| `manager.network.tailscaleName` | The manager's MagicDNS name: `tailscale status --json` → `.Self.DNSName` without the trailing dot.
|
|
165
|
-
| `manager.network.user` | The manager's login name (`id -un`).
|
|
166
|
-
| `manager.neverSleepOnPower` | `true` (workers depend on the manager's services).
|
|
167
|
-
| `workers[].id` | A short lowercase id. The
|
|
168
|
-
| `
|
|
169
|
-
| `workers[].
|
|
170
|
-
| `workers[].
|
|
171
|
-
| `workers[].network.
|
|
172
|
-
| `workers[].network.
|
|
173
|
-
| `workers[].network.
|
|
174
|
-
| `
|
|
175
|
-
| `services.
|
|
176
|
-
| `services.monitoring.
|
|
177
|
-
| `services.monitoring.
|
|
178
|
-
| `services.monitoring.
|
|
179
|
-
| `
|
|
180
|
-
| `
|
|
181
|
-
| `worker.
|
|
182
|
-
| `
|
|
183
|
-
| `
|
|
184
|
-
| `worker.
|
|
185
|
-
| `worker.
|
|
156
|
+
| Field | Value |
|
|
157
|
+
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
|
158
|
+
| `version` | `3` |
|
|
159
|
+
| `source.repository` | `https://github.com/crvouga/dev-fleet.git`, or your fork if you publish one. Machines only accept code from it. |
|
|
160
|
+
| `state` | `{ "bucket", "endpoint"?, "region"?, "prefix"? }` or `{ "directory": "/absolute/path" }`. Never credentials. |
|
|
161
|
+
| `secrets` | See [secrets](#secrets). |
|
|
162
|
+
| `manager.id` | A short lowercase id, `^[a-z][a-z0-9-]{0,62}$`, e.g. `manager-mac`. |
|
|
163
|
+
| `manager.os` | `"macOS"` |
|
|
164
|
+
| `manager.network.tailscaleName` | The manager's MagicDNS name: `tailscale status --json` → `.Self.DNSName` without the trailing dot. |
|
|
165
|
+
| `manager.network.user` | The manager's login name (`id -un`). |
|
|
166
|
+
| `manager.neverSleepOnPower` | `true` (workers depend on the manager's services). |
|
|
167
|
+
| `workers[].id` | A short lowercase id. The first reachable worker is the canary: a new revision must pass there before the others get it. |
|
|
168
|
+
| `canary` | Optional: the id of the worker to prefer as canary. When it is unreachable, the next reachable worker is used. |
|
|
169
|
+
| `workers[].ssh` | The worker's alias in the manager's `~/.ssh/config`. |
|
|
170
|
+
| `workers[].os` | `"linux"` |
|
|
171
|
+
| `workers[].network.tailscaleName` | The worker's MagicDNS name (from `tailscale status --json` → `.Peer[].DNSName` on the manager). |
|
|
172
|
+
| `workers[].network.sshPort` | `22` unless the worker's sshd listens elsewhere. |
|
|
173
|
+
| `workers[].network.user` | The worker account (`ssh WORKER_ALIAS id -un`). |
|
|
174
|
+
| `workers[].network.t3Port` | `3773` |
|
|
175
|
+
| `services.proxy.endpoint` | The public HTTPS origin of the CLI proxy, e.g. `https://cli-proxy.example.com`. |
|
|
176
|
+
| `services.monitoring.hubPort` | `8443` |
|
|
177
|
+
| `services.monitoring.publicUrl` | The public HTTPS origin of the monitoring dashboard. |
|
|
178
|
+
| `services.monitoring.managerAccount` | The email of the monitoring account the manager enrolls machines with. |
|
|
179
|
+
| `services.monitoring.viewerAccount` | The email of the read-only account visitors are signed in as. |
|
|
180
|
+
| `ubuntu` | `{ "release": "26.04", "architectures": ["amd64", "arm64"] }`: a release of the [support matrix](support-matrix.md). |
|
|
181
|
+
| `worker.requiredCapabilities` | Normally all of `github`, `vault`, `t3`, `toolchain`, `workstation`, `worker`. |
|
|
182
|
+
| `worker.githubLogin` | The GitHub login every worker must be signed in as. |
|
|
183
|
+
| `github` (optional) | Who the manager signs in to GitHub as, per-node tokens, required scopes; see [GitHub](github.md). |
|
|
184
|
+
| `worker.vaultUser` | The Vault user each worker's token is issued for. |
|
|
185
|
+
| `worker.toolchain`, `agentProfile` | `"mise.lock"`, `"proxy"` (fixed). |
|
|
186
|
+
| `worker.unattendedSudo`, `persistentT3`, `neverSleep` | `true` (fixed policy). |
|
|
186
187
|
|
|
187
188
|
Use the same values in every place they repeat. Validate before going on:
|
|
188
189
|
|
|
@@ -363,7 +364,7 @@ Read `summary` and `remedy.action` of the failing check. The common ones:
|
|
|
363
364
|
| `ROOT_DEVELOPMENT_USER_REJECTED` | The worker's SSH user is root. Use a normal account with sudo. |
|
|
364
365
|
| `BOOTSTRAP_DISK_CAPACITY_REQUIRED` | Less than 2 GiB free (or too few inodes) where the fleet keeps its files on the worker. |
|
|
365
366
|
| `FLEET_REMOTE_LOCKED` | Another operation is running on that worker. Wait and run again. |
|
|
366
|
-
| `FLEET_PROMOTION_BLOCKED` | The canary (
|
|
367
|
+
| `FLEET_PROMOTION_BLOCKED` | The canary (the report names it under `canary`) failed, so the new revision went no further. Fix the canary first. |
|
|
367
368
|
| `VAULT_LOGIN_REQUIRED` | The manager is not logged in to Vault. The human logs in; the next sync issues worker tokens. |
|
|
368
369
|
| `PROVIDER_UNATTENDED_CREDENTIAL_REQUIRED` | A coding agent has no credential a background service can use. The human signs in on the proxy. |
|
|
369
370
|
| `BESZEL_ACCOUNT_REQUIRED` | The monitoring hub has no account yet. Follow the remedy; other workers are not blocked. |
|
package/docs/declared.md
CHANGED
|
@@ -12,29 +12,29 @@ may only shrink.
|
|
|
12
12
|
|
|
13
13
|
## Declared
|
|
14
14
|
|
|
15
|
-
| What | Declared in | Pinned by | Applied by
|
|
16
|
-
| --------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
|
|
17
|
-
| The fleet: machines, services, accounts | Your `fleet.json` | The dev-fleet version in your `package.json`, which names one commit | `fleet-sync`
|
|
18
|
-
| Tools: languages, CLIs, coding agents | `mise.toml` | `mise.lock`: version and checksum for each platform | `mise install --locked`
|
|
19
|
-
| mise itself | `min_version` and `mise_*_sha256` in `mise.toml` | The digest of its release archive, held here | `bootstrap/bootstrap.sh`
|
|
20
|
-
| JavaScript dependencies | `package.json` | `bun.lock` | `bun install --frozen-lockfile`
|
|
21
|
-
| T3 Code, server and desktop app | `t3_*` under `[vars]` in `mise.toml` | SHA-256 of each artifact | `bootstrap/native-t3.ts`; `ansible/ubuntu/t3-desktop.yml`
|
|
22
|
-
| T3 policy on the manager (updates off) | `ansible/ubuntu/t3/policy.json`, shared with the workers | The commit | `ansible/macos/t3-policy.yml`, once, as an administrator
|
|
23
|
-
| Worker work-in-progress guard | `ansible/ubuntu/wip-guard.py`, `ansible/ubuntu/tasks/recovery.yml` | The commit | `fleet-sync`
|
|
24
|
-
| Ubuntu packages | `packages` and `snapshot` in `ansible/ubuntu/policy.json` | Exact versions from a dated archive snapshot | `ansible/ubuntu/tasks/packages.yml`
|
|
25
|
-
| Tailscale on workers | `tailscale` in `ansible/ubuntu/policy.json` | Exact version; digest of the signing key | `ansible/ubuntu/tailscale.yml`
|
|
26
|
-
| Worker policy: power, SSH, sudo, memory, disk | `ansible/ubuntu/*.yml`, values shared with checks in `policy.json` | The commit | `fleet-sync`
|
|
27
|
-
| Monitoring: hub, tunnel, agents | `ansible/monitoring/*.yml`, `compose.yml`, `lock.json` | Image digests; SHA-256 of each agent archive | `fleet-sync`
|
|
28
|
-
| The manager's private route to the hub | `ansible/monitoring/hub-route-darwin.yml` | The commit | `fleet-sync`
|
|
29
|
-
| The manager's power settings | `pmset` in `ansible/macos/policy.json` | The commit | `ansible/macos/manager.yml`,
|
|
30
|
-
| Ansible on the manager | `ansible/pyproject.toml` | `ansible/uv.lock`: one release, one Python, every dependency hashed | `uv run --locked --isolated`
|
|
31
|
-
| Ansible on a worker | `ansible-core` in `ansible/ubuntu/policy.json` | Exact version | `ansible/ubuntu/tasks/packages.yml`
|
|
32
|
-
| The CLI proxy | `cliproxy/compose.yaml`, `image.lock.json`, `requirements.lock`, `launchd/`, `config.template.yaml` | Image digests; hashed Python dependencies | `devfleet cliproxy install`, and `fleet-sync` after it
|
|
33
|
-
| Agent profiles and T3 providers | `agents/registry.json`, `agents/t3.json` | The commit | `fleet-sync`
|
|
34
|
-
| Worker time service | `ansible/ubuntu/tasks/services.yml`, offsets in `clock` in `ansible/ubuntu/policy.json` | The commit | `fleet-sync`
|
|
35
|
-
| The manager's container watchdog | `ansible/monitoring/container-watchdog.py`, `containerWatchdog` in `ansible/macos/policy.json` | The commit | `fleet-sync` (`hub-darwin.yml`)
|
|
36
|
-
| Docker Desktop on the manager | `docker_desktop_*` under `[vars]` in `mise.toml` | Version and build | You
|
|
37
|
-
| CI | `.github/workflows/` | Each action at a commit; tools from `mise.lock` | GitHub
|
|
15
|
+
| What | Declared in | Pinned by | Applied by | Measured by |
|
|
16
|
+
| --------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | -------------------------------------------- |
|
|
17
|
+
| The fleet: machines, services, accounts | Your `fleet.json` | The dev-fleet version in your `package.json`, which names one commit | `fleet-sync` | `fleet-status` |
|
|
18
|
+
| Tools: languages, CLIs, coding agents | `mise.toml` | `mise.lock`: version and checksum for each platform | `mise install --locked` | Tool and `agent:verification` checks |
|
|
19
|
+
| mise itself | `min_version` and `mise_*_sha256` in `mise.toml` | The digest of its release archive, held here | `bootstrap/bootstrap.sh` | Its version, at every toolchain setup |
|
|
20
|
+
| JavaScript dependencies | `package.json` | `bun.lock` | `bun install --frozen-lockfile` | The install fails on a difference |
|
|
21
|
+
| T3 Code, server and desktop app | `t3_*` under `[vars]` in `mise.toml` | SHA-256 of each artifact | `bootstrap/native-t3.ts`; `ansible/ubuntu/t3-desktop.yml` | T3 checks; `t3:version` on every machine |
|
|
22
|
+
| T3 policy on the manager (updates off) | `ansible/ubuntu/t3/policy.json`, shared with the workers | The commit | `ansible/macos/t3-policy.yml`, once, as an administrator | `t3:updates` |
|
|
23
|
+
| Worker work-in-progress guard | `ansible/ubuntu/wip-guard.py`, `ansible/ubuntu/tasks/recovery.yml` | The commit | `fleet-sync` | `features/ide/work-in-progress.feature` |
|
|
24
|
+
| Ubuntu packages | `packages` and `snapshot` in `ansible/ubuntu/policy.json` | Exact versions from a dated archive snapshot | `ansible/ubuntu/tasks/packages.yml` | `package:*` |
|
|
25
|
+
| Tailscale on workers | `tailscale` in `ansible/ubuntu/policy.json` | Exact version; digest of the signing key | `ansible/ubuntu/tailscale.yml` | `package:tailscale` |
|
|
26
|
+
| Worker policy: power, SSH, sudo, memory, disk | `ansible/ubuntu/*.yml`, values shared with checks in `policy.json` | The commit | `fleet-sync` | Host and resource checks |
|
|
27
|
+
| Monitoring: hub, tunnel, agents | `ansible/monitoring/*.yml`, `compose.yml`, `lock.json` | Image digests; SHA-256 of each agent archive | `fleet-sync` | `beszel:*` |
|
|
28
|
+
| The manager's private route to the hub | `ansible/monitoring/hub-route-darwin.yml` | The commit | `fleet-sync` | The route, read back |
|
|
29
|
+
| The manager's power settings | `pmset` in `ansible/macos/policy.json` | The commit | `ansible/macos/manager.yml`, on every `fleet-sync`, after one grant (`manager-privilege.yml`) | `manager:power` |
|
|
30
|
+
| Ansible on the manager | `ansible/pyproject.toml` | `ansible/uv.lock`: one release, one Python, every dependency hashed | `uv run --locked --isolated` | A hash that differs stops the run |
|
|
31
|
+
| Ansible on a worker | `ansible-core` in `ansible/ubuntu/policy.json` | Exact version | `ansible/ubuntu/tasks/packages.yml` | `package:ansible-core` |
|
|
32
|
+
| The CLI proxy | `cliproxy/compose.yaml`, `image.lock.json`, `requirements.lock`, `launchd/`, `config.template.yaml` | Image digests; hashed Python dependencies | `devfleet cliproxy install`, and `fleet-sync` after it | The proxy's audit |
|
|
33
|
+
| Agent profiles and T3 providers | `agents/registry.json`, `agents/t3.json` | The commit | `fleet-sync` | Profile receipts; the plan after every apply |
|
|
34
|
+
| Worker time service | `ansible/ubuntu/tasks/services.yml`, offsets in `clock` in `ansible/ubuntu/policy.json` | The commit | `fleet-sync` | `worker:clock` |
|
|
35
|
+
| The manager's container watchdog | `ansible/monitoring/container-watchdog.py`, `containerWatchdog` in `ansible/macos/policy.json` | The commit | `fleet-sync` (`hub-darwin.yml`) | `manager:container-watchdog` |
|
|
36
|
+
| Docker Desktop on the manager | `docker_desktop_*` under `[vars]` in `mise.toml` | Version and build | You | `manager:docker` (warns) |
|
|
37
|
+
| CI | `.github/workflows/` | Each action at a commit; tools from `mise.lock` | GitHub | `features/rules/` |
|
|
38
38
|
|
|
39
39
|
A value lives in a policy file when something besides the playbook reads it,
|
|
40
40
|
so the check and the playbook cannot disagree. A value only a playbook uses
|
|
@@ -89,4 +89,4 @@ would settle and the current tools do not is the rest of a worker's packages.
|
|
|
89
89
|
| An Ubuntu package | `packages` in `ansible/ubuntu/policy.json`, and `snapshot` if needed | The VM test in `ansible/ubuntu/tests` |
|
|
90
90
|
| Ansible on the manager | `ansible/pyproject.toml` | `uv lock` in `ansible/` |
|
|
91
91
|
| Beszel | `ansible/monitoring/lock.json`, `compose.yml`, the agent unit | `bun test features/rules` |
|
|
92
|
-
| A manager power setting | `pmset` in `ansible/macos/policy.json` |
|
|
92
|
+
| A manager power setting | `pmset` in `ansible/macos/policy.json` | The next `fleet-sync` applies it |
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# What only a person does on the manager
|
|
2
|
+
|
|
3
|
+
Everything on the manager is declared and converged by `fleet-sync`: the
|
|
4
|
+
toolchain, the power policy (applied by `ansible/macos/manager.yml` on every
|
|
5
|
+
run), the CLI proxy, the monitoring hub and their jobs. What remains are the
|
|
6
|
+
steps that need a person: an identity, an approval or an administrator
|
|
7
|
+
password. Each is done once, and `fleet-sync` measures it afterwards with the
|
|
8
|
+
check named beside it.
|
|
9
|
+
|
|
10
|
+
The list is tested not to grow (`features/rules/manager-setup.feature`): a
|
|
11
|
+
step added here must be added there too, with the check that verifies it.
|
|
12
|
+
|
|
13
|
+
<!-- human-steps -->
|
|
14
|
+
|
|
15
|
+
| Step | Why a person | Verified by |
|
|
16
|
+
| ---------------------------------------------------------------------------------------------------------- | ------------------------------------- | --------------------- |
|
|
17
|
+
| Install Docker Desktop at the release `mise.toml` pins and sign in to it | A licence and an account | `manager:docker` |
|
|
18
|
+
| Sign this Mac in to Tailscale and approve it | A tailnet account and device approval | `tailscale:discovery` |
|
|
19
|
+
| Pass the bucket credentials (`DEVFLEET_BUCKET_*`) or save them with `fleet-state --init` | Credentials of an account you own | `state:bucket` |
|
|
20
|
+
| Log the manager in to Vault | A password or an identity provider | `auth:vault` |
|
|
21
|
+
| Grant the power settings once: `ansible-playbook -e user=$(id -un) -K ansible/macos/manager-privilege.yml` | An administrator password | `manager:power` |
|
|
22
|
+
|
|
23
|
+
<!-- /human-steps -->
|
|
24
|
+
|
|
25
|
+
The grant lets the manager's user run exactly `pmset -c` without a password,
|
|
26
|
+
so later syncs never prompt. Run the same playbook with `-e grant=absent` to
|
|
27
|
+
remove it.
|
package/docs/runbooks.md
CHANGED
|
@@ -8,7 +8,7 @@ person (a login, an approval, a choice) is marked **needs a person**.
|
|
|
8
8
|
`features/rules/runbooks.feature` fails when a code in the source has no
|
|
9
9
|
entry here, or a `devfleet` command quoted in the documents does not exist.
|
|
10
10
|
|
|
11
|
-
There are
|
|
11
|
+
There are 250 codes; 31 need a person.
|
|
12
12
|
|
|
13
13
|
## Bootstrap of a new or returning worker
|
|
14
14
|
|
|
@@ -346,6 +346,20 @@ There are 244 codes; 31 need a person.
|
|
|
346
346
|
- `VAULT_LOGIN_REQUIRED`: vault login required. **Needs a person.**
|
|
347
347
|
- `VAULT_POLICY_PERMISSION_REQUIRED`: vault policy permission required. **Needs a person.**
|
|
348
348
|
- `VAULT_RENEWAL_FAILING`: vault renewal failing.
|
|
349
|
+
- `VAULT_WORKER_POLICY_FAILED`: vault worker policy failed.
|
|
350
|
+
|
|
351
|
+
## The fleet definition and a worker's lifecycle
|
|
352
|
+
|
|
353
|
+
- **Meaning:** `fleet.json` breaks a rule of the fleet definition, the pinned Ansible rejects the inventory it describes, or a seed, replacement or retirement could not change the host keys the manager trusts.
|
|
354
|
+
- **Confirm:** `devfleet fleet-validate` names the field that breaks a rule as `{code, path, expected, actual}`; `devfleet fleet-validate --ansible` also renders the inventory and syntax-checks every playbook.
|
|
355
|
+
- **Fix:** Correct the field the check names in `fleet.json`, then run the command again. For SSH_HOST_KEY_NOT_CHANGED run `devfleet fleet-validate --ansible`, then repeat `devfleet fleet-node-seed`, `devfleet fleet-node-replace` or `devfleet fleet-node-retire`.
|
|
356
|
+
- **Escalate:** An inventory the pinned Ansible rejects for a valid definition is an issue.
|
|
357
|
+
|
|
358
|
+
- `DEFINITION_INVALID`: definition invalid.
|
|
359
|
+
- `DEFINITION_INVENTORY_REJECTED`: definition inventory rejected.
|
|
360
|
+
- `DEFINITION_MISSING`: definition missing.
|
|
361
|
+
- `DEFINITION_UNREADABLE`: definition unreadable.
|
|
362
|
+
- `SSH_HOST_KEY_NOT_CHANGED`: ssh host key not changed.
|
|
349
363
|
|
|
350
364
|
## Fleet operations and the fleet state
|
|
351
365
|
|
package/docs/topology.md
CHANGED
|
@@ -35,9 +35,12 @@ your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.
|
|
|
35
35
|
A new singleton service is declared under `services` and placed on the
|
|
36
36
|
manager; it never names a host.
|
|
37
37
|
- Workers depend on it, so it never sleeps on AC power
|
|
38
|
-
(`pmset` in `ansible/macos/policy.json`, checked as `manager:power
|
|
39
|
-
|
|
40
|
-
|
|
38
|
+
(`pmset` in `ansible/macos/policy.json`, checked as `manager:power` and
|
|
39
|
+
applied by `ansible/macos/manager.yml` on every `fleet-sync` once an
|
|
40
|
+
administrator has given the one grant `ansible/macos/manager-privilege.yml`;
|
|
41
|
+
the failed check prints the command, and
|
|
42
|
+
[docs/manager-setup.md](manager-setup.md) lists every step only a person
|
|
43
|
+
takes). While the
|
|
41
44
|
manager is asleep, closed or offline, workers have no CLI proxy and no
|
|
42
45
|
dashboard; nothing fails over, because a singleton never runs twice.
|
|
43
46
|
- Its state (proxy OAuth/runtime state, the hub database) is backed up off the
|
|
@@ -56,9 +59,15 @@ your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.
|
|
|
56
59
|
checkouts. Durable work lives in Git remotes; credentials come from Vault.
|
|
57
60
|
Work in progress in agent worktrees is pushed to a recovery ref every 15
|
|
58
61
|
minutes; [durability](durability.md) says what a lost machine takes with it.
|
|
59
|
-
-
|
|
60
|
-
fleet
|
|
61
|
-
|
|
62
|
+
- One worker is the rollout canary for a revision new to the fleet: the one
|
|
63
|
+
`fleet-sync --canary` or `canary` in `fleet.json` names, else the first
|
|
64
|
+
declared worker. When it cannot be reached (powered off, being rebuilt),
|
|
65
|
+
the next reachable worker in declared order proves the revision instead and
|
|
66
|
+
the unreachable one is reported as offline, never as a blocked rollout. A
|
|
67
|
+
canary that was deployed and failed stops promotion to the other workers.
|
|
68
|
+
Each worker's applied revision is kept in the fleet state, so one that
|
|
69
|
+
converged keeps its progress when another did not. Repeating a revision
|
|
70
|
+
that already converged everywhere runs all workers at once.
|
|
62
71
|
|
|
63
72
|
## Changing the topology
|
|
64
73
|
|
package/docs/ubuntu-sync.md
CHANGED
|
@@ -188,26 +188,32 @@ verification and the CLIProxy singleton.
|
|
|
188
188
|
| Tailscale | pinned APT package, `tailscaled` enabled, native login required | Tailscale app, human login |
|
|
189
189
|
| Dashboard agent | pinned native binary, hardened `beszel-agent` systemd unit as user `beszel` | not installed |
|
|
190
190
|
| Credential renewal | user timer `personal-worker-auth` renews the node's own Vault token every 10min; `personal-worker-github` signs it in to GitHub from the stored token ([GitHub](github.md)) | the human's own Vault and GitHub sessions |
|
|
191
|
-
| Noninteractive sudo | validated `/etc/sudoers.d/personal-fleet-USER` |
|
|
191
|
+
| Noninteractive sudo | validated `/etc/sudoers.d/personal-fleet-USER` | only `pmset -c`, granted once by `manager-privilege.yml` (`-K`) |
|
|
192
192
|
| T3 worker app | pinned desktop `.deb` is the only T3 process (UI plus embedded server); GDM auto-login, user unit `t3code-desktop.service` bound to the graphical session with `Restart=always`; `personal-worker-recovery` runs a copy of `t3-health.sh` installed in `/usr/local/lib/personal-fleet` every minute: it stops and disables a still-running headless `t3code.service`, reports `T3_NO_GRAPHICAL_SESSION` and restarts the display manager (three times at most) when there is no session, starts a stopped app after `reset-failed`, restarts one whose local API missed three checks, and restarts one on another release than the installed one or with a deferred policy restart once it runs no agent turn; administrator policy | not a worker |
|
|
193
193
|
| T3 providers | the same managed instances (`fleet/agents/t3.json`) through locked T3 administrator policy | reconciled into the human's T3 settings by `fleet-sync` (`t3:providers`) with the ownership-safe `t3-apply` engine, so every T3 offers the identical set; unrelated and unmanaged instances are preserved. `fleet-sync` also applies the same proxy agent profiles (`configuration:apply`) and verifies the shared Vault gateway credential end to end with `provider:*:inference`, exactly as on workers |
|
|
194
194
|
| Singleton services | never; clients of the manager's services only | every singleton runs here: CLI proxy (`cliproxy`), Beszel hub (`workstation/monitoring`) |
|
|
195
195
|
|
|
196
196
|
Credential sync is deliberately indirect: no secret is copied between machines.
|
|
197
|
-
|
|
198
|
-
(`
|
|
199
|
-
`
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
197
|
+
Each worker has a Vault token of its own, kept in a document of its own,
|
|
198
|
+
`<mount>/<project>/workers/<id>` (fields `token` and `accessor`; `<project>`
|
|
199
|
+
is the first segment of the `proxy-client-key` path). The manager issues it
|
|
200
|
+
with `devfleet fleet-node-replace --id ID --apply --yes`: a periodic orphan
|
|
201
|
+
token under the policy `fleet-worker-<id>`, which reads the fleet's shared
|
|
202
|
+
document and the worker's own document and nothing else, so one worker's
|
|
203
|
+
token cannot read another's (`features/fleet-sync/worker-credentials.feature`
|
|
204
|
+
proves it against a Vault development server). The manager's own login needs
|
|
205
|
+
to write policies, create tokens and write the mount.
|
|
206
|
+
|
|
207
|
+
`devfleet fleet-sync` delivers each node's token over SSH stdin (never as an
|
|
208
|
+
argument) whenever that node has no working Vault session. A token still kept
|
|
209
|
+
in the shared document as `FLEET_WORKER_TOKEN_<NODE>` is delivered too, with
|
|
210
|
+
the warning `VAULT_WORKER_TOKEN_SHARED`: every worker can read that document,
|
|
211
|
+
so issue the worker its own token and remove the field. A password login with
|
|
212
|
+
the fleet's shared Vault user on the worker's own terminal happens only with
|
|
213
|
+
`fleet-sync --vault-password-login`. Without a token the node reports
|
|
214
|
+
`VAULT_LOGIN_REQUIRED` and its T3 proxy providers stay unavailable.
|
|
215
|
+
`devfleet fleet-node-retire --id ID --apply --yes` revokes a worker's token by
|
|
216
|
+
its accessor and removes its document and policy.
|
|
211
217
|
|
|
212
218
|
From then on the node's `personal-worker-auth` timer renews that token and its
|
|
213
219
|
`personal-worker-github` timer keeps its GitHub sign-in current
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@crvouga/dev-fleet",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.0",
|
|
4
4
|
"description": "Keeps a set of development machines in sync: one manager and many workers",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": "github:crvouga/dev-fleet",
|
|
@@ -72,5 +72,5 @@
|
|
|
72
72
|
"engines": {
|
|
73
73
|
"node": ">=22"
|
|
74
74
|
},
|
|
75
|
-
"gitHead": "
|
|
75
|
+
"gitHead": "3f13ebe4bda2cf813a68ee45678627b2e21660d5"
|
|
76
76
|
}
|