@crvouga/dev-fleet 0.6.0 → 0.8.0

package/docs/manager-recovery.md

@@ -0,0 +1,78 @@
+# Replacing the manager
+
+This is the one runbook for a manager that is replaced or lost: stolen, a dead
+disk, or a new machine. Other documents link here and do not describe the
+procedure themselves.
+
+## What must exist off the machine
+
+Nothing on the old manager is needed. Before you need this runbook, make sure
+these exist elsewhere:
+
+| What                                | Where                                                                    |
+| ----------------------------------- | ------------------------------------------------------------------------ |
+| Your project: `fleet.json`, the pin | Its Git remote                                                           |
+| The fleet state                     | The bucket `fleet.json` names under `state`                              |
+| The bucket's two credentials        | Your password manager (never the manager alone)                          |
+| Secrets                             | Your `secrets` provider; `file` and `env` secrets must be kept elsewhere |
+| The CLI proxy's encrypted snapshots | Its restic repository (`vault` secrets provider only)                    |
+| Your Tailscale and Docker accounts  | Yours: signing in is the only step only you can do                       |
+
+## Restore
+
+On the new machine, with Git, Node and Docker installed:
+
+1. Give it the manager's local host name: `manager.id` in `fleet.json`
+   (`sudo scutil --set LocalHostName ID` on macOS). `fleet-sync` refuses to
+   run on any other machine.
+2. Sign in to Tailscale and approve the device. Sign in to Docker Desktop if
+   your plan needs it.
+3. Fetch the project and check every prerequisite. Nothing is changed:
+
+   ```sh
+   git clone YOUR-PROJECT && cd YOUR-PROJECT && npm ci
+   export DEVFLEET_BUCKET_ACCESS_KEY_ID=… DEVFLEET_BUCKET_SECRET_ACCESS_KEY=…
+   npx devfleet fleet-restore --dry-run
+   ```
+
+   It fails on anything missing (`restore:*` checks, each with its remedy)
+   and lists what will be restored, in order.
+
+4. Restore, then converge:
+
+   ```sh
+   npx devfleet fleet-restore              # the fleet state, from the bucket
+   npx devfleet fleet-sync --take-over     # the manager, then every worker
+   npx devfleet fleet-state --init         # credentials for runs the manager starts itself
+   npx devfleet cliproxy install           # the proxy, from its snapshots
+   ```
+
+   The proxy's own steps are in [its migration runbook](../cliproxy/docs/migration.md).
+   `--take-over` is needed while the old manager's lease runs (seven days
+   after its last change). Use it only once the old machine is gone: an old
+   manager that wakes up afterwards is refused with `FLEET_MANAGER_LEASE_HELD`.
+
+5. Adopt the T3 instances again: `npx devfleet t3-apply --import ID`.
+
+Workers need nothing: they reach the manager by its Tailscale name, so they
+follow it once the new machine holds that name. If the Tailscale name
+changed, change `manager.network.tailscaleName` in `fleet.json` and run
+`fleet-sync`, which re-points every worker.
+
+## What is lost
+
+The monitoring hub's metric history, alerts and settings made by hand. Its
+accounts and systems are recreated. With `env` or `file` secrets, whatever
+was only on the old manager.
+
+## Coming back by itself
+
+A manager restarts after a power cut only when it is set up to
+(`manager:boot-recovery` in `fleet-status`): `autorestart` and Docker Desktop
+at login are applied by `ansible/macos/manager.yml`. The per-user services
+need a user session. **With FileVault on, a Mac waits at the login window
+after every restart until a person signs in**, so nothing comes back
+unattended. That is the trade-off: FileVault protects the proxy's logins and
+the bucket credentials on a stolen Mac; automatic login without FileVault
+lets the fleet recover from a power cut alone. `fleet-status` reports
+FileVault as a warning with that consequence; the choice is yours.

package/docs/manager-setup.md

@@ -0,0 +1,27 @@
+# What only a person does on the manager
+
+Everything on the manager is declared and converged by `fleet-sync`: the
+toolchain, the power policy (applied by `ansible/macos/manager.yml` on every
+run), the CLI proxy, the monitoring hub and their jobs. What remains are the
+steps that need a person: an identity, an approval or an administrator
+password. Each is done once, and `fleet-sync` measures it afterwards with the
+check named beside it.
+
+The list is tested not to grow (`features/rules/manager-setup.feature`): a
+step added here must be added there too, with the check that verifies it.
+
+<!-- human-steps -->
+
+| Step                                                                                                       | Why a person                          | Verified by           |
+| ---------------------------------------------------------------------------------------------------------- | ------------------------------------- | --------------------- |
+| Install Docker Desktop at the release `mise.toml` pins and sign in to it                                   | A licence and an account              | `manager:docker`      |
+| Sign this Mac in to Tailscale and approve it                                                               | A tailnet account and device approval | `tailscale:discovery` |
+| Pass the bucket credentials (`DEVFLEET_BUCKET_*`) or save them with `fleet-state --init`                   | Credentials of an account you own     | `state:bucket`        |
+| Log the manager in to Vault                                                                                | A password or an identity provider    | `auth:vault`          |
+| Grant the power settings once: `ansible-playbook -e user=$(id -un) -K ansible/macos/manager-privilege.yml` | An administrator password             | `manager:power`       |
+
+<!-- /human-steps -->
+
+The grant lets the manager's user run exactly `pmset -c` without a password,
+so later syncs never prompt. Run the same playbook with `-e grant=absent` to
+remove it.

package/docs/operations.md

@@ -304,6 +304,14 @@ The public CLI exposes `devfleet fleet-repair --list` (static catalog) and
 in this package only executes allowlisted existing owner commands. Native
 account logins and unknown/manual checks always remain operator steps.
 
+Every check ID has an owner in `src/check-owners.ts`: a recipe, or the reason
+only a person can repair it, which the plan prints beside the check
+(`manual`). Machine repairs are one tag of a playbook the worker already
+applies: `recovery` (`worker:service-recovery`), `disk`
+(`worker:capacity:*`), `monitoring` (the Beszel agent) and `network` (the
+Tailscale daemon). `state:*` checks take the fleet state from the bucket
+again. A feature fails when a check is reported that has no owner.
+
 Use `--apply --yes` only after review; `--plan-hash` binds application to the
 previewed steps, targets and checkout/shared-input fingerprints. The remote
 protocol independently requires explicit apply, a supported repair action and

package/docs/runbooks.md

@@ -8,7 +8,7 @@ person (a login, an approval, a choice) is marked **needs a person**.
 `features/rules/runbooks.feature` fails when a code in the source has no
 entry here, or a `devfleet` command quoted in the documents does not exist.
 
-There are 244 codes; 31 need a person.
+There are 250 codes; 31 need a person.
 
 ## Bootstrap of a new or returning worker
 
@@ -346,6 +346,20 @@ There are 244 codes; 31 need a person.
 - `VAULT_LOGIN_REQUIRED`: vault login required. **Needs a person.**
 - `VAULT_POLICY_PERMISSION_REQUIRED`: vault policy permission required. **Needs a person.**
 - `VAULT_RENEWAL_FAILING`: vault renewal failing.
+- `VAULT_WORKER_POLICY_FAILED`: vault worker policy failed.
+
+## The fleet definition and a worker's lifecycle
+
+- **Meaning:** `fleet.json` breaks a rule of the fleet definition, the pinned Ansible rejects the inventory it describes, or a seed, replacement or retirement could not change the host keys the manager trusts.
+- **Confirm:** `devfleet fleet-validate` names the field that breaks a rule as `{code, path, expected, actual}`; `devfleet fleet-validate --ansible` also renders the inventory and syntax-checks every playbook.
+- **Fix:** Correct the field the check names in `fleet.json`, then run the command again. For SSH_HOST_KEY_NOT_CHANGED run `devfleet fleet-validate --ansible`, then repeat `devfleet fleet-node-seed`, `devfleet fleet-node-replace` or `devfleet fleet-node-retire`.
+- **Escalate:** An inventory the pinned Ansible rejects for a valid definition is an issue.
+
+- `DEFINITION_INVALID`: definition invalid.
+- `DEFINITION_INVENTORY_REJECTED`: definition inventory rejected.
+- `DEFINITION_MISSING`: definition missing.
+- `DEFINITION_UNREADABLE`: definition unreadable.
+- `SSH_HOST_KEY_NOT_CHANGED`: ssh host key not changed.
 
 ## Fleet operations and the fleet state
 

package/docs/state.md

@@ -64,9 +64,9 @@ arguments and never written to a report or a log.
 
 **A new fleet.** Nothing more: the first `fleet-sync` seeds the bucket.
 
-**A replacement manager, or a second machine.** Clone your project, `npm ci`,
-and run `fleet-sync` with the credentials. The bucket already holds the fleet,
-so its state is downloaded before anything is changed.
+**A replacement manager.** The bucket already holds the fleet, so its state is
+downloaded before anything is changed. Follow
+[replacing the manager](manager-recovery.md).
 
 **Runs the manager starts by itself.** A worker that hands `fleet-sync` to the
 manager starts a command there that gets no environment and no project. Save
@@ -172,6 +172,7 @@ it records what it applied. Its report carries the result as the
 | `local-ahead`  | This machine has changes the bucket has not received   | The next change, or `--push`, uploads them                 |
 | `bucket-ahead` | Another manager changed the bucket; this copy is stale | The next change or `fleet-sync` takes the bucket's version |
 | `unreachable`  | The bucket did not answer, or did not take an upload   | Changes still apply locally and are uploaded later         |
+| `corrupt`      | The bucket answered with a version that cannot be read | Changes apply locally; `--restore --timestamp` repairs it  |
 | `diverged`     | Both sides changed                                     | Refused. `--restore --yes` takes the bucket's version      |
 
 A stale copy never overwrites the bucket. A machine that was offline while
@@ -210,6 +211,7 @@ once `fleet-sync` has fetched it back.
 | `DEVFLEET_CONFIG`                                                    | A configuration file, instead of the project's `fleet.json`  |
 | `DEVFLEET_STATE`                                                     | Another state directory. Earlier files are not read for it   |
 | `DEVFLEET_BUCKET`                                                    | Another `bucket.json`                                        |
+| `DEVFLEET_MANAGER_HOLDER`                                            | How the manager lease names this machine                     |
 
 How a command reaches the bucket:
 
@@ -254,7 +256,37 @@ mise exec litestream -- litestream restore -no-expand-env \
   ~/.local/share/dev-fleet/state/fleet.db
 ```
 
+`devfleet fleet-state --restore --timestamp 2026-11-01T00:00:00Z --yes` does
+the same and then uploads that version as the bucket's current one. It is the
+remedy for a `corrupt` replica: the bucket answers, but its latest version
+cannot be read.
+
 Litestream is pinned in `mise.toml` and installed the first time it is used.
-The manager is the only machine that runs it. There is one manager, so there is
-one writer; two machines changing the state at the same moment are not
-supported, only detected afterwards.
+The manager is the only machine that runs it.
+
+## One manager at a time
+
+The fleet state carries a **manager lease**: which machine is the manager,
+and until when. Every change takes or renews it, and it reaches the bucket
+with the change. A second machine (a replacement while the old manager still
+runs, or an old manager that wakes up) reads it from the bucket before it
+writes anything and stops with `FLEET_MANAGER_LEASE_HELD`. `fleet-state
+--init` still restores the state on such a machine; it only writes nothing.
+
+The lease runs for seven days after the last change. After that, any machine
+may take it. Before that, `--take-over` (on `fleet-sync`, `fleet-state --init`
+or `fleet-state --import`) makes this machine the manager; use it only once
+the other one is gone ([replacing the manager](manager-recovery.md)).
+
+A machine is named in the lease by a digest of its machine ID
+(`/etc/machine-id` on Linux, the hardware UUID on macOS), never by its host
+name, which a replacement shares. `DEVFLEET_MANAGER_HOLDER` names it instead.
+
+## Schema version
+
+The database records its schema version in `PRAGMA user_version`. The schema
+is a series of numbered migrations (`src/state/migrations/0001-….sql`, …),
+and a command brings an older database forward before it writes. A database a
+newer dev-fleet wrote is refused with `FLEET_STATE_NEWER_THAN_CLI`, read or
+written, and nothing in it changes: install the dev-fleet version the fleet
+runs.

package/docs/topology.md

@@ -35,13 +35,17 @@ your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.
   A new singleton service is declared under `services` and placed on the
   manager; it never names a host.
 - Workers depend on it, so it never sleeps on AC power
-  (`pmset` in `ansible/macos/policy.json`, checked as `manager:power`, applied
-  once by an administrator with `ansible/macos/manager.yml`; the failed check
-  prints the command). While the
+  (`pmset` in `ansible/macos/policy.json`, checked as `manager:power` and
+  applied by `ansible/macos/manager.yml` on every `fleet-sync` once an
+  administrator has given the one grant `ansible/macos/manager-privilege.yml`;
+  the failed check prints the command, and
+  [docs/manager-setup.md](manager-setup.md) lists every step only a person
+  takes). While the
   manager is asleep, closed or offline, workers have no CLI proxy and no
   dashboard; nothing fails over, because a singleton never runs twice.
 - Its state (proxy OAuth/runtime state, the hub database) is backed up off the
-  machine as each service documents, so a replacement manager restores it.
+  machine as each service documents, so a replacement manager restores it
+  ([replacing the manager](manager-recovery.md)).
 
 ## Workers
 
@@ -55,15 +59,19 @@ your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.
   checkouts. Durable work lives in Git remotes; credentials come from Vault.
   Work in progress in agent worktrees is pushed to a recovery ref every 15
   minutes; [durability](durability.md) says what a lost machine takes with it.
-- The first declared worker is the rollout canary for a revision new to the
-  fleet: a failed canary stops promotion to the other workers. Repeating a
-  revision that already converged everywhere runs all workers at once.
+- One worker is the rollout canary for a revision new to the fleet: the one
+  `fleet-sync --canary` or `canary` in `fleet.json` names, else the first
+  declared worker. When it cannot be reached (powered off, being rebuilt),
+  the next reachable worker in declared order proves the revision instead and
+  the unreachable one is reported as offline, never as a blocked rollout. A
+  canary that was deployed and failed stops promotion to the other workers.
+  Each worker's applied revision is kept in the fleet state, so one that
+  converged keeps its progress when another did not. Repeating a revision
+  that already converged everywhere runs all workers at once.
 
 ## Changing the topology
 
 - **Add or replace a worker:** add it to `workers`, enroll it with
   `fleet-sync`, done. Remove it by deleting the entry.
-- **Replace the manager:** edit `manager`, then move each singleton service with
-  its documented fence-and-restore runbook (never two copies at once):
-  [CLI proxy](../cliproxy/docs/migration.md),
-  [Beszel hub](../ansible/monitoring/README.md).
+- **Replace the manager:** follow
+  [replacing the manager](manager-recovery.md).

package/docs/ubuntu-sync.md

@@ -188,26 +188,32 @@ verification and the CLIProxy singleton.
 | Tailscale                   | pinned APT package, `tailscaled` enabled, native login required                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Tailscale app, human login                                                                                                                                                                                                                                                                                                                                                                                |
 | Dashboard agent             | pinned native binary, hardened `beszel-agent` systemd unit as user `beszel`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | not installed                                                                                                                                                                                                                                                                                                                                                                                             |
 | Credential renewal          | user timer `personal-worker-auth` renews the node's own Vault token every 10min; `personal-worker-github` signs it in to GitHub from the stored token ([GitHub](github.md))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | the human's own Vault and GitHub sessions                                                                                                                                                                                                                                                                                                                                                                 |
-| Noninteractive sudo         | validated `/etc/sudoers.d/personal-fleet-USER`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | never; the human runs `manager.yml` once with `-K`                                                                                                                                                                                                                                                                                                                                                        |
+| Noninteractive sudo         | validated `/etc/sudoers.d/personal-fleet-USER`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | only `pmset -c`, granted once by `manager-privilege.yml` (`-K`)                                                                                                                                                                                                                                                                                                                                           |
 | T3 worker app               | pinned desktop `.deb` is the only T3 process (UI plus embedded server); GDM auto-login, user unit `t3code-desktop.service` bound to the graphical session with `Restart=always`; `personal-worker-recovery` runs a copy of `t3-health.sh` installed in `/usr/local/lib/personal-fleet` every minute: it stops and disables a still-running headless `t3code.service`, reports `T3_NO_GRAPHICAL_SESSION` and restarts the display manager (three times at most) when there is no session, starts a stopped app after `reset-failed`, restarts one whose local API missed three checks, and restarts one on another release than the installed one or with a deferred policy restart once it runs no agent turn; administrator policy | not a worker                                                                                                                                                                                                                                                                                                                                                                                              |
 | T3 providers                | the same managed instances (`fleet/agents/t3.json`) through locked T3 administrator policy                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | reconciled into the human's T3 settings by `fleet-sync` (`t3:providers`) with the ownership-safe `t3-apply` engine, so every T3 offers the identical set; unrelated and unmanaged instances are preserved. `fleet-sync` also applies the same proxy agent profiles (`configuration:apply`) and verifies the shared Vault gateway credential end to end with `provider:*:inference`, exactly as on workers |
 | Singleton services          | never; clients of the manager's services only                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | every singleton runs here: CLI proxy (`cliproxy`), Beszel hub (`workstation/monitoring`)                                                                                                                                                                                                                                                                                                                  |
 
 Credential sync is deliberately indirect: no secret is copied between machines.
-OpenTofu issues each enrolled worker its own renewable periodic token
-(`vault.tokens` entries `fleet-worker-<node>`, policy `personal-read`, sunk into
-`secret/personal/prd` as `FLEET_WORKER_TOKEN_<NODE>`). `devfleet fleet-sync`
-delivers each node's token automatically over SSH stdin (never as an argument)
-whenever that node has no working Vault session. Interactive runs first log the
-manager in (`vault login -method=userpass`, password typed on your terminal)
-and fall back to the same password login on a node whose token is not issued.
-Without either, the node reports `VAULT_LOGIN_REQUIRED` and its T3 proxy
-providers stay unavailable. The manual equivalent of the automatic delivery:
-
-```sh
-vault kv get -field=FLEET_WORKER_TOKEN_WORKER_ONE secret/personal/prd \
-  | ssh worker-one 'vault login -address=https://vault.example.com -no-print -'
-```
+Each worker has a Vault token of its own, kept in a document of its own,
+`<mount>/<project>/workers/<id>` (fields `token` and `accessor`; `<project>`
+is the first segment of the `proxy-client-key` path). The manager issues it
+with `devfleet fleet-node-replace --id ID --apply --yes`: a periodic orphan
+token under the policy `fleet-worker-<id>`, which reads the fleet's shared
+document and the worker's own document and nothing else, so one worker's
+token cannot read another's (`features/fleet-sync/worker-credentials.feature`
+proves it against a Vault development server). The manager's own login needs
+to write policies, create tokens and write the mount.
+
+`devfleet fleet-sync` delivers each node's token over SSH stdin (never as an
+argument) whenever that node has no working Vault session. A token still kept
+in the shared document as `FLEET_WORKER_TOKEN_<NODE>` is delivered too, with
+the warning `VAULT_WORKER_TOKEN_SHARED`: every worker can read that document,
+so issue the worker its own token and remove the field. A password login with
+the fleet's shared Vault user on the worker's own terminal happens only with
+`fleet-sync --vault-password-login`. Without a token the node reports
+`VAULT_LOGIN_REQUIRED` and its T3 proxy providers stay unavailable.
+`devfleet fleet-node-retire --id ID --apply --yes` revokes a worker's token by
+its accessor and removes its document and policy.
 
 From then on the node's `personal-worker-auth` timer renews that token and its
 `personal-worker-github` timer keeps its GitHub sign-in current

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@crvouga/dev-fleet",
-  "version": "0.6.0",
+  "version": "0.8.0",
   "description": "Keeps a set of development machines in sync: one manager and many workers",
   "license": "MIT",
   "repository": "github:crvouga/dev-fleet",
@@ -72,5 +72,5 @@
   "engines": {
     "node": ">=22"
   },
-  "gitHead": "9f9718f2853ed7840c846c8dfc8750903c782b87"
+  "gitHead": "3f13ebe4bda2cf813a68ee45678627b2e21660d5"
 }

package/schema/check-ids.json

@@ -66,6 +66,14 @@
       "id": "auth:vault",
       "meaning": "This machine has its own Vault session."
     },
+    {
+      "id": "auth:vault-isolation",
+      "meaning": "A worker reads its Vault token from a document of its own, not from the fleet's shared one."
+    },
+    {
+      "id": "auth:worker-token",
+      "meaning": "A worker has a Vault token of its own that only it and the manager can read, or a retired worker's token is revoked."
+    },
     {
       "id": "beszel:*",
       "meaning": "A monitoring (Beszel) check of this machine."
@@ -150,6 +158,18 @@
       "id": "configuration:operation",
       "meaning": "The remote configuration operation completed."
     },
+    {
+      "id": "definition:inventory",
+      "meaning": "Ansible reads the inventory the fleet definition describes."
+    },
+    {
+      "id": "definition:playbooks",
+      "meaning": "Every playbook passes ansible-playbook --syntax-check."
+    },
+    {
+      "id": "definition:schema",
+      "meaning": "fleet.json follows every rule of the fleet definition."
+    },
     {
       "id": "enrollment:connection",
       "meaning": "The manager has an enrolled connection to this worker."
@@ -218,6 +238,10 @@
       "id": "login:*",
       "meaning": "A native per-machine login is in place."
     },
+    {
+      "id": "manager:boot-recovery",
+      "meaning": "The manager comes back by itself after a power cut or reboot."
+    },
     {
       "id": "manager:docker",
       "meaning": "The manager runs the pinned Docker Desktop."
@@ -302,6 +326,10 @@
       "id": "rollout:evidence",
       "meaning": "A worker's rollout produced verification evidence."
     },
+    {
+      "id": "rollout:next",
+      "meaning": "Names the command that converges a worker after it was replaced."
+    },
     {
       "id": "rollout:operation",
       "meaning": "The remote rollout operation completed."
@@ -386,6 +414,10 @@
       "id": "state:replica",
       "meaning": "The bucket holds this manager's fleet state."
     },
+    {
+      "id": "state:retired",
+      "meaning": "The fleet state records that a retired worker was retired."
+    },
     {
       "id": "t3:api",
       "meaning": "The local T3 API answers."
@@ -534,6 +566,10 @@
       "id": "worker:package:*",
       "meaning": "The pinned Ubuntu package is installed at or past its pin."
     },
+    {
+      "id": "worker:manifest",
+      "meaning": "A worker is made of what the declared pins and a fresh build of the same revision are made of."
+    },
     {
       "id": "worker:power-loss-recovery",
       "meaning": "The worker powers on after AC loss."

package/schema/fleet.schema.json

@@ -58,7 +58,6 @@
     },
     "workers": {
       "type": "array",
-      "minItems": 1,
       "maxItems": 64,
       "items": {
         "type": "object",
@@ -68,7 +67,7 @@
           "id": {
             "type": "string",
             "pattern": "^[a-z][a-z0-9-]{0,62}$",
-            "description": "A short stable id for the worker. The first worker listed is the canary: a revision new to the fleet must pass there before the others receive it."
+            "description": "A short stable id for the worker. A revision new to the fleet must pass on one worker, the canary, before the others receive it: the one named under canary, or else the first reachable worker in this order."
           },
           "ssh": {
             "type": "string",
@@ -119,6 +118,11 @@
         }
       }
     },
+    "canary": {
+      "type": "string",
+      "pattern": "^[a-z][a-z0-9-]{0,62}$",
+      "description": "The id of the worker that proves a revision new to the fleet first. When it is unreachable, the next reachable worker proves it instead. Default: the first reachable worker."
+    },
     "services": {
       "type": "object",
       "additionalProperties": false,
@@ -626,6 +630,22 @@
                       "pattern": "^[A-Za-z0-9_]+$"
                     }
                   }
+                },
+                "tailnet-auth-key": {
+                  "type": "object",
+                  "additionalProperties": false,
+                  "required": ["path", "field"],
+                  "properties": {
+                    "path": {
+                      "type": "string",
+                      "pattern": "^[A-Za-z0-9._/-]+$"
+                    },
+                    "field": {
+                      "type": "string",
+                      "pattern": "^[A-Za-z0-9_]+$"
+                    }
+                  },
+                  "description": "Optional: a short-lived, single-use, tagged Tailscale auth key a new worker joins the tailnet with (fleet-node-seed)."
                 }
               },
               "description": "Required for every secret except github-token, which defaults to the field GITHUB_TOKEN beside proxy-client-key."

package/src/check-owners.ts

@@ -0,0 +1,164 @@
+/** Who repairs a check: a repair recipe (`repair`), or the reason only a
+ * person can (`manual`). Every check ID dev-fleet reports matches one entry,
+ * by its exact ID or by a prefix ending in `*` (`beszel:*`). The most
+ * specific match wins. `login:*` is the sign-in recipe named after the rest
+ * of an `auth:` check's ID.
+ */
+export interface CheckOwner {
+  pattern: string;
+  repair?: string;
+  manual?: string;
+}
+
+const IDENTITY =
+  'It needs a person: a sign-in, an approval or a choice only the operator can make.';
+const EVIDENCE =
+  'It reports how a command or a run went, not the state of a machine: repeat the command once its remedy is followed.';
+const DECLARATION =
+  'The fleet configuration (fleet.json) or the roster is wrong: only the operator edits them.';
+const HARDWARE =
+  'It is a property of the hardware, the firmware or the operating system release, which no command changes.';
+
+export const CHECK_OWNERS: CheckOwner[] = [
+  // Toolchain and workstation: automatic.
+  { pattern: 'tool:*', repair: 'workspace' },
+  { pattern: 'tool:docker-*', repair: 'docker' },
+  { pattern: 'tool:github-auth*', repair: 'login:github' },
+  { pattern: 'dependencies', repair: 'workspace' },
+  { pattern: 'auth:toolchain', repair: 'workspace' },
+  { pattern: 'workstation:*', repair: 'workstation' },
+  { pattern: 'auth:*', repair: 'login:*' },
+  { pattern: 'auth:vscode', manual: IDENTITY },
+  { pattern: 'auth:workstation', manual: IDENTITY },
+  { pattern: 't3:settings', repair: 't3' },
+  { pattern: 't3:api', repair: 't3' },
+  // The checks that fail when a machine breaks.
+  { pattern: 'worker:service-recovery', repair: 'recovery' },
+  { pattern: 'worker:capacity', repair: 'disk' },
+  { pattern: 'worker:capacity:*', repair: 'disk' },
+  { pattern: 'beszel:agent', repair: 'monitoring' },
+  { pattern: 'beszel:service', repair: 'monitoring' },
+  { pattern: 'beszel:installation', repair: 'monitoring' },
+  { pattern: 'beszel:protocol', repair: 'monitoring' },
+  { pattern: 'tailscale:configuration', repair: 'network' },
+  { pattern: 'tailscale:online', repair: 'network' },
+  { pattern: 'tailscale:reachability', repair: 'network' },
+  { pattern: 'state:replica', repair: 'state' },
+  { pattern: 'state:*', repair: 'state' },
+  // What only a person can do.
+  {
+    pattern: 'beszel:*',
+    manual:
+      'The monitoring hub and its accounts are converged by fleet-sync on the manager; follow the remedy, then run fleet-sync.',
+  },
+  {
+    pattern: 'tailscale:*',
+    manual:
+      'Signing a machine in to the tailnet and approving it is a person’s; the daemon itself is repaired by the network recipe.',
+  },
+  {
+    pattern: 'state:bucket',
+    manual:
+      'The bucket and its credentials are the operator’s: name them in fleet.json and pass the credentials.',
+  },
+  { pattern: 'state:definition', manual: DECLARATION },
+  {
+    pattern: 'worker:*',
+    manual:
+      'Re-converge the worker with fleet-sync: it applies the whole worker policy at the approved revision.',
+  },
+  { pattern: 'worker:power-loss-recovery', manual: HARDWARE },
+  { pattern: 'worker:github-principal', manual: IDENTITY },
+  { pattern: 'worker:vault-renewal', manual: IDENTITY },
+  {
+    pattern: 'manager:*',
+    manual:
+      'The manager converges itself in fleet-sync; a setting that needs an administrator is applied with ansible-playbook -K ansible/macos/manager.yml.',
+  },
+  { pattern: 'manager:identity', manual: DECLARATION },
+  {
+    pattern: 'manager:docker',
+    manual:
+      'Docker Desktop is installed and updated by a person; no playbook installs it.',
+  },
+  { pattern: 'macos:*', manual: HARDWARE },
+  { pattern: 'ubuntu:release', manual: HARDWARE },
+  {
+    pattern: 'ubuntu:*',
+    manual:
+      'Re-converge the worker with fleet-sync: it applies the Ubuntu baseline.',
+  },
+  { pattern: 'platform:*', manual: HARDWARE },
+  {
+    pattern: 'ssh:*',
+    manual:
+      'SSH trust (host keys, keys in the agent, the alias) is set up by a person once per machine; dev-fleet never accepts a host key for you.',
+  },
+  { pattern: 'login:*', manual: IDENTITY },
+  { pattern: 'provider:*', manual: IDENTITY },
+  {
+    pattern: 'agent:*',
+    manual:
+      'Re-converge the machine with fleet-sync; an agent that needs a sign-in is a person’s.',
+  },
+  {
+    pattern: 't3:*',
+    manual:
+      'T3 pairing and its private route are approved by a person; fleet-sync applies the rest.',
+  },
+  { pattern: 'vscode:*', manual: IDENTITY },
+  {
+    pattern: 'git:*',
+    manual:
+      'Uncommitted or unpushed work belongs to a person; dev-fleet never resets a checkout.',
+  },
+  {
+    pattern: 'git:configuration',
+    manual:
+      'fleet-sync writes the Git identity and credential helper with ansible/github/git.yml; run it from the manager.',
+  },
+  {
+    pattern: 'github:*',
+    manual:
+      'fleet-sync signs every node in to GitHub from the stored github-token (a worker retries every ten minutes); a token that is missing, expired or short of a scope or organisation needs a person to issue a new one.',
+  },
+  {
+    pattern: 'github:git-credential',
+    manual:
+      'fleet-sync writes the GitHub credential helper with ansible/github/git.yml when the git block is declared; run it from the manager.',
+  },
+  { pattern: 'github:rate-limit', manual: EVIDENCE },
+  { pattern: 'github:workflow', manual: EVIDENCE },
+  { pattern: 'fleet:*', manual: DECLARATION },
+  { pattern: 'definition:*', manual: DECLARATION },
+  { pattern: 'enrollment:*', manual: DECLARATION },
+  { pattern: 'notify:*', manual: DECLARATION },
+  {
+    pattern: 'proxy:*',
+    manual:
+      'The CLI proxy is installed with devfleet cliproxy install and its accounts are signed in by a person.',
+  },
+  {
+    pattern: 'accounts:roster-current',
+    manual:
+      'The roster reaches every node with devfleet fleet-sync --only accounts, run from the manager.',
+  },
+  { pattern: 'accounts:native-login', manual: IDENTITY },
+  { pattern: 'configuration:*', manual: EVIDENCE },
+  { pattern: 'source:*', manual: EVIDENCE },
+  { pattern: 'bootstrap:*', manual: EVIDENCE },
+  { pattern: 'rollout:*', manual: EVIDENCE },
+  { pattern: 'repair:*', manual: EVIDENCE },
+];
+
+const matches = (pattern: string, id: string): boolean =>
+  pattern.endsWith('*') ? id.startsWith(pattern.slice(0, -1)) : id === pattern;
+
+/** The entry that owns a check, or null for an ID nobody registered. */
+export function checkOwner(id: string): CheckOwner | null {
+  return (
+    CHECK_OWNERS.filter((owner) => matches(owner.pattern, id)).sort(
+      (a, b) => b.pattern.length - a.pattern.length
+    )[0] ?? null
+  );
+}