@crvouga/dev-fleet 0.6.0 → 0.8.0

package/README.md

@@ -128,8 +128,10 @@ npm install --save-dev --save-exact @crvouga/dev-fleet@latest
 npx devfleet fleet-sync              # the fleet moves to that release
 ```
 
-Install an older version and run `fleet-sync` to go back. The first worker
-proves a new revision before the others receive it.
+Install an older version and run `fleet-sync` to go back. One worker, the
+canary, proves a new revision before the others receive it: the first
+reachable worker, or the one `canary` in `fleet.json` (or `--canary`) names
+when it is reachable. An offline worker never holds the others back.
 
 ## Your project's own tools
 
@@ -170,16 +172,10 @@ says where the bucket is and the command is passed its credentials:
 npx devfleet fleet-state             # where the state is, and how it stands with the bucket
 ```
 
-The copy on the manager is disposable. **To replace the manager**, clone your
-project on the new machine and run the same two commands:
-
-```sh
-npm ci
-npx devfleet fleet-sync              # with the two bucket credentials in the environment
-```
-
-The roster and the approved revision come back from the bucket, and the
-machine is rebuilt from Git.
+The copy on the manager is disposable: the roster and the approved revision
+come back from the bucket, and the machine is rebuilt from Git.
+[Replacing the manager](docs/manager-recovery.md) is the runbook for a new or
+lost manager.
 
 Some runs are started by a machine rather than by you: a worker that hands
 `fleet-sync` to the manager, for example. They get no environment, so save the
@@ -228,7 +224,26 @@ propagates them without a full sync.
    Ubuntu machine, over SSH
    ([`fresh-box.feature`](features/fresh-box/fresh-box.feature)).
 
-Remove a machine by deleting its entry and running `fleet-sync`.
+A new machine can also join by itself: `npx devfleet fleet-node-seed --id ID
+--output seed.yaml --apply --yes` writes cloud-init user data (or an Ubuntu
+autoinstall file with `--autoinstall`) holding the worker account with your
+manager's key, the privilege rule, a host key the manager pins at once and the
+pinned Tailscale with a short-lived single-use auth key from your secrets
+provider (`tailnet-auth-key`). Boot a stock Ubuntu image with it, then add the
+worker with `fleet-node-add` and run `fleet-sync`.
+
+`npx devfleet fleet-manifest --id ID` measures what a worker is made of
+(packages, active tools, always-on services) with a digest that is the same
+for two workers built from the same revision; its check `worker:manifest`
+names every difference from the declared pins and, with `--reference` (the
+`--json` manifest of a fresh build), from that build.
+
+Remove a machine with `npx devfleet fleet-node-retire --id ID --apply --yes`
+(it revokes the worker's Vault token, forgets its host key and roster entry
+and leaves a tombstone in the fleet state), then delete its entry and run
+`fleet-sync`. To put a fresh machine in place of a worker under the same id,
+run `npx devfleet fleet-node-replace --id ID --host-key "TYPE BASE64" --apply
+--yes` and then `fleet-sync --node ID`.
 
 ## Configuration
 
@@ -237,6 +252,10 @@ Your `fleet.json` is validated by
 package carries: `fleet-state --example` prints a complete example that points
 `$schema` at it, so an editor checks the file as you type.
 [`examples/fleet.json`](examples/fleet.json) is the same example.
+`devfleet fleet-validate` checks the file without contacting anything and
+names the first field that breaks a rule (`{code, path, expected, actual}`);
+`--ansible` also renders the Ansible inventory and syntax-checks every
+playbook with the pinned Ansible.
 
 | Block      | What it declares                                                    |
 | ---------- | ------------------------------------------------------------------- |

package/ansible/macos/manager-privilege.yml

@@ -0,0 +1,37 @@
+---
+# The one privilege grant a manager needs, given once by its administrator:
+#   ansible-playbook -K manager-privilege.yml -e user=$(id -un)
+# It lets that user run exactly `pmset -c` (the AC power settings
+# manager.yml declares) without a password, so every fleet-sync holds the
+# manager's power policy with no prompt. Run again with -e grant=absent to
+# remove it; nothing else on the manager needs root.
+- name: Grant the fleet manager its power settings without a prompt
+  hosts: localhost
+  connection: local
+  become: true
+  gather_facts: false
+  vars:
+    grant: present
+  tasks:
+    - name: Check the request
+      ansible.builtin.assert:
+        that:
+          - "user is match('^[a-z_][a-z0-9_.-]{0,31}$')"
+          - "user != 'root'"
+          - "grant in ['present', 'absent']"
+    - name: Allow exactly the AC power settings
+      ansible.builtin.copy:
+        dest: '/etc/sudoers.d/dev-fleet-manager-{{ user }}'
+        content: |
+          # Managed by dev-fleet: manager power policy
+          {{ user }} ALL=(root) NOPASSWD: /usr/bin/pmset -c *
+        owner: root
+        group: wheel
+        mode: '0440'
+        validate: /usr/sbin/visudo -cf %s
+      when: grant == 'present'
+    - name: Remove the grant
+      ansible.builtin.file:
+        path: '/etc/sudoers.d/dev-fleet-manager-{{ user }}'
+        state: absent
+      when: grant == 'absent'

package/ansible/macos/manager.yml

@@ -1,13 +1,14 @@
 ---
 # Manager power policy. Workers depend on the manager's singleton services
-# (CLI proxy, Beszel hub), so the manager never sleeps on AC power. Sleep on
-# battery is left to the human. The settings are the `pmset` object in
-# policy.json; fleet-sync measures the same object as manager:power. Changing
-# them needs an administrator: ansible-playbook -K manager.yml
+# (CLI proxy, Beszel hub), so the manager never sleeps on AC power and starts
+# again by itself when power returns (autorestart). Sleep on battery is left
+# to the human. The settings are the `pmset` object in policy.json; fleet-sync
+# measures the same object as manager:power and applies this playbook on every
+# run, without a prompt, because `pmset -c` runs through the narrow grant
+# manager-privilege.yml installs once.
 - name: Keep the fleet manager awake on AC power
   hosts: localhost
   connection: local
-  become: true
   gather_facts: false
   vars_files:
     - policy.json
@@ -19,7 +20,15 @@
       register: pmset_custom
     - name: Apply the declared AC power settings
       ansible.builtin.command:
-        argv: [/usr/bin/pmset, -c, '{{ item.key }}', '{{ item.value }}']
+        argv:
+          [
+            /usr/bin/sudo,
+            -n,
+            /usr/bin/pmset,
+            -c,
+            '{{ item.key }}',
+            '{{ item.value }}',
+          ]
       loop: '{{ pmset | dict2items }}'
       loop_control:
         label: '{{ item.key }}'
@@ -29,5 +38,41 @@
       when: observed != item.value
       changed_when: true
 
+# Docker Desktop hosts the CLI proxy and the monitoring hub, so it starts at
+# login: after a power cut the manager comes back without a person. The
+# setting is the `dockerDesktop` object in policy.json; fleet-sync measures it
+# as part of manager:boot-recovery. Only that key of the user's settings file
+# is changed.
+- name: Start Docker Desktop at login
+  hosts: localhost
+  connection: local
+  become: false
+  gather_facts: false
+  vars_files:
+    - policy.json
+  vars:
+    docker_settings: "{{ lookup('ansible.builtin.env', 'HOME') }}/{{ dockerDesktop.settings }}"
+  tasks:
+    - name: Find the Docker Desktop settings
+      ansible.builtin.stat:
+        path: '{{ docker_settings }}'
+      register: docker_settings_file
+    - name: Read the Docker Desktop settings
+      ansible.builtin.slurp:
+        src: '{{ docker_settings }}'
+      register: docker_settings_text
+      when: docker_settings_file.stat.exists
+    - name: Turn on start at login, keeping every other setting
+      ansible.builtin.copy:
+        dest: '{{ docker_settings }}'
+        content: "{{ current | combine({dockerDesktop.key: true}) | to_nice_json(indent=2) }}\n"
+        mode: '0644'
+      vars:
+        current: '{{ docker_settings_text.content | b64decode | from_json }}'
+      # Docker Desktop writes the file the first time it runs.
+      when:
+        - docker_settings_file.stat.exists
+        - (current[dockerDesktop.key] | default(false)) != true
+
 - name: Stage the fleet's T3 policy
   ansible.builtin.import_playbook: t3-policy.yml

package/ansible/macos/policy.json

@@ -4,7 +4,8 @@
   "architectures": ["arm64", "x86_64"],
   "pmset": {
     "sleep": "0",
-    "womp": "1"
+    "womp": "1",
+    "autorestart": "1"
   },
   "beszelDaemon": "dev.devfleet.beszel-agent",
   "disk": {
@@ -16,5 +17,9 @@
     "unhealthyChecks": 3,
     "maxRestartsPerHour": 3,
     "intervalSeconds": 60
+  },
+  "dockerDesktop": {
+    "settings": "Library/Group Containers/group.com.docker/settings-store.json",
+    "key": "AutoStart"
   }
 }

package/ansible/macos/worker-host-key.yml

@@ -0,0 +1,33 @@
+---
+# A worker's pinned SSH host key in the manager's known_hosts. fleet-sync
+# connects with StrictHostKeyChecking=yes, so this is the only way a key is
+# trusted: devfleet fleet-node-replace pins the key of a replacement worker
+# (given by the operator, or by the seed that created it with
+# fleet-node-seed) and devfleet fleet-node-retire forgets it. A key is never
+# learned on first use. Variables: host (the name ssh connects to, with
+# [host]:port for another port), pin (present or absent), key (with present:
+# "TYPE BASE64") and known_hosts (the file).
+- name: Pin or forget a worker's SSH host key on the manager
+  hosts: localhost
+  connection: local
+  gather_facts: false
+  tasks:
+    - name: Check the request
+      ansible.builtin.assert:
+        that:
+          - "host is match('^\\[?[a-zA-Z0-9][a-zA-Z0-9._-]{0,252}(\\]:[0-9]{1,5})?$')"
+          - "pin in ['present', 'absent']"
+          - "pin == 'absent' or (key is match('^(ssh-ed25519|ecdsa-sha2-nistp256|ssh-rsa) [A-Za-z0-9+/=]{16,}$'))"
+    - name: Pin the worker's host key
+      ansible.builtin.known_hosts:
+        name: '{{ host }}'
+        key: '{{ host }} {{ key }}'
+        path: '{{ known_hosts }}'
+        state: present
+      when: pin == 'present'
+    - name: Forget every key of the retired worker
+      ansible.builtin.known_hosts:
+        name: '{{ host }}'
+        path: '{{ known_hosts }}'
+        state: absent
+      when: pin == 'absent'

package/ansible/monitoring/agent.yml

@@ -4,6 +4,8 @@
   connection: local
   become: true
   gather_facts: true
+  # fleet-repair re-applies this play alone for a failing beszel agent check.
+  tags: [monitoring]
   vars:
     beszel_lock: "{{ lookup('ansible.builtin.file', 'lock.json') | from_json }}"
     beszel_arch: "{{ {'x86_64': 'amd64', 'aarch64': 'arm64'}.get(ansible_facts.architecture, 'unsupported') }}"

package/ansible/ubuntu/baseline.yml

@@ -8,24 +8,30 @@
     - policy.json
   pre_tasks:
     - ansible.builtin.import_tasks: tasks/release.yml
+      tags: [always]
     - name: Require an explicit non-root user
       ansible.builtin.assert:
         that:
           - development_user is match('^[a-z_][a-z0-9_-]{0,31}$')
           - development_user != 'root'
         fail_msg: Unsupported user; name the non-root developer account.
+      tags: [always]
     - name: Read the designated existing account
       ansible.builtin.getent:
         database: passwd
         key: '{{ development_user }}'
+      tags: [always]
     - name: Refuse a different user identity or home
       ansible.builtin.assert:
         that:
           - ansible_facts.getent_passwd[development_user][1] | int == development_uid | int
           - ansible_facts.getent_passwd[development_user][4] == development_home
+      tags: [always]
     - ansible.builtin.import_tasks: tasks/dpkg.yml
     # Memory and disk safety must land even when the disk is nearly full.
+    # fleet-repair re-applies them alone (--tags disk) for worker:capacity:*.
     - ansible.builtin.import_tasks: tasks/resources.yml
+      tags: [disk]
     - name: Inspect capacity before package and service changes
       ansible.builtin.command:
         argv:

package/ansible/ubuntu/recovery.yml

@@ -4,6 +4,8 @@
   connection: local
   become: true
   gather_facts: false
+  # fleet-repair re-applies this play alone for worker:service-recovery.
+  tags: [recovery]
   tasks:
     - name: Verify the desktop unit runs before adding any recovery trigger
       ansible.builtin.command:

package/ansible/ubuntu/tailscale.yml

@@ -4,6 +4,8 @@
   connection: local
   become: true
   gather_facts: true
+  # fleet-repair re-applies this play alone for a failing tailscale:* check.
+  tags: [network]
   # The approved version and the signing key's digest: `tailscale` in policy.json.
   vars_files:
     - policy.json

package/bootstrap/worker-seed.yaml

@@ -0,0 +1,61 @@
+# The first boot of a new worker, as cloud-init user data. devfleet
+# fleet-node-seed fills each ${NAME} with a value of the worker it seeds:
+# values from fleet.json, the manager's public key, a host key generated for
+# the machine, the Tailscale version and key digest ansible/ubuntu/policy.json
+# pins, and a short-lived single-use auth key that is deleted once the machine
+# has joined. The sudoers fragment is the one ansible/ubuntu/authorize-sudo.sh
+# writes: its path and marker are already on running machines.
+hostname: '${HOSTNAME}'
+users:
+  - name: '${USER}'
+    groups: [sudo]
+    shell: /bin/bash
+    lock_passwd: true
+    ssh_authorized_keys: ['${MANAGER_KEY}']
+ssh_pwauth: false
+ssh_deletekeys: true
+ssh_genkeytypes: []
+ssh_keys:
+  ed25519_private: '${HOST_KEY_PRIVATE}'
+  ed25519_public: '${HOST_KEY_PUBLIC}'
+write_files:
+  - path: '/etc/sudoers.d/personal-fleet-${USER}'
+    owner: root:root
+    permissions: '0440'
+    content: "# Managed by personal Fleet: designated development user\n${USER} ALL=(ALL:ALL) NOPASSWD: ALL\n"
+  - path: /run/devfleet/tailnet-auth-key
+    owner: root:root
+    permissions: '0600'
+    content: "${AUTH_KEY}\n"
+runcmd:
+  - [visudo, -cf, '/etc/sudoers.d/personal-fleet-${USER}']
+  - [
+      curl,
+      --fail,
+      --silent,
+      --show-error,
+      --location,
+      --output,
+      /usr/share/keyrings/tailscale-archive-keyring.gpg,
+      'https://pkgs.tailscale.com/stable/ubuntu/${CODENAME}.noarmor.gpg',
+    ]
+  - [
+      sh,
+      -c,
+      "echo '${TAILSCALE_KEY_SHA256}  /usr/share/keyrings/tailscale-archive-keyring.gpg' | sha256sum -c -",
+    ]
+  - [
+      sh,
+      -c,
+      "echo 'deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/ubuntu ${CODENAME} main' > /etc/apt/sources.list.d/tailscale.list",
+    ]
+  - [apt-get, update]
+  - [apt-get, install, --yes, 'tailscale=${TAILSCALE_VERSION}']
+  - [
+      tailscale,
+      up,
+      '--auth-key=file:/run/devfleet/tailnet-auth-key',
+      '--hostname=${HOSTNAME}',
+      '--ssh=false',
+    ]
+  - [rm, -f, /run/devfleet/tailnet-auth-key]

package/docs/authentication.md

@@ -59,6 +59,7 @@ Every failure code is in [the runbooks](runbooks.md).
 | `monitoring-backup-secret-key` | when its feature is declared | `DEVFLEET_SECRET_MONITORING_BACKUP_SECRET_KEY` | `~/.config/dev-fleet/secrets/monitoring-backup-secret-key` |
 | `notify-webhook-url`           | when its feature is declared | `DEVFLEET_SECRET_NOTIFY_WEBHOOK_URL`           | `~/.config/dev-fleet/secrets/notify-webhook-url`           |
 | `notify-ntfy-url`              | when its feature is declared | `DEVFLEET_SECRET_NOTIFY_NTFY_URL`              | `~/.config/dev-fleet/secrets/notify-ntfy-url`              |
+| `tailnet-auth-key`             | when its feature is declared | `DEVFLEET_SECRET_TAILNET_AUTH_KEY`             | `~/.config/dev-fleet/secrets/tailnet-auth-key`             |
 
 <!-- end: secrets -->
 
@@ -70,6 +71,7 @@ Every failure code is in [the runbooks](runbooks.md).
 | ----------------------------- | ------------------------------------------ |
 | `auth:github`                 | `src/github/checks.ts`                     |
 | `auth:vault`                  | `src/cli/fleet-sync-vault.ts`              |
+| `auth:vault-isolation`        | `src/cli/fleet-sync-vault.ts`              |
 | `worker:git-identity`         | `src/git-identity.ts`                      |
 | `worker:git-push-auth`        | `src/definition.ts`, `src/git-identity.ts` |
 | `worker:github-sync`          | `src/worker-access-checks.ts`              |

package/docs/consumer-setup.md

@@ -153,36 +153,37 @@ npx devfleet fleet-state --example > fleet.json
 Then replace every placeholder. Each field, from
 [`schema/fleet.schema.json`](../schema/fleet.schema.json):
 
-| Field                                                 | Value                                                                                                                |
-| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
-| `version`                                             | `3`                                                                                                                  |
-| `source.repository`                                   | `https://github.com/crvouga/dev-fleet.git`, or your fork if you publish one. Machines only accept code from it.      |
-| `state`                                               | `{ "bucket", "endpoint"?, "region"?, "prefix"? }` or `{ "directory": "/absolute/path" }`. Never credentials.         |
-| `secrets`                                             | See [secrets](#secrets).                                                                                             |
-| `manager.id`                                          | A short lowercase id, `^[a-z][a-z0-9-]{0,62}$`, e.g. `manager-mac`.                                                  |
-| `manager.os`                                          | `"macOS"`                                                                                                            |
-| `manager.network.tailscaleName`                       | The manager's MagicDNS name: `tailscale status --json` → `.Self.DNSName` without the trailing dot.                   |
-| `manager.network.user`                                | The manager's login name (`id -un`).                                                                                 |
-| `manager.neverSleepOnPower`                           | `true` (workers depend on the manager's services).                                                                   |
-| `workers[].id`                                        | A short lowercase id. The **first** worker is the canary: a new revision must pass there before the others get it.   |
-| `workers[].ssh`                                       | The worker's alias in the manager's `~/.ssh/config`.                                                                 |
-| `workers[].os`                                        | `"linux"`                                                                                                            |
-| `workers[].network.tailscaleName`                     | The worker's MagicDNS name (from `tailscale status --json` → `.Peer[].DNSName` on the manager).                      |
-| `workers[].network.sshPort`                           | `22` unless the worker's sshd listens elsewhere.                                                                     |
-| `workers[].network.user`                              | The worker account (`ssh WORKER_ALIAS id -un`).                                                                      |
-| `workers[].network.t3Port`                            | `3773`                                                                                                               |
-| `services.proxy.endpoint`                             | The public HTTPS origin of the CLI proxy, e.g. `https://cli-proxy.example.com`.                                      |
-| `services.monitoring.hubPort`                         | `8443`                                                                                                               |
-| `services.monitoring.publicUrl`                       | The public HTTPS origin of the monitoring dashboard.                                                                 |
-| `services.monitoring.managerAccount`                  | The email of the monitoring account the manager enrolls machines with.                                               |
-| `services.monitoring.viewerAccount`                   | The email of the read-only account visitors are signed in as.                                                        |
-| `ubuntu`                                              | `{ "release": "26.04", "architectures": ["amd64", "arm64"] }`: a release of the [support matrix](support-matrix.md). |
-| `worker.requiredCapabilities`                         | Normally all of `github`, `vault`, `t3`, `toolchain`, `workstation`, `worker`.                                       |
-| `worker.githubLogin`                                  | The GitHub login every worker must be signed in as.                                                                  |
-| `github` (optional)                                   | Who the manager signs in to GitHub as, per-node tokens, required scopes; see [GitHub](github.md).                    |
-| `worker.vaultUser`                                    | The Vault user each worker's token is issued for.                                                                    |
-| `worker.toolchain`, `agentProfile`                    | `"mise.lock"`, `"proxy"` (fixed).                                                                                    |
-| `worker.unattendedSudo`, `persistentT3`, `neverSleep` | `true` (fixed policy).                                                                                               |
+| Field                                                 | Value                                                                                                                    |
+| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
+| `version`                                             | `3`                                                                                                                      |
+| `source.repository`                                   | `https://github.com/crvouga/dev-fleet.git`, or your fork if you publish one. Machines only accept code from it.          |
+| `state`                                               | `{ "bucket", "endpoint"?, "region"?, "prefix"? }` or `{ "directory": "/absolute/path" }`. Never credentials.             |
+| `secrets`                                             | See [secrets](#secrets).                                                                                                 |
+| `manager.id`                                          | A short lowercase id, `^[a-z][a-z0-9-]{0,62}$`, e.g. `manager-mac`.                                                      |
+| `manager.os`                                          | `"macOS"`                                                                                                                |
+| `manager.network.tailscaleName`                       | The manager's MagicDNS name: `tailscale status --json` → `.Self.DNSName` without the trailing dot.                       |
+| `manager.network.user`                                | The manager's login name (`id -un`).                                                                                     |
+| `manager.neverSleepOnPower`                           | `true` (workers depend on the manager's services).                                                                       |
+| `workers[].id`                                        | A short lowercase id. The first reachable worker is the canary: a new revision must pass there before the others get it. |
+| `canary`                                              | Optional: the id of the worker to prefer as canary. When it is unreachable, the next reachable worker is used.           |
+| `workers[].ssh`                                       | The worker's alias in the manager's `~/.ssh/config`.                                                                     |
+| `workers[].os`                                        | `"linux"`                                                                                                                |
+| `workers[].network.tailscaleName`                     | The worker's MagicDNS name (from `tailscale status --json` → `.Peer[].DNSName` on the manager).                          |
+| `workers[].network.sshPort`                           | `22` unless the worker's sshd listens elsewhere.                                                                         |
+| `workers[].network.user`                              | The worker account (`ssh WORKER_ALIAS id -un`).                                                                          |
+| `workers[].network.t3Port`                            | `3773`                                                                                                                   |
+| `services.proxy.endpoint`                             | The public HTTPS origin of the CLI proxy, e.g. `https://cli-proxy.example.com`.                                          |
+| `services.monitoring.hubPort`                         | `8443`                                                                                                                   |
+| `services.monitoring.publicUrl`                       | The public HTTPS origin of the monitoring dashboard.                                                                     |
+| `services.monitoring.managerAccount`                  | The email of the monitoring account the manager enrolls machines with.                                                   |
+| `services.monitoring.viewerAccount`                   | The email of the read-only account visitors are signed in as.                                                            |
+| `ubuntu`                                              | `{ "release": "26.04", "architectures": ["amd64", "arm64"] }`: a release of the [support matrix](support-matrix.md).     |
+| `worker.requiredCapabilities`                         | Normally all of `github`, `vault`, `t3`, `toolchain`, `workstation`, `worker`.                                           |
+| `worker.githubLogin`                                  | The GitHub login every worker must be signed in as.                                                                      |
+| `github` (optional)                                   | Who the manager signs in to GitHub as, per-node tokens, required scopes; see [GitHub](github.md).                        |
+| `worker.vaultUser`                                    | The Vault user each worker's token is issued for.                                                                        |
+| `worker.toolchain`, `agentProfile`                    | `"mise.lock"`, `"proxy"` (fixed).                                                                                        |
+| `worker.unattendedSudo`, `persistentT3`, `neverSleep` | `true` (fixed policy).                                                                                                   |
 
 Use the same values in every place they repeat. Validate before going on:
 
@@ -363,7 +364,7 @@ Read `summary` and `remedy.action` of the failing check. The common ones:
 | `ROOT_DEVELOPMENT_USER_REJECTED`          | The worker's SSH user is root. Use a normal account with sudo.                                                                |
 | `BOOTSTRAP_DISK_CAPACITY_REQUIRED`        | Less than 2 GiB free (or too few inodes) where the fleet keeps its files on the worker.                                       |
 | `FLEET_REMOTE_LOCKED`                     | Another operation is running on that worker. Wait and run again.                                                              |
-| `FLEET_PROMOTION_BLOCKED`                 | The canary (first worker) failed, so the new revision went no further. Fix the canary first.                                  |
+| `FLEET_PROMOTION_BLOCKED`                 | The canary (the report names it under `canary`) failed, so the new revision went no further. Fix the canary first.            |
 | `VAULT_LOGIN_REQUIRED`                    | The manager is not logged in to Vault. The human logs in; the next sync issues worker tokens.                                 |
 | `PROVIDER_UNATTENDED_CREDENTIAL_REQUIRED` | A coding agent has no credential a background service can use. The human signs in on the proxy.                               |
 | `BESZEL_ACCOUNT_REQUIRED`                 | The monitoring hub has no account yet. Follow the remedy; other workers are not blocked.                                      |
@@ -382,10 +383,8 @@ and codes; leave out everything private.
 - **Upgrade the fleet.** `npm install --save-dev --save-exact @crvouga/dev-fleet@latest`,
   commit, `npx devfleet fleet-sync`. The canary proves the new revision first.
   Install an older version and sync to roll back.
-- **Replace the manager.** On the new Mac: clone the project, `npm ci`, put
-  the bucket credentials in the environment, `npx devfleet fleet-sync`. The
-  roster and the approved revision come back from the bucket. See
-  [state](state.md) for what the bucket does not hold yet.
+- **Replace the manager.** Follow
+  [replacing the manager](manager-recovery.md).
 
 ## Moving from a pinned commit and a wrapper script
 

package/docs/declared.md

@@ -12,29 +12,29 @@ may only shrink.
 
 ## Declared
 
-| What                                          | Declared in                                                                                         | Pinned by                                                            | Applied by                                                | Measured by                                  |
-| --------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | --------------------------------------------------------- | -------------------------------------------- |
-| The fleet: machines, services, accounts       | Your `fleet.json`                                                                                   | The dev-fleet version in your `package.json`, which names one commit | `fleet-sync`                                              | `fleet-status`                               |
-| Tools: languages, CLIs, coding agents         | `mise.toml`                                                                                         | `mise.lock`: version and checksum for each platform                  | `mise install --locked`                                   | Tool and `agent:verification` checks         |
-| mise itself                                   | `min_version` and `mise_*_sha256` in `mise.toml`                                                    | The digest of its release archive, held here                         | `bootstrap/bootstrap.sh`                                  | Its version, at every toolchain setup        |
-| JavaScript dependencies                       | `package.json`                                                                                      | `bun.lock`                                                           | `bun install --frozen-lockfile`                           | The install fails on a difference            |
-| T3 Code, server and desktop app               | `t3_*` under `[vars]` in `mise.toml`                                                                | SHA-256 of each artifact                                             | `bootstrap/native-t3.ts`; `ansible/ubuntu/t3-desktop.yml` | T3 checks; `t3:version` on every machine     |
-| T3 policy on the manager (updates off)        | `ansible/ubuntu/t3/policy.json`, shared with the workers                                            | The commit                                                           | `ansible/macos/t3-policy.yml`, once, as an administrator  | `t3:updates`                                 |
-| Worker work-in-progress guard                 | `ansible/ubuntu/wip-guard.py`, `ansible/ubuntu/tasks/recovery.yml`                                  | The commit                                                           | `fleet-sync`                                              | `features/ide/work-in-progress.feature`      |
-| Ubuntu packages                               | `packages` and `snapshot` in `ansible/ubuntu/policy.json`                                           | Exact versions from a dated archive snapshot                         | `ansible/ubuntu/tasks/packages.yml`                       | `package:*`                                  |
-| Tailscale on workers                          | `tailscale` in `ansible/ubuntu/policy.json`                                                         | Exact version; digest of the signing key                             | `ansible/ubuntu/tailscale.yml`                            | `package:tailscale`                          |
-| Worker policy: power, SSH, sudo, memory, disk | `ansible/ubuntu/*.yml`, values shared with checks in `policy.json`                                  | The commit                                                           | `fleet-sync`                                              | Host and resource checks                     |
-| Monitoring: hub, tunnel, agents               | `ansible/monitoring/*.yml`, `compose.yml`, `lock.json`                                              | Image digests; SHA-256 of each agent archive                         | `fleet-sync`                                              | `beszel:*`                                   |
-| The manager's private route to the hub        | `ansible/monitoring/hub-route-darwin.yml`                                                           | The commit                                                           | `fleet-sync`                                              | The route, read back                         |
-| The manager's power settings                  | `pmset` in `ansible/macos/policy.json`                                                              | The commit                                                           | `ansible/macos/manager.yml`, once, as an administrator    | `manager:power`                              |
-| Ansible on the manager                        | `ansible/pyproject.toml`                                                                            | `ansible/uv.lock`: one release, one Python, every dependency hashed  | `uv run --locked --isolated`                              | A hash that differs stops the run            |
-| Ansible on a worker                           | `ansible-core` in `ansible/ubuntu/policy.json`                                                      | Exact version                                                        | `ansible/ubuntu/tasks/packages.yml`                       | `package:ansible-core`                       |
-| The CLI proxy                                 | `cliproxy/compose.yaml`, `image.lock.json`, `requirements.lock`, `launchd/`, `config.template.yaml` | Image digests; hashed Python dependencies                            | `devfleet cliproxy install`, and `fleet-sync` after it    | The proxy's audit                            |
-| Agent profiles and T3 providers               | `agents/registry.json`, `agents/t3.json`                                                            | The commit                                                           | `fleet-sync`                                              | Profile receipts; the plan after every apply |
-| Worker time service                           | `ansible/ubuntu/tasks/services.yml`, offsets in `clock` in `ansible/ubuntu/policy.json`             | The commit                                                           | `fleet-sync`                                              | `worker:clock`                               |
-| The manager's container watchdog              | `ansible/monitoring/container-watchdog.py`, `containerWatchdog` in `ansible/macos/policy.json`      | The commit                                                           | `fleet-sync` (`hub-darwin.yml`)                           | `manager:container-watchdog`                 |
-| Docker Desktop on the manager                 | `docker_desktop_*` under `[vars]` in `mise.toml`                                                    | Version and build                                                    | You                                                       | `manager:docker` (warns)                     |
-| CI                                            | `.github/workflows/`                                                                                | Each action at a commit; tools from `mise.lock`                      | GitHub                                                    | `features/rules/`                            |
+| What                                          | Declared in                                                                                         | Pinned by                                                            | Applied by                                                                                    | Measured by                                  |
+| --------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | -------------------------------------------- |
+| The fleet: machines, services, accounts       | Your `fleet.json`                                                                                   | The dev-fleet version in your `package.json`, which names one commit | `fleet-sync`                                                                                  | `fleet-status`                               |
+| Tools: languages, CLIs, coding agents         | `mise.toml`                                                                                         | `mise.lock`: version and checksum for each platform                  | `mise install --locked`                                                                       | Tool and `agent:verification` checks         |
+| mise itself                                   | `min_version` and `mise_*_sha256` in `mise.toml`                                                    | The digest of its release archive, held here                         | `bootstrap/bootstrap.sh`                                                                      | Its version, at every toolchain setup        |
+| JavaScript dependencies                       | `package.json`                                                                                      | `bun.lock`                                                           | `bun install --frozen-lockfile`                                                               | The install fails on a difference            |
+| T3 Code, server and desktop app               | `t3_*` under `[vars]` in `mise.toml`                                                                | SHA-256 of each artifact                                             | `bootstrap/native-t3.ts`; `ansible/ubuntu/t3-desktop.yml`                                     | T3 checks; `t3:version` on every machine     |
+| T3 policy on the manager (updates off)        | `ansible/ubuntu/t3/policy.json`, shared with the workers                                            | The commit                                                           | `ansible/macos/t3-policy.yml`, once, as an administrator                                      | `t3:updates`                                 |
+| Worker work-in-progress guard                 | `ansible/ubuntu/wip-guard.py`, `ansible/ubuntu/tasks/recovery.yml`                                  | The commit                                                           | `fleet-sync`                                                                                  | `features/ide/work-in-progress.feature`      |
+| Ubuntu packages                               | `packages` and `snapshot` in `ansible/ubuntu/policy.json`                                           | Exact versions from a dated archive snapshot                         | `ansible/ubuntu/tasks/packages.yml`                                                           | `package:*`                                  |
+| Tailscale on workers                          | `tailscale` in `ansible/ubuntu/policy.json`                                                         | Exact version; digest of the signing key                             | `ansible/ubuntu/tailscale.yml`                                                                | `package:tailscale`                          |
+| Worker policy: power, SSH, sudo, memory, disk | `ansible/ubuntu/*.yml`, values shared with checks in `policy.json`                                  | The commit                                                           | `fleet-sync`                                                                                  | Host and resource checks                     |
+| Monitoring: hub, tunnel, agents               | `ansible/monitoring/*.yml`, `compose.yml`, `lock.json`                                              | Image digests; SHA-256 of each agent archive                         | `fleet-sync`                                                                                  | `beszel:*`                                   |
+| The manager's private route to the hub        | `ansible/monitoring/hub-route-darwin.yml`                                                           | The commit                                                           | `fleet-sync`                                                                                  | The route, read back                         |
+| The manager's power settings                  | `pmset` in `ansible/macos/policy.json`                                                              | The commit                                                           | `ansible/macos/manager.yml`, on every `fleet-sync`, after one grant (`manager-privilege.yml`) | `manager:power`                              |
+| Ansible on the manager                        | `ansible/pyproject.toml`                                                                            | `ansible/uv.lock`: one release, one Python, every dependency hashed  | `uv run --locked --isolated`                                                                  | A hash that differs stops the run            |
+| Ansible on a worker                           | `ansible-core` in `ansible/ubuntu/policy.json`                                                      | Exact version                                                        | `ansible/ubuntu/tasks/packages.yml`                                                           | `package:ansible-core`                       |
+| The CLI proxy                                 | `cliproxy/compose.yaml`, `image.lock.json`, `requirements.lock`, `launchd/`, `config.template.yaml` | Image digests; hashed Python dependencies                            | `devfleet cliproxy install`, and `fleet-sync` after it                                        | The proxy's audit                            |
+| Agent profiles and T3 providers               | `agents/registry.json`, `agents/t3.json`                                                            | The commit                                                           | `fleet-sync`                                                                                  | Profile receipts; the plan after every apply |
+| Worker time service                           | `ansible/ubuntu/tasks/services.yml`, offsets in `clock` in `ansible/ubuntu/policy.json`             | The commit                                                           | `fleet-sync`                                                                                  | `worker:clock`                               |
+| The manager's container watchdog              | `ansible/monitoring/container-watchdog.py`, `containerWatchdog` in `ansible/macos/policy.json`      | The commit                                                           | `fleet-sync` (`hub-darwin.yml`)                                                               | `manager:container-watchdog`                 |
+| Docker Desktop on the manager                 | `docker_desktop_*` under `[vars]` in `mise.toml`                                                    | Version and build                                                    | You                                                                                           | `manager:docker` (warns)                     |
+| CI                                            | `.github/workflows/`                                                                                | Each action at a commit; tools from `mise.lock`                      | GitHub                                                                                        | `features/rules/`                            |
 
 A value lives in a policy file when something besides the playbook reads it,
 so the check and the playbook cannot disagree. A value only a playbook uses
@@ -89,4 +89,4 @@ would settle and the current tools do not is the rest of a worker's packages.
 | An Ubuntu package       | `packages` in `ansible/ubuntu/policy.json`, and `snapshot` if needed | The VM test in `ansible/ubuntu/tests`            |
 | Ansible on the manager  | `ansible/pyproject.toml`                                             | `uv lock` in `ansible/`                          |
 | Beszel                  | `ansible/monitoring/lock.json`, `compose.yml`, the agent unit        | `bun test features/rules`                        |
-| A manager power setting | `pmset` in `ansible/macos/policy.json`                               | Run `manager.yml` once on the manager            |
+| A manager power setting | `pmset` in `ansible/macos/policy.json`                               | The next `fleet-sync` applies it                 |