@crvouga/dev-fleet 0.7.0 → 0.9.0

package/docs/authentication.md

@@ -8,14 +8,14 @@ rewrites them).
 
 ## Where credentials live
 
-| Credential                          | Lives in                                                                                                                      | Read by                                          |
-| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
-| The fleet's secrets (table below)   | The provider `secrets` names in `fleet.json`                                                                                  | The node that needs it, through `src/secrets.ts` |
-| The bucket's two keys               | The environment of the `devfleet` command; `~/.config/dev-fleet/bucket.json` (0600) on the manager after `fleet-state --init` | The manager only                                 |
-| A node's Vault token                | That node's own Vault session (`~/.vault-token`)                                                                              | That node only; never copied                     |
-| A node's GitHub login               | `gh`'s own store on that node                                                                                                 | `gh`, and git through its credential helper      |
-| Provider accounts (Claude, Codex …) | The CLI proxy's store on the manager                                                                                          | The proxy; nodes hold only the proxy client key  |
-| Tailscale and T3 pairing            | Each machine, signed in by a person                                                                                           | That machine                                     |
+| Credential                          | Lives in                                                                                                                       | Read by                                          |
+| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------ |
+| The fleet's secrets (table below)   | The provider `secrets` names in `fleet.json`                                                                                   | The node that needs it, through `src/secrets.ts` |
+| The bucket's two keys               | The environment of the `devfleet` command; `~/.config/dev-fleet/bucket.json` (0600) on the manager after `fleet-state --init`  | The manager only                                 |
+| A node's Vault token                | That node's own Vault session (`~/.vault-token`)                                                                               | That node only; never copied                     |
+| A node's GitHub login               | `gh`'s own store on that node                                                                                                  | `gh`, and git through its credential helper      |
+| Provider accounts (Claude, Codex …) | The CLI proxy's store on the manager                                                                                           | The proxy; nodes hold only the proxy client key  |
+| Tailscale and T3 pairing            | Each machine, signed in by a person; a replaced worker's T3 is paired by the manager with a one-time code it never writes down | That machine                                     |
 
 ## How they reach each node
 
@@ -30,14 +30,19 @@ rewrites them).
   makes `gh` git's credential helper and writes the declared identity
   (`git` in `fleet.json`, `ansible/github/git.yml`) on every node.
 - **The proxy client key** is read by each node from its own secrets provider
-  whenever a launcher starts an agent.
+  whenever a launcher starts an agent. The node keeps the last key it read in
+  `~/.cache/dev-fleet/proxy-client-key` (0600) and uses it only while the
+  provider cannot answer, so an unreachable Vault or an expired session leaves
+  that node's agents working until the Vault checks below are fixed.
 - **The monitoring hub's backup storage keys** are read on the manager by
   `fleet-sync` and handed to the hub's playbook in a private variables file.
 
 ## Rotation and failure
 
-Rotate a secret where the provider keeps it; the next read uses it, nothing
-caches it. A Vault token that is not renewed expires: `worker:vault-renewal`
+Rotate a secret where the provider keeps it; the next read uses it. Only the
+proxy client key is remembered, and only as a fallback; a rotated key reaches
+every node at its next launch, and `retire-key` should wait until every node
+has launched an agent since the rotation. A Vault token that is not renewed expires: `worker:vault-renewal`
 fails and the fix is one login of that machine. A GitHub token that stops
 working fails `auth:github` and `worker:git-push-auth`; store the new one in
 Vault and the worker restores itself within ten minutes. During a proxy
@@ -59,6 +64,8 @@ Every failure code is in [the runbooks](runbooks.md).
 | `monitoring-backup-secret-key` | when its feature is declared | `DEVFLEET_SECRET_MONITORING_BACKUP_SECRET_KEY` | `~/.config/dev-fleet/secrets/monitoring-backup-secret-key` |
 | `notify-webhook-url`           | when its feature is declared | `DEVFLEET_SECRET_NOTIFY_WEBHOOK_URL`           | `~/.config/dev-fleet/secrets/notify-webhook-url`           |
 | `notify-ntfy-url`              | when its feature is declared | `DEVFLEET_SECRET_NOTIFY_NTFY_URL`              | `~/.config/dev-fleet/secrets/notify-ntfy-url`              |
+| `tailnet-auth-key`             | when its feature is declared | `DEVFLEET_SECRET_TAILNET_AUTH_KEY`             | `~/.config/dev-fleet/secrets/tailnet-auth-key`             |
+| `tailnet-admin`                | when its feature is declared | `DEVFLEET_SECRET_TAILNET_ADMIN`                | `~/.config/dev-fleet/secrets/tailnet-admin`                |
 
 <!-- end: secrets -->
 
@@ -70,8 +77,11 @@ Every failure code is in [the runbooks](runbooks.md).
 | ----------------------------- | ------------------------------------------ |
 | `auth:github`                 | `src/github/checks.ts`                     |
 | `auth:vault`                  | `src/cli/fleet-sync-vault.ts`              |
+| `auth:vault-isolation`        | `src/cli/fleet-sync-vault.ts`              |
+| `worker:credential-renewal`   | `src/worker-host-checks.ts`                |
 | `worker:git-identity`         | `src/git-identity.ts`                      |
 | `worker:git-push-auth`        | `src/definition.ts`, `src/git-identity.ts` |
+| `worker:git-signing`          | `src/definition.ts`, `src/git-identity.ts` |
 | `worker:github-sync`          | `src/worker-access-checks.ts`              |
 | `worker:proxy-authentication` | `src/worker-access-checks.ts`              |
 | `worker:vault-renewal`        | `src/worker-access-checks.ts`              |

package/docs/consumer-setup.md

@@ -24,13 +24,13 @@ Nothing else is needed. You never clone dev-fleet, copy its files or write a
 wrapper script around a checkout of it: the npm package is the tool, and the
 version in `package.json` is the commit every machine runs.
 
-| Concept     | Meaning                                                                                                |
-| ----------- | ------------------------------------------------------------------------------------------------------ |
-| manager     | The one macOS machine you run `devfleet fleet-sync` on. Hosts the CLI proxy and the monitoring hub.    |
-| worker      | An Ubuntu 26.04 machine the manager converges over SSH. Stateless: rebuilt from Git and your secrets.  |
-| fleet state | One SQLite database (roster, approved revision, configuration last applied) replicated to your bucket. |
-| bucket      | Any S3-compatible bucket you own, or a directory. The source of truth for the fleet state.             |
-| revision    | The dev-fleet commit the installed package was published from (`npx devfleet --version`).              |
+| Concept     | Meaning                                                                                                        |
+| ----------- | -------------------------------------------------------------------------------------------------------------- |
+| manager     | The one macOS machine you run `devfleet fleet-sync` on. Hosts the CLI proxy and the monitoring hub.            |
+| worker      | An Ubuntu 26.04 or 24.04 machine the manager converges over SSH. Stateless: rebuilt from Git and your secrets. |
+| fleet state | One SQLite database (roster, approved revision, configuration last applied) replicated to your bucket.         |
+| bucket      | Any S3-compatible bucket you own, or a directory. The source of truth for the fleet state.                     |
+| revision    | The dev-fleet commit the installed package was published from (`npx devfleet --version`).                      |
 
 ## Rules for agents
 
@@ -75,25 +75,30 @@ On the manager (the Mac this project is checked out on):
 | Tailscale signed in                                      | `tailscale status --json` (look for `"BackendState":"Running"`) |
 | An SSH alias per worker that works without a prompt      | `ssh -o BatchMode=yes WORKER_ALIAS true` for each worker        |
 
-On every worker:
-
-| Requirement                          | Verify, from the manager                                           |
-| ------------------------------------ | ------------------------------------------------------------------ |
-| Ubuntu 26.04, amd64 or arm64         | `ssh WORKER_ALIAS '. /etc/os-release; echo $VERSION_ID; uname -m'` |
-| A non-root account that can use sudo | `ssh WORKER_ALIAS 'id -u; sudo -n true && echo sudo-ok'`           |
-| Joined to the same tailnet           | the worker appears in `tailscale status` on the manager            |
-
-A worker needs nothing else beforehand: no Git, curl, Python, Node, Bun or
-mise. `fleet-sync` installs them. If `sudo -n true` fails, that is fine for
-the first, interactive run, which asks for the password once and installs a
-sudo rule for the fleet; it is not fine for unattended runs.
-
-The `ssh` value of a worker in `fleet.json` is an alias from the manager's
-`~/.ssh/config`. dev-fleet always connects with `BatchMode=yes` and
-`StrictHostKeyChecking=yes`, so the host key must already be in
-`known_hosts`. **Ask** the human to verify each fingerprint out of band and
-connect once by hand; never add `StrictHostKeyChecking=no` or
-`accept-new`. A typical alias:
+On every worker, check each row of
+[what a machine must offer to be a worker](worker-requirements.md): the
+image, architecture and size, the access, the first-boot data and the
+network. Each row says how to verify it. If `sudo -n true` fails, that is fine
+for the first, interactive run, which asks for the password once and installs
+a sudo rule for the fleet; it is not fine for unattended runs.
+
+A worker is reached in one of two ways, always with `BatchMode=yes` and
+`StrictHostKeyChecking=yes`; never add `StrictHostKeyChecking=no` or
+`accept-new`.
+
+- **From declared data (no alias).** Leave `ssh` out: dev-fleet connects as
+  `network.user` on `network.sshPort` to `network.tailscaleName`, and checks
+  the host key against a known_hosts file of its own, rendered from the key
+  the fleet state pins. The human's `~/.ssh/config` and `~/.ssh/known_hosts`
+  are neither needed nor changed. The key is pinned without trusting a first
+  connection: `npx devfleet fleet-node-seed --id ID --output PATH --apply
+--yes` generates it for a new machine, `npx devfleet fleet-node-replace
+--id ID --host-key "TYPE BASE64" --apply --yes` pins one the human verified
+  out of band, and a worker record (below) may carry it as `hostKey`.
+- **Through the human's own alias.** `ssh` names a Host alias in the
+  manager's `~/.ssh/config`, and its host key must already be in their
+  `known_hosts`. **Ask** the human to verify each fingerprint out of band and
+  connect once by hand. A typical alias:
 
 ```
 Host worker-one
@@ -153,36 +158,37 @@ npx devfleet fleet-state --example > fleet.json
 Then replace every placeholder. Each field, from
 [`schema/fleet.schema.json`](../schema/fleet.schema.json):
 
-| Field                                                 | Value                                                                                                                |
-| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
-| `version`                                             | `3`                                                                                                                  |
-| `source.repository`                                   | `https://github.com/crvouga/dev-fleet.git`, or your fork if you publish one. Machines only accept code from it.      |
-| `state`                                               | `{ "bucket", "endpoint"?, "region"?, "prefix"? }` or `{ "directory": "/absolute/path" }`. Never credentials.         |
-| `secrets`                                             | See [secrets](#secrets).                                                                                             |
-| `manager.id`                                          | A short lowercase id, `^[a-z][a-z0-9-]{0,62}$`, e.g. `manager-mac`.                                                  |
-| `manager.os`                                          | `"macOS"`                                                                                                            |
-| `manager.network.tailscaleName`                       | The manager's MagicDNS name: `tailscale status --json` → `.Self.DNSName` without the trailing dot.                   |
-| `manager.network.user`                                | The manager's login name (`id -un`).                                                                                 |
-| `manager.neverSleepOnPower`                           | `true` (workers depend on the manager's services).                                                                   |
-| `workers[].id`                                        | A short lowercase id. The **first** worker is the canary: a new revision must pass there before the others get it.   |
-| `workers[].ssh`                                       | The worker's alias in the manager's `~/.ssh/config`.                                                                 |
-| `workers[].os`                                        | `"linux"`                                                                                                            |
-| `workers[].network.tailscaleName`                     | The worker's MagicDNS name (from `tailscale status --json` → `.Peer[].DNSName` on the manager).                      |
-| `workers[].network.sshPort`                           | `22` unless the worker's sshd listens elsewhere.                                                                     |
-| `workers[].network.user`                              | The worker account (`ssh WORKER_ALIAS id -un`).                                                                      |
-| `workers[].network.t3Port`                            | `3773`                                                                                                               |
-| `services.proxy.endpoint`                             | The public HTTPS origin of the CLI proxy, e.g. `https://cli-proxy.example.com`.                                      |
-| `services.monitoring.hubPort`                         | `8443`                                                                                                               |
-| `services.monitoring.publicUrl`                       | The public HTTPS origin of the monitoring dashboard.                                                                 |
-| `services.monitoring.managerAccount`                  | The email of the monitoring account the manager enrolls machines with.                                               |
-| `services.monitoring.viewerAccount`                   | The email of the read-only account visitors are signed in as.                                                        |
-| `ubuntu`                                              | `{ "release": "26.04", "architectures": ["amd64", "arm64"] }`: a release of the [support matrix](support-matrix.md). |
-| `worker.requiredCapabilities`                         | Normally all of `github`, `vault`, `t3`, `toolchain`, `workstation`, `worker`.                                       |
-| `worker.githubLogin`                                  | The GitHub login every worker must be signed in as.                                                                  |
-| `github` (optional)                                   | Who the manager signs in to GitHub as, per-node tokens, required scopes; see [GitHub](github.md).                    |
-| `worker.vaultUser`                                    | The Vault user each worker's token is issued for.                                                                    |
-| `worker.toolchain`, `agentProfile`                    | `"mise.lock"`, `"proxy"` (fixed).                                                                                    |
-| `worker.unattendedSudo`, `persistentT3`, `neverSleep` | `true` (fixed policy).                                                                                               |
+| Field                                                 | Value                                                                                                                    |
+| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
+| `version`                                             | `3`                                                                                                                      |
+| `source.repository`                                   | `https://github.com/crvouga/dev-fleet.git`, or your fork if you publish one. Machines only accept code from it.          |
+| `state`                                               | `{ "bucket", "endpoint"?, "region"?, "prefix"? }` or `{ "directory": "/absolute/path" }`. Never credentials.             |
+| `secrets`                                             | See [secrets](#secrets).                                                                                                 |
+| `manager.id`                                          | A short lowercase id, `^[a-z][a-z0-9-]{0,62}$`, e.g. `manager-mac`.                                                      |
+| `manager.os`                                          | `"macOS"`                                                                                                                |
+| `manager.network.tailscaleName`                       | The manager's MagicDNS name: `tailscale status --json` → `.Self.DNSName` without the trailing dot.                       |
+| `manager.network.user`                                | The manager's login name (`id -un`).                                                                                     |
+| `manager.neverSleepOnPower`                           | `true` (workers depend on the manager's services).                                                                       |
+| `workers[].id`                                        | A short lowercase id. The first reachable worker is the canary: a new revision must pass there before the others get it. |
+| `canary`                                              | Optional: the id of the worker to prefer as canary. When it is unreachable, the next reachable worker is used.           |
+| `workers[].ssh`                                       | The worker's alias in the manager's `~/.ssh/config`.                                                                     |
+| `workers[].os`                                        | `"linux"`                                                                                                                |
+| `workers[].network.tailscaleName`                     | The worker's MagicDNS name (from `tailscale status --json` → `.Peer[].DNSName` on the manager).                          |
+| `workers[].network.sshPort`                           | `22` unless the worker's sshd listens elsewhere.                                                                         |
+| `workers[].network.user`                              | The worker account (`ssh WORKER_ALIAS id -un`).                                                                          |
+| `workers[].network.t3Port`                            | `3773`                                                                                                                   |
+| `services.proxy.endpoint`                             | The public HTTPS origin of the CLI proxy, e.g. `https://cli-proxy.example.com`.                                          |
+| `services.monitoring.hubPort`                         | `8443`                                                                                                                   |
+| `services.monitoring.publicUrl`                       | The public HTTPS origin of the monitoring dashboard.                                                                     |
+| `services.monitoring.managerAccount`                  | The email of the monitoring account the manager enrolls machines with.                                                   |
+| `services.monitoring.viewerAccount`                   | The email of the read-only account visitors are signed in as.                                                            |
+| `ubuntu`                                              | `{ "release": "26.04", "architectures": ["amd64", "arm64"] }`: a release of the [support matrix](support-matrix.md).     |
+| `worker.requiredCapabilities`                         | Normally all of `github`, `vault`, `t3`, `toolchain`, `workstation`, `worker`.                                           |
+| `worker.githubLogin`                                  | The GitHub login every worker must be signed in as.                                                                      |
+| `github` (optional)                                   | Who the manager signs in to GitHub as, per-node tokens, required scopes; see [GitHub](github.md).                        |
+| `worker.vaultUser`                                    | The Vault user each worker's token is issued for.                                                                        |
+| `worker.toolchain`, `agentProfile`                    | `"mise.lock"`, `"proxy"` (fixed).                                                                                        |
+| `worker.unattendedSudo`, `persistentT3`, `neverSleep` | `true` (fixed policy).                                                                                                   |
 
 Use the same values in every place they repeat. Validate before going on:
 
@@ -301,6 +307,14 @@ An agent cannot answer those prompts, and `--json`, `--non-interactive` or a
 npx devfleet fleet-sync
 ```
 
+A worker that needs none of those prompts is enrolled without the human: a
+run with `--json` or `--non-interactive` enrolls a declared worker when its
+pinned host key matched, `sudo -n` works on it, and exactly one tailnet device
+matches it (or none, when `fleet.json` declares `tailnet` and its secret
+`tailnet-admin`, so the run joins it with a single-use key). The report lists
+what it enrolled (`enrolled` in the JSON). A machine made with
+`devfleet fleet-node-seed` meets all of these.
+
 It converges the manager first, then the canary worker, then the rest. It
 prints one table and the path of the full JSON report
 (`~/.local/share/personal/fleet/manager/last-sync.json`, readable by the human
@@ -363,7 +377,7 @@ Read `summary` and `remedy.action` of the failing check. The common ones:
 | `ROOT_DEVELOPMENT_USER_REJECTED`          | The worker's SSH user is root. Use a normal account with sudo.                                                                |
 | `BOOTSTRAP_DISK_CAPACITY_REQUIRED`        | Less than 2 GiB free (or too few inodes) where the fleet keeps its files on the worker.                                       |
 | `FLEET_REMOTE_LOCKED`                     | Another operation is running on that worker. Wait and run again.                                                              |
-| `FLEET_PROMOTION_BLOCKED`                 | The canary (first worker) failed, so the new revision went no further. Fix the canary first.                                  |
+| `FLEET_PROMOTION_BLOCKED`                 | The canary (the report names it under `canary`) failed, so the new revision went no further. Fix the canary first.            |
 | `VAULT_LOGIN_REQUIRED`                    | The manager is not logged in to Vault. The human logs in; the next sync issues worker tokens.                                 |
 | `PROVIDER_UNATTENDED_CREDENTIAL_REQUIRED` | A coding agent has no credential a background service can use. The human signs in on the proxy.                               |
 | `BESZEL_ACCOUNT_REQUIRED`                 | The monitoring hub has no account yet. Follow the remedy; other workers are not blocked.                                      |
@@ -381,7 +395,9 @@ and codes; leave out everything private.
 - **Remove a worker.** Delete its entry, commit, run `npx devfleet fleet-sync`.
 - **Upgrade the fleet.** `npm install --save-dev --save-exact @crvouga/dev-fleet@latest`,
   commit, `npx devfleet fleet-sync`. The canary proves the new revision first.
-  Install an older version and sync to roll back.
+  `npx devfleet fleet-sync --rollback` goes back to the revision applied
+  before (the revision, tools and T3; system packages roll forward only, as
+  the [README](../README.md#the-version) says).
 - **Replace the manager.** Follow
   [replacing the manager](manager-recovery.md).
 

package/docs/convergence.md

@@ -0,0 +1,67 @@
+# What converges a machine, and what is not Ansible yet
+
+The goal is one convergence engine: Ansible, on every platform, with the CLI
+as an orchestrator that validates the fleet configuration, runs playbooks,
+reads their results and reports checks. Today three mechanisms change a
+machine: the playbooks under `ansible/`, the bootstrap scripts, and some
+TypeScript. This page lists every step that is not an Ansible task yet, each
+with the reason it is not, so the list can only shrink.
+`features/rules/convergence.feature` fails when a file that changes a machine
+is missing from it.
+
+## Ansible
+
+| Playbook                                  | Converges                                                                     |
+| ----------------------------------------- | ----------------------------------------------------------------------------- |
+| `ansible/ubuntu/baseline.yml`             | A worker's packages, Docker, services, power, user, SSH, disk                 |
+| `ansible/ubuntu/tailscale.yml`            | Tailscale on a worker                                                         |
+| `ansible/ubuntu/t3-desktop.yml`           | The T3 desktop app on a desktop worker                                        |
+| `ansible/ubuntu/t3-headless.yml`          | The T3 server as a lingering user service on a headless worker                |
+| `ansible/ubuntu/t3-policy.yml`            | T3's administrator policy                                                     |
+| `ansible/ubuntu/t3-route.yml`             | T3's private tailnet route                                                    |
+| `ansible/ubuntu/recovery.yml`             | Recovery of a stopped T3 desktop app or server                                |
+| `ansible/ubuntu/worker-account.yml`       | The declared account on an image that lets only root in, from the manager     |
+| `ansible/ubuntu/reboot.yml`               | One worker's first-converge or maintenance reboot, from the manager           |
+| `ansible/monitoring/agent.yml`            | The Beszel agent on a worker                                                  |
+| `ansible/monitoring/agent-darwin.yml`     | The Beszel agent on the manager                                               |
+| `ansible/monitoring/hub-darwin.yml`       | The Beszel hub and its tunnel on the manager                                  |
+| `ansible/monitoring/hub-route-darwin.yml` | The hub's private tailnet route                                               |
+| `ansible/macos/manager.yml`               | The manager's power settings, network time and fleet-watch job, unattended    |
+| `ansible/macos/t3-policy.yml`             | T3's administrator policy on the manager                                      |
+| `ansible/macos/manager-privilege.yml`     | The one administrator grant: sudoers, job log rotation and the T3 policy      |
+| `ansible/macos/worker-host-key.yml`       | A worker's pinned SSH host key in the manager's known_hosts                   |
+| `ansible/macos/vault-renew.yml`           | The launch agent that renews the manager's own Vault session                  |
+| `ansible/github/git.yml`                  | The Git identity and GitHub credentials of every node                         |
+| `ansible/agents/agents.yml`               | Each agent CLI's dedicated profile, rendered from `ansible/agents/templates/` |
+
+Every playbook passes `ansible-lint` in the production profile, every command
+in it says when it changes something and runs, or is skipped, in a dry run
+(`features/rules/playbooks.feature`). The Beszel agent playbook is proven on a
+systemd machine to report no change on a second run and in `--check`, and to
+report drift, and only that drift (`features/monitoring/agent-lifecycle.feature`).
+
+## Not Ansible yet
+
+| Step                                                                                    | Where                                                                                       | Why it is not a task yet                                                                                                    |
+| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
+| Install Git, curl, CA certificates, Python and Ansible                                  | `bootstrap/fleet-remote-bootstrap.sh`                                                       | It must run before Ansible exists on the machine; it is the minimal bootstrap                                               |
+| Create the worker account, its host key, the privilege rule and Tailscale on first boot | `bootstrap/worker-seed.yaml` (written by `fleet-node-seed`)                                 | cloud-init runs it before the manager can reach the machine; fleet-sync converges everything else with Ansible afterwards   |
+| Install mise and the pinned toolchain                                                   | `bootstrap/bootstrap.sh`, `bootstrap/setup.ts`                                              | mise must exist before any tool does, and `mise install --locked` already converges from a lock                             |
+| Install a worker's subset of tools                                                      | `bootstrap/worker-tools.ts`, `bootstrap/worker-features.ts`                                 | It chooses which locked tools `mise install` gets; moving it means a toolchain role                                         |
+| Publish the global mise configuration                                                   | `bootstrap/global-config.ts`                                                                | Derived from `mise.toml` and the agent worktree roots on every run; a `toolchain` role with a template would replace it     |
+| `PATH` lines in shell profiles                                                          | `bootstrap/shell-path.ts`                                                                   | Edits files a person owns, guarded by markers; a `blockinfile` task would replace it                                        |
+| The `devfleet` launcher link                                                            | `bootstrap/install-launcher.sh`                                                             | Points at the fleet-owned checkout of the revision, known only during the run                                               |
+| The T3 server archive                                                                   | `bootstrap/native-t3.ts`                                                                    | Installed with the toolchain, checked against the digest in `mise.toml`; a `t3` role would replace it                       |
+| The AWS CLI on Windows                                                                  | `bootstrap/native-aws.ts`                                                                   | Windows is not a fleet role                                                                                                 |
+| Docker Desktop on the manager                                                           | `bootstrap/docker-desktop.sh`                                                               | A person installs and approves it on macOS; dev-fleet only measures it                                                      |
+| A worker's service credentials from Vault                                               | `bootstrap/worker-github.ts`, `bootstrap/worker-auth.ts`                                    | A login belongs to a machine and a person; it is read from a session, never declared                                        |
+| The manager's own convergence: proxy, profiles, hub, agent                              | `src/cli/fleet-sync-manager.ts`                                                             | Orchestrates the manager playbooks and the proxy's own installer; the proxy is not a playbook yet (`docs/declared.md`)      |
+| Scheduled runs                                                                          | `src/cli/fleet-sync-schedule.ts`, `src/platform/launchd.ts`, `src/platform/systemd-user.ts` | Writes a launchd job or systemd timer from TypeScript; listed in `docs/declared.md` as a schedule that should be a playbook |
+
+## Still to do
+
+Roles with a common interface (`baseline`, `toolchain`, `agents`, `t3`,
+`network`, `monitoring`, `recovery`, `manager`), an inventory generated from
+the fleet configuration, and `site.yml`, `manager.yml` and `workers.yml` that
+bring a fresh manager and worker to ready with no other mutating step. Each
+row of the table above moves into one of those roles.

package/docs/declared.md

@@ -12,29 +12,33 @@ may only shrink.
 
 ## Declared
 
-| What                                          | Declared in                                                                                         | Pinned by                                                            | Applied by                                                | Measured by                                  |
-| --------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | --------------------------------------------------------- | -------------------------------------------- |
-| The fleet: machines, services, accounts       | Your `fleet.json`                                                                                   | The dev-fleet version in your `package.json`, which names one commit | `fleet-sync`                                              | `fleet-status`                               |
-| Tools: languages, CLIs, coding agents         | `mise.toml`                                                                                         | `mise.lock`: version and checksum for each platform                  | `mise install --locked`                                   | Tool and `agent:verification` checks         |
-| mise itself                                   | `min_version` and `mise_*_sha256` in `mise.toml`                                                    | The digest of its release archive, held here                         | `bootstrap/bootstrap.sh`                                  | Its version, at every toolchain setup        |
-| JavaScript dependencies                       | `package.json`                                                                                      | `bun.lock`                                                           | `bun install --frozen-lockfile`                           | The install fails on a difference            |
-| T3 Code, server and desktop app               | `t3_*` under `[vars]` in `mise.toml`                                                                | SHA-256 of each artifact                                             | `bootstrap/native-t3.ts`; `ansible/ubuntu/t3-desktop.yml` | T3 checks; `t3:version` on every machine     |
-| T3 policy on the manager (updates off)        | `ansible/ubuntu/t3/policy.json`, shared with the workers                                            | The commit                                                           | `ansible/macos/t3-policy.yml`, once, as an administrator  | `t3:updates`                                 |
-| Worker work-in-progress guard                 | `ansible/ubuntu/wip-guard.py`, `ansible/ubuntu/tasks/recovery.yml`                                  | The commit                                                           | `fleet-sync`                                              | `features/ide/work-in-progress.feature`      |
-| Ubuntu packages                               | `packages` and `snapshot` in `ansible/ubuntu/policy.json`                                           | Exact versions from a dated archive snapshot                         | `ansible/ubuntu/tasks/packages.yml`                       | `package:*`                                  |
-| Tailscale on workers                          | `tailscale` in `ansible/ubuntu/policy.json`                                                         | Exact version; digest of the signing key                             | `ansible/ubuntu/tailscale.yml`                            | `package:tailscale`                          |
-| Worker policy: power, SSH, sudo, memory, disk | `ansible/ubuntu/*.yml`, values shared with checks in `policy.json`                                  | The commit                                                           | `fleet-sync`                                              | Host and resource checks                     |
-| Monitoring: hub, tunnel, agents               | `ansible/monitoring/*.yml`, `compose.yml`, `lock.json`                                              | Image digests; SHA-256 of each agent archive                         | `fleet-sync`                                              | `beszel:*`                                   |
-| The manager's private route to the hub        | `ansible/monitoring/hub-route-darwin.yml`                                                           | The commit                                                           | `fleet-sync`                                              | The route, read back                         |
-| The manager's power settings                  | `pmset` in `ansible/macos/policy.json`                                                              | The commit                                                           | `ansible/macos/manager.yml`, once, as an administrator    | `manager:power`                              |
-| Ansible on the manager                        | `ansible/pyproject.toml`                                                                            | `ansible/uv.lock`: one release, one Python, every dependency hashed  | `uv run --locked --isolated`                              | A hash that differs stops the run            |
-| Ansible on a worker                           | `ansible-core` in `ansible/ubuntu/policy.json`                                                      | Exact version                                                        | `ansible/ubuntu/tasks/packages.yml`                       | `package:ansible-core`                       |
-| The CLI proxy                                 | `cliproxy/compose.yaml`, `image.lock.json`, `requirements.lock`, `launchd/`, `config.template.yaml` | Image digests; hashed Python dependencies                            | `devfleet cliproxy install`, and `fleet-sync` after it    | The proxy's audit                            |
-| Agent profiles and T3 providers               | `agents/registry.json`, `agents/t3.json`                                                            | The commit                                                           | `fleet-sync`                                              | Profile receipts; the plan after every apply |
-| Worker time service                           | `ansible/ubuntu/tasks/services.yml`, offsets in `clock` in `ansible/ubuntu/policy.json`             | The commit                                                           | `fleet-sync`                                              | `worker:clock`                               |
-| The manager's container watchdog              | `ansible/monitoring/container-watchdog.py`, `containerWatchdog` in `ansible/macos/policy.json`      | The commit                                                           | `fleet-sync` (`hub-darwin.yml`)                           | `manager:container-watchdog`                 |
-| Docker Desktop on the manager                 | `docker_desktop_*` under `[vars]` in `mise.toml`                                                    | Version and build                                                    | You                                                       | `manager:docker` (warns)                     |
-| CI                                            | `.github/workflows/`                                                                                | Each action at a commit; tools from `mise.lock`                      | GitHub                                                    | `features/rules/`                            |
+| What                                                           | Declared in                                                                                         | Pinned by                                                            | Applied by                                                                                    | Measured by                                                     |
+| -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
+| The fleet: machines, services, accounts                        | Your `fleet.json`                                                                                   | The dev-fleet version in your `package.json`, which names one commit | `fleet-sync`                                                                                  | `fleet-status`                                                  |
+| Tools: languages, CLIs, coding agents                          | `mise.toml`                                                                                         | `mise.lock`: version and checksum for each platform                  | `mise install --locked`                                                                       | Tool and `agent:verification` checks                            |
+| mise itself                                                    | `min_version` and `mise_*_sha256` in `mise.toml`                                                    | The digest of its release archive, held here                         | `bootstrap/bootstrap.sh`                                                                      | Its version, at every toolchain setup                           |
+| JavaScript dependencies                                        | `package.json`                                                                                      | `bun.lock`                                                           | `bun install --frozen-lockfile`                                                               | The install fails on a difference                               |
+| T3 Code, server and desktop app                                | `t3_*` under `[vars]` in `mise.toml`                                                                | SHA-256 of each artifact                                             | `bootstrap/native-t3.ts`; `ansible/ubuntu/t3-desktop.yml`                                     | T3 checks; `t3:version` on every machine                        |
+| T3 policy on the manager (updates off, linked environments on) | `ansible/ubuntu/t3/policy.json`, shared with the workers                                            | The commit                                                           | `ansible/macos/t3-policy.yml`, once, as an administrator (`manager-privilege.yml`)            | `t3:updates`                                                    |
+| Worker work-in-progress guard                                  | `ansible/ubuntu/wip-guard.py`, `ansible/ubuntu/tasks/recovery.yml`                                  | The commit                                                           | `fleet-sync`                                                                                  | `features/ide/work-in-progress.feature`                         |
+| Ubuntu packages                                                | `packages` and `snapshot` in `ansible/ubuntu/policy.json`                                           | Exact versions from a dated archive snapshot                         | `ansible/ubuntu/tasks/packages.yml`                                                           | `package:*`                                                     |
+| Tailscale on workers                                           | `tailscale` in `ansible/ubuntu/policy.json`                                                         | Exact version; digest of the signing key                             | `ansible/ubuntu/tailscale.yml`                                                                | `package:tailscale`                                             |
+| Worker policy: power, SSH, sudo, memory, disk                  | `ansible/ubuntu/*.yml`, values shared with checks in `policy.json`                                  | The commit                                                           | `fleet-sync`                                                                                  | Host and resource checks                                        |
+| Monitoring: hub, tunnel, agents                                | `ansible/monitoring/*.yml`, `compose.yml`, `lock.json`                                              | Image digests; SHA-256 of each agent archive                         | `fleet-sync`                                                                                  | `beszel:*`                                                      |
+| The manager's private route to the hub                         | `ansible/monitoring/hub-route-darwin.yml`                                                           | The commit                                                           | `fleet-sync`                                                                                  | The route, read back                                            |
+| The manager's power settings                                   | `pmset` in `ansible/macos/policy.json`                                                              | The commit                                                           | `ansible/macos/manager.yml`, on every `fleet-sync`, after one grant (`manager-privilege.yml`) | `manager:power`                                                 |
+| Ansible on the manager                                         | `ansible/pyproject.toml`                                                                            | `ansible/uv.lock`: one release, one Python, every dependency hashed  | `uv run --locked --isolated`                                                                  | A hash that differs stops the run                               |
+| Ansible on a worker                                            | `ansible-core` in `ansible/ubuntu/policy.json`                                                      | Exact version                                                        | `ansible/ubuntu/tasks/packages.yml`                                                           | `package:ansible-core`                                          |
+| The CLI proxy                                                  | `cliproxy/compose.yaml`, `image.lock.json`, `requirements.lock`, `launchd/`, `config.template.yaml` | Image digests; hashed Python dependencies                            | `devfleet cliproxy install`, and `fleet-sync` after it                                        | The proxy's audit                                               |
+| Agent profiles and T3 providers                                | `agents/registry.json`, `agents/t3.json`; each CLI's configuration from `ansible/agents/templates/` | The commit                                                           | `fleet-sync` (agent profiles through `ansible/agents/agents.yml`)                             | Profile receipts; the plan after every apply; `agent:*:session` |
+| Worker time service                                            | `ansible/ubuntu/tasks/services.yml`, offsets in `clock` in `ansible/ubuntu/policy.json`             | The commit                                                           | `fleet-sync`                                                                                  | `worker:clock`                                                  |
+| The manager's playbook and its receipt                         | `ansible/macos/manager.yml`                                                                         | The commit                                                           | `fleet-sync`, check mode first, then applied; re-applied at least daily                       | `manager:playbook`                                              |
+| The manager's network time                                     | `ansible/macos/manager.yml` (an assert)                                                             | The commit                                                           | macOS                                                                                         | `manager:clock`, `worker:clock-offset`                          |
+| fleet-watch on a schedule                                      | `fleetWatch` in `ansible/macos/policy.json`, `manager.schedule.watchMinutes` in `fleet.json`        | The commit                                                           | `fleet-sync` (`manager.yml`)                                                                  | `manager:watch`                                                 |
+| Rotation of the scheduled jobs' logs                           | `jobLogs` in `ansible/macos/policy.json`                                                            | The commit                                                           | `ansible/macos/manager-privilege.yml`, once, as an administrator                              | `manager:job-logs`                                              |
+| The manager's container watchdog                               | `ansible/monitoring/container-watchdog.py`, `containerWatchdog` in `ansible/macos/policy.json`      | The commit                                                           | `fleet-sync` (`hub-darwin.yml`)                                                               | `manager:container-watchdog`                                    |
+| Docker Desktop on the manager                                  | `docker_desktop_*` under `[vars]` in `mise.toml`                                                    | Version and build                                                    | You                                                                                           | `manager:docker` (warns)                                        |
+| CI                                                             | `.github/workflows/`                                                                                | Each action at a commit; tools from `mise.lock`                      | GitHub                                                                                        | `features/rules/`                                               |
 
 A value lives in a policy file when something besides the playbook reads it,
 so the check and the playbook cannot disagree. A value only a playbook uses
@@ -62,11 +66,11 @@ stays in the playbook: the playbook is its declaration.
   scripts from the declarative files above, not by a playbook.
 - **Files TypeScript and shell still write.** The installed `fleet.json`, the
   `current` and `devfleet` links, the `PATH` lines in shell profiles, the
-  global mise configuration, the T3 server archive, agent profiles and T3
-  settings. Each is derived
+  global mise configuration, the T3 server archive and T3 settings. Each is derived
   from a tracked file and rewritten on every run, but by code.
 - **Logins.** Tailscale, Vault, GitHub, each provider account and T3 pairing
-  belong to a machine and a person. They are checked, never copied.
+  (except a replaced worker's, which the manager pairs itself) belong to a
+  machine and a person. They are checked, never copied.
 - **Windows.** `bootstrap/bootstrap.ps1` exists; Windows is not a fleet role
   and nothing tests it.
 
@@ -80,13 +84,36 @@ moving to it entirely means NixOS workers and nix-darwin on the manager, which
 replaces every unit and path that exists today. The one thing above that Nix
 would settle and the current tools do not is the rest of a worker's packages.
 
+## Keeping pins current
+
+A pin that is only moved by hand goes stale and is then moved in a large
+batch, the riskiest way to change a fleet. Each pin is moved by a machine,
+one pull request at a time, which CI, the fresh workers and the VM test then
+judge like any change, unless it says below why a person moves it.
+
+| Pin                                      | Moved by                                                                                      |
+| ---------------------------------------- | --------------------------------------------------------------------------------------------- |
+| Tools in `mise.toml` and `mise.lock`     | Renovate (`mise`), one pull request per tool                                                  |
+| JavaScript and Python dependencies       | Renovate, with lock file maintenance                                                          |
+| Actions and CI service images            | Renovate, each action at a commit                                                             |
+| mise itself                              | The [Pins workflow](../.github/workflows/pins.yml): `bun bootstrap/pins.ts mise`, weekly      |
+| T3 Code (nightly)                        | The Pins workflow: `bun bootstrap/pins.ts t3`, weekly, every artifact hashed                  |
+| The tunnel image of the monitoring hub   | Renovate, by tag and digest (a custom manager in `renovate.json`)                             |
+| Beszel (hub image and agents)            | A person, because the agents' binary digests need the archives unpacked; `pins.ts` cannot yet |
+| Docker Desktop on the manager            | A person, because a person installs it and dev-fleet only measures the version                |
+| The Ubuntu snapshot and package versions | A person, because a new snapshot moves every package at once and each needs its exact version |
+
+The workflows never name the mise release: they install it through
+[`.github/actions/mise`](../.github/actions/mise/action.yml), which reads
+`min_version` from `mise.toml`.
+
 ## Changing a pin
 
-| To change               | Edit                                                                 | Then                                             |
-| ----------------------- | -------------------------------------------------------------------- | ------------------------------------------------ |
-| A tool                  | Its version in `mise.toml`                                           | Update `mise.lock` with `mise lock`              |
-| mise                    | `min_version` and the four `mise_*_sha256` in `mise.toml`            | Copy the digests from the release's `SHASUMS256` |
-| An Ubuntu package       | `packages` in `ansible/ubuntu/policy.json`, and `snapshot` if needed | The VM test in `ansible/ubuntu/tests`            |
-| Ansible on the manager  | `ansible/pyproject.toml`                                             | `uv lock` in `ansible/`                          |
-| Beszel                  | `ansible/monitoring/lock.json`, `compose.yml`, the agent unit        | `bun test features/rules`                        |
-| A manager power setting | `pmset` in `ansible/macos/policy.json`                               | Run `manager.yml` once on the manager            |
+| To change               | Edit                                                                 | Then                                  |
+| ----------------------- | -------------------------------------------------------------------- | ------------------------------------- |
+| A tool                  | Its version in `mise.toml`                                           | Update `mise.lock` with `mise lock`   |
+| mise                    | `bun bootstrap/pins.ts mise` moves `min_version` and the digests     | The fresh-box run                     |
+| An Ubuntu package       | `packages` in `ansible/ubuntu/policy.json`, and `snapshot` if needed | The VM test in `ansible/ubuntu/tests` |
+| Ansible on the manager  | `ansible/pyproject.toml`                                             | `uv lock` in `ansible/`               |
+| Beszel                  | `ansible/monitoring/lock.json`, `compose.yml`, the agent unit        | `bun test features/rules`             |
+| A manager power setting | `pmset` in `ansible/macos/policy.json`                               | The next `fleet-sync` applies it      |