@crvouga/dev-fleet 0.7.0 → 0.9.0

package/docs/machine-setup.md

@@ -1,6 +1,7 @@
 > This document was written for the fleet dev-fleet was extracted from. Paths and
 > commands were updated; some statements still describe that fleet and are being
-> revised. The README is authoritative.
+> revised. The README is authoritative, and what a machine must offer to be a
+> worker is listed only in [worker requirements](worker-requirements.md).
 
 ## Machine setup: macOS, Windows and Linux
 
@@ -184,7 +185,13 @@ railway login
 Only sign into providers the task actually needs. On fleet machines the Git
 identity and the GitHub credential helper come from the `git` block of
 `fleet.json`: `fleet-sync` writes them in `~/.config/git/config` and checks
-them as `worker:git-identity` and `worker:git-push-auth`. Elsewhere set `git
+them as `worker:git-identity` and `worker:git-push-auth`. With
+`"signing": "ssh"` in that block every node also makes an SSH key of its own
+(`~/.ssh/dev-fleet-git-signing`, never copied off the node) and signs every
+commit and tag with it, checked as `worker:git-signing`; GitHub shows those
+commits as verified once you add each node's `.pub` to the account as a
+signing key (`gh ssh-key add ~/.ssh/dev-fleet-git-signing.pub --type signing`
+on the node). Elsewhere set `git
 config --global user.name` and `user.email` after confirming them; do not guess
 an email or overwrite an existing identity. Enable Git LFS hooks when needed with `git lfs install`.
 
@@ -357,9 +364,44 @@ headless service, so environment identity and pairing are kept. A conflicting
 launcher version requires reviewed maintenance; the recovery timer stops a
 still-running headless service. Routine fleet-sync never ends a running turn
 to load a new pin or policy: the app restarts into them at the first idle
-check (`t3:running-version`, `t3:policy-loaded`). The disposable Ubuntu VM test still exercises the former headless service
-and does not cover the desktop session. Private routing and paired execution
-require separate evidence.
+check (`t3:running-version`, `t3:policy-loaded`). Private routing and paired
+execution require separate evidence.
+
+#### Headless workers
+
+A worker whose session is `headless` (`features.session` in `fleet.json`, or
+the `editing` preset) has no display manager, no graphical session and no
+GPU. It runs the pinned T3 **server**, not the desktop app under a virtual
+display: the pinned release ships the server as its own archive
+(`t3-<version>-linux-<arch>.tar.gz`, the `t3_version` and
+`t3_linux_*_sha256` pins, from the release's `SHA256SUMS`), which the
+toolchain already installs on every worker for policy commands. Its
+`t3 serve` runs the server the desktop app embeds, with no browser; the
+subcommand and its flags were read from the pinned release itself
+(`t3 serve --help` of `0.1.8-nightly.20261106.4262`, whose `--mode web`
+server answered `/.well-known/t3/environment` with that `serverVersion`).
+A virtual display would install and pin a display server and a desktop
+toolkit for one process, which costs the memory a small worker needs.
+
+`ansible/ubuntu/t3-headless.yml` declares `t3code-headless.service`, a user
+unit started by `default.target`: the designated user lingers, so it starts at
+boot with nobody logged in. It listens on `127.0.0.1:3773` like the desktop
+app, with `T3CODE_HOME=%h/.t3`, so the private route, the policy, the provider
+instances, `personal-worker-recovery` (which reads the release from the
+unit's versioned server path) and the checks are the same for both session
+types. Switching a worker between types stops and removes the other type's
+unit and keeps `~/.t3`; GDM auto-login, if the machine had it, is left as it
+was. A headless worker installs none of the desktop-only packages
+(`features.desktop` in `ansible/ubuntu/policy.json`) and reports their checks
+as not applicable. The server prints a one-time pairing token to its own
+journal at start, as `t3 serve` does anywhere; dev-fleet never reads it.
+Pairing is done from the manager: by a person the first time, and by
+`fleet-node-replace` or the next `fleet-sync` for a machine that replaces a
+worker.
+
+The disposable Ubuntu VM test (`ansible/ubuntu/tests/vm.py`) converges each
+session type, reboots with nobody logged in and kills the T3 process to
+prove the recovery timer brings it back.
 
 Normal Ubuntu fleet setup is controller-led:
 `devfleet fleet-sync`. See the
@@ -396,7 +438,10 @@ Controller-driven remote bootstrap passes `--worker` to the shared setup owner.
 It installs the `vars.worker_tools` subset from `mise.toml`, with versions and
 artifacts taken from the same `mise.lock`; native T3 and Ubuntu Docker/service
 policy remain source-pinned. `bun bootstrap/doctor.ts --worker --docker --json`
-checks that profile. It does not require unrelated cloud CLIs on workers.
+checks that profile. It does not require unrelated cloud CLIs on workers. A
+worker whose `containers` feature is off (`features.containers: false`, or the
+`editing` preset) leaves `docker-cli` out of its base tools, and its doctor
+runs with `--no-containers` instead of `--docker`.
 
 Tool profiles add locked tool sets on top of it: `worker.toolProfiles` in
 `fleet.json` (fleet-wide) or `toolProfiles` on one worker selects `web` (pnpm,
@@ -404,7 +449,28 @@ Deno, Yarn and headless Chromium for browser tests), `go`, `rust` (with CMake
 and Ninja) or `jvm` (Temurin), each a `worker_profile_<name>` under `[vars]` in
 `mise.toml`. `worker.tools` adds single tools dev-fleet already pins. The setup
 installs them (`--node` reads the installed configuration) and the doctor
-checks each as `tool:<name>` (`--profiles`, `--tools`).
+checks each as `tool:<name>` (`--profiles`, `--tools`). The base tools include
+`tmux`, for sessions that outlive an SSH connection.
+
+A tool dev-fleet does not ship belongs to the project that needs it: pin it in
+the project's own `mise.toml` and lock it with `mise lock`. On every fleet
+machine dev-fleet's global configuration sets `locked = true` for the
+`project` scope, so mise installs a project's tools in a checkout only from
+that project's `mise.lock` and refuses a tool the lock does not name;
+`mise install --locked` in the checkout installs and verifies them.
+
+There is no `mobile` profile. No mobile toolchain can be pinned with
+checksums for both worker architectures through mise: Flutter publishes no
+Linux arm64 release, the Android SDK's platforms and build tools are fetched
+by `sdkmanager` at run time (outside any lock, behind a licence a person
+accepts), and iOS builds need macOS and Xcode, which workers are not. A
+project that builds for Android on x86_64 workers pins `flutter`, `java` and
+`android-sdk` in its own `mise.toml` as above.
+
+`worker.exposure` (fleet-wide) or `exposure` on one worker says what a worker
+opens beyond loopback and the tailnet; without it, nothing. See
+[Exposure](ubuntu-sync.md#exposure).
+
 The source-owned `worker-auth.ts` timer renews the worker's existing Vault token
 and can restore the declared GitHub account from its authorized Vault service
 credential, without copying another machine's login directory.

package/docs/manager-setup.md

@@ -0,0 +1,34 @@
+# What only a person does on the manager
+
+Everything on the manager is declared and converged by `fleet-sync`: the
+toolchain, the power policy (applied by `ansible/macos/manager.yml` on every
+run), the CLI proxy, the monitoring hub and their jobs. What remains are the
+steps that need a person: an identity, an approval or an administrator
+password. Each is done once, and `fleet-sync` measures it afterwards with the
+check named beside it.
+
+The list is tested not to grow (`features/rules/manager-setup.feature`): a
+step added here must be added there too, with the check that verifies it.
+
+<!-- human-steps -->
+
+| Step                                                                                                       | Why a person                          | Verified by           |
+| ---------------------------------------------------------------------------------------------------------- | ------------------------------------- | --------------------- |
+| Install Docker Desktop at the release `mise.toml` pins and sign in to it                                   | A licence and an account              | `manager:docker`      |
+| Sign this Mac in to Tailscale and approve it                                                               | A tailnet account and device approval | `tailscale:discovery` |
+| Pass the bucket credentials (`DEVFLEET_BUCKET_*`) or save them with `fleet-state --init`                   | Credentials of an account you own     | `state:bucket`        |
+| Log the manager in to Vault                                                                                | A password or an identity provider    | `auth:vault`          |
+| Grant the power settings once: `ansible-playbook -e user=$(id -un) -K ansible/macos/manager-privilege.yml` | An administrator password             | `manager:power`       |
+
+<!-- /human-steps -->
+
+The grant lets the manager's user run exactly `pmset -c` and
+`systemsetup -getusingnetworktime` without a password, so later syncs never
+prompt. The same playbook, run as an administrator, also installs what only
+root may write: the newsyslog drop-in that keeps the logs of the scheduled
+jobs under a fixed size (`manager:job-logs`) and T3's administrator policy
+(`t3:updates`). Every `fleet-sync` then applies `ansible/macos/manager.yml`
+unattended: in check mode first, then for real, with a receipt like the
+workers' and a full re-apply at least once a day (`manager:playbook`). Run the
+grant again when `manager:clock` or `manager:job-logs` asks for it after an
+upgrade, or with `-e grant=absent` to remove it.

package/docs/operations.md

@@ -1,6 +1,7 @@
 > This document was written for the fleet dev-fleet was extracted from. Paths and
 > commands were updated; some statements still describe that fleet and are being
-> revised. The README is authoritative.
+> revised. The README is authoritative, and what a machine must offer to be a
+> worker is listed only in [worker requirements](worker-requirements.md).
 
 # Developer fleet
 
@@ -22,7 +23,7 @@ Use `devfleet fleet-sync` to reconcile all declared workers and
 `devfleet fleet-status --workers` to inspect remaining gaps. See
 [worker-contract.md](worker-contract.md) for bootstrap, ownership and the exact
 scope of readiness. `fleet-status` without flags diagnoses the local roster.
-Workers are Ubuntu 26.04; the manager is macOS and never sleeps on AC power
+Workers are Ubuntu 26.04 or 24.04; the manager is macOS and never sleeps on AC power
 (`ansible/macos/manager.yml`). Machine policy (never sleep, no automatic
 logout, always-on Tailscale/SSH/Docker/Beszel, memory and disk safety, per-node
 credential renewal) is
@@ -134,6 +135,26 @@ expiring: tag them, or disable key expiry for each in the Tailscale admin
 console (tagged nodes do not expire). `tailscale:key-expiry` warns two weeks
 ahead and fails three days ahead.
 
+With `tailnet` in `fleet.json` and the secret `tailnet-admin`, nothing needs
+renewing: every worker joins with a key that carries the fleet's tag, and a
+tagged device has key expiry disabled on the hosted Tailscale. For an enrolled
+worker whose key still expires (one that joined before it was tagged),
+`fleet-sync` disables expiry through the control plane
+(`tailscale:key-expiry-policy`); `tailscale:key-expiry` measures the result on
+the next run. Headscale gives a node registered with a pre-authorised key no
+expiry.
+
+### Maintenance reboots
+
+Kernel and library updates apply only after a reboot (`worker:reboot-pending`).
+`devfleet fleet-sync --maintenance-reboot` reboots each worker that has one
+pending: one at a time, the canary first, only while it runs no agent turn
+(`--force` ends them), through `ansible/ubuntu/reboot.yml`. Each worker is
+converged again once it answers; a worker that does not come back ready stops
+the ones after it. A new worker that was never ready is rebooted once after its
+first converge in any `fleet-sync`, so it reaches ready in one run; a worker
+that was ready before is rebooted only by `--maintenance-reboot`.
+
 To re-join a worker whose key expired, a person signs it in again, because
 only a person's identity can authorise a device:
 
@@ -205,7 +226,8 @@ serially; diagnostics run at most four remote probes concurrently.
   shell PATH and installs frozen Bun dependencies. Review OS-specific effects
   and existing config conflict guards in the shared bootstrap runbook.
 - Record input hashes only after success and stable checkout inputs. Failed
-  setup can leave partial installs; no rollback is claimed. Trusted checkout
+  setup can leave partial installs, which the next sync completes. Going back
+  is `fleet-sync --rollback`, as the [README](../README.md#the-version) says. Trusted checkout
   package lifecycle scripts can execute during installation.
 - No other application repos, credentials, provider settings or cloud resources
   are synchronized. `devfleet fleet-sync` remains the broader convergence.
@@ -258,12 +280,45 @@ A notification carries node and check IDs only. `devfleet fleet-watch --test`
 sends one test notification to every target.
 
 Every run stores what it saw, and when, as `watch.json` beside the fleet state
-in the bucket. That time is the manager's heartbeat. A watcher on the manager
-notices nothing when the manager itself is off, so call the reusable
-`fleet-heartbeat.yml` workflow from the repository that holds your
-`fleet.json` (its header shows the one `uses:` line): it fails, and GitHub
-notifies you, when the heartbeat is older than `max-age-minutes`. `bun run health` remains the separate
-single read-only **cloud platform** report.
+in the bucket. That time is the manager's heartbeat. Nobody has to run it:
+`fleet-sync` installs it on the manager as the launchd job
+`dev.devfleet.fleet-watch` (`ansible/macos/manager.yml`), which runs every
+`manager.schedule.watchMinutes` minutes of `fleet.json` (15 by default) from
+the checkout `fleet-sync` last applied, and `manager:watch` fails when it is
+not loaded. It reads the bucket credentials `fleet-state --init` saved.
+
+A watcher on the manager notices nothing when the manager itself is off, so
+call the reusable `fleet-heartbeat.yml` workflow from the repository that
+holds your `fleet.json` (its header shows the one `uses:` line): it fails, and
+GitHub notifies you, when the heartbeat is older than `max-age-minutes`
+(60 by default). With the watch job every 15 minutes and the workflow every
+30, a manager that is turned off fails the workflow within about 90 minutes
+(the age limit plus one interval of the workflow's schedule; GitHub may delay
+a scheduled run). Keep `max-age-minutes` well above `watchMinutes`.
+
+The CLI proxy is checked off the manager the same way:
+`.github/workflows/cliproxy-health.yml` is a reusable workflow, called with one
+`uses:` line. It needs no stored secret: it reads the Vault address and the
+proxy endpoint from your `fleet.json`, logs in to Vault with GitHub's OIDC
+token (the read-only role `vault-role` names, `github-actions` by default) and
+runs the checks of the dev-fleet release your `package.json` pins. Copy
+[`cliproxy/docs/health-workflow.example.yml`](../cliproxy/docs/health-workflow.example.yml)
+into `.github/workflows/` and name the release tag in its `uses:` line; it
+runs the free readiness check every 15 minutes, a light inference check hourly
+and the full suite nightly, and keeps one tracking issue in your repository.
+
+### The logs of the manager's scheduled jobs
+
+`fleet-sync --schedule` and the watch job write to
+`~/.local/share/personal/fleet/manager/logs/<label>.{out,err}.log`. newsyslog
+rotates each once it passes `jobLogs.maxKiB` of `ansible/macos/policy.json`
+(1024 KiB) and keeps `jobLogs.keep` compressed archives, from the drop-in
+`/etc/newsyslog.d/dev-fleet-USER.conf` that the manager's one administrator
+grant (`ansible/macos/manager-privilege.yml`) installs. `manager:job-logs`
+warns when the drop-in is missing, or a log has grown past twice that size.
+
+`bun run health` remains the separate single read-only **cloud platform**
+report.
 
 ## Development
 

package/docs/releasing.md

@@ -51,7 +51,33 @@ cannot release because publishing was never set up (no tag and nothing on
 npm) fails, and the run summary says why; a merge with nothing new to release
 passes and says so. Re-running a
 release whose version is already on npm skips the publish and finishes the tag
-and GitHub Release.
+and GitHub Release, whether all its jobs or only the failed ones are re-run: a
+`HEAD` that carries its tag but has no GitHub Release is released again.
+
+Nothing is published until the release has been tried. The workflow packs
+the package once, then, for that commit and that tarball:
+
+- installs the tarball in an empty directory, the way a fleet project does,
+  and runs `devfleet --version` and `devfleet fleet-state --example` on Node
+  22 and 24 ([`bootstrap/package-smoke.ts`](../bootstrap/package-smoke.ts)):
+  a broken bin path or a file missing from `files` fails here;
+- runs the [Ubuntu VM test](../.github/workflows/vm.yml) (a real Ubuntu
+  machine: baseline, a second run, drift, reboot, and the production SSH
+  bootstrap of the commit);
+- runs the [tailnet test](../.github/workflows/tailnet.yml) (a worker joins
+  a self-hosted tailnet unattended and leaves no device behind).
+
+Only when all three pass is that same tarball published, and the GitHub Release
+gets notes written from the Conventional Commits since the last release
+([`bootstrap/release-notes.ts`](../bootstrap/release-notes.ts)). A commit
+whose change restarts T3 on the workers, and so ends the agent turns running
+there, says so in a `Worker-Restart: <reason>` footer, and the notes list it
+first.
+
+There is no `next` channel yet: publishing uses npm Trusted Publishing, which
+can publish but cannot move a dist-tag, and moving one would need an npm
+token stored in CI. The VM test before publishing stands in for the soak a
+`next` channel would give.
 
 A hand-pushed tag still works and publishes exactly that version:
 
@@ -62,6 +88,28 @@ git tag vX.Y.Z && git push origin vX.Y.Z
 CI never receives an npm token: it publishes through npm Trusted Publishing
 (OIDC).
 
+## When a release fails
+
+A release that fails publishes nothing, and neither does any merge after it
+until one passes: npm stays at the last version. The
+[Failure issues workflow](../.github/workflows/failures.yml) keeps one open
+issue, `Release is failing on main` (label `ci-failure`), with the run, the
+commit and the jobs and steps that failed; a failed VM test keeps its
+redacted results and the guest's logs as an artifact of the run
+(`gh run download <run>`). The next passing release closes the issue and
+publishes everything since the last tag. The same holds for CI on `main`,
+which the release waits for.
+
+A pull request is merged by a workflow, and a merge made with a workflow's
+own token starts no CI on `main`. [`auto-merge.yml`](../.github/workflows/auto-merge.yml)
+starts it when the newest commit on `main` has no run: when a CI run ends,
+and every hour. A run started that way starts nothing when it ends either,
+so the same hourly run starts the release once CI has passed on the newest
+commit of `main` and no release has run for it, and the Failure issues
+workflow reads the newest finished run of every workflow it follows each
+hour as well. A merge is therefore released within about an hour of its CI
+passing.
+
 ## One-time setup
 
 npm can publish new versions through OIDC but cannot create a package that does

package/docs/runbooks.md

@@ -8,7 +8,7 @@ person (a login, an approval, a choice) is marked **needs a person**.
 `features/rules/runbooks.feature` fails when a code in the source has no
 entry here, or a `devfleet` command quoted in the documents does not exist.
 
-There are 244 codes; 31 need a person.
+There are 301 codes; 46 need a person.
 
 ## Bootstrap of a new or returning worker
 
@@ -34,7 +34,7 @@ There are 244 codes; 31 need a person.
 - **Meaning:** A package transaction could not run or the package database is inconsistent.
 - **Confirm:** `ssh WORKER sudo dpkg --audit` and `sudo apt-get -o Debug::NoLocking=1 check`.
 - **Fix:** Let any running package process finish (it is never killed), free disk under `/var`, then run `devfleet fleet-sync`.
-- **Escalate:** APT_STATE_BROKEN and DPKG_INTERRUPTED need `sudo dpkg --configure -a` by a person once no package process runs.
+- **Escalate:** An interrupted package run is finished by the next `devfleet fleet-sync`: the bootstrap and the baseline each run `dpkg --configure -a` once, bounded. APT_STATE_BROKEN and DPKG_INTERRUPTED mean that repair did not clear `dpkg --audit`: a person repairs the packages it names.
 
 - `APT_DISK_FULL`: apt disk full.
 - `APT_DNS_UNAVAILABLE`: apt dns unavailable.
@@ -104,6 +104,18 @@ There are 244 codes; 31 need a person.
 - `WORKER_RECOVERY_SETUP_FAILED`: worker recovery setup failed.
 - `WORKER_RECOVERY_TIMER_INACTIVE`: worker recovery timer inactive.
 
+## Worker size and features
+
+- **Meaning:** The machine is smaller than the minimum worker (`sizing.minimumWorker` in `ansible/ubuntu/policy.json`), more agent sessions run than its memory policy supports, its declared `features` in `fleet.json` are not ones dev-fleet knows, or something dev-fleet installed for a feature that is now off is still there.
+- **Confirm:** `devfleet fleet-sync --dry-run` shows the `platform:worker-size` check of each worker and, under the plan, what turning a feature off removes; `devfleet fleet-status --workers` shows `worker:features` and `worker:agent-sessions`.
+- **Fix:** Resize a machine below the minimum, then run `devfleet fleet-sync`; nothing was installed on it. For a feature removal, run `devfleet fleet-sync` again. For too many sessions, finish some or move work to a larger worker.
+- **Escalate:** A package another package depends on stays installed until a person removes that one.
+
+- `AGENT_SESSION_CAPACITY_EXHAUSTED`: agent session capacity exhausted: more agent sessions run than the memory policy supports (a warning; the size of a session is an estimate).
+- `WORKER_FEATURES_INVALID`: worker features invalid: the session is neither desktop nor headless.
+- `WORKER_FEATURE_REMOVAL_INCOMPLETE`: worker feature removal incomplete: a Docker package or the docker group membership is still there with containers off.
+- `WORKER_SIZE_BELOW_MINIMUM`: worker size below minimum: less memory, disk or CPUs than the minimum worker. **Needs a person.**
+
 ## Containers on a worker
 
 - **Meaning:** `worker:docker-run` could not start the pinned container or build a one-line image as the worker user.
@@ -119,6 +131,23 @@ There are 244 codes; 31 need a person.
 - `DOCKER_SMOKE_IMAGE_MISSING`: docker smoke image missing.
 - `DOCKER_SOCKET_ACCESS_DENIED`: docker socket access denied.
 
+## Exposure of a worker
+
+- **Meaning:** Something on the worker is reachable from outside the tailnet, or could be: the fleet firewall (`ansible/ubuntu/tasks/firewall.yml`) is not loaded or kept SSH open, a service listens beyond loopback and the tailnet (`worker:exposure`), or the manager reached a fleet port on the worker's public address (`worker:public-reachability`). The exposure policy is `exposure` in `fleet.json` (fleet-wide under `worker`, per worker beside `network`).
+- **Confirm:** On the worker: `sudo nft list table inet personal_fleet`, `sudo ss -H -l -n -t -u -p` and `sudo sshd -T | grep -E 'passwordauthentication|permitrootlogin'`. From the manager: `devfleet fleet-status --workers`.
+- **Fix:** Run `devfleet fleet-sync`. It closes SSH beyond the tailnet only when the manager's session arrives over the tailnet: point the worker's SSH alias at its tailnet name first. Bind a service to `127.0.0.1` or the tailnet address, or declare its port in `publicPorts`.
+- **Escalate:** A firewall the tailnet path could not verify (the previous rules are back) needs `tailscale status` on the worker; a public SSH port is opened on purpose only with `publicSsh`, by a person.
+
+- `FIREWALL_RULES_REJECTED`: firewall rules rejected by `nft --check`; the previous rules stay loaded.
+- `FIREWALL_TAILNET_UNVERIFIED`: the tailnet path did not answer a new probe after the rules loaded; the previous rules were restored.
+- `FIREWALL_UNMANAGED_CONFLICT`: `/etc/personal-fleet/firewall.nft` exists without the fleet's marker. **Needs a person.**
+- `SSH_HARDENING_OVERRIDDEN`: `sshd -T` still allows passwords or root login: an earlier fragment or a Match block wins. **Needs a person.**
+- `WORKER_EXPOSURE_UNOBSERVED`: the addresses, sockets or firewall could not be read with `sudo -n`.
+- `WORKER_FIREWALL_INACTIVE`: the fleet firewall is not loaded.
+- `WORKER_LISTENER_UNRESTRICTED`: a socket listens beyond loopback and the tailnet outside the allow-list; the summary names the port and the owning unit. A fleet service fails; another warns or fails by `otherListeners`.
+- `WORKER_PUBLIC_PORT_UNRESTRICTED`: the manager connected to SSH, T3 or the monitoring port on the worker's public address.
+- `WORKER_PUBLIC_SSH_UNDECLARED`: SSH is still open beyond the tailnet without `publicSsh`, because no session over the tailnet proved the manager's path. **Needs a person.**
+
 ## Headless browser on a worker
 
 - **Meaning:** `worker:browser` could not launch the pinned headless Chromium of the web tool profile.
@@ -133,10 +162,10 @@ There are 244 codes; 31 need a person.
 
 ## Git identity and pushing
 
-- **Meaning:** Commits on the node would carry the wrong author, or git cannot authenticate to GitHub on its own.
+- **Meaning:** Commits on the node would carry the wrong author, would not be signed although `git.signing` is declared, or git cannot authenticate to GitHub on its own.
 - **Confirm:** On the node: `git config --global --get user.name`, and `printf "protocol=https\nhost=github.com\n\n" | GIT_TERMINAL_PROMPT=0 git credential fill >/dev/null`.
 - **Fix:** Declare the `git` block in `fleet.json` and run `devfleet fleet-sync` (`ansible/github/git.yml`). Remove lines in `~/.gitconfig` that override it.
-- **Escalate:** GIT_REMOTE_REFUSED means the fleet account lost access to the declared repository: a person grants it.
+- **Escalate:** GIT_REMOTE_REFUSED means the fleet account lost access to the declared repository: a person grants it. GitHub shows signed commits as verified only once a person adds each node's `~/.ssh/dev-fleet-git-signing.pub` to the account as a signing key.
 
 - `GIT_CONFIGURATION_INCOMPLETE`: git configuration incomplete.
 - `GIT_CREDENTIAL_HELPER_MISSING`: git credential helper missing.
@@ -146,6 +175,9 @@ There are 244 codes; 31 need a person.
 - `GIT_IDENTITY_UNDECLARED`: git identity undeclared.
 - `GIT_REMOTE_REFUSED`: git remote refused. **Needs a person.**
 - `GIT_REMOTE_UNREACHABLE`: git remote unreachable.
+- `GIT_SIGNING_FAILED`: git signing failed: the node's own key is missing or cannot sign without a prompt.
+- `GIT_SIGNING_MISSING`: git signing missing.
+- `GIT_SIGNING_OVERRIDDEN`: git signing overridden.
 
 ## GitHub sign-in
 
@@ -180,12 +212,34 @@ There are 244 codes; 31 need a person.
 - `TAILSCALE_SOURCE_OWNER_CONFLICT`: tailscale source owner conflict. **Needs a person.**
 - `TAILSCALE_UPGRADE_PENDING`: tailscale upgrade pending. **Needs a person.**
 
+## A new worker with no hands on it
+
+- **Meaning:** A machine created without a person (from a pipeline or infrastructure-as-code) could not be enrolled, joined to the tailnet, given its account or brought to ready in one run: its first boot is still running or failed, the tailnet control plane refused a key, the image lets only root in and the account could not be created, or its first-converge or maintenance reboot did not happen.
+- **Confirm:** The `bootstrap:first-boot`, `tailscale:enrolment-key`, `worker:account` and `worker:reboot` checks of the report `devfleet fleet-sync --json` prints, or the `tailscale:device` check of `fleet-node-replace` and `fleet-node-retire` and the `tailscale:leftover` check of `devfleet fleet-status`, which ask the same control plane; on the worker, `cloud-init status --long`.
+- **Fix:** FIRST_BOOT_PENDING and WORKER_REBOOT_BUSY clear on their own: run `devfleet fleet-sync` again later. A TAILNET code names what to fix in `tailnet` of `fleet.json` or in the secret `tailnet-admin`; then run `devfleet fleet-sync`.
+- **Escalate:** A first boot that failed means the image or its user data is wrong: recreate the machine. A root login that cannot create the declared account needs a person on the machine (`useradd --create-home --shell /bin/bash --groups sudo USER`).
+
+- `FIRST_BOOT_FAILED`: first boot failed: cloud-init ended in an error. **Needs a person.**
+- `FIRST_BOOT_PENDING`: first boot pending: cloud-init still runs; retryable.
+- `TAILNET_ADMIN_REQUIRED`: tailnet admin required: fleet.json declares tailnet without the secret. **Needs a person.**
+- `TAILNET_CONTROL_DENIED`: tailnet control denied the tailnet-admin credential. **Needs a person.**
+- `TAILNET_CONTROL_REJECTED`: tailnet control rejected the request (tag or user). **Needs a person.**
+- `TAILNET_CONTROL_RESPONSE_INVALID`: tailnet control response invalid.
+- `TAILNET_CONTROL_UNAVAILABLE`: tailnet control unavailable.
+- `TAILNET_KEY_DELIVERY_FAILED`: tailnet key delivery failed.
+- `TAILNET_KEY_INVALID`: tailnet key invalid.
+- `WORKER_ACCOUNT_CREATION_FAILED`: worker account creation failed. **Needs a person.**
+- `WORKER_ACCOUNT_ROOT_REQUIRED`: worker account root required. **Needs a person.**
+- `WORKER_ACCOUNT_UNDECLARED`: worker account undeclared in workers[].network.user. **Needs a person.**
+- `WORKER_REBOOT_BUSY`: worker reboot busy: an agent turn runs.
+- `WORKER_REBOOT_FAILED`: worker reboot failed.
+
 ## T3 Code
 
 - **Meaning:** The native T3 server or desktop app, its policy, its private route or its providers did not converge or verify.
 - **Confirm:** `devfleet t3-status` and the `t3:*` checks of `devfleet fleet-status`.
 - **Fix:** Run `devfleet fleet-sync`, then `devfleet t3-apply` for providers. A MIGRATION or MAINTENANCE code names a one-time step in its remedy.
-- **Escalate:** Pairing and relinking a peer needs a person in T3 Settings.
+- **Escalate:** A worker's first pairing, and relinking a peer that was not replaced, need a person in T3 Settings (Connections). A replaced worker is paired by `devfleet fleet-node-replace` or the next `devfleet fleet-sync` (`t3:link`).
 
 - `MANAGER_T3_LINK_UNVERIFIED`: manager t3 link unverified.
 - `T3_BASELINE_PREREQUISITES_REQUIRED`: t3 baseline prerequisites required.
@@ -224,6 +278,8 @@ There are 244 codes; 31 need a person.
 - `T3_PROVIDER_NAMES_PENDING`: t3 provider names pending.
 - `T3_PROVIDER_NAMES_UNVERIFIED`: t3 provider names unverified.
 - `T3_RESTART_PENDING`: t3 restart pending.
+- `T3_SERVER_INSTALL_FAILED`: t3 server install failed: the pinned server of a headless worker is missing from the toolchain, or `t3-headless.yml` did not apply.
+- `T3_SERVER_STOPPED`: t3 server stopped: the lingering `t3code-headless.service` of a headless worker is not running.
 - `T3_SETTINGS_BUSY`: t3 settings busy.
 
 ## Monitoring (Beszel)
@@ -295,10 +351,12 @@ There are 244 codes; 31 need a person.
 - **Meaning:** An agent CLI, its profile or a provider behind the CLI proxy did not answer as expected.
 - **Confirm:** The `agent:*` and `provider:*` checks of `devfleet fleet-status`, and `devfleet cliproxy` on the manager.
 - **Fix:** Run `devfleet fleet-sync`; a rate limit or upstream outage clears on its own.
-- **Escalate:** PROVIDER_AUTH_REJECTED needs a person to sign the account in again on the proxy.
+- **Escalate:** PROVIDER_AUTH_REJECTED needs a person to sign the account in again on the proxy. AGENT_PROFILE_RENDER_MISMATCH is a defect of the release (a template of `ansible/agents/agents.yml` no longer renders what `src/agents/render.ts` specifies): report it to dev-fleet. No receipt vouches for what that play wrote, so later applies refuse the file until a person reviews it.
 
 - `AGENT_MEMORY_LIMIT_MISSING`: agent memory limit missing.
 - `AGENT_MODEL_UNSERVED`: agent model unserved.
+- `AGENT_PROFILE_RENDER_MISMATCH`: agent profile render mismatch.
+- `AGENT_PROFILE_REQUEST_INVALID`: agent profile request invalid.
 - `AGENT_PROFILE_STALE`: agent profile stale.
 - `AGENT_SERVICE_ENVIRONMENT_INCOMPLETE`: agent service environment incomplete.
 - `AGENT_SERVICE_ENVIRONMENT_UNVERIFIED`: agent service environment unverified.
@@ -338,7 +396,7 @@ There are 244 codes; 31 need a person.
 ## Vault and credentials
 
 - **Meaning:** A machine cannot read the secrets it needs with its own Vault session.
-- **Confirm:** `vault token lookup` on the machine and the `worker:vault-renewal` check.
+- **Confirm:** `vault token lookup` on the machine and the `worker:vault-renewal` or `manager:vault-renewal` check.
 - **Fix:** Log the machine in once (`devfleet fleet-sync` asks on the manager); renewal is automatic afterwards.
 - **Escalate:** A refused policy needs the Vault administrator.
 
@@ -346,6 +404,39 @@ There are 244 codes; 31 need a person.
 - `VAULT_LOGIN_REQUIRED`: vault login required. **Needs a person.**
 - `VAULT_POLICY_PERMISSION_REQUIRED`: vault policy permission required. **Needs a person.**
 - `VAULT_RENEWAL_FAILING`: vault renewal failing.
+- `VAULT_RENEWAL_MISSING`: vault renewal missing: the manager's renewal job is not loaded. Run `devfleet fleet-sync` on the manager.
+- `VAULT_RENEWAL_NOT_MANAGER`: vault renewal not manager.
+- `VAULT_WORKER_POLICY_FAILED`: vault worker policy failed.
+
+## The fleet definition and a worker's lifecycle
+
+- **Meaning:** `fleet.json` breaks a rule of the fleet definition, the pinned Ansible rejects the inventory it describes, or a seed, replacement or retirement could not change the host keys the manager trusts.
+- **Confirm:** `devfleet fleet-validate` names the field that breaks a rule as `{code, path, expected, actual}`; `devfleet fleet-validate --ansible` also renders the inventory and syntax-checks every playbook.
+- **Fix:** Correct the field the check names in `fleet.json`, then run the command again. For SSH_HOST_KEY_NOT_CHANGED run `devfleet fleet-validate --ansible`, then repeat `devfleet fleet-node-seed`, `devfleet fleet-node-replace` or `devfleet fleet-node-retire`.
+- **Escalate:** An inventory the pinned Ansible rejects for a valid definition is an issue.
+
+- `DEFINITION_INVALID`: definition invalid.
+- `DEFINITION_INVENTORY_REJECTED`: definition inventory rejected.
+- `DEFINITION_MISSING`: definition missing.
+- `DEFINITION_UNREADABLE`: definition unreadable.
+- `SSH_HOST_KEY_NOT_CHANGED`: ssh host key not changed.
+
+## Parked, idle, replaced and retired workers
+
+- **Meaning:** a worker could not be parked or resumed (agent turns under way, a declared `parking.stop` or `parking.start` command failed, the sync after a resume did not end ready), a worker whose record disappeared could not be retired, or the manager's T3 link to a replaced or retired worker could not be removed, or the manager's T3 could not pair with a replaced worker's new environment.
+- **Confirm:** `devfleet fleet-status --workers` lists each worker as active or parked, with its uptime, agent turns and last agent activity; the report of `devfleet fleet-node-park`, `devfleet fleet-node-resume` or `devfleet fleet-sync` names the step that failed.
+- **Fix:** For WORKER_BUSY wait for the turns to end, or repeat `devfleet fleet-node-park` with `--force`. For a failed hook, run the command `fleet.json` names under `parking` by hand with the worker ID, then repeat. For WORKER_RESUME_FAILED follow the sync report and run `devfleet fleet-sync --node ID`. For T3_LINK_UNLINK_FAILED remove the link in the manager's T3 (Settings, Connections). For T3_LINK_PAIRING_FAILED follow the remedy, which names why the link was refused (Linked environments off in the T3 policy, the manager's T3 not running, a worker T3 that did not mint a code), then run `devfleet fleet-sync --node ID`: it pairs a replaced worker until it is paired.
+- **Escalate:** A record retirement that keeps failing at the same step: follow that step's own runbook.
+
+- `T3_LINK_PAIRING_FAILED`: t3 link pairing failed: the manager's T3 could not pair with a replaced worker's new T3 environment. The pairing code never appears in a report.
+- `T3_LINK_UNLINK_FAILED`: t3 link unlink failed.
+- `WORKER_ACTIVITY_UNREADABLE`: worker activity unreadable. **Needs a person.**
+- `WORKER_AUTO_PARK_REFUSED`: worker auto park refused.
+- `WORKER_BUSY`: worker busy. **Needs a person.**
+- `WORKER_RECORD_RETIREMENT_FAILED`: worker record retirement failed.
+- `WORKER_RESUME_FAILED`: worker resume failed.
+- `WORKER_START_HOOK_FAILED`: worker start hook failed.
+- `WORKER_STOP_HOOK_FAILED`: worker stop hook failed.
 
 ## Fleet operations and the fleet state
 
@@ -366,6 +457,7 @@ There are 244 codes; 31 need a person.
 - `FLEET_REMOTE_LOCKED`: fleet remote locked.
 - `FLEET_REMOTE_OPERATION_FAILED`: fleet remote operation failed.
 - `FLEET_REVISION_REQUIRED`: fleet revision required.
+- `FLEET_ROLLBACK_UNAVAILABLE`: fleet rollback unavailable.
 - `FLEET_SCHEDULE_PREVIEW`: fleet schedule preview.
 - `FLEET_SOURCE_UNAVAILABLE`: fleet source unavailable.
 - `FLEET_STATE_DIVERGED`: fleet state diverged. **Needs a person.**
@@ -380,6 +472,19 @@ There are 244 codes; 31 need a person.
 - `REMOTE_CONFIGURATION_INCOMPLETE`: remote configuration incomplete.
 - `REMOTE_PHASE_FAILED`: remote phase failed.
 
+## The manager's own policy and scheduled jobs
+
+- **Meaning:** `ansible/macos/manager.yml` did not pass its check-mode run or its apply, or something it declares (network time, the fleet-watch job, the rotation of the scheduled jobs' logs) is not in place.
+- **Confirm:** The `manager:playbook`, `manager:clock`, `manager:watch` and `manager:job-logs` checks of `devfleet fleet-sync --dry-run`; on the manager, the check-mode run the remedy quotes.
+- **Fix:** Run `devfleet fleet-sync`, which applies the playbook in check mode first, then for real. What needs root (power settings, reading network time, newsyslog) comes from the one grant: `ansible-playbook -e user=$(id -un) -K ansible/macos/manager-privilege.yml`, run again after an upgrade that changed it.
+- **Escalate:** The grant needs an administrator's password, once.
+
+- `MANAGER_CLOCK_UNOBSERVED`: manager clock unobserved: the grant that lets fleet-sync read network time is missing. **Needs a person.**
+- `MANAGER_JOB_LOG_ROTATION_MISSING`: manager job log rotation missing: the newsyslog drop-in the grant installs is missing or outdated. **Needs a person.**
+- `MANAGER_PLAYBOOK_CHECK_FAILED`: manager playbook check failed: nothing was applied.
+- `MANAGER_PLAYBOOK_FAILED`: manager playbook failed.
+- `MANAGER_WATCH_INTERVAL_INVALID`: manager watch interval invalid: `manager.schedule.watchMinutes` is outside 5 to 1440.
+
 ## Notifications (fleet-watch)
 
 - **Meaning:** fleet-watch could not reach a notification target `notify` in `fleet.json` names, or its address is not stored as a secret.

package/docs/state.md

@@ -4,12 +4,13 @@ A fleet is described in two places you own. Your project in Git holds
 `fleet.json` and pins the dev-fleet version. Everything else the manager
 cannot rebuild from Git is the **fleet state**:
 
-| Document     | What it is                                                            |
-| ------------ | --------------------------------------------------------------------- |
-| `roster`     | The enrolled connection to each worker (SSH alias, Tailscale device)  |
-| `enrollment` | The revision `fleet-sync` approved and the one it last applied        |
-| `definition` | The fleet configuration `fleet-sync` last applied: your `fleet.json`  |
-| `history`    | The last 90 days of `fleet-sync` runs (at most 500), and the schedule |
+| Document     | What it is                                                                                                                                                                    |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `roster`     | The enrolled connection to each worker (SSH alias, Tailscale device)                                                                                                          |
+| `enrollment` | The revision `fleet-sync` approved and the one it last applied                                                                                                                |
+| `definition` | The fleet configuration `fleet-sync` last applied: your `fleet.json`                                                                                                          |
+| `history`    | The last 90 days of `fleet-sync` runs (at most 500), and the schedule                                                                                                         |
+| `workers`    | Per worker: its pinned SSH host key, whether it is parked and since when, whether it came from a worker record, and whether the manager is to pair its T3 after a replacement |
 
 `devfleet fleet-watch` also keeps `watch.json` in the bucket, beside the
 database: the health of each worker when it last looked, and when that was
@@ -195,6 +196,12 @@ state that is replicated differently or not at all:
 | Monitoring hub (Beszel) database                     | `~/Library/Application Support/personal-beszel`          | `fleet-sync` recreates accounts and systems; metric history, alerts and settings made by hand are lost |
 | T3 ownership record                                  | `~/.t3/userdata/personal-managed.json`                   | `t3-apply --import ID` adopts instances again                                                          |
 
+The manager reaches a worker that has no SSH alias against
+`~/.local/share/dev-fleet/ssh/known_hosts` (beside the state directory): a
+cache rendered from the host keys the `workers` document pins and the worker
+records declare, rewritten before every session. Delete it and the next
+command writes it again; nothing is ever learned into it.
+
 Everything else under `~/.local/share/personal/fleet` and
 `~/.local/state/personal/fleet` (the copy of the source repository, run
 checkouts, reports, receipts, logs) is a cache that the next `fleet-sync`
@@ -257,9 +264,14 @@ mise exec litestream -- litestream restore -no-expand-env \
 ```
 
 `devfleet fleet-state --restore --timestamp 2026-11-01T00:00:00Z --yes` does
-the same and then uploads that version as the bucket's current one. It is the
-remedy for a `corrupt` replica: the bucket answers, but its latest version
-cannot be read.
+the same and then uploads that version as the bucket's current one, which
+takes the manager lease. It is the remedy for a `corrupt` replica: the bucket
+answers, but its latest version cannot be read. `--yes` consents to replacing
+the bucket, not to replacing another manager: while one holds the lease in the
+bucket's current version, the restore stops with `FLEET_MANAGER_LEASE_HELD`
+before it changes anything, and `--take-over` overrides it. When the current
+version cannot be read, its lease cannot either, and the lease in this
+machine's copy (the last one it saw) decides instead.
 
 Litestream is pinned in `mise.toml` and installed the first time it is used.
 The manager is the only machine that runs it.
@@ -274,8 +286,8 @@ writes anything and stops with `FLEET_MANAGER_LEASE_HELD`. `fleet-state
 --init` still restores the state on such a machine; it only writes nothing.
 
 The lease runs for seven days after the last change. After that, any machine
-may take it. Before that, `--take-over` (on `fleet-sync`, `fleet-state --init`
-or `fleet-state --import`) makes this machine the manager; use it only once
+may take it. Before that, `--take-over` (on `fleet-sync`, `fleet-state --init`,
+`fleet-state --import` or `fleet-state --restore --timestamp`) makes this machine the manager; use it only once
 the other one is gone ([replacing the manager](manager-recovery.md)).
 
 A machine is named in the lease by a digest of its machine ID