@crvouga/dev-fleet 0.8.0 → 0.9.1

package/docs/operations.md

@@ -1,6 +1,7 @@
 > This document was written for the fleet dev-fleet was extracted from. Paths and
 > commands were updated; some statements still describe that fleet and are being
-> revised. The README is authoritative.
+> revised. The README is authoritative, and what a machine must offer to be a
+> worker is listed only in [worker requirements](worker-requirements.md).
 
 # Developer fleet
 
@@ -22,7 +23,7 @@ Use `devfleet fleet-sync` to reconcile all declared workers and
 `devfleet fleet-status --workers` to inspect remaining gaps. See
 [worker-contract.md](worker-contract.md) for bootstrap, ownership and the exact
 scope of readiness. `fleet-status` without flags diagnoses the local roster.
-Workers are Ubuntu 26.04; the manager is macOS and never sleeps on AC power
+Workers are Ubuntu 26.04 or 24.04; the manager is macOS and never sleeps on AC power
 (`ansible/macos/manager.yml`). Machine policy (never sleep, no automatic
 logout, always-on Tailscale/SSH/Docker/Beszel, memory and disk safety, per-node
 credential renewal) is
@@ -134,6 +135,26 @@ expiring: tag them, or disable key expiry for each in the Tailscale admin
 console (tagged nodes do not expire). `tailscale:key-expiry` warns two weeks
 ahead and fails three days ahead.
 
+With `tailnet` in `fleet.json` and the secret `tailnet-admin`, nothing needs
+renewing: every worker joins with a key that carries the fleet's tag, and a
+tagged device has key expiry disabled on the hosted Tailscale. For an enrolled
+worker whose key still expires (one that joined before it was tagged),
+`fleet-sync` disables expiry through the control plane
+(`tailscale:key-expiry-policy`); `tailscale:key-expiry` measures the result on
+the next run. Headscale gives a node registered with a pre-authorised key no
+expiry.
+
+### Maintenance reboots
+
+Kernel and library updates apply only after a reboot (`worker:reboot-pending`).
+`devfleet fleet-sync --maintenance-reboot` reboots each worker that has one
+pending: one at a time, the canary first, only while it runs no agent turn
+(`--force` ends them), through `ansible/ubuntu/reboot.yml`. Each worker is
+converged again once it answers; a worker that does not come back ready stops
+the ones after it. A new worker that was never ready is rebooted once after its
+first converge in any `fleet-sync`, so it reaches ready in one run; a worker
+that was ready before is rebooted only by `--maintenance-reboot`.
+
 To re-join a worker whose key expired, a person signs it in again, because
 only a person's identity can authorise a device:
 
@@ -205,7 +226,8 @@ serially; diagnostics run at most four remote probes concurrently.
   shell PATH and installs frozen Bun dependencies. Review OS-specific effects
   and existing config conflict guards in the shared bootstrap runbook.
 - Record input hashes only after success and stable checkout inputs. Failed
-  setup can leave partial installs; no rollback is claimed. Trusted checkout
+  setup can leave partial installs, which the next sync completes. Going back
+  is `fleet-sync --rollback`, as the [README](../README.md#the-version) says. Trusted checkout
   package lifecycle scripts can execute during installation.
 - No other application repos, credentials, provider settings or cloud resources
   are synchronized. `devfleet fleet-sync` remains the broader convergence.
@@ -258,12 +280,45 @@ A notification carries node and check IDs only. `devfleet fleet-watch --test`
 sends one test notification to every target.
 
 Every run stores what it saw, and when, as `watch.json` beside the fleet state
-in the bucket. That time is the manager's heartbeat. A watcher on the manager
-notices nothing when the manager itself is off, so call the reusable
-`fleet-heartbeat.yml` workflow from the repository that holds your
-`fleet.json` (its header shows the one `uses:` line): it fails, and GitHub
-notifies you, when the heartbeat is older than `max-age-minutes`. `bun run health` remains the separate
-single read-only **cloud platform** report.
+in the bucket. That time is the manager's heartbeat. Nobody has to run it:
+`fleet-sync` installs it on the manager as the launchd job
+`dev.devfleet.fleet-watch` (`ansible/macos/manager.yml`), which runs every
+`manager.schedule.watchMinutes` minutes of `fleet.json` (15 by default) from
+the checkout `fleet-sync` last applied, and `manager:watch` fails when it is
+not loaded. It reads the bucket credentials `fleet-state --init` saved.
+
+A watcher on the manager notices nothing when the manager itself is off, so
+call the reusable `fleet-heartbeat.yml` workflow from the repository that
+holds your `fleet.json` (its header shows the one `uses:` line): it fails, and
+GitHub notifies you, when the heartbeat is older than `max-age-minutes`
+(60 by default). With the watch job every 15 minutes and the workflow every
+30, a manager that is turned off fails the workflow within about 90 minutes
+(the age limit plus one interval of the workflow's schedule; GitHub may delay
+a scheduled run). Keep `max-age-minutes` well above `watchMinutes`.
+
+The CLI proxy is checked off the manager the same way:
+`.github/workflows/cliproxy-health.yml` is a reusable workflow, called with one
+`uses:` line. It needs no stored secret: it reads the Vault address and the
+proxy endpoint from your `fleet.json`, logs in to Vault with GitHub's OIDC
+token (the read-only role `vault-role` names, `github-actions` by default) and
+runs the checks of the dev-fleet release your `package.json` pins. Copy
+[`cliproxy/docs/health-workflow.example.yml`](../cliproxy/docs/health-workflow.example.yml)
+into `.github/workflows/` and name the release tag in its `uses:` line; it
+runs the free readiness check every 15 minutes, a light inference check hourly
+and the full suite nightly, and keeps one tracking issue in your repository.
+
+### The logs of the manager's scheduled jobs
+
+`fleet-sync --schedule` and the watch job write to
+`~/.local/share/personal/fleet/manager/logs/<label>.{out,err}.log`. newsyslog
+rotates each once it passes `jobLogs.maxKiB` of `ansible/macos/policy.json`
+(1024 KiB) and keeps `jobLogs.keep` compressed archives, from the drop-in
+`/etc/newsyslog.d/dev-fleet-USER.conf` that the manager's one administrator
+grant (`ansible/macos/manager-privilege.yml`) installs. `manager:job-logs`
+warns when the drop-in is missing, or a log has grown past twice that size.
+
+`bun run health` remains the separate single read-only **cloud platform**
+report.
 
 ## Development
 

package/docs/releasing.md

@@ -51,7 +51,33 @@ cannot release because publishing was never set up (no tag and nothing on
 npm) fails, and the run summary says why; a merge with nothing new to release
 passes and says so. Re-running a
 release whose version is already on npm skips the publish and finishes the tag
-and GitHub Release.
+and GitHub Release, whether all its jobs or only the failed ones are re-run: a
+`HEAD` that carries its tag but has no GitHub Release is released again.
+
+Nothing is published until the release has been tried. The workflow packs
+the package once, then, for that commit and that tarball:
+
+- installs the tarball in an empty directory, the way a fleet project does,
+  and runs `devfleet --version` and `devfleet fleet-state --example` on Node
+  22 and 24 ([`bootstrap/package-smoke.ts`](../bootstrap/package-smoke.ts)):
+  a broken bin path or a file missing from `files` fails here;
+- runs the [Ubuntu VM test](../.github/workflows/vm.yml) (a real Ubuntu
+  machine: baseline, a second run, drift, reboot, and the production SSH
+  bootstrap of the commit);
+- runs the [tailnet test](../.github/workflows/tailnet.yml) (a worker joins
+  a self-hosted tailnet unattended and leaves no device behind).
+
+Only when all three pass is that same tarball published, and the GitHub Release
+gets notes written from the Conventional Commits since the last release
+([`bootstrap/release-notes.ts`](../bootstrap/release-notes.ts)). A commit
+whose change restarts T3 on the workers, and so ends the agent turns running
+there, says so in a `Worker-Restart: <reason>` footer, and the notes list it
+first.
+
+There is no `next` channel yet: publishing uses npm Trusted Publishing, which
+can publish but cannot move a dist-tag, and moving one would need an npm
+token stored in CI. The VM test before publishing stands in for the soak a
+`next` channel would give.
 
 A hand-pushed tag still works and publishes exactly that version:
 
@@ -62,6 +88,28 @@ git tag vX.Y.Z && git push origin vX.Y.Z
 CI never receives an npm token: it publishes through npm Trusted Publishing
 (OIDC).
 
+## When a release fails
+
+A release that fails publishes nothing, and neither does any merge after it
+until one passes: npm stays at the last version. The
+[Failure issues workflow](../.github/workflows/failures.yml) keeps one open
+issue, `Release is failing on main` (label `ci-failure`), with the run, the
+commit and the jobs and steps that failed; a failed VM test keeps its
+redacted results and the guest's logs as an artifact of the run
+(`gh run download <run>`). The next passing release closes the issue and
+publishes everything since the last tag. The same holds for CI on `main`,
+which the release waits for.
+
+A pull request is merged by a workflow, and a merge made with a workflow's
+own token starts no CI on `main`. [`auto-merge.yml`](../.github/workflows/auto-merge.yml)
+starts it when the newest commit on `main` has no run: when a CI run ends,
+and every hour. A run started that way starts nothing when it ends either,
+so the same hourly run starts the release once CI has passed on the newest
+commit of `main` and no release has run for it, and the Failure issues
+workflow reads the newest finished run of every workflow it follows each
+hour as well. A merge is therefore released within about an hour of its CI
+passing.
+
 ## One-time setup
 
 npm can publish new versions through OIDC but cannot create a package that does

package/docs/runbooks.md

@@ -8,7 +8,7 @@ person (a login, an approval, a choice) is marked **needs a person**.
 `features/rules/runbooks.feature` fails when a code in the source has no
 entry here, or a `devfleet` command quoted in the documents does not exist.
 
-There are 250 codes; 31 need a person.
+There are 301 codes; 46 need a person.
 
 ## Bootstrap of a new or returning worker
 
@@ -34,7 +34,7 @@ There are 250 codes; 31 need a person.
 - **Meaning:** A package transaction could not run or the package database is inconsistent.
 - **Confirm:** `ssh WORKER sudo dpkg --audit` and `sudo apt-get -o Debug::NoLocking=1 check`.
 - **Fix:** Let any running package process finish (it is never killed), free disk under `/var`, then run `devfleet fleet-sync`.
-- **Escalate:** APT_STATE_BROKEN and DPKG_INTERRUPTED need `sudo dpkg --configure -a` by a person once no package process runs.
+- **Escalate:** An interrupted package run is finished by the next `devfleet fleet-sync`: the bootstrap and the baseline each run `dpkg --configure -a` once, bounded. APT_STATE_BROKEN and DPKG_INTERRUPTED mean that repair did not clear `dpkg --audit`: a person repairs the packages it names.
 
 - `APT_DISK_FULL`: apt disk full.
 - `APT_DNS_UNAVAILABLE`: apt dns unavailable.
@@ -104,6 +104,18 @@ There are 250 codes; 31 need a person.
 - `WORKER_RECOVERY_SETUP_FAILED`: worker recovery setup failed.
 - `WORKER_RECOVERY_TIMER_INACTIVE`: worker recovery timer inactive.
 
+## Worker size and features
+
+- **Meaning:** The machine is smaller than the minimum worker (`sizing.minimumWorker` in `ansible/ubuntu/policy.json`), more agent sessions run than its memory policy supports, its declared `features` in `fleet.json` are not ones dev-fleet knows, or something dev-fleet installed for a feature that is now off is still there.
+- **Confirm:** `devfleet fleet-sync --dry-run` shows the `platform:worker-size` check of each worker and, under the plan, what turning a feature off removes; `devfleet fleet-status --workers` shows `worker:features` and `worker:agent-sessions`.
+- **Fix:** Resize a machine below the minimum, then run `devfleet fleet-sync`; nothing was installed on it. For a feature removal, run `devfleet fleet-sync` again. For too many sessions, finish some or move work to a larger worker.
+- **Escalate:** A package another package depends on stays installed until a person removes that one.
+
+- `AGENT_SESSION_CAPACITY_EXHAUSTED`: agent session capacity exhausted: more agent sessions run than the memory policy supports (a warning; the size of a session is an estimate).
+- `WORKER_FEATURES_INVALID`: worker features invalid: the session is neither desktop nor headless.
+- `WORKER_FEATURE_REMOVAL_INCOMPLETE`: worker feature removal incomplete: a Docker package or the docker group membership is still there with containers off.
+- `WORKER_SIZE_BELOW_MINIMUM`: worker size below minimum: less memory, disk or CPUs than the minimum worker. **Needs a person.**
+
 ## Containers on a worker
 
 - **Meaning:** `worker:docker-run` could not start the pinned container or build a one-line image as the worker user.
@@ -119,6 +131,23 @@ There are 250 codes; 31 need a person.
 - `DOCKER_SMOKE_IMAGE_MISSING`: docker smoke image missing.
 - `DOCKER_SOCKET_ACCESS_DENIED`: docker socket access denied.
 
+## Exposure of a worker
+
+- **Meaning:** Something on the worker is reachable from outside the tailnet, or could be: the fleet firewall (`ansible/ubuntu/tasks/firewall.yml`) is not loaded or kept SSH open, a service listens beyond loopback and the tailnet (`worker:exposure`), or the manager reached a fleet port on the worker's public address (`worker:public-reachability`). The exposure policy is `exposure` in `fleet.json` (fleet-wide under `worker`, per worker beside `network`).
+- **Confirm:** On the worker: `sudo nft list table inet personal_fleet`, `sudo ss -H -l -n -t -u -p` and `sudo sshd -T | grep -E 'passwordauthentication|permitrootlogin'`. From the manager: `devfleet fleet-status --workers`.
+- **Fix:** Run `devfleet fleet-sync`. It closes SSH beyond the tailnet only when the manager's session arrives over the tailnet: point the worker's SSH alias at its tailnet name first. Bind a service to `127.0.0.1` or the tailnet address, or declare its port in `publicPorts`.
+- **Escalate:** A firewall the tailnet path could not verify (the previous rules are back) needs `tailscale status` on the worker; a public SSH port is opened on purpose only with `publicSsh`, by a person.
+
+- `FIREWALL_RULES_REJECTED`: firewall rules rejected by `nft --check`; the previous rules stay loaded.
+- `FIREWALL_TAILNET_UNVERIFIED`: the tailnet path did not answer a new probe after the rules loaded; the previous rules were restored.
+- `FIREWALL_UNMANAGED_CONFLICT`: `/etc/personal-fleet/firewall.nft` exists without the fleet's marker. **Needs a person.**
+- `SSH_HARDENING_OVERRIDDEN`: `sshd -T` still allows passwords or root login: an earlier fragment or a Match block wins. **Needs a person.**
+- `WORKER_EXPOSURE_UNOBSERVED`: the addresses, sockets or firewall could not be read with `sudo -n`.
+- `WORKER_FIREWALL_INACTIVE`: the fleet firewall is not loaded.
+- `WORKER_LISTENER_UNRESTRICTED`: a socket listens beyond loopback and the tailnet outside the allow-list; the summary names the port and the owning unit. A fleet service fails; another warns or fails by `otherListeners`.
+- `WORKER_PUBLIC_PORT_UNRESTRICTED`: the manager connected to SSH, T3 or the monitoring port on the worker's public address.
+- `WORKER_PUBLIC_SSH_UNDECLARED`: SSH is still open beyond the tailnet without `publicSsh`, because no session over the tailnet proved the manager's path. **Needs a person.**
+
 ## Headless browser on a worker
 
 - **Meaning:** `worker:browser` could not launch the pinned headless Chromium of the web tool profile.
@@ -133,10 +162,10 @@ There are 250 codes; 31 need a person.
 
 ## Git identity and pushing
 
-- **Meaning:** Commits on the node would carry the wrong author, or git cannot authenticate to GitHub on its own.
+- **Meaning:** Commits on the node would carry the wrong author, would not be signed although `git.signing` is declared, or git cannot authenticate to GitHub on its own.
 - **Confirm:** On the node: `git config --global --get user.name`, and `printf "protocol=https\nhost=github.com\n\n" | GIT_TERMINAL_PROMPT=0 git credential fill >/dev/null`.
 - **Fix:** Declare the `git` block in `fleet.json` and run `devfleet fleet-sync` (`ansible/github/git.yml`). Remove lines in `~/.gitconfig` that override it.
-- **Escalate:** GIT_REMOTE_REFUSED means the fleet account lost access to the declared repository: a person grants it.
+- **Escalate:** GIT_REMOTE_REFUSED means the fleet account lost access to the declared repository: a person grants it. GitHub shows signed commits as verified only once a person adds each node's `~/.ssh/dev-fleet-git-signing.pub` to the account as a signing key.
 
 - `GIT_CONFIGURATION_INCOMPLETE`: git configuration incomplete.
 - `GIT_CREDENTIAL_HELPER_MISSING`: git credential helper missing.
@@ -146,6 +175,9 @@ There are 250 codes; 31 need a person.
 - `GIT_IDENTITY_UNDECLARED`: git identity undeclared.
 - `GIT_REMOTE_REFUSED`: git remote refused. **Needs a person.**
 - `GIT_REMOTE_UNREACHABLE`: git remote unreachable.
+- `GIT_SIGNING_FAILED`: git signing failed: the node's own key is missing or cannot sign without a prompt.
+- `GIT_SIGNING_MISSING`: git signing missing.
+- `GIT_SIGNING_OVERRIDDEN`: git signing overridden.
 
 ## GitHub sign-in
 
@@ -180,12 +212,34 @@ There are 250 codes; 31 need a person.
 - `TAILSCALE_SOURCE_OWNER_CONFLICT`: tailscale source owner conflict. **Needs a person.**
 - `TAILSCALE_UPGRADE_PENDING`: tailscale upgrade pending. **Needs a person.**
 
+## A new worker with no hands on it
+
+- **Meaning:** A machine created without a person (from a pipeline or infrastructure-as-code) could not be enrolled, joined to the tailnet, given its account or brought to ready in one run: its first boot is still running or failed, the tailnet control plane refused a key, the image lets only root in and the account could not be created, or its first-converge or maintenance reboot did not happen.
+- **Confirm:** The `bootstrap:first-boot`, `tailscale:enrolment-key`, `worker:account` and `worker:reboot` checks of the report `devfleet fleet-sync --json` prints, or the `tailscale:device` check of `fleet-node-replace` and `fleet-node-retire` and the `tailscale:leftover` check of `devfleet fleet-status`, which ask the same control plane; on the worker, `cloud-init status --long`.
+- **Fix:** FIRST_BOOT_PENDING and WORKER_REBOOT_BUSY clear on their own: run `devfleet fleet-sync` again later. A TAILNET code names what to fix in `tailnet` of `fleet.json` or in the secret `tailnet-admin`; then run `devfleet fleet-sync`.
+- **Escalate:** A first boot that failed means the image or its user data is wrong: recreate the machine. A root login that cannot create the declared account needs a person on the machine (`useradd --create-home --shell /bin/bash --groups sudo USER`).
+
+- `FIRST_BOOT_FAILED`: first boot failed: cloud-init ended in an error. **Needs a person.**
+- `FIRST_BOOT_PENDING`: first boot pending: cloud-init still runs; retryable.
+- `TAILNET_ADMIN_REQUIRED`: tailnet admin required: fleet.json declares tailnet without the secret. **Needs a person.**
+- `TAILNET_CONTROL_DENIED`: tailnet control denied the tailnet-admin credential. **Needs a person.**
+- `TAILNET_CONTROL_REJECTED`: tailnet control rejected the request (tag or user). **Needs a person.**
+- `TAILNET_CONTROL_RESPONSE_INVALID`: tailnet control response invalid.
+- `TAILNET_CONTROL_UNAVAILABLE`: tailnet control unavailable.
+- `TAILNET_KEY_DELIVERY_FAILED`: tailnet key delivery failed.
+- `TAILNET_KEY_INVALID`: tailnet key invalid.
+- `WORKER_ACCOUNT_CREATION_FAILED`: worker account creation failed. **Needs a person.**
+- `WORKER_ACCOUNT_ROOT_REQUIRED`: worker account root required. **Needs a person.**
+- `WORKER_ACCOUNT_UNDECLARED`: worker account undeclared in workers[].network.user. **Needs a person.**
+- `WORKER_REBOOT_BUSY`: worker reboot busy: an agent turn runs.
+- `WORKER_REBOOT_FAILED`: worker reboot failed.
+
 ## T3 Code
 
 - **Meaning:** The native T3 server or desktop app, its policy, its private route or its providers did not converge or verify.
 - **Confirm:** `devfleet t3-status` and the `t3:*` checks of `devfleet fleet-status`.
 - **Fix:** Run `devfleet fleet-sync`, then `devfleet t3-apply` for providers. A MIGRATION or MAINTENANCE code names a one-time step in its remedy.
-- **Escalate:** Pairing and relinking a peer needs a person in T3 Settings.
+- **Escalate:** A worker's first pairing, and relinking a peer that was not replaced, need a person in T3 Settings (Connections). A replaced worker is paired by `devfleet fleet-node-replace` or the next `devfleet fleet-sync` (`t3:link`).
 
 - `MANAGER_T3_LINK_UNVERIFIED`: manager t3 link unverified.
 - `T3_BASELINE_PREREQUISITES_REQUIRED`: t3 baseline prerequisites required.
@@ -224,6 +278,8 @@ There are 250 codes; 31 need a person.
 - `T3_PROVIDER_NAMES_PENDING`: t3 provider names pending.
 - `T3_PROVIDER_NAMES_UNVERIFIED`: t3 provider names unverified.
 - `T3_RESTART_PENDING`: t3 restart pending.
+- `T3_SERVER_INSTALL_FAILED`: t3 server install failed: the pinned server of a headless worker is missing from the toolchain, or `t3-headless.yml` did not apply.
+- `T3_SERVER_STOPPED`: t3 server stopped: the lingering `t3code-headless.service` of a headless worker is not running.
 - `T3_SETTINGS_BUSY`: t3 settings busy.
 
 ## Monitoring (Beszel)
@@ -295,10 +351,12 @@ There are 250 codes; 31 need a person.
 - **Meaning:** An agent CLI, its profile or a provider behind the CLI proxy did not answer as expected.
 - **Confirm:** The `agent:*` and `provider:*` checks of `devfleet fleet-status`, and `devfleet cliproxy` on the manager.
 - **Fix:** Run `devfleet fleet-sync`; a rate limit or upstream outage clears on its own.
-- **Escalate:** PROVIDER_AUTH_REJECTED needs a person to sign the account in again on the proxy.
+- **Escalate:** PROVIDER_AUTH_REJECTED needs a person to sign the account in again on the proxy. AGENT_PROFILE_RENDER_MISMATCH is a defect of the release (a template of `ansible/agents/agents.yml` no longer renders what `src/agents/render.ts` specifies): report it to dev-fleet. No receipt vouches for what that play wrote, so later applies refuse the file until a person reviews it.
 
 - `AGENT_MEMORY_LIMIT_MISSING`: agent memory limit missing.
 - `AGENT_MODEL_UNSERVED`: agent model unserved.
+- `AGENT_PROFILE_RENDER_MISMATCH`: agent profile render mismatch.
+- `AGENT_PROFILE_REQUEST_INVALID`: agent profile request invalid.
 - `AGENT_PROFILE_STALE`: agent profile stale.
 - `AGENT_SERVICE_ENVIRONMENT_INCOMPLETE`: agent service environment incomplete.
 - `AGENT_SERVICE_ENVIRONMENT_UNVERIFIED`: agent service environment unverified.
@@ -338,7 +396,7 @@ There are 250 codes; 31 need a person.
 ## Vault and credentials
 
 - **Meaning:** A machine cannot read the secrets it needs with its own Vault session.
-- **Confirm:** `vault token lookup` on the machine and the `worker:vault-renewal` check.
+- **Confirm:** `vault token lookup` on the machine and the `worker:vault-renewal` or `manager:vault-renewal` check.
 - **Fix:** Log the machine in once (`devfleet fleet-sync` asks on the manager); renewal is automatic afterwards.
 - **Escalate:** A refused policy needs the Vault administrator.
 
@@ -346,6 +404,8 @@ There are 250 codes; 31 need a person.
 - `VAULT_LOGIN_REQUIRED`: vault login required. **Needs a person.**
 - `VAULT_POLICY_PERMISSION_REQUIRED`: vault policy permission required. **Needs a person.**
 - `VAULT_RENEWAL_FAILING`: vault renewal failing.
+- `VAULT_RENEWAL_MISSING`: vault renewal missing: the manager's renewal job is not loaded. Run `devfleet fleet-sync` on the manager.
+- `VAULT_RENEWAL_NOT_MANAGER`: vault renewal not manager.
 - `VAULT_WORKER_POLICY_FAILED`: vault worker policy failed.
 
 ## The fleet definition and a worker's lifecycle
@@ -361,6 +421,23 @@ There are 250 codes; 31 need a person.
 - `DEFINITION_UNREADABLE`: definition unreadable.
 - `SSH_HOST_KEY_NOT_CHANGED`: ssh host key not changed.
 
+## Parked, idle, replaced and retired workers
+
+- **Meaning:** a worker could not be parked or resumed (agent turns under way, a declared `parking.stop` or `parking.start` command failed, the sync after a resume did not end ready), a worker whose record disappeared could not be retired, or the manager's T3 link to a replaced or retired worker could not be removed, or the manager's T3 could not pair with a replaced worker's new environment.
+- **Confirm:** `devfleet fleet-status --workers` lists each worker as active or parked, with its uptime, agent turns and last agent activity; the report of `devfleet fleet-node-park`, `devfleet fleet-node-resume` or `devfleet fleet-sync` names the step that failed.
+- **Fix:** For WORKER_BUSY wait for the turns to end, or repeat `devfleet fleet-node-park` with `--force`. For a failed hook, run the command `fleet.json` names under `parking` by hand with the worker ID, then repeat. For WORKER_RESUME_FAILED follow the sync report and run `devfleet fleet-sync --node ID`. For T3_LINK_UNLINK_FAILED remove the link in the manager's T3 (Settings, Connections). For T3_LINK_PAIRING_FAILED follow the remedy, which names why the link was refused (Linked environments off in the T3 policy, the manager's T3 not running, a worker T3 that did not mint a code), then run `devfleet fleet-sync --node ID`: it pairs a replaced worker until it is paired.
+- **Escalate:** A record retirement that keeps failing at the same step: follow that step's own runbook.
+
+- `T3_LINK_PAIRING_FAILED`: t3 link pairing failed: the manager's T3 could not pair with a replaced worker's new T3 environment. The pairing code never appears in a report.
+- `T3_LINK_UNLINK_FAILED`: t3 link unlink failed.
+- `WORKER_ACTIVITY_UNREADABLE`: worker activity unreadable. **Needs a person.**
+- `WORKER_AUTO_PARK_REFUSED`: worker auto park refused.
+- `WORKER_BUSY`: worker busy. **Needs a person.**
+- `WORKER_RECORD_RETIREMENT_FAILED`: worker record retirement failed.
+- `WORKER_RESUME_FAILED`: worker resume failed.
+- `WORKER_START_HOOK_FAILED`: worker start hook failed.
+- `WORKER_STOP_HOOK_FAILED`: worker stop hook failed.
+
 ## Fleet operations and the fleet state
 
 - **Meaning:** fleet-sync could not plan, lock, reach a machine, promote a revision or keep the fleet state.
@@ -380,6 +457,7 @@ There are 250 codes; 31 need a person.
 - `FLEET_REMOTE_LOCKED`: fleet remote locked.
 - `FLEET_REMOTE_OPERATION_FAILED`: fleet remote operation failed.
 - `FLEET_REVISION_REQUIRED`: fleet revision required.
+- `FLEET_ROLLBACK_UNAVAILABLE`: fleet rollback unavailable.
 - `FLEET_SCHEDULE_PREVIEW`: fleet schedule preview.
 - `FLEET_SOURCE_UNAVAILABLE`: fleet source unavailable.
 - `FLEET_STATE_DIVERGED`: fleet state diverged. **Needs a person.**
@@ -394,6 +472,19 @@ There are 250 codes; 31 need a person.
 - `REMOTE_CONFIGURATION_INCOMPLETE`: remote configuration incomplete.
 - `REMOTE_PHASE_FAILED`: remote phase failed.
 
+## The manager's own policy and scheduled jobs
+
+- **Meaning:** `ansible/macos/manager.yml` did not pass its check-mode run or its apply, or something it declares (network time, the fleet-watch job, the rotation of the scheduled jobs' logs) is not in place.
+- **Confirm:** The `manager:playbook`, `manager:clock`, `manager:watch` and `manager:job-logs` checks of `devfleet fleet-sync --dry-run`; on the manager, the check-mode run the remedy quotes.
+- **Fix:** Run `devfleet fleet-sync`, which applies the playbook in check mode first, then for real. What needs root (power settings, reading network time, newsyslog) comes from the one grant: `ansible-playbook -e user=$(id -un) -K ansible/macos/manager-privilege.yml`, run again after an upgrade that changed it.
+- **Escalate:** The grant needs an administrator's password, once.
+
+- `MANAGER_CLOCK_UNOBSERVED`: manager clock unobserved: the grant that lets fleet-sync read network time is missing. **Needs a person.**
+- `MANAGER_JOB_LOG_ROTATION_MISSING`: manager job log rotation missing: the newsyslog drop-in the grant installs is missing or outdated. **Needs a person.**
+- `MANAGER_PLAYBOOK_CHECK_FAILED`: manager playbook check failed: nothing was applied.
+- `MANAGER_PLAYBOOK_FAILED`: manager playbook failed.
+- `MANAGER_WATCH_INTERVAL_INVALID`: manager watch interval invalid: `manager.schedule.watchMinutes` is outside 5 to 1440.
+
 ## Notifications (fleet-watch)
 
 - **Meaning:** fleet-watch could not reach a notification target `notify` in `fleet.json` names, or its address is not stored as a secret.

package/docs/state.md

@@ -4,12 +4,13 @@ A fleet is described in two places you own. Your project in Git holds
 `fleet.json` and pins the dev-fleet version. Everything else the manager
 cannot rebuild from Git is the **fleet state**:
 
-| Document     | What it is                                                            |
-| ------------ | --------------------------------------------------------------------- |
-| `roster`     | The enrolled connection to each worker (SSH alias, Tailscale device)  |
-| `enrollment` | The revision `fleet-sync` approved and the one it last applied        |
-| `definition` | The fleet configuration `fleet-sync` last applied: your `fleet.json`  |
-| `history`    | The last 90 days of `fleet-sync` runs (at most 500), and the schedule |
+| Document     | What it is                                                                                                                                                                    |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `roster`     | The enrolled connection to each worker (SSH alias, Tailscale device)                                                                                                          |
+| `enrollment` | The revision `fleet-sync` approved and the one it last applied                                                                                                                |
+| `definition` | The fleet configuration `fleet-sync` last applied: your `fleet.json`                                                                                                          |
+| `history`    | The last 90 days of `fleet-sync` runs (at most 500), and the schedule                                                                                                         |
+| `workers`    | Per worker: its pinned SSH host key, whether it is parked and since when, whether it came from a worker record, and whether the manager is to pair its T3 after a replacement |
 
 `devfleet fleet-watch` also keeps `watch.json` in the bucket, beside the
 database: the health of each worker when it last looked, and when that was
@@ -195,6 +196,12 @@ state that is replicated differently or not at all:
 | Monitoring hub (Beszel) database                     | `~/Library/Application Support/personal-beszel`          | `fleet-sync` recreates accounts and systems; metric history, alerts and settings made by hand are lost |
 | T3 ownership record                                  | `~/.t3/userdata/personal-managed.json`                   | `t3-apply --import ID` adopts instances again                                                          |
 
+The manager reaches a worker that has no SSH alias against
+`~/.local/share/dev-fleet/ssh/known_hosts` (beside the state directory): a
+cache rendered from the host keys the `workers` document pins and the worker
+records declare, rewritten before every session. Delete it and the next
+command writes it again; nothing is ever learned into it.
+
 Everything else under `~/.local/share/personal/fleet` and
 `~/.local/state/personal/fleet` (the copy of the source repository, run
 checkouts, reports, receipts, logs) is a cache that the next `fleet-sync`
@@ -257,9 +264,14 @@ mise exec litestream -- litestream restore -no-expand-env \
 ```
 
 `devfleet fleet-state --restore --timestamp 2026-11-01T00:00:00Z --yes` does
-the same and then uploads that version as the bucket's current one. It is the
-remedy for a `corrupt` replica: the bucket answers, but its latest version
-cannot be read.
+the same and then uploads that version as the bucket's current one, which
+takes the manager lease. It is the remedy for a `corrupt` replica: the bucket
+answers, but its latest version cannot be read. `--yes` consents to replacing
+the bucket, not to replacing another manager: while one holds the lease in the
+bucket's current version, the restore stops with `FLEET_MANAGER_LEASE_HELD`
+before it changes anything, and `--take-over` overrides it. When the current
+version cannot be read, its lease cannot either, and the lease in this
+machine's copy (the last one it saw) decides instead.
 
 Litestream is pinned in `mise.toml` and installed the first time it is used.
 The manager is the only machine that runs it.
@@ -274,8 +286,8 @@ writes anything and stops with `FLEET_MANAGER_LEASE_HELD`. `fleet-state
 --init` still restores the state on such a machine; it only writes nothing.
 
 The lease runs for seven days after the last change. After that, any machine
-may take it. Before that, `--take-over` (on `fleet-sync`, `fleet-state --init`
-or `fleet-state --import`) makes this machine the manager; use it only once
+may take it. Before that, `--take-over` (on `fleet-sync`, `fleet-state --init`,
+`fleet-state --import` or `fleet-state --restore --timestamp`) makes this machine the manager; use it only once
 the other one is gone ([replacing the manager](manager-recovery.md)).
 
 A machine is named in the lease by a digest of its machine ID

package/docs/support-matrix.md

@@ -13,11 +13,12 @@ rest. The default is one macOS manager and Ubuntu workers treated as cattle.
 | -------------- | --------- | ------------------- | --------------- | ----------- | ----------------------------------------------------------------------------------------- |
 | manager        | macOS     | 14 or newer         | arm64, x86_64   | supported   | ansible/macos/policy.json                                                                 |
 | worker         | Ubuntu    | 26.04 (resolute)    | amd64, arm64    | supported   | ansible/ubuntu/policy.json; a machine of its own, physical or virtual                     |
+| worker         | Ubuntu    | 24.04 (noble)       | amd64, arm64    | supported   | ansible/ubuntu/policy.json; a machine of its own, physical or virtual                     |
 | manager        | any other | any                 | any             | refused     | The fleet configuration accepts only a macOS manager.                                     |
 | worker         | Ubuntu    | any other release   | any             | refused     | UBUNTU_BASELINE_MISMATCH                                                                  |
 | worker         | Ubuntu    | an accepted release | any other       | refused     | UBUNTU_BASELINE_MISMATCH                                                                  |
-| worker         | Debian    | any                 | any             | refused     | UBUNTU_BASELINE_MISMATCH                                                                  |
-| worker         | macOS     | any                 | any             | refused     | The fleet configuration accepts only Linux workers.                                       |
+| worker         | Debian    | any                 | any             | refused     | UBUNTU_BASELINE_MISMATCH; not yet supported (see below).                                  |
+| worker         | macOS     | any                 | any             | refused     | The fleet configuration accepts only Linux workers; not yet supported (see below).        |
 | worker         | Windows   | any                 | any             | refused     | The fleet configuration accepts only Linux workers.                                       |
 | worker         | WSL2      | any                 | any             | refused     | WORKER_ENVIRONMENT_UNSUPPORTED                                                            |
 | worker         | container | any                 | any             | refused     | WORKER_ENVIRONMENT_UNSUPPORTED                                                            |
@@ -32,5 +33,40 @@ refuses a worker that is not Linux, the platform probe refuses another
 release, CPU or environment (`ubuntu:release`, `platform:environment`), and
 the bootstrap and the Ubuntu baseline refuse them again on the machine.
 
-Supporting another platform means adding it to the policy that names it, with
-its package pins, a fresh-box job and its row here.
+## Adding a platform or a release
+
+What differs from one worker platform to another is named by the
+`WorkerPlatform` interface (`src/platform/worker-platform.ts`): its policy and
+playbooks, its package manager and pins, its Tailscale key, its service
+manager, its session model and its host checks. Ubuntu is the one
+implementation (`src/platform/ubuntu-worker.ts`).
+
+Another Ubuntu release is one entry in `releases` of
+`ansible/ubuntu/policy.json`: its version, codename and architectures, its
+package and browser library pins read from the snapshot's Packages indexes
+for that release (`packages`, `browserPackages`), and `tailscaleKeySha256`
+only when Tailscale signs its suite with another key than the common one.
+`ansible/ubuntu/tasks/release.yml` selects the entry from the facts, so the
+playbooks need no change of their own. The release is supported once the
+fresh-box features pass on it: CI runs them for every accepted release
+(`DEVFLEET_TEST_FRESH_BOX_RELEASE`), with its image pinned in
+`features/fresh-box/Dockerfile`.
+
+## Not yet supported
+
+These are refused today. What each would need, beyond a row here:
+
+- **Debian workers.** A `WorkerPlatform` of its own with a Debian policy:
+  pins read from snapshot.debian.org for each release, Tailscale's key digest
+  for the Debian suite, the platform probe accepting `ID=debian`, and the
+  bootstrap and `tasks/release.yml` accepting it in place of
+  `UBUNTU_BASELINE_MISMATCH`. The playbooks assume Ubuntu's archive keyring,
+  snapshot service and unattended-upgrades origins, and the GNOME desktop
+  T3 runs in; each needs a Debian equivalent. Then a fresh-box image and job.
+- **macOS workers.** Everything a worker platform names differs: launchd in
+  place of systemd as the service manager, a logged-in user session in place
+  of a lingering systemd user manager, Homebrew or pinned downloads in place
+  of APT, power and sleep policy through `pmset`, and host checks of their
+  own. The fleet configuration would accept `macOS` for a worker, and the
+  convergence would run playbooks under `ansible/macos` for it. No container
+  can stand in for a fresh macOS machine, so the test would be a macOS runner.

package/docs/topology.md

@@ -1,6 +1,7 @@
 > This document was written for the fleet dev-fleet was extracted from. Paths and
 > commands were updated; some statements still describe that fleet and are being
-> revised. The README is authoritative.
+> revised. The README is authoritative, and what a machine must offer to be a
+> worker is listed only in [worker requirements](worker-requirements.md).
 
 # Fleet topology: one manager, many workers
 
@@ -8,10 +9,10 @@ The fleet has exactly two roles, declared in
 your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.schema.json)) and validated by
 [`src/definition.ts`](../src/definition.ts):
 
-| Role        | How many    | Machine                               | Holds state | Runs                                                                                     |
-| ----------- | ----------- | ------------------------------------- | ----------- | ---------------------------------------------------------------------------------------- |
-| **manager** | exactly one | the human's macOS machine (`manager`) | yes         | `fleet-sync`, the T3 desktop that links to every worker, and **every singleton service** |
-| **workers** | any         | Linux boxes (`workers`, Ubuntu 26.04) | no          | the T3 desktop app (UI plus server), agent tools, the Beszel agent; clients only         |
+| Role        | How many    | Machine                                        | Holds state | Runs                                                                                     |
+| ----------- | ----------- | ---------------------------------------------- | ----------- | ---------------------------------------------------------------------------------------- |
+| **manager** | exactly one | the human's macOS machine (`manager`)          | yes         | `fleet-sync`, the T3 desktop that links to every worker, and **every singleton service** |
+| **workers** | any         | Linux boxes (`workers`, Ubuntu 26.04 or 24.04) | no          | the T3 desktop app (UI plus server), agent tools, the Beszel agent; clients only         |
 
 ## Manager
 
@@ -51,7 +52,9 @@ your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.
 
 - Stateless cattle: turn one off, re-image it or add another at will. A worker
   is declared in `workers`, enrolled once (SSH trust, sudo, its own Vault
-  token), and then fully rebuilt by `fleet-sync` from Git and Vault.
+  token), and then fully rebuilt by `fleet-sync` from Git and Vault. What a
+  machine must offer before that is listed in
+  [worker requirements](worker-requirements.md).
 - Workers never host a singleton service. The proxy reports any local install
   on a worker as a duplicate owner, and the hub refuses to install anywhere but
   the manager.
@@ -73,5 +76,63 @@ your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.
 
 - **Add or replace a worker:** add it to `workers`, enroll it with
   `fleet-sync`, done. Remove it by deleting the entry.
+- **Workers an infrastructure tool creates:** `workerRecords` in `fleet.json`
+  names a file (relative to `fleet.json`) or a command that returns worker
+  records, the provider-neutral description of a machine in
+  [`schema/worker-record.schema.json`](../schema/worker-record.schema.json):
+  `id`, `tailnetName` or `address`, `user`, `port`, `hostKey` and optional
+  `labels` and `billing`. Their workers join the inline list; dev-fleet only
+  reads them and never calls a provider or creates a machine. A record that
+  appears is a declared worker, enrolled by the next `fleet-sync`; one that
+  disappears (a file that is gone holds none) is retired by the next
+  `fleet-sync` exactly as `fleet-node-retire` retires one. A worker listed by
+  hand is never retired by itself. An invalid record is refused naming its
+  field, such as `$.workerRecords[0].workers[1].hostKey`.
+  [`examples/workers-from-infrastructure`](../examples/workers-from-infrastructure/main.tf)
+  shows the handoff with a local virtual machine, the seed going in and the
+  record coming out. The canary stays the one `canary` names, so the order
+  of generated records means nothing.
+- **Replace or retire a worker:** `fleet-node-replace` and `fleet-node-retire`
+  take everything of the old machine with it: its pinned host key, its
+  monitoring system (pointed at the new machine, or deleted), the manager's
+  T3 link to its environment, its Vault token and its tailnet device. The
+  manager's T3 pairs with the replacement's environment by itself
+  (`t3:link`): at once when its T3 runs, otherwise in the next `fleet-sync`
+  once it does, and the roster records the new environment. The pairing code
+  is minted on the worker over SSH and handed to the manager's own T3 server
+  in a request body, never on a command line, in a report or in a log. The device is deleted through the
+  control plane fleet.json declares under `tailnet`, asked with the secret
+  `tailnet-admin` (workers are not registered as ephemeral devices); without
+  either, `tailscale:device` is skipped and names the device to remove in the
+  admin console. `fleet-status --workers` warns about a monitoring system that
+  matches no declared worker (`beszel:leftover`), and `fleet-status` about a
+  device carrying the worker tag that is no declared worker, parked or not,
+  and no device the roster binds (`tailscale:leftover`).
+
+## Parked and idle workers
+
+A rented worker costs money while it runs. `devfleet fleet-node-park --id ID`
+records in the fleet state that it is stopped or destroyed on purpose: from
+then on `fleet-sync` and `fleet-status` list it as parked with the time since
+(`worker:parked`, not applicable), never contact it, never make it the
+canary, and it never changes an exit code. Parking refuses while agent turns
+run on it (`WORKER_BUSY`) unless `--force`. `devfleet fleet-node-resume --id
+ID` marks it active and runs `fleet-sync --node ID`, which converges and
+verifies it. `parking.stop` and `parking.start` in `fleet.json` name commands
+(an infrastructure tool, never a provider dev-fleet calls) run with the worker
+ID as their last argument. A parked worker found online on the tailnet is a
+warning (`worker:parked-online`); one that comes back as a new machine goes
+through `fleet-node-replace`.
+
+`fleet-status` reports, for every worker it reaches, its uptime, the agent
+turns under way and the time of the last agent activity, read from what T3
+records (status and times, never a session's content), in its text and as
+`workers` in its JSON. With `parking.idleAfterHours`, a worker with no agent
+activity for longer is `worker:idle` (a warning whose remedy is to park it);
+one with a turn under way never is. `billing.hourlyRate` on a worker (or its
+record) is only displayed and summed: `usage` says how many workers run, how
+many are idle and what the idle ones cost per day. `parking.autoPark: true`
+lets `fleet-watch` park idle workers by itself; it is off unless set.
+
 - **Replace the manager:** follow
   [replacing the manager](manager-recovery.md).