@crvouga/dev-fleet 0.3.0 → 0.5.0

package/docs/operations.md

@@ -4,9 +4,9 @@
 
 # Developer fleet
 
-`@pkgs/fleet` diagnoses development computers: SSH, VS Code Remote-SSH,
+`devfleet` diagnoses development computers: SSH, VS Code Remote-SSH,
 per-machine authentication, T3 Connect and shared toolchain/dependency drift.
-It is a local CLI/library, not a daemon or cloud manager. It reuses the
+It is a local CLI, not a library, a daemon or a cloud manager. It reuses the
 existing mise/workstation owners rather than introducing another installer.
 
 The fleet is one singleton **manager** (the human's macOS machine, which runs
@@ -46,8 +46,9 @@ devfleet fleet-repair --node dev-linux --repair workspace --json   # plan only
 devfleet fleet-repair --node dev-linux --repair workspace --apply --yes --json
 ```
 
-This private package exposes a TypeScript API from `@pkgs/fleet`; the public
-executable is `personal` (`bun run personal` from the checkout). Schema, roster
+dev-fleet is a command with no library API: run it as `npx devfleet` from the
+project that holds your `fleet.json` (`./devfleet` from a checkout of dev-fleet
+itself). Schema, roster
 listing, onboarding and repair previews do not contact nodes or auth services.
 Status never opens a login flow.
 
@@ -61,7 +62,7 @@ remains the sole cloud resource owner; fleet creates no networks, hosts or secre
 
 ```json
 {
-  "$schema": "./packages/fleet/schema.json",
+  "$schema": "./node_modules/@crvouga/dev-fleet/schema/roster.schema.json",
   "version": 1,
   "nodes": [
     {
@@ -97,21 +98,50 @@ remote nodes' required lists. Run `fleet-nodes --schema` for editor/agent valida
 
 ## Evidence and diagnosis
 
-| Check                                                                              | Evidence / next action                                                                                                                                |
-| ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `workspace`, `git:clean`                                                           | HEAD, input availability and dirty-tree status; no paths/diffs/file contents logged.                                                                  |
-| `revision`, `sync:*`                                                               | Compare exact HEAD and SHA-256 of package.json, bun.lock, mise.toml, mise.lock against manager. Missing inputs never pass.                            |
-| `dependencies`                                                                     | Last successful fleet frozen-sync receipt matches input hashes. Missing receipt is unverified, not in-sync.                                           |
-| `auth:toolchain`                                                                   | Existing bootstrap/doctor.ts runs in the node checkout with its Bun, bounded to 90s. Use personal workstation tools check --json for individual pins. |
-| `ssh:agent`                                                                        | Manager agent exposes an identity; missing agent warns because IdentityFile keys may still work.                                                      |
-| `ssh:session`                                                                      | Fresh batch SSH session actually runs the Node/Bun probe. No reused multiplexed connection or agent forwarding.                                       |
-| `ssh:host-key`, `ssh:dns`, `ssh:auth`, `ssh:network`, `ssh:runtime`, `ssh:timeout` | Classified failure and targeted remedy, without raw logs.                                                                                             |
-| `auth:github`                                                                      | gh auth status --active succeeds on this machine; not proof of Git SSH or every repository permission.                                                |
-| `auth:vault`                                                                       | Server-side vault token lookup succeeds in this context; not proof of access to every secret path.                                                    |
-| `auth:vscode`                                                                      | code CLI finds the local Remote-SSH extension. Other profiles/Cursor/GUI PATH/agents can differ.                                                      |
-| `t3:settings`                                                                      | Connect explicitly enabled and not revoked in t3 status --json; settings are not connectivity.                                                        |
-| `t3:api`                                                                           | Read-only t3 environment list --json succeeds against the local API; no pairing/token minting.                                                        |
-| `t3:end-to-end`, `vscode:forwarding`                                               | SKIP: actual relay/peer authorization and VS Code tunnel/server installation require operator testing.                                                |
+| Check                                                                              | Evidence / next action                                                                                                                                                |
+| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `workspace`, `git:clean`                                                           | HEAD, input availability and dirty-tree status; no paths/diffs/file contents logged.                                                                                  |
+| `revision`, `sync:*`                                                               | Compare exact HEAD and SHA-256 of package.json, bun.lock, mise.toml, mise.lock against manager. Missing inputs never pass.                                            |
+| `dependencies`                                                                     | Last successful fleet frozen-sync receipt matches input hashes. Missing receipt is unverified, not in-sync.                                                           |
+| `auth:toolchain`                                                                   | Existing bootstrap/doctor.ts runs in the node checkout with its Bun, bounded to 90s. Run `bun bootstrap/doctor.ts --json` in the node's checkout for individual pins. |
+| `ssh:agent`                                                                        | Manager agent exposes an identity; missing agent warns because IdentityFile keys may still work.                                                                      |
+| `ssh:session`                                                                      | Fresh batch SSH session actually runs the Node/Bun probe. No reused multiplexed connection or agent forwarding.                                                       |
+| `ssh:host-key`, `ssh:dns`, `ssh:auth`, `ssh:network`, `ssh:runtime`, `ssh:timeout` | Classified failure and targeted remedy, without raw logs.                                                                                                             |
+| `auth:github`                                                                      | gh auth status --active succeeds on this machine; not proof of Git SSH or every repository permission.                                                                |
+| `auth:vault`                                                                       | Server-side vault token lookup succeeds in this context; not proof of access to every secret path.                                                                    |
+| `auth:vscode`                                                                      | code CLI finds the local Remote-SSH extension. Other profiles/Cursor/GUI PATH/agents can differ.                                                                      |
+| `t3:settings`                                                                      | Connect explicitly enabled and not revoked in t3 status --json; settings are not connectivity.                                                                        |
+| `t3:api`                                                                           | Read-only t3 environment list --json succeeds against the local API; no pairing/token minting.                                                                        |
+| `t3:end-to-end`, `vscode:forwarding`                                               | SKIP: actual relay/peer authorization and VS Code tunnel/server installation require operator testing.                                                                |
+
+### Machine health
+
+| Check                        | Evidence / next action                                                                                                                                                                                                                                                                                                                             |
+| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `worker:capacity:inodes`     | Free inodes on the home filesystem, against `inodeWarnFreePercent`/`inodeFailFreePercent` in `ansible/ubuntu/policy.json`. The disk guard and disk collector act on the same floors.                                                                                                                                                               |
+| `worker:clock`               | `timedatectl` says the clock is synchronised and a time service is enabled; with a reference time, the offset is held to `clock` in the policy.                                                                                                                                                                                                    |
+| `worker:reboot-pending`      | `/var/run/reboot-required` exists: warns with its age, fails after `rebootPendingFailDays`. Reboot when no agent turn is running.                                                                                                                                                                                                                  |
+| `tailscale:key-expiry`       | The node key expiry Tailscale reports: warns 14 days ahead, fails 3 days ahead. See [Tailscale key expiry](#tailscale-key-expiry).                                                                                                                                                                                                                 |
+| `manager:release`            | `sw_vers` against `minimumMajorVersion` in `ansible/macos/policy.json`.                                                                                                                                                                                                                                                                            |
+| `manager:updates`            | `softwareupdate --list --no-scan` (nothing is downloaded): warns on pending updates, and names one that waits for a restart. Names never shown.                                                                                                                                                                                                    |
+| `manager:disk`               | Free space on the data volume and the space Docker holds, against `disk` in `ansible/macos/policy.json`.                                                                                                                                                                                                                                           |
+| `manager:container-watchdog` | The container watchdog (`ansible/monitoring/container-watchdog.py`, a launch agent `hub-darwin.yml` installs) restarts a labelled container that was unhealthy for `unhealthyChecks` runs, at most `maxRestartsPerHour` times an hour (`containerWatchdog` in `ansible/macos/policy.json`). Warns after a restart; fails once it stops restarting. |
+
+### Tailscale key expiry
+
+A node drops off the tailnet on the day its key expires. Keep workers from
+expiring: tag them, or disable key expiry for each in the Tailscale admin
+console (tagged nodes do not expire). `tailscale:key-expiry` warns two weeks
+ahead and fails three days ahead.
+
+To re-join a worker whose key expired, a person signs it in again, because
+only a person's identity can authorise a device:
+
+1. On the worker's console, run `sudo tailscale up` and open the link it
+   prints, signed in to the fleet's tailnet.
+2. In the admin console, disable key expiry for the worker, or tag it.
+3. From the manager, run `devfleet fleet-sync`: `ansible/ubuntu/tailscale.yml`
+   verifies the login and `tailscale:key-expiry` passes.
 
 An SSH shell is not proof that VS Code or T3 Connect works. Diagnostics check
 **directed manager → node** connectivity, not all pairs. Run from each intended
@@ -147,7 +177,7 @@ need normal repository validation. A green install is not a healthy fleet.
 5. Authenticate separately: GitHub, Git SSH, Vault, coding providers and T3 pairing
    are distinct sessions. Use gh auth login/setup-git, approved vault login and
    operator-driven t3 up --connect as needed. Never put secrets in reports.
-   Workstation/provider configuration remains an explicit `personal workstation sync` operation.
+   Workstation and provider configuration are applied by `devfleet fleet-sync`.
 6. In VS Code's local profile install Remote-SSH, select the same SSH binary,
    alias/config, enable its login terminal temporarily and inspect Output →
    Remote - SSH. Restart after PATH/agent changes. Check disk space, remote
@@ -178,7 +208,7 @@ serially; diagnostics run at most four remote probes concurrently.
   setup can leave partial installs; no rollback is claimed. Trusted checkout
   package lifecycle scripts can execute during installation.
 - No other application repos, credentials, provider settings or cloud resources
-  are synchronized. `personal workstation update`/`personal workstation sync` remain explicit broader convergence.
+  are synchronized. `devfleet fleet-sync` remains the broader convergence.
 - Run doctor again afterward; successful sync does not prove connectivity/auth.
 
 Subprocesses have deadlines and a combined 1 MiB output cap. SSH probes default
@@ -204,17 +234,43 @@ not a way to certify the whole fleet from one shell. Usage errors use
 exception strings. Fleet never prompts; --non-interactive is accepted.
 
 No telemetry or report file is written automatically. Reports still contain
-aliases/hashes: review before sharing publicly. For monitoring, schedule read-only
-`devfleet fleet-status --json` in your existing task runner and alert on exit 1/2;
-do not schedule automatic sync/repairs. `bun run health` remains the separate
+aliases/hashes: review before sharing publicly. Do not schedule automatic
+sync/repairs.
+
+## Notifications and the heartbeat
+
+`devfleet fleet-watch` runs `fleet-status --workers`, compares the result with
+the previous run and notifies only when a worker changes between ready and
+degraded, and again every `repeatHours` (default 24) while it stays degraded.
+Name the targets in `fleet.json`; the address of each is a secret, read through
+the secret provider like any other:
+
+```json
+"notify": { "targets": ["webhook", "ntfy"], "repeatHours": 24 }
+```
+
+| Target    | Secret               | What is sent                                                         |
+| --------- | -------------------- | -------------------------------------------------------------------- |
+| `webhook` | `notify-webhook-url` | A JSON POST: `source`, `event`, `node`, `checks` (check IDs), `text` |
+| `ntfy`    | `notify-ntfy-url`    | A text POST to the topic URL                                         |
+
+A notification carries node and check IDs only. `devfleet fleet-watch --test`
+sends one test notification to every target.
+
+Every run stores what it saw, and when, as `watch.json` beside the fleet state
+in the bucket. That time is the manager's heartbeat. A watcher on the manager
+notices nothing when the manager itself is off, so call the reusable
+`fleet-heartbeat.yml` workflow from the repository that holds your
+`fleet.json` (its header shows the one `uses:` line): it fails, and GitHub
+notifies you, when the heartbeat is older than `max-age-minutes`. `bun run health` remains the separate
 single read-only **cloud platform** report.
 
 ## Development
 
 ```sh
-bun run --cwd packages/fleet tc
-bun run --cwd packages/fleet lint
-bun run --cwd packages/fleet test
+bun run tc
+bun run lint
+bun test
 ```
 
 Tests inject results/use temporary fixtures; they do not contact nodes, login or
@@ -238,7 +294,7 @@ Explicit `services` requirements retain Railway, Vercel, Cloudflare, AWS and npm
 checks. No raw account/auth output is retained. Version checks keep
 per-tool results, including the expected pinned version when available, and require
 Docker Engine/Compose/Buildx readiness. `workstation` delegates to read-only
-`personal workstation doctor --json --non-interactive`; provider configuration presence is not live
+the checkout's `bootstrap/doctor.ts --json`; provider configuration presence is not live
 authorization. Warnings/skips fail the strict whole-tailnet health coverage gate.
 
 ## Repair planning and shared access
@@ -257,7 +313,7 @@ installation never substitutes for authentication or resource authorization.
 
 The workstation owner reads permitted shared provider keys from each machine's
 existing Vault session. For process-scoped shared secret access, use the existing
-`personal infra-vault run --config dev -- COMMAND` workflow on that machine with its intended
+`vault` session on that machine with its intended
 project/config. Infrastructure, access policy and secret seeding remain owned by
 OpenTofu. No OAuth/session/keychain/SSH credential transfer is implemented.
 
@@ -270,7 +326,7 @@ reviewed configuration; without `--dry-run` it applies. `devfleet fleet-sync
 --revision FULL_SHA` rolls the same reviewed revision out to the enrolled fleet
 after approval. The [setup runbook](../agents/setup.md)
 covers canaries, offline/partial failures, per-node receipts and rollback. Both
-delegate rendering to personal workstation and installation to existing mise/bootstrap.
+delegate installation to mise and `bootstrap/setup.ts`.
 They never reset the developer checkout or transfer credentials. Status delegates
 managed-profile drift evidence to the workstation owner; historical rollout receipts
 alone are not a live readiness assertion.

package/docs/runbooks.md

@@ -0,0 +1,377 @@
+# Runbooks
+
+One entry for every failure code dev-fleet reports: in a check's summary, a
+bootstrap phase or an Ansible assertion. Each family says what its codes
+mean, how to confirm one, the fix (a command or a `fleet-sync` that applies
+the playbook that owns it) and when to escalate. A code whose fix needs a
+person (a login, an approval, a choice) is marked **needs a person**.
+`features/rules/runbooks.feature` fails when a code in the source has no
+entry here, or a `devfleet` command quoted in the documents does not exist.
+
+There are 238 codes; 29 need a person.
+
+## Bootstrap of a new or returning worker
+
+- **Meaning:** The remote bootstrap (`bootstrap/fleet-remote-bootstrap.sh`) stopped before node convergence. It reports the phase and the code, never its output.
+- **Confirm:** Run `devfleet fleet-sync --dry-run` and read the `bootstrap:*` check of the worker; `ssh WORKER` and look under `~/.local/share/personal/fleet`.
+- **Fix:** Follow the remedy the check prints, then run `devfleet fleet-sync`. The bootstrap resumes where it stopped.
+- **Escalate:** A platform or architecture code means the machine must be reinstalled with a release [the support matrix](support-matrix.md) lists, on a machine of its own.
+
+- `BOOTSTRAP_CAPACITY_UNVERIFIED`: bootstrap capacity unverified.
+- `BOOTSTRAP_DISK_CAPACITY_REQUIRED`: bootstrap disk capacity required.
+- `CONFIGURATION_INSTALL_FAILED`: configuration install failed.
+- `FLEET_APPROVED_BOOTSTRAP_UNAVAILABLE`: fleet approved bootstrap unavailable.
+- `FLEET_STATE_PATH_UNSAFE`: fleet state path unsafe.
+- `PLATFORM_UNSUPPORTED`: platform unsupported. **Needs a person.**
+- `REMOTE_BOOTSTRAP_EVIDENCE_INVALID`: remote bootstrap evidence invalid.
+- `REMOTE_BOOTSTRAP_EXIT_FAILED`: remote bootstrap exit failed.
+- `ROOT_DEVELOPMENT_USER_REJECTED`: root development user rejected.
+- `UBUNTU_ARCHITECTURE_UNSUPPORTED`: ubuntu architecture unsupported. **Needs a person.**
+- `WORKER_ENVIRONMENT_UNSUPPORTED`: worker environment unsupported: a container or WSL2. **Needs a person.**
+
+## Ubuntu packages
+
+- **Meaning:** A package transaction could not run or the package database is inconsistent.
+- **Confirm:** `ssh WORKER sudo dpkg --audit` and `sudo apt-get -o Debug::NoLocking=1 check`.
+- **Fix:** Let any running package process finish (it is never killed), free disk under `/var`, then run `devfleet fleet-sync`.
+- **Escalate:** APT_STATE_BROKEN and DPKG_INTERRUPTED need `sudo dpkg --configure -a` by a person once no package process runs.
+
+- `APT_DISK_FULL`: apt disk full.
+- `APT_DNS_UNAVAILABLE`: apt dns unavailable.
+- `APT_LOCK_CONTENDED`: apt lock contended.
+- `APT_REPOSITORY_UNAVAILABLE`: apt repository unavailable.
+- `APT_STATE_BROKEN`: apt state broken. **Needs a person.**
+- `DPKG_INTERRUPTED`: dpkg interrupted. **Needs a person.**
+
+## The fleet-owned checkout
+
+- **Meaning:** The checkout of the revision on the machine holds changes fleet-sync did not make, or its bundle changed in transit.
+- **Confirm:** `ssh WORKER git -C ~/.local/share/personal/fleet/revisions/REVISION status`.
+- **Fix:** Remove that revisions directory and run `devfleet fleet-sync`; it is a cache.
+- **Escalate:** A digest mismatch that repeats points at the SSH route.
+
+- `SOURCE_CACHE_CHANGED`: source cache changed.
+- `SOURCE_CHECKOUT_CONFLICT`: source checkout conflict.
+- `SOURCE_DIGEST_MISMATCH`: source digest mismatch.
+- `SOURCE_STATUS_FAILED`: source status failed.
+
+## The pinned toolchain
+
+- **Meaning:** mise, the pinned tools, the global mise configuration or the frozen dependencies did not converge.
+- **Confirm:** Run `bun bootstrap/doctor.ts --json --worker` in the checkout on the machine, or read the `tool:*` checks of `devfleet fleet-status`.
+- **Fix:** Run `devfleet fleet-sync`; it resumes. TOOLCHAIN_CONFIG_CONFLICT means another tool owns `~/.config/mise/config.toml`: move it away.
+- **Escalate:** A tool that fails to download from its pinned artifact needs a new pin in `mise.toml` and `mise.lock` (an issue).
+
+- `LAUNCHER_INSTALLATION_FAILED`: launcher installation failed.
+- `TOOLCHAIN_AWS_FAILED`: toolchain aws failed.
+- `TOOLCHAIN_CONFIG_CONFLICT`: toolchain config conflict.
+- `TOOLCHAIN_CONVERGENCE_FAILED`: toolchain convergence failed.
+- `TOOLCHAIN_FROZEN_DEPENDENCIES_FAILED`: toolchain frozen dependencies failed.
+- `TOOLCHAIN_MISE_FAILED`: toolchain mise failed.
+- `TOOLCHAIN_PATH_CONFIGURATION_FAILED`: toolchain path configuration failed.
+- `TOOLCHAIN_T3_FAILED`: toolchain t3 failed.
+
+## The Ubuntu baseline and its services
+
+- **Meaning:** The Ansible baseline (`ansible/ubuntu/baseline.yml`) did not apply, or what it declares is not what the machine runs.
+- **Confirm:** Read the `worker:*` and `ubuntu:*` checks of `devfleet fleet-status --workers`; on the worker, `systemctl status SERVICE`.
+- **Fix:** Run `devfleet fleet-sync`, which applies the baseline in full when its effect no longer verifies.
+- **Escalate:** Ownership conflicts (sudoers, SSH, Docker from another distribution) need a person to migrate the other owner first.
+
+- `DISK_FREE_BELOW_FLOOR`: disk free below floor.
+- `DISK_FREE_UNREADABLE`: disk free unreadable.
+- `DOCKER_OWNER_CONFLICT`: docker owner conflict. **Needs a person.**
+- `MEMORY_RESERVE_MISSING`: memory reserve missing.
+- `RESOURCE_CAPACITY_REQUIRED`: resource capacity required.
+- `RESOURCE_GUARD_INACTIVE`: resource guard inactive.
+- `SERVICE_NOT_ENABLED`: service not enabled.
+- `SERVICE_NOT_INSTALLED`: service not installed.
+- `SERVICE_NOT_RUNNING`: service not running.
+- `SERVICE_RESTART_POLICY`: service restart policy.
+- `SSH_UNMANAGED_CONFIG_CONFLICT`: ssh unmanaged config conflict.
+- `SUDOERS_OWNERSHIP_CONFLICT`: sudoers ownership conflict. **Needs a person.**
+- `SUDO_AUTHORIZATION_REQUIRED`: sudo authorization required. **Needs a person.**
+- `SWAP_CAPACITY_REQUIRED`: swap capacity required.
+- `SWAP_MISSING`: swap missing.
+- `TMP_QUOTA_EXHAUSTED`: tmp quota exhausted.
+- `UBUNTU_BASELINE_INCOMPLETE`: ubuntu baseline incomplete.
+- `UBUNTU_BASELINE_MISMATCH`: ubuntu baseline mismatch. **Needs a person.**
+- `UBUNTU_LINGER_REQUIRED`: ubuntu linger required.
+- `UBUNTU_MAINTENANCE_REBOOT_PENDING`: ubuntu maintenance reboot pending.
+- `UBUNTU_SESSION_POLICY_PENDING`: ubuntu session policy pending.
+- `UBUNTU_SLEEP_POLICY_DRIFT`: ubuntu sleep policy drift.
+- `WORKER_RECOVERY_SCRIPT_MISSING`: worker recovery script missing.
+- `WORKER_RECOVERY_SETUP_FAILED`: worker recovery setup failed.
+- `WORKER_RECOVERY_TIMER_INACTIVE`: worker recovery timer inactive.
+
+## Containers on a worker
+
+- **Meaning:** `worker:docker-run` could not start the pinned container or build a one-line image as the worker user.
+- **Confirm:** On the worker, as that user: `docker run --rm --network none --pull never IMAGE true`, with IMAGE from `dockerSmokeTest` in `ansible/ubuntu/policy.json`.
+- **Fix:** Run `devfleet fleet-sync`: it starts the engine, adds the user to the docker group and caches the image. A session started earlier keeps its old groups: restart it.
+- **Escalate:** A cgroup or runtime failure that survives a reboot is an issue.
+
+- `DOCKER_BUILD_FAILED`: docker build failed.
+- `DOCKER_DAEMON_UNREACHABLE`: docker daemon unreachable.
+- `DOCKER_DISK_FULL`: docker disk full.
+- `DOCKER_RUN_FAILED`: docker run failed.
+- `DOCKER_RUN_TIMEOUT`: docker run timeout.
+- `DOCKER_SMOKE_IMAGE_MISSING`: docker smoke image missing.
+- `DOCKER_SOCKET_ACCESS_DENIED`: docker socket access denied.
+
+## Headless browser on a worker
+
+- **Meaning:** `worker:browser` could not launch the pinned headless Chromium of the web tool profile.
+- **Confirm:** On the worker run the headless shell under `/opt/personal-fleet/browser/` with `--headless --no-sandbox --dump-dom about:blank`.
+- **Fix:** Run `devfleet fleet-sync`, which installs the pinned libraries and build (`ansible/ubuntu/tasks/browser.yml`).
+- **Escalate:** A missing library the policy does not list is an issue: the list needs it.
+
+- `BROWSER_LAUNCH_FAILED`: browser launch failed.
+- `BROWSER_LAUNCH_TIMEOUT`: browser launch timeout.
+- `BROWSER_LIBRARY_MISSING`: browser library missing.
+- `BROWSER_NOT_INSTALLED`: browser not installed.
+
+## Git identity and pushing
+
+- **Meaning:** Commits on the node would carry the wrong author, or git cannot authenticate to GitHub on its own.
+- **Confirm:** On the node: `git config --global --get user.name`, and `printf "protocol=https\nhost=github.com\n\n" | GIT_TERMINAL_PROMPT=0 git credential fill >/dev/null`.
+- **Fix:** Declare the `git` block in `fleet.json` and run `devfleet fleet-sync` (`ansible/github/git.yml`). Remove lines in `~/.gitconfig` that override it.
+- **Escalate:** GIT_REMOTE_REFUSED means the fleet account lost access to the declared repository: a person grants it.
+
+- `GIT_CONFIGURATION_INCOMPLETE`: git configuration incomplete.
+- `GIT_CREDENTIAL_HELPER_MISSING`: git credential helper missing.
+- `GIT_IDENTITY_MISMATCH`: git identity mismatch.
+- `GIT_IDENTITY_MISSING`: git identity missing.
+- `GIT_IDENTITY_OVERRIDDEN`: git identity overridden.
+- `GIT_IDENTITY_UNDECLARED`: git identity undeclared.
+- `GIT_REMOTE_REFUSED`: git remote refused. **Needs a person.**
+- `GIT_REMOTE_UNREACHABLE`: git remote unreachable.
+
+## Tailscale
+
+- **Meaning:** A machine is not on the tailnet as the fleet declares it.
+- **Confirm:** `tailscale status` on the machine and `devfleet fleet-discover`.
+- **Fix:** Sign the machine in (`sudo tailscale up` on a worker, the app on the manager), then run `devfleet fleet-sync`.
+- **Escalate:** Logins and device approvals need a person.
+
+- `TAILSCALE_DEVICE_REPLACED`: tailscale device replaced. **Needs a person.**
+- `TAILSCALE_ENROLLMENT_REQUIRED`: tailscale enrollment required. **Needs a person.**
+- `TAILSCALE_IDENTITY_CONFLICT`: tailscale identity conflict. **Needs a person.**
+- `TAILSCALE_INTERACTIVE_AUTHORIZATION_REQUIRED`: tailscale interactive authorization required. **Needs a person.**
+- `TAILSCALE_KEY_EXPIRY_UNOBSERVED`: tailscale key expiry unobserved.
+- `TAILSCALE_LOGIN_REQUIRED`: tailscale login required. **Needs a person.**
+- `TAILSCALE_NODE_MISSING`: tailscale node missing.
+- `TAILSCALE_NODE_OFFLINE`: tailscale node offline.
+- `TAILSCALE_SETUP_OR_LOGIN_REQUIRED`: tailscale setup or login required. **Needs a person.**
+- `TAILSCALE_SOURCE_OWNER_CONFLICT`: tailscale source owner conflict. **Needs a person.**
+- `TAILSCALE_UPGRADE_PENDING`: tailscale upgrade pending. **Needs a person.**
+
+## T3 Code
+
+- **Meaning:** The native T3 server or desktop app, its policy, its private route or its providers did not converge or verify.
+- **Confirm:** `devfleet t3-status` and the `t3:*` checks of `devfleet fleet-status`.
+- **Fix:** Run `devfleet fleet-sync`, then `devfleet t3-apply` for providers. A MIGRATION or MAINTENANCE code names a one-time step in its remedy.
+- **Escalate:** Pairing and relinking a peer needs a person in T3 Settings.
+
+- `MANAGER_T3_LINK_UNVERIFIED`: manager t3 link unverified.
+- `T3_BASELINE_PREREQUISITES_REQUIRED`: t3 baseline prerequisites required.
+- `T3_CACHE_DRIFT`: t3 cache drift.
+- `T3_CHECKSUM_MISMATCH`: t3 checksum mismatch.
+- `T3_DESKTOP_INSTALL_FAILED`: t3 desktop install failed.
+- `T3_DESKTOP_STOPPED`: t3 desktop stopped.
+- `T3_DOWNLOAD_UNAVAILABLE`: t3 download unavailable.
+- `T3_ENVIRONMENT_IDENTITY_CHANGED`: t3 environment identity changed.
+- `T3_EXECUTABLE_FAILED`: t3 executable failed.
+- `T3_EXECUTABLE_OWNER_CONFLICT`: t3 executable owner conflict. **Needs a person.**
+- `T3_EXTRACTION_FAILED`: t3 extraction failed.
+- `T3_INSTALLATION_DRIFT`: t3 installation drift.
+- `T3_MANAGER_ENDPOINT_REJECTED`: t3 manager endpoint rejected.
+- `T3_MANAGER_ENDPOINT_UNREACHABLE`: t3 manager endpoint unreachable.
+- `T3_MANAGER_IDENTITY_UNVERIFIED`: t3 manager identity unverified.
+- `T3_MANAGER_ORIGIN_UNRESTRICTED`: t3 manager origin unrestricted.
+- `T3_MANAGER_ORIGIN_UNVERIFIED`: t3 manager origin unverified.
+- `T3_MODELS_MISMATCH`: t3 models mismatch.
+- `T3_MODELS_UNVERIFIED`: t3 models unverified.
+- `T3_POLICY_OWNER_CONFLICT`: t3 policy owner conflict. **Needs a person.**
+- `T3_POLICY_RESTART_FAILED`: t3 policy restart failed.
+- `T3_POLICY_RESTART_PENDING`: t3 policy restart pending.
+- `T3_POLICY_STAGE_FAILED`: t3 policy stage failed.
+- `T3_PRIVATE_IDENTITY_REQUIRED`: t3 private identity required.
+- `T3_PRIVATE_ROUTE_CONFLICT`: t3 private route conflict.
+- `T3_PRIVATE_ROUTE_REQUIRED`: t3 private route required.
+- `T3_PRIVATE_ROUTE_UNVERIFIED`: t3 private route unverified.
+- `T3_PROFILE_CONFLICT`: t3 profile conflict.
+- `T3_PROFILE_PENDING`: t3 profile pending.
+- `T3_PROFILE_RECORD_INVALID`: t3 profile record invalid.
+- `T3_PROFILE_UNVERIFIED`: t3 profile unverified.
+- `T3_PROVIDERS_PENDING`: t3 providers pending.
+- `T3_PROVIDERS_UNVERIFIED`: t3 providers unverified.
+- `T3_PROVIDER_CONFLICT`: t3 provider conflict.
+- `T3_PROVIDER_NAMES_PENDING`: t3 provider names pending.
+- `T3_PROVIDER_NAMES_UNVERIFIED`: t3 provider names unverified.
+- `T3_RESTART_PENDING`: t3 restart pending.
+- `T3_SETTINGS_BUSY`: t3 settings busy.
+
+## Monitoring (Beszel)
+
+- **Meaning:** The hub on the manager, an agent on a machine, an enrollment or the hub backups do not match what the fleet declares.
+- **Confirm:** The `beszel:*` checks of `devfleet fleet-status --workers`; on the manager `docker compose ls` and the hub at `/_/`.
+- **Fix:** Run `devfleet fleet-sync`, which converges the hub, its accounts, its backups and every agent. Hub data is never deleted.
+- **Escalate:** BESZEL_HUB_UPGRADE_PENDING needs a verified backup and a planned upgrade by a person.
+
+- `BESZEL_ACCOUNT_INVALID`: beszel account invalid.
+- `BESZEL_ACCOUNT_REQUIRED`: beszel account required. **Needs a person.**
+- `BESZEL_AGENT_CONNECTION_PENDING`: beszel agent connection pending.
+- `BESZEL_AGENT_DISCONNECTED`: beszel agent disconnected.
+- `BESZEL_AGENT_KEY_REJECTED`: beszel agent key rejected.
+- `BESZEL_AGENT_LOG_UNAVAILABLE`: beszel agent log unavailable.
+- `BESZEL_AGENT_TOKEN_REJECTED`: beszel agent token rejected.
+- `BESZEL_API_UNAVAILABLE`: beszel api unavailable.
+- `BESZEL_API_UNVERIFIED`: beszel api unverified.
+- `BESZEL_AUTH_REQUIRED`: beszel auth required.
+- `BESZEL_BACKUP_CONVERGENCE_FAILED`: beszel backup convergence failed.
+- `BESZEL_BACKUP_NOT_CONVERGED`: beszel backup not converged.
+- `BESZEL_BACKUP_OFFHOST_REQUIRED`: beszel backup offhost required.
+- `BESZEL_BACKUP_STALE`: beszel backup stale.
+- `BESZEL_BACKUP_UNSCHEDULED`: beszel backup unscheduled.
+- `BESZEL_BACKUP_UNVERIFIED`: beszel backup unverified.
+- `BESZEL_CREDENTIALS_REQUIRED`: beszel credentials required. **Needs a person.**
+- `BESZEL_DISK_FULL`: beszel disk full.
+- `BESZEL_DISK_UNVERIFIED`: beszel disk unverified.
+- `BESZEL_DOCKER_DESKTOP_REQUIRED`: beszel docker desktop required.
+- `BESZEL_ENROLLMENT_INCOMPLETE`: beszel enrollment incomplete.
+- `BESZEL_HOST_CONFLICT`: beszel host conflict.
+- `BESZEL_HUB_INSTALLATION_INCOMPLETE`: beszel hub installation incomplete.
+- `BESZEL_HUB_MISMATCH`: beszel hub mismatch.
+- `BESZEL_HUB_SELECTION_REQUIRED`: beszel hub selection required.
+- `BESZEL_HUB_UNHEALTHY`: beszel hub unhealthy.
+- `BESZEL_HUB_UNREACHABLE`: beszel hub unreachable.
+- `BESZEL_HUB_UPGRADE_PENDING`: beszel hub upgrade pending. **Needs a person.**
+- `BESZEL_INPUT_INVALID`: beszel input invalid.
+- `BESZEL_INPUT_LIMIT`: beszel input limit.
+- `BESZEL_INSTALLATION_INCOMPLETE`: beszel installation incomplete.
+- `BESZEL_LEGACY_DAEMON`: beszel legacy daemon.
+- `BESZEL_METRICS_STALE`: beszel metrics stale.
+- `BESZEL_NODE_INVALID`: beszel node invalid.
+- `BESZEL_NODE_MISSING`: beszel node missing.
+- `BESZEL_NODE_TAILNET_REQUIRED`: beszel node tailnet required.
+- `BESZEL_NOT_ENROLLED`: beszel not enrolled.
+- `BESZEL_NOT_HUB`: beszel not hub.
+- `BESZEL_NOT_MANAGER`: beszel not manager.
+- `BESZEL_OPERATION_INCOMPLETE`: beszel operation incomplete.
+- `BESZEL_PRIVATE_ROUTE_CONFLICT`: beszel private route conflict.
+- `BESZEL_PRIVATE_ROUTE_REQUIRED`: beszel private route required.
+- `BESZEL_PRIVATE_ROUTE_UNVERIFIED`: beszel private route unverified.
+- `BESZEL_PROTOCOL`: beszel protocol.
+- `BESZEL_PUBLIC_ENDPOINT_OFFLINE`: beszel public endpoint offline.
+- `BESZEL_REMOTE_EVIDENCE_INVALID`: beszel remote evidence invalid.
+- `BESZEL_RESPONSE_LIMIT`: beszel response limit.
+- `BESZEL_SERVICE_STOPPED`: beszel service stopped.
+- `BESZEL_SOURCE_MISMATCH`: beszel source mismatch.
+- `BESZEL_TAILSCALE_REQUIRED`: beszel tailscale required.
+- `BESZEL_TOKEN_INVALID`: beszel token invalid.
+- `BESZEL_TUNNEL_CREDENTIAL_UNAVAILABLE`: beszel tunnel credential unavailable. **Needs a person.**
+- `BESZEL_VERSION_MISMATCH`: beszel version mismatch.
+- `BESZEL_VIEWER_ROLE`: beszel viewer role.
+- `BESZEL_VIEWER_UNPROVISIONED`: beszel viewer unprovisioned.
+- `BESZEL_VIEWER_UNVERIFIED`: beszel viewer unverified.
+
+## Coding agents and providers
+
+- **Meaning:** An agent CLI, its profile or a provider behind the CLI proxy did not answer as expected.
+- **Confirm:** The `agent:*` and `provider:*` checks of `devfleet fleet-status`, and `devfleet cliproxy` on the manager.
+- **Fix:** Run `devfleet fleet-sync`; a rate limit or upstream outage clears on its own.
+- **Escalate:** PROVIDER_AUTH_REJECTED needs a person to sign the account in again on the proxy.
+
+- `AGENT_MEMORY_LIMIT_MISSING`: agent memory limit missing.
+- `AGENT_MODEL_UNSERVED`: agent model unserved.
+- `AGENT_PROFILE_STALE`: agent profile stale.
+- `AGENT_SERVICE_ENVIRONMENT_INCOMPLETE`: agent service environment incomplete.
+- `AGENT_SERVICE_ENVIRONMENT_UNVERIFIED`: agent service environment unverified.
+- `AGENT_SESSION_UNVERIFIED`: agent session unverified.
+- `AGENT_VAULT_UNAVAILABLE`: agent vault unavailable.
+- `AGENT_VERSION_MISMATCH`: agent version mismatch.
+- `PROVIDER_AUTH_REJECTED`: provider auth rejected. **Needs a person.**
+- `PROVIDER_EMPTY_RESPONSE`: provider empty response.
+- `PROVIDER_MODEL_UNAVAILABLE`: provider model unavailable.
+- `PROVIDER_NETWORK_UNVERIFIED`: provider network unverified.
+- `PROVIDER_OUTPUT_UNVERIFIED`: provider output unverified.
+- `PROVIDER_PROTOCOL_REJECTED`: provider protocol rejected.
+- `PROVIDER_RATE_LIMITED`: provider rate limited.
+- `PROVIDER_RESPONSE_INVALID`: provider response invalid.
+- `PROVIDER_RESPONSE_LIMIT`: provider response limit.
+- `PROVIDER_ROUTE_UNVERIFIED`: provider route unverified.
+- `PROVIDER_STREAM_INCOMPLETE`: provider stream incomplete.
+- `PROVIDER_TIMEOUT`: provider timeout.
+- `PROVIDER_TOOL_ARGUMENTS_INVALID`: provider tool arguments invalid.
+- `PROVIDER_TOOL_INVALID`: provider tool invalid.
+- `PROVIDER_TOOL_MISSING`: provider tool missing.
+- `PROVIDER_TOOL_RESULT_UNVERIFIED`: provider tool result unverified.
+- `PROVIDER_UNATTENDED_CREDENTIAL_REQUIRED`: provider unattended credential required.
+- `PROVIDER_UPSTREAM_UNAVAILABLE`: provider upstream unavailable.
+- `PROXY_RECONCILE_UNVERIFIED`: proxy reconcile unverified.
+- `UNATTENDED_PROXY_AUTHENTICATION_REQUIRED`: unattended proxy authentication required.
+
+## Provider accounts
+
+- **Meaning:** A change to a provider account on the CLI proxy (sign-in, removal, cooldown) did not complete.
+- **Confirm:** `devfleet fleet-accounts` lists every account and its state; `devfleet fleet-accounts --diff` compares this machine.
+- **Fix:** Retry the change with `devfleet fleet-accounts`; see [Provider accounts](accounts.md).
+- **Escalate:** Signing an account in again needs a person.
+
+- `ACCOUNT_CHANGE_FAILED`: account change failed.
+
+## Vault and credentials
+
+- **Meaning:** A machine cannot read the secrets it needs with its own Vault session.
+- **Confirm:** `vault token lookup` on the machine and the `worker:vault-renewal` check.
+- **Fix:** Log the machine in once (`devfleet fleet-sync` asks on the manager); renewal is automatic afterwards.
+- **Escalate:** A refused policy needs the Vault administrator.
+
+- `UNATTENDED_VAULT_IDENTITY_REQUIRED`: unattended vault identity required.
+- `VAULT_LOGIN_REQUIRED`: vault login required. **Needs a person.**
+- `VAULT_POLICY_PERMISSION_REQUIRED`: vault policy permission required. **Needs a person.**
+- `VAULT_RENEWAL_FAILING`: vault renewal failing.
+
+## Fleet operations and the fleet state
+
+- **Meaning:** fleet-sync could not plan, lock, reach a machine, promote a revision or keep the fleet state.
+- **Confirm:** `devfleet fleet-state` for the state, `devfleet fleet-status` for machines, and the report `devfleet fleet-sync` writes.
+- **Fix:** Follow the remedy; most clear by running `devfleet fleet-sync` again once the holder of a lock finishes.
+- **Escalate:** FLEET_STATE_DIVERGED needs a person to choose which copy of the state to keep (docs/state.md).
+
+- `ANSIBLE_VARIABLE_LIMIT`: ansible variable limit.
+- `CONTAINER_WATCHDOG_UNOBSERVED`: container watchdog unobserved.
+- `FLEET_CONNECTION_REQUIRED`: fleet connection required.
+- `FLEET_MANAGER_ENROLLMENT_REQUIRED`: fleet manager enrollment required. **Needs a person.**
+- `FLEET_MANAGER_LOCKED`: fleet manager locked.
+- `FLEET_MANAGER_UNREACHABLE`: fleet manager unreachable.
+- `FLEET_NOT_MANAGER`: fleet not manager.
+- `FLEET_PLAN_STALE`: fleet plan stale.
+- `FLEET_PROMOTION_BLOCKED`: fleet promotion blocked.
+- `FLEET_REMOTE_LOCKED`: fleet remote locked.
+- `FLEET_REMOTE_OPERATION_FAILED`: fleet remote operation failed.
+- `FLEET_REVISION_REQUIRED`: fleet revision required.
+- `FLEET_SCHEDULE_PREVIEW`: fleet schedule preview.
+- `FLEET_SOURCE_UNAVAILABLE`: fleet source unavailable.
+- `FLEET_STATE_DIVERGED`: fleet state diverged. **Needs a person.**
+- `FLEET_VERIFICATION_MISSING`: fleet verification missing.
+- `MANAGER_AGENT_PROFILES_FAILED`: manager agent profiles failed.
+- `MANAGER_DISK_UNOBSERVED`: manager disk unobserved.
+- `MANAGER_DOCKER_DESKTOP_DRIFT`: manager docker desktop drift.
+- `MANAGER_DOCKER_DESKTOP_UNOBSERVED`: manager docker desktop unobserved.
+- `MANAGER_RELEASE_UNOBSERVED`: manager release unobserved.
+- `MANAGER_SLEEPS_ON_POWER`: manager sleeps on power.
+- `MANAGER_UPDATES_UNOBSERVED`: manager updates unobserved.
+- `REMOTE_CONFIGURATION_INCOMPLETE`: remote configuration incomplete.
+- `REMOTE_PHASE_FAILED`: remote phase failed.
+
+## Notifications (fleet-watch)
+
+- **Meaning:** fleet-watch could not reach a notification target `notify` in `fleet.json` names, or its address is not stored as a secret.
+- **Confirm:** Run `devfleet fleet-watch --test` and read the `notify:*` checks.
+- **Fix:** Store the target's address as the secret the remedy names, or fix the endpoint, then run `devfleet fleet-watch --test` again.
+- **Escalate:** A target that keeps refusing needs whoever runs that endpoint.
+
+- `NOTIFY_DELIVERY_FAILED`: notify delivery failed.
+- `NOTIFY_TARGET_UNAVAILABLE`: notify target unavailable.

package/docs/state.md

@@ -4,13 +4,18 @@ A fleet is described in two places you own. Your project in Git holds
 `fleet.json` and pins the dev-fleet version. Everything else the manager
 cannot rebuild from Git is the **fleet state**:
 
-| Document     | What it is                                                           |
-| ------------ | -------------------------------------------------------------------- |
-| `roster`     | The enrolled connection to each worker (SSH alias, Tailscale device) |
-| `enrollment` | The revision `fleet-sync` approved and the one it last applied       |
-| `definition` | The fleet configuration `fleet-sync` last applied: your `fleet.json` |
-
-They live in one SQLite database, and [Litestream](https://litestream.io)
+| Document     | What it is                                                            |
+| ------------ | --------------------------------------------------------------------- |
+| `roster`     | The enrolled connection to each worker (SSH alias, Tailscale device)  |
+| `enrollment` | The revision `fleet-sync` approved and the one it last applied        |
+| `definition` | The fleet configuration `fleet-sync` last applied: your `fleet.json`  |
+| `history`    | The last 90 days of `fleet-sync` runs (at most 500), and the schedule |
+
+`devfleet fleet-watch` also keeps `watch.json` in the bucket, beside the
+database: the health of each worker when it last looked, and when that was
+(the manager's heartbeat). Losing it costs at most one repeated notification.
+
+The three documents live in one SQLite database, and [Litestream](https://litestream.io)
 replicates that database to a storage bucket you own. The bucket is the source
 of truth. The copy on the manager is disposable: delete it, or replace the
 whole machine, and the next command brings the fleet back.
@@ -118,6 +123,7 @@ way, needs no credentials and is the quickest way to try this out.
 ```sh
 devfleet fleet-state            # read-only: where the state is, and how it stands with the bucket
 devfleet fleet-state --example  # print a fleet.json to start a project from
+devfleet fleet-report --since 30d  # read-only: how reliably fleet-sync ran, from the history
 devfleet fleet-state --export   # print the configuration fleet-sync last applied
 devfleet fleet-state --push     # upload changes made while the bucket was unreachable
 devfleet fleet-state --restore  # replace the local copy with the bucket's