@crvouga/dev-fleet 0.1.0 → 0.3.0

package/docs/consumer-setup.md

@@ -0,0 +1,408 @@
+# Set up a fleet from your own project
+
+This is the whole procedure for connecting dev-fleet to your own machines,
+written so a coding agent can do it in one pass from the project that will hold
+the fleet configuration. It assumes nothing about your accounts, hosts or
+domains: every value below in `UPPER_CASE` or under `example.com` is a
+placeholder you replace.
+
+If you are an agent, read the [rules](#rules-for-agents) first, then follow
+the steps in order. Stop and ask the human only at the points marked **Ask**.
+
+## What you end up with
+
+```
+your-fleet/                 any Git repository you own (it may hold other things too)
+  package.json              pins @crvouga/dev-fleet to one exact version
+  package-lock.json         or bun.lock, pnpm-lock.yaml: commit it
+  fleet.json                your machines, services, secrets locations, state bucket
+  .gitignore                keeps credentials and node_modules out of Git
+  AGENTS.md                 tells future agents how to use dev-fleet here
+```
+
+Nothing else is needed. You never clone dev-fleet, copy its files or write a
+wrapper script around a checkout of it: the npm package is the tool, and the
+version in `package.json` is the commit every machine runs.
+
+| Concept     | Meaning                                                                                                |
+| ----------- | ------------------------------------------------------------------------------------------------------ |
+| manager     | The one macOS machine you run `devfleet fleet-sync` on. Hosts the CLI proxy and the monitoring hub.    |
+| worker      | An Ubuntu 26.04 machine the manager converges over SSH. Stateless: rebuilt from Git and your secrets.  |
+| fleet state | One SQLite database (roster, approved revision, configuration last applied) replicated to your bucket. |
+| bucket      | Any S3-compatible bucket you own, or a directory. The source of truth for the fleet state.             |
+| revision    | The dev-fleet commit the installed package was published from (`npx devfleet --version`).              |
+
+## Rules for agents
+
+- **Do not clone, vendor or patch dev-fleet.** Not `node_modules`, not a
+  checkout `fleet-sync` puts on a machine, not a file it manages there. The
+  next install or sync replaces them. A bug or a missing feature is an issue
+  at <https://github.com/crvouga/dev-fleet/issues>; tell the human what is
+  blocked and link it. Issues are public: no credentials, hostnames,
+  accounts, domains or `fleet.json` contents.
+- **Never put a secret in an argument, in `fleet.json`, in a commit or in a
+  log.** Bucket credentials are read only from the environment; every other
+  secret from the provider named under `secrets`.
+- **Do not bypass SSH host keys, forward SSH agents or copy login caches**
+  between machines. Host trust, sudo and sign-ins are the human's steps.
+- **Previews are safe; applies are not.** `--dry-run`, `fleet-status`,
+  `fleet-state` (without flags) and `commands --json` change nothing.
+  `fleet-sync` without `--dry-run` changes every machine.
+- **Use the command, never the code.** dev-fleet has no library API: do not
+  import anything from `node_modules/@crvouga/dev-fleet/`. The project's own
+  tools go in the project's `mise.toml`; the global mise configuration on a
+  fleet machine is dev-fleet's.
+- **Use JSON.** Every diagnostic command takes `--json` and
+  `--non-interactive`. Read the check IDs and `remedy.action` from the
+  output; do not parse the human table.
+- The machine-readable contract (commands, exit codes, safety rules, how to
+  report a bug) is `npx devfleet commands --json`. Read it before running
+  anything else.
+
+## 1. Check what the human must already have
+
+These cannot be installed or decided by dev-fleet. Verify each one; for any
+that fails, **Ask** the human to do it and wait.
+
+On the manager (the Mac this project is checked out on):
+
+| Requirement                                              | Verify                                                          |
+| -------------------------------------------------------- | --------------------------------------------------------------- |
+| macOS 14 or newer                                        | `sw_vers -productVersion`                                       |
+| Git                                                      | `git --version`                                                 |
+| Node 22 or newer (for `npx`)                             | `node --version`                                                |
+| Docker Desktop running (the proxy and the hub run in it) | `docker info --format '{{.ServerVersion}}'`                     |
+| Tailscale signed in                                      | `tailscale status --json` (look for `"BackendState":"Running"`) |
+| An SSH alias per worker that works without a prompt      | `ssh -o BatchMode=yes WORKER_ALIAS true` for each worker        |
+
+On every worker:
+
+| Requirement                          | Verify, from the manager                                           |
+| ------------------------------------ | ------------------------------------------------------------------ |
+| Ubuntu 26.04, amd64 or arm64         | `ssh WORKER_ALIAS '. /etc/os-release; echo $VERSION_ID; uname -m'` |
+| A non-root account that can use sudo | `ssh WORKER_ALIAS 'id -u; sudo -n true && echo sudo-ok'`           |
+| Joined to the same tailnet           | the worker appears in `tailscale status` on the manager            |
+
+A worker needs nothing else beforehand: no Git, curl, Python, Node, Bun or
+mise. `fleet-sync` installs them. If `sudo -n true` fails, that is fine for
+the first, interactive run, which asks for the password once and installs a
+sudo rule for the fleet; it is not fine for unattended runs.
+
+The `ssh` value of a worker in `fleet.json` is an alias from the manager's
+`~/.ssh/config`. dev-fleet always connects with `BatchMode=yes` and
+`StrictHostKeyChecking=yes`, so the host key must already be in
+`known_hosts`. **Ask** the human to verify each fingerprint out of band and
+connect once by hand; never add `StrictHostKeyChecking=no` or
+`accept-new`. A typical alias:
+
+```
+Host worker-one
+  HostName worker-one.tail0a1b2c.ts.net
+  User dev
+```
+
+Also **Ask** for these, which only the human has:
+
+1. **A bucket for the fleet state.** An empty private bucket on any
+   S3-compatible service (AWS S3, Cloudflare R2, Backblaze B2, MinIO) and an
+   access key that can read and write it: bucket name, endpoint (omit for AWS
+   S3), region (`auto` for R2). For a trial, an absolute directory path works
+   instead and needs no credentials.
+2. **Where secrets are kept**, one of `vault`, `sops`, `file` or `env` (see
+   [secrets](#secrets)). With `vault`, the address, the KV v2 mount and the
+   path and field of each secret.
+3. **The public origins** of the CLI proxy and of the monitoring dashboard,
+   and the two monitoring account emails (`services` below). They are
+   placeholders until the human has the domains.
+4. **The GitHub login** every worker signs in as, and the Vault user name.
+
+## 2. Create the project
+
+```sh
+mkdir -p your-fleet && cd your-fleet   # or an existing repository
+git init                               # if it is not one yet
+npm init -y                            # if there is no package.json yet
+npm install --save-dev --save-exact @crvouga/dev-fleet
+npx devfleet --version                 # 1.2.3 (commit 3f2a…): the revision every machine will run
+```
+
+Pin the exact version (`--save-exact`); a range would let a reinstall move the
+whole fleet. Any package manager works (`bun add -d --exact`,
+`pnpm add -D --save-exact`). The CLI runs on Bun, which the package brings as
+a dependency; if your package manager skips install scripts, allow Bun's or
+install Bun yourself.
+
+Add to `.gitignore`:
+
+```gitignore
+node_modules/
+.env
+.env.*
+.fleet.local.json
+```
+
+## 3. Write `fleet.json`
+
+Start from the example the installed package prints, which points `$schema`
+at the schema shipped in the package, so editors validate the file:
+
+```sh
+npx devfleet fleet-state --example > fleet.json
+```
+
+Then replace every placeholder. Each field, from
+[`schema/fleet.schema.json`](../schema/fleet.schema.json):
+
+| Field                                                 | Value                                                                                                              |
+| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
+| `version`                                             | `3`                                                                                                                |
+| `source.repository`                                   | `https://github.com/crvouga/dev-fleet.git`, or your fork if you publish one. Machines only accept code from it.    |
+| `state`                                               | `{ "bucket", "endpoint"?, "region"?, "prefix"? }` or `{ "directory": "/absolute/path" }`. Never credentials.       |
+| `secrets`                                             | See [secrets](#secrets).                                                                                           |
+| `manager.id`                                          | A short lowercase id, `^[a-z][a-z0-9-]{0,62}$`, e.g. `manager-mac`.                                                |
+| `manager.os`                                          | `"macOS"`                                                                                                          |
+| `manager.network.tailscaleName`                       | The manager's MagicDNS name: `tailscale status --json` → `.Self.DNSName` without the trailing dot.                 |
+| `manager.network.user`                                | The manager's login name (`id -un`).                                                                               |
+| `manager.neverSleepOnPower`                           | `true` (workers depend on the manager's services).                                                                 |
+| `workers[].id`                                        | A short lowercase id. The **first** worker is the canary: a new revision must pass there before the others get it. |
+| `workers[].ssh`                                       | The worker's alias in the manager's `~/.ssh/config`.                                                               |
+| `workers[].os`                                        | `"linux"`                                                                                                          |
+| `workers[].network.tailscaleName`                     | The worker's MagicDNS name (from `tailscale status --json` → `.Peer[].DNSName` on the manager).                    |
+| `workers[].network.sshPort`                           | `22` unless the worker's sshd listens elsewhere.                                                                   |
+| `workers[].network.user`                              | The worker account (`ssh WORKER_ALIAS id -un`).                                                                    |
+| `workers[].network.t3Port`                            | `3773`                                                                                                             |
+| `services.proxy.endpoint`                             | The public HTTPS origin of the CLI proxy, e.g. `https://cli-proxy.example.com`.                                    |
+| `services.monitoring.hubPort`                         | `8443`                                                                                                             |
+| `services.monitoring.publicUrl`                       | The public HTTPS origin of the monitoring dashboard.                                                               |
+| `services.monitoring.managerAccount`                  | The email of the monitoring account the manager enrolls machines with.                                             |
+| `services.monitoring.viewerAccount`                   | The email of the read-only account visitors are signed in as.                                                      |
+| `ubuntu`                                              | `{ "release": "26.04", "architectures": ["amd64", "arm64"] }` (fixed).                                             |
+| `worker.requiredCapabilities`                         | Normally all of `github`, `vault`, `t3`, `toolchain`, `workstation`, `worker`.                                     |
+| `worker.githubLogin`                                  | The GitHub login every worker must be signed in as.                                                                |
+| `worker.vaultUser`                                    | The Vault user each worker's token is issued for.                                                                  |
+| `worker.toolchain`, `agentProfile`                    | `"mise.lock"`, `"proxy"` (fixed).                                                                                  |
+| `worker.unattendedSudo`, `persistentT3`, `neverSleep` | `true` (fixed policy).                                                                                             |
+
+Use the same values in every place they repeat. Validate before going on:
+
+```sh
+npx devfleet fleet-state --json        # exits 2 with code "invalid_usage" when fleet.json is invalid
+```
+
+The `state:definition` check names the file it read. A `fleet.json` without a
+`manager` object is not taken for a fleet configuration at all, and the
+command then reports that it found none.
+
+### Secrets
+
+dev-fleet asks for four secrets by purpose: `proxy-client-key`,
+`monitoring-account-password`, `monitoring-tunnel-token` and
+`proxy-tunnel-token`. `secrets.provider` says where they are:
+
+| Provider | `fleet.json`                                                                                                                                         | Where the values are                                                    |
+| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
+| `vault`  | `{ "provider": "vault", "address": "https://vault.example.com", "mount": "secret", "locations": { NAME: { "path", "field" } } }` with all four names | A Vault KV v2 mount. Each worker gets its own renewable token.          |
+| `sops`   | `{ "provider": "sops", "file": "secrets.enc.yaml" }`                                                                                                 | One encrypted document with one top-level key per name.                 |
+| `file`   | `{ "provider": "file" }` (optional `"directory"`)                                                                                                    | One private file per name, default `~/.config/dev-fleet/secrets/NAME`.  |
+| `env`    | `{ "provider": "env" }`                                                                                                                              | `DEVFLEET_SECRET_PROXY_CLIENT_KEY` and so on (upper case, `_` for `-`). |
+
+Only `vault` gives each worker its own login and renewal and gives the CLI
+proxy its encrypted backup; with the other three those two features are not
+available yet. You never write a secret value into the project; the human puts
+them in the provider. **Ask** for that step.
+
+### Bucket credentials
+
+Every command reads them from the environment, never from arguments:
+
+```sh
+export DEVFLEET_BUCKET_ACCESS_KEY_ID=…
+export DEVFLEET_BUCKET_SECRET_ACCESS_KEY=…
+```
+
+Pass them however the human keeps secrets: a password manager's `run`
+command, `vault kv get` into the environment, or a git-ignored `.env` loaded
+with `set -a; . ./.env; set +a`. With neither variable set, the standard AWS
+environment or profile is used. A `"directory"` state needs none.
+
+Runs the manager starts by itself (a worker handing `fleet-sync` to the
+manager) get no environment, so save the credentials on the manager once:
+
+```sh
+npx devfleet fleet-state --init        # takes the bucket from fleet.json, asks on the terminal if the variables are unset
+```
+
+## 4. Tell future agents how to use it
+
+Put this in the project's `AGENTS.md` (or `CLAUDE.md`):
+
+```md
+The fleet is managed with dev-fleet (`npx devfleet`, pinned in package.json).
+Read `npx devfleet commands --json` before using it. Never edit dev-fleet
+itself: not node_modules, not a clone, not what it installs on a machine. A
+bug in dev-fleet or a feature it lacks is a GitHub issue at
+https://github.com/crvouga/dev-fleet/issues; tell me what is blocked and link
+it. Issues are public: no credentials, hostnames, accounts, domains or
+fleet.json contents.
+```
+
+Optionally add scripts, so nobody types the package name:
+
+```json
+"scripts": {
+  "fleet": "devfleet",
+  "fleet:sync": "devfleet fleet-sync",
+  "fleet:status": "devfleet fleet-status --workers"
+}
+```
+
+Commit `package.json`, the lockfile, `fleet.json`, `.gitignore` and
+`AGENTS.md`.
+
+## 5. Preview
+
+Run on the manager, from the project, with the bucket credentials in the
+environment:
+
+```sh
+npx devfleet fleet-state --json
+npx devfleet fleet-sync --dry-run --json
+```
+
+`fleet-state` reports the `state:bucket` and `state:replica` checks. On a new
+fleet the bucket is empty, which is a warning, not a failure: the first sync
+seeds it.
+
+Before the first sync, the dry run is expected to stop with
+`FLEET_MANAGER_ENROLLMENT_REQUIRED`: no worker's SSH connection and tailnet
+identity has been enrolled yet. Anything else is a problem to fix first (see
+[failures](#when-something-fails)).
+
+## 6. The first sync is the human's
+
+The first `fleet-sync` must run in a terminal the human is at. It:
+
+- enrolls each declared worker it reaches over host-key-verified SSH, binding
+  it to its tailnet device;
+- asks once for each worker's sudo password and installs a sudo rule for the
+  fleet account (the password is never stored);
+- with `vault`, asks for the Vault password once and issues each worker its own
+  token.
+
+An agent cannot answer those prompts, and `--json`, `--non-interactive` or a
+`CI` variable turn them off. **Ask** the human to run:
+
+```sh
+npx devfleet fleet-sync
+```
+
+It converges the manager first, then the canary worker, then the rest. It
+prints one table and the path of the full JSON report
+(`~/.local/share/personal/fleet/manager/last-sync.json`, readable by the human
+only). The first run on a bare worker takes several minutes: it installs the
+operating system packages, mise, the pinned toolchain, T3 and the monitoring
+agent.
+
+Then the human signs in, once, to what only they can sign in to:
+
+```sh
+npx devfleet cliproxy install           # the CLI proxy on the manager, if the sync reported it missing
+npx devfleet cliproxy add-account claude
+npx devfleet cliproxy add-account codex
+```
+
+and pairs the T3 desktop app on the manager with each worker as its check
+remedy says (`t3:paired-access`).
+
+## 7. Verify
+
+These are read-only and safe for an agent to run at any time:
+
+```sh
+npx devfleet fleet-status --workers --json    # every worker against the declared policy
+npx devfleet fleet-status --json              # the manager and every enrolled worker
+npx devfleet fleet-state --json               # the bucket holds what this machine has
+npx devfleet fleet-sync --dry-run --json      # what the next sync would do
+```
+
+Exit code `0` is success, `1` a failed check, `2` invalid usage or
+configuration. Each check is `{ id, status, summary, remedy? }` with status
+`pass`, `warn`, `fail` or `skip`. Some checks stay failing until a person
+acts (a sign-in, a pairing, a maintenance reboot); they say so in
+`remedy.action` and do not stop other workers from converging.
+
+Later syncs need no prompt and may be run by an agent when the human asks:
+`npx devfleet fleet-sync --json`.
+
+## When something fails
+
+Read `summary` and `remedy.action` of the failing check. The common ones:
+
+| Code or check                             | Meaning and remedy                                                                                                            |
+| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| `FLEET_MANAGER_ENROLLMENT_REQUIRED`       | No worker is enrolled yet. The human runs the first `fleet-sync` in a terminal.                                               |
+| `FLEET_NOT_MANAGER`                       | You are not on the machine `manager` names. Run there, or from a worker that can SSH to it.                                   |
+| `FLEET_REVISION_REQUIRED`                 | No revision to converge to. Run the installed package (`npx devfleet`), not a copy of its source.                             |
+| `FLEET_SOURCE_UNAVAILABLE`                | The revision could not be fetched from `source.repository`. Check the network and that the URL is right.                      |
+| `FLEET_STATE_DIVERGED`                    | This machine and the bucket hold different fleet states. `fleet-state --json` explains; never pick one without the human.     |
+| `FLEET_MANAGER_LOCKED`                    | Another run holds the manager's lock. Wait for it; do not delete the lock while a run is alive.                               |
+| `FLEET_PLAN_STALE`                        | The plan changed after it was approved. Preview again.                                                                        |
+| `ssh:host-key`                            | Host key unknown or changed. The human verifies the fingerprint out of band; never disable checking.                          |
+| `ssh:auth`, `ssh:dns`, `ssh:network`      | The alias does not work non-interactively. Fix `~/.ssh/config`, the key or Tailscale, then `ssh -o BatchMode=yes ALIAS true`. |
+| `TAILSCALE_NODE_OFFLINE`                  | The worker is off or off the tailnet. It is listed and skipped; others continue.                                              |
+| `TAILSCALE_DEVICE_REPLACED`               | The tailnet device behind a worker changed. Nothing is written to it until the human confirms the replacement.                |
+| `SUDO_AUTHORIZATION_REQUIRED`             | The worker account cannot use sudo without a password. Run the first sync interactively.                                      |
+| `UBUNTU_BASELINE_MISMATCH`                | The worker is not Ubuntu 26.04.                                                                                               |
+| `ROOT_DEVELOPMENT_USER_REJECTED`          | The worker's SSH user is root. Use a normal account with sudo.                                                                |
+| `BOOTSTRAP_DISK_CAPACITY_REQUIRED`        | Less than 2 GiB free (or too few inodes) where the fleet keeps its files on the worker.                                       |
+| `FLEET_REMOTE_LOCKED`                     | Another operation is running on that worker. Wait and run again.                                                              |
+| `FLEET_PROMOTION_BLOCKED`                 | The canary (first worker) failed, so the new revision went no further. Fix the canary first.                                  |
+| `VAULT_LOGIN_REQUIRED`                    | The manager is not logged in to Vault. The human logs in; the next sync issues worker tokens.                                 |
+| `PROVIDER_UNATTENDED_CREDENTIAL_REQUIRED` | A coding agent has no credential a background service can use. The human signs in on the proxy.                               |
+| `BESZEL_ACCOUNT_REQUIRED`                 | The monitoring hub has no account yet. Follow the remedy; other workers are not blocked.                                      |
+
+A failure the remedy does not explain, or one you believe is a defect, is an
+issue at <https://github.com/crvouga/dev-fleet/issues>: include
+`npx devfleet --version`, the command and flags, and the failing check IDs
+and codes; leave out everything private.
+
+## Day to day
+
+- **Add a worker.** Set up its SSH alias and host key and join it to the
+  tailnet (step 1), add it under `workers`, commit, then the human runs
+  `npx devfleet fleet-sync` once in a terminal to enroll it.
+- **Remove a worker.** Delete its entry, commit, run `npx devfleet fleet-sync`.
+- **Upgrade the fleet.** `npm install --save-dev --save-exact @crvouga/dev-fleet@latest`,
+  commit, `npx devfleet fleet-sync`. The canary proves the new revision first.
+  Install an older version and sync to roll back.
+- **Replace the manager.** On the new Mac: clone the project, `npm ci`, put
+  the bucket credentials in the environment, `npx devfleet fleet-sync`. The
+  roster and the approved revision come back from the bucket. See
+  [state](state.md) for what the bucket does not hold yet.
+
+## Moving from a pinned commit and a wrapper script
+
+Some fleets were set up before the package existed: a `fleet.revision` file
+holding a commit, and a script that cloned dev-fleet and ran it from that
+clone. To move such a project to the package:
+
+1. `npm install --save-dev --save-exact @crvouga/dev-fleet@VERSION`, choosing
+   the release whose commit (`npx devfleet --version`) is the one in
+   `fleet.revision` or newer.
+2. Add the `state` block to `fleet.json` if it has none, with the bucket the
+   manager already uses (`~/.config/dev-fleet/bucket.json` on the manager
+   names it).
+3. Replace every call of the wrapper with `npx devfleet …`, then delete the
+   wrapper script and `fleet.revision`.
+4. Run `npx devfleet fleet-sync --dry-run --json`, then `npx devfleet fleet-sync`.
+5. The CLI proxy on the manager may still run from the old clone: run
+   `npx devfleet cliproxy install` once before deleting that clone
+   ([state](state.md) explains why).
+
+## Further reading
+
+- [README](../README.md): what dev-fleet assumes and its known limits.
+- [Fleet state](state.md): the bucket, replication and recovery.
+- [What is declared](declared.md): what is pinned on a machine and what is not.
+- [Topology](topology.md): the manager and worker roles.
+- [CLI proxy](../cliproxy/README.md) and [monitoring](../ansible/monitoring/README.md).

package/docs/declared.md

@@ -0,0 +1,91 @@
+# What is declared, and where
+
+Two machines given the same dev-fleet version and the same `fleet.json` should
+end up the same. That holds only for what is written down in a tracked file,
+pinned to an exact version or digest, applied by a tool that converges
+(Ansible or mise) and measured afterwards. This page lists what is, and what
+is not yet.
+
+`features/rules/declared.feature` keeps the rules below true on every pull
+request. Its tables of exceptions are the same as [not declared yet](#not-declared-yet), and
+may only shrink.
+
+## Declared
+
+| What                                          | Declared in                                                                                         | Pinned by                                                            | Applied by                                                | Measured by                                  |
+| --------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | --------------------------------------------------------- | -------------------------------------------- |
+| The fleet: machines, services, accounts       | Your `fleet.json`                                                                                   | The dev-fleet version in your `package.json`, which names one commit | `fleet-sync`                                              | `fleet-status`                               |
+| Tools: languages, CLIs, coding agents         | `mise.toml`                                                                                         | `mise.lock`: version and checksum for each platform                  | `mise install --locked`                                   | Tool and `agent:verification` checks         |
+| mise itself                                   | `min_version` and `mise_*_sha256` in `mise.toml`                                                    | The digest of its release archive, held here                         | `bootstrap/bootstrap.sh`                                  | Its version, at every toolchain setup        |
+| JavaScript dependencies                       | `package.json`                                                                                      | `bun.lock`                                                           | `bun install --frozen-lockfile`                           | The install fails on a difference            |
+| T3 Code, server and desktop app               | `t3_*` under `[vars]` in `mise.toml`                                                                | SHA-256 of each artifact                                             | `bootstrap/native-t3.ts`; `ansible/ubuntu/t3-desktop.yml` | T3 checks; `t3:version` on every machine     |
+| T3 policy on the manager (updates off)        | `ansible/ubuntu/t3/policy.json`, shared with the workers                                            | The commit                                                           | `ansible/macos/t3-policy.yml`, once, as an administrator  | `t3:updates`                                 |
+| Worker work-in-progress guard                 | `ansible/ubuntu/wip-guard.py`, `ansible/ubuntu/tasks/recovery.yml`                                  | The commit                                                           | `fleet-sync`                                              | `features/ide/work-in-progress.feature`      |
+| Ubuntu packages                               | `packages` and `snapshot` in `ansible/ubuntu/policy.json`                                           | Exact versions from a dated archive snapshot                         | `ansible/ubuntu/tasks/packages.yml`                       | `package:*`                                  |
+| Tailscale on workers                          | `tailscale` in `ansible/ubuntu/policy.json`                                                         | Exact version; digest of the signing key                             | `ansible/ubuntu/tailscale.yml`                            | `package:tailscale`                          |
+| Worker policy: power, SSH, sudo, memory, disk | `ansible/ubuntu/*.yml`, values shared with checks in `policy.json`                                  | The commit                                                           | `fleet-sync`                                              | Host and resource checks                     |
+| Monitoring: hub, tunnel, agents               | `ansible/monitoring/*.yml`, `compose.yml`, `lock.json`                                              | Image digests; SHA-256 of each agent archive                         | `fleet-sync`                                              | `beszel:*`                                   |
+| The manager's private route to the hub        | `ansible/monitoring/hub-route-darwin.yml`                                                           | The commit                                                           | `fleet-sync`                                              | The route, read back                         |
+| The manager's power settings                  | `pmset` in `ansible/macos/policy.json`                                                              | The commit                                                           | `ansible/macos/manager.yml`, once, as an administrator    | `manager:power`                              |
+| Ansible on the manager                        | `ansible/pyproject.toml`                                                                            | `ansible/uv.lock`: one release, one Python, every dependency hashed  | `uv run --locked --isolated`                              | A hash that differs stops the run            |
+| Ansible on a worker                           | `ansible-core` in `ansible/ubuntu/policy.json`                                                      | Exact version                                                        | `ansible/ubuntu/tasks/packages.yml`                       | `package:ansible-core`                       |
+| The CLI proxy                                 | `cliproxy/compose.yaml`, `image.lock.json`, `requirements.lock`, `launchd/`, `config.template.yaml` | Image digests; hashed Python dependencies                            | `devfleet cliproxy install`, and `fleet-sync` after it    | The proxy's audit                            |
+| Agent profiles and T3 providers               | `agents/registry.json`, `agents/t3.json`                                                            | The commit                                                           | `fleet-sync`                                              | Profile receipts; the plan after every apply |
+| Docker Desktop on the manager                 | `docker_desktop_*` under `[vars]` in `mise.toml`                                                    | Version and build                                                    | You                                                       | `manager:docker` (warns)                     |
+| CI                                            | `.github/workflows/`                                                                                | Each action at a commit; tools from `mise.lock`                      | GitHub                                                    | `features/rules/`                            |
+
+A value lives in a policy file when something besides the playbook reads it,
+so the check and the playbook cannot disagree. A value only a playbook uses
+stays in the playbook: the playbook is its declaration.
+
+## Not declared yet
+
+- **A schedule.** Nothing re-applies the fleet on a timer, so a machine drifts
+  until someone runs `fleet-sync`. `fleet-sync --schedule enable` writes a
+  launchd job from TypeScript and has never worked from the published package.
+  It should become a playbook.
+- **The rest of a worker's packages.** Only the packages in `policy.json` are
+  exact. `systemd-oomd` and the T3 desktop libraries are installed at whatever
+  version the machine's own archive holds, and the machine's own updates move
+  everything else. Pinning them needs every worker on the snapshot with its
+  own updates off, so security fixes arrive only when the snapshot moves. That
+  is a policy choice, not a missing line.
+- **What a worker needs before Ansible exists.** `fleet-remote-bootstrap.sh`
+  installs Git, curl, CA certificates, Python and `ansible-core` from the
+  machine's own archive (the snapshot is HTTPS, and a new machine has no CA
+  certificates yet). The baseline then pins each of them.
+- **The manager's applications.** Docker Desktop, Tailscale and Apple's
+  Command Line Tools are installed by a person. Docker Desktop is measured;
+  the other two are only required.
+- **The CLI proxy's lifecycle.** It is installed and started by its own Python
+  scripts from the declarative files above, not by a playbook.
+- **Files TypeScript and shell still write.** The installed `fleet.json`, the
+  `current` and `devfleet` links, the `PATH` lines in shell profiles, the
+  global mise configuration, the T3 server archive, agent profiles and T3
+  settings. Each is derived
+  from a tracked file and rewritten on every run, but by code.
+- **Logins.** Tailscale, Vault, GitHub, each provider account and T3 pairing
+  belong to a machine and a person. They are checked, never copied.
+- **Windows.** `bootstrap/bootstrap.ps1` exists; Windows is not a fleet role
+  and nothing tests it.
+
+## Why not Nix
+
+Nix would give the same three things this has: pinned inputs, verified
+artifacts and a declarative description of the machine. Here they come from
+`mise.lock`, the Ubuntu snapshot and Ansible. Adding Nix would put a second
+package universe on every machine beside the one that is already running, and
+moving to it entirely means NixOS workers and nix-darwin on the manager, which
+replaces every unit and path that exists today. The one thing above that Nix
+would settle and the current tools do not is the rest of a worker's packages.
+
+## Changing a pin
+
+| To change               | Edit                                                                 | Then                                             |
+| ----------------------- | -------------------------------------------------------------------- | ------------------------------------------------ |
+| A tool                  | Its version in `mise.toml`                                           | Update `mise.lock` with `mise lock`              |
+| mise                    | `min_version` and the four `mise_*_sha256` in `mise.toml`            | Copy the digests from the release's `SHASUMS256` |
+| An Ubuntu package       | `packages` in `ansible/ubuntu/policy.json`, and `snapshot` if needed | The VM test in `ansible/ubuntu/tests`            |
+| Ansible on the manager  | `ansible/pyproject.toml`                                             | `uv lock` in `ansible/`                          |
+| Beszel                  | `ansible/monitoring/lock.json`, `compose.yml`, the agent unit        | `bun test features/rules`                        |
+| A manager power setting | `pmset` in `ansible/macos/policy.json`                               | Run `manager.yml` once on the manager            |

package/docs/durability.md

@@ -0,0 +1,26 @@
+# What survives the loss of a machine
+
+This is the one place that says what dev-fleet keeps when a machine is lost.
+
+| Lost        | Kept                                                                                                                                                                                                                                                                      | Lost with it                                                                                             |
+| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
+| A worker    | Everything `fleet-sync` rebuilds from Git and secrets. Work in agent worktrees (`~/.t3/worktrees`) up to the last run of the work-in-progress guard, every 15 minutes, in `refs/recovery/<node>/<branch>` of each worktree's remote. The T3 environment ID in the roster. | Up to 15 minutes of uncommitted work; T3 threads and their transcripts on that worker.                   |
+| The manager | The fleet state, from the bucket ([state](state.md)). The T3 profile, provider instances and policy, from `fleet.json` and Git.                                                                                                                                           | T3's own data on the manager: its linked environments and its threads. They are not in a backup set yet. |
+
+## Recovering work from a lost worker
+
+The guard pushes the files of each worktree as they were (changed and new
+files, not ignored ones) as one commit on top of its `HEAD`, without touching
+the worktree, its index or its branches:
+
+```sh
+git fetch origin 'refs/recovery/*:refs/recovery/*'
+git log refs/recovery/<node>/<branch>
+git switch -c recovered refs/recovery/<node>/<branch>
+```
+
+A worktree whose work is already on a branch of its remote pushes nothing.
+The guard is `ansible/ubuntu/wip-guard.py`, run by the user timer
+`personal-worker-wip-guard.timer` that `ansible/ubuntu/tasks/recovery.yml`
+installs. It pushes with the worker's own Git credentials, so a worktree
+whose remote the worker cannot write to is reported and skipped.