@crvouga/dev-fleet 0.2.0 → 0.4.0

package/docs/accounts.md

@@ -0,0 +1,154 @@
+# Provider accounts
+
+**The rule.** Provider accounts exist only as the CLI proxy's credential files
+on the manager. Every node and IDE gets a view derived from them, and nothing
+on a node holds a provider login of its own. T3 offers one instance per
+account, named `<Provider> <account email>`, and every launcher reaches the
+provider through the proxy.
+
+## The roster
+
+```sh
+npx devfleet fleet-accounts            # provider, index, label, state, when a cooldown clears
+npx devfleet fleet-accounts --diff     # and what this machine offers against it
+```
+
+Each account has one state, derived from its credential file and the proxy's
+management API:
+
+| State             | Meaning                                                                    |
+| ----------------- | -------------------------------------------------------------------------- |
+| `usable`          | Serves requests                                                            |
+| `cooling-down`    | The provider limited it; `until` says when it clears, when the proxy knows |
+| `refresh-failing` | Its token refresh fails for now, and its access token is still valid       |
+| `needs-login`     | The provider refused its refresh token: only signing in again helps        |
+| `removed`         | Disabled on the proxy                                                      |
+
+The index of an account is its position among its provider's accounts, sorted
+by sign-in email. Checks name accounts by provider and index only.
+
+The **roster version** is a digest of the routing prefixes of the accounts.
+The manager computes it from the credential files and every node from the
+proxy's model list, so the two agree exactly when a node offers what the
+proxy holds.
+
+## Checks
+
+| Check                              | Fails when                                                                                             |
+| ---------------------------------- | ------------------------------------------------------------------------------------------------------ |
+| `proxy:account:<provider>`         | The provider has no usable account; it warns below `services.proxy.accounts.minimumUsable` (default 2) |
+| `accounts:roster-current`          | The node offers another roster version than the proxy's                                                |
+| `accounts:native-login`            | (warns) An agent CLI on the node keeps a provider login of its own                                     |
+| `proxy:reachable-from`             | The node gets no answer from the proxy; it warns above two seconds                                     |
+| `agent:<tool>:account-<n>:session` | That account's instance does not run a session; it warns while the account is cooling down             |
+| `agent:<tool>:removed-<n>:session` | The node still offers an instance for an account the proxy no longer holds                             |
+
+Session results are cached for six hours under a key that holds the roster
+version and the version of the client key, so any account change or key
+rotation re-verifies at once.
+
+## Adding, signing in again, removing
+
+```sh
+npx devfleet fleet-accounts --add claude
+npx devfleet fleet-accounts --relogin codex 2
+npx devfleet fleet-accounts --remove codex 2
+```
+
+A sign-in writes into a staging directory beside the credentials; only a
+complete credential is moved in, with one rename, so an interrupted sign-in
+leaves nothing behind. Routing prefixes are applied in the same command. Every
+change is then propagated to every node.
+
+## Propagation without a full sync
+
+```sh
+npx devfleet fleet-sync --only accounts
+```
+
+On the manager and every worker at once, each node re-reads the proxy's
+roster and rewrites only its per-account T3 instances. T3 watches its
+settings, so nothing restarts and no running turn ends. Instance ids derive
+from the account's routing prefix and stay the same. A node that cannot be
+reached catches up on the next propagation or sync; until then
+`accounts:roster-current` reports the version it has.
+
+## Choosing an account
+
+Unpinned requests go through a loopback relay on every node. When the fleet
+declares a policy for a provider, the relay picks the account of each request:
+
+```json
+"services": {
+  "proxy": {
+    "endpoint": "https://cli-proxy.example.com",
+    "accounts": {
+      "minimumUsable": 2,
+      "selection": {
+        "codex": { "strategy": "fill-first", "order": ["a@example.com"] }
+      }
+    }
+  }
+}
+```
+
+| Strategy                 | Order                                            |
+| ------------------------ | ------------------------------------------------ |
+| `fill-first`             | The same order for every request                 |
+| `round-robin`            | Each request starts one account further along    |
+| `least-recently-limited` | The account limited longest ago (or never) first |
+
+The order is the accounts `order` names, in that order, then the others in
+the order of their routing prefix. A quota or authentication refusal sends
+the request on to the next account, and the refused one cools down for its
+`Retry-After` (a minute without one). Without a policy the proxy's own
+fill-first pool chooses. An instance pinned to one account answers "This
+account is cooling down until …" and names the other instances of its
+provider.
+
+## Proxy outages
+
+Every relayed launch (Claude, Codex, Gemini and Antigravity, pinned or not)
+retries a proxy that does not answer for about a minute. If it stays down,
+the agent gets one distinct, retryable error: "The CLI proxy has been
+unreachable since …".
+
+## Checkpoints and the proxy state
+
+Each checkpoint records a short digest of every credential's refresh token,
+never the token. After restoring an older checkpoint, `restore-verify
+RESTORED --newest NEWEST` names exactly the accounts whose refresh token the
+provider has since replaced (`needs-login: <provider> <index>`).
+
+`cliproxy/scripts/lease.py` is a fencing lease on the proxy state: a second
+proxy cannot take it while the first holds it, and taking over an expired
+lease fences the old holder.
+
+## Client keys
+
+`devfleet cliproxy issue-key|rotate-key|retire-key|revoke-key NODE` keep one
+proxy client key per node in the manager's key ring. A rotation keeps the old
+key accepted until `retire-key`, so no running agent fails while nodes move
+over. Revoking one node's key leaves the others working.
+`devfleet cliproxy client-keys` reports each node's key by fingerprint only.
+
+## Not yet done
+
+- The proxy does not notify anyone when an account turns `needs-login`, and
+  no watcher on the manager triggers propagation or a checkpoint when a
+  credential file changes. Run `fleet-sync --only accounts` or
+  `fleet-accounts` instead.
+- Nothing forces a refresh of every account after a restore. The digest
+  comparison above is offline.
+- The lease is not yet taken by the proxy service at start, and its
+  production store (the checkpoint bucket) is not wired.
+- Per-node client keys are not yet distributed to nodes through the secrets
+  provider, so nodes still present the shared key. No
+  `proxy:client-key-current` check exists yet.
+- OpenCode and Grok read the proxy's address from rendered configuration and
+  are not relayed yet. There is no warm standby proxy.
+- The selection policy is applied by the client relay; the proxy itself stays
+  fill-first.
+- A sign-in uses the provider's device code where the proxy offers one.
+  Other providers print a callback URL, and no loopback forward is managed
+  for them.

package/docs/authentication.md

@@ -0,0 +1,90 @@
+# Authentication and accounts
+
+Where each credential dev-fleet uses lives, who can read it, how it reaches
+each node, how it rotates and what happens when it fails. The three tables
+are generated from the code; `features/rules/authentication.feature` fails
+when one is out of date (`DEVFLEET_UPDATE_DOCS=1 bun test features/rules`
+rewrites them).
+
+## Where credentials live
+
+| Credential                          | Lives in                                                                                                                      | Read by                                          |
+| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
+| The fleet's secrets (table below)   | The provider `secrets` names in `fleet.json`                                                                                  | The node that needs it, through `src/secrets.ts` |
+| The bucket's two keys               | The environment of the `devfleet` command; `~/.config/dev-fleet/bucket.json` (0600) on the manager after `fleet-state --init` | The manager only                                 |
+| A node's Vault token                | That node's own Vault session (`~/.vault-token`)                                                                              | That node only; never copied                     |
+| A node's GitHub login               | `gh`'s own store on that node                                                                                                 | `gh`, and git through its credential helper      |
+| Provider accounts (Claude, Codex …) | The CLI proxy's store on the manager                                                                                          | The proxy; nodes hold only the proxy client key  |
+| Tailscale and T3 pairing            | Each machine, signed in by a person                                                                                           | That machine                                     |
+
+## How they reach each node
+
+- **Vault.** A person logs each machine in once (`fleet-sync` asks on the
+  manager and passes the password to the worker over SSH on standard input).
+  From then on the worker's `personal-worker-auth` timer renews its own token
+  (`bootstrap/worker-auth.ts`). Nothing renders provider configuration from
+  Vault on a worker: agent profiles come from `fleet-sync`.
+- **GitHub.** The same job restores the declared account (`worker.githubLogin`)
+  with the GitHub service token it reads from its own Vault session
+  (`bootstrap/worker-github.ts`, on standard input, never argv). `fleet-sync`
+  makes `gh` git's credential helper and writes the declared identity
+  (`git` in `fleet.json`, `ansible/github/git.yml`) on every node.
+- **The proxy client key** is read by each node from its own secrets provider
+  whenever a launcher starts an agent.
+- **The monitoring hub's backup storage keys** are read on the manager by
+  `fleet-sync` and handed to the hub's playbook in a private variables file.
+
+## Rotation and failure
+
+Rotate a secret where the provider keeps it; the next read uses it, nothing
+caches it. A Vault token that is not renewed expires: `worker:vault-renewal`
+fails and the fix is one login of that machine. A GitHub token that stops
+working fails `auth:github` and `worker:git-push-auth`; store the new one in
+Vault and the worker restores itself within ten minutes. During a proxy
+outage agents on workers wait: no worker is configured for native clients.
+Every failure code is in [the runbooks](runbooks.md).
+
+## The fleet's secrets
+
+<!-- generated: secrets -->
+
+| Secret                         | Required                     | `env` provider variable                        | `file` provider file                                       |
+| ------------------------------ | ---------------------------- | ---------------------------------------------- | ---------------------------------------------------------- |
+| `proxy-client-key`             | yes                          | `DEVFLEET_SECRET_PROXY_CLIENT_KEY`             | `~/.config/dev-fleet/secrets/proxy-client-key`             |
+| `monitoring-account-password`  | yes                          | `DEVFLEET_SECRET_MONITORING_ACCOUNT_PASSWORD`  | `~/.config/dev-fleet/secrets/monitoring-account-password`  |
+| `monitoring-tunnel-token`      | yes                          | `DEVFLEET_SECRET_MONITORING_TUNNEL_TOKEN`      | `~/.config/dev-fleet/secrets/monitoring-tunnel-token`      |
+| `proxy-tunnel-token`           | yes                          | `DEVFLEET_SECRET_PROXY_TUNNEL_TOKEN`           | `~/.config/dev-fleet/secrets/proxy-tunnel-token`           |
+| `monitoring-backup-access-key` | when its feature is declared | `DEVFLEET_SECRET_MONITORING_BACKUP_ACCESS_KEY` | `~/.config/dev-fleet/secrets/monitoring-backup-access-key` |
+| `monitoring-backup-secret-key` | when its feature is declared | `DEVFLEET_SECRET_MONITORING_BACKUP_SECRET_KEY` | `~/.config/dev-fleet/secrets/monitoring-backup-secret-key` |
+| `notify-webhook-url`           | when its feature is declared | `DEVFLEET_SECRET_NOTIFY_WEBHOOK_URL`           | `~/.config/dev-fleet/secrets/notify-webhook-url`           |
+| `notify-ntfy-url`              | when its feature is declared | `DEVFLEET_SECRET_NOTIFY_NTFY_URL`              | `~/.config/dev-fleet/secrets/notify-ntfy-url`              |
+
+<!-- end: secrets -->
+
+## Checks of credentials
+
+<!-- generated: checks -->
+
+| Check                         | Measured in                                |
+| ----------------------------- | ------------------------------------------ |
+| `auth:vault`                  | `src/cli/fleet-sync-vault.ts`              |
+| `worker:git-identity`         | `src/git-identity.ts`                      |
+| `worker:git-push-auth`        | `src/definition.ts`, `src/git-identity.ts` |
+| `worker:github-principal`     | `src/worker-access-checks.ts`              |
+| `worker:proxy-authentication` | `src/worker-access-checks.ts`              |
+| `worker:vault-renewal`        | `src/worker-access-checks.ts`              |
+
+<!-- end: checks -->
+
+## Timers
+
+<!-- generated: timers -->
+
+| Timer                            | Declared in                            | Schedule                              |
+| -------------------------------- | -------------------------------------- | ------------------------------------- |
+| `personal-fleet-disk-gc.timer`   | `ansible/ubuntu/tasks/resources.yml`   | OnBootSec=10min, OnCalendar=hourly    |
+| `personal-worker-auth.timer`     | `ansible/ubuntu/tasks/credentials.yml` | OnBootSec=2min, OnUnitActiveSec=10min |
+| `personal-worker-recovery.timer` | `ansible/ubuntu/tasks/recovery.yml`    | OnBootSec=1min, OnUnitActiveSec=1min  |
+| `personal-worker-recovery.timer` | `ansible/ubuntu/tasks/recovery.yml`    | OnBootSec=5min, OnUnitActiveSec=15min |
+
+<!-- end: timers -->

package/docs/declared.md

@@ -18,7 +18,9 @@ may only shrink.
 | Tools: languages, CLIs, coding agents         | `mise.toml`                                                                                         | `mise.lock`: version and checksum for each platform                  | `mise install --locked`                                   | Tool and `agent:verification` checks         |
 | mise itself                                   | `min_version` and `mise_*_sha256` in `mise.toml`                                                    | The digest of its release archive, held here                         | `bootstrap/bootstrap.sh`                                  | Its version, at every toolchain setup        |
 | JavaScript dependencies                       | `package.json`                                                                                      | `bun.lock`                                                           | `bun install --frozen-lockfile`                           | The install fails on a difference            |
-| T3 Code, server and desktop app               | `t3_*` under `[vars]` in `mise.toml`                                                                | SHA-256 of each artifact                                             | `bootstrap/native-t3.ts`; `ansible/ubuntu/t3-desktop.yml` | T3 checks                                    |
+| T3 Code, server and desktop app               | `t3_*` under `[vars]` in `mise.toml`                                                                | SHA-256 of each artifact                                             | `bootstrap/native-t3.ts`; `ansible/ubuntu/t3-desktop.yml` | T3 checks; `t3:version` on every machine     |
+| T3 policy on the manager (updates off)        | `ansible/ubuntu/t3/policy.json`, shared with the workers                                            | The commit                                                           | `ansible/macos/t3-policy.yml`, once, as an administrator  | `t3:updates`                                 |
+| Worker work-in-progress guard                 | `ansible/ubuntu/wip-guard.py`, `ansible/ubuntu/tasks/recovery.yml`                                  | The commit                                                           | `fleet-sync`                                              | `features/ide/work-in-progress.feature`      |
 | Ubuntu packages                               | `packages` and `snapshot` in `ansible/ubuntu/policy.json`                                           | Exact versions from a dated archive snapshot                         | `ansible/ubuntu/tasks/packages.yml`                       | `package:*`                                  |
 | Tailscale on workers                          | `tailscale` in `ansible/ubuntu/policy.json`                                                         | Exact version; digest of the signing key                             | `ansible/ubuntu/tailscale.yml`                            | `package:tailscale`                          |
 | Worker policy: power, SSH, sudo, memory, disk | `ansible/ubuntu/*.yml`, values shared with checks in `policy.json`                                  | The commit                                                           | `fleet-sync`                                              | Host and resource checks                     |
@@ -29,6 +31,8 @@ may only shrink.
 | Ansible on a worker                           | `ansible-core` in `ansible/ubuntu/policy.json`                                                      | Exact version                                                        | `ansible/ubuntu/tasks/packages.yml`                       | `package:ansible-core`                       |
 | The CLI proxy                                 | `cliproxy/compose.yaml`, `image.lock.json`, `requirements.lock`, `launchd/`, `config.template.yaml` | Image digests; hashed Python dependencies                            | `devfleet cliproxy install`, and `fleet-sync` after it    | The proxy's audit                            |
 | Agent profiles and T3 providers               | `agents/registry.json`, `agents/t3.json`                                                            | The commit                                                           | `fleet-sync`                                              | Profile receipts; the plan after every apply |
+| Worker time service                           | `ansible/ubuntu/tasks/services.yml`, offsets in `clock` in `ansible/ubuntu/policy.json`             | The commit                                                           | `fleet-sync`                                              | `worker:clock`                               |
+| The manager's container watchdog              | `ansible/monitoring/container-watchdog.py`, `containerWatchdog` in `ansible/macos/policy.json`      | The commit                                                           | `fleet-sync` (`hub-darwin.yml`)                           | `manager:container-watchdog`                 |
 | Docker Desktop on the manager                 | `docker_desktop_*` under `[vars]` in `mise.toml`                                                    | Version and build                                                    | You                                                       | `manager:docker` (warns)                     |
 | CI                                            | `.github/workflows/`                                                                                | Each action at a commit; tools from `mise.lock`                      | GitHub                                                    | `features/rules/`                            |
 

package/docs/durability.md

@@ -0,0 +1,26 @@
+# What survives the loss of a machine
+
+This is the one place that says what dev-fleet keeps when a machine is lost.
+
+| Lost        | Kept                                                                                                                                                                                                                                                                      | Lost with it                                                                                             |
+| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
+| A worker    | Everything `fleet-sync` rebuilds from Git and secrets. Work in agent worktrees (`~/.t3/worktrees`) up to the last run of the work-in-progress guard, every 15 minutes, in `refs/recovery/<node>/<branch>` of each worktree's remote. The T3 environment ID in the roster. | Up to 15 minutes of uncommitted work; T3 threads and their transcripts on that worker.                   |
+| The manager | The fleet state, from the bucket ([state](state.md)). The T3 profile, provider instances and policy, from `fleet.json` and Git.                                                                                                                                           | T3's own data on the manager: its linked environments and its threads. They are not in a backup set yet. |
+
+## Recovering work from a lost worker
+
+The guard pushes the files of each worktree as they were (changed and new
+files, not ignored ones) as one commit on top of its `HEAD`, without touching
+the worktree, its index or its branches:
+
+```sh
+git fetch origin 'refs/recovery/*:refs/recovery/*'
+git log refs/recovery/<node>/<branch>
+git switch -c recovered refs/recovery/<node>/<branch>
+```
+
+A worktree whose work is already on a branch of its remote pushes nothing.
+The guard is `ansible/ubuntu/wip-guard.py`, run by the user timer
+`personal-worker-wip-guard.timer` that `ansible/ubuntu/tasks/recovery.yml`
+installs. It pushes with the worker's own Git credentials, so a worktree
+whose remote the worker cannot write to is reported and skipped.

package/docs/machine-setup.md

@@ -181,19 +181,21 @@ wrangler login
 railway login
 ```
 
-Only sign into providers the task actually needs. Git identity is personal:
-set `git config --global user.name` and `git config --global user.email` after
-confirming the desired name/email. Do not guess an email or overwrite an existing
-identity. Enable Git LFS hooks when needed with `git lfs install`.
+Only sign into providers the task actually needs. On fleet machines the Git
+identity and the GitHub credential helper come from the `git` block of
+`fleet.json`: `fleet-sync` writes them in `~/.config/git/config` and checks
+them as `worker:git-identity` and `worker:git-push-auth`. Elsewhere set `git
+config --global user.name` and `user.email` after confirming them; do not guess
+an email or overwrite an existing identity. Enable Git LFS hooks when needed with `git lfs install`.
 
 Optional existing workstation/agent settings:
 
 ```sh
 sh bootstrap/install-launcher.sh
-personal workstation doctor
+devfleet fleet-status --local
 # Only for tasks needing the workspace's managed secrets:
 vault login --method=userpass username=example-user
-personal workstation sync
+devfleet fleet-sync
 ```
 
 The machine bootstrap installs tools/dependencies only. `personal:install` additionally
@@ -231,7 +233,7 @@ The manifest installs the **Docker client**. Containers also require a local or
 remote Docker engine, Compose v2 and Buildx plugins. These cannot be installed as
 one identical unprivileged binary on all OSes.
 
-- **macOS:** run `personal workstation tools docker-setup` to install the pinned,
+- **macOS:** run `bash bootstrap/docker-desktop.sh` to install the pinned,
   checksum-verified Docker Desktop bundle from this repository and open it.
   Complete its first-run agreement/setup in the app. Repeated invocation reuses
   an existing installation. The shared mise Docker client remains in use.
@@ -280,11 +282,11 @@ mise install --locked bun
 mise exec --locked bun -- bun bootstrap/setup.ts
 ```
 
-For an installed workstation CLI, `personal workstation update` pulls with a clean-tree/fast-forward
-guard, refreshes the toolchain/global config and frozen dependencies, and runs the
-existing workstation convergence. `personal workstation tools sync` reapplies tool pins/global config
-without pulling or touching project dependencies; `personal workstation tools check --json` verifies
-them. Sync is explicit; Git commits do not silently update other machines.
+On fleet machines `devfleet fleet-sync` refreshes the toolchain, the global
+configuration and frozen dependencies; `mise exec --locked bun -- bun
+bootstrap/setup.ts --tools-only` reapplies tool pins and the global
+configuration without touching project dependencies, and `bun
+bootstrap/doctor.ts --json` verifies them. Sync is explicit; Git commits do not silently update other machines.
 
 For multiple development computers, use [the fleet package](operations.md):
 `devfleet fleet-node-add` registers an existing SSH alias in a local roster,
@@ -352,9 +354,10 @@ only running T3 is the T3 Code desktop app, installed from the same release's
 upstream `SHA256SUMS` omits packages) and started by the `t3code-desktop.service`
 user unit in the GDM auto-login session. It shares `~/.t3` with the former
 headless service, so environment identity and pairing are kept. A conflicting
-launcher version or a still-running headless service requires reviewed
-maintenance; routine fleet-sync never restarts active turns to promote a new
-pin. The disposable Ubuntu VM test still exercises the former headless service
+launcher version requires reviewed maintenance; the recovery timer stops a
+still-running headless service. Routine fleet-sync never ends a running turn
+to load a new pin or policy: the app restarts into them at the first idle
+check (`t3:running-version`, `t3:policy-loaded`). The disposable Ubuntu VM test still exercises the former headless service
 and does not cover the desktop session. Private routing and paired execution
 require separate evidence.
 
@@ -394,6 +397,14 @@ It installs the `vars.worker_tools` subset from `mise.toml`, with versions and
 artifacts taken from the same `mise.lock`; native T3 and Ubuntu Docker/service
 policy remain source-pinned. `bun bootstrap/doctor.ts --worker --docker --json`
 checks that profile. It does not require unrelated cloud CLIs on workers.
+
+Tool profiles add locked tool sets on top of it: `worker.toolProfiles` in
+`fleet.json` (fleet-wide) or `toolProfiles` on one worker selects `web` (pnpm,
+Deno, Yarn and headless Chromium for browser tests), `go`, `rust` (with CMake
+and Ninja) or `jvm` (Temurin), each a `worker_profile_<name>` under `[vars]` in
+`mise.toml`. `worker.tools` adds single tools dev-fleet already pins. The setup
+installs them (`--node` reads the installed configuration) and the doctor
+checks each as `tool:<name>` (`--profiles`, `--tools`).
 The source-owned `worker-auth.ts` timer renews the worker's existing Vault token
 and can restore the declared GitHub account from its authorized Vault service
 credential, without copying another machine's login directory.

package/docs/operations.md

@@ -4,9 +4,9 @@
 
 # Developer fleet
 
-`@pkgs/fleet` diagnoses development computers: SSH, VS Code Remote-SSH,
+`devfleet` diagnoses development computers: SSH, VS Code Remote-SSH,
 per-machine authentication, T3 Connect and shared toolchain/dependency drift.
-It is a local CLI/library, not a daemon or cloud manager. It reuses the
+It is a local CLI, not a library, a daemon or a cloud manager. It reuses the
 existing mise/workstation owners rather than introducing another installer.
 
 The fleet is one singleton **manager** (the human's macOS machine, which runs
@@ -46,8 +46,9 @@ devfleet fleet-repair --node dev-linux --repair workspace --json   # plan only
 devfleet fleet-repair --node dev-linux --repair workspace --apply --yes --json
 ```
 
-This private package exposes a TypeScript API from `@pkgs/fleet`; the public
-executable is `personal` (`bun run personal` from the checkout). Schema, roster
+dev-fleet is a command with no library API: run it as `npx devfleet` from the
+project that holds your `fleet.json` (`./devfleet` from a checkout of dev-fleet
+itself). Schema, roster
 listing, onboarding and repair previews do not contact nodes or auth services.
 Status never opens a login flow.
 
@@ -61,7 +62,7 @@ remains the sole cloud resource owner; fleet creates no networks, hosts or secre
 
 ```json
 {
-  "$schema": "./packages/fleet/schema.json",
+  "$schema": "./node_modules/@crvouga/dev-fleet/schema/roster.schema.json",
   "version": 1,
   "nodes": [
     {
@@ -97,21 +98,50 @@ remote nodes' required lists. Run `fleet-nodes --schema` for editor/agent valida
 
 ## Evidence and diagnosis
 
-| Check                                                                              | Evidence / next action                                                                                                                                |
-| ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `workspace`, `git:clean`                                                           | HEAD, input availability and dirty-tree status; no paths/diffs/file contents logged.                                                                  |
-| `revision`, `sync:*`                                                               | Compare exact HEAD and SHA-256 of package.json, bun.lock, mise.toml, mise.lock against manager. Missing inputs never pass.                            |
-| `dependencies`                                                                     | Last successful fleet frozen-sync receipt matches input hashes. Missing receipt is unverified, not in-sync.                                           |
-| `auth:toolchain`                                                                   | Existing bootstrap/doctor.ts runs in the node checkout with its Bun, bounded to 90s. Use personal workstation tools check --json for individual pins. |
-| `ssh:agent`                                                                        | Manager agent exposes an identity; missing agent warns because IdentityFile keys may still work.                                                      |
-| `ssh:session`                                                                      | Fresh batch SSH session actually runs the Node/Bun probe. No reused multiplexed connection or agent forwarding.                                       |
-| `ssh:host-key`, `ssh:dns`, `ssh:auth`, `ssh:network`, `ssh:runtime`, `ssh:timeout` | Classified failure and targeted remedy, without raw logs.                                                                                             |
-| `auth:github`                                                                      | gh auth status --active succeeds on this machine; not proof of Git SSH or every repository permission.                                                |
-| `auth:vault`                                                                       | Server-side vault token lookup succeeds in this context; not proof of access to every secret path.                                                    |
-| `auth:vscode`                                                                      | code CLI finds the local Remote-SSH extension. Other profiles/Cursor/GUI PATH/agents can differ.                                                      |
-| `t3:settings`                                                                      | Connect explicitly enabled and not revoked in t3 status --json; settings are not connectivity.                                                        |
-| `t3:api`                                                                           | Read-only t3 environment list --json succeeds against the local API; no pairing/token minting.                                                        |
-| `t3:end-to-end`, `vscode:forwarding`                                               | SKIP: actual relay/peer authorization and VS Code tunnel/server installation require operator testing.                                                |
+| Check                                                                              | Evidence / next action                                                                                                                                                |
+| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `workspace`, `git:clean`                                                           | HEAD, input availability and dirty-tree status; no paths/diffs/file contents logged.                                                                                  |
+| `revision`, `sync:*`                                                               | Compare exact HEAD and SHA-256 of package.json, bun.lock, mise.toml, mise.lock against manager. Missing inputs never pass.                                            |
+| `dependencies`                                                                     | Last successful fleet frozen-sync receipt matches input hashes. Missing receipt is unverified, not in-sync.                                                           |
+| `auth:toolchain`                                                                   | Existing bootstrap/doctor.ts runs in the node checkout with its Bun, bounded to 90s. Run `bun bootstrap/doctor.ts --json` in the node's checkout for individual pins. |
+| `ssh:agent`                                                                        | Manager agent exposes an identity; missing agent warns because IdentityFile keys may still work.                                                                      |
+| `ssh:session`                                                                      | Fresh batch SSH session actually runs the Node/Bun probe. No reused multiplexed connection or agent forwarding.                                                       |
+| `ssh:host-key`, `ssh:dns`, `ssh:auth`, `ssh:network`, `ssh:runtime`, `ssh:timeout` | Classified failure and targeted remedy, without raw logs.                                                                                                             |
+| `auth:github`                                                                      | gh auth status --active succeeds on this machine; not proof of Git SSH or every repository permission.                                                                |
+| `auth:vault`                                                                       | Server-side vault token lookup succeeds in this context; not proof of access to every secret path.                                                                    |
+| `auth:vscode`                                                                      | code CLI finds the local Remote-SSH extension. Other profiles/Cursor/GUI PATH/agents can differ.                                                                      |
+| `t3:settings`                                                                      | Connect explicitly enabled and not revoked in t3 status --json; settings are not connectivity.                                                                        |
+| `t3:api`                                                                           | Read-only t3 environment list --json succeeds against the local API; no pairing/token minting.                                                                        |
+| `t3:end-to-end`, `vscode:forwarding`                                               | SKIP: actual relay/peer authorization and VS Code tunnel/server installation require operator testing.                                                                |
+
+### Machine health
+
+| Check                        | Evidence / next action                                                                                                                                                                                                                                                                                                                             |
+| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `worker:capacity:inodes`     | Free inodes on the home filesystem, against `inodeWarnFreePercent`/`inodeFailFreePercent` in `ansible/ubuntu/policy.json`. The disk guard and disk collector act on the same floors.                                                                                                                                                               |
+| `worker:clock`               | `timedatectl` says the clock is synchronised and a time service is enabled; with a reference time, the offset is held to `clock` in the policy.                                                                                                                                                                                                    |
+| `worker:reboot-pending`      | `/var/run/reboot-required` exists: warns with its age, fails after `rebootPendingFailDays`. Reboot when no agent turn is running.                                                                                                                                                                                                                  |
+| `tailscale:key-expiry`       | The node key expiry Tailscale reports: warns 14 days ahead, fails 3 days ahead. See [Tailscale key expiry](#tailscale-key-expiry).                                                                                                                                                                                                                 |
+| `manager:release`            | `sw_vers` against `minimumMajorVersion` in `ansible/macos/policy.json`.                                                                                                                                                                                                                                                                            |
+| `manager:updates`            | `softwareupdate --list --no-scan` (nothing is downloaded): warns on pending updates, and names one that waits for a restart. Names never shown.                                                                                                                                                                                                    |
+| `manager:disk`               | Free space on the data volume and the space Docker holds, against `disk` in `ansible/macos/policy.json`.                                                                                                                                                                                                                                           |
+| `manager:container-watchdog` | The container watchdog (`ansible/monitoring/container-watchdog.py`, a launch agent `hub-darwin.yml` installs) restarts a labelled container that was unhealthy for `unhealthyChecks` runs, at most `maxRestartsPerHour` times an hour (`containerWatchdog` in `ansible/macos/policy.json`). Warns after a restart; fails once it stops restarting. |
+
+### Tailscale key expiry
+
+A node drops off the tailnet on the day its key expires. Keep workers from
+expiring: tag them, or disable key expiry for each in the Tailscale admin
+console (tagged nodes do not expire). `tailscale:key-expiry` warns two weeks
+ahead and fails three days ahead.
+
+To re-join a worker whose key expired, a person signs it in again, because
+only a person's identity can authorise a device:
+
+1. On the worker's console, run `sudo tailscale up` and open the link it
+   prints, signed in to the fleet's tailnet.
+2. In the admin console, disable key expiry for the worker, or tag it.
+3. From the manager, run `devfleet fleet-sync`: `ansible/ubuntu/tailscale.yml`
+   verifies the login and `tailscale:key-expiry` passes.
 
 An SSH shell is not proof that VS Code or T3 Connect works. Diagnostics check
 **directed manager → node** connectivity, not all pairs. Run from each intended
@@ -147,7 +177,7 @@ need normal repository validation. A green install is not a healthy fleet.
 5. Authenticate separately: GitHub, Git SSH, Vault, coding providers and T3 pairing
    are distinct sessions. Use gh auth login/setup-git, approved vault login and
    operator-driven t3 up --connect as needed. Never put secrets in reports.
-   Workstation/provider configuration remains an explicit `personal workstation sync` operation.
+   Workstation and provider configuration are applied by `devfleet fleet-sync`.
 6. In VS Code's local profile install Remote-SSH, select the same SSH binary,
    alias/config, enable its login terminal temporarily and inspect Output →
    Remote - SSH. Restart after PATH/agent changes. Check disk space, remote
@@ -178,7 +208,7 @@ serially; diagnostics run at most four remote probes concurrently.
   setup can leave partial installs; no rollback is claimed. Trusted checkout
   package lifecycle scripts can execute during installation.
 - No other application repos, credentials, provider settings or cloud resources
-  are synchronized. `personal workstation update`/`personal workstation sync` remain explicit broader convergence.
+  are synchronized. `devfleet fleet-sync` remains the broader convergence.
 - Run doctor again afterward; successful sync does not prove connectivity/auth.
 
 Subprocesses have deadlines and a combined 1 MiB output cap. SSH probes default
@@ -204,17 +234,43 @@ not a way to certify the whole fleet from one shell. Usage errors use
 exception strings. Fleet never prompts; --non-interactive is accepted.
 
 No telemetry or report file is written automatically. Reports still contain
-aliases/hashes: review before sharing publicly. For monitoring, schedule read-only
-`devfleet fleet-status --json` in your existing task runner and alert on exit 1/2;
-do not schedule automatic sync/repairs. `bun run health` remains the separate
+aliases/hashes: review before sharing publicly. Do not schedule automatic
+sync/repairs.
+
+## Notifications and the heartbeat
+
+`devfleet fleet-watch` runs `fleet-status --workers`, compares the result with
+the previous run and notifies only when a worker changes between ready and
+degraded, and again every `repeatHours` (default 24) while it stays degraded.
+Name the targets in `fleet.json`; the address of each is a secret, read through
+the secret provider like any other:
+
+```json
+"notify": { "targets": ["webhook", "ntfy"], "repeatHours": 24 }
+```
+
+| Target    | Secret               | What is sent                                                         |
+| --------- | -------------------- | -------------------------------------------------------------------- |
+| `webhook` | `notify-webhook-url` | A JSON POST: `source`, `event`, `node`, `checks` (check IDs), `text` |
+| `ntfy`    | `notify-ntfy-url`    | A text POST to the topic URL                                         |
+
+A notification carries node and check IDs only. `devfleet fleet-watch --test`
+sends one test notification to every target.
+
+Every run stores what it saw, and when, as `watch.json` beside the fleet state
+in the bucket. That time is the manager's heartbeat. A watcher on the manager
+notices nothing when the manager itself is off, so call the reusable
+`fleet-heartbeat.yml` workflow from the repository that holds your
+`fleet.json` (its header shows the one `uses:` line): it fails, and GitHub
+notifies you, when the heartbeat is older than `max-age-minutes`. `bun run health` remains the separate
 single read-only **cloud platform** report.
 
 ## Development
 
 ```sh
-bun run --cwd packages/fleet tc
-bun run --cwd packages/fleet lint
-bun run --cwd packages/fleet test
+bun run tc
+bun run lint
+bun test
 ```
 
 Tests inject results/use temporary fixtures; they do not contact nodes, login or
@@ -238,7 +294,7 @@ Explicit `services` requirements retain Railway, Vercel, Cloudflare, AWS and npm
 checks. No raw account/auth output is retained. Version checks keep
 per-tool results, including the expected pinned version when available, and require
 Docker Engine/Compose/Buildx readiness. `workstation` delegates to read-only
-`personal workstation doctor --json --non-interactive`; provider configuration presence is not live
+the checkout's `bootstrap/doctor.ts --json`; provider configuration presence is not live
 authorization. Warnings/skips fail the strict whole-tailnet health coverage gate.
 
 ## Repair planning and shared access
@@ -257,7 +313,7 @@ installation never substitutes for authentication or resource authorization.
 
 The workstation owner reads permitted shared provider keys from each machine's
 existing Vault session. For process-scoped shared secret access, use the existing
-`personal infra-vault run --config dev -- COMMAND` workflow on that machine with its intended
+`vault` session on that machine with its intended
 project/config. Infrastructure, access policy and secret seeding remain owned by
 OpenTofu. No OAuth/session/keychain/SSH credential transfer is implemented.
 
@@ -270,7 +326,7 @@ reviewed configuration; without `--dry-run` it applies. `devfleet fleet-sync
 --revision FULL_SHA` rolls the same reviewed revision out to the enrolled fleet
 after approval. The [setup runbook](../agents/setup.md)
 covers canaries, offline/partial failures, per-node receipts and rollback. Both
-delegate rendering to personal workstation and installation to existing mise/bootstrap.
+delegate installation to mise and `bootstrap/setup.ts`.
 They never reset the developer checkout or transfer credentials. Status delegates
 managed-profile drift evidence to the workstation owner; historical rollout receipts
 alone are not a live readiness assertion.