@crvouga/dev-fleet 0.3.0 → 0.4.0

package/docs/machine-setup.md

@@ -181,19 +181,21 @@ wrangler login
 railway login
 ```
 
-Only sign into providers the task actually needs. Git identity is personal:
-set `git config --global user.name` and `git config --global user.email` after
-confirming the desired name/email. Do not guess an email or overwrite an existing
-identity. Enable Git LFS hooks when needed with `git lfs install`.
+Only sign into providers the task actually needs. On fleet machines the Git
+identity and the GitHub credential helper come from the `git` block of
+`fleet.json`: `fleet-sync` writes them in `~/.config/git/config` and checks
+them as `worker:git-identity` and `worker:git-push-auth`. Elsewhere set `git
+config --global user.name` and `user.email` after confirming them; do not guess
+an email or overwrite an existing identity. Enable Git LFS hooks when needed with `git lfs install`.
 
 Optional existing workstation/agent settings:
 
 ```sh
 sh bootstrap/install-launcher.sh
-personal workstation doctor
+devfleet fleet-status --local
 # Only for tasks needing the workspace's managed secrets:
 vault login --method=userpass username=example-user
-personal workstation sync
+devfleet fleet-sync
 ```
 
 The machine bootstrap installs tools/dependencies only. `personal:install` additionally
@@ -231,7 +233,7 @@ The manifest installs the **Docker client**. Containers also require a local or
 remote Docker engine, Compose v2 and Buildx plugins. These cannot be installed as
 one identical unprivileged binary on all OSes.
 
-- **macOS:** run `personal workstation tools docker-setup` to install the pinned,
+- **macOS:** run `bash bootstrap/docker-desktop.sh` to install the pinned,
   checksum-verified Docker Desktop bundle from this repository and open it.
   Complete its first-run agreement/setup in the app. Repeated invocation reuses
   an existing installation. The shared mise Docker client remains in use.
@@ -280,11 +282,11 @@ mise install --locked bun
 mise exec --locked bun -- bun bootstrap/setup.ts
 ```
 
-For an installed workstation CLI, `personal workstation update` pulls with a clean-tree/fast-forward
-guard, refreshes the toolchain/global config and frozen dependencies, and runs the
-existing workstation convergence. `personal workstation tools sync` reapplies tool pins/global config
-without pulling or touching project dependencies; `personal workstation tools check --json` verifies
-them. Sync is explicit; Git commits do not silently update other machines.
+On fleet machines `devfleet fleet-sync` refreshes the toolchain, the global
+configuration and frozen dependencies; `mise exec --locked bun -- bun
+bootstrap/setup.ts --tools-only` reapplies tool pins and the global
+configuration without touching project dependencies, and `bun
+bootstrap/doctor.ts --json` verifies them. Sync is explicit; Git commits do not silently update other machines.
 
 For multiple development computers, use [the fleet package](operations.md):
 `devfleet fleet-node-add` registers an existing SSH alias in a local roster,
@@ -395,6 +397,14 @@ It installs the `vars.worker_tools` subset from `mise.toml`, with versions and
 artifacts taken from the same `mise.lock`; native T3 and Ubuntu Docker/service
 policy remain source-pinned. `bun bootstrap/doctor.ts --worker --docker --json`
 checks that profile. It does not require unrelated cloud CLIs on workers.
+
+Tool profiles add locked tool sets on top of it: `worker.toolProfiles` in
+`fleet.json` (fleet-wide) or `toolProfiles` on one worker selects `web` (pnpm,
+Deno, Yarn and headless Chromium for browser tests), `go`, `rust` (with CMake
+and Ninja) or `jvm` (Temurin), each a `worker_profile_<name>` under `[vars]` in
+`mise.toml`. `worker.tools` adds single tools dev-fleet already pins. The setup
+installs them (`--node` reads the installed configuration) and the doctor
+checks each as `tool:<name>` (`--profiles`, `--tools`).
 The source-owned `worker-auth.ts` timer renews the worker's existing Vault token
 and can restore the declared GitHub account from its authorized Vault service
 credential, without copying another machine's login directory.

package/docs/operations.md

@@ -4,9 +4,9 @@
 
 # Developer fleet
 
-`@pkgs/fleet` diagnoses development computers: SSH, VS Code Remote-SSH,
+`devfleet` diagnoses development computers: SSH, VS Code Remote-SSH,
 per-machine authentication, T3 Connect and shared toolchain/dependency drift.
-It is a local CLI/library, not a daemon or cloud manager. It reuses the
+It is a local CLI, not a library, a daemon or a cloud manager. It reuses the
 existing mise/workstation owners rather than introducing another installer.
 
 The fleet is one singleton **manager** (the human's macOS machine, which runs
@@ -46,8 +46,9 @@ devfleet fleet-repair --node dev-linux --repair workspace --json   # plan only
 devfleet fleet-repair --node dev-linux --repair workspace --apply --yes --json
 ```
 
-This private package exposes a TypeScript API from `@pkgs/fleet`; the public
-executable is `personal` (`bun run personal` from the checkout). Schema, roster
+dev-fleet is a command with no library API: run it as `npx devfleet` from the
+project that holds your `fleet.json` (`./devfleet` from a checkout of dev-fleet
+itself). Schema, roster
 listing, onboarding and repair previews do not contact nodes or auth services.
 Status never opens a login flow.
 
@@ -61,7 +62,7 @@ remains the sole cloud resource owner; fleet creates no networks, hosts or secre
 
 ```json
 {
-  "$schema": "./packages/fleet/schema.json",
+  "$schema": "./node_modules/@crvouga/dev-fleet/schema/roster.schema.json",
   "version": 1,
   "nodes": [
     {
@@ -97,21 +98,50 @@ remote nodes' required lists. Run `fleet-nodes --schema` for editor/agent valida
 
 ## Evidence and diagnosis
 
-| Check                                                                              | Evidence / next action                                                                                                                                |
-| ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `workspace`, `git:clean`                                                           | HEAD, input availability and dirty-tree status; no paths/diffs/file contents logged.                                                                  |
-| `revision`, `sync:*`                                                               | Compare exact HEAD and SHA-256 of package.json, bun.lock, mise.toml, mise.lock against manager. Missing inputs never pass.                            |
-| `dependencies`                                                                     | Last successful fleet frozen-sync receipt matches input hashes. Missing receipt is unverified, not in-sync.                                           |
-| `auth:toolchain`                                                                   | Existing bootstrap/doctor.ts runs in the node checkout with its Bun, bounded to 90s. Use personal workstation tools check --json for individual pins. |
-| `ssh:agent`                                                                        | Manager agent exposes an identity; missing agent warns because IdentityFile keys may still work.                                                      |
-| `ssh:session`                                                                      | Fresh batch SSH session actually runs the Node/Bun probe. No reused multiplexed connection or agent forwarding.                                       |
-| `ssh:host-key`, `ssh:dns`, `ssh:auth`, `ssh:network`, `ssh:runtime`, `ssh:timeout` | Classified failure and targeted remedy, without raw logs.                                                                                             |
-| `auth:github`                                                                      | gh auth status --active succeeds on this machine; not proof of Git SSH or every repository permission.                                                |
-| `auth:vault`                                                                       | Server-side vault token lookup succeeds in this context; not proof of access to every secret path.                                                    |
-| `auth:vscode`                                                                      | code CLI finds the local Remote-SSH extension. Other profiles/Cursor/GUI PATH/agents can differ.                                                      |
-| `t3:settings`                                                                      | Connect explicitly enabled and not revoked in t3 status --json; settings are not connectivity.                                                        |
-| `t3:api`                                                                           | Read-only t3 environment list --json succeeds against the local API; no pairing/token minting.                                                        |
-| `t3:end-to-end`, `vscode:forwarding`                                               | SKIP: actual relay/peer authorization and VS Code tunnel/server installation require operator testing.                                                |
+| Check                                                                              | Evidence / next action                                                                                                                                                |
+| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `workspace`, `git:clean`                                                           | HEAD, input availability and dirty-tree status; no paths/diffs/file contents logged.                                                                                  |
+| `revision`, `sync:*`                                                               | Compare exact HEAD and SHA-256 of package.json, bun.lock, mise.toml, mise.lock against manager. Missing inputs never pass.                                            |
+| `dependencies`                                                                     | Last successful fleet frozen-sync receipt matches input hashes. Missing receipt is unverified, not in-sync.                                                           |
+| `auth:toolchain`                                                                   | Existing bootstrap/doctor.ts runs in the node checkout with its Bun, bounded to 90s. Run `bun bootstrap/doctor.ts --json` in the node's checkout for individual pins. |
+| `ssh:agent`                                                                        | Manager agent exposes an identity; missing agent warns because IdentityFile keys may still work.                                                                      |
+| `ssh:session`                                                                      | Fresh batch SSH session actually runs the Node/Bun probe. No reused multiplexed connection or agent forwarding.                                                       |
+| `ssh:host-key`, `ssh:dns`, `ssh:auth`, `ssh:network`, `ssh:runtime`, `ssh:timeout` | Classified failure and targeted remedy, without raw logs.                                                                                                             |
+| `auth:github`                                                                      | gh auth status --active succeeds on this machine; not proof of Git SSH or every repository permission.                                                                |
+| `auth:vault`                                                                       | Server-side vault token lookup succeeds in this context; not proof of access to every secret path.                                                                    |
+| `auth:vscode`                                                                      | code CLI finds the local Remote-SSH extension. Other profiles/Cursor/GUI PATH/agents can differ.                                                                      |
+| `t3:settings`                                                                      | Connect explicitly enabled and not revoked in t3 status --json; settings are not connectivity.                                                                        |
+| `t3:api`                                                                           | Read-only t3 environment list --json succeeds against the local API; no pairing/token minting.                                                                        |
+| `t3:end-to-end`, `vscode:forwarding`                                               | SKIP: actual relay/peer authorization and VS Code tunnel/server installation require operator testing.                                                                |
+
+### Machine health
+
+| Check                        | Evidence / next action                                                                                                                                                                                                                                                                                                                             |
+| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `worker:capacity:inodes`     | Free inodes on the home filesystem, against `inodeWarnFreePercent`/`inodeFailFreePercent` in `ansible/ubuntu/policy.json`. The disk guard and disk collector act on the same floors.                                                                                                                                                               |
+| `worker:clock`               | `timedatectl` says the clock is synchronised and a time service is enabled; with a reference time, the offset is held to `clock` in the policy.                                                                                                                                                                                                    |
+| `worker:reboot-pending`      | `/var/run/reboot-required` exists: warns with its age, fails after `rebootPendingFailDays`. Reboot when no agent turn is running.                                                                                                                                                                                                                  |
+| `tailscale:key-expiry`       | The node key expiry Tailscale reports: warns 14 days ahead, fails 3 days ahead. See [Tailscale key expiry](#tailscale-key-expiry).                                                                                                                                                                                                                 |
+| `manager:release`            | `sw_vers` against `minimumMajorVersion` in `ansible/macos/policy.json`.                                                                                                                                                                                                                                                                            |
+| `manager:updates`            | `softwareupdate --list --no-scan` (nothing is downloaded): warns on pending updates, and names one that waits for a restart. Names never shown.                                                                                                                                                                                                    |
+| `manager:disk`               | Free space on the data volume and the space Docker holds, against `disk` in `ansible/macos/policy.json`.                                                                                                                                                                                                                                           |
+| `manager:container-watchdog` | The container watchdog (`ansible/monitoring/container-watchdog.py`, a launch agent `hub-darwin.yml` installs) restarts a labelled container that was unhealthy for `unhealthyChecks` runs, at most `maxRestartsPerHour` times an hour (`containerWatchdog` in `ansible/macos/policy.json`). Warns after a restart; fails once it stops restarting. |
+
+### Tailscale key expiry
+
+A node drops off the tailnet on the day its key expires. Keep workers from
+expiring: tag them, or disable key expiry for each in the Tailscale admin
+console (tagged nodes do not expire). `tailscale:key-expiry` warns two weeks
+ahead and fails three days ahead.
+
+To re-join a worker whose key expired, a person signs it in again, because
+only a person's identity can authorise a device:
+
+1. On the worker's console, run `sudo tailscale up` and open the link it
+   prints, signed in to the fleet's tailnet.
+2. In the admin console, disable key expiry for the worker, or tag it.
+3. From the manager, run `devfleet fleet-sync`: `ansible/ubuntu/tailscale.yml`
+   verifies the login and `tailscale:key-expiry` passes.
 
 An SSH shell is not proof that VS Code or T3 Connect works. Diagnostics check
 **directed manager → node** connectivity, not all pairs. Run from each intended
@@ -147,7 +177,7 @@ need normal repository validation. A green install is not a healthy fleet.
 5. Authenticate separately: GitHub, Git SSH, Vault, coding providers and T3 pairing
    are distinct sessions. Use gh auth login/setup-git, approved vault login and
    operator-driven t3 up --connect as needed. Never put secrets in reports.
-   Workstation/provider configuration remains an explicit `personal workstation sync` operation.
+   Workstation and provider configuration are applied by `devfleet fleet-sync`.
 6. In VS Code's local profile install Remote-SSH, select the same SSH binary,
    alias/config, enable its login terminal temporarily and inspect Output →
    Remote - SSH. Restart after PATH/agent changes. Check disk space, remote
@@ -178,7 +208,7 @@ serially; diagnostics run at most four remote probes concurrently.
   setup can leave partial installs; no rollback is claimed. Trusted checkout
   package lifecycle scripts can execute during installation.
 - No other application repos, credentials, provider settings or cloud resources
-  are synchronized. `personal workstation update`/`personal workstation sync` remain explicit broader convergence.
+  are synchronized. `devfleet fleet-sync` remains the broader convergence.
 - Run doctor again afterward; successful sync does not prove connectivity/auth.
 
 Subprocesses have deadlines and a combined 1 MiB output cap. SSH probes default
@@ -204,17 +234,43 @@ not a way to certify the whole fleet from one shell. Usage errors use
 exception strings. Fleet never prompts; --non-interactive is accepted.
 
 No telemetry or report file is written automatically. Reports still contain
-aliases/hashes: review before sharing publicly. For monitoring, schedule read-only
-`devfleet fleet-status --json` in your existing task runner and alert on exit 1/2;
-do not schedule automatic sync/repairs. `bun run health` remains the separate
+aliases/hashes: review before sharing publicly. Do not schedule automatic
+sync/repairs.
+
+## Notifications and the heartbeat
+
+`devfleet fleet-watch` runs `fleet-status --workers`, compares the result with
+the previous run and notifies only when a worker changes between ready and
+degraded, and again every `repeatHours` (default 24) while it stays degraded.
+Name the targets in `fleet.json`; the address of each is a secret, read through
+the secret provider like any other:
+
+```json
+"notify": { "targets": ["webhook", "ntfy"], "repeatHours": 24 }
+```
+
+| Target    | Secret               | What is sent                                                         |
+| --------- | -------------------- | -------------------------------------------------------------------- |
+| `webhook` | `notify-webhook-url` | A JSON POST: `source`, `event`, `node`, `checks` (check IDs), `text` |
+| `ntfy`    | `notify-ntfy-url`    | A text POST to the topic URL                                         |
+
+A notification carries node and check IDs only. `devfleet fleet-watch --test`
+sends one test notification to every target.
+
+Every run stores what it saw, and when, as `watch.json` beside the fleet state
+in the bucket. That time is the manager's heartbeat. A watcher on the manager
+notices nothing when the manager itself is off, so call the reusable
+`fleet-heartbeat.yml` workflow from the repository that holds your
+`fleet.json` (its header shows the one `uses:` line): it fails, and GitHub
+notifies you, when the heartbeat is older than `max-age-minutes`. `bun run health` remains the separate
 single read-only **cloud platform** report.
 
 ## Development
 
 ```sh
-bun run --cwd packages/fleet tc
-bun run --cwd packages/fleet lint
-bun run --cwd packages/fleet test
+bun run tc
+bun run lint
+bun test
 ```
 
 Tests inject results/use temporary fixtures; they do not contact nodes, login or
@@ -238,7 +294,7 @@ Explicit `services` requirements retain Railway, Vercel, Cloudflare, AWS and npm
 checks. No raw account/auth output is retained. Version checks keep
 per-tool results, including the expected pinned version when available, and require
 Docker Engine/Compose/Buildx readiness. `workstation` delegates to read-only
-`personal workstation doctor --json --non-interactive`; provider configuration presence is not live
+the checkout's `bootstrap/doctor.ts --json`; provider configuration presence is not live
 authorization. Warnings/skips fail the strict whole-tailnet health coverage gate.
 
 ## Repair planning and shared access
@@ -257,7 +313,7 @@ installation never substitutes for authentication or resource authorization.
 
 The workstation owner reads permitted shared provider keys from each machine's
 existing Vault session. For process-scoped shared secret access, use the existing
-`personal infra-vault run --config dev -- COMMAND` workflow on that machine with its intended
+`vault` session on that machine with its intended
 project/config. Infrastructure, access policy and secret seeding remain owned by
 OpenTofu. No OAuth/session/keychain/SSH credential transfer is implemented.
 
@@ -270,7 +326,7 @@ reviewed configuration; without `--dry-run` it applies. `devfleet fleet-sync
 --revision FULL_SHA` rolls the same reviewed revision out to the enrolled fleet
 after approval. The [setup runbook](../agents/setup.md)
 covers canaries, offline/partial failures, per-node receipts and rollback. Both
-delegate rendering to personal workstation and installation to existing mise/bootstrap.
+delegate installation to mise and `bootstrap/setup.ts`.
 They never reset the developer checkout or transfer credentials. Status delegates
 managed-profile drift evidence to the workstation owner; historical rollout receipts
 alone are not a live readiness assertion.