@crvouga/dev-fleet 0.1.0 → 0.2.0

package/docs/operations.md

@@ -0,0 +1,276 @@
+> This document was written for the fleet dev-fleet was extracted from. Paths and
+> commands were updated; some statements still describe that fleet and are being
+> revised. The README is authoritative.
+
+# Developer fleet
+
+`@pkgs/fleet` diagnoses development computers: SSH, VS Code Remote-SSH,
+per-machine authentication, T3 Connect and shared toolchain/dependency drift.
+It is a local CLI/library, not a daemon or cloud manager. It reuses the
+existing mise/workstation owners rather than introducing another installer.
+
+The fleet is one singleton **manager** (the human's macOS machine, which runs
+`fleet-sync` and every singleton service such as the CLI proxy and Beszel hub)
+and stateless Linux **workers**. See [topology](topology.md).
+
+See [Ubuntu fleet-sync implementation and operator status](ubuntu-sync.md) for
+the manager-driven public command and its outstanding full-machine gates.
+
+## Worker contract
+
+Use `devfleet fleet-sync` to reconcile all declared workers and
+`devfleet fleet-status --workers` to inspect remaining gaps. See
+[worker-contract.md](worker-contract.md) for bootstrap, ownership and the exact
+scope of readiness. `fleet-status` without flags diagnoses the local roster.
+Workers are Ubuntu 26.04; the manager is macOS and never sleeps on AC power
+(`ansible/macos/manager.yml`). Machine policy (never sleep, no automatic
+logout, always-on Tailscale/SSH/Docker/Beszel, memory and disk safety, per-node
+credential renewal) is
+declared in Ansible and measured by `fleet-status --workers`, which also verifies
+the centralized Beszel dashboard. See the
+[platform matrix](ubuntu-sync.md#machine-policy-by-platform).
+
+## Quick start
+
+Run from the repository root after `bun install --frozen-lockfile`:
+
+```sh
+devfleet fleet-node-add --id dev-linux --ssh dev-linux --path /home/dev/workspace --json
+devfleet fleet-node-add --id dev-linux --ssh dev-linux --path /home/dev/workspace --apply --yes --json
+devfleet fleet-nodes --json                        # validate the roster; no network
+devfleet fleet-nodes --onboard --node dev-linux --json
+devfleet fleet-status --json                       # manager + all roster nodes
+devfleet fleet-status --node dev-linux --json
+devfleet fleet-status --local --json               # no roster/SSH needed
+devfleet fleet-repair --node dev-linux --repair workspace --json   # plan only
+devfleet fleet-repair --node dev-linux --repair workspace --apply --yes --json
+```
+
+This private package exposes a TypeScript API from `@pkgs/fleet`; the public
+executable is `personal` (`bun run personal` from the checkout). Schema, roster
+listing, onboarding and repair previews do not contact nodes or auth services.
+Status never opens a login flow.
+
+## Roster
+
+The roster is kept in the [fleet state](state.md) and replicated to its bucket.
+A `.fleet.local.json` found from the current working directory (gitignored), or
+`--config PATH`, names a roster file instead (keep it out of Git yourself). This describes
+**existing developer computers**, not desired cloud infrastructure. OpenTofu
+remains the sole cloud resource owner; fleet creates no networks, hosts or secrets.
+
+```json
+{
+  "$schema": "./packages/fleet/schema.json",
+  "version": 1,
+  "nodes": [
+    {
+      "id": "dev-linux",
+      "ssh": "dev-linux",
+      "workspace": "/home/developer/workspace",
+      "runtime": "/home/developer/.local/share/mise/shims/node",
+      "required": ["github", "vault", "t3", "toolchain"]
+    }
+  ]
+}
+```
+
+- `id`: unique lowercase name, up to 63 characters; 1–64 nodes per roster.
+- `ssh`: existing alias/hostname. Put User, Port, IdentityFile, ProxyJump and
+  keychain configuration in normal SSH config. user@host/IPv6 should use aliases.
+- `workspace`: absolute POSIX checkout path. Spaces/apostrophes work; relative
+  paths, tilde expansion and control characters are rejected.
+- `runtime`: optional absolute Node/Bun executable or mise shim. Defaults to `node` on
+  the **non-login SSH PATH**. Configure explicitly when GUI terminals work but SSH
+  cannot find Node. A shim also needs mise's environment/PATH working.
+- `required`: `github`, `vault`, `t3`, `vscode`, `toolchain`. Defaults to all except
+  vscode. Set vscode on editor/manager machines, not just on headless targets.
+  `[]` still checks checkout/dependencies/SSH. Unknown fields fail closed.
+
+The manager reference defaults to where the CLI runs from: the published
+package, or the dev-fleet checkout a source run is started in. `--workspace PATH`
+changes that reference/local sync target. Nodes are compared to **that checkout**,
+including uncommitted lockfile edits, not an implicitly fetched origin/main.
+Manager capabilities default to all five; headless managers can pass
+`--require github,vault,t3,toolchain` or `--require none`. This does not override
+remote nodes' required lists. Run `fleet-nodes --schema` for editor/agent validation.
+
+## Evidence and diagnosis
+
+| Check                                                                              | Evidence / next action                                                                                                                                |
+| ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `workspace`, `git:clean`                                                           | HEAD, input availability and dirty-tree status; no paths/diffs/file contents logged.                                                                  |
+| `revision`, `sync:*`                                                               | Compare exact HEAD and SHA-256 of package.json, bun.lock, mise.toml, mise.lock against manager. Missing inputs never pass.                            |
+| `dependencies`                                                                     | Last successful fleet frozen-sync receipt matches input hashes. Missing receipt is unverified, not in-sync.                                           |
+| `auth:toolchain`                                                                   | Existing bootstrap/doctor.ts runs in the node checkout with its Bun, bounded to 90s. Use personal workstation tools check --json for individual pins. |
+| `ssh:agent`                                                                        | Manager agent exposes an identity; missing agent warns because IdentityFile keys may still work.                                                      |
+| `ssh:session`                                                                      | Fresh batch SSH session actually runs the Node/Bun probe. No reused multiplexed connection or agent forwarding.                                       |
+| `ssh:host-key`, `ssh:dns`, `ssh:auth`, `ssh:network`, `ssh:runtime`, `ssh:timeout` | Classified failure and targeted remedy, without raw logs.                                                                                             |
+| `auth:github`                                                                      | gh auth status --active succeeds on this machine; not proof of Git SSH or every repository permission.                                                |
+| `auth:vault`                                                                       | Server-side vault token lookup succeeds in this context; not proof of access to every secret path.                                                    |
+| `auth:vscode`                                                                      | code CLI finds the local Remote-SSH extension. Other profiles/Cursor/GUI PATH/agents can differ.                                                      |
+| `t3:settings`                                                                      | Connect explicitly enabled and not revoked in t3 status --json; settings are not connectivity.                                                        |
+| `t3:api`                                                                           | Read-only t3 environment list --json succeeds against the local API; no pairing/token minting.                                                        |
+| `t3:end-to-end`, `vscode:forwarding`                                               | SKIP: actual relay/peer authorization and VS Code tunnel/server installation require operator testing.                                                |
+
+An SSH shell is not proof that VS Code or T3 Connect works. Diagnostics check
+**directed manager → node** connectivity, not all pairs. Run from each intended
+manager to check reverse directions. No host-key acceptance, known_hosts edits,
+sshd changes, agent forwarding, password retries or credential copying happen.
+SSH config/executables/hosts are trusted operator inputs; configured ProxyCommand
+or Match commands can execute the operator's own code.
+
+Fingerprints also cover all flat workspace package manifests and the files under
+`.mise/locks`, so uncommitted dependency/sidecar edits cannot hide behind an
+unchanged root lockfile. Missing/unreadable inputs cannot prove synchronization.
+
+The receipt (`node_modules/.fleet-sync.json`) is historical evidence of completed
+frozen setup, **not a full integrity audit**. Manual node_modules edits/deletions
+can invalidate that assumption; resync when in doubt. Native package builds still
+need normal repository validation. A green install is not a healthy fleet.
+
+## Add a computer
+
+1. Review and run the shared [machine bootstrap](../bootstrap/bootstrap.sh) on
+   the new machine. Reopen terminal/editor. Do not invent a second installer.
+2. As operator, enable OS SSH access on a trusted private network. Generate a
+   distinct machine key, authorize only its public key and verify the server
+   fingerprint out of band. Never expose public SSH or copy private keys/auth
+   folders. Cloud provisioning remains an OpenTofu task.
+3. Configure the manager's SSH alias, append the node to the local roster,
+   then run `fleet-nodes` and `fleet-nodes --onboard --node ID --json`. Onboarding
+   emits a ready-to-edit SSH stanza, VS Code settings and ordered instructions;
+   it does not overwrite user config. Test `ssh ALIAS` manually first.
+4. Ensure noninteractive shell startup is quiet and Node (or explicit Bun runtime) is reachable. Fleet
+   bundles/streams its own dependency-free probe over stdin: the remote computer
+   needs Node or Bun, **not a matching installation of this package** or its dependencies.
+5. Authenticate separately: GitHub, Git SSH, Vault, coding providers and T3 pairing
+   are distinct sessions. Use gh auth login/setup-git, approved vault login and
+   operator-driven t3 up --connect as needed. Never put secrets in reports.
+   Workstation/provider configuration remains an explicit `personal workstation sync` operation.
+6. In VS Code's local profile install Remote-SSH, select the same SSH binary,
+   alias/config, enable its login terminal temporarily and inspect Output →
+   Remote - SSH. Restart after PATH/agent changes. Check disk space, remote
+   server prerequisites and TCP forwarding policy. Onboarding includes a
+   loopback-only forwarding test against a known service, not a public listener.
+7. Run doctor, review remedies and explicitly sync if needed. Actually open the
+   remote workspace in VS Code and test the linked peer in T3 Settings →
+   Connections. These are the end-to-end acceptance gates.
+
+Remote nodes require macOS/Linux/WSL and a **POSIX SSH shell**. Native Windows
+supports local commands/diagnostics; use a WSL SSH endpoint for remote checks.
+Fleet does not guess cmd.exe/PowerShell quoting. Use the existing native Windows
+bootstrap and run local reports when a WSL endpoint is unavailable.
+
+## Explicit sync
+
+`sync` is a plan unless **both --apply --yes** are present. Selected nodes apply
+serially; diagnostics run at most four remote probes concurrently.
+
+- `--pull` requires a clean attached branch, then runs git pull --ff-only against
+  its upstream. No stash/reset/force/rebase/discard. Without it, the branch stays.
+- Invalidate the old receipt, then delegate to
+  `mise exec --locked bun -- bun bootstrap/setup.ts --non-interactive`.
+  This existing owner installs locked tools, publishes user mise config, sets
+  shell PATH and installs frozen Bun dependencies. Review OS-specific effects
+  and existing config conflict guards in the shared bootstrap runbook.
+- Record input hashes only after success and stable checkout inputs. Failed
+  setup can leave partial installs; no rollback is claimed. Trusted checkout
+  package lifecycle scripts can execute during installation.
+- No other application repos, credentials, provider settings or cloud resources
+  are synchronized. `personal workstation update`/`personal workstation sync` remain explicit broader convergence.
+- Run doctor again afterward; successful sync does not prove connectivity/auth.
+
+Subprocesses have deadlines and a combined 1 MiB output cap. SSH probes default
+to 120s per node; --timeout MS allows 1s–30m. Applied sync defaults to a 25m SSH
+envelope and 20m setup deadline. A killed/disconnected SSH session does **not**
+guarantee remote installers stop: inspect the node before retrying. POSIX local
+subprocess groups are killed at their deadline.
+
+## Agent contract
+
+`--json` emits one versioned document on stdout, with no progress/auth/child
+output. Bun's script runner may print its command to stderr. Reports have
+`version: 1`, command, ok, generatedAt, nodes and pass/warn/fail/skip counts.
+Node checks have stable id, status, safe summary and optional remedy.action;
+snapshots include platform and revision/hash fingerprints. Preserve skips and
+warnings: neither is a pass. A plan is not completed work.
+
+Exit 0: no failing checks (warnings/skips remain visible). Exit 1: health or
+operation failure. Exit 2: usage/config error. `doctor --strict` also fails on
+warnings/skips, including intentional end-to-end skips: it is a **coverage gate**,
+not a way to certify the whole fleet from one shell. Usage errors use
+`code: invalid_usage`; unexpected errors use `operation_failed` without exposing
+exception strings. Fleet never prompts; --non-interactive is accepted.
+
+No telemetry or report file is written automatically. Reports still contain
+aliases/hashes: review before sharing publicly. For monitoring, schedule read-only
+`devfleet fleet-status --json` in your existing task runner and alert on exit 1/2;
+do not schedule automatic sync/repairs. `bun run health` remains the separate
+single read-only **cloud platform** report.
+
+## Development
+
+```sh
+bun run --cwd packages/fleet tc
+bun run --cwd packages/fleet lint
+bun run --cwd packages/fleet test
+```
+
+Tests inject results/use temporary fixtures; they do not contact nodes, login or
+change global config. The transport smoke test runs the bundled probe locally
+against a temporary checkout, not an SSH server.
+
+## Tailscale fleet-status --tailnet in the public CLI
+
+`devfleet fleet-discover --json` lists computers visible to the local signed-in
+Tailscale daemon, including offline devices, while explicitly excluding iOS,
+Android and tvOS. `devfleet fleet-status --tailnet --json` checks them all and reports
+unregistered/ambiguous computers and registered nodes missing from the inventory.
+Register actual SSH aliases and checkout paths; the checker does not guess them.
+Existing verified-host-key SSH policy and bounded probes apply. Tailscale daemon
+health errors invalidate complete discovery. ACL-hidden devices cannot be certified.
+
+Health probes each registered remote node's declared `required` capabilities; it
+no longer adds cloud-service logins or direct provider authentication implicitly.
+The manager default covers T3, toolchain and workstation configuration.
+Explicit `services` requirements retain Railway, Vercel, Cloudflare, AWS and npm
+checks. No raw account/auth output is retained. Version checks keep
+per-tool results, including the expected pinned version when available, and require
+Docker Engine/Compose/Buildx readiness. `workstation` delegates to read-only
+`personal workstation doctor --json --non-interactive`; provider configuration presence is not live
+authorization. Warnings/skips fail the strict whole-tailnet health coverage gate.
+
+## Repair planning and shared access
+
+The public CLI exposes `devfleet fleet-repair --list` (static catalog) and
+`devfleet fleet-repair` (read-only inspection/plan by default). The repair library
+in this package only executes allowlisted existing owner commands. Native
+account logins and unknown/manual checks always remain operator steps.
+
+Use `--apply --yes` only after review; `--plan-hash` binds application to the
+previewed steps, targets and checkout/shared-input fingerprints. The remote
+protocol independently requires explicit apply, a supported repair action and
+expected input fingerprints. It preserves verified SSH and disables forwarding.
+After owner commands, the manager obtains fresh health checks. Successful
+installation never substitutes for authentication or resource authorization.
+
+The workstation owner reads permitted shared provider keys from each machine's
+existing Vault session. For process-scoped shared secret access, use the existing
+`personal infra-vault run --config dev -- COMMAND` workflow on that machine with its intended
+project/config. Infrastructure, access policy and secret seeding remain owned by
+OpenTofu. No OAuth/session/keychain/SSH credential transfer is implemented.
+
+All commands are in the `devfleet` entry point.
+
+## Reviewed workstation configuration rollout
+
+`devfleet fleet-node-sync --revision FULL_SHA --dry-run` previews this node's
+reviewed configuration; without `--dry-run` it applies. `devfleet fleet-sync
+--revision FULL_SHA` rolls the same reviewed revision out to the enrolled fleet
+after approval. The [setup runbook](../agents/setup.md)
+covers canaries, offline/partial failures, per-node receipts and rollback. Both
+delegate rendering to personal workstation and installation to existing mise/bootstrap.
+They never reset the developer checkout or transfer credentials. Status delegates
+managed-profile drift evidence to the workstation owner; historical rollout receipts
+alone are not a live readiness assertion.

package/docs/releasing.md

@@ -0,0 +1,103 @@
+# Releasing
+
+dev-fleet is published to npm as `@crvouga/dev-fleet`. The package is how
+dev-fleet is used: a fleet is a project that depends on it and runs `devfleet`
+from there (see the [README](../README.md#how-you-use-it)). Nobody clones this
+repository to run a fleet.
+
+```sh
+npm install --save-dev --save-exact @crvouga/dev-fleet
+npx devfleet fleet-sync
+```
+
+The package ships the TypeScript sources, the Ansible policy, the bootstrap
+scripts and the proxy scripts, because the CLI reads them at run time. Node
+will not run TypeScript from `node_modules`, so the `devfleet` entry is a small
+JavaScript launcher (`bin/devfleet.mjs`) that runs the CLI on the Bun this
+release pins (a dependency of the package), falling back to a Bun on `PATH`.
+
+Machines do not run the package. `fleet-sync` rebuilds every machine from one
+git commit of this repository, and the package says which: the Release
+workflow writes the commit it publishes from into the package's `package.json`
+as `gitHead` ([`src/package-info.ts`](../src/package-info.ts) reads it). Run
+from the package, `fleet-sync` fetches that commit from the repository
+`fleet.json` names under `source` into a cache the manager owns, and converges
+the manager and every worker from checkouts of it. So a version of the package
+is a revision of the fleet, and the commit a release names must stay reachable
+in the source repository.
+
+## What a release is
+
+Every merge to `main` is a release. When CI passes on `main`, the
+[Release workflow](../.github/workflows/release.yml) picks the next version from
+the commits since the newest `vX.Y.Z` tag (or, before there is one, the newest
+version on npm)
+([`bootstrap/release-version.ts`](../bootstrap/release-version.ts)), publishes it
+to `@crvouga/dev-fleet` with provenance, pushes the tag and creates the GitHub
+Release. The commit messages choose the bump:
+
+| Commits since the last tag                     | Bump                                 |
+| ---------------------------------------------- | ------------------------------------ |
+| any `feat!:`, `fix(x)!:` or `BREAKING CHANGE:` | major (minor while the major is `0`) |
+| otherwise any `feat:`                          | minor                                |
+| otherwise                                      | patch                                |
+
+`package.json` stays at `0.0.0-development` and carries no `gitHead`; the
+workflow stamps the version and the commit into the package it publishes, so
+nobody edits a version or runs `npm publish` by hand after the one-time setup
+below. Merges that land while a release runs
+wait, and the newest one releases everything since the last tag. A run that
+cannot release because publishing was never set up (no tag and nothing on
+npm) fails, and the run summary says why; a merge with nothing new to release
+passes and says so. Re-running a
+release whose version is already on npm skips the publish and finishes the tag
+and GitHub Release.
+
+A hand-pushed tag still works and publishes exactly that version:
+
+```sh
+git tag vX.Y.Z && git push origin vX.Y.Z
+```
+
+CI never receives an npm token: it publishes through npm Trusted Publishing
+(OIDC).
+
+## One-time setup
+
+npm can publish new versions through OIDC but cannot create a package that does
+not exist yet. One command does the one-time work with your own npm login, from
+`main` with a clean tree:
+
+```sh
+bun run bootstrap
+```
+
+It signs in to npm (browser approval), runs `bun run check`, publishes `0.1.0`
+from a throwaway checkout of `HEAD` (so your working tree is never edited),
+makes `release.yml` the package's trusted publisher, and tags `v0.1.0` on the
+commit npm recorded. When npm recorded none (as for the `0.1.0` that is on
+npm), there is nothing to tag: until the first tag exists, the Release
+workflow counts from the newest version on npm and every commit up to `HEAD`
+picks the bump. Every step measures first and skips what is already done,
+so run it again after any interruption. It needs npm 11.10 or newer (for
+`npm trust`) and an npm account that owns the `@crvouga` scope. Flags: `--yes`
+skips the confirmation, `--skip-check` skips `bun run check`.
+
+Verify from a machine that has never seen the package:
+
+```sh
+npx --yes @crvouga/dev-fleet --version
+```
+
+If `npm trust` is unavailable, set the publisher in the browser instead:
+<https://www.npmjs.com/package/@crvouga/dev-fleet/access> → Trusted Publisher →
+GitHub Actions → organization `crvouga`, repository `dev-fleet`, workflow
+filename `release.yml`, environment empty.
+
+## Why the package looks like this
+
+- `files` is an allow-list; tests and Python caches are excluded. `features/rules/package.feature`
+  fails if a file the CLI reads is missing from the tarball.
+- `bun` is a dependency pinned to the same version as `packageManager` and the
+  `mise.toml` pin; bump the three together.
+- The provenance statement links each version to the workflow run that built it.

package/docs/state.md

@@ -0,0 +1,254 @@
+# Fleet state
+
+A fleet is described in two places you own. Your project in Git holds
+`fleet.json` and pins the dev-fleet version. Everything else the manager
+cannot rebuild from Git is the **fleet state**:
+
+| Document     | What it is                                                           |
+| ------------ | -------------------------------------------------------------------- |
+| `roster`     | The enrolled connection to each worker (SSH alias, Tailscale device) |
+| `enrollment` | The revision `fleet-sync` approved and the one it last applied       |
+| `definition` | The fleet configuration `fleet-sync` last applied: your `fleet.json` |
+
+They live in one SQLite database, and [Litestream](https://litestream.io)
+replicates that database to a storage bucket you own. The bucket is the source
+of truth. The copy on the manager is disposable: delete it, or replace the
+whole machine, and the next command brings the fleet back.
+
+```
+your project (Git)
+    fleet.json                          the configuration; says where the bucket is
+    package.json                        the dev-fleet version
+~/.local/share/dev-fleet/state/         the local copy of the state; safe to delete
+    fleet.db                            the database Litestream replicates
+    definition.json                     the stored configuration as a file, rewritten from fleet.db
+    replicated.json, litestream.yml     bookkeeping
+~/.config/dev-fleet/bucket.json         optional: credentials saved for unattended runs
+```
+
+## Set it up
+
+Any S3-compatible bucket works: AWS S3, Cloudflare R2, Backblaze B2, MinIO.
+Create an empty private bucket and an access key that can read and write it.
+Name the bucket in `fleet.json`:
+
+```json
+"state": {
+  "bucket": "my-fleet-state",
+  "endpoint": "https://ACCOUNT.r2.cloudflarestorage.com",
+  "region": "auto",
+  "prefix": "dev-fleet"
+}
+```
+
+Omit `endpoint` for AWS S3. `prefix` defaults to `dev-fleet`; one bucket can
+hold several fleets under different prefixes. The block never holds
+credentials, and a `fleet.json` that does is refused.
+
+Every devfleet command reads the credentials from its environment:
+
+```sh
+export DEVFLEET_BUCKET_ACCESS_KEY_ID=…
+export DEVFLEET_BUCKET_SECRET_ACCESS_KEY=…
+npx devfleet fleet-sync
+```
+
+Set both or neither. With neither, and nothing saved on the machine, the
+standard AWS environment or profile is used. Credentials are never taken from
+arguments and never written to a report or a log.
+
+**A new fleet.** Nothing more: the first `fleet-sync` seeds the bucket.
+
+**A replacement manager, or a second machine.** Clone your project, `npm ci`,
+and run `fleet-sync` with the credentials. The bucket already holds the fleet,
+so its state is downloaded before anything is changed.
+
+**Runs the manager starts by itself.** A worker that hands `fleet-sync` to the
+manager starts a command there that gets no environment and no project. Save
+the credentials on the manager once:
+
+```sh
+npx devfleet fleet-state --init
+```
+
+It takes the bucket from `fleet.json` and the credentials from the environment,
+or asks for them on the terminal, and writes both to
+`~/.config/dev-fleet/bucket.json` (readable by you alone). Such a run then
+uses the configuration last applied and the saved credentials. Without them it
+still converges the fleet from the local copy and reports that the bucket was
+not reached; the next run that has credentials uploads what is pending.
+
+**A fleet that kept its configuration in the bucket.** Before projects, the
+configuration was stored with `fleet-state --import` and the bucket was named
+with `fleet-state --init --bucket …`. To move such a fleet into a project, on
+the manager:
+
+```sh
+mkdir my-fleet && cd my-fleet && git init && npm init -y
+npm install --save-dev --save-exact @crvouga/dev-fleet
+npx devfleet fleet-state --export > fleet.json
+```
+
+Add the `state` block to `fleet.json` with the same bucket, endpoint, region
+and prefix as `~/.config/dev-fleet/bucket.json`, then run
+`npx devfleet fleet-sync`. The credentials already saved for that bucket keep
+working.
+
+The clone of dev-fleet the fleet used to run from is then no longer read by
+`fleet-sync`, with one exception: the CLI proxy's installed code is a Git
+worktree of the repository it was installed from. Install the proxy again
+from the project (`npx devfleet cliproxy install`, which takes the revision
+`fleet-sync` applied from the manager's own copy of the source) before you
+delete the clone.
+
+Without a `fleet.json`, `fleet-state --init --bucket NAME --endpoint https://HOST`
+still connects a bucket and restores the configuration stored in it.
+
+On a manager older than the fleet state itself, `--init` (or the first
+`fleet-sync`) also moves the configuration, the roster and the approved
+revision from the files that held them into the state. Those files are left in
+place and no longer read.
+
+`"state": { "directory": "/Volumes/backup/fleet" }` replicates to a directory
+instead of a bucket (an attached disk, a mounted share). It behaves the same
+way, needs no credentials and is the quickest way to try this out.
+
+## Day to day
+
+```sh
+devfleet fleet-state            # read-only: where the state is, and how it stands with the bucket
+devfleet fleet-state --example  # print a fleet.json to start a project from
+devfleet fleet-state --export   # print the configuration fleet-sync last applied
+devfleet fleet-state --push     # upload changes made while the bucket was unreachable
+devfleet fleet-state --restore  # replace the local copy with the bucket's
+```
+
+To change the fleet, for example to add a worker, edit `fleet.json`, commit
+and run `devfleet fleet-sync` from the project. `fleet-sync` validates the
+file, installs it on every machine and records it in the state.
+
+Which configuration a command uses, in this order:
+
+1. The file `DEVFLEET_CONFIG` names.
+2. The closest `fleet.json` in the working directory or above it.
+3. The configuration `fleet-sync` last applied, from the fleet state (the
+   devfleet CLI on the manager).
+4. The copy `fleet-sync` installed at `~/.config/dev-fleet/fleet.json`
+   (launchers and scheduled jobs on every machine read only this one).
+
+`fleet-state --import PATH` stores a configuration in the state without
+running `fleet-sync`.
+
+## How it stays correct
+
+Litestream does not run as a daemon. A devfleet command that changes the state
+runs it once, so the command has not finished until the bucket has the change
+and nothing has to be kept running on the manager. Every change follows the
+same three steps:
+
+1. **Download** the bucket's current version and compare it with the local copy.
+2. **Change** the local copy, on top of the bucket's version.
+3. **Upload**, then download again. The change counts only once a restore from
+   the bucket returns it.
+
+`fleet-sync` does step 1 before it reads the configuration and step 3 after
+it records what it applied. Its report carries the result as the
+`state:replica` check.
+
+`fleet-state` reports one of these:
+
+| Standing       | Meaning                                                | What happens                                               |
+| -------------- | ------------------------------------------------------ | ---------------------------------------------------------- |
+| `in-sync`      | The bucket and this machine hold the same version      | Nothing to do                                              |
+| `local-only`   | No bucket is configured                                | A warning on every `fleet-sync`; name one in `fleet.json`  |
+| `empty`        | Neither side holds any state                           | The first `fleet-sync` records it                          |
+| `unpublished`  | The bucket holds nothing yet                           | The next change, or `--push`, seeds it                     |
+| `local-ahead`  | This machine has changes the bucket has not received   | The next change, or `--push`, uploads them                 |
+| `bucket-ahead` | Another manager changed the bucket; this copy is stale | The next change or `fleet-sync` takes the bucket's version |
+| `unreachable`  | The bucket did not answer, or did not take an upload   | Changes still apply locally and are uploaded later         |
+| `diverged`     | Both sides changed                                     | Refused. `--restore --yes` takes the bucket's version      |
+
+A stale copy never overwrites the bucket. A machine that was offline while
+another one changed the fleet is brought up to date before it can change
+anything, and if it also changed something itself it stops and says so. The
+bucket wins: export what you still need, then `--restore --yes`.
+
+An unreachable bucket does not block `fleet-sync`. The fleet converges, the
+report warns, and the next command uploads what is pending.
+
+## What is not in the bucket yet
+
+The fleet state covers what `fleet-sync` itself needs. The manager also holds
+state that is replicated differently or not at all:
+
+| State                                                | Where                                                    | If the manager is lost                                                                                 |
+| ---------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
+| Secrets                                              | Your `secrets` provider (`vault`, `sops`, `file`, `env`) | Unaffected for `vault`; `file` and `env` secrets must be supplied again                                |
+| CLI proxy: provider logins, keys, Tailscale identity | `~/Library/Application Support/cliproxy-fleet/state`     | Restored from its own encrypted restic snapshots ([cliproxy](../cliproxy/README.md)); needs `vault`    |
+| Monitoring hub (Beszel) database                     | `~/Library/Application Support/personal-beszel`          | `fleet-sync` recreates accounts and systems; metric history, alerts and settings made by hand are lost |
+| T3 ownership record                                  | `~/.t3/userdata/personal-managed.json`                   | `t3-apply --import ID` adopts instances again                                                          |
+
+Everything else under `~/.local/share/personal/fleet` and
+`~/.local/state/personal/fleet` (the copy of the source repository, run
+checkouts, reports, receipts, logs) is a cache that the next `fleet-sync`
+rebuilds. The CLI proxy's installed code is a Git worktree of that copy of the
+source repository: after deleting it, run `devfleet cliproxy install` again
+once `fleet-sync` has fetched it back.
+
+## Reference
+
+| Setting                                                              | Meaning                                                      |
+| -------------------------------------------------------------------- | ------------------------------------------------------------ |
+| `state` in `fleet.json`                                              | Where the bucket is                                          |
+| `DEVFLEET_BUCKET_ACCESS_KEY_ID`, `DEVFLEET_BUCKET_SECRET_ACCESS_KEY` | The bucket's credentials, for the command they are passed to |
+| `DEVFLEET_CONFIG`                                                    | A configuration file, instead of the project's `fleet.json`  |
+| `DEVFLEET_STATE`                                                     | Another state directory. Earlier files are not read for it   |
+| `DEVFLEET_BUCKET`                                                    | Another `bucket.json`                                        |
+
+How a command reaches the bucket:
+
+- **Where it is**: the `state` block of the configuration; without one, the
+  bucket saved in `bucket.json`.
+- **Credentials**: the two variables; otherwise the ones saved in
+  `bucket.json`, only when they were saved for that same bucket; otherwise the
+  standard AWS environment or profile.
+
+`fleet-state --json` reports both as `bucket.location` (`fleet.json` or
+`bucket.json`) and `bucket.credentials` (`environment`, `bucket.json` or
+`aws`), never a credential.
+
+`bucket.json`, written by `fleet-state --init`:
+
+```json
+{
+  "version": 1,
+  "bucket": "my-fleet-state",
+  "endpoint": "https://ACCOUNT.r2.cloudflarestorage.com",
+  "region": "auto",
+  "prefix": "dev-fleet",
+  "accessKeyId": "…",
+  "secretAccessKey": "…"
+}
+```
+
+It must be readable by you alone (`chmod 600`); devfleet refuses to use it
+otherwise. The keys reach Litestream through its environment and are never
+written to its configuration, a report or a log. To rotate them, pass the new
+ones to `fleet-state --init`.
+
+The database is stored under `PREFIX/fleet.db/`. Each change adds a few
+kilobytes and Litestream keeps thirty days of snapshots, so a restore to an
+earlier point is possible with Litestream directly (with the bucket's keys in
+`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`):
+
+```sh
+mise exec litestream -- litestream restore -no-expand-env \
+  -config ~/.local/share/dev-fleet/state/litestream.yml \
+  -timestamp 2026-11-01T00:00:00Z -o /tmp/fleet-then.db \
+  ~/.local/share/dev-fleet/state/fleet.db
+```
+
+Litestream is pinned in `mise.toml` and installed the first time it is used.
+The manager is the only machine that runs it. There is one manager, so there is
+one writer; two machines changing the state at the same moment are not
+supported, only detected afterwards.