@crvouga/dev-fleet 0.8.0 → 0.9.1

package/docs/authentication.md

@@ -8,14 +8,14 @@ rewrites them).
 
 ## Where credentials live
 
-| Credential                          | Lives in                                                                                                                      | Read by                                          |
-| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
-| The fleet's secrets (table below)   | The provider `secrets` names in `fleet.json`                                                                                  | The node that needs it, through `src/secrets.ts` |
-| The bucket's two keys               | The environment of the `devfleet` command; `~/.config/dev-fleet/bucket.json` (0600) on the manager after `fleet-state --init` | The manager only                                 |
-| A node's Vault token                | That node's own Vault session (`~/.vault-token`)                                                                              | That node only; never copied                     |
-| A node's GitHub login               | `gh`'s own store on that node                                                                                                 | `gh`, and git through its credential helper      |
-| Provider accounts (Claude, Codex …) | The CLI proxy's store on the manager                                                                                          | The proxy; nodes hold only the proxy client key  |
-| Tailscale and T3 pairing            | Each machine, signed in by a person                                                                                           | That machine                                     |
+| Credential                          | Lives in                                                                                                                       | Read by                                          |
+| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------ |
+| The fleet's secrets (table below)   | The provider `secrets` names in `fleet.json`                                                                                   | The node that needs it, through `src/secrets.ts` |
+| The bucket's two keys               | The environment of the `devfleet` command; `~/.config/dev-fleet/bucket.json` (0600) on the manager after `fleet-state --init`  | The manager only                                 |
+| A node's Vault token                | That node's own Vault session (`~/.vault-token`)                                                                               | That node only; never copied                     |
+| A node's GitHub login               | `gh`'s own store on that node                                                                                                  | `gh`, and git through its credential helper      |
+| Provider accounts (Claude, Codex …) | The CLI proxy's store on the manager                                                                                           | The proxy; nodes hold only the proxy client key  |
+| Tailscale and T3 pairing            | Each machine, signed in by a person; a replaced worker's T3 is paired by the manager with a one-time code it never writes down | That machine                                     |
 
 ## How they reach each node
 
@@ -30,14 +30,19 @@ rewrites them).
   makes `gh` git's credential helper and writes the declared identity
   (`git` in `fleet.json`, `ansible/github/git.yml`) on every node.
 - **The proxy client key** is read by each node from its own secrets provider
-  whenever a launcher starts an agent.
+  whenever a launcher starts an agent. The node keeps the last key it read in
+  `~/.cache/dev-fleet/proxy-client-key` (0600) and uses it only while the
+  provider cannot answer, so an unreachable Vault or an expired session leaves
+  that node's agents working until the Vault checks below are fixed.
 - **The monitoring hub's backup storage keys** are read on the manager by
   `fleet-sync` and handed to the hub's playbook in a private variables file.
 
 ## Rotation and failure
 
-Rotate a secret where the provider keeps it; the next read uses it, nothing
-caches it. A Vault token that is not renewed expires: `worker:vault-renewal`
+Rotate a secret where the provider keeps it; the next read uses it. Only the
+proxy client key is remembered, and only as a fallback; a rotated key reaches
+every node at its next launch, and `retire-key` should wait until every node
+has launched an agent since the rotation. A Vault token that is not renewed expires: `worker:vault-renewal`
 fails and the fix is one login of that machine. A GitHub token that stops
 working fails `auth:github` and `worker:git-push-auth`; store the new one in
 Vault and the worker restores itself within ten minutes. During a proxy
@@ -60,6 +65,7 @@ Every failure code is in [the runbooks](runbooks.md).
 | `notify-webhook-url`           | when its feature is declared | `DEVFLEET_SECRET_NOTIFY_WEBHOOK_URL`           | `~/.config/dev-fleet/secrets/notify-webhook-url`           |
 | `notify-ntfy-url`              | when its feature is declared | `DEVFLEET_SECRET_NOTIFY_NTFY_URL`              | `~/.config/dev-fleet/secrets/notify-ntfy-url`              |
 | `tailnet-auth-key`             | when its feature is declared | `DEVFLEET_SECRET_TAILNET_AUTH_KEY`             | `~/.config/dev-fleet/secrets/tailnet-auth-key`             |
+| `tailnet-admin`                | when its feature is declared | `DEVFLEET_SECRET_TAILNET_ADMIN`                | `~/.config/dev-fleet/secrets/tailnet-admin`                |
 
 <!-- end: secrets -->
 
@@ -72,8 +78,10 @@ Every failure code is in [the runbooks](runbooks.md).
 | `auth:github`                 | `src/github/checks.ts`                     |
 | `auth:vault`                  | `src/cli/fleet-sync-vault.ts`              |
 | `auth:vault-isolation`        | `src/cli/fleet-sync-vault.ts`              |
+| `worker:credential-renewal`   | `src/worker-host-checks.ts`                |
 | `worker:git-identity`         | `src/git-identity.ts`                      |
 | `worker:git-push-auth`        | `src/definition.ts`, `src/git-identity.ts` |
+| `worker:git-signing`          | `src/definition.ts`, `src/git-identity.ts` |
 | `worker:github-sync`          | `src/worker-access-checks.ts`              |
 | `worker:proxy-authentication` | `src/worker-access-checks.ts`              |
 | `worker:vault-renewal`        | `src/worker-access-checks.ts`              |

package/docs/consumer-setup.md

@@ -24,13 +24,13 @@ Nothing else is needed. You never clone dev-fleet, copy its files or write a
 wrapper script around a checkout of it: the npm package is the tool, and the
 version in `package.json` is the commit every machine runs.
 
-| Concept     | Meaning                                                                                                |
-| ----------- | ------------------------------------------------------------------------------------------------------ |
-| manager     | The one macOS machine you run `devfleet fleet-sync` on. Hosts the CLI proxy and the monitoring hub.    |
-| worker      | An Ubuntu 26.04 machine the manager converges over SSH. Stateless: rebuilt from Git and your secrets.  |
-| fleet state | One SQLite database (roster, approved revision, configuration last applied) replicated to your bucket. |
-| bucket      | Any S3-compatible bucket you own, or a directory. The source of truth for the fleet state.             |
-| revision    | The dev-fleet commit the installed package was published from (`npx devfleet --version`).              |
+| Concept     | Meaning                                                                                                        |
+| ----------- | -------------------------------------------------------------------------------------------------------------- |
+| manager     | The one macOS machine you run `devfleet fleet-sync` on. Hosts the CLI proxy and the monitoring hub.            |
+| worker      | An Ubuntu 26.04 or 24.04 machine the manager converges over SSH. Stateless: rebuilt from Git and your secrets. |
+| fleet state | One SQLite database (roster, approved revision, configuration last applied) replicated to your bucket.         |
+| bucket      | Any S3-compatible bucket you own, or a directory. The source of truth for the fleet state.                     |
+| revision    | The dev-fleet commit the installed package was published from (`npx devfleet --version`).                      |
 
 ## Rules for agents
 
@@ -75,25 +75,30 @@ On the manager (the Mac this project is checked out on):
 | Tailscale signed in                                      | `tailscale status --json` (look for `"BackendState":"Running"`) |
 | An SSH alias per worker that works without a prompt      | `ssh -o BatchMode=yes WORKER_ALIAS true` for each worker        |
 
-On every worker:
-
-| Requirement                          | Verify, from the manager                                           |
-| ------------------------------------ | ------------------------------------------------------------------ |
-| Ubuntu 26.04, amd64 or arm64         | `ssh WORKER_ALIAS '. /etc/os-release; echo $VERSION_ID; uname -m'` |
-| A non-root account that can use sudo | `ssh WORKER_ALIAS 'id -u; sudo -n true && echo sudo-ok'`           |
-| Joined to the same tailnet           | the worker appears in `tailscale status` on the manager            |
-
-A worker needs nothing else beforehand: no Git, curl, Python, Node, Bun or
-mise. `fleet-sync` installs them. If `sudo -n true` fails, that is fine for
-the first, interactive run, which asks for the password once and installs a
-sudo rule for the fleet; it is not fine for unattended runs.
-
-The `ssh` value of a worker in `fleet.json` is an alias from the manager's
-`~/.ssh/config`. dev-fleet always connects with `BatchMode=yes` and
-`StrictHostKeyChecking=yes`, so the host key must already be in
-`known_hosts`. **Ask** the human to verify each fingerprint out of band and
-connect once by hand; never add `StrictHostKeyChecking=no` or
-`accept-new`. A typical alias:
+On every worker, check each row of
+[what a machine must offer to be a worker](worker-requirements.md): the
+image, architecture and size, the access, the first-boot data and the
+network. Each row says how to verify it. If `sudo -n true` fails, that is fine
+for the first, interactive run, which asks for the password once and installs
+a sudo rule for the fleet; it is not fine for unattended runs.
+
+A worker is reached in one of two ways, always with `BatchMode=yes` and
+`StrictHostKeyChecking=yes`; never add `StrictHostKeyChecking=no` or
+`accept-new`.
+
+- **From declared data (no alias).** Leave `ssh` out: dev-fleet connects as
+  `network.user` on `network.sshPort` to `network.tailscaleName`, and checks
+  the host key against a known_hosts file of its own, rendered from the key
+  the fleet state pins. The human's `~/.ssh/config` and `~/.ssh/known_hosts`
+  are neither needed nor changed. The key is pinned without trusting a first
+  connection: `npx devfleet fleet-node-seed --id ID --output PATH --apply
+--yes` generates it for a new machine, `npx devfleet fleet-node-replace
+--id ID --host-key "TYPE BASE64" --apply --yes` pins one the human verified
+  out of band, and a worker record (below) may carry it as `hostKey`.
+- **Through the human's own alias.** `ssh` names a Host alias in the
+  manager's `~/.ssh/config`, and its host key must already be in their
+  `known_hosts`. **Ask** the human to verify each fingerprint out of band and
+  connect once by hand. A typical alias:
 
 ```
 Host worker-one
@@ -302,6 +307,14 @@ An agent cannot answer those prompts, and `--json`, `--non-interactive` or a
 npx devfleet fleet-sync
 ```
 
+A worker that needs none of those prompts is enrolled without the human: a
+run with `--json` or `--non-interactive` enrolls a declared worker when its
+pinned host key matched, `sudo -n` works on it, and exactly one tailnet device
+matches it (or none, when `fleet.json` declares `tailnet` and its secret
+`tailnet-admin`, so the run joins it with a single-use key). The report lists
+what it enrolled (`enrolled` in the JSON). A machine made with
+`devfleet fleet-node-seed` meets all of these.
+
 It converges the manager first, then the canary worker, then the rest. It
 prints one table and the path of the full JSON report
 (`~/.local/share/personal/fleet/manager/last-sync.json`, readable by the human
@@ -382,7 +395,9 @@ and codes; leave out everything private.
 - **Remove a worker.** Delete its entry, commit, run `npx devfleet fleet-sync`.
 - **Upgrade the fleet.** `npm install --save-dev --save-exact @crvouga/dev-fleet@latest`,
   commit, `npx devfleet fleet-sync`. The canary proves the new revision first.
-  Install an older version and sync to roll back.
+  `npx devfleet fleet-sync --rollback` goes back to the revision applied
+  before (the revision, tools and T3; system packages roll forward only, as
+  the [README](../README.md#the-version) says).
 - **Replace the manager.** Follow
   [replacing the manager](manager-recovery.md).
 

package/docs/convergence.md

@@ -0,0 +1,67 @@
+# What converges a machine, and what is not Ansible yet
+
+The goal is one convergence engine: Ansible, on every platform, with the CLI
+as an orchestrator that validates the fleet configuration, runs playbooks,
+reads their results and reports checks. Today three mechanisms change a
+machine: the playbooks under `ansible/`, the bootstrap scripts, and some
+TypeScript. This page lists every step that is not an Ansible task yet, each
+with the reason it is not, so the list can only shrink.
+`features/rules/convergence.feature` fails when a file that changes a machine
+is missing from it.
+
+## Ansible
+
+| Playbook                                  | Converges                                                                     |
+| ----------------------------------------- | ----------------------------------------------------------------------------- |
+| `ansible/ubuntu/baseline.yml`             | A worker's packages, Docker, services, power, user, SSH, disk                 |
+| `ansible/ubuntu/tailscale.yml`            | Tailscale on a worker                                                         |
+| `ansible/ubuntu/t3-desktop.yml`           | The T3 desktop app on a desktop worker                                        |
+| `ansible/ubuntu/t3-headless.yml`          | The T3 server as a lingering user service on a headless worker                |
+| `ansible/ubuntu/t3-policy.yml`            | T3's administrator policy                                                     |
+| `ansible/ubuntu/t3-route.yml`             | T3's private tailnet route                                                    |
+| `ansible/ubuntu/recovery.yml`             | Recovery of a stopped T3 desktop app or server                                |
+| `ansible/ubuntu/worker-account.yml`       | The declared account on an image that lets only root in, from the manager     |
+| `ansible/ubuntu/reboot.yml`               | One worker's first-converge or maintenance reboot, from the manager           |
+| `ansible/monitoring/agent.yml`            | The Beszel agent on a worker                                                  |
+| `ansible/monitoring/agent-darwin.yml`     | The Beszel agent on the manager                                               |
+| `ansible/monitoring/hub-darwin.yml`       | The Beszel hub and its tunnel on the manager                                  |
+| `ansible/monitoring/hub-route-darwin.yml` | The hub's private tailnet route                                               |
+| `ansible/macos/manager.yml`               | The manager's power settings, network time and fleet-watch job, unattended    |
+| `ansible/macos/t3-policy.yml`             | T3's administrator policy on the manager                                      |
+| `ansible/macos/manager-privilege.yml`     | The one administrator grant: sudoers, job log rotation and the T3 policy      |
+| `ansible/macos/worker-host-key.yml`       | A worker's pinned SSH host key in the manager's known_hosts                   |
+| `ansible/macos/vault-renew.yml`           | The launch agent that renews the manager's own Vault session                  |
+| `ansible/github/git.yml`                  | The Git identity and GitHub credentials of every node                         |
+| `ansible/agents/agents.yml`               | Each agent CLI's dedicated profile, rendered from `ansible/agents/templates/` |
+
+Every playbook passes `ansible-lint` in the production profile, every command
+in it says when it changes something and runs, or is skipped, in a dry run
+(`features/rules/playbooks.feature`). The Beszel agent playbook is proven on a
+systemd machine to report no change on a second run and in `--check`, and to
+report drift, and only that drift (`features/monitoring/agent-lifecycle.feature`).
+
+## Not Ansible yet
+
+| Step                                                                                    | Where                                                                                       | Why it is not a task yet                                                                                                    |
+| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
+| Install Git, curl, CA certificates, Python and Ansible                                  | `bootstrap/fleet-remote-bootstrap.sh`                                                       | It must run before Ansible exists on the machine; it is the minimal bootstrap                                               |
+| Create the worker account, its host key, the privilege rule and Tailscale on first boot | `bootstrap/worker-seed.yaml` (written by `fleet-node-seed`)                                 | cloud-init runs it before the manager can reach the machine; fleet-sync converges everything else with Ansible afterwards   |
+| Install mise and the pinned toolchain                                                   | `bootstrap/bootstrap.sh`, `bootstrap/setup.ts`                                              | mise must exist before any tool does, and `mise install --locked` already converges from a lock                             |
+| Install a worker's subset of tools                                                      | `bootstrap/worker-tools.ts`, `bootstrap/worker-features.ts`                                 | It chooses which locked tools `mise install` gets; moving it means a toolchain role                                         |
+| Publish the global mise configuration                                                   | `bootstrap/global-config.ts`                                                                | Derived from `mise.toml` and the agent worktree roots on every run; a `toolchain` role with a template would replace it     |
+| `PATH` lines in shell profiles                                                          | `bootstrap/shell-path.ts`                                                                   | Edits files a person owns, guarded by markers; a `blockinfile` task would replace it                                        |
+| The `devfleet` launcher link                                                            | `bootstrap/install-launcher.sh`                                                             | Points at the fleet-owned checkout of the revision, known only during the run                                               |
+| The T3 server archive                                                                   | `bootstrap/native-t3.ts`                                                                    | Installed with the toolchain, checked against the digest in `mise.toml`; a `t3` role would replace it                       |
+| The AWS CLI on Windows                                                                  | `bootstrap/native-aws.ts`                                                                   | Windows is not a fleet role                                                                                                 |
+| Docker Desktop on the manager                                                           | `bootstrap/docker-desktop.sh`                                                               | A person installs and approves it on macOS; dev-fleet only measures it                                                      |
+| A worker's service credentials from Vault                                               | `bootstrap/worker-github.ts`, `bootstrap/worker-auth.ts`                                    | A login belongs to a machine and a person; it is read from a session, never declared                                        |
+| The manager's own convergence: proxy, profiles, hub, agent                              | `src/cli/fleet-sync-manager.ts`                                                             | Orchestrates the manager playbooks and the proxy's own installer; the proxy is not a playbook yet (`docs/declared.md`)      |
+| Scheduled runs                                                                          | `src/cli/fleet-sync-schedule.ts`, `src/platform/launchd.ts`, `src/platform/systemd-user.ts` | Writes a launchd job or systemd timer from TypeScript; listed in `docs/declared.md` as a schedule that should be a playbook |
+
+## Still to do
+
+Roles with a common interface (`baseline`, `toolchain`, `agents`, `t3`,
+`network`, `monitoring`, `recovery`, `manager`), an inventory generated from
+the fleet configuration, and `site.yml`, `manager.yml` and `workers.yml` that
+bring a fresh manager and worker to ready with no other mutating step. Each
+row of the table above moves into one of those roles.

package/docs/declared.md

@@ -12,29 +12,33 @@ may only shrink.
 
 ## Declared
 
-| What                                          | Declared in                                                                                         | Pinned by                                                            | Applied by                                                                                    | Measured by                                  |
-| --------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | -------------------------------------------- |
-| The fleet: machines, services, accounts       | Your `fleet.json`                                                                                   | The dev-fleet version in your `package.json`, which names one commit | `fleet-sync`                                                                                  | `fleet-status`                               |
-| Tools: languages, CLIs, coding agents         | `mise.toml`                                                                                         | `mise.lock`: version and checksum for each platform                  | `mise install --locked`                                                                       | Tool and `agent:verification` checks         |
-| mise itself                                   | `min_version` and `mise_*_sha256` in `mise.toml`                                                    | The digest of its release archive, held here                         | `bootstrap/bootstrap.sh`                                                                      | Its version, at every toolchain setup        |
-| JavaScript dependencies                       | `package.json`                                                                                      | `bun.lock`                                                           | `bun install --frozen-lockfile`                                                               | The install fails on a difference            |
-| T3 Code, server and desktop app               | `t3_*` under `[vars]` in `mise.toml`                                                                | SHA-256 of each artifact                                             | `bootstrap/native-t3.ts`; `ansible/ubuntu/t3-desktop.yml`                                     | T3 checks; `t3:version` on every machine     |
-| T3 policy on the manager (updates off)        | `ansible/ubuntu/t3/policy.json`, shared with the workers                                            | The commit                                                           | `ansible/macos/t3-policy.yml`, once, as an administrator                                      | `t3:updates`                                 |
-| Worker work-in-progress guard                 | `ansible/ubuntu/wip-guard.py`, `ansible/ubuntu/tasks/recovery.yml`                                  | The commit                                                           | `fleet-sync`                                                                                  | `features/ide/work-in-progress.feature`      |
-| Ubuntu packages                               | `packages` and `snapshot` in `ansible/ubuntu/policy.json`                                           | Exact versions from a dated archive snapshot                         | `ansible/ubuntu/tasks/packages.yml`                                                           | `package:*`                                  |
-| Tailscale on workers                          | `tailscale` in `ansible/ubuntu/policy.json`                                                         | Exact version; digest of the signing key                             | `ansible/ubuntu/tailscale.yml`                                                                | `package:tailscale`                          |
-| Worker policy: power, SSH, sudo, memory, disk | `ansible/ubuntu/*.yml`, values shared with checks in `policy.json`                                  | The commit                                                           | `fleet-sync`                                                                                  | Host and resource checks                     |
-| Monitoring: hub, tunnel, agents               | `ansible/monitoring/*.yml`, `compose.yml`, `lock.json`                                              | Image digests; SHA-256 of each agent archive                         | `fleet-sync`                                                                                  | `beszel:*`                                   |
-| The manager's private route to the hub        | `ansible/monitoring/hub-route-darwin.yml`                                                           | The commit                                                           | `fleet-sync`                                                                                  | The route, read back                         |
-| The manager's power settings                  | `pmset` in `ansible/macos/policy.json`                                                              | The commit                                                           | `ansible/macos/manager.yml`, on every `fleet-sync`, after one grant (`manager-privilege.yml`) | `manager:power`                              |
-| Ansible on the manager                        | `ansible/pyproject.toml`                                                                            | `ansible/uv.lock`: one release, one Python, every dependency hashed  | `uv run --locked --isolated`                                                                  | A hash that differs stops the run            |
-| Ansible on a worker                           | `ansible-core` in `ansible/ubuntu/policy.json`                                                      | Exact version                                                        | `ansible/ubuntu/tasks/packages.yml`                                                           | `package:ansible-core`                       |
-| The CLI proxy                                 | `cliproxy/compose.yaml`, `image.lock.json`, `requirements.lock`, `launchd/`, `config.template.yaml` | Image digests; hashed Python dependencies                            | `devfleet cliproxy install`, and `fleet-sync` after it                                        | The proxy's audit                            |
-| Agent profiles and T3 providers               | `agents/registry.json`, `agents/t3.json`                                                            | The commit                                                           | `fleet-sync`                                                                                  | Profile receipts; the plan after every apply |
-| Worker time service                           | `ansible/ubuntu/tasks/services.yml`, offsets in `clock` in `ansible/ubuntu/policy.json`             | The commit                                                           | `fleet-sync`                                                                                  | `worker:clock`                               |
-| The manager's container watchdog              | `ansible/monitoring/container-watchdog.py`, `containerWatchdog` in `ansible/macos/policy.json`      | The commit                                                           | `fleet-sync` (`hub-darwin.yml`)                                                               | `manager:container-watchdog`                 |
-| Docker Desktop on the manager                 | `docker_desktop_*` under `[vars]` in `mise.toml`                                                    | Version and build                                                    | You                                                                                           | `manager:docker` (warns)                     |
-| CI                                            | `.github/workflows/`                                                                                | Each action at a commit; tools from `mise.lock`                      | GitHub                                                                                        | `features/rules/`                            |
+| What                                                           | Declared in                                                                                         | Pinned by                                                            | Applied by                                                                                    | Measured by                                                     |
+| -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
+| The fleet: machines, services, accounts                        | Your `fleet.json`                                                                                   | The dev-fleet version in your `package.json`, which names one commit | `fleet-sync`                                                                                  | `fleet-status`                                                  |
+| Tools: languages, CLIs, coding agents                          | `mise.toml`                                                                                         | `mise.lock`: version and checksum for each platform                  | `mise install --locked`                                                                       | Tool and `agent:verification` checks                            |
+| mise itself                                                    | `min_version` and `mise_*_sha256` in `mise.toml`                                                    | The digest of its release archive, held here                         | `bootstrap/bootstrap.sh`                                                                      | Its version, at every toolchain setup                           |
+| JavaScript dependencies                                        | `package.json`                                                                                      | `bun.lock`                                                           | `bun install --frozen-lockfile`                                                               | The install fails on a difference                               |
+| T3 Code, server and desktop app                                | `t3_*` under `[vars]` in `mise.toml`                                                                | SHA-256 of each artifact                                             | `bootstrap/native-t3.ts`; `ansible/ubuntu/t3-desktop.yml`                                     | T3 checks; `t3:version` on every machine                        |
+| T3 policy on the manager (updates off, linked environments on) | `ansible/ubuntu/t3/policy.json`, shared with the workers                                            | The commit                                                           | `ansible/macos/t3-policy.yml`, once, as an administrator (`manager-privilege.yml`)            | `t3:updates`                                                    |
+| Worker work-in-progress guard                                  | `ansible/ubuntu/wip-guard.py`, `ansible/ubuntu/tasks/recovery.yml`                                  | The commit                                                           | `fleet-sync`                                                                                  | `features/ide/work-in-progress.feature`                         |
+| Ubuntu packages                                                | `packages` and `snapshot` in `ansible/ubuntu/policy.json`                                           | Exact versions from a dated archive snapshot                         | `ansible/ubuntu/tasks/packages.yml`                                                           | `package:*`                                                     |
+| Tailscale on workers                                           | `tailscale` in `ansible/ubuntu/policy.json`                                                         | Exact version; digest of the signing key                             | `ansible/ubuntu/tailscale.yml`                                                                | `package:tailscale`                                             |
+| Worker policy: power, SSH, sudo, memory, disk                  | `ansible/ubuntu/*.yml`, values shared with checks in `policy.json`                                  | The commit                                                           | `fleet-sync`                                                                                  | Host and resource checks                                        |
+| Monitoring: hub, tunnel, agents                                | `ansible/monitoring/*.yml`, `compose.yml`, `lock.json`                                              | Image digests; SHA-256 of each agent archive                         | `fleet-sync`                                                                                  | `beszel:*`                                                      |
+| The manager's private route to the hub                         | `ansible/monitoring/hub-route-darwin.yml`                                                           | The commit                                                           | `fleet-sync`                                                                                  | The route, read back                                            |
+| The manager's power settings                                   | `pmset` in `ansible/macos/policy.json`                                                              | The commit                                                           | `ansible/macos/manager.yml`, on every `fleet-sync`, after one grant (`manager-privilege.yml`) | `manager:power`                                                 |
+| Ansible on the manager                                         | `ansible/pyproject.toml`                                                                            | `ansible/uv.lock`: one release, one Python, every dependency hashed  | `uv run --locked --isolated`                                                                  | A hash that differs stops the run                               |
+| Ansible on a worker                                            | `ansible-core` in `ansible/ubuntu/policy.json`                                                      | Exact version                                                        | `ansible/ubuntu/tasks/packages.yml`                                                           | `package:ansible-core`                                          |
+| The CLI proxy                                                  | `cliproxy/compose.yaml`, `image.lock.json`, `requirements.lock`, `launchd/`, `config.template.yaml` | Image digests; hashed Python dependencies                            | `devfleet cliproxy install`, and `fleet-sync` after it                                        | The proxy's audit                                               |
+| Agent profiles and T3 providers                                | `agents/registry.json`, `agents/t3.json`; each CLI's configuration from `ansible/agents/templates/` | The commit                                                           | `fleet-sync` (agent profiles through `ansible/agents/agents.yml`)                             | Profile receipts; the plan after every apply; `agent:*:session` |
+| Worker time service                                            | `ansible/ubuntu/tasks/services.yml`, offsets in `clock` in `ansible/ubuntu/policy.json`             | The commit                                                           | `fleet-sync`                                                                                  | `worker:clock`                                                  |
+| The manager's playbook and its receipt                         | `ansible/macos/manager.yml`                                                                         | The commit                                                           | `fleet-sync`, check mode first, then applied; re-applied at least daily                       | `manager:playbook`                                              |
+| The manager's network time                                     | `ansible/macos/manager.yml` (an assert)                                                             | The commit                                                           | macOS                                                                                         | `manager:clock`, `worker:clock-offset`                          |
+| fleet-watch on a schedule                                      | `fleetWatch` in `ansible/macos/policy.json`, `manager.schedule.watchMinutes` in `fleet.json`        | The commit                                                           | `fleet-sync` (`manager.yml`)                                                                  | `manager:watch`                                                 |
+| Rotation of the scheduled jobs' logs                           | `jobLogs` in `ansible/macos/policy.json`                                                            | The commit                                                           | `ansible/macos/manager-privilege.yml`, once, as an administrator                              | `manager:job-logs`                                              |
+| The manager's container watchdog                               | `ansible/monitoring/container-watchdog.py`, `containerWatchdog` in `ansible/macos/policy.json`      | The commit                                                           | `fleet-sync` (`hub-darwin.yml`)                                                               | `manager:container-watchdog`                                    |
+| Docker Desktop on the manager                                  | `docker_desktop_*` under `[vars]` in `mise.toml`                                                    | Version and build                                                    | You                                                                                           | `manager:docker` (warns)                                        |
+| CI                                                             | `.github/workflows/`                                                                                | Each action at a commit; tools from `mise.lock`                      | GitHub                                                                                        | `features/rules/`                                               |
 
 A value lives in a policy file when something besides the playbook reads it,
 so the check and the playbook cannot disagree. A value only a playbook uses
@@ -62,11 +66,11 @@ stays in the playbook: the playbook is its declaration.
   scripts from the declarative files above, not by a playbook.
 - **Files TypeScript and shell still write.** The installed `fleet.json`, the
   `current` and `devfleet` links, the `PATH` lines in shell profiles, the
-  global mise configuration, the T3 server archive, agent profiles and T3
-  settings. Each is derived
+  global mise configuration, the T3 server archive and T3 settings. Each is derived
   from a tracked file and rewritten on every run, but by code.
 - **Logins.** Tailscale, Vault, GitHub, each provider account and T3 pairing
-  belong to a machine and a person. They are checked, never copied.
+  (except a replaced worker's, which the manager pairs itself) belong to a
+  machine and a person. They are checked, never copied.
 - **Windows.** `bootstrap/bootstrap.ps1` exists; Windows is not a fleet role
   and nothing tests it.
 
@@ -80,13 +84,36 @@ moving to it entirely means NixOS workers and nix-darwin on the manager, which
 replaces every unit and path that exists today. The one thing above that Nix
 would settle and the current tools do not is the rest of a worker's packages.
 
+## Keeping pins current
+
+A pin that is only moved by hand goes stale and is then moved in a large
+batch, the riskiest way to change a fleet. Each pin is moved by a machine,
+one pull request at a time, which CI, the fresh workers and the VM test then
+judge like any change, unless it says below why a person moves it.
+
+| Pin                                      | Moved by                                                                                      |
+| ---------------------------------------- | --------------------------------------------------------------------------------------------- |
+| Tools in `mise.toml` and `mise.lock`     | Renovate (`mise`), one pull request per tool                                                  |
+| JavaScript and Python dependencies       | Renovate, with lock file maintenance                                                          |
+| Actions and CI service images            | Renovate, each action at a commit                                                             |
+| mise itself                              | The [Pins workflow](../.github/workflows/pins.yml): `bun bootstrap/pins.ts mise`, weekly      |
+| T3 Code (nightly)                        | The Pins workflow: `bun bootstrap/pins.ts t3`, weekly, every artifact hashed                  |
+| The tunnel image of the monitoring hub   | Renovate, by tag and digest (a custom manager in `renovate.json`)                             |
+| Beszel (hub image and agents)            | A person, because the agents' binary digests need the archives unpacked; `pins.ts` cannot yet |
+| Docker Desktop on the manager            | A person, because a person installs it and dev-fleet only measures the version                |
+| The Ubuntu snapshot and package versions | A person, because a new snapshot moves every package at once and each needs its exact version |
+
+The workflows never name the mise release: they install it through
+[`.github/actions/mise`](../.github/actions/mise/action.yml), which reads
+`min_version` from `mise.toml`.
+
 ## Changing a pin
 
-| To change               | Edit                                                                 | Then                                             |
-| ----------------------- | -------------------------------------------------------------------- | ------------------------------------------------ |
-| A tool                  | Its version in `mise.toml`                                           | Update `mise.lock` with `mise lock`              |
-| mise                    | `min_version` and the four `mise_*_sha256` in `mise.toml`            | Copy the digests from the release's `SHASUMS256` |
-| An Ubuntu package       | `packages` in `ansible/ubuntu/policy.json`, and `snapshot` if needed | The VM test in `ansible/ubuntu/tests`            |
-| Ansible on the manager  | `ansible/pyproject.toml`                                             | `uv lock` in `ansible/`                          |
-| Beszel                  | `ansible/monitoring/lock.json`, `compose.yml`, the agent unit        | `bun test features/rules`                        |
-| A manager power setting | `pmset` in `ansible/macos/policy.json`                               | The next `fleet-sync` applies it                 |
+| To change               | Edit                                                                 | Then                                  |
+| ----------------------- | -------------------------------------------------------------------- | ------------------------------------- |
+| A tool                  | Its version in `mise.toml`                                           | Update `mise.lock` with `mise lock`   |
+| mise                    | `bun bootstrap/pins.ts mise` moves `min_version` and the digests     | The fresh-box run                     |
+| An Ubuntu package       | `packages` in `ansible/ubuntu/policy.json`, and `snapshot` if needed | The VM test in `ansible/ubuntu/tests` |
+| Ansible on the manager  | `ansible/pyproject.toml`                                             | `uv lock` in `ansible/`               |
+| Beszel                  | `ansible/monitoring/lock.json`, `compose.yml`, the agent unit        | `bun test features/rules`             |
+| A manager power setting | `pmset` in `ansible/macos/policy.json`                               | The next `fleet-sync` applies it      |

package/docs/machine-setup.md

@@ -1,6 +1,7 @@
 > This document was written for the fleet dev-fleet was extracted from. Paths and
 > commands were updated; some statements still describe that fleet and are being
-> revised. The README is authoritative.
+> revised. The README is authoritative, and what a machine must offer to be a
+> worker is listed only in [worker requirements](worker-requirements.md).
 
 ## Machine setup: macOS, Windows and Linux
 
@@ -184,7 +185,13 @@ railway login
 Only sign into providers the task actually needs. On fleet machines the Git
 identity and the GitHub credential helper come from the `git` block of
 `fleet.json`: `fleet-sync` writes them in `~/.config/git/config` and checks
-them as `worker:git-identity` and `worker:git-push-auth`. Elsewhere set `git
+them as `worker:git-identity` and `worker:git-push-auth`. With
+`"signing": "ssh"` in that block every node also makes an SSH key of its own
+(`~/.ssh/dev-fleet-git-signing`, never copied off the node) and signs every
+commit and tag with it, checked as `worker:git-signing`; GitHub shows those
+commits as verified once you add each node's `.pub` to the account as a
+signing key (`gh ssh-key add ~/.ssh/dev-fleet-git-signing.pub --type signing`
+on the node). Elsewhere set `git
 config --global user.name` and `user.email` after confirming them; do not guess
 an email or overwrite an existing identity. Enable Git LFS hooks when needed with `git lfs install`.
 
@@ -357,9 +364,44 @@ headless service, so environment identity and pairing are kept. A conflicting
 launcher version requires reviewed maintenance; the recovery timer stops a
 still-running headless service. Routine fleet-sync never ends a running turn
 to load a new pin or policy: the app restarts into them at the first idle
-check (`t3:running-version`, `t3:policy-loaded`). The disposable Ubuntu VM test still exercises the former headless service
-and does not cover the desktop session. Private routing and paired execution
-require separate evidence.
+check (`t3:running-version`, `t3:policy-loaded`). Private routing and paired
+execution require separate evidence.
+
+#### Headless workers
+
+A worker whose session is `headless` (`features.session` in `fleet.json`, or
+the `editing` preset) has no display manager, no graphical session and no
+GPU. It runs the pinned T3 **server**, not the desktop app under a virtual
+display: the pinned release ships the server as its own archive
+(`t3-<version>-linux-<arch>.tar.gz`, the `t3_version` and
+`t3_linux_*_sha256` pins, from the release's `SHA256SUMS`), which the
+toolchain already installs on every worker for policy commands. Its
+`t3 serve` runs the server the desktop app embeds, with no browser; the
+subcommand and its flags were read from the pinned release itself
+(`t3 serve --help` of `0.1.8-nightly.20261106.4262`, whose `--mode web`
+server answered `/.well-known/t3/environment` with that `serverVersion`).
+A virtual display would install and pin a display server and a desktop
+toolkit for one process, which costs the memory a small worker needs.
+
+`ansible/ubuntu/t3-headless.yml` declares `t3code-headless.service`, a user
+unit started by `default.target`: the designated user lingers, so it starts at
+boot with nobody logged in. It listens on `127.0.0.1:3773` like the desktop
+app, with `T3CODE_HOME=%h/.t3`, so the private route, the policy, the provider
+instances, `personal-worker-recovery` (which reads the release from the
+unit's versioned server path) and the checks are the same for both session
+types. Switching a worker between types stops and removes the other type's
+unit and keeps `~/.t3`; GDM auto-login, if the machine had it, is left as it
+was. A headless worker installs none of the desktop-only packages
+(`features.desktop` in `ansible/ubuntu/policy.json`) and reports their checks
+as not applicable. The server prints a one-time pairing token to its own
+journal at start, as `t3 serve` does anywhere; dev-fleet never reads it.
+Pairing is done from the manager: by a person the first time, and by
+`fleet-node-replace` or the next `fleet-sync` for a machine that replaces a
+worker.
+
+The disposable Ubuntu VM test (`ansible/ubuntu/tests/vm.py`) converges each
+session type, reboots with nobody logged in and kills the T3 process to
+prove the recovery timer brings it back.
 
 Normal Ubuntu fleet setup is controller-led:
 `devfleet fleet-sync`. See the
@@ -396,7 +438,10 @@ Controller-driven remote bootstrap passes `--worker` to the shared setup owner.
 It installs the `vars.worker_tools` subset from `mise.toml`, with versions and
 artifacts taken from the same `mise.lock`; native T3 and Ubuntu Docker/service
 policy remain source-pinned. `bun bootstrap/doctor.ts --worker --docker --json`
-checks that profile. It does not require unrelated cloud CLIs on workers.
+checks that profile. It does not require unrelated cloud CLIs on workers. A
+worker whose `containers` feature is off (`features.containers: false`, or the
+`editing` preset) leaves `docker-cli` out of its base tools, and its doctor
+runs with `--no-containers` instead of `--docker`.
 
 Tool profiles add locked tool sets on top of it: `worker.toolProfiles` in
 `fleet.json` (fleet-wide) or `toolProfiles` on one worker selects `web` (pnpm,
@@ -404,7 +449,28 @@ Deno, Yarn and headless Chromium for browser tests), `go`, `rust` (with CMake
 and Ninja) or `jvm` (Temurin), each a `worker_profile_<name>` under `[vars]` in
 `mise.toml`. `worker.tools` adds single tools dev-fleet already pins. The setup
 installs them (`--node` reads the installed configuration) and the doctor
-checks each as `tool:<name>` (`--profiles`, `--tools`).
+checks each as `tool:<name>` (`--profiles`, `--tools`). The base tools include
+`tmux`, for sessions that outlive an SSH connection.
+
+A tool dev-fleet does not ship belongs to the project that needs it: pin it in
+the project's own `mise.toml` and lock it with `mise lock`. On every fleet
+machine dev-fleet's global configuration sets `locked = true` for the
+`project` scope, so mise installs a project's tools in a checkout only from
+that project's `mise.lock` and refuses a tool the lock does not name;
+`mise install --locked` in the checkout installs and verifies them.
+
+There is no `mobile` profile. No mobile toolchain can be pinned with
+checksums for both worker architectures through mise: Flutter publishes no
+Linux arm64 release, the Android SDK's platforms and build tools are fetched
+by `sdkmanager` at run time (outside any lock, behind a licence a person
+accepts), and iOS builds need macOS and Xcode, which workers are not. A
+project that builds for Android on x86_64 workers pins `flutter`, `java` and
+`android-sdk` in its own `mise.toml` as above.
+
+`worker.exposure` (fleet-wide) or `exposure` on one worker says what a worker
+opens beyond loopback and the tailnet; without it, nothing. See
+[Exposure](ubuntu-sync.md#exposure).
+
 The source-owned `worker-auth.ts` timer renews the worker's existing Vault token
 and can restore the declared GitHub account from its authorized Vault service
 credential, without copying another machine's login directory.

package/docs/manager-setup.md

@@ -22,6 +22,13 @@ step added here must be added there too, with the check that verifies it.
 
 <!-- /human-steps -->
 
-The grant lets the manager's user run exactly `pmset -c` without a password,
-so later syncs never prompt. Run the same playbook with `-e grant=absent` to
-remove it.
+The grant lets the manager's user run exactly `pmset -c` and
+`systemsetup -getusingnetworktime` without a password, so later syncs never
+prompt. The same playbook, run as an administrator, also installs what only
+root may write: the newsyslog drop-in that keeps the logs of the scheduled
+jobs under a fixed size (`manager:job-logs`) and T3's administrator policy
+(`t3:updates`). Every `fleet-sync` then applies `ansible/macos/manager.yml`
+unattended: in check mode first, then for real, with a receipt like the
+workers' and a full re-apply at least once a day (`manager:playbook`). Run the
+grant again when `manager:clock` or `manager:job-logs` asks for it after an
+upgrade, or with `-e grant=absent` to remove it.