@crvouga/dev-fleet 0.8.0 → 0.9.0

package/docs/ubuntu-sync.md

@@ -1,6 +1,7 @@
 > This document was written for the fleet dev-fleet was extracted from. Paths and
 > commands were updated; some statements still describe that fleet and are being
-> revised. The README is authoritative.
+> revised. The README is authoritative, and what a machine must offer to be a
+> worker is listed only in [worker requirements](worker-requirements.md).
 
 # Developer fleet: one manager (macOS) and stateless workers (Ubuntu)
 
@@ -10,7 +11,7 @@ What is proven, and where:
 
 | Evidence                                                                 | Where                                                                                                                                     |
 | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
-| A bare Ubuntu 26.04 machine bootstrapped over real SSH, then repeated    | `features/fresh-box/fresh-box.feature`, on every pull request (amd64 and arm64)                                                           |
+| A bare Ubuntu machine bootstrapped over real SSH, then repeated          | `features/fresh-box/fresh-box.feature`, on every pull request (each accepted release, amd64 and arm64)                                    |
 | One run over several machines, one versioned report, no partial receipt  | `features/fresh-box/fleet.feature`                                                                                                        |
 | Interrupted checkout, package installation and toolchain setup resumed   | `features/fresh-box/fleet.feature`                                                                                                        |
 | Overlapping runs, a dropped connection, a held APT lock, no DNS, no disk | `features/fresh-box/fleet.feature`                                                                                                        |
@@ -86,9 +87,11 @@ existing SSH aliases, tests strict SSH access, and offers one enrollment
 confirmation for the verified connections. The apply approval binds a fingerprint
 of the selected revision, policy and connection records; changed bindings fail
 with `FLEET_PLAN_STALE` before source preparation or remote mutation. It stores private overrides mode 0600. Unknown host keys are never accepted automatically. Compare fingerprints
-out of band before enrolling trust. Unreachable machines remain listed; first
-reachability requires an existing trusted LAN SSH route or physical setup of
-SSH/Tailscale. No software on the manager can bootstrap a disconnected box.
+out of band before enrolling trust. Unreachable machines remain listed; what a machine
+needs to be reachable the first time (SSH with a key, a known host key, the
+tailnet, or first-boot data that provides them) is listed in
+[worker requirements](worker-requirements.md). No software on the manager can
+bootstrap a disconnected box.
 
 A previously configured manager uses saved connection bindings and its approved
 revision. A changed Tailscale device ID is a replacement gate, even if the old
@@ -177,21 +180,21 @@ independently. Bespoke TypeScript remains only where a native CLI or API has no
 declarative owner: T3 service lifecycle, Beszel enrollment tokens, provider
 verification and the CLIProxy singleton.
 
-| Policy                      | Workers (Ubuntu 26.04, stateless)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Manager (macOS, singleton)                                                                                                                                                                                                                                                                                                                                                                                |
-| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Never sleep                 | systemd sleep targets masked; logind lid/idle policy; GNOME power plugin set to `nothing`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | never sleeps on AC power (`pmset -c sleep 0`, `macos/manager.yml`, verified as `manager:power`); battery sleep is left to the human                                                                                                                                                                                                                                                                       |
-| No automatic logout or lock | GNOME `lock-enabled false`, `idle-delay 0`; designated-user lingering                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | not managed (the human uses it)                                                                                                                                                                                                                                                                                                                                                                           |
-| Always-on services          | drop-ins `Restart=always`, `StartLimitIntervalSec=0` for ssh, tailscaled, docker, beszel-agent                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | singleton services under launchd and Docker Desktop (see the last row)                                                                                                                                                                                                                                                                                                                                    |
-| T3 health and connection    | `fleet-sync` and `fleet-status --workers` require `worker:t3-health` (local API answers), `t3:manager-endpoint` (manager HTTPS over Tailscale) and `t3:paired-access` (manager T3 link reachable)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | the human's T3 desktop holds the links to every worker (`t3:paired-access`)                                                                                                                                                                                                                                                                                                                               |
-| Memory safety               | `systemd-oomd` (swap 90%, pressure 60%); `user.slice` `MemoryMax=90%`; each `t3code-*.scope` agent `MemoryMax=30%`, `OOMPolicy=continue`; swap file when none                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
-| Disk safety                 | `personal-fleet-disk-guard` kills only the top user-cgroup writer below 3 GiB free, never deletes files; hourly `personal-fleet-disk-gc.timer` deletes git-ignored build output (`node_modules`, `.cache`, `dist`, ...) in checkouts idle 7d (12h under 15 GiB free) with no process inside, plus unused Docker images and build cache, never source, worktrees or volumes; journal 1G; `/tmp` 25% of RAM, aged 2d hourly, a measured write probe (`TMP_QUOTA_EXHAUSTED`), and T3 agents use disk-backed `/var/tmp` (`TMPDIR`, aged 14d); Docker log rotation                                                                                                                                                                       | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
-| Tailscale                   | pinned APT package, `tailscaled` enabled, native login required                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Tailscale app, human login                                                                                                                                                                                                                                                                                                                                                                                |
-| Dashboard agent             | pinned native binary, hardened `beszel-agent` systemd unit as user `beszel`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | not installed                                                                                                                                                                                                                                                                                                                                                                                             |
-| Credential renewal          | user timer `personal-worker-auth` renews the node's own Vault token every 10min; `personal-worker-github` signs it in to GitHub from the stored token ([GitHub](github.md))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | the human's own Vault and GitHub sessions                                                                                                                                                                                                                                                                                                                                                                 |
-| Noninteractive sudo         | validated `/etc/sudoers.d/personal-fleet-USER`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | only `pmset -c`, granted once by `manager-privilege.yml` (`-K`)                                                                                                                                                                                                                                                                                                                                           |
-| T3 worker app               | pinned desktop `.deb` is the only T3 process (UI plus embedded server); GDM auto-login, user unit `t3code-desktop.service` bound to the graphical session with `Restart=always`; `personal-worker-recovery` runs a copy of `t3-health.sh` installed in `/usr/local/lib/personal-fleet` every minute: it stops and disables a still-running headless `t3code.service`, reports `T3_NO_GRAPHICAL_SESSION` and restarts the display manager (three times at most) when there is no session, starts a stopped app after `reset-failed`, restarts one whose local API missed three checks, and restarts one on another release than the installed one or with a deferred policy restart once it runs no agent turn; administrator policy | not a worker                                                                                                                                                                                                                                                                                                                                                                                              |
-| T3 providers                | the same managed instances (`fleet/agents/t3.json`) through locked T3 administrator policy                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | reconciled into the human's T3 settings by `fleet-sync` (`t3:providers`) with the ownership-safe `t3-apply` engine, so every T3 offers the identical set; unrelated and unmanaged instances are preserved. `fleet-sync` also applies the same proxy agent profiles (`configuration:apply`) and verifies the shared Vault gateway credential end to end with `provider:*:inference`, exactly as on workers |
-| Singleton services          | never; clients of the manager's services only                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | every singleton runs here: CLI proxy (`cliproxy`), Beszel hub (`workstation/monitoring`)                                                                                                                                                                                                                                                                                                                  |
+| Policy                      | Workers (Ubuntu 26.04 or 24.04, stateless)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Manager (macOS, singleton)                                                                                                                                                                                                                                                                                                                                                                                |
+| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Never sleep                 | systemd sleep targets masked; logind lid/idle policy; GNOME power plugin set to `nothing`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | never sleeps on AC power (`pmset -c sleep 0`, `macos/manager.yml`, verified as `manager:power`); battery sleep is left to the human                                                                                                                                                                                                                                                                       |
+| No automatic logout or lock | GNOME `lock-enabled false`, `idle-delay 0`; designated-user lingering                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | not managed (the human uses it)                                                                                                                                                                                                                                                                                                                                                                           |
+| Always-on services          | drop-ins `Restart=always`, `StartLimitIntervalSec=0` for ssh, tailscaled, docker, beszel-agent                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | singleton services under launchd and Docker Desktop (see the last row)                                                                                                                                                                                                                                                                                                                                    |
+| T3 health and connection    | `fleet-sync` and `fleet-status --workers` require `worker:t3-health` (local API answers), `t3:manager-endpoint` (manager HTTPS over Tailscale) and `t3:paired-access` (manager T3 link reachable)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | the human's T3 desktop holds the links to every worker (`t3:paired-access`)                                                                                                                                                                                                                                                                                                                               |
+| Memory safety               | `systemd-oomd` (swap 90%, pressure 60%); `user.slice` `MemoryMax=90%`; each `t3code-*.scope` agent `MemoryMax=30%`, `OOMPolicy=continue`; swap file when none                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
+| Disk safety                 | `personal-fleet-disk-guard` kills only the top user-cgroup writer below 3 GiB free, never deletes files; hourly `personal-fleet-disk-gc.timer` deletes git-ignored build output (`node_modules`, `.cache`, `dist`, ...) in checkouts idle 7d (12h under 15 GiB free) with no process inside, plus unused Docker images and build cache, never source, worktrees or volumes; journal 1G; `/tmp` 25% of RAM, aged 2d hourly, a measured write probe (`TMP_QUOTA_EXHAUSTED`), and T3 agents use disk-backed `/var/tmp` (`TMPDIR`, aged 14d); Docker log rotation                                                                                                                                                                                                                                                                                                                                        | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
+| Tailscale                   | pinned APT package, `tailscaled` enabled, native login required                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | Tailscale app, human login                                                                                                                                                                                                                                                                                                                                                                                |
+| Dashboard agent             | pinned native binary, hardened `beszel-agent` systemd unit as user `beszel`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | not installed                                                                                                                                                                                                                                                                                                                                                                                             |
+| Credential renewal          | user timer `personal-worker-auth` renews the node's own Vault token every 10min; `personal-worker-github` signs it in to GitHub from the stored token ([GitHub](github.md))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | the human's own Vault and GitHub sessions                                                                                                                                                                                                                                                                                                                                                                 |
+| Noninteractive sudo         | validated `/etc/sudoers.d/personal-fleet-USER`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | only `pmset -c`, granted once by `manager-privilege.yml` (`-K`)                                                                                                                                                                                                                                                                                                                                           |
+| T3 worker app               | pinned desktop `.deb` is the only T3 process (UI plus embedded server) on a desktop worker, and the pinned server as the lingering user unit `t3code-headless.service` on a headless one (`features.session`; no session is waited for); GDM auto-login, user unit `t3code-desktop.service` bound to the graphical session with `Restart=always`; `personal-worker-recovery` runs a copy of `t3-health.sh` installed in `/usr/local/lib/personal-fleet` every minute: it stops and disables a still-running headless `t3code.service`, reports `T3_NO_GRAPHICAL_SESSION` and restarts the display manager (three times at most) when there is no session, starts a stopped app after `reset-failed`, restarts one whose local API missed three checks, and restarts one on another release than the installed one or with a deferred policy restart once it runs no agent turn; administrator policy | not a worker                                                                                                                                                                                                                                                                                                                                                                                              |
+| T3 providers                | the same managed instances (`fleet/agents/t3.json`) through locked T3 administrator policy                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | reconciled into the human's T3 settings by `fleet-sync` (`t3:providers`) with the ownership-safe `t3-apply` engine, so every T3 offers the identical set; unrelated and unmanaged instances are preserved. `fleet-sync` also applies the same proxy agent profiles (`configuration:apply`) and verifies the shared Vault gateway credential end to end with `provider:*:inference`, exactly as on workers |
+| Singleton services          | never; clients of the manager's services only                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | every singleton runs here: CLI proxy (`cliproxy`), Beszel hub (`workstation/monitoring`)                                                                                                                                                                                                                                                                                                                  |
 
 Credential sync is deliberately indirect: no secret is copied between machines.
 Each worker has a Vault token of its own, kept in a document of its own,
@@ -236,14 +239,58 @@ GNOME lock/idle changes apply to the designated user only. Effective running
 session/logind behavior still needs a maintenance/reboot acceptance check; writing
 a file does not prove logout/reboot survival. The sudoers fragment is validated
 with visudo before installation; this user and its agents can administer the
-machine. SSH authentication remains enabled.
+machine.
+
+SSH changes validate the complete configuration before reload and restore the
+previous fragments on validation failure. sshd takes keys only and refuses
+root (`10-personal-fleet-exposure.conf`, which sorts before an image's own
+fragments; `SSH_HARDENING_OVERRIDDEN` when `sshd -T` still shows otherwise).
+Existing listeners and socket activation are preserved; unmanaged fragments
+require review. The manager makes a **new** strict SSH connection after apply.
+
+### Exposure
+
+Every worker, with a public address or without one, admits inbound traffic
+only over loopback and the tailnet (`ansible/ubuntu/tasks/firewall.yml`, the
+`inet personal_fleet` nftables table, loaded at boot by
+`personal-fleet-firewall.service`). Open beyond it: tailscaled's UDP port,
+DHCP replies, ICMP, and what the exposure policy opens. Where Docker is
+installed, a published container port is dropped from outside the tailnet
+before Docker's own chains see it. Docker's and Tailscale's tables are left to
+their owners.
+
+The policy is `exposure` in `fleet.json`, fleet-wide under `worker` and per
+worker beside `network`; a field a worker names replaces the fleet-wide one:
+
+```json
+"exposure": {
+  "publicSsh": false,
+  "publicPorts": [{ "protocol": "udp", "port": 51820 }],
+  "otherListeners": "warn"
+}
+```
 
-SSH forwarding changes validate the complete configuration before reload and
-restore the previous fragment on validation failure. Existing listeners,
-authentication and firewall rules are preserved. Existing socket activation is preserved for forwarding-only changes; listener
-migrations and unmanaged fragments require review. The manager makes a **new**
-strict SSH connection after apply. Private-listener/firewall acceptance and
-failure-after-reload recovery still require disposable integration testing.
+`publicSsh` is break-glass access: SSH stays open beyond the tailnet.
+Without it the public SSH port closes only when the session running the sync
+arrived over the tailnet, after the tailnet path answers a new probe and sshd
+a new connection on the tailnet address; a failed verification restores the
+previous rules (`FIREWALL_TAILNET_UNVERIFIED`). A session from a public
+address, a tailnet that is not up, or a run without a session that finds the
+port open keeps it open, and `worker:exposure` reports
+`WORKER_PUBLIC_SSH_UNDECLARED` until the manager's SSH alias names the
+worker's tailnet name.
+
+`worker:exposure` (on the worker) lists listening sockets and fails on one
+bound beyond loopback and the tailnet outside the allow-list (sshd, tailscaled,
+the system's DHCP and name-resolution listeners, `publicPorts`), naming the
+port and its unit. A fleet service there always fails; anything else, such as
+an agent's development server, warns or fails by `otherListeners`.
+`worker:public-reachability` (from the manager, in `fleet-status --workers`)
+tries the SSH, T3 and monitoring ports on the public address Tailscale knows
+for the worker; it never blocks readiness, because the manager may share the
+worker's network. Both are not applicable without an address outside the
+tailnet. Closing the public path on a machine with two interfaces still needs
+the disposable VM or a real machine to prove.
 
 The manager is the only configuration coordinator. No per-node pull timer is
 installed. Manager schedule controls use the existing platform adapter:
@@ -327,7 +374,10 @@ A worker whose T3 has not loaded the staged provider policy, or runs an older
 release than the one installed, is restarted only while it runs no agent turn;
 otherwise sync reports the pending restart and the recovery timer applies it
 at the first idle check, failing the check after
-`t3RestartMaxDeferralHours` (`ansible/ubuntu/policy.json`). A pending proxy
+`t3RestartMaxDeferralHours` (`ansible/ubuntu/policy.json`).
+`fleet-sync --restart-busy` passes `t3_restart_busy` to the plays, which then
+restart a busy T3 at once and end its turns; the `--dry-run` plan says, per
+T3 restart, whether running turns end. A pending proxy
 update is installed even when it recreates the gateway. The manager finishes before the workers
 start, so they are verified against the updated proxy. `--dry-run` previews.
 

package/docs/worker-contract.md

@@ -1,6 +1,7 @@
 > This document was written for the fleet dev-fleet was extracted from. Paths and
 > commands were updated; some statements still describe that fleet and are being
-> revised. The README is authoritative.
+> revised. The README is authoritative, and what a machine must offer to be a
+> worker is listed only in [worker requirements](worker-requirements.md).
 
 # Declarative remote workers
 
@@ -12,11 +13,16 @@ devfleet fleet-status --workers
 ```
 
 The manager needs Node/npm, Git, SSH, Tailscale, and its own authorized T3 link.
-Initially each worker (an Ubuntu release the [support matrix](support-matrix.md)
-lists) needs a
-designated non-root account, trusted SSH access and administrator authorization. Standard Ubuntu cloud-image users
+What each worker must offer beforehand is listed once, in
+[worker requirements](worker-requirements.md). Standard Ubuntu cloud-image users
 already commonly have noninteractive sudo; otherwise establish the repository's
-`authorize-sudo.sh` policy once during administrator enrollment. Routine commands
+`authorize-sudo.sh` policy once during administrator enrollment. An image that
+lets only root in (many rented servers) needs no hands either: when the
+account `workers[].network.user` names cannot log in yet, or the SSH alias
+logs in as root, `fleet-sync` creates that account once over the root login
+(`ansible/ubuntu/worker-account.yml`: the key root was reached with and the
+fleet's sudo rule) and continues as it. The developer toolchain never runs as
+root, and the SSH policy turns root login off afterwards. Routine commands
 never ask for or store an SSH/sudo password. Unknown host keys, a different device
 identity, or missing privilege stop with a named action rather than bypassing trust.
 
@@ -27,18 +33,110 @@ Ansible policy. Developer checkouts and dirty work stay intact. A published CLI
 uses its source provenance; a source project action explicitly selects checkout
 HEAD. Private rosters contain connection paths and identities, not weaker policy.
 
-| Requirement                        | Declaration and enforcement                                                                                                                                                                                |
-| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| One consistent dependency set      | `mise.toml` worker profile, `mise.lock`, Ubuntu snapshot/package pins; exact versions and deployed input hashes checked live                                                                               |
-| Common service access              | Canonical required capabilities; declared GitHub account, worker's own renewable Vault session, authenticated singleton gateway catalog                                                                    |
-| One gateway host                   | The CLI proxy runs only on the manager; ownership/audit/recovery code rejects any install on a worker and missing singleton state                                                                          |
-| T3 connected and persistent        | Pinned desktop app as the only T3 process, auto-login session unit enabled/active with restart policy, bounded recovery timer, private endpoint, manager link matched to SSH-observed environment identity |
-| No idle sleep and session survival | systemd sleep masks, logind policy, designated-user lingering; running policy and boot timing checked, not inferred from files alone                                                                       |
-| Recovery after power loss          | Required boot services and recovery timers; recognized firmware restore-after-AC settings inspected when exposed; macOS `autorestart` where supported                                                      |
-| Always-on services                 | `Restart=always` drop-ins for ssh, tailscaled, docker and beszel-agent (launchd `KeepAlive` on macOS); health requires the running unit to report that policy                                              |
-| Central dashboard                  | Pinned Beszel hub on the manager, every worker enrolled with its own token; health verifies hub reachability, registration, connection and fresh samples                                                   |
-| No routine password prompts        | Verified BatchMode SSH, `sudo -n`, source-owned NOPASSWD policy; missing initial authorization is a failure                                                                                                |
-| Keep service authorization alive   | Native service credentials stay per-node; Vault self-renewal timer and GitHub recovery from authorized Vault credentials; expiring T3 links are reported before expiry                                     |
+| Requirement                        | Declaration and enforcement                                                                                                                                                                                                                                                   |
+| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| One consistent dependency set      | `mise.toml` worker profile, `mise.lock`, Ubuntu snapshot/package pins; exact versions and deployed input hashes checked live                                                                                                                                                  |
+| Common service access              | Canonical required capabilities; declared GitHub account, worker's own renewable Vault session, authenticated singleton gateway catalog                                                                                                                                       |
+| One gateway host                   | The CLI proxy runs only on the manager; ownership/audit/recovery code rejects any install on a worker and missing singleton state                                                                                                                                             |
+| T3 connected and persistent        | One pinned T3 process: the desktop app in the auto-login session, or on a headless worker the pinned server as a lingering user unit; enabled/active with restart policy, bounded recovery timer, private endpoint, manager link matched to SSH-observed environment identity |
+| No idle sleep and session survival | systemd sleep masks, logind policy, designated-user lingering; running policy and boot timing checked, not inferred from files alone                                                                                                                                          |
+| Recovery after power loss          | Required boot services and recovery timers; recognized firmware restore-after-AC settings inspected when exposed (not applicable on a virtual machine); macOS `autorestart` where supported                                                                                   |
+| Always-on services                 | `Restart=always` drop-ins for ssh, tailscaled, docker (with `containers`) and beszel-agent (launchd `KeepAlive` on macOS); health requires the running unit to report that policy                                                                                             |
+| Central dashboard                  | Pinned Beszel hub on the manager, every worker enrolled with its own token; health verifies hub reachability, registration, connection and fresh samples                                                                                                                      |
+| No routine password prompts        | Verified BatchMode SSH, `sudo -n`, source-owned NOPASSWD policy; missing initial authorization is a failure                                                                                                                                                                   |
+| Keep service authorization alive   | Native service credentials stay per-node; Vault self-renewal timer and GitHub recovery from authorized Vault credentials; expiring T3 links are reported before expiry                                                                                                        |
+
+## Worker features
+
+What a worker is built with beyond the base is declared in `fleet.json`:
+fleet-wide under `worker.features`, or for one worker under its own
+`features`. Each block may name a `preset` first; its own fields override
+it. A fleet that names no feature gets what every worker had before features
+existed.
+
+| Feature      | Values                | Default   | What it decides                                                                                                                                    |
+| ------------ | --------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `containers` | `true`, `false`       | `true`    | Docker: the Ubuntu packages, `docker.service`, the developer's `docker` group membership, the `docker-cli` tool and the Docker step of the disk GC |
+| `session`    | `desktop`, `headless` | `desktop` | How T3 runs: the desktop app in an auto-login graphical session, or the pinned server as a lingering user service with no display manager          |
+| `preset`     | `editing`             | none      | `editing`: no containers, a headless session and only the base tools (it adds no tool profile)                                                     |
+
+What each feature installs is listed once, under `features` in
+`ansible/ubuntu/policy.json`. A worker without it installs none of it, and its
+checks (`worker:package:*`, `worker:service:docker`, `worker:docker-run`,
+`tool:docker-cli`) report **not applicable** with the reason. Turning
+`containers` off on an existing worker removes what dev-fleet installed for it:
+the packages (not purged: images and volumes under `/var/lib/docker` stay
+until a person deletes them), the restart drop-in and the developer's group
+membership. `fleet-sync --dry-run` lists what each worker will lose (`plan`,
+under `removals`), and `worker:features` reports what the run removed and
+fails with `WORKER_FEATURE_REMOVAL_INCOMPLETE` while any of it remains. A
+headless worker installs no desktop-only package but removes none: a desktop
+the machine came with may depend on them.
+
+## The class of a machine
+
+The probe records each worker's class before anything is installed
+(`platform:machine-class`): **physical**, **virtual** (systemd-detect-virt
+names a hypervisor) or unsupported (a container or WSL2, refused by
+`platform:environment`). A virtual machine has no lid, no battery and no
+firmware its guest can read, and its provider restarts it after a power loss,
+so `worker:power-loss-recovery` and `ubuntu:lid-policy` report **not
+applicable**, with the reason and the worker's optional `powerRecoveryNote`
+(for example "provider restarts instances"). The sleep masks are still
+checked: a guest can be suspended by its own idle policy. On a physical
+machine nothing changes. A not-applicable check is counted apart, shown with
+its reason in text and in `--json`, and never fails a run, not even with
+`--strict`.
+
+## Size
+
+The minimum size of a worker for the editing workload is stated in
+[worker requirements](worker-requirements.md), from `sizing.minimumWorker` in
+`ansible/ubuntu/policy.json`. The probe measures it before anything is
+installed and refuses a smaller machine with `WORKER_SIZE_BELOW_MINIMUM`
+(`platform:worker-size`); the baseline asserts the same before it installs a
+package.
+
+The disk dev-fleet itself installs is measured on every pull request: the
+fresh-box CI job converges a stock Ubuntu image of each accepted release on
+amd64 and arm64, measures what each part takes, and fails when a part
+outgrows its budget in `sizing.footprintMiB`
+(`features/fresh-box/fresh-box.feature`). Measured on Ubuntu 26.04:
+
+| Part                                         | amd64        | arm64        | Budget   |
+| -------------------------------------------- | ------------ | ------------ | -------- |
+| Base tools with the agent CLIs, and `go`     | 5184 MiB     | 4830 MiB     | 6144 MiB |
+| The fleet's checkouts (up to four revisions) | 189 MiB each | 189 MiB each | 1024 MiB |
+| Tool profile `web` (its tools)               |              | 135 MiB      | 256 MiB  |
+| Tool profile `rust`                          |              | 1562 MiB     | 2048 MiB |
+| Tool profile `jvm`                           | 437 MiB      | 303 MiB      | 640 MiB  |
+
+What is not measured there, and stays an estimate: Ubuntu Server itself
+(about 3 to 5 GiB), the system packages of the baseline, the headless
+Chromium the `web` profile installs as packages, the swap file (half of
+memory), and the engine and images of `containers`. The minimum adds those
+to the measured parts with room for the disk floors below.
+
+The disk and memory lines of the worker policy scale with the machine: each
+is a share of the root filesystem or of memory between a lower and an upper
+bound (`sizing.thresholds`), and the upper bounds are the fixed numbers every
+worker had before.
+
+| Line                               | Share         | Lower    | Upper     |
+| ---------------------------------- | ------------- | -------- | --------- |
+| Swap file (created only when none) | 50% of memory | 1024 MiB | 8192 MiB  |
+| Free disk the baseline keeps       | 5% of disk    | 2048 MiB | 4096 MiB  |
+| Disk guard floor                   | 3% of disk    | 1024 MiB | 3072 MiB  |
+| Disk GC pressure line              | 15% of disk   | 4096 MiB | 15360 MiB |
+| `worker:capacity:disk` warns under | 10% of disk   | 3072 MiB | 10240 MiB |
+| `worker:capacity:disk` fails under | 2% of disk    | 1024 MiB | 2048 MiB  |
+
+`worker:agent-sessions` reports how many agent sessions run and how many the
+memory policy supports at once: the memory user workloads may use (90%), less
+what T3 keeps for itself (1536 MiB), in sessions of an estimated 1024 MiB each
+(`sizing.agentSessionMiB`, also an estimate). More sessions than that warn
+with `AGENT_SESSION_CAPACITY_EXHAUSTED`; each agent is still capped at 30% of
+memory, so the largest is stopped under pressure rather than the machine.
 
 ## Package pins and security updates
 
@@ -64,12 +162,17 @@ one:
 A package run cut off by a power loss is finished by the next sync: the
 baseline waits for the APT lock, runs `dpkg --configure -a` once (bounded to
 ten minutes) and audits again; only a database still inconsistent stops with
-`APT_STATE_BROKEN`.
+`APT_STATE_BROKEN`. When the cut-off run left one of the bootstrap's own
+packages unconfigured (an upgrade of `python3` or `git`, say), the bootstrap
+makes the same repair before it installs what it lacks, whoever started the
+run, waiting up to two minutes for a package process that holds the lock;
+only a database still inconsistent stops it with `DPKG_INTERRUPTED`.
 
 Health does not mutate configuration, renew tokens, log in, install software, or
 make billable inference requests. It audits the immutable desired worker deployment,
 not an unrelated developer branch. Every declared worker must be enrolled and
-reachable. Extra service hosts are listed as outside scope. Required failures,
+reachable, unless it is parked (`devfleet fleet-node-park`): a parked worker
+is listed as parked and never contacted. Extra service hosts are listed as outside scope. Required failures,
 warnings and unverified checks prevent a ready result; explicitly non-required
 checks remain visible without inventing a pass.
 

package/docs/worker-requirements.md

@@ -0,0 +1,107 @@
+# What a machine must offer to be a worker
+
+The one list of what a machine needs before it can be a worker, wherever it
+runs: a computer at home, a virtual machine on your own host or an instance
+rented from any provider. Everything else, `fleet-sync` installs. No vendor is
+required; any provider that boots a stock Ubuntu image with SSH and cloud-init
+user data works the same way.
+
+Each requirement that a check enforces names the check (from
+[schema/check-ids.json](../schema/check-ids.json)) and the failure code it
+reports (each has an entry in the [runbooks](runbooks.md)). A dash means the
+check reports no code of its own; its summary and remedy say what failed.
+`features/rules/worker-requirements.feature` fails when a check or code here
+does not exist, when a check marked as a worker requirement in the registry
+has no row here, or when the image, architecture and size rows disagree with
+`ansible/ubuntu/policy.json`.
+
+## The machine
+
+| Requirement                                                                                                                                                                                                      | How to check it                                       | Check                  | Failure code                     |
+| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ---------------------- | -------------------------------- |
+| A stock Ubuntu image of an accepted release: **26.04** (resolute) or **24.04** (noble). No distribution upgrade is ever made.                                                                                    | `. /etc/os-release; echo $VERSION_ID`                 | `ubuntu:release`       | `UBUNTU_BASELINE_MISMATCH`       |
+| An accepted architecture: **amd64** (x86_64) or **arm64** (aarch64).                                                                                                                                             | `uname -m`                                            | `ubuntu:release`       | `UBUNTU_BASELINE_MISMATCH`       |
+| A machine of its own: physical, or a full virtual machine (KVM, Xen, VMware, Hyper-V, or any hypervisor `systemd-detect-virt --vm` names). A container (Docker, LXC, systemd-nspawn, OpenVZ) or WSL2 is refused. | `systemd-detect-virt`                                 | `platform:environment` | `WORKER_ENVIRONMENT_UNSUPPORTED` |
+| At least **2 CPUs**, **3584 MiB** of memory (a 4 GiB instance) and a **20480 MiB** root filesystem. What dev-fleet installs is measured in CI; the operating system is estimated.                                | `nproc; free -m; df -m /`                             | `platform:worker-size` | `WORKER_SIZE_BELOW_MINIMUM`      |
+| Survives one reboot: the first converge ends with a reboot while no agent runs, and the worker must answer SSH again within 15 minutes.                                                                          | `devfleet fleet-status --workers` after the first run | `worker:reboot`        | `WORKER_REBOOT_FAILED`           |
+
+The minimum size is `sizing.minimumWorker` in `ansible/ubuntu/policy.json`;
+[the worker contract](worker-contract.md#size) explains the estimate and what
+grows with the machine. The accepted releases and architectures are `releases`
+in the same file; [the support matrix](support-matrix.md) lists every
+platform and why the others are refused.
+
+## Access
+
+| Requirement                                                                                                                                                                                          | How to check it                          | Check                     | Failure code                        |
+| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | ------------------------- | ----------------------------------- |
+| SSH with a key, from the manager, without a password.                                                                                                                                                | `ssh -o BatchMode=yes WORKER true`       | `ssh:auth`                | -                                   |
+| A host key the manager knows before the first connection: generated by `devfleet fleet-node-seed`, given as `hostKey` in a worker record, or pinned with `devfleet fleet-node-replace --host-key`.   | `devfleet fleet-status --workers`        | `ssh:host-key`            | -                                   |
+| A non-root account that may use `sudo` without a password, as `workers[].network.user` names it. An image that lets only root in is fine: `fleet-sync` creates the account over the root login once. | `ssh WORKER 'id -u; sudo -n true'`       | `bootstrap:privilege`     | `SUDO_AUTHORIZATION_REQUIRED`       |
+| With only a root login, `workers[].network.user` names the account to create.                                                                                                                        | `fleet.json`                             | `worker:account`          | `WORKER_ACCOUNT_UNDECLARED`         |
+| Joined to the fleet's tailnet: by the seed at first boot, or by `fleet-sync` with a single-use key the control plane `fleet.json` declares (`tailnet`) mints for it.                                 | `tailscale status --json` on the manager | `tailscale:configuration` | `TAILSCALE_SETUP_OR_LOGIN_REQUIRED` |
+
+## First-boot data
+
+A rented machine is set up from cloud-init user data and nothing else.
+`devfleet fleet-node-seed --id ID --output PATH` writes it: the account with
+the manager's key, the sudo rule, a host key generated for the machine, the
+pinned Tailscale and a short-lived single-use key. Any image that runs
+cloud-init accepts it (`--autoinstall` writes an Ubuntu autoinstall file for a
+machine installed from the ISO). The seed holds a secret: pass it to the
+provider as user data and delete it afterwards.
+
+| Requirement                                                                                                     | How to check it            | Check                  | Failure code                              |
+| --------------------------------------------------------------------------------------------------------------- | -------------------------- | ---------------------- | ----------------------------------------- |
+| The image runs cloud-init, or nothing at first boot. `fleet-sync` waits for a first boot that is still running. | `cloud-init status --wait` | `bootstrap:first-boot` | `FIRST_BOOT_FAILED`, `FIRST_BOOT_PENDING` |
+
+## Network
+
+No inbound connection from the internet is needed: the manager reaches a
+worker over the tailnet. A fixed or public address is not needed either.
+Outbound, the worker opens HTTPS (TCP 443) to:
+
+- the Ubuntu archive its image is configured with, and `snapshot.ubuntu.com`
+  (every pinned package);
+- `pkgs.tailscale.com` (Tailscale) and the fleet's tailnet control plane:
+  `tailnet.url` in `fleet.json`, or the hosted control plane when it names
+  none; Tailscale also uses UDP 41641 and its relays (UDP 3478, TCP 443);
+- `github.com`, `api.github.com` and GitHub's release download hosts, and the
+  other download hosts `mise.lock` names
+  (`grep -o 'https://[^/"]*' mise.lock | sort -u` lists them);
+- the CLI proxy at `services.proxy.endpoint`, and the secret store
+  `fleet.json` names (Vault's address with `vault`).
+
+| Requirement                                                                                                                                                | How to check it                                | Check                        | Failure code                                               |
+| ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | ---------------------------- | ---------------------------------------------------------- |
+| Reaches the Ubuntu archive and the snapshot.                                                                                                               | `ssh WORKER 'sudo apt-get update'`             | `bootstrap:prerequisites`    | `APT_REPOSITORY_UNAVAILABLE`, `APT_DNS_UNAVAILABLE`        |
+| Reaches GitHub.                                                                                                                                            | `ssh WORKER 'curl -sI https://api.github.com'` | `worker:github-sync`         | `GITHUB_UNREACHABLE`                                       |
+| Reaches the CLI proxy.                                                                                                                                     | `devfleet fleet-status --workers`              | `proxy:reachable-from`       | -                                                          |
+| Accepts no inbound connection beyond loopback and the tailnet, but what `exposure` in `fleet.json` declares. The fleet firewall closes the rest by itself. | `sudo nft list table inet personal_fleet`      | `worker:exposure`            | `WORKER_LISTENER_UNRESTRICTED`, `WORKER_FIREWALL_INACTIVE` |
+| Its public address, when it has one, answers on no fleet port.                                                                                             | `devfleet fleet-status --workers`              | `worker:public-reachability` | `WORKER_PUBLIC_PORT_UNRESTRICTED`                          |
+
+A provider's own firewall (a security group) may stay closed to all inbound
+traffic: Tailscale needs none, and SSH from the internet is never needed once
+the seed has joined the machine to the tailnet.
+
+## What is not needed
+
+- A display, a desktop or a GPU: `"session": "headless"` (the `editing`
+  preset) runs T3 as a service with no graphical session.
+- A public inbound port, a fixed address or a domain name.
+- Git, curl, Python, Node, Bun or mise beforehand: `fleet-sync` installs them.
+- Docker beforehand: the `containers` feature (on unless a worker turns it
+  off) installs it.
+- A password for SSH or `sudo`, or anyone at the console.
+- An account with any particular provider.
+
+## From infrastructure as code
+
+[examples/workers-from-infrastructure](../examples/workers-from-infrastructure/main.tf)
+shows the handoff with a local virtual machine: the seed from
+`devfleet fleet-node-seed` goes in as user data, and a worker record
+(`schema/worker-record.schema.json`) comes out for `fleet.json` to read. For a
+hosted provider only these change: the provider block and its credentials, the
+image (the provider's stock Ubuntu image of an accepted release), the instance
+type (at least the size above) and the network (a private network, or an
+inbound rule set with nothing open). The seed and the record stay the same.

package/examples/fleet.json

@@ -110,6 +110,7 @@
     "vaultUser": "example-user",
     "toolchain": "mise.lock",
     "toolProfiles": ["go"],
+    "features": { "containers": true, "session": "desktop" },
     "agentProfile": "proxy",
     "unattendedSudo": true,
     "persistentT3": true,