@crvouga/dev-fleet 0.7.0 → 0.9.0

package/docs/support-matrix.md

@@ -13,11 +13,12 @@ rest. The default is one macOS manager and Ubuntu workers treated as cattle.
 | -------------- | --------- | ------------------- | --------------- | ----------- | ----------------------------------------------------------------------------------------- |
 | manager        | macOS     | 14 or newer         | arm64, x86_64   | supported   | ansible/macos/policy.json                                                                 |
 | worker         | Ubuntu    | 26.04 (resolute)    | amd64, arm64    | supported   | ansible/ubuntu/policy.json; a machine of its own, physical or virtual                     |
+| worker         | Ubuntu    | 24.04 (noble)       | amd64, arm64    | supported   | ansible/ubuntu/policy.json; a machine of its own, physical or virtual                     |
 | manager        | any other | any                 | any             | refused     | The fleet configuration accepts only a macOS manager.                                     |
 | worker         | Ubuntu    | any other release   | any             | refused     | UBUNTU_BASELINE_MISMATCH                                                                  |
 | worker         | Ubuntu    | an accepted release | any other       | refused     | UBUNTU_BASELINE_MISMATCH                                                                  |
-| worker         | Debian    | any                 | any             | refused     | UBUNTU_BASELINE_MISMATCH                                                                  |
-| worker         | macOS     | any                 | any             | refused     | The fleet configuration accepts only Linux workers.                                       |
+| worker         | Debian    | any                 | any             | refused     | UBUNTU_BASELINE_MISMATCH; not yet supported (see below).                                  |
+| worker         | macOS     | any                 | any             | refused     | The fleet configuration accepts only Linux workers; not yet supported (see below).        |
 | worker         | Windows   | any                 | any             | refused     | The fleet configuration accepts only Linux workers.                                       |
 | worker         | WSL2      | any                 | any             | refused     | WORKER_ENVIRONMENT_UNSUPPORTED                                                            |
 | worker         | container | any                 | any             | refused     | WORKER_ENVIRONMENT_UNSUPPORTED                                                            |
@@ -32,5 +33,40 @@ refuses a worker that is not Linux, the platform probe refuses another
 release, CPU or environment (`ubuntu:release`, `platform:environment`), and
 the bootstrap and the Ubuntu baseline refuse them again on the machine.
 
-Supporting another platform means adding it to the policy that names it, with
-its package pins, a fresh-box job and its row here.
+## Adding a platform or a release
+
+What differs from one worker platform to another is named by the
+`WorkerPlatform` interface (`src/platform/worker-platform.ts`): its policy and
+playbooks, its package manager and pins, its Tailscale key, its service
+manager, its session model and its host checks. Ubuntu is the one
+implementation (`src/platform/ubuntu-worker.ts`).
+
+Another Ubuntu release is one entry in `releases` of
+`ansible/ubuntu/policy.json`: its version, codename and architectures, its
+package and browser library pins read from the snapshot's Packages indexes
+for that release (`packages`, `browserPackages`), and `tailscaleKeySha256`
+only when Tailscale signs its suite with another key than the common one.
+`ansible/ubuntu/tasks/release.yml` selects the entry from the facts, so the
+playbooks need no change of their own. The release is supported once the
+fresh-box features pass on it: CI runs them for every accepted release
+(`DEVFLEET_TEST_FRESH_BOX_RELEASE`), with its image pinned in
+`features/fresh-box/Dockerfile`.
+
+## Not yet supported
+
+These are refused today. What each would need, beyond a row here:
+
+- **Debian workers.** A `WorkerPlatform` of its own with a Debian policy:
+  pins read from snapshot.debian.org for each release, Tailscale's key digest
+  for the Debian suite, the platform probe accepting `ID=debian`, and the
+  bootstrap and `tasks/release.yml` accepting it in place of
+  `UBUNTU_BASELINE_MISMATCH`. The playbooks assume Ubuntu's archive keyring,
+  snapshot service and unattended-upgrades origins, and the GNOME desktop
+  T3 runs in; each needs a Debian equivalent. Then a fresh-box image and job.
+- **macOS workers.** Everything a worker platform names differs: launchd in
+  place of systemd as the service manager, a logged-in user session in place
+  of a lingering systemd user manager, Homebrew or pinned downloads in place
+  of APT, power and sleep policy through `pmset`, and host checks of their
+  own. The fleet configuration would accept `macOS` for a worker, and the
+  convergence would run playbooks under `ansible/macos` for it. No container
+  can stand in for a fresh macOS machine, so the test would be a macOS runner.

package/docs/topology.md

@@ -1,6 +1,7 @@
 > This document was written for the fleet dev-fleet was extracted from. Paths and
 > commands were updated; some statements still describe that fleet and are being
-> revised. The README is authoritative.
+> revised. The README is authoritative, and what a machine must offer to be a
+> worker is listed only in [worker requirements](worker-requirements.md).
 
 # Fleet topology: one manager, many workers
 
@@ -8,10 +9,10 @@ The fleet has exactly two roles, declared in
 your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.schema.json)) and validated by
 [`src/definition.ts`](../src/definition.ts):
 
-| Role        | How many    | Machine                               | Holds state | Runs                                                                                     |
-| ----------- | ----------- | ------------------------------------- | ----------- | ---------------------------------------------------------------------------------------- |
-| **manager** | exactly one | the human's macOS machine (`manager`) | yes         | `fleet-sync`, the T3 desktop that links to every worker, and **every singleton service** |
-| **workers** | any         | Linux boxes (`workers`, Ubuntu 26.04) | no          | the T3 desktop app (UI plus server), agent tools, the Beszel agent; clients only         |
+| Role        | How many    | Machine                                        | Holds state | Runs                                                                                     |
+| ----------- | ----------- | ---------------------------------------------- | ----------- | ---------------------------------------------------------------------------------------- |
+| **manager** | exactly one | the human's macOS machine (`manager`)          | yes         | `fleet-sync`, the T3 desktop that links to every worker, and **every singleton service** |
+| **workers** | any         | Linux boxes (`workers`, Ubuntu 26.04 or 24.04) | no          | the T3 desktop app (UI plus server), agent tools, the Beszel agent; clients only         |
 
 ## Manager
 
@@ -35,9 +36,12 @@ your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.
   A new singleton service is declared under `services` and placed on the
   manager; it never names a host.
 - Workers depend on it, so it never sleeps on AC power
-  (`pmset` in `ansible/macos/policy.json`, checked as `manager:power`, applied
-  once by an administrator with `ansible/macos/manager.yml`; the failed check
-  prints the command). While the
+  (`pmset` in `ansible/macos/policy.json`, checked as `manager:power` and
+  applied by `ansible/macos/manager.yml` on every `fleet-sync` once an
+  administrator has given the one grant `ansible/macos/manager-privilege.yml`;
+  the failed check prints the command, and
+  [docs/manager-setup.md](manager-setup.md) lists every step only a person
+  takes). While the
   manager is asleep, closed or offline, workers have no CLI proxy and no
   dashboard; nothing fails over, because a singleton never runs twice.
 - Its state (proxy OAuth/runtime state, the hub database) is backed up off the
@@ -48,7 +52,9 @@ your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.
 
 - Stateless cattle: turn one off, re-image it or add another at will. A worker
   is declared in `workers`, enrolled once (SSH trust, sudo, its own Vault
-  token), and then fully rebuilt by `fleet-sync` from Git and Vault.
+  token), and then fully rebuilt by `fleet-sync` from Git and Vault. What a
+  machine must offer before that is listed in
+  [worker requirements](worker-requirements.md).
 - Workers never host a singleton service. The proxy reports any local install
   on a worker as a duplicate owner, and the hub refuses to install anywhere but
   the manager.
@@ -56,13 +62,77 @@ your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.
   checkouts. Durable work lives in Git remotes; credentials come from Vault.
   Work in progress in agent worktrees is pushed to a recovery ref every 15
   minutes; [durability](durability.md) says what a lost machine takes with it.
-- The first declared worker is the rollout canary for a revision new to the
-  fleet: a failed canary stops promotion to the other workers. Repeating a
-  revision that already converged everywhere runs all workers at once.
+- One worker is the rollout canary for a revision new to the fleet: the one
+  `fleet-sync --canary` or `canary` in `fleet.json` names, else the first
+  declared worker. When it cannot be reached (powered off, being rebuilt),
+  the next reachable worker in declared order proves the revision instead and
+  the unreachable one is reported as offline, never as a blocked rollout. A
+  canary that was deployed and failed stops promotion to the other workers.
+  Each worker's applied revision is kept in the fleet state, so one that
+  converged keeps its progress when another did not. Repeating a revision
+  that already converged everywhere runs all workers at once.
 
 ## Changing the topology
 
 - **Add or replace a worker:** add it to `workers`, enroll it with
   `fleet-sync`, done. Remove it by deleting the entry.
+- **Workers an infrastructure tool creates:** `workerRecords` in `fleet.json`
+  names a file (relative to `fleet.json`) or a command that returns worker
+  records, the provider-neutral description of a machine in
+  [`schema/worker-record.schema.json`](../schema/worker-record.schema.json):
+  `id`, `tailnetName` or `address`, `user`, `port`, `hostKey` and optional
+  `labels` and `billing`. Their workers join the inline list; dev-fleet only
+  reads them and never calls a provider or creates a machine. A record that
+  appears is a declared worker, enrolled by the next `fleet-sync`; one that
+  disappears (a file that is gone holds none) is retired by the next
+  `fleet-sync` exactly as `fleet-node-retire` retires one. A worker listed by
+  hand is never retired by itself. An invalid record is refused naming its
+  field, such as `$.workerRecords[0].workers[1].hostKey`.
+  [`examples/workers-from-infrastructure`](../examples/workers-from-infrastructure/main.tf)
+  shows the handoff with a local virtual machine, the seed going in and the
+  record coming out. The canary stays the one `canary` names, so the order
+  of generated records means nothing.
+- **Replace or retire a worker:** `fleet-node-replace` and `fleet-node-retire`
+  take everything of the old machine with it: its pinned host key, its
+  monitoring system (pointed at the new machine, or deleted), the manager's
+  T3 link to its environment, its Vault token and its tailnet device. The
+  manager's T3 pairs with the replacement's environment by itself
+  (`t3:link`): at once when its T3 runs, otherwise in the next `fleet-sync`
+  once it does, and the roster records the new environment. The pairing code
+  is minted on the worker over SSH and handed to the manager's own T3 server
+  in a request body, never on a command line, in a report or in a log. The device is deleted through the
+  control plane fleet.json declares under `tailnet`, asked with the secret
+  `tailnet-admin` (workers are not registered as ephemeral devices); without
+  either, `tailscale:device` is skipped and names the device to remove in the
+  admin console. `fleet-status --workers` warns about a monitoring system that
+  matches no declared worker (`beszel:leftover`), and `fleet-status` about a
+  device carrying the worker tag that is no declared worker, parked or not,
+  and no device the roster binds (`tailscale:leftover`).
+
+## Parked and idle workers
+
+A rented worker costs money while it runs. `devfleet fleet-node-park --id ID`
+records in the fleet state that it is stopped or destroyed on purpose: from
+then on `fleet-sync` and `fleet-status` list it as parked with the time since
+(`worker:parked`, not applicable), never contact it, never make it the
+canary, and it never changes an exit code. Parking refuses while agent turns
+run on it (`WORKER_BUSY`) unless `--force`. `devfleet fleet-node-resume --id
+ID` marks it active and runs `fleet-sync --node ID`, which converges and
+verifies it. `parking.stop` and `parking.start` in `fleet.json` name commands
+(an infrastructure tool, never a provider dev-fleet calls) run with the worker
+ID as their last argument. A parked worker found online on the tailnet is a
+warning (`worker:parked-online`); one that comes back as a new machine goes
+through `fleet-node-replace`.
+
+`fleet-status` reports, for every worker it reaches, its uptime, the agent
+turns under way and the time of the last agent activity, read from what T3
+records (status and times, never a session's content), in its text and as
+`workers` in its JSON. With `parking.idleAfterHours`, a worker with no agent
+activity for longer is `worker:idle` (a warning whose remedy is to park it);
+one with a turn under way never is. `billing.hourlyRate` on a worker (or its
+record) is only displayed and summed: `usage` says how many workers run, how
+many are idle and what the idle ones cost per day. `parking.autoPark: true`
+lets `fleet-watch` park idle workers by itself; it is off unless set.
+
 - **Replace the manager:** follow
   [replacing the manager](manager-recovery.md).

package/docs/ubuntu-sync.md

@@ -1,6 +1,7 @@
 > This document was written for the fleet dev-fleet was extracted from. Paths and
 > commands were updated; some statements still describe that fleet and are being
-> revised. The README is authoritative.
+> revised. The README is authoritative, and what a machine must offer to be a
+> worker is listed only in [worker requirements](worker-requirements.md).
 
 # Developer fleet: one manager (macOS) and stateless workers (Ubuntu)
 
@@ -10,7 +11,7 @@ What is proven, and where:
 
 | Evidence                                                                 | Where                                                                                                                                     |
 | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
-| A bare Ubuntu 26.04 machine bootstrapped over real SSH, then repeated    | `features/fresh-box/fresh-box.feature`, on every pull request (amd64 and arm64)                                                           |
+| A bare Ubuntu machine bootstrapped over real SSH, then repeated          | `features/fresh-box/fresh-box.feature`, on every pull request (each accepted release, amd64 and arm64)                                    |
 | One run over several machines, one versioned report, no partial receipt  | `features/fresh-box/fleet.feature`                                                                                                        |
 | Interrupted checkout, package installation and toolchain setup resumed   | `features/fresh-box/fleet.feature`                                                                                                        |
 | Overlapping runs, a dropped connection, a held APT lock, no DNS, no disk | `features/fresh-box/fleet.feature`                                                                                                        |
@@ -86,9 +87,11 @@ existing SSH aliases, tests strict SSH access, and offers one enrollment
 confirmation for the verified connections. The apply approval binds a fingerprint
 of the selected revision, policy and connection records; changed bindings fail
 with `FLEET_PLAN_STALE` before source preparation or remote mutation. It stores private overrides mode 0600. Unknown host keys are never accepted automatically. Compare fingerprints
-out of band before enrolling trust. Unreachable machines remain listed; first
-reachability requires an existing trusted LAN SSH route or physical setup of
-SSH/Tailscale. No software on the manager can bootstrap a disconnected box.
+out of band before enrolling trust. Unreachable machines remain listed; what a machine
+needs to be reachable the first time (SSH with a key, a known host key, the
+tailnet, or first-boot data that provides them) is listed in
+[worker requirements](worker-requirements.md). No software on the manager can
+bootstrap a disconnected box.
 
 A previously configured manager uses saved connection bindings and its approved
 revision. A changed Tailscale device ID is a replacement gate, even if the old
@@ -177,37 +180,43 @@ independently. Bespoke TypeScript remains only where a native CLI or API has no
 declarative owner: T3 service lifecycle, Beszel enrollment tokens, provider
 verification and the CLIProxy singleton.
 
-| Policy                      | Workers (Ubuntu 26.04, stateless)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Manager (macOS, singleton)                                                                                                                                                                                                                                                                                                                                                                                |
-| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Never sleep                 | systemd sleep targets masked; logind lid/idle policy; GNOME power plugin set to `nothing`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | never sleeps on AC power (`pmset -c sleep 0`, `macos/manager.yml`, verified as `manager:power`); battery sleep is left to the human                                                                                                                                                                                                                                                                       |
-| No automatic logout or lock | GNOME `lock-enabled false`, `idle-delay 0`; designated-user lingering                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | not managed (the human uses it)                                                                                                                                                                                                                                                                                                                                                                           |
-| Always-on services          | drop-ins `Restart=always`, `StartLimitIntervalSec=0` for ssh, tailscaled, docker, beszel-agent                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | singleton services under launchd and Docker Desktop (see the last row)                                                                                                                                                                                                                                                                                                                                    |
-| T3 health and connection    | `fleet-sync` and `fleet-status --workers` require `worker:t3-health` (local API answers), `t3:manager-endpoint` (manager HTTPS over Tailscale) and `t3:paired-access` (manager T3 link reachable)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | the human's T3 desktop holds the links to every worker (`t3:paired-access`)                                                                                                                                                                                                                                                                                                                               |
-| Memory safety               | `systemd-oomd` (swap 90%, pressure 60%); `user.slice` `MemoryMax=90%`; each `t3code-*.scope` agent `MemoryMax=30%`, `OOMPolicy=continue`; swap file when none                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
-| Disk safety                 | `personal-fleet-disk-guard` kills only the top user-cgroup writer below 3 GiB free, never deletes files; hourly `personal-fleet-disk-gc.timer` deletes git-ignored build output (`node_modules`, `.cache`, `dist`, ...) in checkouts idle 7d (12h under 15 GiB free) with no process inside, plus unused Docker images and build cache, never source, worktrees or volumes; journal 1G; `/tmp` 25% of RAM, aged 2d hourly, a measured write probe (`TMP_QUOTA_EXHAUSTED`), and T3 agents use disk-backed `/var/tmp` (`TMPDIR`, aged 14d); Docker log rotation                                                                                                                                                                       | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
-| Tailscale                   | pinned APT package, `tailscaled` enabled, native login required                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Tailscale app, human login                                                                                                                                                                                                                                                                                                                                                                                |
-| Dashboard agent             | pinned native binary, hardened `beszel-agent` systemd unit as user `beszel`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | not installed                                                                                                                                                                                                                                                                                                                                                                                             |
-| Credential renewal          | user timer `personal-worker-auth` renews the node's own Vault token every 10min; `personal-worker-github` signs it in to GitHub from the stored token ([GitHub](github.md))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | the human's own Vault and GitHub sessions                                                                                                                                                                                                                                                                                                                                                                 |
-| Noninteractive sudo         | validated `/etc/sudoers.d/personal-fleet-USER`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | never; the human runs `manager.yml` once with `-K`                                                                                                                                                                                                                                                                                                                                                        |
-| T3 worker app               | pinned desktop `.deb` is the only T3 process (UI plus embedded server); GDM auto-login, user unit `t3code-desktop.service` bound to the graphical session with `Restart=always`; `personal-worker-recovery` runs a copy of `t3-health.sh` installed in `/usr/local/lib/personal-fleet` every minute: it stops and disables a still-running headless `t3code.service`, reports `T3_NO_GRAPHICAL_SESSION` and restarts the display manager (three times at most) when there is no session, starts a stopped app after `reset-failed`, restarts one whose local API missed three checks, and restarts one on another release than the installed one or with a deferred policy restart once it runs no agent turn; administrator policy | not a worker                                                                                                                                                                                                                                                                                                                                                                                              |
-| T3 providers                | the same managed instances (`fleet/agents/t3.json`) through locked T3 administrator policy                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | reconciled into the human's T3 settings by `fleet-sync` (`t3:providers`) with the ownership-safe `t3-apply` engine, so every T3 offers the identical set; unrelated and unmanaged instances are preserved. `fleet-sync` also applies the same proxy agent profiles (`configuration:apply`) and verifies the shared Vault gateway credential end to end with `provider:*:inference`, exactly as on workers |
-| Singleton services          | never; clients of the manager's services only                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | every singleton runs here: CLI proxy (`cliproxy`), Beszel hub (`workstation/monitoring`)                                                                                                                                                                                                                                                                                                                  |
+| Policy                      | Workers (Ubuntu 26.04 or 24.04, stateless)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Manager (macOS, singleton)                                                                                                                                                                                                                                                                                                                                                                                |
+| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Never sleep                 | systemd sleep targets masked; logind lid/idle policy; GNOME power plugin set to `nothing`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | never sleeps on AC power (`pmset -c sleep 0`, `macos/manager.yml`, verified as `manager:power`); battery sleep is left to the human                                                                                                                                                                                                                                                                       |
+| No automatic logout or lock | GNOME `lock-enabled false`, `idle-delay 0`; designated-user lingering                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | not managed (the human uses it)                                                                                                                                                                                                                                                                                                                                                                           |
+| Always-on services          | drop-ins `Restart=always`, `StartLimitIntervalSec=0` for ssh, tailscaled, docker, beszel-agent                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | singleton services under launchd and Docker Desktop (see the last row)                                                                                                                                                                                                                                                                                                                                    |
+| T3 health and connection    | `fleet-sync` and `fleet-status --workers` require `worker:t3-health` (local API answers), `t3:manager-endpoint` (manager HTTPS over Tailscale) and `t3:paired-access` (manager T3 link reachable)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | the human's T3 desktop holds the links to every worker (`t3:paired-access`)                                                                                                                                                                                                                                                                                                                               |
+| Memory safety               | `systemd-oomd` (swap 90%, pressure 60%); `user.slice` `MemoryMax=90%`; each `t3code-*.scope` agent `MemoryMax=30%`, `OOMPolicy=continue`; swap file when none                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
+| Disk safety                 | `personal-fleet-disk-guard` kills only the top user-cgroup writer below 3 GiB free, never deletes files; hourly `personal-fleet-disk-gc.timer` deletes git-ignored build output (`node_modules`, `.cache`, `dist`, ...) in checkouts idle 7d (12h under 15 GiB free) with no process inside, plus unused Docker images and build cache, never source, worktrees or volumes; journal 1G; `/tmp` 25% of RAM, aged 2d hourly, a measured write probe (`TMP_QUOTA_EXHAUSTED`), and T3 agents use disk-backed `/var/tmp` (`TMPDIR`, aged 14d); Docker log rotation                                                                                                                                                                                                                                                                                                                                        | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
+| Tailscale                   | pinned APT package, `tailscaled` enabled, native login required                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | Tailscale app, human login                                                                                                                                                                                                                                                                                                                                                                                |
+| Dashboard agent             | pinned native binary, hardened `beszel-agent` systemd unit as user `beszel`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | not installed                                                                                                                                                                                                                                                                                                                                                                                             |
+| Credential renewal          | user timer `personal-worker-auth` renews the node's own Vault token every 10min; `personal-worker-github` signs it in to GitHub from the stored token ([GitHub](github.md))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | the human's own Vault and GitHub sessions                                                                                                                                                                                                                                                                                                                                                                 |
+| Noninteractive sudo         | validated `/etc/sudoers.d/personal-fleet-USER`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | only `pmset -c`, granted once by `manager-privilege.yml` (`-K`)                                                                                                                                                                                                                                                                                                                                           |
+| T3 worker app               | pinned desktop `.deb` is the only T3 process (UI plus embedded server) on a desktop worker, and the pinned server as the lingering user unit `t3code-headless.service` on a headless one (`features.session`; no session is waited for); GDM auto-login, user unit `t3code-desktop.service` bound to the graphical session with `Restart=always`; `personal-worker-recovery` runs a copy of `t3-health.sh` installed in `/usr/local/lib/personal-fleet` every minute: it stops and disables a still-running headless `t3code.service`, reports `T3_NO_GRAPHICAL_SESSION` and restarts the display manager (three times at most) when there is no session, starts a stopped app after `reset-failed`, restarts one whose local API missed three checks, and restarts one on another release than the installed one or with a deferred policy restart once it runs no agent turn; administrator policy | not a worker                                                                                                                                                                                                                                                                                                                                                                                              |
+| T3 providers                | the same managed instances (`fleet/agents/t3.json`) through locked T3 administrator policy                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | reconciled into the human's T3 settings by `fleet-sync` (`t3:providers`) with the ownership-safe `t3-apply` engine, so every T3 offers the identical set; unrelated and unmanaged instances are preserved. `fleet-sync` also applies the same proxy agent profiles (`configuration:apply`) and verifies the shared Vault gateway credential end to end with `provider:*:inference`, exactly as on workers |
+| Singleton services          | never; clients of the manager's services only                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | every singleton runs here: CLI proxy (`cliproxy`), Beszel hub (`workstation/monitoring`)                                                                                                                                                                                                                                                                                                                  |
 
 Credential sync is deliberately indirect: no secret is copied between machines.
-OpenTofu issues each enrolled worker its own renewable periodic token
-(`vault.tokens` entries `fleet-worker-<node>`, policy `personal-read`, sunk into
-`secret/personal/prd` as `FLEET_WORKER_TOKEN_<NODE>`). `devfleet fleet-sync`
-delivers each node's token automatically over SSH stdin (never as an argument)
-whenever that node has no working Vault session. Interactive runs first log the
-manager in (`vault login -method=userpass`, password typed on your terminal)
-and fall back to the same password login on a node whose token is not issued.
-Without either, the node reports `VAULT_LOGIN_REQUIRED` and its T3 proxy
-providers stay unavailable. The manual equivalent of the automatic delivery:
-
-```sh
-vault kv get -field=FLEET_WORKER_TOKEN_WORKER_ONE secret/personal/prd \
-  | ssh worker-one 'vault login -address=https://vault.example.com -no-print -'
-```
+Each worker has a Vault token of its own, kept in a document of its own,
+`<mount>/<project>/workers/<id>` (fields `token` and `accessor`; `<project>`
+is the first segment of the `proxy-client-key` path). The manager issues it
+with `devfleet fleet-node-replace --id ID --apply --yes`: a periodic orphan
+token under the policy `fleet-worker-<id>`, which reads the fleet's shared
+document and the worker's own document and nothing else, so one worker's
+token cannot read another's (`features/fleet-sync/worker-credentials.feature`
+proves it against a Vault development server). The manager's own login needs
+to write policies, create tokens and write the mount.
+
+`devfleet fleet-sync` delivers each node's token over SSH stdin (never as an
+argument) whenever that node has no working Vault session. A token still kept
+in the shared document as `FLEET_WORKER_TOKEN_<NODE>` is delivered too, with
+the warning `VAULT_WORKER_TOKEN_SHARED`: every worker can read that document,
+so issue the worker its own token and remove the field. A password login with
+the fleet's shared Vault user on the worker's own terminal happens only with
+`fleet-sync --vault-password-login`. Without a token the node reports
+`VAULT_LOGIN_REQUIRED` and its T3 proxy providers stay unavailable.
+`devfleet fleet-node-retire --id ID --apply --yes` revokes a worker's token by
+its accessor and removes its document and policy.
 
 From then on the node's `personal-worker-auth` timer renews that token and its
 `personal-worker-github` timer keeps its GitHub sign-in current
@@ -230,14 +239,58 @@ GNOME lock/idle changes apply to the designated user only. Effective running
 session/logind behavior still needs a maintenance/reboot acceptance check; writing
 a file does not prove logout/reboot survival. The sudoers fragment is validated
 with visudo before installation; this user and its agents can administer the
-machine. SSH authentication remains enabled.
+machine.
+
+SSH changes validate the complete configuration before reload and restore the
+previous fragments on validation failure. sshd takes keys only and refuses
+root (`10-personal-fleet-exposure.conf`, which sorts before an image's own
+fragments; `SSH_HARDENING_OVERRIDDEN` when `sshd -T` still shows otherwise).
+Existing listeners and socket activation are preserved; unmanaged fragments
+require review. The manager makes a **new** strict SSH connection after apply.
+
+### Exposure
+
+Every worker, with a public address or without one, admits inbound traffic
+only over loopback and the tailnet (`ansible/ubuntu/tasks/firewall.yml`, the
+`inet personal_fleet` nftables table, loaded at boot by
+`personal-fleet-firewall.service`). Open beyond it: tailscaled's UDP port,
+DHCP replies, ICMP, and what the exposure policy opens. Where Docker is
+installed, a published container port is dropped from outside the tailnet
+before Docker's own chains see it. Docker's and Tailscale's tables are left to
+their owners.
+
+The policy is `exposure` in `fleet.json`, fleet-wide under `worker` and per
+worker beside `network`; a field a worker names replaces the fleet-wide one:
+
+```json
+"exposure": {
+  "publicSsh": false,
+  "publicPorts": [{ "protocol": "udp", "port": 51820 }],
+  "otherListeners": "warn"
+}
+```
 
-SSH forwarding changes validate the complete configuration before reload and
-restore the previous fragment on validation failure. Existing listeners,
-authentication and firewall rules are preserved. Existing socket activation is preserved for forwarding-only changes; listener
-migrations and unmanaged fragments require review. The manager makes a **new**
-strict SSH connection after apply. Private-listener/firewall acceptance and
-failure-after-reload recovery still require disposable integration testing.
+`publicSsh` is break-glass access: SSH stays open beyond the tailnet.
+Without it the public SSH port closes only when the session running the sync
+arrived over the tailnet, after the tailnet path answers a new probe and sshd
+a new connection on the tailnet address; a failed verification restores the
+previous rules (`FIREWALL_TAILNET_UNVERIFIED`). A session from a public
+address, a tailnet that is not up, or a run without a session that finds the
+port open keeps it open, and `worker:exposure` reports
+`WORKER_PUBLIC_SSH_UNDECLARED` until the manager's SSH alias names the
+worker's tailnet name.
+
+`worker:exposure` (on the worker) lists listening sockets and fails on one
+bound beyond loopback and the tailnet outside the allow-list (sshd, tailscaled,
+the system's DHCP and name-resolution listeners, `publicPorts`), naming the
+port and its unit. A fleet service there always fails; anything else, such as
+an agent's development server, warns or fails by `otherListeners`.
+`worker:public-reachability` (from the manager, in `fleet-status --workers`)
+tries the SSH, T3 and monitoring ports on the public address Tailscale knows
+for the worker; it never blocks readiness, because the manager may share the
+worker's network. Both are not applicable without an address outside the
+tailnet. Closing the public path on a machine with two interfaces still needs
+the disposable VM or a real machine to prove.
 
 The manager is the only configuration coordinator. No per-node pull timer is
 installed. Manager schedule controls use the existing platform adapter:
@@ -321,7 +374,10 @@ A worker whose T3 has not loaded the staged provider policy, or runs an older
 release than the one installed, is restarted only while it runs no agent turn;
 otherwise sync reports the pending restart and the recovery timer applies it
 at the first idle check, failing the check after
-`t3RestartMaxDeferralHours` (`ansible/ubuntu/policy.json`). A pending proxy
+`t3RestartMaxDeferralHours` (`ansible/ubuntu/policy.json`).
+`fleet-sync --restart-busy` passes `t3_restart_busy` to the plays, which then
+restart a busy T3 at once and end its turns; the `--dry-run` plan says, per
+T3 restart, whether running turns end. A pending proxy
 update is installed even when it recreates the gateway. The manager finishes before the workers
 start, so they are verified against the updated proxy. `--dry-run` previews.