@crvouga/dev-fleet 0.7.0 → 0.9.0

package/docs/worker-contract.md

@@ -1,6 +1,7 @@
 > This document was written for the fleet dev-fleet was extracted from. Paths and
 > commands were updated; some statements still describe that fleet and are being
-> revised. The README is authoritative.
+> revised. The README is authoritative, and what a machine must offer to be a
+> worker is listed only in [worker requirements](worker-requirements.md).
 
 # Declarative remote workers
 
@@ -12,11 +13,16 @@ devfleet fleet-status --workers
 ```
 
 The manager needs Node/npm, Git, SSH, Tailscale, and its own authorized T3 link.
-Initially each worker (an Ubuntu release the [support matrix](support-matrix.md)
-lists) needs a
-designated non-root account, trusted SSH access and administrator authorization. Standard Ubuntu cloud-image users
+What each worker must offer beforehand is listed once, in
+[worker requirements](worker-requirements.md). Standard Ubuntu cloud-image users
 already commonly have noninteractive sudo; otherwise establish the repository's
-`authorize-sudo.sh` policy once during administrator enrollment. Routine commands
+`authorize-sudo.sh` policy once during administrator enrollment. An image that
+lets only root in (many rented servers) needs no hands either: when the
+account `workers[].network.user` names cannot log in yet, or the SSH alias
+logs in as root, `fleet-sync` creates that account once over the root login
+(`ansible/ubuntu/worker-account.yml`: the key root was reached with and the
+fleet's sudo rule) and continues as it. The developer toolchain never runs as
+root, and the SSH policy turns root login off afterwards. Routine commands
 never ask for or store an SSH/sudo password. Unknown host keys, a different device
 identity, or missing privilege stop with a named action rather than bypassing trust.
 
@@ -27,18 +33,110 @@ Ansible policy. Developer checkouts and dirty work stay intact. A published CLI
 uses its source provenance; a source project action explicitly selects checkout
 HEAD. Private rosters contain connection paths and identities, not weaker policy.
 
-| Requirement                        | Declaration and enforcement                                                                                                                                                                                |
-| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| One consistent dependency set      | `mise.toml` worker profile, `mise.lock`, Ubuntu snapshot/package pins; exact versions and deployed input hashes checked live                                                                               |
-| Common service access              | Canonical required capabilities; declared GitHub account, worker's own renewable Vault session, authenticated singleton gateway catalog                                                                    |
-| One gateway host                   | The CLI proxy runs only on the manager; ownership/audit/recovery code rejects any install on a worker and missing singleton state                                                                          |
-| T3 connected and persistent        | Pinned desktop app as the only T3 process, auto-login session unit enabled/active with restart policy, bounded recovery timer, private endpoint, manager link matched to SSH-observed environment identity |
-| No idle sleep and session survival | systemd sleep masks, logind policy, designated-user lingering; running policy and boot timing checked, not inferred from files alone                                                                       |
-| Recovery after power loss          | Required boot services and recovery timers; recognized firmware restore-after-AC settings inspected when exposed; macOS `autorestart` where supported                                                      |
-| Always-on services                 | `Restart=always` drop-ins for ssh, tailscaled, docker and beszel-agent (launchd `KeepAlive` on macOS); health requires the running unit to report that policy                                              |
-| Central dashboard                  | Pinned Beszel hub on the manager, every worker enrolled with its own token; health verifies hub reachability, registration, connection and fresh samples                                                   |
-| No routine password prompts        | Verified BatchMode SSH, `sudo -n`, source-owned NOPASSWD policy; missing initial authorization is a failure                                                                                                |
-| Keep service authorization alive   | Native service credentials stay per-node; Vault self-renewal timer and GitHub recovery from authorized Vault credentials; expiring T3 links are reported before expiry                                     |
+| Requirement                        | Declaration and enforcement                                                                                                                                                                                                                                                   |
+| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| One consistent dependency set      | `mise.toml` worker profile, `mise.lock`, Ubuntu snapshot/package pins; exact versions and deployed input hashes checked live                                                                                                                                                  |
+| Common service access              | Canonical required capabilities; declared GitHub account, worker's own renewable Vault session, authenticated singleton gateway catalog                                                                                                                                       |
+| One gateway host                   | The CLI proxy runs only on the manager; ownership/audit/recovery code rejects any install on a worker and missing singleton state                                                                                                                                             |
+| T3 connected and persistent        | One pinned T3 process: the desktop app in the auto-login session, or on a headless worker the pinned server as a lingering user unit; enabled/active with restart policy, bounded recovery timer, private endpoint, manager link matched to SSH-observed environment identity |
+| No idle sleep and session survival | systemd sleep masks, logind policy, designated-user lingering; running policy and boot timing checked, not inferred from files alone                                                                                                                                          |
+| Recovery after power loss          | Required boot services and recovery timers; recognized firmware restore-after-AC settings inspected when exposed (not applicable on a virtual machine); macOS `autorestart` where supported                                                                                   |
+| Always-on services                 | `Restart=always` drop-ins for ssh, tailscaled, docker (with `containers`) and beszel-agent (launchd `KeepAlive` on macOS); health requires the running unit to report that policy                                                                                             |
+| Central dashboard                  | Pinned Beszel hub on the manager, every worker enrolled with its own token; health verifies hub reachability, registration, connection and fresh samples                                                                                                                      |
+| No routine password prompts        | Verified BatchMode SSH, `sudo -n`, source-owned NOPASSWD policy; missing initial authorization is a failure                                                                                                                                                                   |
+| Keep service authorization alive   | Native service credentials stay per-node; Vault self-renewal timer and GitHub recovery from authorized Vault credentials; expiring T3 links are reported before expiry                                                                                                        |
+
+## Worker features
+
+What a worker is built with beyond the base is declared in `fleet.json`:
+fleet-wide under `worker.features`, or for one worker under its own
+`features`. Each block may name a `preset` first; its own fields override
+it. A fleet that names no feature gets what every worker had before features
+existed.
+
+| Feature      | Values                | Default   | What it decides                                                                                                                                    |
+| ------------ | --------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `containers` | `true`, `false`       | `true`    | Docker: the Ubuntu packages, `docker.service`, the developer's `docker` group membership, the `docker-cli` tool and the Docker step of the disk GC |
+| `session`    | `desktop`, `headless` | `desktop` | How T3 runs: the desktop app in an auto-login graphical session, or the pinned server as a lingering user service with no display manager          |
+| `preset`     | `editing`             | none      | `editing`: no containers, a headless session and only the base tools (it adds no tool profile)                                                     |
+
+What each feature installs is listed once, under `features` in
+`ansible/ubuntu/policy.json`. A worker without it installs none of it, and its
+checks (`worker:package:*`, `worker:service:docker`, `worker:docker-run`,
+`tool:docker-cli`) report **not applicable** with the reason. Turning
+`containers` off on an existing worker removes what dev-fleet installed for it:
+the packages (not purged: images and volumes under `/var/lib/docker` stay
+until a person deletes them), the restart drop-in and the developer's group
+membership. `fleet-sync --dry-run` lists what each worker will lose (`plan`,
+under `removals`), and `worker:features` reports what the run removed and
+fails with `WORKER_FEATURE_REMOVAL_INCOMPLETE` while any of it remains. A
+headless worker installs no desktop-only package but removes none: a desktop
+the machine came with may depend on them.
+
+## The class of a machine
+
+The probe records each worker's class before anything is installed
+(`platform:machine-class`): **physical**, **virtual** (systemd-detect-virt
+names a hypervisor) or unsupported (a container or WSL2, refused by
+`platform:environment`). A virtual machine has no lid, no battery and no
+firmware its guest can read, and its provider restarts it after a power loss,
+so `worker:power-loss-recovery` and `ubuntu:lid-policy` report **not
+applicable**, with the reason and the worker's optional `powerRecoveryNote`
+(for example "provider restarts instances"). The sleep masks are still
+checked: a guest can be suspended by its own idle policy. On a physical
+machine nothing changes. A not-applicable check is counted apart, shown with
+its reason in text and in `--json`, and never fails a run, not even with
+`--strict`.
+
+## Size
+
+The minimum size of a worker for the editing workload is stated in
+[worker requirements](worker-requirements.md), from `sizing.minimumWorker` in
+`ansible/ubuntu/policy.json`. The probe measures it before anything is
+installed and refuses a smaller machine with `WORKER_SIZE_BELOW_MINIMUM`
+(`platform:worker-size`); the baseline asserts the same before it installs a
+package.
+
+The disk dev-fleet itself installs is measured on every pull request: the
+fresh-box CI job converges a stock Ubuntu image of each accepted release on
+amd64 and arm64, measures what each part takes, and fails when a part
+outgrows its budget in `sizing.footprintMiB`
+(`features/fresh-box/fresh-box.feature`). Measured on Ubuntu 26.04:
+
+| Part                                         | amd64        | arm64        | Budget   |
+| -------------------------------------------- | ------------ | ------------ | -------- |
+| Base tools with the agent CLIs, and `go`     | 5184 MiB     | 4830 MiB     | 6144 MiB |
+| The fleet's checkouts (up to four revisions) | 189 MiB each | 189 MiB each | 1024 MiB |
+| Tool profile `web` (its tools)               |              | 135 MiB      | 256 MiB  |
+| Tool profile `rust`                          |              | 1562 MiB     | 2048 MiB |
+| Tool profile `jvm`                           | 437 MiB      | 303 MiB      | 640 MiB  |
+
+What is not measured there, and stays an estimate: Ubuntu Server itself
+(about 3 to 5 GiB), the system packages of the baseline, the headless
+Chromium the `web` profile installs as packages, the swap file (half of
+memory), and the engine and images of `containers`. The minimum adds those
+to the measured parts with room for the disk floors below.
+
+The disk and memory lines of the worker policy scale with the machine: each
+is a share of the root filesystem or of memory between a lower and an upper
+bound (`sizing.thresholds`), and the upper bounds are the fixed numbers every
+worker had before.
+
+| Line                               | Share         | Lower    | Upper     |
+| ---------------------------------- | ------------- | -------- | --------- |
+| Swap file (created only when none) | 50% of memory | 1024 MiB | 8192 MiB  |
+| Free disk the baseline keeps       | 5% of disk    | 2048 MiB | 4096 MiB  |
+| Disk guard floor                   | 3% of disk    | 1024 MiB | 3072 MiB  |
+| Disk GC pressure line              | 15% of disk   | 4096 MiB | 15360 MiB |
+| `worker:capacity:disk` warns under | 10% of disk   | 3072 MiB | 10240 MiB |
+| `worker:capacity:disk` fails under | 2% of disk    | 1024 MiB | 2048 MiB  |
+
+`worker:agent-sessions` reports how many agent sessions run and how many the
+memory policy supports at once: the memory user workloads may use (90%), less
+what T3 keeps for itself (1536 MiB), in sessions of an estimated 1024 MiB each
+(`sizing.agentSessionMiB`, also an estimate). More sessions than that warn
+with `AGENT_SESSION_CAPACITY_EXHAUSTED`; each agent is still capped at 30% of
+memory, so the largest is stopped under pressure rather than the machine.
 
 ## Package pins and security updates
 
@@ -64,12 +162,17 @@ one:
 A package run cut off by a power loss is finished by the next sync: the
 baseline waits for the APT lock, runs `dpkg --configure -a` once (bounded to
 ten minutes) and audits again; only a database still inconsistent stops with
-`APT_STATE_BROKEN`.
+`APT_STATE_BROKEN`. When the cut-off run left one of the bootstrap's own
+packages unconfigured (an upgrade of `python3` or `git`, say), the bootstrap
+makes the same repair before it installs what it lacks, whoever started the
+run, waiting up to two minutes for a package process that holds the lock;
+only a database still inconsistent stops it with `DPKG_INTERRUPTED`.
 
 Health does not mutate configuration, renew tokens, log in, install software, or
 make billable inference requests. It audits the immutable desired worker deployment,
 not an unrelated developer branch. Every declared worker must be enrolled and
-reachable. Extra service hosts are listed as outside scope. Required failures,
+reachable, unless it is parked (`devfleet fleet-node-park`): a parked worker
+is listed as parked and never contacted. Extra service hosts are listed as outside scope. Required failures,
 warnings and unverified checks prevent a ready result; explicitly non-required
 checks remain visible without inventing a pass.
 

package/docs/worker-requirements.md

@@ -0,0 +1,107 @@
+# What a machine must offer to be a worker
+
+The one list of what a machine needs before it can be a worker, wherever it
+runs: a computer at home, a virtual machine on your own host or an instance
+rented from any provider. Everything else, `fleet-sync` installs. No vendor is
+required; any provider that boots a stock Ubuntu image with SSH and cloud-init
+user data works the same way.
+
+Each requirement that a check enforces names the check (from
+[schema/check-ids.json](../schema/check-ids.json)) and the failure code it
+reports (each has an entry in the [runbooks](runbooks.md)). A dash means the
+check reports no code of its own; its summary and remedy say what failed.
+`features/rules/worker-requirements.feature` fails when a check or code here
+does not exist, when a check marked as a worker requirement in the registry
+has no row here, or when the image, architecture and size rows disagree with
+`ansible/ubuntu/policy.json`.
+
+## The machine
+
+| Requirement                                                                                                                                                                                                      | How to check it                                       | Check                  | Failure code                     |
+| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ---------------------- | -------------------------------- |
+| A stock Ubuntu image of an accepted release: **26.04** (resolute) or **24.04** (noble). No distribution upgrade is ever made.                                                                                    | `. /etc/os-release; echo $VERSION_ID`                 | `ubuntu:release`       | `UBUNTU_BASELINE_MISMATCH`       |
+| An accepted architecture: **amd64** (x86_64) or **arm64** (aarch64).                                                                                                                                             | `uname -m`                                            | `ubuntu:release`       | `UBUNTU_BASELINE_MISMATCH`       |
+| A machine of its own: physical, or a full virtual machine (KVM, Xen, VMware, Hyper-V, or any hypervisor `systemd-detect-virt --vm` names). A container (Docker, LXC, systemd-nspawn, OpenVZ) or WSL2 is refused. | `systemd-detect-virt`                                 | `platform:environment` | `WORKER_ENVIRONMENT_UNSUPPORTED` |
+| At least **2 CPUs**, **3584 MiB** of memory (a 4 GiB instance) and a **20480 MiB** root filesystem. What dev-fleet installs is measured in CI; the operating system is estimated.                                | `nproc; free -m; df -m /`                             | `platform:worker-size` | `WORKER_SIZE_BELOW_MINIMUM`      |
+| Survives one reboot: the first converge ends with a reboot while no agent runs, and the worker must answer SSH again within 15 minutes.                                                                          | `devfleet fleet-status --workers` after the first run | `worker:reboot`        | `WORKER_REBOOT_FAILED`           |
+
+The minimum size is `sizing.minimumWorker` in `ansible/ubuntu/policy.json`;
+[the worker contract](worker-contract.md#size) explains the estimate and what
+grows with the machine. The accepted releases and architectures are `releases`
+in the same file; [the support matrix](support-matrix.md) lists every
+platform and why the others are refused.
+
+## Access
+
+| Requirement                                                                                                                                                                                          | How to check it                          | Check                     | Failure code                        |
+| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | ------------------------- | ----------------------------------- |
+| SSH with a key, from the manager, without a password.                                                                                                                                                | `ssh -o BatchMode=yes WORKER true`       | `ssh:auth`                | -                                   |
+| A host key the manager knows before the first connection: generated by `devfleet fleet-node-seed`, given as `hostKey` in a worker record, or pinned with `devfleet fleet-node-replace --host-key`.   | `devfleet fleet-status --workers`        | `ssh:host-key`            | -                                   |
+| A non-root account that may use `sudo` without a password, as `workers[].network.user` names it. An image that lets only root in is fine: `fleet-sync` creates the account over the root login once. | `ssh WORKER 'id -u; sudo -n true'`       | `bootstrap:privilege`     | `SUDO_AUTHORIZATION_REQUIRED`       |
+| With only a root login, `workers[].network.user` names the account to create.                                                                                                                        | `fleet.json`                             | `worker:account`          | `WORKER_ACCOUNT_UNDECLARED`         |
+| Joined to the fleet's tailnet: by the seed at first boot, or by `fleet-sync` with a single-use key the control plane `fleet.json` declares (`tailnet`) mints for it.                                 | `tailscale status --json` on the manager | `tailscale:configuration` | `TAILSCALE_SETUP_OR_LOGIN_REQUIRED` |
+
+## First-boot data
+
+A rented machine is set up from cloud-init user data and nothing else.
+`devfleet fleet-node-seed --id ID --output PATH` writes it: the account with
+the manager's key, the sudo rule, a host key generated for the machine, the
+pinned Tailscale and a short-lived single-use key. Any image that runs
+cloud-init accepts it (`--autoinstall` writes an Ubuntu autoinstall file for a
+machine installed from the ISO). The seed holds a secret: pass it to the
+provider as user data and delete it afterwards.
+
+| Requirement                                                                                                     | How to check it            | Check                  | Failure code                              |
+| --------------------------------------------------------------------------------------------------------------- | -------------------------- | ---------------------- | ----------------------------------------- |
+| The image runs cloud-init, or nothing at first boot. `fleet-sync` waits for a first boot that is still running. | `cloud-init status --wait` | `bootstrap:first-boot` | `FIRST_BOOT_FAILED`, `FIRST_BOOT_PENDING` |
+
+## Network
+
+No inbound connection from the internet is needed: the manager reaches a
+worker over the tailnet. A fixed or public address is not needed either.
+Outbound, the worker opens HTTPS (TCP 443) to:
+
+- the Ubuntu archive its image is configured with, and `snapshot.ubuntu.com`
+  (every pinned package);
+- `pkgs.tailscale.com` (Tailscale) and the fleet's tailnet control plane:
+  `tailnet.url` in `fleet.json`, or the hosted control plane when it names
+  none; Tailscale also uses UDP 41641 and its relays (UDP 3478, TCP 443);
+- `github.com`, `api.github.com` and GitHub's release download hosts, and the
+  other download hosts `mise.lock` names
+  (`grep -o 'https://[^/"]*' mise.lock | sort -u` lists them);
+- the CLI proxy at `services.proxy.endpoint`, and the secret store
+  `fleet.json` names (Vault's address with `vault`).
+
+| Requirement                                                                                                                                                | How to check it                                | Check                        | Failure code                                               |
+| ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | ---------------------------- | ---------------------------------------------------------- |
+| Reaches the Ubuntu archive and the snapshot.                                                                                                               | `ssh WORKER 'sudo apt-get update'`             | `bootstrap:prerequisites`    | `APT_REPOSITORY_UNAVAILABLE`, `APT_DNS_UNAVAILABLE`        |
+| Reaches GitHub.                                                                                                                                            | `ssh WORKER 'curl -sI https://api.github.com'` | `worker:github-sync`         | `GITHUB_UNREACHABLE`                                       |
+| Reaches the CLI proxy.                                                                                                                                     | `devfleet fleet-status --workers`              | `proxy:reachable-from`       | -                                                          |
+| Accepts no inbound connection beyond loopback and the tailnet, but what `exposure` in `fleet.json` declares. The fleet firewall closes the rest by itself. | `sudo nft list table inet personal_fleet`      | `worker:exposure`            | `WORKER_LISTENER_UNRESTRICTED`, `WORKER_FIREWALL_INACTIVE` |
+| Its public address, when it has one, answers on no fleet port.                                                                                             | `devfleet fleet-status --workers`              | `worker:public-reachability` | `WORKER_PUBLIC_PORT_UNRESTRICTED`                          |
+
+A provider's own firewall (a security group) may stay closed to all inbound
+traffic: Tailscale needs none, and SSH from the internet is never needed once
+the seed has joined the machine to the tailnet.
+
+## What is not needed
+
+- A display, a desktop or a GPU: `"session": "headless"` (the `editing`
+  preset) runs T3 as a service with no graphical session.
+- A public inbound port, a fixed address or a domain name.
+- Git, curl, Python, Node, Bun or mise beforehand: `fleet-sync` installs them.
+- Docker beforehand: the `containers` feature (on unless a worker turns it
+  off) installs it.
+- A password for SSH or `sudo`, or anyone at the console.
+- An account with any particular provider.
+
+## From infrastructure as code
+
+[examples/workers-from-infrastructure](../examples/workers-from-infrastructure/main.tf)
+shows the handoff with a local virtual machine: the seed from
+`devfleet fleet-node-seed` goes in as user data, and a worker record
+(`schema/worker-record.schema.json`) comes out for `fleet.json` to read. For a
+hosted provider only these change: the provider block and its credentials, the
+image (the provider's stock Ubuntu image of an accepted release), the instance
+type (at least the size above) and the network (a private network, or an
+inbound rule set with nothing open). The seed and the record stay the same.

package/examples/fleet.json

@@ -110,6 +110,7 @@
     "vaultUser": "example-user",
     "toolchain": "mise.lock",
     "toolProfiles": ["go"],
+    "features": { "containers": true, "session": "desktop" },
     "agentProfile": "proxy",
     "unattendedSudo": true,
     "persistentT3": true,

package/examples/workers-from-infrastructure/main.tf

@@ -0,0 +1,132 @@
+# Workers from infrastructure as code: an example, with a local virtual
+# machine (libvirt) standing in for the provider. Any tool or provider that
+# can boot a stock Ubuntu image with cloud-init user data and write a file
+# works the same way; dev-fleet never calls a provider and never creates a
+# machine. It reads the worker records this writes (worker-records.json, see
+# schema/worker-record.schema.json) because fleet.json names the file:
+#
+#   "workerRecords": [{ "file": "infra/worker-records.json" }]
+#
+# The handoff, from the fleet project (OpenTofu installed, libvirt on this
+# machine):
+#
+#   1. Declare the worker. With only `tailnet_name` and `user` set, this
+#      writes its record and creates nothing else:
+#        tofu -chdir=infra apply -var-file=workers.tfvars
+#   2. Seed it. dev-fleet generates its host key, pins it in the fleet state
+#      and writes the cloud-init user data:
+#        npx devfleet fleet-node-seed --id rented-one \
+#          --output infra/seed/rented-one.yaml --apply --yes --json
+#      Put the path in `seed` (and, if you like, the printed hostKey in
+#      `host_key`, so that the record carries it too).
+#   3. Create the machine, then converge it:
+#        tofu -chdir=infra apply -var-file=workers.tfvars
+#        npx devfleet fleet-sync --non-interactive
+#   4. Remove it: drop it from `workers` and apply, or `tofu destroy`. Its
+#      record is gone, so the next `npx devfleet fleet-sync` retires it: its
+#      token, pinned host key, monitoring system and T3 link are removed.
+#
+# What the machine must offer (image, architecture, size, access, network) is
+# docs/worker-requirements.md. For a hosted provider only these change: the
+# provider block and its credentials, the image (its stock Ubuntu image of an
+# accepted release, in place of `ubuntu_image`), the instance type (at least
+# the minimum size, in place of `memory_mib` and `vcpus`) and the network (a
+# private network, or inbound rules with nothing open, in place of
+# `network_interface`). The seed goes in as the instance's user data and the
+# records come out the same way.
+#
+# This example cannot be applied in dev-fleet's CI (it needs a libvirt host
+# and a tailnet), so it is not proven there; the record format and what
+# fleet-sync does with records are (features/cli/fleet-validate.feature,
+# features/fleet-sync/node-lifecycle.feature). Pin the provider in your own
+# project: `tofu init` writes .terraform.lock.hcl.
+
+terraform {
+  required_providers {
+    libvirt = { source = "dmacvicar/libvirt" }
+    local   = { source = "hashicorp/local" }
+  }
+}
+
+provider "libvirt" {
+  uri = "qemu:///system"
+}
+
+variable "ubuntu_image" {
+  description = "A stock Ubuntu cloud image (a URL or a local path) of the release fleet.json names."
+  type        = string
+}
+
+variable "workers" {
+  description = "The workers, by fleet ID. A worker with a seed is created; one without is only declared."
+  type = map(object({
+    tailnet_name = string
+    user         = string
+    seed         = optional(string)
+    host_key     = optional(string)
+    memory_mib   = optional(number, 8192)
+    vcpus        = optional(number, 4)
+    hourly_rate  = optional(number)
+  }))
+}
+
+locals {
+  seeded = { for id, worker in var.workers : id => worker if worker.seed != null }
+}
+
+resource "libvirt_volume" "image" {
+  name   = "devfleet-ubuntu.qcow2"
+  source = var.ubuntu_image
+}
+
+resource "libvirt_volume" "disk" {
+  for_each       = local.seeded
+  name           = "${each.key}.qcow2"
+  base_volume_id = libvirt_volume.image.id
+  size           = 64 * 1024 * 1024 * 1024
+}
+
+resource "libvirt_cloudinit_disk" "seed" {
+  for_each  = local.seeded
+  name      = "${each.key}-seed.iso"
+  user_data = file(each.value.seed)
+}
+
+resource "libvirt_domain" "worker" {
+  for_each  = local.seeded
+  name      = each.key
+  memory    = each.value.memory_mib
+  vcpu      = each.value.vcpus
+  cloudinit = libvirt_cloudinit_disk.seed[each.key].id
+
+  disk {
+    volume_id = libvirt_volume.disk[each.key].id
+  }
+
+  network_interface {
+    network_name = "default"
+  }
+}
+
+# The handoff: one record per declared worker. Only what dev-fleet needs to
+# reach the machine, and nothing that names this tool or the provider.
+resource "local_file" "worker_records" {
+  filename        = "${path.module}/worker-records.json"
+  file_permission = "0644"
+  content = jsonencode({
+    version = 1
+    workers = [
+      for id, worker in var.workers : merge(
+        { id = id, tailnetName = worker.tailnet_name, user = worker.user },
+        worker.host_key == null ? {} : { hostKey = worker.host_key },
+        worker.hourly_rate == null ? {} : { billing = { hourlyRate = worker.hourly_rate } },
+        { labels = { source = "local-vm" } },
+      )
+    ]
+  })
+}
+
+output "fleet_workers" {
+  description = "The same records, for a fleet.json that runs a command instead: { \"command\": [\"tofu\", \"-chdir=infra\", \"output\", \"-json\", \"fleet_workers\"] }."
+  value       = jsondecode(local_file.worker_records.content)
+}

package/examples/workers-from-infrastructure/worker-records.json

@@ -0,0 +1,14 @@
+{
+  "$schema": "../../schema/worker-record.schema.json",
+  "version": 1,
+  "workers": [
+    {
+      "id": "rented-one",
+      "tailnetName": "rented-one.tail0a1b2c.ts.net",
+      "user": "dev",
+      "hostKey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExampleHostKeyOfARentedWorker",
+      "billing": { "hourlyRate": 0.5 },
+      "labels": { "source": "local-vm" }
+    }
+  ]
+}

package/llms.txt

@@ -1,6 +1,6 @@
 # dev-fleet
 
-> dev-fleet keeps a set of development machines in sync: one macOS **manager** and any number of Ubuntu 26.04 **workers** reached over SSH and Tailscale. It is used as the npm package `@crvouga/dev-fleet` (CLI `devfleet`) from a project of your own that holds `fleet.json` and pins the package version; one command, `devfleet fleet-sync`, converges every machine to the commit that version was published from. The fleet state lives in an S3-compatible bucket you own.
+> dev-fleet keeps a set of development machines in sync: one macOS **manager** and any number of Ubuntu (26.04 or 24.04) **workers** reached over SSH and Tailscale. It is used as the npm package `@crvouga/dev-fleet` (CLI `devfleet`) from a project of your own that holds `fleet.json` and pins the package version; one command, `devfleet fleet-sync`, converges every machine to the commit that version was published from. The fleet state lives in an S3-compatible bucket you own.
 
 Rules for an agent working in a project that uses dev-fleet:
 
@@ -16,10 +16,12 @@ Rules for an agent working in a project that uses dev-fleet:
 - [README](https://github.com/crvouga/dev-fleet/blob/main/README.md): what dev-fleet assumes, quickstart, versions, adding a machine, known limits
 - [Example fleet.json](https://github.com/crvouga/dev-fleet/blob/main/examples/fleet.json): a complete configuration with placeholders (`npx devfleet fleet-state --example` prints it)
 - [fleet.json schema](https://github.com/crvouga/dev-fleet/blob/main/schema/fleet.schema.json): JSON Schema for the fleet configuration
+- [Check-ID registry](https://github.com/crvouga/dev-fleet/blob/main/schema/check-ids.json): every check ID a report can carry, with what it means, whether it can block readiness (severity) and what repairs it (a fleet-repair recipe, or why only a person can); generated from the code
 
 ## Operate it
 
 - [Fleet state](https://github.com/crvouga/dev-fleet/blob/main/docs/state.md): the bucket, replication, replacing the manager, what the bucket does not hold yet
+- [Worker requirements](https://github.com/crvouga/dev-fleet/blob/main/docs/worker-requirements.md): what a machine must offer to be a worker (image, architecture, size, access, first-boot data, network), each with its check and failure code, and what changes for a hosted provider
 - [Topology](https://github.com/crvouga/dev-fleet/blob/main/docs/topology.md): the manager and worker roles and the canary
 - [What is declared, and where](https://github.com/crvouga/dev-fleet/blob/main/docs/declared.md): what is pinned on a machine and what is not
 - [Provider accounts](https://github.com/crvouga/dev-fleet/blob/main/docs/accounts.md): the account roster, its states and checks, adding or removing an account, propagation to every node, account selection

package/mise.lock

@@ -1355,6 +1355,35 @@ url = "https://github.com/getsops/sops/releases/download/v3.13.3/sops-v3.13.3.am
 url_api = "https://api.github.com/repos/getsops/sops/releases/assets/486808200"
 provenance = "cosign"
 
+[[tools.tmux]]
+version = "3.8"
+backend = "aqua:tmux/tmux-builds"
+specifiers = ["3.8"]
+
+[tools.tmux."platforms.linux-arm64"]
+checksum = "sha256:f399311b1159705e69839eaf4f6f68c8b8540dc3a8538b87c30dedb9c3c7fc23"
+checksum_source = "mise-versions"
+url = "https://github.com/tmux/tmux-builds/releases/download/v3.8/tmux-3.8-linux-arm64.tar.gz"
+url_api = "https://api.github.com/repos/tmux/tmux-builds/releases/assets/721688993"
+
+[tools.tmux."platforms.linux-x64"]
+checksum = "sha256:147849ff18b62a8aa3d5d8f75dd07c3f9fdfad529f932236afdec838fc1d15e1"
+checksum_source = "mise-versions"
+url = "https://github.com/tmux/tmux-builds/releases/download/v3.8/tmux-3.8-linux-x86_64.tar.gz"
+url_api = "https://api.github.com/repos/tmux/tmux-builds/releases/assets/721688991"
+
+[tools.tmux."platforms.macos-arm64"]
+checksum = "sha256:ec926bbf28ddf55d0dc7c9791d43e03e251ddab392d763f7fbb38d4b8408d051"
+checksum_source = "mise-versions"
+url = "https://github.com/tmux/tmux-builds/releases/download/v3.8/tmux-3.8-macos-arm64.tar.gz"
+url_api = "https://api.github.com/repos/tmux/tmux-builds/releases/assets/721688988"
+
+[tools.tmux."platforms.macos-x64"]
+checksum = "sha256:e7b92e51d9e9610ce80d1c477da1e6ff6a6398ea6b6b4d4ff7b19cf60f594555"
+checksum_source = "mise-versions"
+url = "https://github.com/tmux/tmux-builds/releases/download/v3.8/tmux-3.8-macos-x86_64.tar.gz"
+url_api = "https://api.github.com/repos/tmux/tmux-builds/releases/assets/721688994"
+
 [[tools.usql]]
 version = "0.21.6"
 backend = "aqua:xo/usql"

package/mise.toml

@@ -1,7 +1,7 @@
 # Shared workstation toolchain. Bootstrap installs this globally; projects may override it.
 # Commit mise.lock whenever changing these pins.
-# min_version is also the exact mise bootstrap/bootstrap.sh installs; bump the
-# mise_*_sha256 digests under [vars] with it.
+# min_version is also the exact mise bootstrap/bootstrap.sh installs; the
+# mise_*_sha256 digests under [vars] move with it (bun bootstrap/pins.ts mise).
 min_version = "2026.10.24"
 
 [settings]
@@ -30,6 +30,8 @@ go = "1.27.3"
 jq = "1.8.2"
 yq = "4.54.1"
 ripgrep = "15.2.0"
+# Terminal multiplexer for long agent and build sessions on a worker.
+tmux = { version = "3.8", os = ["linux", "macos"] }
 fd = "10.5.0"
 fzf = "0.74.5"
 bat = "0.26.1"
@@ -109,7 +111,7 @@ docker_desktop_arm64_sha256 = "9c6f13ce4edda2abf20d801552b6acb96acd3d64b3e91b50c
 docker_desktop_amd64_sha256 = "96b53f9a4a807b6be6207d18f360b9778ff481e1d94b5027bdf1ae2a22d837cb"
 
 # Minimal remote development baseline; versions are resolved exclusively by tools + mise.lock.
-worker_tools = "bun,node,gh,python,uv,docker-cli,vault,jq,codex,claude,aqua:anomalyco/opencode,gemini,copilot,http:grok,http:antigravity"
+worker_tools = "bun,node,gh,python,uv,docker-cli,vault,jq,tmux,codex,claude,aqua:anomalyco/opencode,gemini,copilot,http:grok,http:antigravity"
 # Tool profiles a fleet adds to its workers (`worker.toolProfiles` in
 # fleet.json, fleet-wide or per worker): one `worker_profile_<name>` each,
 # naming tools pinned above. bootstrap/worker-tools.ts lists the names.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@crvouga/dev-fleet",
-  "version": "0.7.0",
+  "version": "0.9.0",
   "description": "Keeps a set of development machines in sync: one manager and many workers",
   "license": "MIT",
   "repository": "github:crvouga/dev-fleet",
@@ -72,5 +72,5 @@
   "engines": {
     "node": ">=22"
   },
-  "gitHead": "77efce7ad99a3d521169a5e773ad6b97ca21bc75"
+  "gitHead": "92f24582645e8a05f47124be1076a28f637ec347"
 }