@crvouga/dev-fleet 0.2.0 → 0.4.0

package/docs/runbooks.md

@@ -0,0 +1,380 @@
+# Runbooks
+
+One entry for every failure code dev-fleet reports: in a check's summary, a
+bootstrap phase or an Ansible assertion. Each family says what its codes
+mean, how to confirm one, the fix (a command or a `fleet-sync` that applies
+the playbook that owns it) and when to escalate. A code whose fix needs a
+person (a login, an approval, a choice) is marked **needs a person**.
+`features/rules/runbooks.feature` fails when a code in the source has no
+entry here, or a `devfleet` command quoted in the documents does not exist.
+
+There are 241 codes; 31 need a person.
+
+## Bootstrap of a new or returning worker
+
+- **Meaning:** The remote bootstrap (`bootstrap/fleet-remote-bootstrap.sh`) stopped before node convergence. It reports the phase and the code, never its output.
+- **Confirm:** Run `devfleet fleet-sync --dry-run` and read the `bootstrap:*` check of the worker; `ssh WORKER` and look under `~/.local/share/personal/fleet`.
+- **Fix:** Follow the remedy the check prints, then run `devfleet fleet-sync`. The bootstrap resumes where it stopped.
+- **Escalate:** A platform or architecture code means the machine must be reinstalled: Ubuntu 26.04 on x86_64 or aarch64.
+
+- `BOOTSTRAP_CAPACITY_UNVERIFIED`: bootstrap capacity unverified.
+- `BOOTSTRAP_DISK_CAPACITY_REQUIRED`: bootstrap disk capacity required.
+- `BOOTSTRAP_LOCK_TOOL_REQUIRED`: bootstrap lock tool required.
+- `CONFIGURATION_INSTALL_FAILED`: configuration install failed.
+- `FLEET_APPROVED_BOOTSTRAP_UNAVAILABLE`: fleet approved bootstrap unavailable.
+- `FLEET_STATE_PATH_UNSAFE`: fleet state path unsafe.
+- `MACOS_ARCHITECTURE_UNSUPPORTED`: macos architecture unsupported. **Needs a person.**
+- `MACOS_BASELINE_MISMATCH`: macos baseline mismatch. **Needs a person.**
+- `PLATFORM_UNSUPPORTED`: platform unsupported. **Needs a person.**
+- `REMOTE_BOOTSTRAP_EVIDENCE_INVALID`: remote bootstrap evidence invalid.
+- `REMOTE_BOOTSTRAP_EXIT_FAILED`: remote bootstrap exit failed.
+- `ROOT_DEVELOPMENT_USER_REJECTED`: root development user rejected.
+- `UBUNTU_ARCHITECTURE_UNSUPPORTED`: ubuntu architecture unsupported. **Needs a person.**
+- `XCODE_COMMAND_LINE_TOOLS_REQUIRED`: xcode command line tools required. **Needs a person.**
+
+## Ubuntu packages
+
+- **Meaning:** A package transaction could not run or the package database is inconsistent.
+- **Confirm:** `ssh WORKER sudo dpkg --audit` and `sudo apt-get -o Debug::NoLocking=1 check`.
+- **Fix:** Let any running package process finish (it is never killed), free disk under `/var`, then run `devfleet fleet-sync`.
+- **Escalate:** APT_STATE_BROKEN and DPKG_INTERRUPTED need `sudo dpkg --configure -a` by a person once no package process runs.
+
+- `APT_DISK_FULL`: apt disk full.
+- `APT_DNS_UNAVAILABLE`: apt dns unavailable.
+- `APT_LOCK_CONTENDED`: apt lock contended.
+- `APT_REPOSITORY_UNAVAILABLE`: apt repository unavailable.
+- `APT_STATE_BROKEN`: apt state broken. **Needs a person.**
+- `DPKG_INTERRUPTED`: dpkg interrupted. **Needs a person.**
+
+## The fleet-owned checkout
+
+- **Meaning:** The checkout of the revision on the machine holds changes fleet-sync did not make, or its bundle changed in transit.
+- **Confirm:** `ssh WORKER git -C ~/.local/share/personal/fleet/revisions/REVISION status`.
+- **Fix:** Remove that revisions directory and run `devfleet fleet-sync`; it is a cache.
+- **Escalate:** A digest mismatch that repeats points at the SSH route.
+
+- `SOURCE_CACHE_CHANGED`: source cache changed.
+- `SOURCE_CHECKOUT_CONFLICT`: source checkout conflict.
+- `SOURCE_DIGEST_MISMATCH`: source digest mismatch.
+- `SOURCE_STATUS_FAILED`: source status failed.
+
+## The pinned toolchain
+
+- **Meaning:** mise, the pinned tools, the global mise configuration or the frozen dependencies did not converge.
+- **Confirm:** Run `bun bootstrap/doctor.ts --json --worker` in the checkout on the machine, or read the `tool:*` checks of `devfleet fleet-status`.
+- **Fix:** Run `devfleet fleet-sync`; it resumes. TOOLCHAIN_CONFIG_CONFLICT means another tool owns `~/.config/mise/config.toml`: move it away.
+- **Escalate:** A tool that fails to download from its pinned artifact needs a new pin in `mise.toml` and `mise.lock` (an issue).
+
+- `LAUNCHER_INSTALLATION_FAILED`: launcher installation failed.
+- `TOOLCHAIN_AWS_FAILED`: toolchain aws failed.
+- `TOOLCHAIN_CONFIG_CONFLICT`: toolchain config conflict.
+- `TOOLCHAIN_CONVERGENCE_FAILED`: toolchain convergence failed.
+- `TOOLCHAIN_FROZEN_DEPENDENCIES_FAILED`: toolchain frozen dependencies failed.
+- `TOOLCHAIN_MISE_FAILED`: toolchain mise failed.
+- `TOOLCHAIN_PATH_CONFIGURATION_FAILED`: toolchain path configuration failed.
+- `TOOLCHAIN_T3_FAILED`: toolchain t3 failed.
+
+## The Ubuntu baseline and its services
+
+- **Meaning:** The Ansible baseline (`ansible/ubuntu/baseline.yml`) did not apply, or what it declares is not what the machine runs.
+- **Confirm:** Read the `worker:*` and `ubuntu:*` checks of `devfleet fleet-status --workers`; on the worker, `systemctl status SERVICE`.
+- **Fix:** Run `devfleet fleet-sync`, which applies the baseline in full when its effect no longer verifies.
+- **Escalate:** Ownership conflicts (sudoers, SSH, Docker from another distribution) need a person to migrate the other owner first.
+
+- `DISK_FREE_BELOW_FLOOR`: disk free below floor.
+- `DISK_FREE_UNREADABLE`: disk free unreadable.
+- `DOCKER_OWNER_CONFLICT`: docker owner conflict. **Needs a person.**
+- `MEMORY_RESERVE_MISSING`: memory reserve missing.
+- `RESOURCE_CAPACITY_REQUIRED`: resource capacity required.
+- `RESOURCE_GUARD_INACTIVE`: resource guard inactive.
+- `SERVICE_NOT_ENABLED`: service not enabled.
+- `SERVICE_NOT_INSTALLED`: service not installed.
+- `SERVICE_NOT_RUNNING`: service not running.
+- `SERVICE_RESTART_POLICY`: service restart policy.
+- `SSH_UNMANAGED_CONFIG_CONFLICT`: ssh unmanaged config conflict.
+- `SUDOERS_OWNERSHIP_CONFLICT`: sudoers ownership conflict. **Needs a person.**
+- `SUDO_AUTHORIZATION_REQUIRED`: sudo authorization required. **Needs a person.**
+- `SWAP_CAPACITY_REQUIRED`: swap capacity required.
+- `SWAP_MISSING`: swap missing.
+- `TMP_QUOTA_EXHAUSTED`: tmp quota exhausted.
+- `UBUNTU_BASELINE_INCOMPLETE`: ubuntu baseline incomplete.
+- `UBUNTU_BASELINE_MISMATCH`: ubuntu baseline mismatch. **Needs a person.**
+- `UBUNTU_LINGER_REQUIRED`: ubuntu linger required.
+- `UBUNTU_MAINTENANCE_REBOOT_PENDING`: ubuntu maintenance reboot pending.
+- `UBUNTU_SESSION_POLICY_PENDING`: ubuntu session policy pending.
+- `UBUNTU_SLEEP_POLICY_DRIFT`: ubuntu sleep policy drift.
+- `WORKER_RECOVERY_SCRIPT_MISSING`: worker recovery script missing.
+- `WORKER_RECOVERY_SETUP_FAILED`: worker recovery setup failed.
+- `WORKER_RECOVERY_TIMER_INACTIVE`: worker recovery timer inactive.
+
+## Containers on a worker
+
+- **Meaning:** `worker:docker-run` could not start the pinned container or build a one-line image as the worker user.
+- **Confirm:** On the worker, as that user: `docker run --rm --network none --pull never IMAGE true`, with IMAGE from `dockerSmokeTest` in `ansible/ubuntu/policy.json`.
+- **Fix:** Run `devfleet fleet-sync`: it starts the engine, adds the user to the docker group and caches the image. A session started earlier keeps its old groups: restart it.
+- **Escalate:** A cgroup or runtime failure that survives a reboot is an issue.
+
+- `DOCKER_BUILD_FAILED`: docker build failed.
+- `DOCKER_DAEMON_UNREACHABLE`: docker daemon unreachable.
+- `DOCKER_DISK_FULL`: docker disk full.
+- `DOCKER_RUN_FAILED`: docker run failed.
+- `DOCKER_RUN_TIMEOUT`: docker run timeout.
+- `DOCKER_SMOKE_IMAGE_MISSING`: docker smoke image missing.
+- `DOCKER_SOCKET_ACCESS_DENIED`: docker socket access denied.
+
+## Headless browser on a worker
+
+- **Meaning:** `worker:browser` could not launch the pinned headless Chromium of the web tool profile.
+- **Confirm:** On the worker run the headless shell under `/opt/personal-fleet/browser/` with `--headless --no-sandbox --dump-dom about:blank`.
+- **Fix:** Run `devfleet fleet-sync`, which installs the pinned libraries and build (`ansible/ubuntu/tasks/browser.yml`).
+- **Escalate:** A missing library the policy does not list is an issue: the list needs it.
+
+- `BROWSER_LAUNCH_FAILED`: browser launch failed.
+- `BROWSER_LAUNCH_TIMEOUT`: browser launch timeout.
+- `BROWSER_LIBRARY_MISSING`: browser library missing.
+- `BROWSER_NOT_INSTALLED`: browser not installed.
+
+## Git identity and pushing
+
+- **Meaning:** Commits on the node would carry the wrong author, or git cannot authenticate to GitHub on its own.
+- **Confirm:** On the node: `git config --global --get user.name`, and `printf "protocol=https\nhost=github.com\n\n" | GIT_TERMINAL_PROMPT=0 git credential fill >/dev/null`.
+- **Fix:** Declare the `git` block in `fleet.json` and run `devfleet fleet-sync` (`ansible/github/git.yml`). Remove lines in `~/.gitconfig` that override it.
+- **Escalate:** GIT_REMOTE_REFUSED means the fleet account lost access to the declared repository: a person grants it.
+
+- `GIT_CONFIGURATION_INCOMPLETE`: git configuration incomplete.
+- `GIT_CREDENTIAL_HELPER_MISSING`: git credential helper missing.
+- `GIT_IDENTITY_MISMATCH`: git identity mismatch.
+- `GIT_IDENTITY_MISSING`: git identity missing.
+- `GIT_IDENTITY_OVERRIDDEN`: git identity overridden.
+- `GIT_IDENTITY_UNDECLARED`: git identity undeclared.
+- `GIT_REMOTE_REFUSED`: git remote refused. **Needs a person.**
+- `GIT_REMOTE_UNREACHABLE`: git remote unreachable.
+
+## Tailscale
+
+- **Meaning:** A machine is not on the tailnet as the fleet declares it.
+- **Confirm:** `tailscale status` on the machine and `devfleet fleet-discover`.
+- **Fix:** Sign the machine in (`sudo tailscale up` on a worker, the app on the manager), then run `devfleet fleet-sync`.
+- **Escalate:** Logins and device approvals need a person.
+
+- `TAILSCALE_DEVICE_REPLACED`: tailscale device replaced. **Needs a person.**
+- `TAILSCALE_ENROLLMENT_REQUIRED`: tailscale enrollment required. **Needs a person.**
+- `TAILSCALE_IDENTITY_CONFLICT`: tailscale identity conflict. **Needs a person.**
+- `TAILSCALE_INTERACTIVE_AUTHORIZATION_REQUIRED`: tailscale interactive authorization required. **Needs a person.**
+- `TAILSCALE_KEY_EXPIRY_UNOBSERVED`: tailscale key expiry unobserved.
+- `TAILSCALE_LOGIN_REQUIRED`: tailscale login required. **Needs a person.**
+- `TAILSCALE_NODE_MISSING`: tailscale node missing.
+- `TAILSCALE_NODE_OFFLINE`: tailscale node offline.
+- `TAILSCALE_SETUP_OR_LOGIN_REQUIRED`: tailscale setup or login required. **Needs a person.**
+- `TAILSCALE_SOURCE_OWNER_CONFLICT`: tailscale source owner conflict. **Needs a person.**
+- `TAILSCALE_UPGRADE_PENDING`: tailscale upgrade pending. **Needs a person.**
+
+## T3 Code
+
+- **Meaning:** The native T3 server or desktop app, its policy, its private route or its providers did not converge or verify.
+- **Confirm:** `devfleet t3-status` and the `t3:*` checks of `devfleet fleet-status`.
+- **Fix:** Run `devfleet fleet-sync`, then `devfleet t3-apply` for providers. A MIGRATION or MAINTENANCE code names a one-time step in its remedy.
+- **Escalate:** Pairing and relinking a peer needs a person in T3 Settings.
+
+- `MANAGER_T3_LINK_UNVERIFIED`: manager t3 link unverified.
+- `T3_BASELINE_PREREQUISITES_REQUIRED`: t3 baseline prerequisites required.
+- `T3_CACHE_DRIFT`: t3 cache drift.
+- `T3_CHECKSUM_MISMATCH`: t3 checksum mismatch.
+- `T3_DESKTOP_INSTALL_FAILED`: t3 desktop install failed.
+- `T3_DESKTOP_STOPPED`: t3 desktop stopped.
+- `T3_DOWNLOAD_UNAVAILABLE`: t3 download unavailable.
+- `T3_ENVIRONMENT_IDENTITY_CHANGED`: t3 environment identity changed.
+- `T3_EXECUTABLE_FAILED`: t3 executable failed.
+- `T3_EXECUTABLE_OWNER_CONFLICT`: t3 executable owner conflict. **Needs a person.**
+- `T3_EXTRACTION_FAILED`: t3 extraction failed.
+- `T3_INSTALLATION_DRIFT`: t3 installation drift.
+- `T3_MANAGER_ENDPOINT_REJECTED`: t3 manager endpoint rejected.
+- `T3_MANAGER_ENDPOINT_UNREACHABLE`: t3 manager endpoint unreachable.
+- `T3_MANAGER_IDENTITY_UNVERIFIED`: t3 manager identity unverified.
+- `T3_MANAGER_ORIGIN_UNRESTRICTED`: t3 manager origin unrestricted.
+- `T3_MANAGER_ORIGIN_UNVERIFIED`: t3 manager origin unverified.
+- `T3_MODELS_MISMATCH`: t3 models mismatch.
+- `T3_MODELS_UNVERIFIED`: t3 models unverified.
+- `T3_POLICY_OWNER_CONFLICT`: t3 policy owner conflict. **Needs a person.**
+- `T3_POLICY_RESTART_FAILED`: t3 policy restart failed.
+- `T3_POLICY_RESTART_PENDING`: t3 policy restart pending.
+- `T3_POLICY_STAGE_FAILED`: t3 policy stage failed.
+- `T3_PRIVATE_IDENTITY_REQUIRED`: t3 private identity required.
+- `T3_PRIVATE_ROUTE_CONFLICT`: t3 private route conflict.
+- `T3_PRIVATE_ROUTE_REQUIRED`: t3 private route required.
+- `T3_PRIVATE_ROUTE_UNVERIFIED`: t3 private route unverified.
+- `T3_PROFILE_CONFLICT`: t3 profile conflict.
+- `T3_PROFILE_PENDING`: t3 profile pending.
+- `T3_PROFILE_RECORD_INVALID`: t3 profile record invalid.
+- `T3_PROFILE_UNVERIFIED`: t3 profile unverified.
+- `T3_PROVIDERS_PENDING`: t3 providers pending.
+- `T3_PROVIDERS_UNVERIFIED`: t3 providers unverified.
+- `T3_PROVIDER_CONFLICT`: t3 provider conflict.
+- `T3_PROVIDER_NAMES_PENDING`: t3 provider names pending.
+- `T3_PROVIDER_NAMES_UNVERIFIED`: t3 provider names unverified.
+- `T3_RESTART_PENDING`: t3 restart pending.
+- `T3_SETTINGS_BUSY`: t3 settings busy.
+
+## Monitoring (Beszel)
+
+- **Meaning:** The hub on the manager, an agent on a machine, an enrollment or the hub backups do not match what the fleet declares.
+- **Confirm:** The `beszel:*` checks of `devfleet fleet-status --workers`; on the manager `docker compose ls` and the hub at `/_/`.
+- **Fix:** Run `devfleet fleet-sync`, which converges the hub, its accounts, its backups and every agent. Hub data is never deleted.
+- **Escalate:** BESZEL_HUB_UPGRADE_PENDING needs a verified backup and a planned upgrade by a person.
+
+- `BESZEL_ACCOUNT_INVALID`: beszel account invalid.
+- `BESZEL_ACCOUNT_REQUIRED`: beszel account required. **Needs a person.**
+- `BESZEL_AGENT_CONNECTION_PENDING`: beszel agent connection pending.
+- `BESZEL_AGENT_DISCONNECTED`: beszel agent disconnected.
+- `BESZEL_AGENT_KEY_REJECTED`: beszel agent key rejected.
+- `BESZEL_AGENT_LOG_UNAVAILABLE`: beszel agent log unavailable.
+- `BESZEL_AGENT_TOKEN_REJECTED`: beszel agent token rejected.
+- `BESZEL_API_UNAVAILABLE`: beszel api unavailable.
+- `BESZEL_API_UNVERIFIED`: beszel api unverified.
+- `BESZEL_AUTH_REQUIRED`: beszel auth required.
+- `BESZEL_BACKUP_CONVERGENCE_FAILED`: beszel backup convergence failed.
+- `BESZEL_BACKUP_NOT_CONVERGED`: beszel backup not converged.
+- `BESZEL_BACKUP_OFFHOST_REQUIRED`: beszel backup offhost required.
+- `BESZEL_BACKUP_STALE`: beszel backup stale.
+- `BESZEL_BACKUP_UNSCHEDULED`: beszel backup unscheduled.
+- `BESZEL_BACKUP_UNVERIFIED`: beszel backup unverified.
+- `BESZEL_CREDENTIALS_REQUIRED`: beszel credentials required. **Needs a person.**
+- `BESZEL_DISK_FULL`: beszel disk full.
+- `BESZEL_DISK_UNVERIFIED`: beszel disk unverified.
+- `BESZEL_DOCKER_DESKTOP_REQUIRED`: beszel docker desktop required.
+- `BESZEL_ENROLLMENT_INCOMPLETE`: beszel enrollment incomplete.
+- `BESZEL_HOST_CONFLICT`: beszel host conflict.
+- `BESZEL_HUB_INSTALLATION_INCOMPLETE`: beszel hub installation incomplete.
+- `BESZEL_HUB_MISMATCH`: beszel hub mismatch.
+- `BESZEL_HUB_SELECTION_REQUIRED`: beszel hub selection required.
+- `BESZEL_HUB_UNHEALTHY`: beszel hub unhealthy.
+- `BESZEL_HUB_UNREACHABLE`: beszel hub unreachable.
+- `BESZEL_HUB_UPGRADE_PENDING`: beszel hub upgrade pending. **Needs a person.**
+- `BESZEL_INPUT_INVALID`: beszel input invalid.
+- `BESZEL_INPUT_LIMIT`: beszel input limit.
+- `BESZEL_INSTALLATION_INCOMPLETE`: beszel installation incomplete.
+- `BESZEL_LEGACY_DAEMON`: beszel legacy daemon.
+- `BESZEL_METRICS_STALE`: beszel metrics stale.
+- `BESZEL_NODE_INVALID`: beszel node invalid.
+- `BESZEL_NODE_MISSING`: beszel node missing.
+- `BESZEL_NODE_TAILNET_REQUIRED`: beszel node tailnet required.
+- `BESZEL_NOT_ENROLLED`: beszel not enrolled.
+- `BESZEL_NOT_HUB`: beszel not hub.
+- `BESZEL_NOT_MANAGER`: beszel not manager.
+- `BESZEL_OPERATION_INCOMPLETE`: beszel operation incomplete.
+- `BESZEL_PRIVATE_ROUTE_CONFLICT`: beszel private route conflict.
+- `BESZEL_PRIVATE_ROUTE_REQUIRED`: beszel private route required.
+- `BESZEL_PRIVATE_ROUTE_UNVERIFIED`: beszel private route unverified.
+- `BESZEL_PROTOCOL`: beszel protocol.
+- `BESZEL_PUBLIC_ENDPOINT_OFFLINE`: beszel public endpoint offline.
+- `BESZEL_REMOTE_EVIDENCE_INVALID`: beszel remote evidence invalid.
+- `BESZEL_RESPONSE_LIMIT`: beszel response limit.
+- `BESZEL_SERVICE_STOPPED`: beszel service stopped.
+- `BESZEL_SOURCE_MISMATCH`: beszel source mismatch.
+- `BESZEL_TAILSCALE_REQUIRED`: beszel tailscale required.
+- `BESZEL_TOKEN_INVALID`: beszel token invalid.
+- `BESZEL_TUNNEL_CREDENTIAL_UNAVAILABLE`: beszel tunnel credential unavailable. **Needs a person.**
+- `BESZEL_VERSION_MISMATCH`: beszel version mismatch.
+- `BESZEL_VIEWER_ROLE`: beszel viewer role.
+- `BESZEL_VIEWER_UNPROVISIONED`: beszel viewer unprovisioned.
+- `BESZEL_VIEWER_UNVERIFIED`: beszel viewer unverified.
+
+## Coding agents and providers
+
+- **Meaning:** An agent CLI, its profile or a provider behind the CLI proxy did not answer as expected.
+- **Confirm:** The `agent:*` and `provider:*` checks of `devfleet fleet-status`, and `devfleet cliproxy` on the manager.
+- **Fix:** Run `devfleet fleet-sync`; a rate limit or upstream outage clears on its own.
+- **Escalate:** PROVIDER_AUTH_REJECTED needs a person to sign the account in again on the proxy.
+
+- `AGENT_MEMORY_LIMIT_MISSING`: agent memory limit missing.
+- `AGENT_MODEL_UNSERVED`: agent model unserved.
+- `AGENT_PROFILE_STALE`: agent profile stale.
+- `AGENT_SERVICE_ENVIRONMENT_INCOMPLETE`: agent service environment incomplete.
+- `AGENT_SERVICE_ENVIRONMENT_UNVERIFIED`: agent service environment unverified.
+- `AGENT_SESSION_UNVERIFIED`: agent session unverified.
+- `AGENT_VAULT_UNAVAILABLE`: agent vault unavailable.
+- `AGENT_VERSION_MISMATCH`: agent version mismatch.
+- `PROVIDER_AUTH_REJECTED`: provider auth rejected. **Needs a person.**
+- `PROVIDER_EMPTY_RESPONSE`: provider empty response.
+- `PROVIDER_MODEL_UNAVAILABLE`: provider model unavailable.
+- `PROVIDER_NETWORK_UNVERIFIED`: provider network unverified.
+- `PROVIDER_OUTPUT_UNVERIFIED`: provider output unverified.
+- `PROVIDER_PROTOCOL_REJECTED`: provider protocol rejected.
+- `PROVIDER_RATE_LIMITED`: provider rate limited.
+- `PROVIDER_RESPONSE_INVALID`: provider response invalid.
+- `PROVIDER_RESPONSE_LIMIT`: provider response limit.
+- `PROVIDER_ROUTE_UNVERIFIED`: provider route unverified.
+- `PROVIDER_STREAM_INCOMPLETE`: provider stream incomplete.
+- `PROVIDER_TIMEOUT`: provider timeout.
+- `PROVIDER_TOOL_ARGUMENTS_INVALID`: provider tool arguments invalid.
+- `PROVIDER_TOOL_INVALID`: provider tool invalid.
+- `PROVIDER_TOOL_MISSING`: provider tool missing.
+- `PROVIDER_TOOL_RESULT_UNVERIFIED`: provider tool result unverified.
+- `PROVIDER_UNATTENDED_CREDENTIAL_REQUIRED`: provider unattended credential required.
+- `PROVIDER_UPSTREAM_UNAVAILABLE`: provider upstream unavailable.
+- `PROXY_RECONCILE_UNVERIFIED`: proxy reconcile unverified.
+- `UNATTENDED_PROXY_AUTHENTICATION_REQUIRED`: unattended proxy authentication required.
+
+## Provider accounts
+
+- **Meaning:** A change to a provider account on the CLI proxy (sign-in, removal, cooldown) did not complete.
+- **Confirm:** `devfleet fleet-accounts` lists every account and its state; `devfleet fleet-accounts --diff` compares this machine.
+- **Fix:** Retry the change with `devfleet fleet-accounts`; see [Provider accounts](accounts.md).
+- **Escalate:** Signing an account in again needs a person.
+
+- `ACCOUNT_CHANGE_FAILED`: account change failed.
+
+## Vault and credentials
+
+- **Meaning:** A machine cannot read the secrets it needs with its own Vault session.
+- **Confirm:** `vault token lookup` on the machine and the `worker:vault-renewal` check.
+- **Fix:** Log the machine in once (`devfleet fleet-sync` asks on the manager); renewal is automatic afterwards.
+- **Escalate:** A refused policy needs the Vault administrator.
+
+- `UNATTENDED_VAULT_IDENTITY_REQUIRED`: unattended vault identity required.
+- `VAULT_LOGIN_REQUIRED`: vault login required. **Needs a person.**
+- `VAULT_POLICY_PERMISSION_REQUIRED`: vault policy permission required. **Needs a person.**
+- `VAULT_RENEWAL_FAILING`: vault renewal failing.
+
+## Fleet operations and the fleet state
+
+- **Meaning:** fleet-sync could not plan, lock, reach a machine, promote a revision or keep the fleet state.
+- **Confirm:** `devfleet fleet-state` for the state, `devfleet fleet-status` for machines, and the report `devfleet fleet-sync` writes.
+- **Fix:** Follow the remedy; most clear by running `devfleet fleet-sync` again once the holder of a lock finishes.
+- **Escalate:** FLEET_STATE_DIVERGED needs a person to choose which copy of the state to keep (docs/state.md).
+
+- `ANSIBLE_VARIABLE_LIMIT`: ansible variable limit.
+- `CONTAINER_WATCHDOG_UNOBSERVED`: container watchdog unobserved.
+- `FLEET_CONNECTION_REQUIRED`: fleet connection required.
+- `FLEET_MANAGER_ENROLLMENT_REQUIRED`: fleet manager enrollment required. **Needs a person.**
+- `FLEET_MANAGER_LOCKED`: fleet manager locked.
+- `FLEET_MANAGER_UNREACHABLE`: fleet manager unreachable.
+- `FLEET_NOT_MANAGER`: fleet not manager.
+- `FLEET_PLAN_STALE`: fleet plan stale.
+- `FLEET_PROMOTION_BLOCKED`: fleet promotion blocked.
+- `FLEET_REMOTE_LOCKED`: fleet remote locked.
+- `FLEET_REMOTE_OPERATION_FAILED`: fleet remote operation failed.
+- `FLEET_REVISION_REQUIRED`: fleet revision required.
+- `FLEET_SCHEDULE_PREVIEW`: fleet schedule preview.
+- `FLEET_SOURCE_UNAVAILABLE`: fleet source unavailable.
+- `FLEET_STATE_DIVERGED`: fleet state diverged. **Needs a person.**
+- `FLEET_VERIFICATION_MISSING`: fleet verification missing.
+- `MANAGER_AGENT_PROFILES_FAILED`: manager agent profiles failed.
+- `MANAGER_DISK_UNOBSERVED`: manager disk unobserved.
+- `MANAGER_DOCKER_DESKTOP_DRIFT`: manager docker desktop drift.
+- `MANAGER_DOCKER_DESKTOP_UNOBSERVED`: manager docker desktop unobserved.
+- `MANAGER_RELEASE_UNOBSERVED`: manager release unobserved.
+- `MANAGER_SLEEPS_ON_POWER`: manager sleeps on power.
+- `MANAGER_UPDATES_UNOBSERVED`: manager updates unobserved.
+- `REMOTE_CONFIGURATION_INCOMPLETE`: remote configuration incomplete.
+- `REMOTE_PHASE_FAILED`: remote phase failed.
+
+## Notifications (fleet-watch)
+
+- **Meaning:** fleet-watch could not reach a notification target `notify` in `fleet.json` names, or its address is not stored as a secret.
+- **Confirm:** Run `devfleet fleet-watch --test` and read the `notify:*` checks.
+- **Fix:** Store the target's address as the secret the remedy names, or fix the endpoint, then run `devfleet fleet-watch --test` again.
+- **Escalate:** A target that keeps refusing needs whoever runs that endpoint.
+
+- `NOTIFY_DELIVERY_FAILED`: notify delivery failed.
+- `NOTIFY_TARGET_UNAVAILABLE`: notify target unavailable.

package/docs/state.md

@@ -4,13 +4,18 @@ A fleet is described in two places you own. Your project in Git holds
 `fleet.json` and pins the dev-fleet version. Everything else the manager
 cannot rebuild from Git is the **fleet state**:
 
-| Document     | What it is                                                           |
-| ------------ | -------------------------------------------------------------------- |
-| `roster`     | The enrolled connection to each worker (SSH alias, Tailscale device) |
-| `enrollment` | The revision `fleet-sync` approved and the one it last applied       |
-| `definition` | The fleet configuration `fleet-sync` last applied: your `fleet.json` |
-
-They live in one SQLite database, and [Litestream](https://litestream.io)
+| Document     | What it is                                                            |
+| ------------ | --------------------------------------------------------------------- |
+| `roster`     | The enrolled connection to each worker (SSH alias, Tailscale device)  |
+| `enrollment` | The revision `fleet-sync` approved and the one it last applied        |
+| `definition` | The fleet configuration `fleet-sync` last applied: your `fleet.json`  |
+| `history`    | The last 90 days of `fleet-sync` runs (at most 500), and the schedule |
+
+`devfleet fleet-watch` also keeps `watch.json` in the bucket, beside the
+database: the health of each worker when it last looked, and when that was
+(the manager's heartbeat). Losing it costs at most one repeated notification.
+
+The three documents live in one SQLite database, and [Litestream](https://litestream.io)
 replicates that database to a storage bucket you own. The bucket is the source
 of truth. The copy on the manager is disposable: delete it, or replace the
 whole machine, and the next command brings the fleet back.
@@ -118,6 +123,7 @@ way, needs no credentials and is the quickest way to try this out.
 ```sh
 devfleet fleet-state            # read-only: where the state is, and how it stands with the bucket
 devfleet fleet-state --example  # print a fleet.json to start a project from
+devfleet fleet-report --since 30d  # read-only: how reliably fleet-sync ran, from the history
 devfleet fleet-state --export   # print the configuration fleet-sync last applied
 devfleet fleet-state --push     # upload changes made while the bucket was unreachable
 devfleet fleet-state --restore  # replace the local copy with the bucket's

package/docs/topology.md

@@ -53,6 +53,8 @@ your project's `fleet.json` (version 3, checked by [the schema](../schema/fleet.
   the manager.
 - Anything on a worker is disposable: T3 sessions and worktrees, caches,
   checkouts. Durable work lives in Git remotes; credentials come from Vault.
+  Work in progress in agent worktrees is pushed to a recovery ref every 15
+  minutes; [durability](durability.md) says what a lost machine takes with it.
 - The first declared worker is the rollout canary for a revision new to the
   fleet: a failed canary stops promotion to the other workers. Repeating a
   revision that already converged everywhere runs all workers at once.

package/docs/ubuntu-sync.md

@@ -117,9 +117,8 @@ included, to the exact version in `ansible/ubuntu/policy.json`. The interactive
   roster in the [fleet state](state.md), using the existing roster schema.
   Preserve workspace/runtime paths, capabilities and explicit `tailscaleId`
   bindings.
-- Workstation network consumers derive their inventory from Fleet. The old file
-  is retained only as `packages/fleet/migrations/workstation-network-v1.json` for
-  lossless identity-path migration; conflicts fail rather than silently winning.
+- Workstation network consumers derive their inventory from the fleet
+  configuration; conflicts fail rather than silently winning.
 - Manager approval: the `enrollment` document in the [fleet state](state.md).
   Manager deployment lock: `~/.local/share/personal/fleet/manager/sync.lock`.
 - Remote source: `~/.local/share/personal/fleet/revisions/FULL_SHA`.
@@ -179,21 +178,21 @@ independently. Bespoke TypeScript remains only where a native CLI or API has no
 declarative owner: T3 service lifecycle, Beszel enrollment tokens, provider
 verification and the CLIProxy singleton.
 
-| Policy                      | Workers (Ubuntu 26.04, stateless)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | Manager (macOS, singleton)                                                                                                                                                                                                                                                                                                                                                                                |
-| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Never sleep                 | systemd sleep targets masked; logind lid/idle policy; GNOME power plugin set to `nothing`                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | never sleeps on AC power (`pmset -c sleep 0`, `macos/manager.yml`, verified as `manager:power`); battery sleep is left to the human                                                                                                                                                                                                                                                                       |
-| No automatic logout or lock | GNOME `lock-enabled false`, `idle-delay 0`; designated-user lingering                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | not managed (the human uses it)                                                                                                                                                                                                                                                                                                                                                                           |
-| Always-on services          | drop-ins `Restart=always`, `StartLimitIntervalSec=0` for ssh, tailscaled, docker, beszel-agent                                                                                                                                                                                                                                                                                                                                                                                                                                                                | singleton services under launchd and Docker Desktop (see the last row)                                                                                                                                                                                                                                                                                                                                    |
-| T3 health and connection    | `fleet-sync` and `fleet-status --workers` require `worker:t3-health` (local API answers), `t3:manager-endpoint` (manager HTTPS over Tailscale) and `t3:paired-access` (manager T3 link reachable)                                                                                                                                                                                                                                                                                                                                                             | the human's T3 desktop holds the links to every worker (`t3:paired-access`)                                                                                                                                                                                                                                                                                                                               |
-| Memory safety               | `systemd-oomd` (swap 90%, pressure 60%); `user.slice` `MemoryMax=90%`; each `t3code-*.scope` agent `MemoryMax=30%`, `OOMPolicy=continue`; swap file when none                                                                                                                                                                                                                                                                                                                                                                                                 | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
-| Disk safety                 | `personal-fleet-disk-guard` kills only the top user-cgroup writer below 3 GiB free, never deletes files; hourly `personal-fleet-disk-gc.timer` deletes git-ignored build output (`node_modules`, `.cache`, `dist`, ...) in checkouts idle 7d (12h under 15 GiB free) with no process inside, plus unused Docker images and build cache, never source, worktrees or volumes; journal 1G; `/tmp` 25% of RAM, aged 2d hourly, a measured write probe (`TMP_QUOTA_EXHAUSTED`), and T3 agents use disk-backed `/var/tmp` (`TMPDIR`, aged 14d); Docker log rotation | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
-| Tailscale                   | pinned APT package, `tailscaled` enabled, native login required                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | Tailscale app, human login                                                                                                                                                                                                                                                                                                                                                                                |
-| Dashboard agent             | pinned native binary, hardened `beszel-agent` systemd unit as user `beszel`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | not installed                                                                                                                                                                                                                                                                                                                                                                                             |
-| Credential renewal          | user timer `personal-worker-auth` renews the node's own Vault token and GitHub login every 10min                                                                                                                                                                                                                                                                                                                                                                                                                                                              | the human's own Vault and GitHub sessions                                                                                                                                                                                                                                                                                                                                                                 |
-| Noninteractive sudo         | validated `/etc/sudoers.d/personal-fleet-USER`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | never; the human runs `manager.yml` once with `-K`                                                                                                                                                                                                                                                                                                                                                        |
-| T3 worker app               | pinned desktop `.deb` is the only T3 process (UI plus embedded server); GDM auto-login, user unit `t3code-desktop.service` bound to the graphical session with `Restart=always`; `personal-worker-recovery` checks every minute, starts a stopped app and restarts one whose local API missed three checks; administrator policy. A still-running headless `t3code.service` reports `T3_DESKTOP_MIGRATION_REQUIRED` (stopping it ends active turns)                                                                                                           | not a worker                                                                                                                                                                                                                                                                                                                                                                                              |
-| T3 providers                | the same managed instances (`fleet/agents/t3.json`) through locked T3 administrator policy                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | reconciled into the human's T3 settings by `fleet-sync` (`t3:providers`) with the ownership-safe `t3-apply` engine, so every T3 offers the identical set; unrelated and unmanaged instances are preserved. `fleet-sync` also applies the same proxy agent profiles (`configuration:apply`) and verifies the shared Vault gateway credential end to end with `provider:*:inference`, exactly as on workers |
-| Singleton services          | never; clients of the manager's services only                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | every singleton runs here: CLI proxy (`cliproxy`), Beszel hub (`workstation/monitoring`)                                                                                                                                                                                                                                                                                                                  |
+| Policy                      | Workers (Ubuntu 26.04, stateless)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Manager (macOS, singleton)                                                                                                                                                                                                                                                                                                                                                                                |
+| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Never sleep                 | systemd sleep targets masked; logind lid/idle policy; GNOME power plugin set to `nothing`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | never sleeps on AC power (`pmset -c sleep 0`, `macos/manager.yml`, verified as `manager:power`); battery sleep is left to the human                                                                                                                                                                                                                                                                       |
+| No automatic logout or lock | GNOME `lock-enabled false`, `idle-delay 0`; designated-user lingering                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | not managed (the human uses it)                                                                                                                                                                                                                                                                                                                                                                           |
+| Always-on services          | drop-ins `Restart=always`, `StartLimitIntervalSec=0` for ssh, tailscaled, docker, beszel-agent                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | singleton services under launchd and Docker Desktop (see the last row)                                                                                                                                                                                                                                                                                                                                    |
+| T3 health and connection    | `fleet-sync` and `fleet-status --workers` require `worker:t3-health` (local API answers), `t3:manager-endpoint` (manager HTTPS over Tailscale) and `t3:paired-access` (manager T3 link reachable)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | the human's T3 desktop holds the links to every worker (`t3:paired-access`)                                                                                                                                                                                                                                                                                                                               |
+| Memory safety               | `systemd-oomd` (swap 90%, pressure 60%); `user.slice` `MemoryMax=90%`; each `t3code-*.scope` agent `MemoryMax=30%`, `OOMPolicy=continue`; swap file when none                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
+| Disk safety                 | `personal-fleet-disk-guard` kills only the top user-cgroup writer below 3 GiB free, never deletes files; hourly `personal-fleet-disk-gc.timer` deletes git-ignored build output (`node_modules`, `.cache`, `dist`, ...) in checkouts idle 7d (12h under 15 GiB free) with no process inside, plus unused Docker images and build cache, never source, worktrees or volumes; journal 1G; `/tmp` 25% of RAM, aged 2d hourly, a measured write probe (`TMP_QUOTA_EXHAUSTED`), and T3 agents use disk-backed `/var/tmp` (`TMPDIR`, aged 14d); Docker log rotation                                                                                                                                                                       | not managed                                                                                                                                                                                                                                                                                                                                                                                               |
+| Tailscale                   | pinned APT package, `tailscaled` enabled, native login required                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Tailscale app, human login                                                                                                                                                                                                                                                                                                                                                                                |
+| Dashboard agent             | pinned native binary, hardened `beszel-agent` systemd unit as user `beszel`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | not installed                                                                                                                                                                                                                                                                                                                                                                                             |
+| Credential renewal          | user timer `personal-worker-auth` renews the node's own Vault token and GitHub login every 10min                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | the human's own Vault and GitHub sessions                                                                                                                                                                                                                                                                                                                                                                 |
+| Noninteractive sudo         | validated `/etc/sudoers.d/personal-fleet-USER`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | never; the human runs `manager.yml` once with `-K`                                                                                                                                                                                                                                                                                                                                                        |
+| T3 worker app               | pinned desktop `.deb` is the only T3 process (UI plus embedded server); GDM auto-login, user unit `t3code-desktop.service` bound to the graphical session with `Restart=always`; `personal-worker-recovery` runs a copy of `t3-health.sh` installed in `/usr/local/lib/personal-fleet` every minute: it stops and disables a still-running headless `t3code.service`, reports `T3_NO_GRAPHICAL_SESSION` and restarts the display manager (three times at most) when there is no session, starts a stopped app after `reset-failed`, restarts one whose local API missed three checks, and restarts one on another release than the installed one or with a deferred policy restart once it runs no agent turn; administrator policy | not a worker                                                                                                                                                                                                                                                                                                                                                                                              |
+| T3 providers                | the same managed instances (`fleet/agents/t3.json`) through locked T3 administrator policy                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | reconciled into the human's T3 settings by `fleet-sync` (`t3:providers`) with the ownership-safe `t3-apply` engine, so every T3 offers the identical set; unrelated and unmanaged instances are preserved. `fleet-sync` also applies the same proxy agent profiles (`configuration:apply`) and verifies the shared Vault gateway credential end to end with `provider:*:inference`, exactly as on workers |
+| Singleton services          | never; clients of the manager's services only                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | every singleton runs here: CLI proxy (`cliproxy`), Beszel hub (`workstation/monitoring`)                                                                                                                                                                                                                                                                                                                  |
 
 Credential sync is deliberately indirect: no secret is copied between machines.
 OpenTofu issues each enrolled worker its own renewable periodic token
@@ -211,8 +210,10 @@ vault kv get -field=FLEET_WORKER_TOKEN_WORKER_ONE secret/personal/prd \
   | ssh worker-one 'vault login -address=https://vault.example.com -no-print -'
 ```
 
-From then on the node's `personal-worker-auth` timer renews that token and its
-GitHub login, and renders provider configuration from Vault through Workstation.
+From then on the node's `personal-worker-auth` timer renews that token and
+restores its GitHub login from it (`bootstrap/worker-auth.ts`). It renders no
+provider configuration: agent profiles are applied by `fleet-sync`. See
+[Authentication and accounts](authentication.md).
 
 ## Ubuntu and remote-work safety
 
@@ -257,15 +258,18 @@ the existing LaunchAgent and requires a user session. Unattended manager boot
 acceptance and log-retention configuration remain gates. No node pull scheduler
 is installed or necessary.
 
-Sync never reboots a box. It does restart a worker's T3 when a new provider
-policy is staged, which ends that T3's running turns. Explicit
+Sync never reboots a box and never ends a running agent turn. A worker's T3
+that must restart for a new provider policy or release is restarted only while
+it runs no agent turn; otherwise the restart is pending
+(`T3_POLICY_RESTART_PENDING`, `T3_RESTART_PENDING`) and the recovery timer
+applies it at the first idle check. Explicit
 maintenance reboot/reconnect orchestration remains an acceptance gate. Kernel,
 driver, firmware, hardware and power failures are outside any “never freezes”
 promise. Inspect disk/inodes, swap/memory pressure, `systemctl --failed`, and prior
 boot journal evidence locally; sanitize it before sharing. Do not enable automatic
 reboot-on-pressure or kill active work to obtain a green report.
 
-T3 0.1.7's supported administrator policy at `/etc/t3code/policy.json`
+The pinned T3's (`t3_version` in `mise.toml`) supported administrator policy at `/etc/t3code/policy.json`
 disables independent updates and declares the existing managed Codex instance
 plus an OpenCode instance for all five gateway models. Fleet reuses the
 existing `agents/t3.json` renderer; it does not edit the live settings database.
@@ -303,17 +307,20 @@ launch; nothing is copied between nodes.
 
 **Ansible applies, TypeScript measures.** On a worker the T3 lifecycle is
 declared in playbooks: `t3-policy.yml` stages the provider policy and restarts
-a running T3 that has not loaded it, `t3-desktop.yml` installs the app, retires
+a running T3 that has not loaded it once it runs no agent turn, `t3-desktop.yml` installs the app, retires
 the former headless server and starts it, and `t3-route.yml` publishes the
 private tailnet route when port 443 has no owner. The TypeScript around them
 only refuses conflicts and reports what it observes afterwards. Each sync also
 prunes old revision checkouts on the worker, keeping the new one, the one T3
 launches from and the two most recent others.
 
-**Every sync applies everything.** `fleet-sync` never waits for a quiet
-moment: a worker whose T3 has not loaded the staged provider policy is
-restarted (ending its running turns), and a pending proxy update is installed
-even when it recreates the gateway. The manager finishes before the workers
+**Every sync applies everything, except a T3 restart that would end a turn.**
+A worker whose T3 has not loaded the staged provider policy, or runs an older
+release than the one installed, is restarted only while it runs no agent turn;
+otherwise sync reports the pending restart and the recovery timer applies it
+at the first idle check, failing the check after
+`t3RestartMaxDeferralHours` (`ansible/ubuntu/policy.json`). A pending proxy
+update is installed even when it recreates the gateway. The manager finishes before the workers
 start, so they are verified against the updated proxy. `--dry-run` previews.
 
 **Accounts are never listed in this repository.** The proxy's credential files

package/examples/fleet.json

@@ -29,6 +29,14 @@
       "proxy-tunnel-token": {
         "path": "cliproxy-fleet/tunnel",
         "field": "TUNNEL_TOKEN"
+      },
+      "monitoring-backup-access-key": {
+        "path": "beszel/backups",
+        "field": "ACCESS_KEY_ID"
+      },
+      "monitoring-backup-secret-key": {
+        "path": "beszel/backups",
+        "field": "SECRET_ACCESS_KEY"
       }
     }
   },
@@ -73,7 +81,16 @@
       "hubPort": 8443,
       "publicUrl": "https://beszel.example.com",
       "managerAccount": "fleet-manager@example.com",
-      "viewerAccount": "public-viewer@example.com"
+      "viewerAccount": "public-viewer@example.com",
+      "backups": {
+        "schedule": "0 3 * * *",
+        "keep": 7,
+        "storage": {
+          "bucket": "my-fleet-hub-backups",
+          "endpoint": "https://s3.example.com",
+          "region": "auto"
+        }
+      }
     }
   },
   "ubuntu": {
@@ -92,6 +109,7 @@
     "githubLogin": "example-user",
     "vaultUser": "example-user",
     "toolchain": "mise.lock",
+    "toolProfiles": ["go"],
     "agentProfile": "proxy",
     "unattendedSudo": true,
     "persistentT3": true,