@wpelevator/tug 0.12.0 → 0.13.0

package/docs/real-host-evaluation.md

@@ -0,0 +1,224 @@
+# Real-host evaluation, 31 October 2026
+
+Two WordPress sites on the existing test hosts, using the current CLI against the control plane already running there. This is an operator trial, not a passing CI job. The roadmap's WordPress picture was the target. What the tool can deploy today is a smaller thing, and the gap shows up as soon as you follow a normal Compose file.
+
+## Result
+
+Both sites are installed and serving over public HTTPS.
+
+**Version context.** The trial ran on the earlier runtime. Its control API was plain HTTP on port 8080 with bearer tokens, `/healthz` was unauthenticated, agents applied each project by running `docker compose up` on the node, and cross-node names were called service DNS. The current implementation differs: control serves a single mutual-TLS port (`8443`) with certificates from its own CA and no unauthenticated route, agents use the Docker Engine API with no Compose plugin on nodes, and the feature is called cross-node service names (hosts-file entries). Statements below about HTTP, port 8080, tokens, Compose behavior and `/healthz` describe the trial hosts, not the current product; [Control plane](control-plane.md), [Nodes](nodes.md) and [Architecture](architecture.md) describe the current one. All references to WireGuard, mesh membership, overlay addresses, `--mesh`, `TUG_MESH` and `node-rotate-wg-key` likewise describe the removed Tug-owned mesh implementation, retained only as historical evidence; they are not current capabilities or tests that still apply. Findings are kept as evidence, not rewritten.
+
+This document preserves what the installed `m4-amd64` hosts did during the trial. Since then, current `main` has made `tug deploy` include the node's apply error, made `doctor --node` wait for its queued result, kept lifecycle events quiet after the first change, and added an outbound agent wake channel so deployments and queued commands do not normally wait for the next periodic poll. The remaining findings are still open unless a paragraph says otherwise.
+
+| Site | URL | Node | Host | Cert |
+|---|---|---|---|---|
+| Press A | https://a.178.128.250.108.sslip.io | `app` | 142.93.140.20 | Let's Encrypt, CN=YE1, through 29 Jan 2027 |
+| Press B | https://b.178.128.250.108.sslip.io | `edge` | 178.128.250.108 | Let's Encrypt, CN=YE2, through 29 Jan 2027 |
+
+Each site is `wordpress:6.8-apache` plus `mariadb:11.4` on the same node, with a persistent volume for `/var/www/html` and another for `/var/lib/mysql`. Ingress is the managed Caddy on `edge`. HTTP returns 308 to HTTPS. The homepages return 200 with the site title. The admin user is `editor`. The password is only in `/tmp/tug-wp-eval/credentials.txt` on the operator machine, mode `0600`, and is not in this repo.
+
+The second `wp1` deploy, images already pulled, reached `succeeded` in 20 seconds. `wp2`, including the image pull onto the other host, took 57 seconds. After both were up, `app` had 263 MB available of 961 MB and `edge` had 331 MB. MariaDB RSS was 85 MB and 116 MB only because the compose file sets `--innodb-buffer-pool-size=48M`. WordPress RSS was about 120 MB on each host.
+
+This did not exercise the original specification's topology (plan.md, since removed in favor of the roadmap's [reference topology](roadmap.md#reference-topology)): three PHP-FPM nodes, a front node speaking FastCGI, a separate database node, Memcached, and an NFS docroot. Those needed M6 and M7, which have since been implemented but not validated on real hosts. With two 1 GB hosts, that layout would not fit even if the fields existed.
+
+## What I followed, and what I actually did
+
+The README quick start says install control over SSH, `tug init` against an HTTPS URL, set a secret, `node add`, deploy, then use `status`, `events`, `logs`, `restart` and `doctor`. The hosts were already past install. Control is bound to `10.110.0.3:8080`, not the public address. Port 8080 on `142.93.140.20` refuses connections. The working operator path is an SSH tunnel to `127.0.0.1:18082` and a config file outside the repo (`/tmp/tug-real-m4/config.json`, project still set to `realcheck`). That tunnel pattern is one example buried in the CLI and nodes docs. It is not the quick start.
+
+The agents are `tug:m4-amd64`, not `ghcr.io/wpelevator/tug:0.2.0`. `tug doctor` on the laptop reports Docker Compose v5.6.1. The same command's control check reports v2.32.4, which is the plugin inside the runtime image. `tug status` does not show the image tag or the CLI version the node is running. An operator comparing those two Compose versions will think the install is inconsistent. It is, but status will not say so.
+
+I did not reinstall from the published image. The cluster was already serving, and a rebuild would have tested the installer again rather than the application path. The version skew is still invisible.
+
+## What worked
+
+- Two projects on one ingress node. Caddy picked up `a.178.128.250.108.sslip.io` and `b.178.128.250.108.sslip.io` without dropping the existing `178.128.250.108.sslip.io` route.
+- Certificates issued with `TUG_ACME_EMAIL` unset. DNS for `*.178.128.250.108.sslip.io` is the whole trick, and it is not mentioned in the docs.
+- `secret set` from stdin, before the project row existed. Values stayed out of argv. `secret list` shows names and versions only.
+- Placement, persistent volume records, and `volume list` naming the node.
+- Health checks: TCP to MariaDB, HTTP to WordPress. Once the compose file was one Compose would accept, both deploys converged without raising the 300 second default timeout.
+- Binding the app port to the overlay address on `app` (`10.210.0.1:8080:80`) and to `127.0.0.1:8080:80` on `edge` kept port 8080 off the public interfaces. Caddy still reached both sites.
+
+## Failures and friction
+
+### The first deploy failed, and the CLI hid the reason
+
+The first compose file used the dependency every WordPress example uses:
+
+```yaml
+depends_on:
+  db:
+    condition: service_healthy
+```
+
+MariaDB 11.4 has no Docker `HEALTHCHECK`. `x-tug.health` is not one either. Tug strips `x-tug` before writing the node compose file, then runs `docker compose up`. Compose refused to start WordPress:
+
+```text
+dependency failed to start: container wp1-db-1 has no healthcheck configured
+```
+
+`tug deploy` printed `deployment dep_73508669cffa774f failed: one or more assigned nodes reported failed health` and exited in 50 seconds. That is shorter than the health retries in the file, so it was not a timeout. `tug events --project wp1` showed `service.pulling` and `service.starting`, then nothing about the failure. `tug logs wordpress` printed nothing, because the container was still `Created`. `tug logs db` showed MariaDB ready for connections, which sends you looking at WordPress. The compose error exists only inside the `status_json` string on the node status row. I got it by SSH, then by parsing `tug status`.
+
+Both visibility gaps are closed on current `main` and not on these hosts. [PR #13](https://github.com/wpelevator/tug/pull/13) emits one `service.failed` event with the apply error, and retries of the same failure stay quiet. [PR #17](https://github.com/wpelevator/tug/pull/17) also includes the reporting node's apply or rollout error in the deployment failure that `tug deploy` prints.
+
+A plain `depends_on: [db]` worked. The WordPress image already waits for the database. The docs say `depends_on` is a hint and that readiness comes from `x-tug.health`. They do not say that `condition: service_healthy` fails the deploy, or that you must add a Docker `healthcheck:` block yourself if you want Compose to wait.
+
+### JSON is the interface
+
+`tug deploy` on success prints the deployment row and the full desired state, including the compose file. The word `succeeded` is a field you scroll to. There is no human summary and no `--quiet`. The docs say there is no global `--json` because JSON is already the default. That is honest, and it is also why a failed deploy is hard to read: the one string a person needs is not in the error line.
+
+`tug status` returns database rows. Health is a snake_case column plus a stringified JSON blob (`status_json`) that holds the camelCase agent report. Finding out that `wp1` is serving means parsing a string inside a string. CLI docs say new structured results use camelCase and that unchanged API responses may keep database names. Status is the command an operator runs first, and it is the unchanged dump.
+
+`doctor --node edge` returned `"ok": true` and a command id. It does not wait. `restart` and `node-rotate-wg-key` do. The command description says "run" diagnostics. The nodes doc says "queues". Both are true, and the result never comes back to the terminal.
+
+### The config project is a trap
+
+The tunnel config's `project` is `realcheck`. `tug logs wordpress`, without `--project`, posted to that project and returned `404 {"error":"service not found"}`, then told me to run `--help`. It did not say which project it used. `events` is project-scoped the same way. `status` is control-wide. After `init`, every later command inherits whatever project you set, and the error looks like a missing service.
+
+### Ingress bind addresses are insider knowledge
+
+Caddy dials `127.0.0.1` for a service on the ingress node and the overlay address for any other node (`src/ingress.ts`). A published port of `8080:80` listens on `0.0.0.0`, so the app is also on the public IP, next to Caddy, with no TLS. Binding the overlay IP on the app node and `127.0.0.1` on the edge node is what makes the ingress work without that exposure. The networking doc says the edge uses the published host port. It does not say which address that port has to be on, or that the overlay IP is something you have to look up and paste into the compose file. If the node is re-registered and gets a new overlay address, the compose file is wrong and nothing in `tug deploy` will say so before Compose tries the bind.
+
+### Cross-node service names (then called service DNS) do not get you a remote database
+
+Names injected with `extra_hosts` point at the node address, and the client must use the published host port. A MariaDB that is not published is reachable only on the Compose network, which means the same node. Publishing `3306` so WordPress on the other node can use the name puts the database on an address the mesh can route to. Binding it to the overlay IP only is the same undocumented trick as the app port. The generated `wp2` compose file had no `extra_hosts`, which is correct: both of its services are on `edge`.
+
+So the deployable WordPress shape is app and database on one node, one project per site. The original specification's shape, app nodes talking to a database node over a service name, is not the shape this tool deploys. Copying that example fails for more than missing nodes. It uses `domains`, `upstream`, `upstreamProtocol`, `sticky`, and `type: shared` / `driver: nfs`. Current validation accepts `placement`, `replicas` (must equal the node count), `health`, `ingress`, and `rollout`. The rest is either ignored or not implemented. At the time of this evaluation M6 (label placement, `replicas: N` spread) and M7 (NFS docroot) were not built; both are implemented now but unvalidated on real hosts. Ingress is an HTTP reverse proxy, not FastCGI.
+
+Rolling the WordPress container would be the wrong next step. A rolling revision publishes a different host port and shares the named volume with the old revision. Two PHP processes on one `wp-content` is how you corrupt uploads. The rollout docs say volumes are shared and that databases should stay on `recreate`. They do not say that about a PHP docroot. I left both sites on recreate.
+
+### Secrets and environment are easy to get half-right
+
+A value that is not exactly `${NAME}` makes Tug treat the environment key as a secret name. `WORDPRESS_DB_HOST: db` does not require a secret, and the literal is what the container gets. A project secret with that same name would also be released to the node and written into `.env`. The `.env` writer does not quote values. A password containing `#`, `$`, a space, or a newline breaks interpolation. Nothing in the secrets section says "use a substitution for the password, a literal for the hostname, and keep the password free of compose metacharacters." I used hex passwords because I had read `writeEnv`, not because the docs told me to.
+
+### Steady state is still a health check every poll
+
+`reconcile` compares container config hashes and runs the health checks on every poll, not only when the desired version changes. After `wp1` was already `serving`, the WordPress access log still showed user-agent `node`, from `172.20.0.1`, requesting `/` and following the 302 to `/wp-admin/install.php` about every 9 seconds. Node's fetch follows redirects. The docs mention that. They do not mention that a redirect to the public hostname would send the health check out through Caddy and back, or that the checker keeps doing it after the deploy has succeeded. For this site the installer returned 200, so the check passed. A redirect to a non-2xx URL would flap a healthy site.
+
+On the image these hosts are running, those polls also wrote `service.pulling`, `starting`, `health_checking` and `serving` every time. [PR #13](https://github.com/wpelevator/tug/pull/13), merged after this trial, stops that. The agent keeps a fingerprint and emits lifecycle events only when the revision, rendered compose or secrets change. The apply and the health check still run on every poll, which the new observability page states on purpose. The access-log noise is not fixed. The event flood is, once these agents are replaced.
+
+### What I still had to do by SSH
+
+M5 (`exec`, `logs -f`, `ps`) is the stated next milestone, and this run needed it on the first failure. Completing the WordPress install was a POST to `install.php` from the laptop, which is fine. Finding out why the container never started was not. Current `main` now puts the last apply error in the deployment failure and waits for `doctor --node`, but it still has no live shell or log stream.
+
+`volume list` records the volume and the node. It does not show whether the Docker volume exists or how large it is. I did not test `volume remove`. I also did not re-run rolling deploys, rollback, or `--build-local` on these sites. Those already passed on the nginx fixture. They are not what a WordPress install hits first.
+
+## Documentation
+
+The pages disagree with the path that works.
+
+- The README's steps 1 and 2 assume a public HTTPS control URL. This cluster's API is HTTP on a VPC address, on purpose. The quick start should show the tunnel, and it should say that `control setup` publishes `PORT:8080` on all interfaces unless you change the rendered compose. The bind to `10.110.0.3` is not a documented flag.
+- There is no compose example with a database, a secret, a volume, and an ingress host. The samples are nginx, busybox, and redis, placed on `vps-a`, `vps-b`, and `vps-c`. These nodes are named `app` and `edge`. A person copying the sample will fail placement and learn nothing about `depends_on` or TLS.
+- `sslip.io` is how both real-host runs got a trusted certificate without owning a domain. It belongs in the networking page next to `tls: auto`, including the rate-limit warning: sslip.io is one registered domain, and Let's Encrypt's per-domain cap is shared with everyone else using it. Two hostnames worked today. A third batch might not.
+- Preflight checks free disk (2048 MB under `/var/lib` for control, 1024 MB for a node) and never checks RAM. Both hosts have 1 GB and no swap. An unmodified MariaDB on the control host, next to the agent and the control container, is a bad idea. The control-plane page says the control VPS is not in the request path. It does not say not to put a database on that same 1 GB machine. This cluster does, because there are only two machines.
+- plan.md is still the "original specification" linked from the README, and its WordPress compose file is not a file this CLI accepts. The roadmap already says M6 and M7 serve that topology. The README does not. Someone who starts at the README and opens plan.md will try to deploy it. Resolved since: plan.md was removed and the [roadmap](roadmap.md#reference-topology) now holds the reference topology.
+
+## What I would change first
+
+1. Add one worked example: WordPress or any HTTP app plus a database, secrets via `${DB_PASSWORD}`, `depends_on` without `service_healthy`, ingress on an sslip.io name, and a note on the 1 GB MariaDB flag. State that app and database have to share a node until a published port or M7 exists.
+2. Make `tug deploy` print one line on success (`wp1 wp1-2 succeeded, 2 services`) and keep the JSON behind a flag. Same for `status`: project, version, health, node, without a nested string.
+3. Name the selected project in "service not found" errors.
+4. Stop publishing application ports on `0.0.0.0` by default when an ingress is set. Bind the overlay address, or `127.0.0.1` on the ingress node, in the generated file. The operator should not have to paste `10.210.0.1` into the source compose.
+5. Say in the README that the plan.md WordPress stack is not deployable yet, and point at the example that is. (Done differently: plan.md was removed and the roadmap describes the topology Tug targets.)
+
+I would not treat this run as evidence for rolling deploys, image build, or rollback. Those were not on this path. I would treat it as evidence that a person can get two real sites up, and that the first failure still requires SSH.
+
+
+## M6, M7 and enrollment on fresh hosts
+
+**Removed mesh implementation:** the commands, flags, environment variables, overlay addresses and packet-path results in this section are historical. Tug no longer provides that mesh, and these mesh checks are not a current validation baseline.
+
+31 October 2026, on the same two Ubuntu 24.04 droplets (1 vCPU, 1 GB, amd64, a shared VPC), after wiping both: containers, volumes, images, `wg0` and all Tug state. Control ran on 142.93.140.20 with `TUG_MESH=true`. The Tug image was built for amd64 from the branch and loaded with `docker save`.
+
+| Check | Result |
+|---|---|
+| `node add --mesh --label ...` on two hosts (`a`: `app`,`storage`; `b`: `app`) | Both online with labels; WireGuard mesh up |
+| `global: true` + `placement.labels: [app]` | One `web` replica on each labelled node |
+| `replicas: 1` + labels | One `worker`, on the alphabetically first matching node |
+| `tug scale worker 2` / `1` | Second replica started on `b`, then removed; `0` and unknown services rejected |
+| NFS volume (`x-tug.type: nfs`, exporter `a`; the volume type has since been removed) | `b` mounted `10.210.0.1:/srv/tug/shared` over the WireGuard overlay (NFSv3); a file written on `b` appeared on `a`; the exporter bind-mounted the directory |
+| `ps`, `inspect`, `exec`, `logs -f` against real nodes | Worked |
+| `node enroll --install-docker --mesh` cloud-config on a host with Docker purged | Docker Engine and Compose installed, agent started, node enrolled with its label |
+| Enrolled node with no public address behind `ufw` deny-incoming | WireGuard handshake and overlay ping to `a` worked; NFS mount worked |
+
+What this does not prove: the cloud-config was applied by re-running cloud-init (`cloud-init clean`, then its stages) with the config in `cloud.cfg.d`, not from first-boot user data, and the agent image came through an SSH-tunnelled throwaway registry. The firewalled node has a public IP, so this is not NAT. Debian, Fedora and Rocky bootstraps, arm64, and certificate renewal were not run.
+
+### Findings
+
+- **Fixed: dead nodes held rollouts open.** After `b`'s host was repurposed as `c`, the old `b` stayed registered with its labels. The next deploy placed `global` replicas on `b`, which never answered, and the deployment stayed `rolling_out` until cancelled. Label and `global` placement, and `tug scale`, now skip nodes not seen within `TUG_STALE_SECONDS`. `scale` also no longer spreads a labelled service onto nodes that do not match its labels when none match.
+- **Fixed: there was no way to remove a node.** `tug node remove <node>` now deletes it (refused while a current deployment places something on it, or while its agent still reports, unless `--force`). Run on the hosts: removing the dead `b` succeeded and `a`'s WireGuard peer for it was dropped, while removing `a` was refused with the services placed on it.
+- **Fixed: `control setup` could not enable the mesh.** `TUG_MESH` had to be added to the generated compose file by hand; `control setup --mesh` now does it (unit-tested only).
+- **Closed by removal: NFS needed host preparation Tug does not do.** The exporter needed an NFS server and `/etc/exports`, consumers `nfs-common`. The `nfs` volume type was removed rather than growing a host installer.
+- **Open: `cloud-init clean` regenerates SSH host keys.** Expected, but it makes the next SSH to the host fail strict checking.
+
+## Pull-once and backup recipe on a fresh host, 31 October 2026
+
+A single freshly wiped Ubuntu 24.04 host (Docker 29.9.2, 1 GB RAM) with control and one agent built from `main` after PR 25.
+
+| Check | Result |
+|---|---|
+| Unchanged project left alone for three minutes of agent polls | No container was recreated (same container IDs and start times) and no `service.pulling` event or pull line in the agent log appeared after the deploy finished |
+| `examples/wordpress-backup` deployed with `tug deploy` | WordPress, MariaDB 11.4.8 and the backup sidecar converged and WordPress answered on its published port |
+| `tug exec backup -- backup.sh once`, then `restic snapshots` | Exit 0, one `db` and one `files` snapshot, `restic check` reported no errors |
+| Restore drill into a scratch project (`db` and `backup` only) | The two seeded database rows and the seeded file came back; the repository was a local path, not S3 |
+
+### Second node and S3
+
+The second host (a cloud-init rebuilt Ubuntu 24.04, Docker 29.9.2) was wiped and joined with `node add` from the PR 27 branch; its agent came up with `net=host`. `node add` over an already registered node name worked.
+
+| Check | Result |
+|---|---|
+| `wp1` redeployed with a `backup-edge` sidecar and an `uploads` volume placed on the second node | Succeeded in 53 s; the sidecar image was built on control and downloaded by the second node's agent; the sidecar was healthy |
+| Both sidecars `backup.sh once` into one S3-compatible repository | Snapshots tagged host `app` (`db`, `files`) and host `edge` (`files`) in one repository; `restic check` reported no errors |
+| Restore of the `edge` snapshot from the first node into a scratch directory | The text file matched and the 3 MB random file had the same SHA-256 as the original |
+
+The store was RustFS in a container on the second node with random credentials, because the `quay.io/minio/minio` image answers 401 and the `minio/minio` Docker Hub image can no longer be pulled. It is not a hosted provider, and the repository was reached over plain HTTP on a public address, so treat the result as a protocol check, not a production recommendation.
+
+### Findings
+
+- **Fixed: health checks cannot reach containers from a bridge-networked agent.** At the time, `node add` started the agent on the default bridge. On Docker 29.9.2 it could not dial the container IPs of a project's Compose network, so every TCP or HTTP health check timed out (`tcp timeout 172.19.0.2:3306`) while MariaDB was ready, and the first deploy failed. Restarting the agent with `--network host` fixed it. Earlier runs on older Docker did not hit this. Current `node add` and `node enroll` always start the agent with `--network host` for these health checks; no added network capability is required.
+- **Open: `--build-local-image` builds for the CLI machine's architecture.** On an arm64 Mac it produced an arm64 image that the amd64 host could not run (`exec format error`). It should build for the target's architecture.
+- **Open: a deploy that fails health leaves the project half-up.** The containers kept running after the failed deployment, which hid the real cause until the agent's view was checked.
+
+## Generic backups, restore and `tug run` on fresh hosts, 1 November 2026
+
+The `examples/wordpress` project on two freshly reimaged 1 GB droplets, with the runtime built from the branch (`tug:local`, amd64) and nothing else on the hosts. The hosts were Ubuntu 26.04.1 LTS, newer than the 24.04 of earlier runs. Control and node `app` ran on `tug-1` and node `app2` on `tug-2`. The same example was deployed twice: as project `blog` on `app` and as project `shop` on `app2`, each with its own database and uploads volume. Backups went to an S3-compatible server (`rustfs`) that Tug itself deployed as a third project (`storage`) on `app`, reachable only on the private network. This is an operator trial, not a CI job.
+
+### What ran, and the result
+
+| Step | Result on both sites |
+|---|---|
+| `control setup --install-docker --build-local-image`, `node add` for both nodes | Worked on Ubuntu 26.04: Docker CE 29.9.2 came from Docker's own `resolute` suite, control and both agents came up, nodes registered online |
+| `tug deploy` of the example | Converged with health checks passing; WordPress answered `302` to its installer, then served after a scripted install |
+| `backup run` | Database dump (`mariadb-dump` run in the container, 107 KiB) and the uploads volume (14.1 MiB) both succeeded in about ten seconds |
+| `backup list --check` | Exit 0 with a healthy target list |
+| Damage: update the site title in the database, delete the active theme, create a stray file | The site rendered a blank page, as expected |
+| `restore db --confirm` | Completed in about 17 to 19 seconds: WordPress stopped, safety backup, dump loaded, WordPress started |
+| `restore wp-content --confirm` | Completed; the site title was present **immediately** when the command returned, with no sleep. The theme was back and the stray file gone |
+| `restore wp-content --scratch --confirm` | New volume with the files, the live volume untouched; it restored the last real backup, not the safety snapshot taken a moment earlier |
+| Scheduled runs | `shop` was redeployed with a two-minute schedule: the scheduler noticed the change within its 30-second tick and started both targets by itself at the next slot (trigger `schedule`); both succeeded. Redeploying the nightly schedule put the next run at 03:00 UTC again |
+| `tug run wordpress -- ...` | Saw the service's environment (`WORDPRESS_DB_HOST=db`); exit codes 0 and 7 were returned; no helper containers were left on either node |
+| Repository contents | Each project has its own repository (`tug-wp/blog`, `tug-wp/shop`), four snapshots each (two backups, two safety snapshots), hosts `app` and `app2`; `restic check` passed when run from the *other* server, a machine that never held the data |
+| Memory | `tug-1` (control, agent, S3, MariaDB, WordPress) used 508 of 956 MB; `tug-2` used 420 MB; no out-of-memory events |
+
+### Findings
+
+- **The updater can starve a small host.** On the earlier installs Ubuntu's `unattended-upgrades` and `apt-check` had run for over an hour on `tug-1`, holding it at load 16 until Docker commands timed out. The fresh hosts had the apt timers disabled before anything else was installed. This is now noted in [Nodes](nodes.md).
+- **Control and a node on one host compete for port 8080.** The example publishes `8080:80`, control publishes `8080`, so the copy for `tug-1` used `8081`. Control on a runtime node is [M15](roadmap.md#m15--flexible-node-roles).
+- **Named volumes need no ownership fixing.** The S3 server runs as uid 10001. As a bind mount it failed with `Permission denied`; as a named volume deployed by Tug it started cleanly.
+- **Scratch-restore volumes are left behind**, as documented. They were removed by hand.
+
+Not covered by this run: a real AWS S3 bucket (the S3 server is `rustfs`, which speaks the same protocol), a failing backup on a real host (`backup list --check` was only seen exiting 0 here; its failing path is covered by unit tests), a control restart or a long control outage, a node behind NAT, ingress and certificates (the hosts had no DNS names, so the sites were reached on their private addresses), and a project that spans both nodes.
+
+## Recovery drill with the published 0.9.0 release, 2 November 2026
+
+Two disposable amd64 Ubuntu hosts were used. The published npm CLI and `ghcr.io/wpelevator/tug:0.9.0` image ran control and the original agent on one host; an agent on the other host served as the replacement workload node. A dedicated RustFS container on the replacement host stored workload backups over the private network; this was an S3-protocol exercise over HTTP, not a hosted S3 or encrypted-transport validation. One host already had an unrelated control container and data, which were left in place. The release predates removal of Tug’s optional mesh; no mesh was enabled.
+
+The WordPress example deployed successfully, WordPress was initialized, and an upload sentinel was created. A manual backup produced a MariaDB dump and a `wp-content` snapshot. A separate control backup and the repository credentials were copied off the control host into private local files. The original workload agent and containers were stopped to simulate workload loss while keeping control available. The project was redeployed with placement changed to the replacement node, then the database and uploads were restored there with Tug’s normal safety backups. The recovered database contained the original site title, the uploaded sentinel matched exactly, and HTTP returned 200. This tested a manual re-placement and restore, not an automatic host failover; external DNS, HTTPS and a second public request path were not tested.
+
+Immediately after the replacement deployment succeeded, the restore plan still named the old node because backup schedules had not yet caught up. After a later scheduler tick it named the replacement; the restore was not confirmed until the target had been checked. The branch now refreshes schedules when planning and queueing a restore and has a regression test. This fix was not part of the published release used for the drill.
+
+For control recovery, an updated control backup was copied off-host. The test control container was stopped and removed, and its data directory was moved aside intact. A new control was bootstrapped on the same machine with fresh storage and a temporary credential. The backup was copied into its mounted data directory, restored through the API and activated by restarting control. The original credential and CA worked again, the replacement agent reconnected, the recovered backup repository credentials matched the escrowed copy, and the WordPress site still returned HTTP 200. This first pass demonstrates fresh-directory recovery on the same machine, not loss of its underlying disk. The test did not verify certificate renewal, automatic off-host backup, or a restore after the backup destination itself is lost. No measured recovery time or data-loss objective is claimed from this exploratory run.
+
+A second control backup cycle included a small Tug-built image. After restoring that backup into another fresh control data directory, the disposable probe container and its local image were removed from the replacement node. Its agent downloaded the retained artifact from restored control and recreated the container; the file baked into the image contained the expected marker. An initial remote-build attempt launched from the repository directory with an absolute Compose path failed with `source context has unsafe entries or exceeds expanded limits`; running the same deployment from its own clean context directory succeeded. The cause of the first rejection was not established.
+
+A further control recovery used the other VPS: a separate control container was bootstrapped on a distinct port and empty data directory without replacing that host’s pre-existing control instance. The off-host backup containing the built image was copied into the new data directory, restored using the temporary bootstrap credential, and activated by restart. The original test control was stopped to prevent two active authorities. The original operator credential then worked with a control URL override, and `node replace` repointed the replacement agent to the recovered control; the node reported online and the backup catalog was present. The probe’s local container and image were deleted once more, and the agent re-downloaded the restored built image from control on the new host; the image marker matched and WordPress returned HTTP 200. This confirms manual cross-host control recovery for this topology, but the replacement host also held the test object store and site, so it did not test simultaneous loss of that host or its object store. `node add` for the existing node returned 409 and required the identity-replacement command.

package/docs/releases.md

@@ -0,0 +1,39 @@
+# Releases
+
+Tug is published as the npm package [`@wpelevator/tug`](https://www.npmjs.com/package/@wpelevator/tug) (bins `tug` and `tug-runtime`) and as the multi-arch (`linux/amd64`, `linux/arm64`) Docker image `ghcr.io/wpelevator/tug`. The package and image always share one version, and the CLI's install commands default to the image with the CLI's own version.
+
+## Changesets
+
+Versions come from [Changesets](https://github.com/changesets/changesets). Every pull request with a user-facing change (CLI, API, agent behavior, image contents, docs for a new behavior) adds a changeset:
+
+```bash
+npm run changeset
+```
+
+It asks for the bump (`patch`, `minor` or `major`) and a summary, and writes a Markdown file in `.changeset/` that you commit with the PR. The summary becomes the changelog entry, so write it for users. Changes with no user-facing effect (CI, refactors, tests) need none.
+
+## Release flow
+
+The npm package includes the compiled runtime in `dist/` and the documentation in `docs/`, as listed in `package.json`'s `files` allowlist. npm also includes the root README and package metadata. To inspect the release contents locally without publishing, run `npm pack --dry-run`; its `prepack` hook builds the runtime first.
+
+1. A PR with a changeset merges to `main`.
+2. `release.yml` runs on every push to `main`. While unreleased changesets exist, `changesets/action` opens or updates a "chore: version packages" PR that runs `npm run version-packages` (bumps `package.json`, updates `package-lock.json`, writes `CHANGELOG.md` and deletes the consumed changesets).
+3. Merging that PR is the release. The same workflow then runs `npm run release` (`changeset publish`), which publishes `@wpelevator/tug` to npm with provenance and creates the git tag.
+4. When a package was published, the `image` job builds the multi-arch image from that commit and pushes `ghcr.io/wpelevator/tug` tagged `X.Y.Z`, `X.Y` and `latest`.
+5. Independently, every push to `main` pushes `ghcr.io/wpelevator/tug:edge`, the latest unreleased build.
+
+Nothing is released by hand: do not run `npm publish` or create tags yourself. Pull requests created by the workflow with `GITHUB_TOKEN` do not trigger other workflows, so CI does not start on the "Version Packages" PR by itself; push an empty commit to its branch or close and reopen it if you want checks on it.
+
+## npm trusted publishing
+
+The workflow publishes with npm [trusted publishing](https://docs.npmjs.com/trusted-publishers) over OIDC, so there is no `NPM_TOKEN` secret. The package on npmjs.com is configured with GitHub repository `wpelevator/tug` and workflow file `release.yml`, so that file must keep its name. The job has `permissions: id-token: write` and installs npm >= 11.5.1, the first version that supports trusted publishing.
+
+## One-time repository setup
+
+- Settings > Actions > General > Workflow permissions: allow "Read and write permissions" and "Allow GitHub Actions to create and approve pull requests", so `changesets/action` can open the Version Packages PR.
+- After the first image push, open the `tug` package under the organization's Packages and set its visibility to public (new GHCR packages are private by default), so hosts can pull without logging in. Make sure the package is linked to the repository and that Actions has write access to it (Package settings > Manage Actions access).
+- Optionally protect `main` and require the CI checks, so a release only comes from tested code.
+
+## Local development image
+
+To try an unreleased build, `tug control setup` and `tug node add` accept `--build-local-image`, which builds `tug:local` from the checkout and copies it over SSH, or `--image ghcr.io/wpelevator/tug:edge`. See [Nodes](nodes.md) and [Control plane](control-plane.md).

package/docs/roadmap.md

@@ -0,0 +1,234 @@
+# Roadmap
+
+This is the canonical statement of what Tug is for, the shape of deployment it targets, and the milestones that get it there. A milestone that is being worked on gets a `plan-{task-name}.md` working file (see [AGENTS.md](../AGENTS.md)); when it lands, its lasting decisions move into the docs and its item here is checked off.
+
+## Goal
+
+Run a few Docker Compose applications across a handful of ordinary Linux VPSes from one small self-hosted control plane, without Kubernetes, a hosted PaaS or a cloud provider's API.
+
+The operator this is for runs one to a few dozen sites on two to ten VPSes, deploys from a laptop or CI, and needs the day-2 jobs covered as well as the deploy: backups and restores, one-off commands such as `wp` CLI and database upgrades, logs, safe rollouts and rollbacks, and an audit of who changed what. Feature parity with other tools is not the goal; [uncloud](https://github.com/psviderski/uncloud) and Kamal are useful baselines for what operators expect (see [Positioning](#positioning)).
+
+## Principles
+
+1. **Compose stays the app format.** `docker compose up` works locally; `x-tug` adds deployment metadata, and Compose features that cannot work across nodes are rejected rather than half-supported ([Compose extensions](compose.md#not-supported)). Compose is the input, not the node runtime: nodes need only the Docker Engine ([M14](#m14--engine-api-agent)).
+2. **Production runs immutable images.** Application code, including WordPress core, themes and plugins, ships inside an image built on control or on the operator's machine. Every image a deployment uses is pinned to a digest when the deployment is created ([M13](#m13--pinned-images-and-one-delivery-path-done)). Nothing is built or synced into a running node during reconciliation, so an image rollback restores the earlier image and configuration when retained artifacts remain available. It does not reverse database changes, uploads or external side effects.
+3. **Agents reconcile desired state and only dial out.** Normal deploys need no SSH and no inbound port on nodes, so nodes behind NAT or a deny-all firewall work. Running containers keep serving while control is down.
+4. **One writer.** Desired state, locks, secrets and history live in one control plane. This keeps ordering, rollback and audit simple; availability comes from control backups and, if needed later, a warm standby.
+5. **Stateful things are pinned and backed up.** Databases and data volumes stay on their placement node unless the operator moves them, are never deleted by a deploy, and every first-class stateful service has a backup and restore path.
+6. **Private by default.** Bind ports to the configured node address or loopback unless the operator explicitly requests a public bind. Reachability, firewalling and workload-traffic encryption depend on the operator's network; a private address alone is not an isolation guarantee.
+7. **Start simple.** No distributed storage, no autoscaling, no bin-packing scheduler, no database failover. Single points of failure are allowed when they are documented. A single node needs nothing beyond Docker, the agent and control: ingress and other subsystems stay optional layers on top of that core, and multi-node workloads use the operator's existing network.
+
+Non-goals: Kubernetes compatibility, provisioning VPSes through provider APIs, autoscaling, distributed block or file storage, database HA, a multi-tenant hosted control plane.
+
+## Reference topology
+
+These are the shapes Tug is tested against and the ones the docs should steer people toward.
+
+**Small: one node per site, or several sites per node.** App and database on the same node, the managed Caddy ingress on that node, data in local named volumes. This is what most WordPress sites need and what was validated first ([Real-host evaluation](real-host-evaluation.md)). Many sites per node is the common case, so clear project boundaries, ingress routing for many hostnames and per-site backups matter more than multi-node features. Project boundaries are not isolation between mutually untrusted workloads.
+
+**Scaled WordPress (a possible future topology, not a validated support promise):**
+
+```text
+Internet -> ingress node (managed Caddy, TLS)
+              | HTTP, over the operator's private network
+              v
+         app1, app2, app3: nginx + php-fpm, image with core/themes/plugins
+              | uploads             | SQL / cache
+              v                     v
+         uploads store         db node (MariaDB), cache (Memcached)
+         (S3-compatible or a file server you run, wp-content/uploads only)
+```
+
+- WordPress code is in the image (Bedrock or Composer style), so every app node runs exactly the deployed version and a rolling rollout or rollback swaps images atomically per node.
+- Only `wp-content/uploads` is shared: preferably on S3-compatible object storage through an offload plugin, otherwise on a file server you run and mount with plain Compose `driver_opts`. PHP code is never served from a network file system, which avoids opcache stat latency, mixed versions during a sync and non-atomic rollbacks.
+- Each app node serves its own static files with nginx next to php-fpm, and the ingress balances HTTP across app nodes. There is no FastCGI across hosts and no front node that needs the shared filesystem.
+- The database, cache, uploads store and ingress are each single instances. Scaling the app tier adds PHP capacity, not availability; the docs say so, and backups are the recovery story.
+
+The earlier specification (a front nginx speaking FastCGI to app nodes over a shared NFS docroot, with WordPress files synced to NFS on deploy) was dropped for the reasons above.
+
+## Positioning
+
+Keep the single-writer control plane rather than decentralizing it like uncloud. Deploy locks, ordered deployments and rollbacks, secret versions and build-then-deploy sequencing are simple and correct with one writer and hard to get right with several. Invest in what a central API makes easy and distributed control makes hard: deploy safety, scheduled backups with a catalog, auditability, CI automation and multi-operator access.
+
+Alternatives considered:
+
+- **uncloud** (decentralized, WireGuard mesh, container-routable networking, unregistry image push). Closest in spirit. Tug differs by having one control plane, agents that only dial out and no Tug-owned workload overlay: multi-node traffic uses the operator's existing private network.
+- **Docker Swarm** overlaps Tug's placement, reconciliation, rolling updates, secrets and node mTLS, and adds continuous task rescheduling, service DNS/VIPs, overlay networking and replicated managers. A single manager works; three tolerate one manager failure. Manager and overlay connectivity must be available on the private network, which can be a suitable VPN behind NAT. Tug instead integrates source builds, registry-free application image delivery, managed HTTPS, actor-attributed deployment history, scoped operator access and workload backup/restore. Docker has events and supports authorization plugins, so those Tug advantages concern the built-in application workflow. Not adopted; it remains the yardstick for how much Tug should build itself. See the source-linked [Docker Swarm comparison](swarm-comparison.md) for implemented differences, failure behavior and workload fit.
+- **Kamal** pushes over SSH with no control plane. Simpler, but no reconciliation, no central audit or scoped CI credentials, and no state for scheduled jobs such as backups.
+
+Re-check uncloud's releases at the start of each milestone; baseline rechecked on 2026-10-31: [v0.21.0](https://github.com/psviderski/uncloud/releases/tag/v0.21.0), adding shared Caddy certificate storage, systemd socket activation and runtime templates to the existing mesh, DNS and deployment features.
+
+## Milestones
+
+Numbers are identifiers, not order; the order is in [Sequencing](#sequencing).
+
+### M1 — TypeScript adoption (done)
+
+Typed `src/protocol.ts` control↔agent contract and injectable agent runners. Source and tests run as `.ts` through Node type stripping; only the Docker image is compiled to `dist/`. See [Development](development.md) and [Architecture](architecture.md).
+
+### M2 — Cross-node networking, service names, ingress/TLS (done)
+
+Health-aware cross-node service names (hosts-file `extra_hosts` entries, not DNS) are on by default with a `TUG_SERVICE_NAMES=false` opt-out, and the managed Caddy ingress provides optional `x-tug.ingress` routes and automatic TLS. Names and upstreams resolve to each node's `address` on the operator's private network; service names need published host ports. Tug owns no overlay or workload-traffic encryption. See [Networking](networking.md). Stable rolling-service name/port and avoiding client recreation on endpoint changes remain in [Later](#later--only-if-needed).
+
+### M3 — Zero-downtime rolling deploys + automatic rollback (done)
+
+Rolling rollouts for ingress-fronted services (start the new revision on its own host port, health-check it, switch the Caddy upstream, drain the old one), `x-tug.rollout` options and opt-in `x-tug.rollback: auto`. See [Architecture](architecture.md#rolling-rollouts-and-automatic-rollback) and [Compose extensions](compose.md#rolling-rollouts). The numbers it is held to are in [Testing](testing.md#metrics). Rolling services have no stable cross-node name and port; see [Later](#later--only-if-needed).
+
+### M4 — Images without a registry, bare-host install (done)
+
+Control stores built images as saved image files and agents download them over their own authenticated connection, so no registry or daemon change is needed. `deploy --build-local` uploads an image built on the operator's machine. `--install-docker` bootstraps Docker on apt and dnf hosts and in `node enroll` cloud-init. See [Architecture](architecture.md#image-artifacts). Each deploy transfers the whole image; layer reuse is in [Later](#later--only-if-needed).
+
+### M5 — Streaming channel between agent and control (mostly done)
+
+Agent-initiated WebSocket with long-poll fallback that wakes agents when work is queued and carries live sessions: non-TTY `tug exec` with stdin, `logs -f`, `ps` and `inspect` ([Architecture](architecture.md#agent-wake-channel), [Observability](observability.md#live-commands)). Remaining, at low priority: dropping the long-poll fallback (the WebSocket plus the periodic poll cover a lost socket), agent and daemon log streams, and terminating the remote process when an exec session is cancelled. PTY allocation and resize for `exec -t` shipped with [M14](#m14--engine-api-agent) through the Engine API.
+
+### M6 — Placement and scheduling (done)
+
+Node labels, `placement.labels`, `replicas: N` spread across matching nodes, `global: true` and `tug scale`. Scheduling stays central and simple: spread by count, skip stale nodes, no bin-packing. See [Compose extensions](compose.md).
+
+### M7 — Shared volumes (removed)
+
+An `x-tug.type: nfs` volume type was built and verified on real hosts, then removed: it mounted a share but could not run the NFS server, so users still prepared the host, and installing and exporting from Tug would make it a host configuration tool. Shared storage is the operator's: S3-compatible object storage, or a file server mounted with plain Compose `driver: local` and `driver_opts` ([Volumes](compose.md#volumes)).
+
+### M8 — Central API advantages (partly done)
+
+Shipped: scoped operator credentials (`read`, `deploy`, `admin`, per project), a GitHub Actions deploy example and credential names in the audit trail ([Access control](access-control.md)). Remaining: outbound webhooks for deployment state changes, failed health checks and backup results (shared with M9 alerting). The read-only web dashboard is dropped to [Later](#later--only-if-needed): `tug status`, `/metrics` and webhooks cover the need.
+
+### M9 — Observability
+
+Make behavior visible without adding required services: event hygiene (done), structured logs, a Prometheus `/metrics` endpoint on control that also carries node and service stats reported by agents (Prometheus cannot scrape nodes behind NAT, which is why agents report through control), a readiness endpoint, sample alert rules and webhooks. Design and status are in [Observability](observability.md). Keep agent-reported stats small; point operators who want depth at cAdvisor or node_exporter deployed as ordinary services.
+
+### M10 — Private-by-default port exposure (done)
+
+A published port such as `3306:3306` listened on every interface of the node, and Docker's iptables rules bypass host firewalls such as ufw, so a database published for cross-node service names was reachable from the internet. Now published ports without a host IP are bound to `127.0.0.1` when only the local ingress dials them, and otherwise to the node's `address` with a `ports.public` warning event when it is public; an explicit host IP (including `0.0.0.0`) is left as written, and `tug doctor --nodes` reports containers listening on a public address. See [Networking](networking.md#published-ports-private-by-default). This was a prerequisite for recommending the scaled topology on an operator-provided private network.
+
+### M11 — Backups, restores and run-once jobs
+
+Working plan: `plan-backups.md`.
+
+The stateful half of principle 5, and something a central control plane does better than distributed schedulers: one schedule, one catalog, retention and restore.
+
+- **Done: generic backups.** `x-tug.backup` marks a service or a volume. Tug has no database support of its own: a service backup is a `command` run in the running container whose output is stored (database recipes are documentation), a volume backup archives its files. Control schedules them as agent commands on the placement node; the agent streams into restic in a helper container, uploading to a destination configured on control (S3-compatible, encrypted before leaving the node), one repository per project. Retention, a catalog, `backup run` and `backup list`, and `backup.*` events for completed, failed, missed and stale runs. See [Backups](backups.md).
+- **Done: restore.** `tug restore` locks the project, stops the services that write to the target, takes a safety backup, restores through the service's own `restore` command or the volume's files, and starts them again; an interrupted restore keeps the writers stopped until resumed or released. A scratch restore of a volume touches nothing live.
+- **Done: `tug run`.** `tug run SERVICE -- cmd` runs a one-off container with the service's image, environment, secrets, network and mounts, returns the exit code and is audited.
+- Not done: `x-tug.release` (a command that runs once per deployment before or after rollout, recorded on the deployment, a failure stopping the rollout), SFTP and postgres-free extras such as a weekly `restic check`/prune, a restore drill, `destination test`, and failed or missed backups as webhooks (M8).
+- Moving a stateful service to another node (`tug move SERVICE --to NODE`) as backup, stop, restore and re-place, since the pieces are the same.
+
+### M12 — Container networking over the mesh (cancelled)
+
+Cancelled when Tug stopped providing a workload overlay. Tug will not build the proposed per-node container subnets, managed resolver or routable container network. The two operator-facing limitations this milestone was meant to solve—stable cross-node name and port for rolling services, and client recreation when `extra_hosts` endpoints change—are tracked independently in [Later](#later--only-if-needed) and do not assume a Tug-owned overlay.
+
+### M13 — Pinned images and one delivery path (done)
+
+Pin image identity for reliable image rollback, and leave one way for Tug-built images to reach a node. Mutable data and external side effects are not rolled back.
+
+- **Digest pinning.** Control resolves each distinct published image tag once per deployment and stores `image@sha256:…` in desired state and canonical Compose before publication. Explicit pins need no lookup. Multi-platform indexes stay intact. Resolution failure leaves current desired state unchanged; rollback and scaling reuse stored pins. Nodes pull by digest from upstream. Private HTTPS registries use the encrypted project secret `TUG_REGISTRY_AUTH` and Engine API authentication, released only to assigned services. See [Control plane](control-plane.md#published-images-and-private-registries).
+- **One built-image delivery path.** Remote and local builds always travel as retained saved archives from control. Built-image registry configuration, `--no-build-upload`, registry pull tokens and their rotate/revoke commands, registry auth hooks and node `docker login` are removed. Each node has one agent credential. CI-built images can be pushed independently and deployed through `image:`.
+
+Each deploy still transfers whole Tug-built images; a control-hosted image store, mirroring of published images and layer reuse are in [Later](#later--only-if-needed) until image size hurts.
+
+### M14 — Engine API agent
+
+The agent stops running the `docker compose` CLI and reconciles containers through the Docker Engine API. The agent already works around Compose: rolling revisions are separate Compose projects with hand-made networks and volumes carrying copied Compose labels, `depends_on` is stripped across them, and secret changes need forced recreates.
+
+- The CLI normalizes the Compose file with `docker compose config` (Compose does interpolation, `extends`, profiles and `env_file`; secret variables are left as sentinels for the agent to substitute) and sends the canonical definition (read from Compose's YAML rendering, since its JSON rendering drops `x-tug`); control validates an explicit allowlist of supported keys, including resource limits; agents never parse Compose YAML. Nodes need only the Docker Engine, not the Compose plugin. The control-plane host keeps using Compose for its own install unless that moves to `docker run`.
+- One reconciler for every service: containers carry `tug.*` labels with a hash of their resolved configuration, including the image reference and the versions of the secrets they use, and are recreated when it changes. `recreate` and `rolling` become two strategies of the same loop. Tug names its own networks, volumes and containers; service names are network aliases.
+- Secrets are substituted into the container environment by the agent; no `.env` file on disk. Health checks stay agent-run, because images often lack the tools a Docker `HEALTHCHECK` would need.
+- `exec -t` with a real PTY and resize, and `logs`, `ps` and `inspect` from the API instead of CLI output.
+
+### M15 — Flexible node roles
+
+- **Control on a runtime node.** Installing control and a node on the same host is supported, so a small setup needs no dedicated control VPS. Control's API is then served through the node's managed Caddy, since both need ports 80 and 443. Keep a direct control port bound to loopback or a private node address as break-glass access, so a broken ingress or agent never locks out the fix path.
+- **Builder nodes.** Builds run on a node labelled `builder`: control queues a build command, the agent downloads the source context from control, builds it and uploads the saved image to control the way `deploy --build-local` does. Without a builder node, control builds, as today. This keeps untrusted Dockerfile builds and their CPU and memory use off the host that holds the secrets database.
+
+## Later / only if needed
+
+- Warm-standby control plane (WAL shipping or periodic backup plus promote), see [Positioning](#positioning).
+- A stable cross-node name and port for rolling services, so clients need not switch to an ingress hostname or discover the serving revision's port. Revisit only with a concrete multi-node need; a solution must work over the operator's network and must not assume a Tug-owned overlay.
+- Avoid recreating client containers when cross-node `extra_hosts` endpoints change, while preserving health-aware names and zero-downtime rollouts. This is an endpoint-update problem, not a reason to recreate a Tug-owned overlay.
+- A host-level agent helper or standalone agent binary, for work a container should not do, such as host mounts outside Docker. Shares the build with the standalone CLI binary ([Standalone binary](standalone-binary.md)).
+- A control-hosted OCI image store, mirroring of published images and an agent-local registry endpoint for layer reuse, once whole-image transfers or upstream rate limits hurt. Mirroring is opt-in; it makes control a bandwidth and disk bottleneck and a dependency for every pull.
+- Standalone Bun-compiled `tug` binaries and a Homebrew tap, so operators need no Node.js.
+- A minimal read-only web dashboard over `/api/v1/status` and `/api/v1/events`.
+- Load-balanced service names and multiple routes per service, wildcard certificates.
+- Managed DNS convenience domain for ingress, similar to uncloud's `*.uncloud.run`.
+
+## Product decisions and reasoning
+
+Tug serves a trusted operator or small trusted team running Compose applications on one or a few ordinary VPSes. The primary topology is an application and its state on one node, including several independent applications on one node; additional nodes first add capacity and placement for independent applications. A scaled single application is a later, demand-driven use case. Judge work by whether it makes installing, operating and recovering those applications easier and safer, not by feature parity or the number of orchestration milestones completed.
+
+The control plane earns its cost through understandable state, reliable deployments, backup visibility and recovery. Prefer a proven, repeatable recovery journey over additional scheduling, networking or infrastructure features. Restic is the backup repository mechanism; Tug's responsibility is to make backup status, credentials, off-host copies and actual restore outcomes understandable. Do not set recovery time or data-loss targets until a representative exercise measures them.
+
+Deployment authors are trusted infrastructure operators. Project-scoped credentials are operational least-privilege controls that reduce accidental access and limit a compromised credential's API authority; they do not isolate mutually untrusted code, shared kernels, application networks, builds or workload secrets. An operator who can deploy or execute code must be trusted with the workload's secrets and the nodes they can select. Use separate hosts or control planes for mutually untrusted authors. Continue fixing concrete authorization, confidentiality and data-safety defects, but do not add a complex tenant-isolation policy without an explicit user need and a coherent end-to-end threat model. An earlier project node/hostname grants and host-feature restrictions experiment remains separate from the recovery-first branch; evaluate it against this contract before adopting it. It is not a product decision to support hostile tenants.
+
+Every subsystem Tug owns carries maintenance and failure costs. Keep the outbound agent while its NAT and unattended reconciliation benefits justify it, but a bounded SSH-controller experiment is legitimate if it preserves the operational guarantees users need. Compare control-hosted image archives with an existing registry on deployment, recovery, storage, security and operational cost before expanding either path. Keep the Compose subset explicit and predictable, use the operator's private network for cross-node workloads, and prefer simple placement over a general scheduler. Remove or narrow an existing feature if its cost exceeds its demonstrated value.
+
+Claims must match verified behavior. Image rollback does not undo data, schema or external side effects; private port binding depends on the configured address and network; scoped credentials are not tenant isolation. Deployment success needs a documented definition that distinguishes node health from ingress applying the intended revision and, if relevant, an external probe. Each claim needs tests at the cheapest useful layer and real-host evidence for the end-to-end operating journey.
+
+## Sequencing
+
+1. Review the current security work against the trusted-operator contract. Preserve independently useful data-safety and confidentiality fixes, assess grants and restrictions on their own merits, and do not claim a tenant-isolation boundary.
+2. Prove recovery on disposable hosts using published artifacts: install WordPress, deploy a meaningful change, back up database, uploads, control state, credentials and image artifacts; recover a lost workload host and then a lost control plane onto fresh storage, reconnect the fleet and verify the site. Record elapsed time, data-loss window, manual actions and gaps; fix and retest failures. The control and workload backups are separate and both must be recoverable.
+3. Improve the ordinary operator experience: clear human-readable status and errors, backup freshness and actionable alerts, and explicit deployment progress and success semantics including ingress acknowledgement where promised.
+4. Harden the actual production path: bounded work and storage, safe build inputs, credential handling, and repeatable release, install and upgrade checks.
+5. Reconsider flexible node roles, release hooks, scaled WordPress, additional networking, dashboards and stronger isolation only for a demonstrated need. Co-locating control is attractive for small installations but requires an independently tested off-host recovery path and an explicit host trust model.
+
+M10 is done; M12 is cancelled and its residual limitations are deferred. M14 and much of M11 are implemented; the current priority is verifying the complete operating journey rather than repeating those milestones. The milestone descriptions and checklist record existing work and candidates, not commitments that override this sequence.
+
+Guided onboarding (`tug setup`, an inferred node name, and `node add` offering Docker, the inter-node address and ingress in a terminal) answers the first-run friction recorded in [real-host runs](real-host-evaluation.md). It prompts only in a terminal outside CI; scripts pass the same choices as flags. The prompts have unit and CLI tests but have not yet been exercised on real hosts, so the next real-host run should follow the README quick start as written.
+
+Historical validation: [real-host runs](real-host-evaluation.md) on two Ubuntu 24.04 hosts covered SSH install with the Docker bootstrap, the now-removed WireGuard mesh, an agent behind bridge NAT, a publicly trusted Let's Encrypt certificate from the managed Caddy, cross-node service names as then implemented, remote and `--build-local` deploys, rollback from a retained image, a rolling rollout through the public ingress with no failed requests (over 2,200 sampled), a failing version rejected while the old revision kept serving, automatic rollback, and on freshly wiped hosts label placement, `global`, `tug scale`, `ps`/`inspect`/`exec`/`logs -f` and a cloud-init enrolled node behind a deny-all-inbound firewall ([M6, M7 and enrollment](real-host-evaluation.md#m6-m7-and-enrollment-on-fresh-hosts)). The mesh and mesh-based name results are retained as history, not evidence for the current plain-address network path. Not yet validated on real hosts: the current plain-address cross-node service-name, ingress and private-port path, a node behind real NAT, certificate renewal, arm64 nodes and dnf-based hosts.
+
+The test layers, the metrics milestones are held to and the backlog of mocked tests are in [Testing](testing.md).
+
+## Checklist
+
+- [x] M1 TypeScript adoption merged
+- [x] M2 networking, service names, ingress/TLS: default-on cross-node service names over node `address` values, `x-tug.ingress`, managed Caddy with `auto`/`custom`/`off` TLS ([Networking](networking.md))
+- [x] M3 rolling deploys with ingress upstream switching, `x-tug.rollout` (`strategy`, `drain`; `maxSurge` not implemented), opt-in automatic rollback, run on real hosts through the public ingress
+- [x] M4 images stored on control, retention tied to rollback history, agent image pruning, `deploy --build-local`; M13 makes this the only built-image delivery path
+- [x] M4 opt-in Docker install for `node add`/`control setup` (verified on Ubuntu 24.04) and `node enroll --install-docker` cloud-init (verified on a cloud-init host)
+- [ ] M4 dnf bootstrap verified on real Fedora and Rocky/AlmaLinux hosts
+- [x] M5 wake channel, faster convergence, non-TTY `exec` with stdin, `logs -f`, `ps`, `inspect`
+- [ ] M5 agent/daemon log streams
+- [ ] M5 long-poll wake fallback removed; WebSocket plus periodic poll
+- [ ] M5 remote process termination when an exec session is cancelled or disconnected
+- [x] M6 node labels, label placement, replicas spread, `global` mode, `scale` command, verified on real hosts
+- [x] M7 `nfs` volume type, removed: shared storage is the operator's (S3-compatible storage or `driver_opts`)
+- [x] M8 scoped operator credentials, CI deploy example, credential names in the audit trail ([Access control](access-control.md))
+- [ ] M8 outbound webhooks
+- [x] M9 event hygiene ([Architecture](architecture.md#events)) and observability design ([Observability](observability.md))
+- [ ] M9 structured logs (`TUG_LOG_FORMAT`, `TUG_LOG_LEVEL`)
+- [ ] M9 `GET /metrics` on control (Prometheus text, `read` credential), with agent-reported node and service stats
+- [ ] M9 an authenticated `/api/v1/readyz`, sample alert rules, and the metrics documentation test gate
+- [x] M10 published ports bound to the node address or loopback by default; `doctor` reports public listeners
+- [x] M11 spec written
+- [x] M11 `x-tug.backup` (generic: a command in a service, or a volume's files), scheduled backups to S3-compatible storage with restic, catalog, `backup run`, `backup list`, `restore` (verified against real Docker and on two real hosts with an S3-compatible server and a scheduled run: [Real-host evaluation](real-host-evaluation.md#generic-backups-restore-and-tug-run-on-fresh-hosts-1-november-2026); not yet against AWS S3)
+- [x] M11 `tug run` one-off containers
+- [ ] M11 `x-tug.release` run-once hooks
+- [ ] M11 weekly maintenance (`restic check`, prune), restore drill, `backup-destination-test`, SFTP
+- [x] M11 `restore` waits for the restored writers to be serving again (five minutes, then it finishes with a warning)
+- [ ] M11 list and remove scratch-restore volumes, and expire safety snapshots (they accumulate for ever and only `restic forget` removes them)
+- [x] M11 `backup list --check`: exit nonzero when any target is stale, failed or missed, as an interim for cron and monitors until webhooks (M8) exist
+- [x] M11 `memory` under `x-tug.backup` for the restic helper's memory limit
+- [x] A failed health check says when the agent runs outside a container (`doctor` does not check it separately)
+- [x] `TUG_ROLE=agent` works in the runtime image: it no longer sets a default command
+- [ ] A `tug run` option to start without the service's volumes (`--no-volumes`), and a decision on whether it should take the project lock
+- [ ] M11 `tug move` for stateful services
+- [x] M12 cancelled with the Tug-owned overlay; stable rolling-service name/port and endpoint-change recreation moved to [Later](#later--only-if-needed)
+- [x] M13 stopgap: published images pulled only when a deployment revision is new to the node
+- [x] M13 digest pinning for every `image:` at deploy time, used by rollback
+- [x] M13 bring-your-own registry, registry pull tokens and `docker login` removed; one node credential
+- [x] M14 Compose normalized on the operator machine; allowlist validation on control
+- [x] M14 Engine API reconciler with config-hash recreation, `recreate` and `rolling` strategies, no `.env` on disk
+- [x] M14 `exec -t` with PTY and resize; `logs`, `ps`, `inspect` and ingress on the Engine API
+- [x] M14 Compose plugin no longer required on nodes (the runtime image no longer ships it; the control-plane host still uses Compose to run the control container)
+- [x] One mutual-TLS port for operators, CI and nodes, certificates from control's private CA, immutable node IDs, and `tug doctor` certificate expiry and clock skew ([Control plane](control-plane.md#authentication-and-pki), [Nodes](nodes.md))
+- [ ] M15 control on a runtime node behind the managed Caddy, with a break-glass control port
+- [ ] M15 builds on `builder`-labelled nodes, uploaded to control as saved images
+- [ ] Scaled WordPress reference example (code in image, shared uploads) validated on real hosts
+- [ ] Mocked tests from the [Testing](testing.md#backlog-of-mocked-tests-proposed) backlog: revisions started per deployment and old-schema database behavior
+- [x] M13 uncloud release baseline rechecked and updated in [Positioning](#positioning); repeat at the start of each milestone
+- [x] Guided onboarding: grouped `tug <subject> <verb>` commands, `tug setup`, name inference and `node add` prompts that never run without a terminal
+- [ ] Follow the README quick start, prompts included, on fresh hosts and record the result in [real-host evaluation](real-host-evaluation.md)
+- [ ] Each milestone: tests added, README.md and docs updated, its own plan file folded into docs and deleted when done