@wpelevator/tug 0.11.0 → 0.13.0

package/docs/observability.md

@@ -0,0 +1,213 @@
+# Observability
+
+**Status: mostly proposed.** Tug today has an audit trail, a few read commands and a certificate and connectivity check, and nothing that measures CPU, memory, requests or latency. Sections marked **Proposed** describe a setup that is not built; the milestone is [M9](roadmap.md#m9--observability). Nothing marked proposed works yet, and none of the names, flags or environment variables in those sections exist in the code.
+
+The aim is the smallest setup that answers the day-to-day questions: is every node reporting, did the last deployment succeed, did an automatic rollback happen, will a certificate expire, is the control database growing. It follows the shape of the rest of Tug: one central control plane, agents that only dial out, and no required extra services.
+
+## What exists today
+
+Verified against the code on `main`.
+
+| Capability | What it does | Limits |
+| --- | --- | --- |
+| Audit trail | Deployments and events record the acting credential, and `system:auto-rollback` for automatic rollbacks. See [Access control](access-control.md). | Answers who acted, not how the system behaves |
+| `tug status` | Summarizes deployments, errors and node reports; `--json` prints `GET /api/v1/status` as JSON: projects, the 20 latest deployments, nodes with `status` and `last_seen_at`, and per-node project status including health, rollout state, port and rollout error. | Node health is a last observation, not a live probe; a node is marked `stale` only when this endpoint is called, using `TUG_STALE_SECONDS` (default 60). The current deployment can be outside the 20 returned rows. |
+| `tug events` | Lists events from `GET /api/v1/events`, newest first, at most 500 per call, 100 by default. | Agents report on change only, see [Events](#events) |
+| `tug doctor` | Local checks (Node.js, Docker, the Compose plugin `deploy` needs), control checks (SQLite, CA and server certificate expiry, each node's certificate expiry and connectivity, each online node's observed clock skew, master key, Docker, ingress state) and, with `--nodes` or `--node`, host diagnostics through each agent. It waits for those results. Clock skew is unknown, not zero, for a node that is offline. | A point-in-time check, not a time series |
+| `tug logs <service>` | Prints a service log tail; `--follow` streams logs through the live channel. | Following logs requires an active WebSocket session. |
+| Reachability | Control has no unauthenticated route, not even a health check: every request needs a client certificate. Monitor TCP reachability of the mTLS port for liveness, and use an authenticated read (`tug doctor`, `tug status`) for readiness. | A TCP check proves the listener accepts connections, not that the database works |
+| Process logs | Control and agent write plain `console.log` and `console.error` text. | No timestamps, levels, structure or request ids |
+
+What does not exist: a `/metrics` endpoint, CPU, memory, disk or restart data, request, build or poll metrics, structured or leveled logs, alerting, outbound webhooks (M8, unchecked) and a dashboard (M8, unchecked). The managed Caddy ingress does not enable Caddy's metrics: the generated Caddyfile has no global options beyond the optional ACME email.
+
+### Events
+
+Agents report lifecycle events for the recreated services of a project (`service.pulling`, `starting`, `health_checking` and `serving`) only when something about the project changes: a new desired revision, a different Compose file rendered for the node, or different secret versions. A failed apply produces one `service.failed` event (level `error`) naming the service whose check failed and why, retries of the same failure add nothing, and recovery produces one `service.serving` event. Agents still apply and health-check the project on every poll, so only the events are quiet. A `ports.public` event (level `warn`) says published ports were bound to a public node address ([Networking](networking.md#published-ports-private-by-default)); it is sent once per change, not per poll. Control deletes events older than `TUG_EVENT_RETENTION_DAYS` (default 30) and keeps at most `TUG_EVENT_MAX_ROWS` (default 100000), at startup and then at most once an hour while agents report; `0` turns a limit off. See [Architecture](architecture.md#events) and [Configuration](configuration.md#control-plane).
+
+Before [PR #13](https://github.com/wpelevator/tug/pull/13) agents emitted those four events for every non-rolling service on every poll, whether or not anything changed. On a real control plane with nothing deploying that was about 71 events a minute, roughly 100,000 a day, nothing deleted them, and the default `tug events` view (the last 100) covered about a minute and a half and hid real events such as an automatic rollback.
+
+Known limit: a container that crashes and is restarted by the next poll produces no event unless its health check fails, so crash loops would show up only in the restart counts proposed under [node and service stats](#node-and-service-stats-reported-by-agents).
+
+## Proposed
+
+Everything below is **Proposed** unless a status line says otherwise.
+
+| Piece | Section | Status |
+| --- | --- | --- |
+| Structured logs | [Structured logs](#structured-logs) | Proposed |
+| Metrics on control | [Metrics on control](#metrics-on-control) | Proposed |
+| Node and service stats from agents | [Node and service stats reported by agents](#node-and-service-stats-reported-by-agents) | Proposed |
+| Readiness endpoint | [Health endpoints](#health-endpoints) | Proposed |
+| Ingress metrics | [Ingress metrics](#ingress-metrics) | Proposed, with open questions |
+| Sample alerts and webhooks | [Alerting](#alerting) | Proposed |
+| Live commands | [Live commands](#live-commands) | Partly built in M5 |
+| Metrics test gate | [Test gate](#test-gate) | Proposed |
+
+### Structured logs
+
+Status: proposed.
+
+Control and agent would write one JSON object per line with `time`, `level`, `component`, `msg` and context fields where they apply: `requestId`, `node`, `project`, `service`, `deploymentId`. Two environment variables would select the output: `TUG_LOG_FORMAT=json|text` and `TUG_LOG_LEVEL` (`debug`, `info`, `warn`, `error`). Text would stay the default so existing installs read the same, and JSON is meant for `docker logs` piped to whatever the operator already uses.
+
+Credential private keys, join keys and secret values are never logged, and the logger would redact known field names as a second line of defence. The CLI keeps its human-readable output; only the long-running processes change. Every existing `console.log` and `console.error` in `src/control.ts` and `src/agent.ts` would move to the logger.
+
+### Metrics on control
+
+Status: proposed.
+
+Control would serve `GET /metrics` in the Prometheus text format. It needs the `read` role from [Access control](access-control.md#roles), so Prometheus would present the client certificate and key of a `read` operator credential (`tls_config` with `cert_file`, `key_file` and `ca_file` extracted from the credential file); a credential limited to projects would see only those projects' series. Adding the route means adding it to `ROUTE_ROLES` in `src/credentials.ts` and to the endpoint table, which the existing test already requires.
+
+Naming follows the Prometheus conventions: a `tug_` prefix, base units (`_seconds`, `_bytes`), `_total` for counters. Labels are bounded: project, node and service names are operator-defined and few; route labels use the route pattern (`/api/v1/deployments/:id`), never the raw path; deployment IDs, command IDs, versions and error messages are never labels.
+
+| Metric | Type | Labels | Meaning |
+| --- | --- | --- | --- |
+| `tug_build_info` | gauge | `version` | Constant 1; the running control version |
+| `tug_deployments` | gauge | `project`, `state` | Deployments currently in each state (`uploading`, `building`, `pending_dependencies`, `rolling_out`, `succeeded`, `failed`, `build_failed`, `cancelled`) |
+| `tug_deployment_duration_seconds` | histogram | `project`, `outcome` | Time from creation to a terminal state |
+| `tug_rollout_outcomes_total` | counter | `project`, `outcome` | Rollout results: `rolled_out`, `rejected` (a candidate failed its health check while the old revision kept serving), `auto_rollback` |
+| `tug_nodes` | gauge | `status` | Registered nodes by `online` or `stale`, computed from `last_seen_at` at scrape time rather than stored |
+| `tug_node_poll_age_seconds` | gauge | `node` | Seconds since the node last reported |
+| `tug_api_requests_total` | counter | `route`, `method`, `status_class` | API requests by route pattern and status class (`2xx`, `4xx`, `5xx`) |
+| `tug_api_request_duration_seconds` | histogram | `route` | API request latency |
+| `tug_builds_total` | counter | `outcome` | Builds by result (`complete`, `failed`) |
+| `tug_build_duration_seconds` | histogram | none | Remote build duration |
+| `tug_image_archive_bytes` | gauge | none | Total size of retained image archives under the data directory |
+| `tug_db_size_bytes` | gauge | none | Size of the control SQLite database file |
+| `tug_events_rows` | gauge | none | Rows in the `events` table |
+| `tug_project_locks` | gauge | none | Project operation locks currently held |
+| `tug_writer_queue_depth` | gauge | none | Operations waiting for the single writer |
+
+Gauges that describe state (`tug_deployments`, `tug_nodes`, `tug_events_rows`, `tug_project_locks`) would be computed from the database at scrape time, so they stay correct across control restarts. Counters and histograms live in process memory and reset on restart, which Prometheus handles as a counter reset.
+
+### Node and service stats reported by agents
+
+Status: proposed.
+
+Agents already send a status report on every poll. The proposal adds a `stats` block to that report and has control expose it on the same `/metrics` endpoint, so one scrape target covers everything:
+
+| Metric | Type | Labels | Meaning |
+| --- | --- | --- | --- |
+| `tug_service_cpu_seconds_total` | counter | `node`, `project`, `service` | CPU time used by the service's containers |
+| `tug_service_memory_bytes` | gauge | `node`, `project`, `service` | Memory in use |
+| `tug_service_up` | gauge | `node`, `project`, `service` | 1 when the container is running, 0 otherwise |
+| `tug_service_restarts_total` | counter | `node`, `project`, `service` | Container restarts since the container was created |
+| `tug_node_disk_free_bytes` | gauge | `node`, `path` | Free space on the Docker data and Tug state paths |
+| `tug_node_load1` | gauge | `node` | One-minute load average |
+
+The agent would read these from `docker stats --no-stream` and `docker inspect`, and control would drop series for services that no longer exist. A node that stops reporting keeps its last values, so alerts use `tug_node_poll_age_seconds` to tell fresh data from stale data.
+
+Why report through control: agents sit behind NAT and only dial out, so Prometheus often cannot reach them. Reporting through the existing authenticated channel needs no inbound port, no extra credential and no firewall change, and adds one scrape target regardless of node count. The cost is resolution: values are as fresh as the poll interval (`TUG_POLL_INTERVAL_MS`), and the status report grows with the number of services.
+
+The alternative is scraping an exporter on each agent directly over the operator's private network from [Networking](networking.md#bring-your-own-network). It gives finer resolution and keeps control out of the data path, but it requires a listener on every node, firewall rules that admit only Prometheus and a service-discovery list. It does not work where Prometheus cannot route to the node addresses, including nodes behind NAT. If both are ever wanted, the metric names above can be reused so dashboards do not change.
+
+### Health endpoints
+
+Status: proposed.
+
+Control has no unauthenticated route (there is no `/healthz`), so liveness is TCP reachability of the mTLS port and the container restart policy. An authenticated `GET /api/v1/readyz` with the `read` role would return 200 only when control can do its job: it can read the database, and the writer queue is not stuck (the operation at the head of the queue has not been running longer than a limit). Otherwise it returns 503 with the names of the failing checks and nothing more. An external uptime check would present a `read` credential.
+
+### Ingress metrics
+
+Status: proposed, with open questions.
+
+The managed Caddy would enable its metrics with the global option `servers { metrics }` in the generated Caddyfile (see `renderCaddyfile` in `src/ingress.ts`), served by Caddy's admin endpoint, which listens on `localhost:2019` inside the container by default. That gives request counts by status class and request duration histograms. To reach the same scrape target, the agent on the ingress node would read the admin metrics and include a filtered subset (request counts, status classes, latency buckets, certificate expiry per host) in its status report, and control would re-export it as `tug_ingress_*` series labelled by `node` and `host`.
+
+Open questions:
+
+- Whether re-exporting parsed Caddy metrics through the status report is worth the complexity, compared with publishing Caddy's admin metrics port on a private node address, firewalling it to Prometheus and scraping it directly.
+- Which certificate expiry signal Caddy exposes reliably across versions; the alert below assumes a `tug_ingress_certificate_expiry_timestamp_seconds` gauge that may have to be derived from the certificate files.
+- Label cardinality: one `host` label per route is bounded by the number of routes, but per-upstream labels are not planned.
+
+### Alerting
+
+Status: proposed.
+
+Tug would not bundle Prometheus or Alertmanager. A sample scrape configuration:
+
+```yaml
+scrape_configs:
+  - job_name: tug
+    metrics_path: /metrics
+    scheme: https
+    tls_config:
+      ca_file: /etc/prometheus/tug-ca.pem
+      cert_file: /etc/prometheus/tug-read.crt
+      key_file: /etc/prometheus/tug-read.key
+    static_configs:
+      - targets: ['control.example.com']
+```
+
+Sample alert rules, written against the proposed metric names:
+
+```yaml
+groups:
+  - name: tug
+    rules:
+      - alert: TugNodeStale
+        expr: tug_node_poll_age_seconds > 120
+        for: 2m
+        annotations: { summary: "Node {{ $labels.node }} has not reported for over two minutes" }
+      - alert: TugDeploymentFailed
+        expr: increase(tug_deployment_duration_seconds_count{outcome=~"failed|build_failed"}[10m]) > 0
+        annotations: { summary: "A deployment of {{ $labels.project }} failed" }
+      - alert: TugAutomaticRollback
+        expr: increase(tug_rollout_outcomes_total{outcome="auto_rollback"}[10m]) > 0
+        annotations: { summary: "{{ $labels.project }} was rolled back automatically" }
+      - alert: TugCertificateExpiresSoon
+        expr: tug_ingress_certificate_expiry_timestamp_seconds - time() < 14 * 86400
+        annotations: { summary: "Certificate for {{ $labels.host }} expires in under 14 days" }
+      - alert: TugDatabaseGrowing
+        expr: predict_linear(tug_db_size_bytes[6h], 7 * 86400) > 5 * 1024 ^ 3
+        annotations: { summary: "Control database is on track to pass 5 GiB within a week" }
+      - alert: TugAgentPollAgeHigh
+        expr: tug_node_poll_age_seconds > 30
+        for: 10m
+        annotations: { summary: "Node {{ $labels.node }} polls much slower than its interval" }
+```
+
+Thresholds are starting points. The last rule assumes the default poll interval and should be set from `TUG_POLL_INTERVAL_MS`.
+
+Outbound webhooks (M8) are the lightweight alternative for operators without Prometheus: control would POST a signed JSON body to configured URLs. The events worth sending are a deployment reaching `succeeded`, `failed`, `build_failed` or `cancelled`, a rollout candidate rejected, an automatic rollback, a node going stale and recovering, and an ingress node reporting `failed`. Webhooks send state changes, never metrics, and depend on event hygiene so that an event always means something happened.
+
+### Live commands
+
+Status: non-TTY container commands built in M5; agent/daemon logs and `top` remain proposed.
+
+`tug logs <service>` keeps its durable tail through the command queue, while `-f` follows the current container over the live channel. `tug exec`, `tug run`, `tug ps` and `tug inspect <service>` use that channel too. Agent and daemon log streams are still proposed for troubleshooting. These commands answer what is happening right now on a node and complement metrics, which answer what has been happening. An optional `tug top` would render the agent-reported stats as a refreshing table of CPU, memory and restarts per service, reading `GET /metrics` or a read-only JSON view rather than a new data path.
+
+### Test gate
+
+Status: proposed.
+
+Modelled on madplay's `@magplay/observability` package: metrics sit behind a small injectable interface whose default is a no-op, so unit tests of control and agent logic need no registry; a small closed outcome taxonomy (`success`, `failure`, `timeout`, `rejected`, `cancelled`) with one helper that maps an error to an outcome; and a CI check (`test:observability`) that fails when the code and the documentation disagree.
+
+For Tug the equivalent check would assert:
+
+- every metric registered in code appears in this page's metric tables with the same name, type and label set, and every row in the tables is registered in code;
+- every label name is on an allow-list, and none is `id`, `version`, `error`, `path` or `message`, so cardinality stays bounded;
+- `GET /metrics` with a certificate for an unknown key returns 401 and with a `read` credential returns 200, and `ROUTE_ROLES` lists the route;
+- a scrape after a credential is created and used contains no credential private key, join key or secret value;
+- with the no-op default, the existing control and agent tests behave unchanged.
+
+## Rollout order
+
+1. Event hygiene (done, [PR #13](https://github.com/wpelevator/tug/pull/13)). Independent of everything else.
+2. Structured logs and the `/metrics` endpoint on control with the control-side metrics and the readiness endpoint, plus the test gate so later metrics cannot ship undocumented.
+3. Agent-reported node and service stats, then ingress metrics once the open questions are settled.
+4. Sample alert rules (they need steps 2 and 3) and outbound webhooks.
+
+Dependencies: the remaining agent/daemon live log streams and TTY backend extend M5, while webhooks and the dashboard are M8 items. Steps 1 to 3 need neither.
+
+## Open questions and non-goals
+
+Open questions:
+
+- Where the first dashboard JSON would live: in this repository (for example a `dashboards/` directory) or published separately, and whether the M8 read-only web dashboard makes it unnecessary.
+- Whether agents should send stats on every poll or on a slower interval of their own.
+- Whether `/metrics` should accept the `root` admin credential as well as named `read` credentials (an admin passes every role check today).
+- How the ingress metrics reach control (see [Ingress metrics](#ingress-metrics)).
+- Whether control should keep a short history of stats for `tug top` and a dashboard without Prometheus.
+
+Non-goals: a log aggregation system (JSON on stdout is the interface), distributed tracing, and a bundled Prometheus, Alertmanager or Grafana. Tug would expose metrics and document how to scrape them; running the monitoring stack is the operator's choice.
+
+See also [Architecture](architecture.md), [Control plane](control-plane.md), [Networking](networking.md), [Testing](testing.md) and the [Roadmap](roadmap.md).

package/docs/real-host-evaluation.md

@@ -0,0 +1,224 @@
+# Real-host evaluation, 31 October 2026
+
+Two WordPress sites on the existing test hosts, using the current CLI against the control plane already running there. This is an operator trial, not a passing CI job. The roadmap's WordPress picture was the target. What the tool can deploy today is a smaller thing, and the gap shows up as soon as you follow a normal Compose file.
+
+## Result
+
+Both sites are installed and serving over public HTTPS.
+
+**Version context.** The trial ran on the earlier runtime. Its control API was plain HTTP on port 8080 with bearer tokens, `/healthz` was unauthenticated, agents applied each project by running `docker compose up` on the node, and cross-node names were called service DNS. The current implementation differs: control serves a single mutual-TLS port (`8443`) with certificates from its own CA and no unauthenticated route, agents use the Docker Engine API with no Compose plugin on nodes, and the feature is called cross-node service names (hosts-file entries). Statements below about HTTP, port 8080, tokens, Compose behavior and `/healthz` describe the trial hosts, not the current product; [Control plane](control-plane.md), [Nodes](nodes.md) and [Architecture](architecture.md) describe the current one. All references to WireGuard, mesh membership, overlay addresses, `--mesh`, `TUG_MESH` and `node-rotate-wg-key` likewise describe the removed Tug-owned mesh implementation, retained only as historical evidence; they are not current capabilities or tests that still apply. Findings are kept as evidence, not rewritten.
+
+This document preserves what the installed `m4-amd64` hosts did during the trial. Since then, current `main` has made `tug deploy` include the node's apply error, made `doctor --node` wait for its queued result, kept lifecycle events quiet after the first change, and added an outbound agent wake channel so deployments and queued commands do not normally wait for the next periodic poll. The remaining findings are still open unless a paragraph says otherwise.
+
+| Site | URL | Node | Host | Cert |
+|---|---|---|---|---|
+| Press A | https://a.178.128.250.108.sslip.io | `app` | 142.93.140.20 | Let's Encrypt, CN=YE1, through 29 Jan 2027 |
+| Press B | https://b.178.128.250.108.sslip.io | `edge` | 178.128.250.108 | Let's Encrypt, CN=YE2, through 29 Jan 2027 |
+
+Each site is `wordpress:6.8-apache` plus `mariadb:11.4` on the same node, with a persistent volume for `/var/www/html` and another for `/var/lib/mysql`. Ingress is the managed Caddy on `edge`. HTTP returns 308 to HTTPS. The homepages return 200 with the site title. The admin user is `editor`. The password is only in `/tmp/tug-wp-eval/credentials.txt` on the operator machine, mode `0600`, and is not in this repo.
+
+The second `wp1` deploy, images already pulled, reached `succeeded` in 20 seconds. `wp2`, including the image pull onto the other host, took 57 seconds. After both were up, `app` had 263 MB available of 961 MB and `edge` had 331 MB. MariaDB RSS was 85 MB and 116 MB only because the compose file sets `--innodb-buffer-pool-size=48M`. WordPress RSS was about 120 MB on each host.
+
+This did not exercise the original specification's topology (plan.md, since removed in favor of the roadmap's [reference topology](roadmap.md#reference-topology)): three PHP-FPM nodes, a front node speaking FastCGI, a separate database node, Memcached, and an NFS docroot. Those needed M6 and M7, which have since been implemented but not validated on real hosts. With two 1 GB hosts, that layout would not fit even if the fields existed.
+
+## What I followed, and what I actually did
+
+The README quick start says install control over SSH, `tug init` against an HTTPS URL, set a secret, `node add`, deploy, then use `status`, `events`, `logs`, `restart` and `doctor`. The hosts were already past install. Control is bound to `10.110.0.3:8080`, not the public address. Port 8080 on `142.93.140.20` refuses connections. The working operator path is an SSH tunnel to `127.0.0.1:18082` and a config file outside the repo (`/tmp/tug-real-m4/config.json`, project still set to `realcheck`). That tunnel pattern is one example buried in the CLI and nodes docs. It is not the quick start.
+
+The agents are `tug:m4-amd64`, not `ghcr.io/wpelevator/tug:0.2.0`. `tug doctor` on the laptop reports Docker Compose v5.6.1. The same command's control check reports v2.32.4, which is the plugin inside the runtime image. `tug status` does not show the image tag or the CLI version the node is running. An operator comparing those two Compose versions will think the install is inconsistent. It is, but status will not say so.
+
+I did not reinstall from the published image. The cluster was already serving, and a rebuild would have tested the installer again rather than the application path. The version skew is still invisible.
+
+## What worked
+
+- Two projects on one ingress node. Caddy picked up `a.178.128.250.108.sslip.io` and `b.178.128.250.108.sslip.io` without dropping the existing `178.128.250.108.sslip.io` route.
+- Certificates issued with `TUG_ACME_EMAIL` unset. DNS for `*.178.128.250.108.sslip.io` is the whole trick, and it is not mentioned in the docs.
+- `secret set` from stdin, before the project row existed. Values stayed out of argv. `secret list` shows names and versions only.
+- Placement, persistent volume records, and `volume list` naming the node.
+- Health checks: TCP to MariaDB, HTTP to WordPress. Once the compose file was one Compose would accept, both deploys converged without raising the 300 second default timeout.
+- Binding the app port to the overlay address on `app` (`10.210.0.1:8080:80`) and to `127.0.0.1:8080:80` on `edge` kept port 8080 off the public interfaces. Caddy still reached both sites.
+
+## Failures and friction
+
+### The first deploy failed, and the CLI hid the reason
+
+The first compose file used the dependency every WordPress example uses:
+
+```yaml
+depends_on:
+  db:
+    condition: service_healthy
+```
+
+MariaDB 11.4 has no Docker `HEALTHCHECK`. `x-tug.health` is not one either. Tug strips `x-tug` before writing the node compose file, then runs `docker compose up`. Compose refused to start WordPress:
+
+```text
+dependency failed to start: container wp1-db-1 has no healthcheck configured
+```
+
+`tug deploy` printed `deployment dep_73508669cffa774f failed: one or more assigned nodes reported failed health` and exited in 50 seconds. That is shorter than the health retries in the file, so it was not a timeout. `tug events --project wp1` showed `service.pulling` and `service.starting`, then nothing about the failure. `tug logs wordpress` printed nothing, because the container was still `Created`. `tug logs db` showed MariaDB ready for connections, which sends you looking at WordPress. The compose error exists only inside the `status_json` string on the node status row. I got it by SSH, then by parsing `tug status`.
+
+Both visibility gaps are closed on current `main` and not on these hosts. [PR #13](https://github.com/wpelevator/tug/pull/13) emits one `service.failed` event with the apply error, and retries of the same failure stay quiet. [PR #17](https://github.com/wpelevator/tug/pull/17) also includes the reporting node's apply or rollout error in the deployment failure that `tug deploy` prints.
+
+A plain `depends_on: [db]` worked. The WordPress image already waits for the database. The docs say `depends_on` is a hint and that readiness comes from `x-tug.health`. They do not say that `condition: service_healthy` fails the deploy, or that you must add a Docker `healthcheck:` block yourself if you want Compose to wait.
+
+### JSON is the interface
+
+`tug deploy` on success prints the deployment row and the full desired state, including the compose file. The word `succeeded` is a field you scroll to. There is no human summary and no `--quiet`. The docs say there is no global `--json` because JSON is already the default. That is honest, and it is also why a failed deploy is hard to read: the one string a person needs is not in the error line.
+
+`tug status` returns database rows. Health is a snake_case column plus a stringified JSON blob (`status_json`) that holds the camelCase agent report. Finding out that `wp1` is serving means parsing a string inside a string. CLI docs say new structured results use camelCase and that unchanged API responses may keep database names. Status is the command an operator runs first, and it is the unchanged dump.
+
+`doctor --node edge` returned `"ok": true` and a command id. It does not wait. `restart` and `node-rotate-wg-key` do. The command description says "run" diagnostics. The nodes doc says "queues". Both are true, and the result never comes back to the terminal.
+
+### The config project is a trap
+
+The tunnel config's `project` is `realcheck`. `tug logs wordpress`, without `--project`, posted to that project and returned `404 {"error":"service not found"}`, then told me to run `--help`. It did not say which project it used. `events` is project-scoped the same way. `status` is control-wide. After `init`, every later command inherits whatever project you set, and the error looks like a missing service.
+
+### Ingress bind addresses are insider knowledge
+
+Caddy dials `127.0.0.1` for a service on the ingress node and the overlay address for any other node (`src/ingress.ts`). A published port of `8080:80` listens on `0.0.0.0`, so the app is also on the public IP, next to Caddy, with no TLS. Binding the overlay IP on the app node and `127.0.0.1` on the edge node is what makes the ingress work without that exposure. The networking doc says the edge uses the published host port. It does not say which address that port has to be on, or that the overlay IP is something you have to look up and paste into the compose file. If the node is re-registered and gets a new overlay address, the compose file is wrong and nothing in `tug deploy` will say so before Compose tries the bind.
+
+### Cross-node service names (then called service DNS) do not get you a remote database
+
+Names injected with `extra_hosts` point at the node address, and the client must use the published host port. A MariaDB that is not published is reachable only on the Compose network, which means the same node. Publishing `3306` so WordPress on the other node can use the name puts the database on an address the mesh can route to. Binding it to the overlay IP only is the same undocumented trick as the app port. The generated `wp2` compose file had no `extra_hosts`, which is correct: both of its services are on `edge`.
+
+So the deployable WordPress shape is app and database on one node, one project per site. The original specification's shape, app nodes talking to a database node over a service name, is not the shape this tool deploys. Copying that example fails for more than missing nodes. It uses `domains`, `upstream`, `upstreamProtocol`, `sticky`, and `type: shared` / `driver: nfs`. Current validation accepts `placement`, `replicas` (must equal the node count), `health`, `ingress`, and `rollout`. The rest is either ignored or not implemented. At the time of this evaluation M6 (label placement, `replicas: N` spread) and M7 (NFS docroot) were not built; both are implemented now but unvalidated on real hosts. Ingress is an HTTP reverse proxy, not FastCGI.
+
+Rolling the WordPress container would be the wrong next step. A rolling revision publishes a different host port and shares the named volume with the old revision. Two PHP processes on one `wp-content` is how you corrupt uploads. The rollout docs say volumes are shared and that databases should stay on `recreate`. They do not say that about a PHP docroot. I left both sites on recreate.
+
+### Secrets and environment are easy to get half-right
+
+A value that is not exactly `${NAME}` makes Tug treat the environment key as a secret name. `WORDPRESS_DB_HOST: db` does not require a secret, and the literal is what the container gets. A project secret with that same name would also be released to the node and written into `.env`. The `.env` writer does not quote values. A password containing `#`, `$`, a space, or a newline breaks interpolation. Nothing in the secrets section says "use a substitution for the password, a literal for the hostname, and keep the password free of compose metacharacters." I used hex passwords because I had read `writeEnv`, not because the docs told me to.
+
+### Steady state is still a health check every poll
+
+`reconcile` compares container config hashes and runs the health checks on every poll, not only when the desired version changes. After `wp1` was already `serving`, the WordPress access log still showed user-agent `node`, from `172.20.0.1`, requesting `/` and following the 302 to `/wp-admin/install.php` about every 9 seconds. Node's fetch follows redirects. The docs mention that. They do not mention that a redirect to the public hostname would send the health check out through Caddy and back, or that the checker keeps doing it after the deploy has succeeded. For this site the installer returned 200, so the check passed. A redirect to a non-2xx URL would flap a healthy site.
+
+On the image these hosts are running, those polls also wrote `service.pulling`, `starting`, `health_checking` and `serving` every time. [PR #13](https://github.com/wpelevator/tug/pull/13), merged after this trial, stops that. The agent keeps a fingerprint and emits lifecycle events only when the revision, rendered compose or secrets change. The apply and the health check still run on every poll, which the new observability page states on purpose. The access-log noise is not fixed. The event flood is, once these agents are replaced.
+
+### What I still had to do by SSH
+
+M5 (`exec`, `logs -f`, `ps`) is the stated next milestone, and this run needed it on the first failure. Completing the WordPress install was a POST to `install.php` from the laptop, which is fine. Finding out why the container never started was not. Current `main` now puts the last apply error in the deployment failure and waits for `doctor --node`, but it still has no live shell or log stream.
+
+`volume list` records the volume and the node. It does not show whether the Docker volume exists or how large it is. I did not test `volume remove`. I also did not re-run rolling deploys, rollback, or `--build-local` on these sites. Those already passed on the nginx fixture. They are not what a WordPress install hits first.
+
+## Documentation
+
+The pages disagree with the path that works.
+
+- The README's steps 1 and 2 assume a public HTTPS control URL. This cluster's API is HTTP on a VPC address, on purpose. The quick start should show the tunnel, and it should say that `control setup` publishes `PORT:8080` on all interfaces unless you change the rendered compose. The bind to `10.110.0.3` is not a documented flag.
+- There is no compose example with a database, a secret, a volume, and an ingress host. The samples are nginx, busybox, and redis, placed on `vps-a`, `vps-b`, and `vps-c`. These nodes are named `app` and `edge`. A person copying the sample will fail placement and learn nothing about `depends_on` or TLS.
+- `sslip.io` is how both real-host runs got a trusted certificate without owning a domain. It belongs in the networking page next to `tls: auto`, including the rate-limit warning: sslip.io is one registered domain, and Let's Encrypt's per-domain cap is shared with everyone else using it. Two hostnames worked today. A third batch might not.
+- Preflight checks free disk (2048 MB under `/var/lib` for control, 1024 MB for a node) and never checks RAM. Both hosts have 1 GB and no swap. An unmodified MariaDB on the control host, next to the agent and the control container, is a bad idea. The control-plane page says the control VPS is not in the request path. It does not say not to put a database on that same 1 GB machine. This cluster does, because there are only two machines.
+- plan.md is still the "original specification" linked from the README, and its WordPress compose file is not a file this CLI accepts. The roadmap already says M6 and M7 serve that topology. The README does not. Someone who starts at the README and opens plan.md will try to deploy it. Resolved since: plan.md was removed and the [roadmap](roadmap.md#reference-topology) now holds the reference topology.
+
+## What I would change first
+
+1. Add one worked example: WordPress or any HTTP app plus a database, secrets via `${DB_PASSWORD}`, `depends_on` without `service_healthy`, ingress on an sslip.io name, and a note on the 1 GB MariaDB flag. State that app and database have to share a node until a published port or M7 exists.
+2. Make `tug deploy` print one line on success (`wp1 wp1-2 succeeded, 2 services`) and keep the JSON behind a flag. Same for `status`: project, version, health, node, without a nested string.
+3. Name the selected project in "service not found" errors.
+4. Stop publishing application ports on `0.0.0.0` by default when an ingress is set. Bind the overlay address, or `127.0.0.1` on the ingress node, in the generated file. The operator should not have to paste `10.210.0.1` into the source compose.
+5. Say in the README that the plan.md WordPress stack is not deployable yet, and point at the example that is. (Done differently: plan.md was removed and the roadmap describes the topology Tug targets.)
+
+I would not treat this run as evidence for rolling deploys, image build, or rollback. Those were not on this path. I would treat it as evidence that a person can get two real sites up, and that the first failure still requires SSH.
+
+
+## M6, M7 and enrollment on fresh hosts
+
+**Removed mesh implementation:** the commands, flags, environment variables, overlay addresses and packet-path results in this section are historical. Tug no longer provides that mesh, and these mesh checks are not a current validation baseline.
+
+31 October 2026, on the same two Ubuntu 24.04 droplets (1 vCPU, 1 GB, amd64, a shared VPC), after wiping both: containers, volumes, images, `wg0` and all Tug state. Control ran on 142.93.140.20 with `TUG_MESH=true`. The Tug image was built for amd64 from the branch and loaded with `docker save`.
+
+| Check | Result |
+|---|---|
+| `node add --mesh --label ...` on two hosts (`a`: `app`,`storage`; `b`: `app`) | Both online with labels; WireGuard mesh up |
+| `global: true` + `placement.labels: [app]` | One `web` replica on each labelled node |
+| `replicas: 1` + labels | One `worker`, on the alphabetically first matching node |
+| `tug scale worker 2` / `1` | Second replica started on `b`, then removed; `0` and unknown services rejected |
+| NFS volume (`x-tug.type: nfs`, exporter `a`; the volume type has since been removed) | `b` mounted `10.210.0.1:/srv/tug/shared` over the WireGuard overlay (NFSv3); a file written on `b` appeared on `a`; the exporter bind-mounted the directory |
+| `ps`, `inspect`, `exec`, `logs -f` against real nodes | Worked |
+| `node enroll --install-docker --mesh` cloud-config on a host with Docker purged | Docker Engine and Compose installed, agent started, node enrolled with its label |
+| Enrolled node with no public address behind `ufw` deny-incoming | WireGuard handshake and overlay ping to `a` worked; NFS mount worked |
+
+What this does not prove: the cloud-config was applied by re-running cloud-init (`cloud-init clean`, then its stages) with the config in `cloud.cfg.d`, not from first-boot user data, and the agent image came through an SSH-tunnelled throwaway registry. The firewalled node has a public IP, so this is not NAT. Debian, Fedora and Rocky bootstraps, arm64, and certificate renewal were not run.
+
+### Findings
+
+- **Fixed: dead nodes held rollouts open.** After `b`'s host was repurposed as `c`, the old `b` stayed registered with its labels. The next deploy placed `global` replicas on `b`, which never answered, and the deployment stayed `rolling_out` until cancelled. Label and `global` placement, and `tug scale`, now skip nodes not seen within `TUG_STALE_SECONDS`. `scale` also no longer spreads a labelled service onto nodes that do not match its labels when none match.
+- **Fixed: there was no way to remove a node.** `tug node remove <node>` now deletes it (refused while a current deployment places something on it, or while its agent still reports, unless `--force`). Run on the hosts: removing the dead `b` succeeded and `a`'s WireGuard peer for it was dropped, while removing `a` was refused with the services placed on it.
+- **Fixed: `control setup` could not enable the mesh.** `TUG_MESH` had to be added to the generated compose file by hand; `control setup --mesh` now does it (unit-tested only).
+- **Closed by removal: NFS needed host preparation Tug does not do.** The exporter needed an NFS server and `/etc/exports`, consumers `nfs-common`. The `nfs` volume type was removed rather than growing a host installer.
+- **Open: `cloud-init clean` regenerates SSH host keys.** Expected, but it makes the next SSH to the host fail strict checking.
+
+## Pull-once and backup recipe on a fresh host, 31 October 2026
+
+A single freshly wiped Ubuntu 24.04 host (Docker 29.9.2, 1 GB RAM) with control and one agent built from `main` after PR 25.
+
+| Check | Result |
+|---|---|
+| Unchanged project left alone for three minutes of agent polls | No container was recreated (same container IDs and start times) and no `service.pulling` event or pull line in the agent log appeared after the deploy finished |
+| `examples/wordpress-backup` deployed with `tug deploy` | WordPress, MariaDB 11.4.8 and the backup sidecar converged and WordPress answered on its published port |
+| `tug exec backup -- backup.sh once`, then `restic snapshots` | Exit 0, one `db` and one `files` snapshot, `restic check` reported no errors |
+| Restore drill into a scratch project (`db` and `backup` only) | The two seeded database rows and the seeded file came back; the repository was a local path, not S3 |
+
+### Second node and S3
+
+The second host (a cloud-init rebuilt Ubuntu 24.04, Docker 29.9.2) was wiped and joined with `node add` from the PR 27 branch; its agent came up with `net=host`. `node add` over an already registered node name worked.
+
+| Check | Result |
+|---|---|
+| `wp1` redeployed with a `backup-edge` sidecar and an `uploads` volume placed on the second node | Succeeded in 53 s; the sidecar image was built on control and downloaded by the second node's agent; the sidecar was healthy |
+| Both sidecars `backup.sh once` into one S3-compatible repository | Snapshots tagged host `app` (`db`, `files`) and host `edge` (`files`) in one repository; `restic check` reported no errors |
+| Restore of the `edge` snapshot from the first node into a scratch directory | The text file matched and the 3 MB random file had the same SHA-256 as the original |
+
+The store was RustFS in a container on the second node with random credentials, because the `quay.io/minio/minio` image answers 401 and the `minio/minio` Docker Hub image can no longer be pulled. It is not a hosted provider, and the repository was reached over plain HTTP on a public address, so treat the result as a protocol check, not a production recommendation.
+
+### Findings
+
+- **Fixed: health checks cannot reach containers from a bridge-networked agent.** At the time, `node add` started the agent on the default bridge. On Docker 29.9.2 it could not dial the container IPs of a project's Compose network, so every TCP or HTTP health check timed out (`tcp timeout 172.19.0.2:3306`) while MariaDB was ready, and the first deploy failed. Restarting the agent with `--network host` fixed it. Earlier runs on older Docker did not hit this. Current `node add` and `node enroll` always start the agent with `--network host` for these health checks; no added network capability is required.
+- **Open: `--build-local-image` builds for the CLI machine's architecture.** On an arm64 Mac it produced an arm64 image that the amd64 host could not run (`exec format error`). It should build for the target's architecture.
+- **Open: a deploy that fails health leaves the project half-up.** The containers kept running after the failed deployment, which hid the real cause until the agent's view was checked.
+
+## Generic backups, restore and `tug run` on fresh hosts, 1 November 2026
+
+The `examples/wordpress` project on two freshly reimaged 1 GB droplets, with the runtime built from the branch (`tug:local`, amd64) and nothing else on the hosts. The hosts were Ubuntu 26.04.1 LTS, newer than the 24.04 of earlier runs. Control and node `app` ran on `tug-1` and node `app2` on `tug-2`. The same example was deployed twice: as project `blog` on `app` and as project `shop` on `app2`, each with its own database and uploads volume. Backups went to an S3-compatible server (`rustfs`) that Tug itself deployed as a third project (`storage`) on `app`, reachable only on the private network. This is an operator trial, not a CI job.
+
+### What ran, and the result
+
+| Step | Result on both sites |
+|---|---|
+| `control setup --install-docker --build-local-image`, `node add` for both nodes | Worked on Ubuntu 26.04: Docker CE 29.9.2 came from Docker's own `resolute` suite, control and both agents came up, nodes registered online |
+| `tug deploy` of the example | Converged with health checks passing; WordPress answered `302` to its installer, then served after a scripted install |
+| `backup run` | Database dump (`mariadb-dump` run in the container, 107 KiB) and the uploads volume (14.1 MiB) both succeeded in about ten seconds |
+| `backup list --check` | Exit 0 with a healthy target list |
+| Damage: update the site title in the database, delete the active theme, create a stray file | The site rendered a blank page, as expected |
+| `restore db --confirm` | Completed in about 17 to 19 seconds: WordPress stopped, safety backup, dump loaded, WordPress started |
+| `restore wp-content --confirm` | Completed; the site title was present **immediately** when the command returned, with no sleep. The theme was back and the stray file gone |
+| `restore wp-content --scratch --confirm` | New volume with the files, the live volume untouched; it restored the last real backup, not the safety snapshot taken a moment earlier |
+| Scheduled runs | `shop` was redeployed with a two-minute schedule: the scheduler noticed the change within its 30-second tick and started both targets by itself at the next slot (trigger `schedule`); both succeeded. Redeploying the nightly schedule put the next run at 03:00 UTC again |
+| `tug run wordpress -- ...` | Saw the service's environment (`WORDPRESS_DB_HOST=db`); exit codes 0 and 7 were returned; no helper containers were left on either node |
+| Repository contents | Each project has its own repository (`tug-wp/blog`, `tug-wp/shop`), four snapshots each (two backups, two safety snapshots), hosts `app` and `app2`; `restic check` passed when run from the *other* server, a machine that never held the data |
+| Memory | `tug-1` (control, agent, S3, MariaDB, WordPress) used 508 of 956 MB; `tug-2` used 420 MB; no out-of-memory events |
+
+### Findings
+
+- **The updater can starve a small host.** On the earlier installs Ubuntu's `unattended-upgrades` and `apt-check` had run for over an hour on `tug-1`, holding it at load 16 until Docker commands timed out. The fresh hosts had the apt timers disabled before anything else was installed. This is now noted in [Nodes](nodes.md).
+- **Control and a node on one host compete for port 8080.** The example publishes `8080:80`, control publishes `8080`, so the copy for `tug-1` used `8081`. Control on a runtime node is [M15](roadmap.md#m15--flexible-node-roles).
+- **Named volumes need no ownership fixing.** The S3 server runs as uid 10001. As a bind mount it failed with `Permission denied`; as a named volume deployed by Tug it started cleanly.
+- **Scratch-restore volumes are left behind**, as documented. They were removed by hand.
+
+Not covered by this run: a real AWS S3 bucket (the S3 server is `rustfs`, which speaks the same protocol), a failing backup on a real host (`backup list --check` was only seen exiting 0 here; its failing path is covered by unit tests), a control restart or a long control outage, a node behind NAT, ingress and certificates (the hosts had no DNS names, so the sites were reached on their private addresses), and a project that spans both nodes.
+
+## Recovery drill with the published 0.9.0 release, 2 November 2026
+
+Two disposable amd64 Ubuntu hosts were used. The published npm CLI and `ghcr.io/wpelevator/tug:0.9.0` image ran control and the original agent on one host; an agent on the other host served as the replacement workload node. A dedicated RustFS container on the replacement host stored workload backups over the private network; this was an S3-protocol exercise over HTTP, not a hosted S3 or encrypted-transport validation. One host already had an unrelated control container and data, which were left in place. The release predates removal of Tug’s optional mesh; no mesh was enabled.
+
+The WordPress example deployed successfully, WordPress was initialized, and an upload sentinel was created. A manual backup produced a MariaDB dump and a `wp-content` snapshot. A separate control backup and the repository credentials were copied off the control host into private local files. The original workload agent and containers were stopped to simulate workload loss while keeping control available. The project was redeployed with placement changed to the replacement node, then the database and uploads were restored there with Tug’s normal safety backups. The recovered database contained the original site title, the uploaded sentinel matched exactly, and HTTP returned 200. This tested a manual re-placement and restore, not an automatic host failover; external DNS, HTTPS and a second public request path were not tested.
+
+Immediately after the replacement deployment succeeded, the restore plan still named the old node because backup schedules had not yet caught up. After a later scheduler tick it named the replacement; the restore was not confirmed until the target had been checked. The branch now refreshes schedules when planning and queueing a restore and has a regression test. This fix was not part of the published release used for the drill.
+
+For control recovery, an updated control backup was copied off-host. The test control container was stopped and removed, and its data directory was moved aside intact. A new control was bootstrapped on the same machine with fresh storage and a temporary credential. The backup was copied into its mounted data directory, restored through the API and activated by restarting control. The original credential and CA worked again, the replacement agent reconnected, the recovered backup repository credentials matched the escrowed copy, and the WordPress site still returned HTTP 200. This first pass demonstrates fresh-directory recovery on the same machine, not loss of its underlying disk. The test did not verify certificate renewal, automatic off-host backup, or a restore after the backup destination itself is lost. No measured recovery time or data-loss objective is claimed from this exploratory run.
+
+A second control backup cycle included a small Tug-built image. After restoring that backup into another fresh control data directory, the disposable probe container and its local image were removed from the replacement node. Its agent downloaded the retained artifact from restored control and recreated the container; the file baked into the image contained the expected marker. An initial remote-build attempt launched from the repository directory with an absolute Compose path failed with `source context has unsafe entries or exceeds expanded limits`; running the same deployment from its own clean context directory succeeded. The cause of the first rejection was not established.
+
+A further control recovery used the other VPS: a separate control container was bootstrapped on a distinct port and empty data directory without replacing that host’s pre-existing control instance. The off-host backup containing the built image was copied into the new data directory, restored using the temporary bootstrap credential, and activated by restart. The original test control was stopped to prevent two active authorities. The original operator credential then worked with a control URL override, and `node replace` repointed the replacement agent to the recovered control; the node reported online and the backup catalog was present. The probe’s local container and image were deleted once more, and the agent re-downloaded the restored built image from control on the new host; the image marker matched and WordPress returned HTTP 200. This confirms manual cross-host control recovery for this topology, but the replacement host also held the test object store and site, so it did not test simultaneous loss of that host or its object store. `node add` for the existing node returned 409 and required the identity-replacement command.

package/docs/releases.md

@@ -0,0 +1,39 @@
+# Releases
+
+Tug is published as the npm package [`@wpelevator/tug`](https://www.npmjs.com/package/@wpelevator/tug) (bins `tug` and `tug-runtime`) and as the multi-arch (`linux/amd64`, `linux/arm64`) Docker image `ghcr.io/wpelevator/tug`. The package and image always share one version, and the CLI's install commands default to the image with the CLI's own version.
+
+## Changesets
+
+Versions come from [Changesets](https://github.com/changesets/changesets). Every pull request with a user-facing change (CLI, API, agent behavior, image contents, docs for a new behavior) adds a changeset:
+
+```bash
+npm run changeset
+```
+
+It asks for the bump (`patch`, `minor` or `major`) and a summary, and writes a Markdown file in `.changeset/` that you commit with the PR. The summary becomes the changelog entry, so write it for users. Changes with no user-facing effect (CI, refactors, tests) need none.
+
+## Release flow
+
+The npm package includes the compiled runtime in `dist/` and the documentation in `docs/`, as listed in `package.json`'s `files` allowlist. npm also includes the root README and package metadata. To inspect the release contents locally without publishing, run `npm pack --dry-run`; its `prepack` hook builds the runtime first.
+
+1. A PR with a changeset merges to `main`.
+2. `release.yml` runs on every push to `main`. While unreleased changesets exist, `changesets/action` opens or updates a "chore: version packages" PR that runs `npm run version-packages` (bumps `package.json`, updates `package-lock.json`, writes `CHANGELOG.md` and deletes the consumed changesets).
+3. Merging that PR is the release. The same workflow then runs `npm run release` (`changeset publish`), which publishes `@wpelevator/tug` to npm with provenance and creates the git tag.
+4. When a package was published, the `image` job builds the multi-arch image from that commit and pushes `ghcr.io/wpelevator/tug` tagged `X.Y.Z`, `X.Y` and `latest`.
+5. Independently, every push to `main` pushes `ghcr.io/wpelevator/tug:edge`, the latest unreleased build.
+
+Nothing is released by hand: do not run `npm publish` or create tags yourself. Pull requests created by the workflow with `GITHUB_TOKEN` do not trigger other workflows, so CI does not start on the "Version Packages" PR by itself; push an empty commit to its branch or close and reopen it if you want checks on it.
+
+## npm trusted publishing
+
+The workflow publishes with npm [trusted publishing](https://docs.npmjs.com/trusted-publishers) over OIDC, so there is no `NPM_TOKEN` secret. The package on npmjs.com is configured with GitHub repository `wpelevator/tug` and workflow file `release.yml`, so that file must keep its name. The job has `permissions: id-token: write` and installs npm >= 11.5.1, the first version that supports trusted publishing.
+
+## One-time repository setup
+
+- Settings > Actions > General > Workflow permissions: allow "Read and write permissions" and "Allow GitHub Actions to create and approve pull requests", so `changesets/action` can open the Version Packages PR.
+- After the first image push, open the `tug` package under the organization's Packages and set its visibility to public (new GHCR packages are private by default), so hosts can pull without logging in. Make sure the package is linked to the repository and that Actions has write access to it (Package settings > Manage Actions access).
+- Optionally protect `main` and require the CI checks, so a release only comes from tested code.
+
+## Local development image
+
+To try an unreleased build, `tug control setup` and `tug node add` accept `--build-local-image`, which builds `tug:local` from the checkout and copies it over SSH, or `--image ghcr.io/wpelevator/tug:edge`. See [Nodes](nodes.md) and [Control plane](control-plane.md).