@wpelevator/tug 0.12.0 → 0.13.0

package/docs/networking.md

@@ -0,0 +1,150 @@
+# Networking: cross-node service names and ingress
+
+Two layers let services on different hosts find each other and let the internet find them. Both use the private network you already operate; Tug does not create an overlay network. The control plane stays single-writer: it resolves endpoints and renders configuration, but never carries workload traffic.
+
+| Layer | What it adds | Turned on by |
+|---|---|---|
+| A. Cross-node service names | Services on other nodes resolve by name to their node's `address` (hosts-file entries, not DNS), only while healthy | On by default; set `TUG_SERVICE_NAMES=false` on control to opt out |
+| B. Ingress and TLS | A managed Caddy on an ingress node routes hostnames to services anywhere, with automatic HTTPS | `tug node ingress <node>` plus `x-tug.ingress` on a service |
+
+Independently of the layers, [published ports are private by default](#published-ports-private-by-default): Tug binds them to the address other nodes and the ingress use, never to every interface unless you wrote that.
+
+## Bring your own network
+
+A node's `address` is its sole inter-node address. Set it with `node add --address`, the node's `address` config key or `TUG_NODE_ADDRESS` at enrollment to a private IP on one of the host's interfaces. Every node that runs a client or an ingress upstream must be able to route to that IP. A hostname is also accepted, but it must resolve on the node to an address Docker can bind. Update a registered address with `tug node address <name> <address>`. Do not use a cloud VM's public 1:1 NAT address when that address is absent from its interfaces; register its private interface address instead.
+
+The network can be a VPC, LAN, existing VPN or another routed private network. Tug does not allocate inter-node addresses, create tunnels, configure routes or provide NAT traversal for workload traffic. Agents can still dial out to control through NAT, but cross-node service names and remote ingress upstreams work only where this workload network is reachable.
+
+**Tug does not encrypt node-to-node workload traffic.** Mutual TLS protects connections to control, not connections between application containers, published ports and ingress upstreams. Use application-layer TLS or an operator-managed encrypted network where the traffic needs confidentiality or integrity.
+
+Firewall the workload network so only the required peers can reach the published service ports. Allow the ingress nodes to reach their upstream ports, and allow application nodes to reach only the services they consume; there is no need to open every port between every node. Publish ports 80 and 443 on ingress nodes only as required by the configured HTTP and ACME flows. Keep databases and other internal services off public interfaces and use a provider firewall as a second boundary: Docker's iptables rules can bypass host firewalls such as ufw. An explicit `0.0.0.0` or `[::]` host IP in Compose is a request for all-interface exposure.
+
+A single-node project needs no inter-node network. When only the managed Caddy on the same node reaches an ingress port, Tug binds that port to loopback.
+
+## Layer A: cross-node service names
+
+Decision: `extra_hosts` injection, not a DNS service. The names are hosts-file entries written into containers at creation, so there is no DNS server, no round-robin answer and no update between reconciliations; "service DNS" is reserved for a possible future implementation. Names for services on the same node are Docker's own network DNS and are unrelated. Cross-node names add no long-running process and need nothing new on the nodes; the cost is that records only change when the agent re-renders the project, which already happens on every poll.
+
+Cross-node service names are enabled by default. Set `TUG_SERVICE_NAMES=false` on control to disable them. While enabled, the work response carries `serviceEndpoints`: for each project on the node, each service to the list of `{ node, address }` where
+
+- the service is placed on that node,
+- the node has polled within `TUG_STALE_SECONDS`, and
+- the node's latest status reports that service as `rolloutState: serving` (so a failing health check withdraws the record).
+
+`address` is the node's configured `address`. The agent resolves a hostname address on the client node before writing it into `extra_hosts`; endpoints that cannot resolve are omitted until resolution succeeds, so the resulting entry is a `name:ip` value; a hostname must therefore resolve on the node. The agent's container definition (`containerSpecs`) adds `extra_hosts` to every service on the node for every service that runs elsewhere, as both `<service>` and `<service>.<project>.tug`. Services placed on the same node keep resolving through the project network's Docker DNS, and entries you wrote yourself in `extra_hosts` win. A service on several nodes gets one entry per serving node.
+
+Consequences:
+
+- The name resolves to the node's address, not the container. The target service must publish its port (`ports: ["3306:3306"]`) and clients connect to the published host port. Tug binds that port to the same address the name resolves to ([Published ports](#published-ports-private-by-default)); the operator's network and firewall determine who can reach it. Rolling services are the exception, see [Rolling services and service names](#rolling-services-and-service-names).
+- `/etc/hosts` has no load balancing: most resolvers use the first entry, so multi-node services give failover on node loss rather than round robin. Real load balancing is a non-goal.
+- Records appear after the target reports `serving` and the client's node re-renders its container definitions on its next poll, which recreates the client container if its `extra_hosts` changed. Order dependent services with health checks and expect one extra recreate on first deploy.
+
+## Layer B: ingress and TLS
+
+Decision: Caddy, as one managed container per ingress node, configured by a Caddyfile that control renders.
+
+```yaml
+services:
+  web:
+    image: nginx:1.27-alpine
+    ports:
+      - "8080:80"
+    x-tug:
+      placement:
+        nodes: [app1]
+      ingress:
+        host: app.example.com
+        port: 80
+        tls: auto
+```
+
+An ingress hostname belongs to one project: a deploy or rollback is refused with `409` if another project's current deployment already routes the host (case-insensitive). `tug node ingress edge1` marks a node as an ingress node (`--disable` removes the role). `port` may be the container port or the published host port; control resolves it to the published host port, and `validateCompose` rejects an ingress port that is not published, because the edge reaches services on `<node address>:<published port>`.
+
+Caddy dials each upstream where its port is bound ([Published ports](#published-ports-private-by-default)): `127.0.0.1` for a service whose ingress port listens on loopback on the ingress node, otherwise the node's `address`, including for services on the ingress node itself. Remote addresses must be reachable from the ingress node over the operator's network.
+
+Control aggregates the `ingress` blocks of every project's current deployment, and for each ingress node renders:
+
+- one site block per host with `reverse_proxy` to every serving endpoint, using `127.0.0.1` for loopback-bound services on the ingress node and the node `address` for the rest;
+- no route at all for a host without a serving upstream (reported in `ingress.warnings` rather than serving errors), and only the first project for a host claimed twice;
+- with `TUG_ACME_EMAIL`, a global `email` for ACME registration.
+
+TLS modes:
+
+- `auto` (default): Caddy obtains and renews certificates with ACME. Ports 80 and 443 must be reachable from the internet and the host's DNS record must point at the ingress node. A name such as `app.<ingress-ip>.sslip.io` resolves to that address without a domain you own. Let's Encrypt limits certificates per registered domain, and `sslip.io` is shared, so a burst of new hostnames can be refused.
+- `custom`: `cert` and `key` name two project secrets holding the PEM certificate chain and private key (`tug secret set WEB_CERT --project blog --value "$(cat fullchain.pem)"`). Deploy fails if either secret is missing. Control decrypts them only for the ingress node, which receives them as files in the work response; they never reach the nodes running the service, and they never appear in the Caddyfile.
+- `off`: plain HTTP on port 80, for development.
+
+The agent on an ingress node creates the `tug-caddy` container with host networking and three named volumes (`/data` for ACME accounts and certificates, `/config`, `/etc/caddy`), so certificates survive container recreation and image updates. Config is uploaded with the Engine API archive endpoint (which works whether the agent runs on the host or in a container) and applied with `caddy reload`, which swaps config gracefully without dropping connections or restarting the container. If Caddy rejects the config the agent restores the last applied files, reports `failed` with Caddy's error in the node's ingress status, and retries on the next poll; `tug doctor` flags ingress nodes that are not `serving`. Removing the ingress role removes the container but keeps the volumes. The Caddy image is `caddy:2-alpine` (override with `TUG_CADDY_IMAGE` on the agent).
+
+For services with `x-tug.rollout.strategy: rolling`, the upstream port is the one the serving revision reports instead of the declared published port (see below).
+
+Caddy's automatic HTTPS protects the client-to-ingress connection. It does not encrypt the subsequent HTTP connection from Caddy to a service on another node; that upstream connection uses the operator's network.
+
+## Published ports: private by default
+
+A `ports` entry such as `"3306:3306"` would listen on every interface of the node, and Docker's iptables rules bypass host firewalls such as ufw, so the port would be on the internet unless a provider firewall blocks it. Tug therefore rewrites every `ports` entry that names no host IP in the container definition the agent builds for the node:
+
+| Situation | Bound to |
+|---|---|
+| The service's ingress port, when only the managed Caddy on this node dials it | `127.0.0.1` |
+| Otherwise | the node's `address` (a warning event when it is public) |
+| The node has no `address` | `127.0.0.1` |
+
+When a node `address` is configured, it is the same one other nodes resolve the service to and the ingress dials, so cross-node service names and upstreams keep working when the private network is reachable.
+
+"Only the managed Caddy on this node" means: this node is the only ingress node, the service runs on it, and every service of the project runs only on nodes where this service also runs, so no other node reaches it through a service name. That is the [small reference topology](roadmap.md#reference-topology) (app, database and Caddy on one node). Only the entry that publishes `x-tug.ingress.port` moves to loopback; other ports of the service follow the rest of the table.
+
+Details:
+
+- Short syntax gets the address prefixed (`10.0.0.8:3306:3306`, `10.0.0.8:5353:53/udp`, `10.0.0.8:8000-8010:8000-8010`); a container-only entry such as `"80"` becomes `10.0.0.8::80`, and an IPv6 address is bracketed. Long syntax gets `host_ip`.
+- An entry that already names a host IP is left exactly as written, including `0.0.0.0` and `[::]`. That is how you publish something on purpose: `"0.0.0.0:2222:22"`.
+- Control sends the binding in the work response (`ports`, `PortBinding` in `src/protocol.ts`): the address and, per project, the loopback services. A hostname `address` is resolved on the node, only when some port of the project binds to it; if it does not resolve the project fails to apply rather than falling back to every interface. The address must belong to one of the node's interfaces, otherwise Docker cannot bind it (`cannot assign requested address`).
+- With a public `address`, ports are bound to that public address and the agent sends one `ports.public` warning event naming the services (again whenever the set changes). Register a private `address`, or write the host IP in `ports` to make the exposure explicit.
+- Rolling revisions keep the host IP when their host port is remapped, so revision ports (`TUG_ROLLOUT_PORTS`) are bound the same way.
+- `tug doctor --nodes` runs a `ports.public` check on every agent: it lists containers of Tug projects (and their rolling revisions) listening on `0.0.0.0`, `::` or a public IP. It is a warning, not a failure, because an explicit `0.0.0.0` may be intended.
+
+## Rolling services and service names
+
+A service with `x-tug.rollout.strategy: rolling` ([Compose extensions](compose.md#rolling-rollouts)) is addressed through the ingress, and the design deliberately keeps service names stable while its port is not.
+
+Decision: each revision of a rolling service publishes the ingress port on its own host port from `TUG_ROLLOUT_PORTS` (default `20000-20999`) so old and new revisions can run at the same time. The agent reports the serving revision's port in its status (`serviceStates.<service>.port`), control puts it into the endpoint (`ServiceEndpoint.port`) and the Caddyfile upstream becomes `<node address>:<revision port>`. A new revision is therefore invisible to the ingress until it is healthy, and a failed one is never routed to because the node keeps reporting the old revision (still `serving`) with the old port.
+
+What this means for names:
+
+- `extra_hosts` entries are `name -> node address` with no port, so a rollout does not change them. Dependents on other nodes are never recreated by a rollout, and a rollout never restarts services that did not change. This is why per-revision ports are safe for `extra_hosts` injection even though a hosts file cannot carry ports.
+- The name of a rolling service on another node still resolves (to the node address, as long as that node reports the service serving), but clients dialing `web:8080` find nothing on the declared port, because rolling revisions do not hold it. Cross-node clients of a rolling service should use its ingress hostname. Giving rolling services a stable name and port would need another mechanism that owns the declared port; it is not implemented.
+- On the same node, services in the main project reach a rolling service by name over the shared project network: every revision joins `<project>_default` with the service name as an alias, so the name resolves to whichever revisions are running. During the overlap (start, health check and drain) that can be both revisions, and a not-yet-healthy new revision can receive same-node name traffic. Put anything that must not see an unhealthy revision behind the ingress.
+- Reaching a rolling service by its declared host port (for example `curl localhost:8080` on the node) does not work; the port moves with every rollout.
+- The injected `extra_hosts` are part of a revision's identity, so a rolling service rolls again when another node's service starts or stops serving. On a real first deploy `web` rolled once without the `client` entry and again as soon as `client` on the edge node became serving. The extra rollout is graceful, because the ingress only switches to a healthy revision, but it does start a new container.
+
+Ingress takeover timing: the new port reaches Caddy after the app node reports it (next poll), control renders a new Caddyfile and the ingress node polls and runs `caddy reload` (graceful, in-flight requests finish). The old revision is removed only after `drain`, which defaults to 30s, so there is no gap. Keep `drain` above the two poll intervals.
+
+## Sequence
+
+```mermaid
+sequenceDiagram
+    participant Op as Operator
+    participant Ctrl as Control plane
+    participant A as Agent app1
+    participant E as Agent edge1 (ingress)
+    participant Caddy as Caddy on edge1
+
+    Op->>Ctrl: node add app1 / edge1 with private address values
+    Op->>Ctrl: node ingress edge1
+    Op->>Ctrl: deploy (web on app1 with x-tug.ingress)
+    loop every poll
+        A->>Ctrl: GET /agent/v1/work
+        Ctrl-->>A: desiredStates + serviceEndpoints + port bindings
+        A->>A: reconcile containers with extra_hosts and private-address ports
+        A->>Ctrl: POST /agent/v1/status (web serving)
+        E->>Ctrl: GET /agent/v1/work
+        Ctrl-->>E: serviceEndpoints + ingress (Caddyfile, certs, routes) + port bindings
+        E->>Caddy: docker cp config, caddy reload (only if config hash changed)
+        E->>Ctrl: POST /agent/v1/status (ingress serving)
+    end
+    Note over Caddy,A: client request: Caddy terminates TLS, proxies over the operator's network to app1 address:8080
+```
+
+## Verifying
+
+`npm run test:integration:network` starts three privileged fake VPSes with plain numeric node `address` values (`vps-c` as ingress node), deploys `docker-compose.network-sample.yml`, and checks a container on `vps-b` fetching `http://web:8080` by its cross-node service name, Caddy on `vps-c` serving `app.test` from `vps-a`, and published ports bound to the intended private node addresses rather than every interface. See [Development](development.md#integration-test).

package/docs/nodes.md

@@ -0,0 +1,143 @@
+# Nodes
+
+A node is a registered Docker host on which Tug places services, and the agent is the container on it that fetches work from the control plane and manages the node's containers through the Docker Engine API. Nodes need Docker Engine only, not the Compose plugin. Add nodes either by pushing an install over SSH or by letting a fresh host enroll itself.
+
+Node operations that act on existing nodes can offer registered names when omitted. `tug node replace --select` and `tug node repair --select` choose several registered nodes with SSH targets in local config; `--all` continues to walk every configured node. Use `tug node address --to <address>` to choose the node while supplying its new address. See [interactive selection](cli.md#interactive-selection).
+
+## Node names and IDs
+
+A node has a **name** you choose (used in `x-tug.placement.nodes`, the CLI and the config file) and an **ID** that control generates (`node_` followed by random hex). The ID is immutable: it is never derived from the name, never changes when the identity is replaced, and is never reused. The node's certificate names its ID (`tug:node:<id>`), and every record control keeps for the node (status, commands, backup schedules and volumes) refers to the ID. When `node remove` frees a name, a node created later with that name gets a new ID, so certificates issued for the removed node can never authenticate the new one, even if the same key is installed again, and leftover records of the removed node do not attach to it. `tug status` and `tug events` show IDs next to names where a record carries one.
+
+## Install over SSH (`node add`)
+
+```bash
+tug node add root@app1.example.com --name app1 --address 10.0.0.11 --install-docker
+```
+
+The node generates its own key and certificate request; an admin credential authorizes the identity at that moment and the certificate is installed back over SSH, so no bootstrap secret travels and the private key never leaves the node's state volume. In order, the command runs SSH preflight checks, runs `agent identity` on the host (printing the CSR for the pending key in `/var/lib/tug/agent-pending.json`), posts it to `POST /api/v1/nodes/:name/identity` with `mode: create`, pipes the response into `agent install-cert` (which validates that the certificate names this node, chains to the CA in the response and matches the pending key before durably writing `/var/lib/tug/agent.json`), writes `/etc/tug/agent.env` and starts the agent container. Every step is safe to repeat: rerunning after a partial failure reuses the pending key, and `create` with the identical key is an idempotent retry that keeps the first call's metadata. No secret appears in an SSH command line or log. The command then waits for a status report from the new identity (`--no-wait` skips it).
+
+`--address` is the node's sole inter-node address: other nodes and the ingress use it to reach services, service names resolve to it, and published ports bind to it ([Published ports](networking.md#published-ports-private-by-default)). Give a private IP on one of the host's interfaces and make it reachable from the other nodes that need it. Control rejects an `--address` that is not an IP or hostname without a port. Tug does not create or encrypt this workload network; see [Bring your own network](networking.md#bring-your-own-network).
+
+Built application images are downloaded directly from control. Published images are pulled by their stored digest from upstream registries, with private credentials released through control when needed. No node registry login is required.
+
+Small hosts: a 1 GB VPS is enough for one site (a database and WordPress together use a few hundred MB), but not for much else. Ubuntu's `unattended-upgrades` and `apt-check` can run for hours on such a host and starve Docker of memory and CPU until commands time out; during testing one pegged a 1 GB node at load 16. Tug does not manage host updates; if the host is small, schedule them for a quiet window or disable the timers (`systemctl disable --now apt-daily.timer apt-daily-upgrade.timer`) and patch deliberately. A control plane and a node on the same host also compete for port 8443 if a service publishes it, which is why the WordPress example is not meant to share a host with control until [M15](roadmap.md#m15--flexible-node-roles).
+
+Preflight checks fail early, with a fix hint, unless the host has Docker Engine (no Compose plugin needed) accessible to the SSH user, writable `/opt`, `/var/lib` and `/etc/tug` paths, at least 1024 MB free under `/var/lib`, and can open a TCP connection to the control port. Preflight checks reachability only and copies no operator credential to the host; real TLS authorization happens when the agent first connects.
+
+The global `--control-url` changes only where the operator CLI connects; the server is still verified by its generated control name. When agents must dial a different address, pass `--agent-control-url`; it is written to the agent environment as `TUG_CONTROL_URL` and is likewise a connect target only. For a local SSH tunnel to control and a private endpoint for nodes: `tug --control-url https://127.0.0.1:18082 node add root@app1.example.com --name app1 --agent-control-url https://10.110.0.3:8443`. Both overrides must be `https://`.
+
+### Certificates, renewal and recovery
+
+Node certificates last 90 days. The agent renews at two thirds of the lifetime plus up to ten percent random jitter by sending a CSR for its existing key to `POST /agent/v1/renew`, saves the new certificate durably, then swaps its HTTP client and reconnects the stream. If renewal fails it keeps working on the old certificate until that expires. Authorization follows the key, so old certificates stay valid until they expire and routine renewal closes no connection. Control enforces a minimum interval between renewals per node (`TUG_RENEW_MIN_INTERVAL_MS`, default one hour).
+
+| Situation | Command |
+| --- | --- |
+| Cut a node off immediately, without SSH or deleting anything | `tug node disable <node>` (also removes it from service endpoints, ingress and label-based scheduling; explicit `placement.nodes` entries are left alone) |
+| Certificate expired, key lost or suspected compromised, node disabled, CA loss, or an older backup restored | `tug node replace <node> [ssh-target]` installs a new key (`mode: replace`); the old key's certificates stop working at once and their sockets close |
+| `agent.json` or `agent-pending.json` is corrupt | `tug node repair <node> [ssh-target]` moves the corrupt documents aside (never service volumes or agent event state) and replaces the identity |
+| Every node after a CA loss or older restore | recover root on the control-plane host (`control setup --recover-root`), then `tug node replace --all` |
+
+`node add --all` and `node replace --all` walk the SSH targets declared in `tug.yml` one node at a time, continue after failures and print a per-node result. Suspected key compromise is `node disable` followed by `node replace`. `node remove` deletes the node (its ID is retired for good) and revokes any unconsumed join key.
+
+Moving services to new servers and retiring old nodes is covered in [Migrating to new servers](migration.md).
+
+## Declaring nodes in the config file
+
+A repository can record its servers in the [config file](cli.md#config-file) so nobody has to remember `root@203.0.113.7`:
+
+```yaml
+# tug.yml
+nodeDefaults:
+  installDocker: true
+nodes:
+  app1:
+    ssh: root@203.0.113.7
+    address: 10.0.0.7
+    label: [web, ingress]
+```
+
+`tug node add --name app1` (or `tug node replace app1`) then runs as if `root@203.0.113.7 --address 10.0.0.7 --label web --label ingress --install-docker` had been typed; `tug node register app1` and `tug node enroll --name app1` read the same entry. Flags override the file key by key, and a `label` list on the command line replaces the file's list. An `<ssh-target>` argument overrides `ssh`; without one, a node with no `ssh` in the file is an error that names the missing key. The argument is never guessed to be a node name, because `app1` is also a valid hostname. `nodeDefaults` apply to every node, declared or not. Batch identity commands (`node add`, `node replace` and `node repair` with `--all`) use the same precedence for every supported key: explicit flags, then the node entry, then `nodeDefaults`, then command defaults. This includes `image`, `agentControlUrl` and `installDocker`; `false` and empty lists remain explicit values.
+
+To change a registered node’s address, run `tug node address app1 10.0.0.12`. Control wakes the agents so port bindings and service endpoints converge to the new address. The address must already be available on the node and reachable by its peers.
+
+The control plane stays the source of truth. Only `node add`, `node replace`, `node repair`, `node register` and `node enroll` read `nodes`, as flag defaults; `status`, `deploy`, `scale`, `ps`, `logs`, `exec`, `node remove` and every other command ask control and never consult the file. Re-running `node add` with the same key is an idempotent retry that keeps the first call's metadata; `node replace` installs a new key and keeps the node's ID, metadata and placement. It also updates the node's `address` when `--address` is supplied, with the same validation as registration. Editing the file without re-running anything changes nothing in control.
+
+## Self-service enrollment (`node enroll`)
+
+For hosts you cannot SSH to or do not want to touch: cloud-init and user-data provisioning, autoscaling and golden images, hosts behind NAT or in private networks, and delegated onboarding by someone without admin credentials. Mint a join key with an admin credential, then run the printed `docker run` on the fresh host or put it in user-data. The agent dials out to control.
+
+```bash
+tug node enroll --name app1 --address 10.0.0.11
+# prints, e.g.:
+#   docker run -d --name tug-agent --restart unless-stopped \
+#     --network host \
+#     -v /var/run/docker.sock:/var/run/docker.sock \
+#     -v tug-agent:/var/lib/tug \
+#     -e TUG_JOIN_KEY=<key> -e TUG_NODE_ADDRESS=10.0.0.11 \
+#     ghcr.io/wpelevator/tug:<CLI version> agent
+```
+
+The join key (`TUG_JOIN_KEY`) is base64url-encoded JSON `{ v: 2, credential, name, connectUrl? }`. The credential is a **bootstrap certificate** with SAN `tug:enroll:<id>` and its private key; it can reach `POST /agent/v1/enroll` and nothing else. The agent generates its own node key (into `agent-pending.json`), sends a CSR with the bootstrap certificate, validates and installs the returned node certificate exactly as `agent install-cert` does, and from then on ignores the join key. On restart the agent loads `/var/lib/tug/agent.json` and skips enrollment.
+
+Join keys are single-use, name-bound and short-lived (default one hour). There is no `--uses` and no reusable or unbound key, so golden-image and autoscaling flows mint one key per instance, for example into each instance's cloud-init. Name and labels come from the control-side row, never from the join key or the request. Enrollment is create-only: an existing name is refused and a join key can never replace a node. A retry with the same CSR key after a lost response signs another certificate and changes nothing stored. Minting again for a name revokes its previous unconsumed key, and `node remove` revokes it too. The join key is an **enrollment credential** and as sensitive as any bootstrap secret: it contains a private key, a saved user-data document contains a credential, and in cloud user-data it may be readable from the provider's metadata service by processes on the host. Single use, short life, name binding and create-only enrollment limit the damage.
+
+Options:
+
+- `--name <node>` (required) binds the node name.
+- `--address <addr>` supplies the private node address to the generated agent command and cloud-init. When omitted, set `TUG_NODE_ADDRESS` on the agent before its first enrollment if cross-node networking is needed.
+- `--agent-control-url <url>` sets `connectUrl`, a connect-only target; verification still uses the control name from the pinned CA.
+- `--ttl <seconds>` sets the key lifetime (default `3600`).
+- `--label <label>` sets node labels (repeatable).
+- `--image <tag>` sets the image in the printed command (default `ghcr.io/wpelevator/tug:<CLI version>`).
+- `--json` prints machine-readable output.
+- `--install-docker` prints a complete cloud-init document that installs Docker Engine before starting the agent. With `--json`, the result includes both `command` and `cloudInit` fields.
+
+To keep the key out of shell history and `docker inspect`, mount it as a `0600` file and set `TUG_JOIN_KEY_FILE=/path/to/key` instead of `TUG_JOIN_KEY`. Without an identity or a join key the agent refuses to start and points to `tug node add`, `tug node enroll` or `tug node repair`.
+
+## Bare-host Docker installation
+
+`control setup --install-docker` and `node add --install-docker` run the same Docker bootstrap over SSH before Docker preflight or runtime-image transfer. Without the flag, Docker Engine must already work (and the Compose plugin on the control-plane host, which runs the control container with Compose). The SSH target must be root for installation; Tug does not modify group membership or configure passwordless sudo.
+
+For a host created by your cloud provider, save `tug node enroll --name app1 --address 10.0.0.11 --install-docker` as user-data. Its stdout starts with `#cloud-config`, includes the bootstrap script and starts the agent only after Docker installation succeeds. The default version-matched Tug runtime image must be accessible from the host; an overridden development `tug:local` image must already be loaded there. Join keys are short-lived and should be treated as credentials when saving user-data.
+
+Bootstrap requires systemd and supports these releases:
+
+| Family | Releases | Docker repository |
+| --- | --- | --- |
+| Debian (apt) | 12 `bookworm`, 13 `trixie` | [Debian](https://docs.docker.com/engine/install/debian/) |
+| Ubuntu (apt) | 22.04 `jammy`, 24.04 `noble`, 26.04 `resolute`, 26.10 `stonking` | [Ubuntu](https://docs.docker.com/engine/install/ubuntu/) |
+| Fedora (dnf) | 42, 43, 44 | [Fedora](https://docs.docker.com/engine/install/fedora/) |
+| RHEL (dnf) | 9, 10 | [RHEL](https://docs.docker.com/engine/install/rhel/) |
+| Rocky Linux, AlmaLinux, CentOS Stream (dnf) | 9, 10 | [CentOS](https://docs.docker.com/engine/install/centos/) |
+
+It adds Docker's official apt or dnf repository, installs Engine and Buildx, and enables Docker (the control-plane host also gets the Compose plugin, because `control setup` runs the control container with Compose; nodes do not). A working Engine installation is preserved. If only Compose is missing, it installs the plugin without upgrading Engine. An inaccessible existing daemon, unsupported OS or release, or conflicting distribution packages (`docker.io` and `podman-docker` on Debian/Ubuntu; `docker`, `moby-engine` and `podman-docker` on RPM hosts) produce a clear error for manual resolution; Tug does not remove them or alter Docker daemon trust settings. Other distributions, such as Alpine or Arch, and hosts without systemd need Docker installed manually.
+
+## Image
+
+`node add` and `node enroll` default `--image` to `ghcr.io/wpelevator/tug:<version>`, where `<version>` is the installed CLI's own package version, so the CLI and the agents it installs stay on the same release. Pass `--image <tag>` to run another image, for example `ghcr.io/wpelevator/tug:edge` for the latest build of `main` or a mirror in your own registry.
+
+`node add --build-local-image` is for development: it builds the image from the current checkout, copies it to the host with `docker save` / `scp` / `docker load` and runs it, tagging it `tug:local` unless `--image` is also given. It needs a Tug checkout (the Dockerfile) in the working directory. See [Releases](releases.md) for how images are published. The image is built for the target host's architecture (read with `uname -m` over SSH), so an arm64 laptop can install an amd64 VPS; x86_64 and aarch64 hosts are supported.
+
+## Node labels
+
+`node add`, `node enroll` and `node register` accept a repeatable `--label <label>`. Labels are plain string tags stored on the node and listed by the node list API; services select nodes that carry all of their `x-tug.placement.labels` (see [Compose extensions](compose.md#services)). They are set when the node is created (`node add`, `node register --mode create` or the enrollment credential); `node replace` keeps them. They are unrelated to Docker container labels.
+
+## Agent host networking
+
+The agent always runs with host networking (`--network host`) because it dials container IPs directly for TCP and HTTP health checks, which a bridge-networked agent cannot reach on current Docker (29.x). On Docker Desktop the daemon and container addresses live inside its VM, which is why the agent belongs in a container on that daemon rather than in a plain host process. Host networking needs no added Linux capability. Inter-node workload reachability is a separate concern owned by the operator's network; see [Networking](networking.md#bring-your-own-network).
+
+## CSR in, certificate out (`node register`)
+
+`tug node register <name> --csr node.csr [--mode create|replace]` is the primitive behind the SSH commands: it posts a CSR you generated on the node (for example with `agent identity`) to the identity endpoint and prints the certificate, CA and public URL. Use it for local development or when you drive the steps yourself; install the output with `agent install-cert`.
+
+## Status and diagnostics
+
+Nodes are `online` while they contact control and become `stale` after `TUG_STALE_SECONDS` (default `60`) without an authenticated work fetch, status report or wake wait. Agents keep an outbound WebSocket open so control can wake them for deployments and commands; if WebSockets are unavailable they use an HTTP long poll, and `TUG_POLL_INTERVAL_MS` remains the periodic reconciliation fallback. `tug status` lists nodes, deployments and per-node project status; `tug doctor --nodes` (or `--node <name>`) runs Docker Engine checks, plus Caddy checks on ingress nodes, through each agent and waits for the result; the wait defaults to 60 seconds. Online and offline are connectivity states only: a `stale` node has not contacted control recently, which does not prove the machine is off. `tug doctor` also reports each node's certificate expiry and, for nodes that are reporting now, the observed clock skew (see [Doctor](#doctor-certificate-expiry-and-clock-skew)).
+
+### Doctor: certificate expiry and clock skew
+
+`tug doctor` always reports the control CA and server certificate expiry and each node certificate's expiry, with warnings when renewal is overdue (CA under 90 days, server certificate under 30, a node certificate under 14) and a failure for an expired certificate. For nodes it distinguishes **observed** clock skew from **unknown**: agents send their clock with every status report, control records the difference, and doctor shows it only for a node that is online now. A node that is stale, has never reported or is disabled has an unknown clock, not a zero skew. The figure includes request latency. A skew of 30 seconds or more is a warning; five minutes or more fails, because certificates are back-dated five minutes and a node that far ahead or behind cannot validate new certificates (fix time synchronization on the host).
+
+## Configuration
+
+See [Configuration](configuration.md#agent) for agent environment variables.

package/docs/observability.md

@@ -0,0 +1,213 @@
+# Observability
+
+**Status: mostly proposed.** Tug today has an audit trail, a few read commands and a certificate and connectivity check, and nothing that measures CPU, memory, requests or latency. Sections marked **Proposed** describe a setup that is not built; the milestone is [M9](roadmap.md#m9--observability). Nothing marked proposed works yet, and none of the names, flags or environment variables in those sections exist in the code.
+
+The aim is the smallest setup that answers the day-to-day questions: is every node reporting, did the last deployment succeed, did an automatic rollback happen, will a certificate expire, is the control database growing. It follows the shape of the rest of Tug: one central control plane, agents that only dial out, and no required extra services.
+
+## What exists today
+
+Verified against the code on `main`.
+
+| Capability | What it does | Limits |
+| --- | --- | --- |
+| Audit trail | Deployments and events record the acting credential, and `system:auto-rollback` for automatic rollbacks. See [Access control](access-control.md). | Answers who acted, not how the system behaves |
+| `tug status` | Summarizes deployments, errors and node reports; `--json` prints `GET /api/v1/status` as JSON: projects, the 20 latest deployments, nodes with `status` and `last_seen_at`, and per-node project status including health, rollout state, port and rollout error. | Node health is a last observation, not a live probe; a node is marked `stale` only when this endpoint is called, using `TUG_STALE_SECONDS` (default 60). The current deployment can be outside the 20 returned rows. |
+| `tug events` | Lists events from `GET /api/v1/events`, newest first, at most 500 per call, 100 by default. | Agents report on change only, see [Events](#events) |
+| `tug doctor` | Local checks (Node.js, Docker, the Compose plugin `deploy` needs), control checks (SQLite, CA and server certificate expiry, each node's certificate expiry and connectivity, each online node's observed clock skew, master key, Docker, ingress state) and, with `--nodes` or `--node`, host diagnostics through each agent. It waits for those results. Clock skew is unknown, not zero, for a node that is offline. | A point-in-time check, not a time series |
+| `tug logs <service>` | Prints a service log tail; `--follow` streams logs through the live channel. | Following logs requires an active WebSocket session. |
+| Reachability | Control has no unauthenticated route, not even a health check: every request needs a client certificate. Monitor TCP reachability of the mTLS port for liveness, and use an authenticated read (`tug doctor`, `tug status`) for readiness. | A TCP check proves the listener accepts connections, not that the database works |
+| Process logs | Control and agent write plain `console.log` and `console.error` text. | No timestamps, levels, structure or request ids |
+
+What does not exist: a `/metrics` endpoint, CPU, memory, disk or restart data, request, build or poll metrics, structured or leveled logs, alerting, outbound webhooks (M8, unchecked) and a dashboard (M8, unchecked). The managed Caddy ingress does not enable Caddy's metrics: the generated Caddyfile has no global options beyond the optional ACME email.
+
+### Events
+
+Agents report lifecycle events for the recreated services of a project (`service.pulling`, `starting`, `health_checking` and `serving`) only when something about the project changes: a new desired revision, a different Compose file rendered for the node, or different secret versions. A failed apply produces one `service.failed` event (level `error`) naming the service whose check failed and why, retries of the same failure add nothing, and recovery produces one `service.serving` event. Agents still apply and health-check the project on every poll, so only the events are quiet. A `ports.public` event (level `warn`) says published ports were bound to a public node address ([Networking](networking.md#published-ports-private-by-default)); it is sent once per change, not per poll. Control deletes events older than `TUG_EVENT_RETENTION_DAYS` (default 30) and keeps at most `TUG_EVENT_MAX_ROWS` (default 100000), at startup and then at most once an hour while agents report; `0` turns a limit off. See [Architecture](architecture.md#events) and [Configuration](configuration.md#control-plane).
+
+Before [PR #13](https://github.com/wpelevator/tug/pull/13) agents emitted those four events for every non-rolling service on every poll, whether or not anything changed. On a real control plane with nothing deploying that was about 71 events a minute, roughly 100,000 a day, nothing deleted them, and the default `tug events` view (the last 100) covered about a minute and a half and hid real events such as an automatic rollback.
+
+Known limit: a container that crashes and is restarted by the next poll produces no event unless its health check fails, so crash loops would show up only in the restart counts proposed under [node and service stats](#node-and-service-stats-reported-by-agents).
+
+## Proposed
+
+Everything below is **Proposed** unless a status line says otherwise.
+
+| Piece | Section | Status |
+| --- | --- | --- |
+| Structured logs | [Structured logs](#structured-logs) | Proposed |
+| Metrics on control | [Metrics on control](#metrics-on-control) | Proposed |
+| Node and service stats from agents | [Node and service stats reported by agents](#node-and-service-stats-reported-by-agents) | Proposed |
+| Readiness endpoint | [Health endpoints](#health-endpoints) | Proposed |
+| Ingress metrics | [Ingress metrics](#ingress-metrics) | Proposed, with open questions |
+| Sample alerts and webhooks | [Alerting](#alerting) | Proposed |
+| Live commands | [Live commands](#live-commands) | Partly built in M5 |
+| Metrics test gate | [Test gate](#test-gate) | Proposed |
+
+### Structured logs
+
+Status: proposed.
+
+Control and agent would write one JSON object per line with `time`, `level`, `component`, `msg` and context fields where they apply: `requestId`, `node`, `project`, `service`, `deploymentId`. Two environment variables would select the output: `TUG_LOG_FORMAT=json|text` and `TUG_LOG_LEVEL` (`debug`, `info`, `warn`, `error`). Text would stay the default so existing installs read the same, and JSON is meant for `docker logs` piped to whatever the operator already uses.
+
+Credential private keys, join keys and secret values are never logged, and the logger would redact known field names as a second line of defence. The CLI keeps its human-readable output; only the long-running processes change. Every existing `console.log` and `console.error` in `src/control.ts` and `src/agent.ts` would move to the logger.
+
+### Metrics on control
+
+Status: proposed.
+
+Control would serve `GET /metrics` in the Prometheus text format. It needs the `read` role from [Access control](access-control.md#roles), so Prometheus would present the client certificate and key of a `read` operator credential (`tls_config` with `cert_file`, `key_file` and `ca_file` extracted from the credential file); a credential limited to projects would see only those projects' series. Adding the route means adding it to `ROUTE_ROLES` in `src/credentials.ts` and to the endpoint table, which the existing test already requires.
+
+Naming follows the Prometheus conventions: a `tug_` prefix, base units (`_seconds`, `_bytes`), `_total` for counters. Labels are bounded: project, node and service names are operator-defined and few; route labels use the route pattern (`/api/v1/deployments/:id`), never the raw path; deployment IDs, command IDs, versions and error messages are never labels.
+
+| Metric | Type | Labels | Meaning |
+| --- | --- | --- | --- |
+| `tug_build_info` | gauge | `version` | Constant 1; the running control version |
+| `tug_deployments` | gauge | `project`, `state` | Deployments currently in each state (`uploading`, `building`, `pending_dependencies`, `rolling_out`, `succeeded`, `failed`, `build_failed`, `cancelled`) |
+| `tug_deployment_duration_seconds` | histogram | `project`, `outcome` | Time from creation to a terminal state |
+| `tug_rollout_outcomes_total` | counter | `project`, `outcome` | Rollout results: `rolled_out`, `rejected` (a candidate failed its health check while the old revision kept serving), `auto_rollback` |
+| `tug_nodes` | gauge | `status` | Registered nodes by `online` or `stale`, computed from `last_seen_at` at scrape time rather than stored |
+| `tug_node_poll_age_seconds` | gauge | `node` | Seconds since the node last reported |
+| `tug_api_requests_total` | counter | `route`, `method`, `status_class` | API requests by route pattern and status class (`2xx`, `4xx`, `5xx`) |
+| `tug_api_request_duration_seconds` | histogram | `route` | API request latency |
+| `tug_builds_total` | counter | `outcome` | Builds by result (`complete`, `failed`) |
+| `tug_build_duration_seconds` | histogram | none | Remote build duration |
+| `tug_image_archive_bytes` | gauge | none | Total size of retained image archives under the data directory |
+| `tug_db_size_bytes` | gauge | none | Size of the control SQLite database file |
+| `tug_events_rows` | gauge | none | Rows in the `events` table |
+| `tug_project_locks` | gauge | none | Project operation locks currently held |
+| `tug_writer_queue_depth` | gauge | none | Operations waiting for the single writer |
+
+Gauges that describe state (`tug_deployments`, `tug_nodes`, `tug_events_rows`, `tug_project_locks`) would be computed from the database at scrape time, so they stay correct across control restarts. Counters and histograms live in process memory and reset on restart, which Prometheus handles as a counter reset.
+
+### Node and service stats reported by agents
+
+Status: proposed.
+
+Agents already send a status report on every poll. The proposal adds a `stats` block to that report and has control expose it on the same `/metrics` endpoint, so one scrape target covers everything:
+
+| Metric | Type | Labels | Meaning |
+| --- | --- | --- | --- |
+| `tug_service_cpu_seconds_total` | counter | `node`, `project`, `service` | CPU time used by the service's containers |
+| `tug_service_memory_bytes` | gauge | `node`, `project`, `service` | Memory in use |
+| `tug_service_up` | gauge | `node`, `project`, `service` | 1 when the container is running, 0 otherwise |
+| `tug_service_restarts_total` | counter | `node`, `project`, `service` | Container restarts since the container was created |
+| `tug_node_disk_free_bytes` | gauge | `node`, `path` | Free space on the Docker data and Tug state paths |
+| `tug_node_load1` | gauge | `node` | One-minute load average |
+
+The agent would read these from `docker stats --no-stream` and `docker inspect`, and control would drop series for services that no longer exist. A node that stops reporting keeps its last values, so alerts use `tug_node_poll_age_seconds` to tell fresh data from stale data.
+
+Why report through control: agents sit behind NAT and only dial out, so Prometheus often cannot reach them. Reporting through the existing authenticated channel needs no inbound port, no extra credential and no firewall change, and adds one scrape target regardless of node count. The cost is resolution: values are as fresh as the poll interval (`TUG_POLL_INTERVAL_MS`), and the status report grows with the number of services.
+
+The alternative is scraping an exporter on each agent directly over the operator's private network from [Networking](networking.md#bring-your-own-network). It gives finer resolution and keeps control out of the data path, but it requires a listener on every node, firewall rules that admit only Prometheus and a service-discovery list. It does not work where Prometheus cannot route to the node addresses, including nodes behind NAT. If both are ever wanted, the metric names above can be reused so dashboards do not change.
+
+### Health endpoints
+
+Status: proposed.
+
+Control has no unauthenticated route (there is no `/healthz`), so liveness is TCP reachability of the mTLS port and the container restart policy. An authenticated `GET /api/v1/readyz` with the `read` role would return 200 only when control can do its job: it can read the database, and the writer queue is not stuck (the operation at the head of the queue has not been running longer than a limit). Otherwise it returns 503 with the names of the failing checks and nothing more. An external uptime check would present a `read` credential.
+
+### Ingress metrics
+
+Status: proposed, with open questions.
+
+The managed Caddy would enable its metrics with the global option `servers { metrics }` in the generated Caddyfile (see `renderCaddyfile` in `src/ingress.ts`), served by Caddy's admin endpoint, which listens on `localhost:2019` inside the container by default. That gives request counts by status class and request duration histograms. To reach the same scrape target, the agent on the ingress node would read the admin metrics and include a filtered subset (request counts, status classes, latency buckets, certificate expiry per host) in its status report, and control would re-export it as `tug_ingress_*` series labelled by `node` and `host`.
+
+Open questions:
+
+- Whether re-exporting parsed Caddy metrics through the status report is worth the complexity, compared with publishing Caddy's admin metrics port on a private node address, firewalling it to Prometheus and scraping it directly.
+- Which certificate expiry signal Caddy exposes reliably across versions; the alert below assumes a `tug_ingress_certificate_expiry_timestamp_seconds` gauge that may have to be derived from the certificate files.
+- Label cardinality: one `host` label per route is bounded by the number of routes, but per-upstream labels are not planned.
+
+### Alerting
+
+Status: proposed.
+
+Tug would not bundle Prometheus or Alertmanager. A sample scrape configuration:
+
+```yaml
+scrape_configs:
+  - job_name: tug
+    metrics_path: /metrics
+    scheme: https
+    tls_config:
+      ca_file: /etc/prometheus/tug-ca.pem
+      cert_file: /etc/prometheus/tug-read.crt
+      key_file: /etc/prometheus/tug-read.key
+    static_configs:
+      - targets: ['control.example.com']
+```
+
+Sample alert rules, written against the proposed metric names:
+
+```yaml
+groups:
+  - name: tug
+    rules:
+      - alert: TugNodeStale
+        expr: tug_node_poll_age_seconds > 120
+        for: 2m
+        annotations: { summary: "Node {{ $labels.node }} has not reported for over two minutes" }
+      - alert: TugDeploymentFailed
+        expr: increase(tug_deployment_duration_seconds_count{outcome=~"failed|build_failed"}[10m]) > 0
+        annotations: { summary: "A deployment of {{ $labels.project }} failed" }
+      - alert: TugAutomaticRollback
+        expr: increase(tug_rollout_outcomes_total{outcome="auto_rollback"}[10m]) > 0
+        annotations: { summary: "{{ $labels.project }} was rolled back automatically" }
+      - alert: TugCertificateExpiresSoon
+        expr: tug_ingress_certificate_expiry_timestamp_seconds - time() < 14 * 86400
+        annotations: { summary: "Certificate for {{ $labels.host }} expires in under 14 days" }
+      - alert: TugDatabaseGrowing
+        expr: predict_linear(tug_db_size_bytes[6h], 7 * 86400) > 5 * 1024 ^ 3
+        annotations: { summary: "Control database is on track to pass 5 GiB within a week" }
+      - alert: TugAgentPollAgeHigh
+        expr: tug_node_poll_age_seconds > 30
+        for: 10m
+        annotations: { summary: "Node {{ $labels.node }} polls much slower than its interval" }
+```
+
+Thresholds are starting points. The last rule assumes the default poll interval and should be set from `TUG_POLL_INTERVAL_MS`.
+
+Outbound webhooks (M8) are the lightweight alternative for operators without Prometheus: control would POST a signed JSON body to configured URLs. The events worth sending are a deployment reaching `succeeded`, `failed`, `build_failed` or `cancelled`, a rollout candidate rejected, an automatic rollback, a node going stale and recovering, and an ingress node reporting `failed`. Webhooks send state changes, never metrics, and depend on event hygiene so that an event always means something happened.
+
+### Live commands
+
+Status: non-TTY container commands built in M5; agent/daemon logs and `top` remain proposed.
+
+`tug logs <service>` keeps its durable tail through the command queue, while `-f` follows the current container over the live channel. `tug exec`, `tug run`, `tug ps` and `tug inspect <service>` use that channel too. Agent and daemon log streams are still proposed for troubleshooting. These commands answer what is happening right now on a node and complement metrics, which answer what has been happening. An optional `tug top` would render the agent-reported stats as a refreshing table of CPU, memory and restarts per service, reading `GET /metrics` or a read-only JSON view rather than a new data path.
+
+### Test gate
+
+Status: proposed.
+
+Modelled on madplay's `@magplay/observability` package: metrics sit behind a small injectable interface whose default is a no-op, so unit tests of control and agent logic need no registry; a small closed outcome taxonomy (`success`, `failure`, `timeout`, `rejected`, `cancelled`) with one helper that maps an error to an outcome; and a CI check (`test:observability`) that fails when the code and the documentation disagree.
+
+For Tug the equivalent check would assert:
+
+- every metric registered in code appears in this page's metric tables with the same name, type and label set, and every row in the tables is registered in code;
+- every label name is on an allow-list, and none is `id`, `version`, `error`, `path` or `message`, so cardinality stays bounded;
+- `GET /metrics` with a certificate for an unknown key returns 401 and with a `read` credential returns 200, and `ROUTE_ROLES` lists the route;
+- a scrape after a credential is created and used contains no credential private key, join key or secret value;
+- with the no-op default, the existing control and agent tests behave unchanged.
+
+## Rollout order
+
+1. Event hygiene (done, [PR #13](https://github.com/wpelevator/tug/pull/13)). Independent of everything else.
+2. Structured logs and the `/metrics` endpoint on control with the control-side metrics and the readiness endpoint, plus the test gate so later metrics cannot ship undocumented.
+3. Agent-reported node and service stats, then ingress metrics once the open questions are settled.
+4. Sample alert rules (they need steps 2 and 3) and outbound webhooks.
+
+Dependencies: the remaining agent/daemon live log streams and TTY backend extend M5, while webhooks and the dashboard are M8 items. Steps 1 to 3 need neither.
+
+## Open questions and non-goals
+
+Open questions:
+
+- Where the first dashboard JSON would live: in this repository (for example a `dashboards/` directory) or published separately, and whether the M8 read-only web dashboard makes it unnecessary.
+- Whether agents should send stats on every poll or on a slower interval of their own.
+- Whether `/metrics` should accept the `root` admin credential as well as named `read` credentials (an admin passes every role check today).
+- How the ingress metrics reach control (see [Ingress metrics](#ingress-metrics)).
+- Whether control should keep a short history of stats for `tug top` and a dashboard without Prometheus.
+
+Non-goals: a log aggregation system (JSON on stdout is the interface), distributed tracing, and a bundled Prometheus, Alertmanager or Grafana. Tug would expose metrics and document how to scrape them; running the monitoring stack is the operator's choice.
+
+See also [Architecture](architecture.md), [Control plane](control-plane.md), [Networking](networking.md), [Testing](testing.md) and the [Roadmap](roadmap.md).