@wpelevator/tug 0.11.0 → 0.13.0

package/docs/getting-started.md

@@ -0,0 +1,75 @@
+# Deploy your first service
+
+Install Tug's control plane and agent on one Linux VPS, deploy nginx, and visit its welcome page.
+
+## Before you start
+
+You need a VPS with a public IP and SSH access as root, Node.js 22.18 or newer and Docker Compose v2 on your computer. The VPS must run Debian, Ubuntu, Fedora or a RHEL-compatible distribution if you want Tug to install Docker; otherwise install Docker Engine and the Compose plugin there first. The VPS needs outbound access to pull images. Allow your computer and the agent to reach control on TCP 8443, and allow your computer to reach TCP 8080 for this temporary example. Check your provider firewall as well as any host firewall.
+
+Replace `203.0.113.5` below with your VPS address. These commands assume an interactive terminal, where Tug explains and confirms the installation choices. For automation see [Setup in the CLI reference](cli.md#setup).
+
+## 1. Install Tug and control
+
+On your computer, create a directory for this project and install the CLI:
+
+```sh
+mkdir my-first-tug && cd my-first-tug
+npm install -g @wpelevator/tug
+tug setup root@203.0.113.5
+```
+
+Allow Tug to install Docker if necessary and confirm the plan. Setup installs control and imports a root operator credential to your computer. The credential has full administrative access; keep it private. Setup creates `tug.yml` for shared defaults and a local overlay pointing to the credential. Keep working in this directory: the CLI searches the current directory for configuration, not its parents. See [project and configuration selection](cli.md#arguments-options-and-configuration).
+
+```sh
+tug status
+```
+
+You should see a control-wide status summary. If it cannot connect, check the address printed during setup and TCP 8443; see [Troubleshooting](troubleshooting.md).
+
+## 2. Add the same VPS as a node
+
+```sh
+tug node add root@203.0.113.5 --name app1
+tug node list
+```
+
+Accept Docker installation if prompted, choose an address belonging to the VPS and confirm the plan. You can decline managed ingress for this example and enable it later when you add a hostname. A private interface address is useful when you add a second server. The agent connects out to control and should appear in the node list with a recent contact time. The control plane and agent now share the VPS.
+
+## 3. Deploy and visit a service
+
+Save this as `compose.yaml` in the same directory:
+
+```yaml
+services:
+  web:
+    image: nginx:1.27-alpine
+    ports:
+      - "0.0.0.0:8080:80"
+    x-tug:
+      health:
+        type: http
+        port: 80
+        path: /
+```
+
+```sh
+tug deploy
+tug status
+tug logs web
+curl http://203.0.113.5:8080/
+```
+
+The deploy should report success, status should show `web` serving on `app1`, and curl should return the nginx welcome page. The explicit `0.0.0.0` publishes the example on all IPv4 interfaces; restrict TCP 8080 to your computer in the provider firewall while trying it. For a real site, use [managed ingress and HTTPS](networking.md#layer-b-ingress-and-tls), remove this explicit public binding, and redeploy. Tug otherwise binds published ports to the node address or loopback; a public node address can still make them public. See [port exposure](networking.md#published-ports-private-by-default).
+
+With one node, omitted placement selects it automatically. The HTTP check verifies nginx responds before Tug marks it serving; see [Health checks](healthchecks.md). If the deploy fails or the page is unreachable, start with [Troubleshooting](troubleshooting.md).
+
+## Choose a next step
+
+- [Add a hostname and HTTPS](networking.md#layer-b-ingress-and-tls): enable ingress, point DNS at the VPS and route a service.
+- [Deploy an application and database](../README.md#your-compose-file-plus-a-little-x-tug): use a volume and a project secret, then explore the [WordPress example](../examples/wordpress/README.md).
+- [Back up and recover](backups.md): protect workload data and the control plane separately, then rehearse a restore.
+- [Add another node](service-placement.md): pin stateful services and spread a stateless service across servers.
+- [Choose health checks and safer rollouts](healthchecks.md): understand readiness, failures and rollback.
+- [Give CI or teammates access](access-control.md): create credentials appropriate for trusted operators.
+
+The service and control plane will keep running after this tutorial. Before retiring a host that contains data, follow [Retire a host](migration.md#retire-a-host); teardown can delete volumes and credentials. The [CLI reference](cli.md), [Compose reference](compose.md) and [configuration reference](configuration.md) describe all options.

package/docs/healthchecks.md

@@ -0,0 +1,51 @@
+# Health checks and safer deployments
+
+A health check tells Tug when a service is ready to serve. During a deployment, the agent checks the container and reports failures to control. Ingress uses serving endpoints, and a rolling deployment waits for a healthy replacement before switching traffic. If you omit `x-tug.health`, Tug does not run a Tug readiness probe for that service; a running container alone does not prove the application is ready.
+
+## Choose a check
+
+| Type | What it tests | Useful for |
+| --- | --- | --- |
+| `docker` | The container is running and, if present, its Docker `HEALTHCHECK` reports healthy. Without a Docker health check it only verifies the container is running. | Images that already define a meaningful health check. |
+| `tcp` | The agent can connect to a port on the container IP. | A database or protocol server when a port-open check is sufficient. It does not prove a query will succeed. |
+| `http` | An HTTP request to the container IP returns 2xx after redirects. | A web app with a dedicated readiness endpoint. |
+
+This example gives MariaDB a TCP check and nginx an HTTP check:
+
+```yaml
+services:
+  db:
+    image: mariadb:11.4.8
+    environment:
+      MARIADB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
+    x-tug:
+      health:
+        type: tcp
+        port: 3306
+  web:
+    image: nginx:1.27-alpine
+    depends_on:
+      db:
+        condition: service_healthy
+    x-tug:
+      health:
+        type: http
+        port: 80
+        path: /
+```
+
+Set `DB_ROOT_PASSWORD` with `tug secret set DB_ROOT_PASSWORD` before deploying. The nginx service in this example does not use the database. See the [WordPress example](../examples/wordpress/README.md) for an app that does.
+
+Checks default to 10 attempts, a two-second interval and a two-second timeout. Increase the interval or retries for a slow-starting application. HTTP checks use the container IP and port, not the public hostname or ingress; select an endpoint that returns 2xx without relying on public redirects. A process-only endpoint can pass while a database or migration is broken; decide which dependencies must be ready before traffic is sent. Conversely, a check that depends on a service stopped during restore can make its own service unhealthy during that restore. Field details are in the [Compose reference](compose.md#services).
+
+## Dependencies and failures
+
+`depends_on` orders services on the same node; `condition: service_healthy` waits for the dependency's `x-tug.health`, or its Docker health check if there is no Tug check. Without either, the fallback checks only that the dependency container is running. Applications still need connection retries, particularly across nodes or when a dependency later restarts. Rolling services have additional ordering limitations described in [Rolling rollouts](compose.md#rolling-rollouts).
+
+If a check fails, `tug deploy` reports the error and `tug status`, `tug events` and `tug logs <service>` help diagnose it. Check the container port and path, the application's startup time and the service logs. A successful TCP check does not verify credentials or application data. See [Troubleshooting](troubleshooting.md).
+
+## Roll forward safely
+
+The default rollout recreates a changed service, which can interrupt traffic. A stateless service behind managed ingress can opt into `x-tug.rollout.strategy: rolling`; Tug starts and checks the new revision before ingress switches to it, then drains the old one. Rolling requires ingress, an explicit Tug health check and exactly one published port. It is not a database failover mechanism, and overlapping revisions can share mounted volumes. Review [rolling requirements and limits](compose.md#rolling-rollouts) before enabling it.
+
+`tug rollback` returns a project to a previous deployment; top-level `x-tug: { rollback: auto }` opts into automatic rollback for eligible failures. It cannot undo database writes or schema changes. See [automatic rollback](compose.md#automatic-rollback) and plan compatible migrations separately.

package/docs/migration.md

@@ -0,0 +1,113 @@
+# Migrating to new servers
+
+How to move a Tug installation, or one tier of it, to new servers and retire the old ones. Tug has no single "migrate" command: a move is a short sequence of the ordinary commands (`node add`, `deploy`, `backup`, `restore`, `node remove`), and the order is what keeps your data safe.
+
+Each section says what was exercised on a real host and what comes only from reading the docs. Treat the second kind as a checklist to rehearse on disposable servers first.
+
+| Section | Status |
+| --- | --- |
+| [Before you start](#before-you-start) | Reference |
+| [Use labels so a move is only `node add`](#use-labels-so-a-move-is-only-node-add) | Placement behaviour is documented; the exact edit was not rehearsed |
+| [Move the web tier](#move-the-web-tier) | Not exercised in a live run |
+| [Move the database](#move-the-database) | Restore across nodes was exercised; the full move was not |
+| [Move the object store](#move-the-object-store) | Not exercised; Tug has no data move for it |
+| [Move the control plane](#move-the-control-plane) | Recovery on another host was exercised on 0.9.0, see [Control plane](control-plane.md#recover-control-on-another-host) |
+| [Retire a host](#retire-a-host) | Manual; `node remove` only removes the control-plane record |
+
+## Before you start
+
+- **What lives where.** A host holds nothing a rebuild cannot recreate, except named volumes, any data you keep outside Docker (an object store on an attached disk) and, on the control host, the control state. Your Compose file, `tug.yml` and the images you build are the source of truth for everything else.
+- **Escrow the secrets that cannot be regenerated.** Keep the root credential and the control backup's master key off the servers, and keep the backup repository password (`tug backup password`). Without them a lost control host cannot be recovered. See [Backups](backups.md).
+- **Take fresh backups first.** `tug backup run` for every target you are about to move, then `tug backup list --check` to confirm they succeeded. For control, `tug control backup --no-images` is enough for a migration, because build archives are rebuilt by deploying again.
+- **Trim old build archives** on control with `tug image prune --keep 3` (add `--dry-run` to preview) so the control backup and any transfer stay small. Never delete files under `/var/lib/tug-control/images/` by hand: the database would still list them and nodes would get a 404 when they try to download them.
+- **Private network.** New nodes need a private address on the same network as the old ones, because cross-node service names and published ports use it. See [Networking](networking.md).
+
+## Use labels so a move is only `node add`
+
+If a service pins explicit nodes (`x-tug.placement.nodes: [web-1]`), moving it means editing the Compose file. If it selects nodes by label, moving it only means adding a node that carries the label and removing the old one:
+
+```yaml
+services:
+  php-fpm:
+    x-tug:
+      placement:
+        labels: [web]
+```
+
+```bash
+tug node add root@NEW --name web-2 --address 10.0.0.9 --label web --install-docker -y
+```
+
+Nodes must carry all of a service's labels, and nodes whose agent has not been seen recently are skipped when labels are resolved. Keep explicit `nodes:` only where a service must stay on one named host, such as a database that owns a local volume. Details are in [Compose extensions](compose.md) and [Nodes](nodes.md#node-labels).
+
+`tug scale` stays within a service's explicit `placement.nodes`, so it is not a way to move a pinned service. Change the placement and deploy instead.
+
+## Move the web tier
+
+A web tier whose database and uploads live elsewhere is stateless, so it is the cheapest move.
+
+1. Add the new node with the labels the web services select, and `--ingress` if it should handle ingress.
+2. Run `tug deploy`. Services placed by label are scheduled onto the new node when the placement is re-resolved at deploy. If you pin explicit nodes, change them in the Compose file first.
+3. Check `tug status` shows the services serving on the new node.
+4. Remove the old node once nothing is placed on it: `tug node remove web-1`. It refuses while a current deployment places services there or while its agent still reports; `--force` overrides both.
+5. Then [retire the host](#retire-a-host).
+
+Ingress: certificates are issued per ingress node and live in that node's Caddy volumes, so a new ingress node requests new ones. If the public IP changes, point DNS at it first, and keep Let's Encrypt rate limits in mind if you use a shared wildcard service such as `sslip.io`.
+
+## Move the database
+
+This is the move that can lose data, so the order matters. Deploying a database to a new node creates its volume there **empty**. If the application is still running and can reach that empty database, it will happily write to it or fail in confusing ways.
+
+1. **Add the new node** on the same private network, with the label the database selects (or pin it by name).
+2. **Back up now:** `tug backup run db`, then confirm with `tug backup list --check`.
+3. **Change the database placement and deploy with `--move`:** `tug deploy --move db`. Without the flag Tug refuses the deploy, because the database would start empty on the new node. With it, the services that write to the database are held (stopped) so nothing connects to the empty one. Writers are found from `depends_on` and from host names in environment variables (`WORDPRESS_DB_HOST=db`); check them with `tug restore run db` without `--confirm`, which prints the plan.
+4. **Restore the snapshot into it:** `tug restore run db --confirm`, adding `--writers php-fpm` if the plan missed a writer. The restore takes over the holds, takes a safety snapshot of the current data first and starts the writers when it finishes.
+5. **Changed your mind?** `tug hold release` starts the held services without restoring. Only do that if the empty database is acceptable.
+6. **Verify** with the application and `tug backup list --check`.
+7. **Clean up the old volume.** `tug volume remove db-data --delete-data` removes the record and the Docker volume on its node, but it refuses while a service in the current deployment still references the volume. Check `tug volume list` and, if the old copy is still there, remove the leftover Docker volume on the old host by hand.
+
+If the restore fails before it changes any data, the writers stay held, so nothing connects to the empty database. Fix the cause and run the restore again; `tug hold release` is the way to start them without a restore.
+
+While a writer is held, a sibling service whose health check depends on it (an nginx check that requests a PHP page) reports a failed apply for the duration. Give such services a health check that does not depend on the writer, such as a static `/healthz`. See [Restore](backups.md#restore). If the writers never report held, the restore fails after five minutes and the error names the node, the writer and the apply error the node reported.
+
+Before relying on this procedure, rehearse it on throwaway servers with a copy of your data.
+
+## Move the object store
+
+An object store such as SeaweedFS or MinIO is a plain Docker volume to Tug. Tug does not copy volumes between nodes, so you choose how the data travels:
+
+- **Small stores:** give the volume an `x-tug.backup` entry so Tug archives its files with restic, move the service, and restore the volume with `tug restore run <volume> --confirm`. See [Backups](backups.md).
+- **Large stores:** use the store's own replication or a one-off copy (`rsync` of the data directory with the store stopped), then point the service at the new location.
+- **Disposable stores:** deploy a fresh one and re-upload, after `tug volume remove <volume> --delete-data` on the old node.
+
+Whatever you choose, the buckets your applications and backups use must exist before they are used, and Tug does not create them. Create them with the store's own tooling (for example `aws s3api create-bucket` through an SSH tunnel to the private port). The access keys that the application, and `tug backup destination set`, use must match what the new store was started with.
+
+## Move the control plane
+
+To recover a lost control server, see [Recover control on another host](control-plane.md#recover-control-on-another-host). To change only its address, see [Moving the control plane](control-plane.md#moving-the-control-plane). For a planned move:
+
+1. Take `tug control backup --no-images` on the old control and copy the archive off the host.
+2. Follow the recovery steps on the new host. Stop or isolate the old control before the new one serves, because two active control planes with divergent state are not supported.
+3. After the restore, repoint every node with `tug node replace <node> <ssh-target> --control-url … --agent-control-url …`.
+4. Deploy your projects again so control rebuilds the build archives it no longer has. Rolling back to a deployment whose archive was not restored is refused until you do.
+
+Workload data is not part of the control backup. Back it up and restore it separately as above.
+
+## Retire a host
+
+`tug node remove <name> --teardown` removes the node's record from control and then cleans the host over SSH (the target defaults to `nodes.<name>.ssh` in `tug.yml`). It refuses a node that a current deployment still places services on, stops the agent and ingress, removes the Tug containers and networks, and deletes `/etc/tug` and `/var/lib/tug`. Project volumes and ingress certificates are kept; add `--delete-data` to delete them too, only once the data has been moved or backed up. It asks for confirmation (`-y` to skip, required without a terminal) and refuses a host that runs the control plane. If the cleanup fails after the record was removed, run the same command again.
+
+Plain `tug node remove <name>` only deletes the record and leaves the host running.
+
+On a former control host remove the control stack and its data, which includes the master key and the private CA: take the backup first, then `docker compose -p tug-control down -v` and `rm -rf /opt/tug-control /var/lib/tug-control`. Remove Docker itself only if nothing else uses it.
+
+Locally, delete the host from `tug.yml`, and remove stale credentials under `~/.config/tug/credentials/` and any `TUG_CONTROL_URL` you exported.
+
+Names to look for when cleaning up by hand: containers are `tug-<project>-<service>-<hash8>`, networks `tug_<project>`, volumes `tug_v<length>_<project>_<volume>`, and Tug resources carry `tug.project` labels. Review what a command will match before running it; the volume and container commands above delete data.
+
+## Known limits
+
+- A move does not copy the data: you restore it from a backup, and the old volume is not removed for you.
+- A control host is retired by hand.
+- Scratch restore volumes and the safety snapshot of every in-place restore are kept. See the limits in [Backups](backups.md).
+- The control archive does not contain workload data.

package/docs/networking.md

@@ -0,0 +1,150 @@
+# Networking: cross-node service names and ingress
+
+Two layers let services on different hosts find each other and let the internet find them. Both use the private network you already operate; Tug does not create an overlay network. The control plane stays single-writer: it resolves endpoints and renders configuration, but never carries workload traffic.
+
+| Layer | What it adds | Turned on by |
+|---|---|---|
+| A. Cross-node service names | Services on other nodes resolve by name to their node's `address` (hosts-file entries, not DNS), only while healthy | On by default; set `TUG_SERVICE_NAMES=false` on control to opt out |
+| B. Ingress and TLS | A managed Caddy on an ingress node routes hostnames to services anywhere, with automatic HTTPS | `tug node ingress <node>` plus `x-tug.ingress` on a service |
+
+Independently of the layers, [published ports are private by default](#published-ports-private-by-default): Tug binds them to the address other nodes and the ingress use, never to every interface unless you wrote that.
+
+## Bring your own network
+
+A node's `address` is its sole inter-node address. Set it with `node add --address`, the node's `address` config key or `TUG_NODE_ADDRESS` at enrollment to a private IP on one of the host's interfaces. Every node that runs a client or an ingress upstream must be able to route to that IP. A hostname is also accepted, but it must resolve on the node to an address Docker can bind. Update a registered address with `tug node address <name> <address>`. Do not use a cloud VM's public 1:1 NAT address when that address is absent from its interfaces; register its private interface address instead.
+
+The network can be a VPC, LAN, existing VPN or another routed private network. Tug does not allocate inter-node addresses, create tunnels, configure routes or provide NAT traversal for workload traffic. Agents can still dial out to control through NAT, but cross-node service names and remote ingress upstreams work only where this workload network is reachable.
+
+**Tug does not encrypt node-to-node workload traffic.** Mutual TLS protects connections to control, not connections between application containers, published ports and ingress upstreams. Use application-layer TLS or an operator-managed encrypted network where the traffic needs confidentiality or integrity.
+
+Firewall the workload network so only the required peers can reach the published service ports. Allow the ingress nodes to reach their upstream ports, and allow application nodes to reach only the services they consume; there is no need to open every port between every node. Publish ports 80 and 443 on ingress nodes only as required by the configured HTTP and ACME flows. Keep databases and other internal services off public interfaces and use a provider firewall as a second boundary: Docker's iptables rules can bypass host firewalls such as ufw. An explicit `0.0.0.0` or `[::]` host IP in Compose is a request for all-interface exposure.
+
+A single-node project needs no inter-node network. When only the managed Caddy on the same node reaches an ingress port, Tug binds that port to loopback.
+
+## Layer A: cross-node service names
+
+Decision: `extra_hosts` injection, not a DNS service. The names are hosts-file entries written into containers at creation, so there is no DNS server, no round-robin answer and no update between reconciliations; "service DNS" is reserved for a possible future implementation. Names for services on the same node are Docker's own network DNS and are unrelated. Cross-node names add no long-running process and need nothing new on the nodes; the cost is that records only change when the agent re-renders the project, which already happens on every poll.
+
+Cross-node service names are enabled by default. Set `TUG_SERVICE_NAMES=false` on control to disable them. While enabled, the work response carries `serviceEndpoints`: for each project on the node, each service to the list of `{ node, address }` where
+
+- the service is placed on that node,
+- the node has polled within `TUG_STALE_SECONDS`, and
+- the node's latest status reports that service as `rolloutState: serving` (so a failing health check withdraws the record).
+
+`address` is the node's configured `address`. The agent resolves a hostname address on the client node before writing it into `extra_hosts`; endpoints that cannot resolve are omitted until resolution succeeds, so the resulting entry is a `name:ip` value; a hostname must therefore resolve on the node. The agent's container definition (`containerSpecs`) adds `extra_hosts` to every service on the node for every service that runs elsewhere, as both `<service>` and `<service>.<project>.tug`. Services placed on the same node keep resolving through the project network's Docker DNS, and entries you wrote yourself in `extra_hosts` win. A service on several nodes gets one entry per serving node.
+
+Consequences:
+
+- The name resolves to the node's address, not the container. The target service must publish its port (`ports: ["3306:3306"]`) and clients connect to the published host port. Tug binds that port to the same address the name resolves to ([Published ports](#published-ports-private-by-default)); the operator's network and firewall determine who can reach it. Rolling services are the exception, see [Rolling services and service names](#rolling-services-and-service-names).
+- `/etc/hosts` has no load balancing: most resolvers use the first entry, so multi-node services give failover on node loss rather than round robin. Real load balancing is a non-goal.
+- Records appear after the target reports `serving` and the client's node re-renders its container definitions on its next poll, which recreates the client container if its `extra_hosts` changed. Order dependent services with health checks and expect one extra recreate on first deploy.
+
+## Layer B: ingress and TLS
+
+Decision: Caddy, as one managed container per ingress node, configured by a Caddyfile that control renders.
+
+```yaml
+services:
+  web:
+    image: nginx:1.27-alpine
+    ports:
+      - "8080:80"
+    x-tug:
+      placement:
+        nodes: [app1]
+      ingress:
+        host: app.example.com
+        port: 80
+        tls: auto
+```
+
+An ingress hostname belongs to one project: a deploy or rollback is refused with `409` if another project's current deployment already routes the host (case-insensitive). `tug node ingress edge1` marks a node as an ingress node (`--disable` removes the role). `port` may be the container port or the published host port; control resolves it to the published host port, and `validateCompose` rejects an ingress port that is not published, because the edge reaches services on `<node address>:<published port>`.
+
+Caddy dials each upstream where its port is bound ([Published ports](#published-ports-private-by-default)): `127.0.0.1` for a service whose ingress port listens on loopback on the ingress node, otherwise the node's `address`, including for services on the ingress node itself. Remote addresses must be reachable from the ingress node over the operator's network.
+
+Control aggregates the `ingress` blocks of every project's current deployment, and for each ingress node renders:
+
+- one site block per host with `reverse_proxy` to every serving endpoint, using `127.0.0.1` for loopback-bound services on the ingress node and the node `address` for the rest;
+- no route at all for a host without a serving upstream (reported in `ingress.warnings` rather than serving errors), and only the first project for a host claimed twice;
+- with `TUG_ACME_EMAIL`, a global `email` for ACME registration.
+
+TLS modes:
+
+- `auto` (default): Caddy obtains and renews certificates with ACME. Ports 80 and 443 must be reachable from the internet and the host's DNS record must point at the ingress node. A name such as `app.<ingress-ip>.sslip.io` resolves to that address without a domain you own. Let's Encrypt limits certificates per registered domain, and `sslip.io` is shared, so a burst of new hostnames can be refused.
+- `custom`: `cert` and `key` name two project secrets holding the PEM certificate chain and private key (`tug secret set WEB_CERT --project blog --value "$(cat fullchain.pem)"`). Deploy fails if either secret is missing. Control decrypts them only for the ingress node, which receives them as files in the work response; they never reach the nodes running the service, and they never appear in the Caddyfile.
+- `off`: plain HTTP on port 80, for development.
+
+The agent on an ingress node creates the `tug-caddy` container with host networking and three named volumes (`/data` for ACME accounts and certificates, `/config`, `/etc/caddy`), so certificates survive container recreation and image updates. Config is uploaded with the Engine API archive endpoint (which works whether the agent runs on the host or in a container) and applied with `caddy reload`, which swaps config gracefully without dropping connections or restarting the container. If Caddy rejects the config the agent restores the last applied files, reports `failed` with Caddy's error in the node's ingress status, and retries on the next poll; `tug doctor` flags ingress nodes that are not `serving`. Removing the ingress role removes the container but keeps the volumes. The Caddy image is `caddy:2-alpine` (override with `TUG_CADDY_IMAGE` on the agent).
+
+For services with `x-tug.rollout.strategy: rolling`, the upstream port is the one the serving revision reports instead of the declared published port (see below).
+
+Caddy's automatic HTTPS protects the client-to-ingress connection. It does not encrypt the subsequent HTTP connection from Caddy to a service on another node; that upstream connection uses the operator's network.
+
+## Published ports: private by default
+
+A `ports` entry such as `"3306:3306"` would listen on every interface of the node, and Docker's iptables rules bypass host firewalls such as ufw, so the port would be on the internet unless a provider firewall blocks it. Tug therefore rewrites every `ports` entry that names no host IP in the container definition the agent builds for the node:
+
+| Situation | Bound to |
+|---|---|
+| The service's ingress port, when only the managed Caddy on this node dials it | `127.0.0.1` |
+| Otherwise | the node's `address` (a warning event when it is public) |
+| The node has no `address` | `127.0.0.1` |
+
+When a node `address` is configured, it is the same one other nodes resolve the service to and the ingress dials, so cross-node service names and upstreams keep working when the private network is reachable.
+
+"Only the managed Caddy on this node" means: this node is the only ingress node, the service runs on it, and every service of the project runs only on nodes where this service also runs, so no other node reaches it through a service name. That is the [small reference topology](roadmap.md#reference-topology) (app, database and Caddy on one node). Only the entry that publishes `x-tug.ingress.port` moves to loopback; other ports of the service follow the rest of the table.
+
+Details:
+
+- Short syntax gets the address prefixed (`10.0.0.8:3306:3306`, `10.0.0.8:5353:53/udp`, `10.0.0.8:8000-8010:8000-8010`); a container-only entry such as `"80"` becomes `10.0.0.8::80`, and an IPv6 address is bracketed. Long syntax gets `host_ip`.
+- An entry that already names a host IP is left exactly as written, including `0.0.0.0` and `[::]`. That is how you publish something on purpose: `"0.0.0.0:2222:22"`.
+- Control sends the binding in the work response (`ports`, `PortBinding` in `src/protocol.ts`): the address and, per project, the loopback services. A hostname `address` is resolved on the node, only when some port of the project binds to it; if it does not resolve the project fails to apply rather than falling back to every interface. The address must belong to one of the node's interfaces, otherwise Docker cannot bind it (`cannot assign requested address`).
+- With a public `address`, ports are bound to that public address and the agent sends one `ports.public` warning event naming the services (again whenever the set changes). Register a private `address`, or write the host IP in `ports` to make the exposure explicit.
+- Rolling revisions keep the host IP when their host port is remapped, so revision ports (`TUG_ROLLOUT_PORTS`) are bound the same way.
+- `tug doctor --nodes` runs a `ports.public` check on every agent: it lists containers of Tug projects (and their rolling revisions) listening on `0.0.0.0`, `::` or a public IP. It is a warning, not a failure, because an explicit `0.0.0.0` may be intended.
+
+## Rolling services and service names
+
+A service with `x-tug.rollout.strategy: rolling` ([Compose extensions](compose.md#rolling-rollouts)) is addressed through the ingress, and the design deliberately keeps service names stable while its port is not.
+
+Decision: each revision of a rolling service publishes the ingress port on its own host port from `TUG_ROLLOUT_PORTS` (default `20000-20999`) so old and new revisions can run at the same time. The agent reports the serving revision's port in its status (`serviceStates.<service>.port`), control puts it into the endpoint (`ServiceEndpoint.port`) and the Caddyfile upstream becomes `<node address>:<revision port>`. A new revision is therefore invisible to the ingress until it is healthy, and a failed one is never routed to because the node keeps reporting the old revision (still `serving`) with the old port.
+
+What this means for names:
+
+- `extra_hosts` entries are `name -> node address` with no port, so a rollout does not change them. Dependents on other nodes are never recreated by a rollout, and a rollout never restarts services that did not change. This is why per-revision ports are safe for `extra_hosts` injection even though a hosts file cannot carry ports.
+- The name of a rolling service on another node still resolves (to the node address, as long as that node reports the service serving), but clients dialing `web:8080` find nothing on the declared port, because rolling revisions do not hold it. Cross-node clients of a rolling service should use its ingress hostname. Giving rolling services a stable name and port would need another mechanism that owns the declared port; it is not implemented.
+- On the same node, services in the main project reach a rolling service by name over the shared project network: every revision joins `<project>_default` with the service name as an alias, so the name resolves to whichever revisions are running. During the overlap (start, health check and drain) that can be both revisions, and a not-yet-healthy new revision can receive same-node name traffic. Put anything that must not see an unhealthy revision behind the ingress.
+- Reaching a rolling service by its declared host port (for example `curl localhost:8080` on the node) does not work; the port moves with every rollout.
+- The injected `extra_hosts` are part of a revision's identity, so a rolling service rolls again when another node's service starts or stops serving. On a real first deploy `web` rolled once without the `client` entry and again as soon as `client` on the edge node became serving. The extra rollout is graceful, because the ingress only switches to a healthy revision, but it does start a new container.
+
+Ingress takeover timing: the new port reaches Caddy after the app node reports it (next poll), control renders a new Caddyfile and the ingress node polls and runs `caddy reload` (graceful, in-flight requests finish). The old revision is removed only after `drain`, which defaults to 30s, so there is no gap. Keep `drain` above the two poll intervals.
+
+## Sequence
+
+```mermaid
+sequenceDiagram
+    participant Op as Operator
+    participant Ctrl as Control plane
+    participant A as Agent app1
+    participant E as Agent edge1 (ingress)
+    participant Caddy as Caddy on edge1
+
+    Op->>Ctrl: node add app1 / edge1 with private address values
+    Op->>Ctrl: node ingress edge1
+    Op->>Ctrl: deploy (web on app1 with x-tug.ingress)
+    loop every poll
+        A->>Ctrl: GET /agent/v1/work
+        Ctrl-->>A: desiredStates + serviceEndpoints + port bindings
+        A->>A: reconcile containers with extra_hosts and private-address ports
+        A->>Ctrl: POST /agent/v1/status (web serving)
+        E->>Ctrl: GET /agent/v1/work
+        Ctrl-->>E: serviceEndpoints + ingress (Caddyfile, certs, routes) + port bindings
+        E->>Caddy: docker cp config, caddy reload (only if config hash changed)
+        E->>Ctrl: POST /agent/v1/status (ingress serving)
+    end
+    Note over Caddy,A: client request: Caddy terminates TLS, proxies over the operator's network to app1 address:8080
+```
+
+## Verifying
+
+`npm run test:integration:network` starts three privileged fake VPSes with plain numeric node `address` values (`vps-c` as ingress node), deploys `docker-compose.network-sample.yml`, and checks a container on `vps-b` fetching `http://web:8080` by its cross-node service name, Caddy on `vps-c` serving `app.test` from `vps-a`, and published ports bound to the intended private node addresses rather than every interface. See [Development](development.md#integration-test).

package/docs/nodes.md

@@ -0,0 +1,143 @@
+# Nodes
+
+A node is a registered Docker host on which Tug places services, and the agent is the container on it that fetches work from the control plane and manages the node's containers through the Docker Engine API. Nodes need Docker Engine only, not the Compose plugin. Add nodes either by pushing an install over SSH or by letting a fresh host enroll itself.
+
+Node operations that act on existing nodes can offer registered names when omitted. `tug node replace --select` and `tug node repair --select` choose several registered nodes with SSH targets in local config; `--all` continues to walk every configured node. Use `tug node address --to <address>` to choose the node while supplying its new address. See [interactive selection](cli.md#interactive-selection).
+
+## Node names and IDs
+
+A node has a **name** you choose (used in `x-tug.placement.nodes`, the CLI and the config file) and an **ID** that control generates (`node_` followed by random hex). The ID is immutable: it is never derived from the name, never changes when the identity is replaced, and is never reused. The node's certificate names its ID (`tug:node:<id>`), and every record control keeps for the node (status, commands, backup schedules and volumes) refers to the ID. When `node remove` frees a name, a node created later with that name gets a new ID, so certificates issued for the removed node can never authenticate the new one, even if the same key is installed again, and leftover records of the removed node do not attach to it. `tug status` and `tug events` show IDs next to names where a record carries one.
+
+## Install over SSH (`node add`)
+
+```bash
+tug node add root@app1.example.com --name app1 --address 10.0.0.11 --install-docker
+```
+
+The node generates its own key and certificate request; an admin credential authorizes the identity at that moment and the certificate is installed back over SSH, so no bootstrap secret travels and the private key never leaves the node's state volume. In order, the command runs SSH preflight checks, runs `agent identity` on the host (printing the CSR for the pending key in `/var/lib/tug/agent-pending.json`), posts it to `POST /api/v1/nodes/:name/identity` with `mode: create`, pipes the response into `agent install-cert` (which validates that the certificate names this node, chains to the CA in the response and matches the pending key before durably writing `/var/lib/tug/agent.json`), writes `/etc/tug/agent.env` and starts the agent container. Every step is safe to repeat: rerunning after a partial failure reuses the pending key, and `create` with the identical key is an idempotent retry that keeps the first call's metadata. No secret appears in an SSH command line or log. The command then waits for a status report from the new identity (`--no-wait` skips it).
+
+`--address` is the node's sole inter-node address: other nodes and the ingress use it to reach services, service names resolve to it, and published ports bind to it ([Published ports](networking.md#published-ports-private-by-default)). Give a private IP on one of the host's interfaces and make it reachable from the other nodes that need it. Control rejects an `--address` that is not an IP or hostname without a port. Tug does not create or encrypt this workload network; see [Bring your own network](networking.md#bring-your-own-network).
+
+Built application images are downloaded directly from control. Published images are pulled by their stored digest from upstream registries, with private credentials released through control when needed. No node registry login is required.
+
+Small hosts: a 1 GB VPS is enough for one site (a database and WordPress together use a few hundred MB), but not for much else. Ubuntu's `unattended-upgrades` and `apt-check` can run for hours on such a host and starve Docker of memory and CPU until commands time out; during testing one pegged a 1 GB node at load 16. Tug does not manage host updates; if the host is small, schedule them for a quiet window or disable the timers (`systemctl disable --now apt-daily.timer apt-daily-upgrade.timer`) and patch deliberately. A control plane and a node on the same host also compete for port 8443 if a service publishes it, which is why the WordPress example is not meant to share a host with control until [M15](roadmap.md#m15--flexible-node-roles).
+
+Preflight checks fail early, with a fix hint, unless the host has Docker Engine (no Compose plugin needed) accessible to the SSH user, writable `/opt`, `/var/lib` and `/etc/tug` paths, at least 1024 MB free under `/var/lib`, and can open a TCP connection to the control port. Preflight checks reachability only and copies no operator credential to the host; real TLS authorization happens when the agent first connects.
+
+The global `--control-url` changes only where the operator CLI connects; the server is still verified by its generated control name. When agents must dial a different address, pass `--agent-control-url`; it is written to the agent environment as `TUG_CONTROL_URL` and is likewise a connect target only. For a local SSH tunnel to control and a private endpoint for nodes: `tug --control-url https://127.0.0.1:18082 node add root@app1.example.com --name app1 --agent-control-url https://10.110.0.3:8443`. Both overrides must be `https://`.
+
+### Certificates, renewal and recovery
+
+Node certificates last 90 days. The agent renews at two thirds of the lifetime plus up to ten percent random jitter by sending a CSR for its existing key to `POST /agent/v1/renew`, saves the new certificate durably, then swaps its HTTP client and reconnects the stream. If renewal fails it keeps working on the old certificate until that expires. Authorization follows the key, so old certificates stay valid until they expire and routine renewal closes no connection. Control enforces a minimum interval between renewals per node (`TUG_RENEW_MIN_INTERVAL_MS`, default one hour).
+
+| Situation | Command |
+| --- | --- |
+| Cut a node off immediately, without SSH or deleting anything | `tug node disable <node>` (also removes it from service endpoints, ingress and label-based scheduling; explicit `placement.nodes` entries are left alone) |
+| Certificate expired, key lost or suspected compromised, node disabled, CA loss, or an older backup restored | `tug node replace <node> [ssh-target]` installs a new key (`mode: replace`); the old key's certificates stop working at once and their sockets close |
+| `agent.json` or `agent-pending.json` is corrupt | `tug node repair <node> [ssh-target]` moves the corrupt documents aside (never service volumes or agent event state) and replaces the identity |
+| Every node after a CA loss or older restore | recover root on the control-plane host (`control setup --recover-root`), then `tug node replace --all` |
+
+`node add --all` and `node replace --all` walk the SSH targets declared in `tug.yml` one node at a time, continue after failures and print a per-node result. Suspected key compromise is `node disable` followed by `node replace`. `node remove` deletes the node (its ID is retired for good) and revokes any unconsumed join key.
+
+Moving services to new servers and retiring old nodes is covered in [Migrating to new servers](migration.md).
+
+## Declaring nodes in the config file
+
+A repository can record its servers in the [config file](cli.md#config-file) so nobody has to remember `root@203.0.113.7`:
+
+```yaml
+# tug.yml
+nodeDefaults:
+  installDocker: true
+nodes:
+  app1:
+    ssh: root@203.0.113.7
+    address: 10.0.0.7
+    label: [web, ingress]
+```
+
+`tug node add --name app1` (or `tug node replace app1`) then runs as if `root@203.0.113.7 --address 10.0.0.7 --label web --label ingress --install-docker` had been typed; `tug node register app1` and `tug node enroll --name app1` read the same entry. Flags override the file key by key, and a `label` list on the command line replaces the file's list. An `<ssh-target>` argument overrides `ssh`; without one, a node with no `ssh` in the file is an error that names the missing key. The argument is never guessed to be a node name, because `app1` is also a valid hostname. `nodeDefaults` apply to every node, declared or not. Batch identity commands (`node add`, `node replace` and `node repair` with `--all`) use the same precedence for every supported key: explicit flags, then the node entry, then `nodeDefaults`, then command defaults. This includes `image`, `agentControlUrl` and `installDocker`; `false` and empty lists remain explicit values.
+
+To change a registered node’s address, run `tug node address app1 10.0.0.12`. Control wakes the agents so port bindings and service endpoints converge to the new address. The address must already be available on the node and reachable by its peers.
+
+The control plane stays the source of truth. Only `node add`, `node replace`, `node repair`, `node register` and `node enroll` read `nodes`, as flag defaults; `status`, `deploy`, `scale`, `ps`, `logs`, `exec`, `node remove` and every other command ask control and never consult the file. Re-running `node add` with the same key is an idempotent retry that keeps the first call's metadata; `node replace` installs a new key and keeps the node's ID, metadata and placement. It also updates the node's `address` when `--address` is supplied, with the same validation as registration. Editing the file without re-running anything changes nothing in control.
+
+## Self-service enrollment (`node enroll`)
+
+For hosts you cannot SSH to or do not want to touch: cloud-init and user-data provisioning, autoscaling and golden images, hosts behind NAT or in private networks, and delegated onboarding by someone without admin credentials. Mint a join key with an admin credential, then run the printed `docker run` on the fresh host or put it in user-data. The agent dials out to control.
+
+```bash
+tug node enroll --name app1 --address 10.0.0.11
+# prints, e.g.:
+#   docker run -d --name tug-agent --restart unless-stopped \
+#     --network host \
+#     -v /var/run/docker.sock:/var/run/docker.sock \
+#     -v tug-agent:/var/lib/tug \
+#     -e TUG_JOIN_KEY=<key> -e TUG_NODE_ADDRESS=10.0.0.11 \
+#     ghcr.io/wpelevator/tug:<CLI version> agent
+```
+
+The join key (`TUG_JOIN_KEY`) is base64url-encoded JSON `{ v: 2, credential, name, connectUrl? }`. The credential is a **bootstrap certificate** with SAN `tug:enroll:<id>` and its private key; it can reach `POST /agent/v1/enroll` and nothing else. The agent generates its own node key (into `agent-pending.json`), sends a CSR with the bootstrap certificate, validates and installs the returned node certificate exactly as `agent install-cert` does, and from then on ignores the join key. On restart the agent loads `/var/lib/tug/agent.json` and skips enrollment.
+
+Join keys are single-use, name-bound and short-lived (default one hour). There is no `--uses` and no reusable or unbound key, so golden-image and autoscaling flows mint one key per instance, for example into each instance's cloud-init. Name and labels come from the control-side row, never from the join key or the request. Enrollment is create-only: an existing name is refused and a join key can never replace a node. A retry with the same CSR key after a lost response signs another certificate and changes nothing stored. Minting again for a name revokes its previous unconsumed key, and `node remove` revokes it too. The join key is an **enrollment credential** and as sensitive as any bootstrap secret: it contains a private key, a saved user-data document contains a credential, and in cloud user-data it may be readable from the provider's metadata service by processes on the host. Single use, short life, name binding and create-only enrollment limit the damage.
+
+Options:
+
+- `--name <node>` (required) binds the node name.
+- `--address <addr>` supplies the private node address to the generated agent command and cloud-init. When omitted, set `TUG_NODE_ADDRESS` on the agent before its first enrollment if cross-node networking is needed.
+- `--agent-control-url <url>` sets `connectUrl`, a connect-only target; verification still uses the control name from the pinned CA.
+- `--ttl <seconds>` sets the key lifetime (default `3600`).
+- `--label <label>` sets node labels (repeatable).
+- `--image <tag>` sets the image in the printed command (default `ghcr.io/wpelevator/tug:<CLI version>`).
+- `--json` prints machine-readable output.
+- `--install-docker` prints a complete cloud-init document that installs Docker Engine before starting the agent. With `--json`, the result includes both `command` and `cloudInit` fields.
+
+To keep the key out of shell history and `docker inspect`, mount it as a `0600` file and set `TUG_JOIN_KEY_FILE=/path/to/key` instead of `TUG_JOIN_KEY`. Without an identity or a join key the agent refuses to start and points to `tug node add`, `tug node enroll` or `tug node repair`.
+
+## Bare-host Docker installation
+
+`control setup --install-docker` and `node add --install-docker` run the same Docker bootstrap over SSH before Docker preflight or runtime-image transfer. Without the flag, Docker Engine must already work (and the Compose plugin on the control-plane host, which runs the control container with Compose). The SSH target must be root for installation; Tug does not modify group membership or configure passwordless sudo.
+
+For a host created by your cloud provider, save `tug node enroll --name app1 --address 10.0.0.11 --install-docker` as user-data. Its stdout starts with `#cloud-config`, includes the bootstrap script and starts the agent only after Docker installation succeeds. The default version-matched Tug runtime image must be accessible from the host; an overridden development `tug:local` image must already be loaded there. Join keys are short-lived and should be treated as credentials when saving user-data.
+
+Bootstrap requires systemd and supports these releases:
+
+| Family | Releases | Docker repository |
+| --- | --- | --- |
+| Debian (apt) | 12 `bookworm`, 13 `trixie` | [Debian](https://docs.docker.com/engine/install/debian/) |
+| Ubuntu (apt) | 22.04 `jammy`, 24.04 `noble`, 26.04 `resolute`, 26.10 `stonking` | [Ubuntu](https://docs.docker.com/engine/install/ubuntu/) |
+| Fedora (dnf) | 42, 43, 44 | [Fedora](https://docs.docker.com/engine/install/fedora/) |
+| RHEL (dnf) | 9, 10 | [RHEL](https://docs.docker.com/engine/install/rhel/) |
+| Rocky Linux, AlmaLinux, CentOS Stream (dnf) | 9, 10 | [CentOS](https://docs.docker.com/engine/install/centos/) |
+
+It adds Docker's official apt or dnf repository, installs Engine and Buildx, and enables Docker (the control-plane host also gets the Compose plugin, because `control setup` runs the control container with Compose; nodes do not). A working Engine installation is preserved. If only Compose is missing, it installs the plugin without upgrading Engine. An inaccessible existing daemon, unsupported OS or release, or conflicting distribution packages (`docker.io` and `podman-docker` on Debian/Ubuntu; `docker`, `moby-engine` and `podman-docker` on RPM hosts) produce a clear error for manual resolution; Tug does not remove them or alter Docker daemon trust settings. Other distributions, such as Alpine or Arch, and hosts without systemd need Docker installed manually.
+
+## Image
+
+`node add` and `node enroll` default `--image` to `ghcr.io/wpelevator/tug:<version>`, where `<version>` is the installed CLI's own package version, so the CLI and the agents it installs stay on the same release. Pass `--image <tag>` to run another image, for example `ghcr.io/wpelevator/tug:edge` for the latest build of `main` or a mirror in your own registry.
+
+`node add --build-local-image` is for development: it builds the image from the current checkout, copies it to the host with `docker save` / `scp` / `docker load` and runs it, tagging it `tug:local` unless `--image` is also given. It needs a Tug checkout (the Dockerfile) in the working directory. See [Releases](releases.md) for how images are published. The image is built for the target host's architecture (read with `uname -m` over SSH), so an arm64 laptop can install an amd64 VPS; x86_64 and aarch64 hosts are supported.
+
+## Node labels
+
+`node add`, `node enroll` and `node register` accept a repeatable `--label <label>`. Labels are plain string tags stored on the node and listed by the node list API; services select nodes that carry all of their `x-tug.placement.labels` (see [Compose extensions](compose.md#services)). They are set when the node is created (`node add`, `node register --mode create` or the enrollment credential); `node replace` keeps them. They are unrelated to Docker container labels.
+
+## Agent host networking
+
+The agent always runs with host networking (`--network host`) because it dials container IPs directly for TCP and HTTP health checks, which a bridge-networked agent cannot reach on current Docker (29.x). On Docker Desktop the daemon and container addresses live inside its VM, which is why the agent belongs in a container on that daemon rather than in a plain host process. Host networking needs no added Linux capability. Inter-node workload reachability is a separate concern owned by the operator's network; see [Networking](networking.md#bring-your-own-network).
+
+## CSR in, certificate out (`node register`)
+
+`tug node register <name> --csr node.csr [--mode create|replace]` is the primitive behind the SSH commands: it posts a CSR you generated on the node (for example with `agent identity`) to the identity endpoint and prints the certificate, CA and public URL. Use it for local development or when you drive the steps yourself; install the output with `agent install-cert`.
+
+## Status and diagnostics
+
+Nodes are `online` while they contact control and become `stale` after `TUG_STALE_SECONDS` (default `60`) without an authenticated work fetch, status report or wake wait. Agents keep an outbound WebSocket open so control can wake them for deployments and commands; if WebSockets are unavailable they use an HTTP long poll, and `TUG_POLL_INTERVAL_MS` remains the periodic reconciliation fallback. `tug status` lists nodes, deployments and per-node project status; `tug doctor --nodes` (or `--node <name>`) runs Docker Engine checks, plus Caddy checks on ingress nodes, through each agent and waits for the result; the wait defaults to 60 seconds. Online and offline are connectivity states only: a `stale` node has not contacted control recently, which does not prove the machine is off. `tug doctor` also reports each node's certificate expiry and, for nodes that are reporting now, the observed clock skew (see [Doctor](#doctor-certificate-expiry-and-clock-skew)).
+
+### Doctor: certificate expiry and clock skew
+
+`tug doctor` always reports the control CA and server certificate expiry and each node certificate's expiry, with warnings when renewal is overdue (CA under 90 days, server certificate under 30, a node certificate under 14) and a failure for an expired certificate. For nodes it distinguishes **observed** clock skew from **unknown**: agents send their clock with every status report, control records the difference, and doctor shows it only for a node that is online now. A node that is stale, has never reported or is disabled has an unknown clock, not a zero skew. The figure includes request latency. A skew of 30 seconds or more is a warning; five minutes or more fails, because certificates are back-dated five minutes and a node that far ahead or behind cannot validate new certificates (fix time synchronization on the host).
+
+## Configuration
+
+See [Configuration](configuration.md#agent) for agent environment variables.