@wpelevator/tug 0.12.0 → 0.13.0

package/docs/control-plane.md

@@ -0,0 +1,112 @@
+# Control plane
+
+## Install over SSH
+
+```bash
+tug control setup root@control.example.com
+```
+
+The address clients use by default comes from `controlUrl` in the CLI config file (`--config <file>` selects one), including any overlay override. Without it, the address is the SSH target's host plus the published port (`https://control.example.com:8443`), which `control setup` prints. Pass `--control-url` (or set `TUG_CONTROL_URL`) to override either default, for example when you install through an SSH alias or a private address but clients reach control elsewhere. The credential URL is never used as an installation address. A configured URL keeps its public port even when `--port` publishes a different Docker host port. It is only an address: clients verify the generated control name, not this URL. `control setup` runs SSH preflight checks on the host (Docker, Docker Compose, writable Tug paths, at least 2048 MB free under `/var/lib`, the mTLS port free), then starts the `control` container from `src/templates/control-compose.yml`, publishing one port (default `8443`). Add `--install-docker` to install Docker Engine and Compose first on a supported Debian, Ubuntu, Fedora and RHEL-compatible host.
+
+On a fresh install it then issues the **root** credential on the host (`docker exec ... control issue-credential --name root --role admin` over SSH) and imports it straight into the local config as a `0600` file (`--credential-out <file>` optionally writes an extra copy; there is no default file); it never lands in container logs or a permanent file on control. Upgrading by running `control setup` again preserves credentials and never reissues root. If the root credential is lost, `tug control setup <ssh-target> --control-url https://control.example.com:8443 --recover-root` replaces only root's key without the API, closes its sessions within a second and audits the recovery; other admins are unaffected.
+
+The control container runs the published image `ghcr.io/wpelevator/tug:<CLI version>` (the installed CLI's own package version, so control and the CLI match). Pass `--image <tag>` to use another tag, or `--build-local-image` to build the image from the current checkout and copy it to the host over SSH with `docker save` / `scp` / `docker load` (tagged `tug:local` unless `--image` is given); the latter is for development and needs the repository's Dockerfile in the working directory. See [Releases](releases.md). Running `control setup` again over an existing control upgrades it in place: the port preflight accepts the port when the existing `tug-control-control-1` container holds it, and still fails when anything else does.
+
+Built images always travel as compressed archives over the authenticated control connection. Published `image:` services pull directly from their upstream registry by the digest resolved on control before deployment publication.
+
+`control setup` (also available as `tug setup <ssh-target>`) imports the root credential into the local config automatically, the same way `tug init` does, so no separate step is needed. In a terminal it shows a plan and asks for confirmation (`-y, --yes` skips it) and offers to install Docker when it is missing. `--no-import` skips the import. To import a credential from elsewhere (for example one issued on another machine), run:
+
+```bash
+tug init root.credential.json
+```
+
+`tug init` copies the credential (a file, or `-` for stdin) into a `0600` file under `~/.config/tug/credentials/` and points the git-ignored `tug.local.yml` at it. CI passes `TUG_CREDENTIAL` (the encoded credential) or `TUG_CREDENTIAL_FILE`; precedence is `--credential <file>`, then the environment (setting both is an error), then the overlay's `credential` path. Tracked configuration may not name a credential.
+
+A release publishes the npm package a few minutes before its container image. A freshly updated CLI therefore fails `tug setup` and `tug node add` before touching the host, with a message that the image is not published yet; retry shortly, or pass `--image ghcr.io/wpelevator/tug:latest`.
+
+## Authentication and PKI
+
+Control listens on **one** TLS 1.3 port that requires a client certificate from every peer. There is no plain HTTP listener, no unauthenticated route (not even a health check: monitor TCP reachability separately from authenticated readiness) and no bearer token. Operators, CI and nodes all use certificates signed by control's private CA; see [Access control](access-control.md) for identity classes, roles and key-based authorization.
+
+- **CA.** A self-signed P-256 CA (10-year lifetime) is created only for a fresh installation and kept at `<data dir>/pki/ca.key` (`0600`) and `ca.crt`. Missing, corrupt or mismatched CA material alongside existing state is a startup error. CA rotation is out of scope; replacing the CA means redistributing trust and credentials.
+- **Control name.** Control has a generated name, `control-<first 12 hex of the CA key hash>.tug`, derived from its CA. It is stable for the life of the CA, needs no configuration and survives backup and restore with `pki/`. Clients recompute it from the CA they pin and verify the server against it, so verification never depends on the address they dial.
+- **Server certificate.** SANs are the control name, the host of `TUG_CONTROL_URL` (required, a credential-free `https://` origin including the published port; the default address written into credentials) and optional `TUG_EXTRA_SANS` (comma-separated DNS names and IPs). The server certificate lasts 90 days, is renewed at two thirds of its lifetime while running, and the TLS context reloads without a restart. Control warns when the CA nears expiry.
+- **Profiles.** Control assigns every certificate: it verifies CSR signatures and P-256 keys and never copies requester-supplied subjects, SANs, validity or extensions. Leaves are `CA=false` with one assigned identity URI, back-dated five minutes and capped by the CA's validity.
+- **Connections.** Chain validation rejects missing or foreign certificates during the handshake, before any HTTP is parsed. TLS session resumption is disabled and any reused session is closed, so every connection presents its certificate. A post-handshake gate then checks the database, and every request and WebSocket upgrade checks it again. Disable, replace, remove, revoke, root recovery and certificate expiry close the affected sockets (API-originated changes immediately, host-side changes within a second).
+- **Clients.** The CLI and agent pin the credential's CA and verify the control name derived from the CA. `--control-url` / `--agent-control-url` change only the TCP connect target (for example `127.0.0.1:18082` through an SSH tunnel), and the credential's stored URL is just the default target. Moving control to a new IP or DNS name therefore invalidates no credential; see [Moving the control plane](#moving-the-control-plane).
+
+Deployment boundaries: TCP passthrough works. TLS termination ahead of control does not (it breaks the client certificate), and TLS interception fails certificate validation. Browsers need explicit CA trust and an imported client credential; browser setup is out of scope. Control's mutual TLS does not encrypt workload traffic between nodes; that traffic uses the operator's network ([Networking](networking.md#bring-your-own-network)).
+
+### Backups and authorization history
+
+`control backup` includes `pki/` (except the regenerable server certificate) and the key bindings. Restore validates the CA before changing live state, closes live sessions and requires a restart. Because authorization follows keys, a snapshot older than a certificate renewal does not lock a node out. An older snapshot still forgets nodes and operators created after it, restores keys that were replaced afterwards, and revives principals disabled or revoked afterwards. After an older restore: recover root on the host if needed (`--recover-root`), reapply revocations and disables, run `tug node replace` for nodes whose key changed and `tug node add` for nodes created after the snapshot (`tug node replace --all` after a CA loss).
+
+## Image delivery without a registry
+
+Remote builds save immutable images under `<data dir>/images/<build ID>.tar.gz`. Agents download only the images assigned to them through `GET /agent/v1/images/:buildId`, verify the compressed SHA-256 and byte count, validate the saved image metadata and load the archive. The agent loads those artifacts without a registry pull. No registry certificate, daemon trust change or inbound connection to a node is needed; downloads use the same mTLS connection as every other agent request.
+
+`tug deploy --build-local` builds application images on the operator machine and uploads archives instead of source. Control validates and imports them, gives them immutable tags and serves them to assigned agents. Builds are single-platform Linux images (amd64 or arm64); use a Docker builder matching target nodes. Agents reject an incompatible image platform before starting it. This flag is distinct from the installers' development-only `--build-local-image`, which builds the Tug runtime itself.
+
+All published desired-state history is retained, including rollback and lagging-node references. Its image archives remain available until you run `tug image prune`, which deletes the archives of everything except the newest `--keep N` (default 5) deployments of each project, the current deployment and any version a node still reports; `--dry-run` lists what would go. The desired states stay in history, but rolling back to one whose archive was pruned is refused and the service must be deployed again. Run it to reclaim disk, since each source build adds a full compressed image under `images/`. Cleanup removes failed unpublished builds and orphan archives. Agents prune only their recorded, unused Tug images absent from their authorized retained set and never run global Docker prune. Disk usage therefore grows with retained successful builds. Each transfer sends a complete compressed image, with no layer reuse.
+
+## Published images and private registries
+
+Control resolves each distinct published image tag once per deployment through the registry HTTPS manifest API and stores its SHA-256 digest in both desired service state and canonical Compose. Multi-platform images retain their complete index digest, allowing each node to select its architecture from that same immutable index. An explicit SHA-256 pin is validated and kept without a network lookup. If resolution fails, the deployment fails before replacing current desired state. Rollback and scaling reuse the stored pins without resolving tags again. Registries must retain pinned manifests and layers for rollback and fresh-node pulls; Tug does not mirror published images.
+
+Private registry credentials live in the encrypted project secret `TUG_REGISTRY_AUTH`, a JSON object keyed by registry host (including a port when applicable):
+
+```json
+{"ghcr.io":{"username":"ci-user","password":"pull-token"},"docker.io":{"username":"hub-user","password":"hub-token"}}
+```
+
+Store it using `tug secret set TUG_REGISTRY_AUTH --project site` with the value on stdin. Use `docker.io` for Docker Hub, including short image names. HTTPS registries with Basic or Bearer authentication are supported; credential helpers and plaintext registries are not. Use credentials with pull access only. Control decrypts the secret for resolution; assigned agents receive only credentials for their published image hosts through the authenticated secrets endpoint and pass them directly as Engine API `X-Registry-Auth`. Credentials stay out of desired state, container environments and Docker login files; referencing the reserved secret in a service definition is rejected. Secret updates apply to future pull requests, including retries and rollback; keep access to retained image digests when rotating credentials.
+
+Tug issues one node identity (key and certificate) per node. It does not push built images, manage registry users or log agents into registries. Teams building images in CI can push them themselves and deploy with `image:`.
+
+## Backup and restore
+
+`tug control backup` takes a consistent SQLite snapshot (including operator credential records) and writes a `0600` `.tar.gz` (in a `0700` directory) with `tug.db`, `master.key`, `pki/` (the private CA), retained `images/` archives (leave them out with `tug control backup --no-images`; deploy again to rebuild them after a restore) and metadata to `TUG_CONTROL_BACKUP_DIR` (default `<data dir>/control backups`). `tug control restore <artifactRef> --confirm` overwrites the database, master key and image store from such an archive; restart control afterwards. SQLite snapshot creation and restore integrity checking run synchronously and pause control API traffic, WebSocket handling and timers. Duration grows with data size and storage speed; the synthetic benchmark and reference measurements are in [Testing](testing.md#sqlite-contract-and-blocking-measurements). Take larger backups outside latency-sensitive rollout windows and keep project lease durations above expected blocking time. Restore validates through a read-only connection before replacing live data, then rejects database-dependent requests with HTTP 503 until restart. This is the control-plane backup: it covers disaster recovery of the control plane only and does not capture workload volume contents. Back up project data (volumes, databases) with `x-tug.backup` as described in [Backups](backups.md), which also shows a scheduled `control backup`.
+
+Running containers keep serving while control is down; agents just cannot pick up new desired state until it is back.
+
+### Recover control on another host
+
+For a planned move of the whole installation, including workloads, see [Migrating to new servers](migration.md).
+
+This procedure was exercised with the published 0.9.0 release on two VPSes; see [Real-host evaluation](real-host-evaluation.md#recovery-drill-with-the-published-090-release-2-november-2026) for the exact scope and limitations. Restore the workload data separately using its project backups. Keep the control backup encrypted off-host and keep the original operator credential and backup repository passwords separately; the control archive contains the master key and private CA.
+
+1. Stop or isolate the old control plane before directing agents to the replacement. Two active control planes with divergent state are not a supported failover setup. Running workload containers continue serving while control is unavailable.
+2. Install the matching Tug release on the replacement host with `tug control setup`, writing its temporary bootstrap credential to a private file. This initializes a new control and CA solely to authorize the restore.
+3. Copy the decrypted control archive into the replacement control data directory with mode `0600`. The standard installer mounts host `/var/lib/tug-control` as `/data`: for example, a host file `/var/lib/tug-control/restore-input.tar.gz` is addressed to the API as `/data/restore-input.tar.gz`. The archive path is read on the control host; the CLI does not upload a local file.
+4. Run `tug control restore /data/restore-input.tar.gz --confirm --credential <bootstrap-credential>`, then restart the control container. Restore replaces the bootstrap CA, credentials, state and image store; the temporary credential no longer authenticates after restart.
+5. Connect with the original operator credential and use `--control-url https://<replacement-host>:8443` if the address changed. Verify the recovered projects, deployments and backup catalog. Agents that still point to the old address must be repointed; the tested 0.9.0 path was `tug node replace <node> <ssh-target> --control-url https://<replacement-host>:8443 --agent-control-url https://<replacement-host>:8443 --credential <original-credential>`. Verify that each node returns online and that the site serves the expected data.
+
+A snapshot older than later node or operator changes does not contain those changes; follow the identity and revocation recovery notes in [Authentication and PKI](#authentication-and-pki). This sequence does not replace a separate, tested backup of workload volumes or databases.
+
+## Moving the control plane
+
+Because clients verify the generated control name, the address is only a default:
+
+1. Move or re-address the host and set the new `TUG_CONTROL_URL` on control (or rerun `control setup` with the new `--control-url`). Control adds the new address to its certificate on restart.
+2. Until credentials are refreshed, point clients at the new address: `tug --control-url https://NEW:8443 ...` or `TUG_CONTROL_URL`, and for an agent set `TUG_CONTROL_URL` in `/etc/tug/agent.env`.
+3. Optionally reissue credentials so they carry the new default: `control setup --recover-root` for root, `credential create` for others, `node replace` for nodes.
+
+No credential becomes invalid because the address changed. The control name only changes if the CA itself is replaced, which requires redistributing every credential.
+
+## Lost or compromised something
+
+| Situation | What to do |
+| --- | --- |
+| Lost the root credential | `tug control setup <ssh-target> --recover-root` |
+| A CI credential leaked | `tug credential revoke <name>`; live sessions close within a second |
+| A node is suspect | `tug node disable <node>` now, then `tug node replace` |
+| A node's identity is corrupt or expired | `tug node repair` or `tug node replace` |
+| Restored an older control backup | Recover root if needed, reapply revocations, then `node replace --all` ([Control plane](control-plane.md#backups-and-authorization-history)) |
+
+## Access control
+
+Besides the `root` admin credential, control accepts named operator credentials with `read`, `deploy` or `admin` roles, optionally limited to projects, for CI and teammates. Every deployment and control event records the acting credential name. See [Access control](access-control.md) for the endpoint table, the audit trail and a GitHub Actions deploy example. Control stores each credential's name, role, scope and public-key hash (never a private key); these records travel with control backups.
+
+## Configuration
+
+See [Configuration](configuration.md#control-plane) for control environment variables.

package/docs/development.md

@@ -0,0 +1,140 @@
+# Development
+
+The source is TypeScript. Development and tests run the `.ts` files directly through Node's built-in type stripping, so there is no build step, but it needs Node >= 22.18 (`engines.node`); development and CI use Node 24 (`.nvmrc`). SQLite uses built-in `node:sqlite`; installation requires no SQLite npm addon, Python or C++ compiler. The module is marked active development on Node 22.18.0 and can emit an experimental warning on supported Node releases; warnings are not globally suppressed. Compilation to `dist/` happens when building the Docker image and when packing the npm package (`prepack`).
+
+```bash
+npm install
+npm test
+npm run typecheck
+npm run lint
+npm run format
+npm run build
+npm run test:smoke:dist
+```
+
+Code style is enforced by Prettier (`.prettierrc.json`: 100 columns, single quotes, trailing commas; indentation comes from `.editorconfig`). `npm run lint` runs ESLint and then `prettier --check .`, and CI runs it, so an unformatted file fails the build. `npm run format` rewrites files with Prettier and then applies ESLint's autofixes. `eslint-config-prettier` switches off the ESLint rules that would fight the formatter, so ESLint only reports correctness problems. The one style rule it adds is `max-depth` (5) with `no-else-return`: early `return` guards are the house style, but a loop or state machine whose body needs several `continue`/`break` exits, or nests deeper than that, should be split into a function per item or per state that returns early instead (see `serviceTarget` in `src/backup-config.ts` and the `advance*` handlers in `src/backup-restore.ts`). `no-continue` is deliberately off: a one-line `continue` guard at the top of a short loop is fine. Prettier covers TypeScript, JavaScript and JSON; Markdown, YAML, the install templates and generated files are listed in `.prettierignore`. Prettier moves a trailing `// eslint-disable-line` comment when it rewraps a statement, so put suppressions on their own line above the code they cover (`// eslint-disable-next-line`).
+
+`npm test` runs `node --test 'src/**/*.test.ts'`. `npm run typecheck` runs `tsc --noEmit` over `src/` and `tests/` with `strict` on. `npm run build` compiles `src/` to `dist/` (JavaScript plus `.d.ts`) and copies `src/templates` to `dist/templates`. `npm run test:smoke:dist` builds and checks that `control setup` and `node add` render their templates from `dist/`.
+
+Import specifiers in the source use the `.ts` extension (`import { x } from './util.ts'`) because Node's type stripping does not remap `.js` to `.ts`; `tsc` rewrites them to `.js` in `dist/` (`rewriteRelativeImportExtensions`). Only erasable TypeScript syntax is allowed (`erasableSyntaxOnly`: no enums or constructor parameter properties).
+
+Unit tests sit next to the source files as `src/*.test.ts`. The agent integration test lives in `tests/integration/`. [Testing](testing.md) explains how the test layers fit together and what each one is expected to catch.
+
+## Docker image
+
+The runtime Dockerfile is multi-stage: a build stage runs `npm ci && npm run build`, and the final image ships `dist/` plus production dependencies only, with entrypoint `node dist/main.js`. `tsc` does not copy `src/templates/*`, so `npm run build` copies them next to the emitted `dist/cli.js`; the CLI resolves them relative to the running file, so `node src/cli.ts` and `node dist/cli.js` both work.
+
+The runtime image ships the Docker client (control uses it to load, tag and remove image archives through the mounted socket) and the compiled `tug-runtime` entry (`dist/main.js`). It deliberately has no Compose plugin: agents drive the Docker Engine API, and the control-plane host runs the control container with the host's own Compose (`control setup` checks for it). It is based on `node:24-alpine` (musl), which keeps the image about half the size of a Debian one; the Docker client comes from Alpine's `docker-cli` package, so the Docker daemon, containerd and runc are not installed.
+
+Run `npm run test:smoke:image` to build the actual shipped runtime image and execute its Docker client and Node binaries, the compiled control/agent database initialization (a writable snapshot and read-only validation) and `tools/scripts/verify-mtls-runtime.js`, which issues certificates with `@peculiar/x509`, serves mTLS with the production TLS options and calls it through the undici transport, including the loopback connect-target override that still verifies the generated control name. Nothing from the host socket or CLI plugins is mounted. To test an already-built image, set `TUG_RUNTIME_IMAGE=<tag>`; to check another Dockerfile, set `TUG_RUNTIME_DOCKERFILE=<path>`. CI runs this packaging smoke before the integration test, so the fake-VPS image cannot mask missing runtime tools.
+
+## Continuous integration and releases
+
+`.github/workflows/ci.yml` runs on pushes to `main` and on pull requests: typecheck and lint on Node 24 (the version in `.nvmrc`), unit tests and the dist smoke test on the exact minimum Node 22.18.0 and current Node 22 and 24, plus the Docker integration tests (fake VPS containers, registry-free image delivery, plain-address cross-node networking and rolling rollouts) as separate jobs. `.github/workflows/release.yml` versions and publishes; see [Releases](releases.md). Every user-facing change needs a changeset (`npm run changeset`).
+
+### CI caching
+
+The `.github/actions/setup` composite action installs Node.js with the npm download cache (keyed on the lockfile) and runs `npm ci`; every job uses it. The integration jobs prebuild their images with `.github/actions/build-images` (Buildx and the GitHub Actions layer cache, scopes `runtime` and `fake-vps`) and set `TUG_COMPOSE_BUILD=` so the integration scripts run `docker compose up` (on the CI host, to start the test stack) without `--build`; the Compose files tag those images `tug-runtime:integration` and `tug-fake-vps:integration`. Locally the variable is unset and the scripts keep passing `--build`. Only pushes to `main` write the cache, because a pull request can read the default branch's cache but not another pull request's. The release workflow's `edge` job exports the `runtime` scope and the versioned `image` job reads it. The Dockerfiles pin the Node.js major (`node:24-*`), so an upstream LTS bump cannot silently invalidate the cache or change the shipped runtime.
+
+Published images are `ghcr.io/wpelevator/tug`. To test an unreleased CLI against a checkout, use `--build-local-image` on `control setup` and `node add` (builds `tug:local` from the working directory and copies it over SSH). The dev Compose files (`docker-compose*.yml`) and the integration tests always build the image from the checkout with `build: .` and never pull from GHCR.
+
+## Compose stack
+
+`docker-compose.yml` defines:
+
+- `control`: the control plane on its single mTLS port `8443`, with SQLite, the master key, the private CA and saved image archives in a named volume, plus the Docker socket for builds. It requires `TUG_CONTROL_URL`, the HTTPS origin clients connect to.
+- `agent`: an optional profile service (applying nothing, `TUG_APPLY=false`); run it with `--profile agent` once you have an enrollment credential for it.
+
+```bash
+export TUG_CONTROL_URL=https://localhost:8443
+docker compose up --build control
+# issue the root operator credential on the control-plane host (fresh state only):
+docker compose exec control node dist/main.js control issue-credential --name root --role admin > tug-root.credential.json
+node src/cli.ts init tug-root.credential.json
+# prepare a node, then start the agent with the join key it prints:
+node src/cli.ts node enroll --name dev-node --agent-control-url https://127.0.0.1:8443
+TUG_JOIN_KEY=... docker compose --profile agent up agent
+```
+
+Control does not print or store a credential on startup; `issue-credential` writes the credential to stdout and keeps no private key. The agent keeps its identity in the `agent-state` volume, so it ignores the join key after the first start.
+
+## Host mode
+
+```bash
+export TUG_CONTROL_URL=https://localhost:8443
+npm run start:control
+node src/main.ts control issue-credential --name root --role admin > tug-root.credential.json
+node src/cli.ts init tug-root.credential.json
+```
+
+Host mode shares `./data` (or `TUG_DATA_DIR`) between control and `issue-credential`. Register a development node the way `node add` does, with the node making its own key:
+
+```bash
+export TUG_STATE_DIR=$PWD/data/agent TUG_NODE_NAME=dev-node
+node src/main.ts agent identity > dev-node.csr
+node src/cli.ts node register dev-node --csr dev-node.csr --address 127.0.0.1 | node src/main.ts agent install-cert
+```
+
+Start an agent that reports status without applying Docker changes:
+
+```bash
+export TUG_APPLY=false
+npm run start:agent
+```
+
+Deploy a Compose project without uploading a build context:
+
+```bash
+node src/cli.ts deploy -f docker-compose.yml
+node src/cli.ts status
+```
+
+Services with `build:` upload the current directory and control builds and stores compressed saved images for agents. `deploy --build-local` instead builds on the operator daemon and uploads archives; the platform must match target nodes. Published-image-only applications upload no context and control pins their registry digests before publication.
+
+## Integration test
+
+`docker-compose.sample.yml` is a three-service sample app with explicit placement: `web` on `vps-a`, `worker` on `vps-b`, `cache` on `vps-c`.
+
+`docker-compose.integration.yml` starts a control plane and three fake VPSes. On macOS/Docker Desktop each fake VPS is a privileged Linux container with its own Docker daemon and Tug agent, so agents apply containers into isolated Docker environments through their own Engine API rather than the host daemon.
+
+```bash
+npm run test:integration
+```
+
+The test starts control, runs a provisioner that issues the root credential and each fake VPS's node identity (the `agent identity` / `agent install-cert` flow, written to the shared `/tokens` volume), sets a sample secret, deploys `docker-compose.sample.yml`, waits for all three agents to report the desired revision, verifies published images run by the digest pinned on control, and tears the stack down. Set `KEEP_INTEGRATION_STACK=true` to keep containers and volumes for inspection. The integration network defaults to subnet `10.89.240.0/24`; override it with `TUG_TEST_SUBNET` if that overlaps with local Docker or VPN networks. For concurrent runs, set a unique `COMPOSE_PROJECT_NAME`, `TUG_TEST_PORT` and matching `TUG_CONTROL_URL` (an `https://` connect target; the credential still verifies `https://control:8443`).
+
+### Registry-free image integration
+
+```bash
+npm run test:integration:no-registry
+```
+
+The Node-driven `tools/scripts/integration-no-registry.js` uses `docker-compose.integration-no-registry.yml` with isolated control and node Docker daemons and no registry service or registry configuration. It verifies remote builds with changed application contents under a reused display version, rollback, agent restart, local application builds and backup/restore with a fresh image download. Separate daemons ensure a node cannot accidentally reuse control's already-built image instead of downloading the archive. The API port defaults to `18081` (`TUG_IMAGE_TEST_PORT` overrides it); use a unique `COMPOSE_PROJECT_NAME` when running alongside other tests. `KEEP_INTEGRATION_STACK=true` preserves the stack for inspection. These container tests do not establish real SSH installation, public ACME certificate issuance or a real-host private-network/NAT baseline.
+
+### Rolling rollout integration test
+
+```bash
+npm run test:integration:rollout
+```
+
+Layers `docker-compose.integration-rollout.yml` over the same stack with a single fake VPS (`vps-a`) that runs the app and is the Caddy ingress node, then runs `tools/scripts/integration-rollout.js`. It deploys v1 of a rolling nginx service next to an unchanged recreated service, keeps requesting the app through Caddy from inside the VPS every ~50 ms while v2 rolls out (zero failed requests, v1 then v2 observed, the old container removed after the drain, the recreated service's container id unchanged), then deploys v3 whose health check fails and asserts that v2 kept serving the whole time, v3 never received a request, control recorded `deployment.auto_rollback` and the rollback deployment succeeded. It uses the compose project `tug-it-rollout` and the same host ports as the other integration tests, so run them one at a time; `KEEP_INTEGRATION_STACK=true` keeps the stack.
+
+## Test VPS with Lima
+
+`template-vps.yml` is a [Lima](https://lima-vm.io/) template for an Ubuntu VM with Docker and Node.js installed, useful as a real SSH target for `control setup` and `node add`:
+
+```bash
+limactl start --name=vps template-vps.yml
+```
+
+### Networking integration test
+
+```bash
+npm run test:integration:network
+```
+
+Layers `docker-compose.integration-network.yml` over the same stack with plain numeric node `address` values and `vps-c` marked as an ingress node, deploys `docker-compose.network-sample.yml` (`web` on `vps-a`, `client` on `vps-b`) and runs `tools/scripts/integration-verify-network.js`: a container on `vps-b` fetches `http://web:8080` by its cross-node service name, Caddy on `vps-c` serves `Host: app.test` from `vps-a`, and generated published-port bindings use the intended private node addresses rather than every interface. It uses the compose project `tug-it-net` and the same ports as the plain integration test, so run them one at a time.
+
+## Running the agent outside Docker
+
+Run the agent in a container, as the installers do (`--network host`, the Docker socket mounted). An agent started as a plain host process, for example `node src/main.ts agent` on macOS, can create containers but cannot reach their IP addresses, so every TCP or HTTP health check times out and the deploy fails with a message such as `tcp timeout 172.21.0.2:3306` that does not mention where the agent runs. On Docker Desktop the Docker daemon lives in a VM, so container addresses are only reachable from inside it. When the agent is not in a container, a failed TCP or HTTP health check says so in its error and points at `docker run`; `doctor` does not check it separately.

package/docs/getting-started.md

@@ -0,0 +1,75 @@
+# Deploy your first service
+
+Install Tug's control plane and agent on one Linux VPS, deploy nginx, and visit its welcome page.
+
+## Before you start
+
+You need a VPS with a public IP and SSH access as root, Node.js 22.18 or newer and Docker Compose v2 on your computer. The VPS must run Debian, Ubuntu, Fedora or a RHEL-compatible distribution if you want Tug to install Docker; otherwise install Docker Engine and the Compose plugin there first. The VPS needs outbound access to pull images. Allow your computer and the agent to reach control on TCP 8443, and allow your computer to reach TCP 8080 for this temporary example. Check your provider firewall as well as any host firewall.
+
+Replace `203.0.113.5` below with your VPS address. These commands assume an interactive terminal, where Tug explains and confirms the installation choices. For automation see [Setup in the CLI reference](cli.md#setup).
+
+## 1. Install Tug and control
+
+On your computer, create a directory for this project and install the CLI:
+
+```sh
+mkdir my-first-tug && cd my-first-tug
+npm install -g @wpelevator/tug
+tug setup root@203.0.113.5
+```
+
+Allow Tug to install Docker if necessary and confirm the plan. Setup installs control and imports a root operator credential to your computer. The credential has full administrative access; keep it private. Setup creates `tug.yml` for shared defaults and a local overlay pointing to the credential. Keep working in this directory: the CLI searches the current directory for configuration, not its parents. See [project and configuration selection](cli.md#arguments-options-and-configuration).
+
+```sh
+tug status
+```
+
+You should see a control-wide status summary. If it cannot connect, check the address printed during setup and TCP 8443; see [Troubleshooting](troubleshooting.md).
+
+## 2. Add the same VPS as a node
+
+```sh
+tug node add root@203.0.113.5 --name app1
+tug node list
+```
+
+Accept Docker installation if prompted, choose an address belonging to the VPS and confirm the plan. You can decline managed ingress for this example and enable it later when you add a hostname. A private interface address is useful when you add a second server. The agent connects out to control and should appear in the node list with a recent contact time. The control plane and agent now share the VPS.
+
+## 3. Deploy and visit a service
+
+Save this as `compose.yaml` in the same directory:
+
+```yaml
+services:
+  web:
+    image: nginx:1.27-alpine
+    ports:
+      - "0.0.0.0:8080:80"
+    x-tug:
+      health:
+        type: http
+        port: 80
+        path: /
+```
+
+```sh
+tug deploy
+tug status
+tug logs web
+curl http://203.0.113.5:8080/
+```
+
+The deploy should report success, status should show `web` serving on `app1`, and curl should return the nginx welcome page. The explicit `0.0.0.0` publishes the example on all IPv4 interfaces; restrict TCP 8080 to your computer in the provider firewall while trying it. For a real site, use [managed ingress and HTTPS](networking.md#layer-b-ingress-and-tls), remove this explicit public binding, and redeploy. Tug otherwise binds published ports to the node address or loopback; a public node address can still make them public. See [port exposure](networking.md#published-ports-private-by-default).
+
+With one node, omitted placement selects it automatically. The HTTP check verifies nginx responds before Tug marks it serving; see [Health checks](healthchecks.md). If the deploy fails or the page is unreachable, start with [Troubleshooting](troubleshooting.md).
+
+## Choose a next step
+
+- [Add a hostname and HTTPS](networking.md#layer-b-ingress-and-tls): enable ingress, point DNS at the VPS and route a service.
+- [Deploy an application and database](../README.md#your-compose-file-plus-a-little-x-tug): use a volume and a project secret, then explore the [WordPress example](../examples/wordpress/README.md).
+- [Back up and recover](backups.md): protect workload data and the control plane separately, then rehearse a restore.
+- [Add another node](service-placement.md): pin stateful services and spread a stateless service across servers.
+- [Choose health checks and safer rollouts](healthchecks.md): understand readiness, failures and rollback.
+- [Give CI or teammates access](access-control.md): create credentials appropriate for trusted operators.
+
+The service and control plane will keep running after this tutorial. Before retiring a host that contains data, follow [Retire a host](migration.md#retire-a-host); teardown can delete volumes and credentials. The [CLI reference](cli.md), [Compose reference](compose.md) and [configuration reference](configuration.md) describe all options.

package/docs/healthchecks.md

@@ -0,0 +1,51 @@
+# Health checks and safer deployments
+
+A health check tells Tug when a service is ready to serve. During a deployment, the agent checks the container and reports failures to control. Ingress uses serving endpoints, and a rolling deployment waits for a healthy replacement before switching traffic. If you omit `x-tug.health`, Tug does not run a Tug readiness probe for that service; a running container alone does not prove the application is ready.
+
+## Choose a check
+
+| Type | What it tests | Useful for |
+| --- | --- | --- |
+| `docker` | The container is running and, if present, its Docker `HEALTHCHECK` reports healthy. Without a Docker health check it only verifies the container is running. | Images that already define a meaningful health check. |
+| `tcp` | The agent can connect to a port on the container IP. | A database or protocol server when a port-open check is sufficient. It does not prove a query will succeed. |
+| `http` | An HTTP request to the container IP returns 2xx after redirects. | A web app with a dedicated readiness endpoint. |
+
+This example gives MariaDB a TCP check and nginx an HTTP check:
+
+```yaml
+services:
+  db:
+    image: mariadb:11.4.8
+    environment:
+      MARIADB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
+    x-tug:
+      health:
+        type: tcp
+        port: 3306
+  web:
+    image: nginx:1.27-alpine
+    depends_on:
+      db:
+        condition: service_healthy
+    x-tug:
+      health:
+        type: http
+        port: 80
+        path: /
+```
+
+Set `DB_ROOT_PASSWORD` with `tug secret set DB_ROOT_PASSWORD` before deploying. The nginx service in this example does not use the database. See the [WordPress example](../examples/wordpress/README.md) for an app that does.
+
+Checks default to 10 attempts, a two-second interval and a two-second timeout. Increase the interval or retries for a slow-starting application. HTTP checks use the container IP and port, not the public hostname or ingress; select an endpoint that returns 2xx without relying on public redirects. A process-only endpoint can pass while a database or migration is broken; decide which dependencies must be ready before traffic is sent. Conversely, a check that depends on a service stopped during restore can make its own service unhealthy during that restore. Field details are in the [Compose reference](compose.md#services).
+
+## Dependencies and failures
+
+`depends_on` orders services on the same node; `condition: service_healthy` waits for the dependency's `x-tug.health`, or its Docker health check if there is no Tug check. Without either, the fallback checks only that the dependency container is running. Applications still need connection retries, particularly across nodes or when a dependency later restarts. Rolling services have additional ordering limitations described in [Rolling rollouts](compose.md#rolling-rollouts).
+
+If a check fails, `tug deploy` reports the error and `tug status`, `tug events` and `tug logs <service>` help diagnose it. Check the container port and path, the application's startup time and the service logs. A successful TCP check does not verify credentials or application data. See [Troubleshooting](troubleshooting.md).
+
+## Roll forward safely
+
+The default rollout recreates a changed service, which can interrupt traffic. A stateless service behind managed ingress can opt into `x-tug.rollout.strategy: rolling`; Tug starts and checks the new revision before ingress switches to it, then drains the old one. Rolling requires ingress, an explicit Tug health check and exactly one published port. It is not a database failover mechanism, and overlapping revisions can share mounted volumes. Review [rolling requirements and limits](compose.md#rolling-rollouts) before enabling it.
+
+`tug rollback` returns a project to a previous deployment; top-level `x-tug: { rollback: auto }` opts into automatic rollback for eligible failures. It cannot undo database writes or schema changes. See [automatic rollback](compose.md#automatic-rollback) and plan compatible migrations separately.

package/docs/migration.md

@@ -0,0 +1,113 @@
+# Migrating to new servers
+
+How to move a Tug installation, or one tier of it, to new servers and retire the old ones. Tug has no single "migrate" command: a move is a short sequence of the ordinary commands (`node add`, `deploy`, `backup`, `restore`, `node remove`), and the order is what keeps your data safe.
+
+Each section says what was exercised on a real host and what comes only from reading the docs. Treat the second kind as a checklist to rehearse on disposable servers first.
+
+| Section | Status |
+| --- | --- |
+| [Before you start](#before-you-start) | Reference |
+| [Use labels so a move is only `node add`](#use-labels-so-a-move-is-only-node-add) | Placement behaviour is documented; the exact edit was not rehearsed |
+| [Move the web tier](#move-the-web-tier) | Not exercised in a live run |
+| [Move the database](#move-the-database) | Restore across nodes was exercised; the full move was not |
+| [Move the object store](#move-the-object-store) | Not exercised; Tug has no data move for it |
+| [Move the control plane](#move-the-control-plane) | Recovery on another host was exercised on 0.9.0, see [Control plane](control-plane.md#recover-control-on-another-host) |
+| [Retire a host](#retire-a-host) | Manual; `node remove` only removes the control-plane record |
+
+## Before you start
+
+- **What lives where.** A host holds nothing a rebuild cannot recreate, except named volumes, any data you keep outside Docker (an object store on an attached disk) and, on the control host, the control state. Your Compose file, `tug.yml` and the images you build are the source of truth for everything else.
+- **Escrow the secrets that cannot be regenerated.** Keep the root credential and the control backup's master key off the servers, and keep the backup repository password (`tug backup password`). Without them a lost control host cannot be recovered. See [Backups](backups.md).
+- **Take fresh backups first.** `tug backup run` for every target you are about to move, then `tug backup list --check` to confirm they succeeded. For control, `tug control backup --no-images` is enough for a migration, because build archives are rebuilt by deploying again.
+- **Trim old build archives** on control with `tug image prune --keep 3` (add `--dry-run` to preview) so the control backup and any transfer stay small. Never delete files under `/var/lib/tug-control/images/` by hand: the database would still list them and nodes would get a 404 when they try to download them.
+- **Private network.** New nodes need a private address on the same network as the old ones, because cross-node service names and published ports use it. See [Networking](networking.md).
+
+## Use labels so a move is only `node add`
+
+If a service pins explicit nodes (`x-tug.placement.nodes: [web-1]`), moving it means editing the Compose file. If it selects nodes by label, moving it only means adding a node that carries the label and removing the old one:
+
+```yaml
+services:
+  php-fpm:
+    x-tug:
+      placement:
+        labels: [web]
+```
+
+```bash
+tug node add root@NEW --name web-2 --address 10.0.0.9 --label web --install-docker -y
+```
+
+Nodes must carry all of a service's labels, and nodes whose agent has not been seen recently are skipped when labels are resolved. Keep explicit `nodes:` only where a service must stay on one named host, such as a database that owns a local volume. Details are in [Compose extensions](compose.md) and [Nodes](nodes.md#node-labels).
+
+`tug scale` stays within a service's explicit `placement.nodes`, so it is not a way to move a pinned service. Change the placement and deploy instead.
+
+## Move the web tier
+
+A web tier whose database and uploads live elsewhere is stateless, so it is the cheapest move.
+
+1. Add the new node with the labels the web services select, and `--ingress` if it should handle ingress.
+2. Run `tug deploy`. Services placed by label are scheduled onto the new node when the placement is re-resolved at deploy. If you pin explicit nodes, change them in the Compose file first.
+3. Check `tug status` shows the services serving on the new node.
+4. Remove the old node once nothing is placed on it: `tug node remove web-1`. It refuses while a current deployment places services there or while its agent still reports; `--force` overrides both.
+5. Then [retire the host](#retire-a-host).
+
+Ingress: certificates are issued per ingress node and live in that node's Caddy volumes, so a new ingress node requests new ones. If the public IP changes, point DNS at it first, and keep Let's Encrypt rate limits in mind if you use a shared wildcard service such as `sslip.io`.
+
+## Move the database
+
+This is the move that can lose data, so the order matters. Deploying a database to a new node creates its volume there **empty**. If the application is still running and can reach that empty database, it will happily write to it or fail in confusing ways.
+
+1. **Add the new node** on the same private network, with the label the database selects (or pin it by name).
+2. **Back up now:** `tug backup run db`, then confirm with `tug backup list --check`.
+3. **Change the database placement and deploy with `--move`:** `tug deploy --move db`. Without the flag Tug refuses the deploy, because the database would start empty on the new node. With it, the services that write to the database are held (stopped) so nothing connects to the empty one. Writers are found from `depends_on` and from host names in environment variables (`WORDPRESS_DB_HOST=db`); check them with `tug restore run db` without `--confirm`, which prints the plan.
+4. **Restore the snapshot into it:** `tug restore run db --confirm`, adding `--writers php-fpm` if the plan missed a writer. The restore takes over the holds, takes a safety snapshot of the current data first and starts the writers when it finishes.
+5. **Changed your mind?** `tug hold release` starts the held services without restoring. Only do that if the empty database is acceptable.
+6. **Verify** with the application and `tug backup list --check`.
+7. **Clean up the old volume.** `tug volume remove db-data --delete-data` removes the record and the Docker volume on its node, but it refuses while a service in the current deployment still references the volume. Check `tug volume list` and, if the old copy is still there, remove the leftover Docker volume on the old host by hand.
+
+If the restore fails before it changes any data, the writers stay held, so nothing connects to the empty database. Fix the cause and run the restore again; `tug hold release` is the way to start them without a restore.
+
+While a writer is held, a sibling service whose health check depends on it (an nginx check that requests a PHP page) reports a failed apply for the duration. Give such services a health check that does not depend on the writer, such as a static `/healthz`. See [Restore](backups.md#restore). If the writers never report held, the restore fails after five minutes and the error names the node, the writer and the apply error the node reported.
+
+Before relying on this procedure, rehearse it on throwaway servers with a copy of your data.
+
+## Move the object store
+
+An object store such as SeaweedFS or MinIO is a plain Docker volume to Tug. Tug does not copy volumes between nodes, so you choose how the data travels:
+
+- **Small stores:** give the volume an `x-tug.backup` entry so Tug archives its files with restic, move the service, and restore the volume with `tug restore run <volume> --confirm`. See [Backups](backups.md).
+- **Large stores:** use the store's own replication or a one-off copy (`rsync` of the data directory with the store stopped), then point the service at the new location.
+- **Disposable stores:** deploy a fresh one and re-upload, after `tug volume remove <volume> --delete-data` on the old node.
+
+Whatever you choose, the buckets your applications and backups use must exist before they are used, and Tug does not create them. Create them with the store's own tooling (for example `aws s3api create-bucket` through an SSH tunnel to the private port). The access keys that the application, and `tug backup destination set`, use must match what the new store was started with.
+
+## Move the control plane
+
+To recover a lost control server, see [Recover control on another host](control-plane.md#recover-control-on-another-host). To change only its address, see [Moving the control plane](control-plane.md#moving-the-control-plane). For a planned move:
+
+1. Take `tug control backup --no-images` on the old control and copy the archive off the host.
+2. Follow the recovery steps on the new host. Stop or isolate the old control before the new one serves, because two active control planes with divergent state are not supported.
+3. After the restore, repoint every node with `tug node replace <node> <ssh-target> --control-url … --agent-control-url …`.
+4. Deploy your projects again so control rebuilds the build archives it no longer has. Rolling back to a deployment whose archive was not restored is refused until you do.
+
+Workload data is not part of the control backup. Back it up and restore it separately as above.
+
+## Retire a host
+
+`tug node remove <name> --teardown` removes the node's record from control and then cleans the host over SSH (the target defaults to `nodes.<name>.ssh` in `tug.yml`). It refuses a node that a current deployment still places services on, stops the agent and ingress, removes the Tug containers and networks, and deletes `/etc/tug` and `/var/lib/tug`. Project volumes and ingress certificates are kept; add `--delete-data` to delete them too, only once the data has been moved or backed up. It asks for confirmation (`-y` to skip, required without a terminal) and refuses a host that runs the control plane. If the cleanup fails after the record was removed, run the same command again.
+
+Plain `tug node remove <name>` only deletes the record and leaves the host running.
+
+On a former control host remove the control stack and its data, which includes the master key and the private CA: take the backup first, then `docker compose -p tug-control down -v` and `rm -rf /opt/tug-control /var/lib/tug-control`. Remove Docker itself only if nothing else uses it.
+
+Locally, delete the host from `tug.yml`, and remove stale credentials under `~/.config/tug/credentials/` and any `TUG_CONTROL_URL` you exported.
+
+Names to look for when cleaning up by hand: containers are `tug-<project>-<service>-<hash8>`, networks `tug_<project>`, volumes `tug_v<length>_<project>_<volume>`, and Tug resources carry `tug.project` labels. Review what a command will match before running it; the volume and container commands above delete data.
+
+## Known limits
+
+- A move does not copy the data: you restore it from a backup, and the old volume is not removed for you.
+- A control host is retired by hand.
+- Scratch restore volumes and the safety snapshot of every in-place restore are kept. See the limits in [Backups](backups.md).
+- The control archive does not contain workload data.