@wpelevator/tug 0.11.0 → 0.13.0

package/docs/backups.md

@@ -0,0 +1,160 @@
+# Backups and recovery
+
+Tug schedules, verifies and restores backups itself. You mark a service or a volume with `x-tug.backup`, give control one S3-compatible destination, and agents run [restic](https://restic.net) on the node that holds the data. There is no backup container to build or keep alive.
+
+## Plan for recovery
+
+Workload snapshots contain application data; control-plane archives contain Tug state and keys. Store both away from the VPS, and keep their credentials and passwords somewhere you can reach if the server is lost.
+
+| What | How to protect it | What to keep for recovery |
+| --- | --- | --- |
+| Database | Back up a logical dump from the running service with `x-tug.backup.command`; copying a live database volume is generally not consistent. | Repository password, storage credentials, a suitable restore command and application configuration. |
+| Files in a named volume | Mark the volume with `x-tug.backup`; live files are crash-consistent, not necessarily application-consistent. | Repository password and storage credentials; test a scratch restore. |
+| External object storage | Configure separate protection with the storage provider. | Provider credentials and a way to restore objects to a compatible point in time with the database. |
+| Control state, secrets, private CA and master key | Run `tug control backup` and copy the archive off-host encrypted; it does not include workload data. | The archive and an original operator credential; see [control recovery](control-plane.md#recover-control-on-another-host). |
+| Compose file and source | Keep them in version control. | Access to the repository and any external dependencies. |
+
+Tug backs up each target separately. If your application needs a consistent point-in-time copy, plan how to stop writes and coordinate external storage. [The WordPress example](../examples/wordpress/README.md) illustrates a database and files on one node.
+
+## Set up your first backup
+
+Choose S3-compatible storage reachable from the workload node and credentials allowed to use the bucket. Configure one destination for control, then mark each target in your Compose file as shown in the [backup reference](compose.md#backups). A top-level schedule only supplies defaults; it does not enable any targets. For a complete database and volume configuration, use the [WordPress example](../examples/wordpress/README.md).
+
+```sh
+# Supply the values through your shell or secret manager; do not commit them.
+export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
+tug backup destination set s3:https://s3.example.com/bucket/tug-backups --env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY
+tug deploy
+tug backup run
+tug backup list --check
+tug backup password
+```
+
+Replace the endpoint and bucket with your own. `backup run` waits for the configured targets and reports failure; inspect `backup list --check` to confirm each target succeeded. Store the printed repository password and storage credentials securely off-host. The password is derived from control's master key; losing both the password and master key makes the repository unreadable. `tug backup destination show` shows the destination without credential values. Credentials are encrypted on control and released to a node running a backup. Each project has its own restic repository under `<destination>/<project>`; commands use the [resolved project](cli.md#arguments-options-and-configuration).
+
+Also [back up control](#control-plane-backup), copy the archive off-host, and practice a [restore drill](#restore-drill). No backup notification is pushed automatically; monitor `tug backup list --check`.
+
+## Backup destination
+
+The control plane uses one destination. Running `tug backup destination set` replaces it. Setting or removing a destination updates its configuration and stored credentials atomically; a database failure leaves the previous configuration and credentials intact. Each project uses its own repository. The complete command and credential options are in the [CLI reference](cli.md#backups).
+
+## Scheduled backups
+
+Mark a service or a volume with `x-tug.backup` ([Compose extensions](compose.md#backups)) and Tug backs it up on its node, on the schedule you give, to the project's repository. Control runs a scheduler that ticks every 30 seconds (`TUG_BACKUP_TICK_MS`); at each slot it queues a `backup.run` command with a deadline. The agent runs it as a background job, so reconciliation and health checks are never held up by a long backup, and one node runs one backup at a time unless `TUG_BACKUP_CONCURRENCY` says otherwise.
+
+- **Services** run your `command` through an exec in the running container and stream its output into `restic backup --stdin`, so no staging disk is needed. The command sees the container's own environment, including its secrets, so passwords are read from the environment (`$MARIADB_ROOT_PASSWORD`) and never appear in Tug's configuration, payloads or events. A command that fails, writes nothing or lacks the `verify` text is never kept: the partial snapshot is forgotten and the run fails. The service must be running on the node.
+- **Volumes** are mounted read-only into the restic container and archived, with your `exclude` patterns. A volume that does not exist on the node fails the run instead of backing up an empty one. Live volumes are crash-consistent, which suits uploads and static files.
+- **Retention** runs after each successful backup (`restic forget --prune` per node and target). Snapshots tagged `trigger:pre-restore` are kept regardless.
+
+- **Images** for restic come from `TUG_RESTIC_IMAGE` on control (default `restic/restic:0.17.3`); a digest reference is recommended. Nodes pull it outbound like any published image.
+- **Restic runs in a helper container** with a memory limit, labelled `tug.job`, removed when it finishes. An agent that restarts mid-job removes leftovers and reports the job failed.
+
+Before mounting a managed volume, backup and restore check its project and volume ownership labels; a scratch restore checks its scratch volume's project and restore labels. A conflicting or unlabelled volume is refused instead of being read or overwritten.
+
+## Recipes
+
+Tug has no database support of its own; each recipe is a command for the service container. In a Compose file use `$$` for variables that the container shell must expand: a single `$` is interpolated on the operator machine before deployment. Read credentials from the container environment, write the data to standard output, and give `verify` the line the tool prints on success.
+
+```yaml
+# MariaDB (use mysqldump and MYSQL_ROOT_PASSWORD for MySQL)
+backup:
+  command: mariadb-dump -u root -p"$$MARIADB_ROOT_PASSWORD" --single-transaction --quick --routines --events --triggers --add-drop-database --databases wordpress
+  filename: wordpress.sql
+  verify: "-- Dump completed"
+
+# PostgreSQL
+backup:
+  command: pg_dump -U "$$POSTGRES_USER" --clean --if-exists "$$POSTGRES_DB"
+  filename: app.sql
+  verify: "PostgreSQL database dump complete"
+
+# SQLite file in a volume mounted into the service
+backup:
+  command: sqlite3 /data/app.db .dump
+  filename: app.sql
+  verify: COMMIT
+
+# A directory as a tar stream
+backup:
+  command: tar -C /var/www/html -cf - wp-content/themes
+  filename: themes.tar
+```
+
+`mariadb-dump` and `mysqldump` do not dump users and grants; they come from the image's init environment. A service has one backup command, so to include PostgreSQL roles chain them: `sh -c 'pg_dumpall --globals-only; pg_dump ...'`. Restore with the same tool: `restic dump <snapshot> wordpress.sql | mariadb ...`.
+
+### Events and what to watch
+
+`tug events` shows `backup.completed`, `backup.failed`, `backup.missed`, `backup.stale` and `backup.no_destination`. A backup is `missed` when its node did not pick it up before its deadline (`node_offline`) or when control was not running at the slot: a week of downtime produces one missed event and one catch-up run per target, not a burst. A failed scheduled run is retried once after ten minutes. A target with no success for two periods raises `backup.stale` once a day, which catches a destination key that has quietly stopped working. While control is down no backup starts, which is the price of one scheduler; running jobs finish and report when control returns.
+
+`tug backup run` takes a backup now and `tug backup list` shows the catalog ([CLI](cli.md#backups)).
+
+### Restore
+
+In a terminal, `tug backup run --select` chooses a subset of enabled targets. `tug restore run` offers a target when omitted; add `--select-snapshot` to choose a successful catalog backup or `--select-writers` to override writer discovery. `tug restore resume` and `tug restore release` offer interrupted restores when their ID is omitted. Existing preview and confirmation requirements still apply. See [interactive selection](cli.md#interactive-selection).
+
+```bash
+tug restore run db                       # prints the plan, changes nothing
+tug restore run db --confirm             # latest backup of the db service
+tug restore run uploads --scratch --confirm   # files into a new volume, touching nothing live
+```
+
+A restore plan and its queued restore refresh backup placement from the latest successful deployment before choosing a node, so moving a project does not leave a restore aimed at the previous node until the next scheduler tick. Check the printed target before confirming.
+
+Restores are as generic as backups. A service is restored by running its `x-tug.backup.restore` command in the running container with the stored file on standard input (`restic dump` piped in), so a MariaDB service uses `restore: sh -c 'mariadb -uroot -p"$$MARIADB_ROOT_PASSWORD"'`. A volume is emptied and filled from the snapshot, or restored into a new volume with `--scratch`. A service without a `restore` command is refused rather than guessed at.
+
+`tug restore` returns when the data is back and the services it stopped report serving again, or after five minutes with a warning if they do not, so a script can probe the site as soon as it returns. An in-place restore is deliberately careful:
+
+1. It **locks the project**, so a deploy, rollback or secret change gets a clear refusal while it runs.
+2. It **stops the writers**: the `writers` you named, or discovered from `depends_on`, or host names in environment variables such as `WORDPRESS_DB_HOST=db` or `mysql://u@db:3306/app` (for a service), or volume mounts (for a volume), including everything that depends on them. If none are found it refuses unless you pass `--writers` or `--no-writers`, because an app that reaches the database without `depends_on` is the hole in discovery and the plan makes you look.
+3. It takes a **safety backup** of the current data, tagged `trigger:pre-restore` and exempt from retention (skip with `--no-safety-backup`). If that fails, nothing has changed: the writers start again and the lock is dropped.
+4. It restores, then starts the writers and unlocks.
+
+While a writer is held, a service that health-checks through it (an nginx check that requests a PHP page, for example) is unhealthy, so its node reports a failed apply for the duration. Tug still sees the writer as held and continues, but the deployment looks failed until the hold ends. Give such services a health check that does not depend on the writer (a static `/healthz`) to keep them serving through a restore. If a writer never reports held within five minutes, the restore fails and the error names the node, the writer's state and the apply error that node reported.
+
+Only a snapshot this target produced can be restored into it (its `project:` and `target:` tags are checked). If the restore itself fails, the writers stay stopped and the project locked, because a half-restored database serving traffic is worse than an outage. The restore is `interrupted`; `tug restore resume <id>` tries again and `tug restore release <id>` starts the services anyway. Scheduled backups of a target are skipped while it is being restored. A restore is recorded with `restore.started`, `restore.completed`, `restore.failed` and `restore.released` events.
+
+A scratch restore of a volume is also the restore drill: it proves a backup can be read without touching anything live. Databases are the same drill by hand: `restic dump <snapshot> wordpress.sql` into a throwaway container.
+
+## Recover a site on a replacement node
+
+This is a manual procedure, not automatic failover. A WordPress and MariaDB project was recovered this way with the published 0.9.0 release in the [real-host drill](real-host-evaluation.md#recovery-drill-with-the-published-090-release-2-november-2026). The original host was simulated as unavailable while control and the backup destination stayed available. Restore control first if it is also lost.
+
+1. Fence the lost node or otherwise ensure its old workloads cannot write to shared services. Enroll the replacement node and verify it is online. Keep the recovery site private or otherwise block public traffic while its new database and volumes are empty; do not expose a fresh WordPress installer.
+2. Change the project’s service and volume placement to the replacement node and deploy the same project using the retained images, configuration and secrets. The replacement uses fresh local volumes; deploying alone does not restore their contents. Keep the old backup repository and its password available.
+3. Check `tug backup list` and run `tug restore run db` and `tug restore run wp-content` without `--confirm`. Verify each plan names the replacement node and the intended pre-loss snapshot. If the latest snapshot is not the desired one, use `--from` with the intended backup run or snapshot ID. Older releases may briefly show the former node until the backup scheduler catches up; never confirm a plan aimed at the wrong host.
+4. Restore the database and uploads with the printed commands and `--confirm`. Tug normally takes a safety backup before each in-place restore and stops the writers; wait for each restore to finish and investigate any warning before continuing. Verify known database content and uploaded files, probe the application, then re-enable traffic and update DNS or routing as needed.
+
+This does not restore data that was never backed up, external services, DNS, or changes after the selected snapshots. A live workload can write new data during recovery unless traffic and other writers are fenced; control alone cannot fence an unreachable or compromised host.
+
+## Control plane backup
+
+`tug control backup` covers the control plane only, not workload data. Take it with an `admin` operator credential and copy the archive off the control-plane host. The archive holds the master key, every secret and the private CA key (which can issue credentials for the whole installation), so treat it like a password vault and store it encrypted. Create a dedicated credential for the job: `tug credential create backup --role admin` and store it at `/etc/tug/backup.credential.json` (mode `0600`). Example `/etc/cron.d/tug-control-backup` on the control-plane host (the runtime image runs the CLI; adjust to how you invoke `tug` there):
+
+```cron
+15 2 * * * root TUG_CONTROL_URL=https://127.0.0.1:8443 TUG_CREDENTIAL_FILE=/etc/tug/backup.credential.json tug control backup > /var/log/tug-control-backup.json 2>&1 && restic -r s3:https://s3.eu-central-1.amazonaws.com/acme-backups/control backup /var/lib/tug/control-backups --password-file /etc/tug/restic.pass
+```
+
+Put the S3 credentials in the crontab environment or a `restic.env` file with mode `0600`. Archives accumulate in `TUG_CONTROL_BACKUP_DIR`; prune old files with `find ... -mtime +14 -delete`. The cron line has not been run on a real control-plane host; verify that `tug` is on the cron `PATH` and that the data directory matches your install (`TUG_CONTROL_BACKUP_DIR`, see [Configuration](configuration.md)). `TUG_CONTROL_URL` is a connect target only; the certificate is still verified against the public URL in the credential. Restore with `tug control restore <path> --confirm` and restart control ([Control plane](control-plane.md#backup-and-restore)).
+
+## Restore drill
+
+A backup you have not restored is a hope. Start with an isolated restore to verify the backup without changing live data. Time the exercise and record what you needed.
+
+1. **Check the repository and the catalog.** `tug backup list` shows every target with its last success; the newest run should be recent and `succeeded`. `tug backup password` prints the repository and its password, so you can also inspect it directly with `restic snapshots` from any machine that can reach the bucket.
+2. **Restore files into a scratch volume.** `tug restore run wp-content --scratch --confirm` reads the latest snapshot into a new volume next to the live one and touches nothing live. List it with `docker run --rm -v <volume>:/data alpine ls /data` on the node, then remove the volume with `docker volume rm <volume>`.
+3. **Restore the database into an isolated throwaway database.** Use the original project repository and its escrowed password to read the snapshot with restic into a separate database container. Verify known data without connecting the test container to production services, mail or scheduled jobs. This manual end-to-end drill has not yet been validated on a fresh host.
+4. **Write down how long it took.** That is your real recovery time.
+
+Tug does not restore a snapshot into a different project: repositories and snapshot tags are project-specific. For an in-place restore, inspect `tug restore run db` first; confirmation stops writers, takes a safety backup and changes the live database. See [Restore](#restore).
+
+## Known limits
+
+- One failed night is visible as a `backup.failed` or `backup.stale` event, not a push notification, until [webhooks](roadmap.md#m8--central-api-advantages-partly-done) exist. Watch `tug events`, or point a monitor at `tug backup list`.
+- While control is down no backup starts. Running jobs finish and report when control returns, and a long gap yields one missed event and one catch-up run per target.
+- Live volumes are crash-consistent, not application-consistent. Use a command backup for anything with its own consistency (databases).
+- A volume is backed up from the one node that holds it. A volume you mount from a file server with `driver_opts` is read through that node's mount; back up the file server's own disk where you can.
+- Repository maintenance (`restic check`, a read-verify of the data) and SFTP destinations are not built yet.
+- A scratch restore leaves its volume (`tug_r<project-length>_<project>_<id>`) behind, and the safety snapshot of every in-place restore is kept for ever. There is no command to list or remove either yet: remove scratch volumes with `docker volume rm` on the node, and safety snapshots with `restic forget` using the repository from `tug backup password`.
+- The restic helper container defaults to a 512 MB memory limit, which is half of a 1 GB host; lower it with `memory` under `x-tug.backup`. Restic needs more memory for larger repositories, so test the limit against your data.
+- Failures are not pushed anywhere until webhooks exist. Run `tug backup list --check` from cron or a monitor: it exits nonzero when a target is stale, failed or missed.
+- Cancelling `tug exec` does not stop the remote process.

package/docs/cli.md

@@ -0,0 +1,182 @@
+# CLI reference
+
+Run `tug --help` to list commands and `tug <command> --help` for command options, units and defaults. Commands read `tug <subject> <verb> [params]`, such as `tug node add` and `tug secret set`, and `tug <subject> --help` lists a subject's verbs. Everyday project commands (`deploy`, `status`, `logs`, `exec` and the like) stay at the top level, and `tug setup` is a shortcut for `tug control setup`. There are no colon names or dash-joined legacy names.
+
+| Subject | Verbs |
+|---|---|
+| (top level) | `setup`, `init`, `deploy` (and `deploy cancel`), `status`, `rollback`, `restart`, `scale`, `logs`, `exec`, `run`, `ps`, `inspect`, `events`, `doctor` |
+| `control` | `setup`, `backup`, `restore` |
+| `node` | `add`, `list` (`ls`), `replace`, `repair`, `remove`, `disable`, `ingress`, `address`, `register`, `enroll` |
+| `secret` | `set`, `list`, `remove` |
+| `backup` | `run`, `list`, `password`, `destination set`, `destination show`, `destination remove` |
+| `restore` | `run`, `list`, `resume`, `release` |
+| `credential` | `create`, `list`, `revoke` |
+| `volume` | `list`, `remove` |
+| `hold` | `release` |
+| `image` | `prune` |
+
+## Arguments, options and configuration
+
+A positional argument names the primary target: the SSH host for `node add`, node for `node register`, service for `logs`, secret name for `secret set` or backup archive for `control restore`. Named options qualify the operation. For example, `node add root@app1.example.com --name app1` distinguishes the SSH destination from the registered node name. Commands reject unknown options and extra positional arguments instead of ignoring typos. Numeric options accept bounded integers; invalid values and contradictory options fail before an API request or Docker/SSH operation.
+
+| Global option | Environment | Config key | Default |
+|---|---|---|---|
+| `--config <file>` | `TUG_CONFIG` | — | `tug.yml`, `tug.yaml`, `tug.json` or `.tug.json` in the current directory |
+| `--control-url <url>` (operator API endpoint) | `TUG_CONTROL_URL` | `controlUrl` | the credential's default address (an override is a connect target only) |
+| `--project <name>` | `TUG_PROJECT` | `project` | `site` |
+| `--credential <file>` | `TUG_CREDENTIAL` (inline) or `TUG_CREDENTIAL_FILE` | `credential` (a file path), only in the untracked overlay | No credential |
+
+Global options can appear before or after a command. Explicit options take precedence over environment variables, then the overlay, then the tracked config, then defaults. Relative `--config` paths resolve from the working directory. A missing default config is allowed; an explicitly selected missing file, invalid YAML or JSON, an unknown key or an invalid value is an error. `init` can create a new selected file. `--project` applies only to project-scoped operations; status and node operations retain their control-wide scope.
+
+Most commands print the API result as JSON; `status` prints a readable summary unless given `--json`. `init` prints the written config path, `exec` and `logs` write container output to stdout/stderr, and enrollment prints a runnable command or cloud-init document. `ps` and `inspect` print live Docker data as JSON. `node enroll --json` returns JSON with `command`, and also `cloudInit` with `--install-docker`; its stdout contains no explanatory prose. There is no global `--json` flag. Install preflight and image-transfer tools may also print progress. Errors go to stderr and exit nonzero.
+
+Credentials are never command-line arguments, since arguments can appear in process listings and shell history; use `TUG_CREDENTIAL`, `TUG_CREDENTIAL_FILE` or `tug init`. `secret set --value` has the same exposure; omit it to read the value from stdin. The credential path belongs in the untracked overlay (see below), never in the tracked config.
+
+`tug deploy` prints progress to stderr in terminals and CI, including with `--json`: source preparation, upload/control preparation, per-service build/export/cleanup for `--build-local`, and deployment state changes while waiting. Each stage appears immediately and repeats every ten seconds with elapsed seconds for that stage, so long builds do not leave the CLI silent. Repeated polling of the same deployment state does not reset its timer. Remote upload and image building share one stage because the control API completes both in the deployment request; these updates show elapsed time, not upload percentages or Docker build logs. Stdout still contains only the final summary or JSON result. `--no-wait` reports preparation progress and returns after control publishes desired state, without waiting for rollout.
+
+The same progress reporting covers `rollback`, `backup run`, `restore run`, control-plane backup/restore, control setup, node add/replace/repair and node host teardown. Setup reports Docker installation, host checks, identity creation and installation; `--build-local-image` additionally reports image building, export, transfer and loading on the host. Node installation reports the wait for a fresh agent report. Interactive setup questions and confirmation finish before these stages start. Backups identify pending targets and their reported states, restores report their state changes, and rollback reports deployment states. The shared command wait used by `restart`, non-followed `logs` and node diagnostics in `doctor` also reports queued/running/terminal states. Heartbeats stop on completion, failure or timeout; existing timeouts and exit codes still apply. `--no-wait` skips the corresponding remote wait and its progress. `node enroll` only prepares enrollment instructions and does not wait for the host to enroll.
+
+Both remote source uploads and `--build-local` use a temporary source snapshot of the current working directory. Tug applies the root `.dockerignore` using its supported pattern matcher and always excludes `.git`, `node_modules`, `data`, credential documents named `*.credential.json`, the Tug overlays and `.tug.json` at any depth, root Tug config names, and the resolved config, overlay and credential file paths. These mandatory exclusions cannot be re-included by ignore rules. The snapshot is validated before use and unsafe links are rejected. Docker then applies its own ignore rules to the local build. Tug does not detect arbitrary secret names or claim full Docker ignore compatibility: review your build contexts and Dockerfile-specific ignore rules, especially for nested or multiple contexts. A broad `COPY .` can still include secrets that Tug cannot identify.
+
+## Interactive selection
+
+In a terminal outside CI, omit an existing target to fetch choices from control: `tug logs`, `tug exec`, `tug inspect`, `tug restart`, `tug deploy cancel`, `tug restore run`, `tug restore resume`, `tug restore release`, `tug secret remove`, `tug volume remove` and `tug credential revoke`. Node operations `replace`, `repair`, `disable`, `remove` and `ingress` also offer registered nodes when their name is omitted. Use `tug scale --replicas 3` or `tug node address --to 10.0.0.2` to supply the new value while choosing the target. `tug doctor --node` offers a node; plain `doctor` retains its existing behavior.
+
+Menus show numbered choices with identifying details. Enter a number, or comma-separated numbers for multi-choice; enter `q` to cancel. Blank input never selects anything, even if there is only one choice. Menus and questions go to stderr, keeping JSON and command output on stdout usable. Without a terminal, or with `CI` set, missing targets fail with an explicit-target error and selection never reads stdin. Explicit targets skip their selection menu and discovery request.
+
+Live `exec`, `run`, `inspect`, `logs --follow` and `ps` offer a node when placement is ambiguous and `--node` is omitted. The choices come from current desired placement, with node status shown alongside each name; stale or disabled placements remain visible and control still validates the operation. Pass both the service and `--node` to avoid discovery for live commands. To select a service while supplying a remote command, use `tug exec --select -- sh -c 'echo hello'` or `tug run --select -- wp core update-db`; with `--select`, all positional arguments belong to the remote command.
+
+Commands whose omitted targets already mean “all” retain that behavior. Use these explicit selection options for a subset:
+
+| Command | Choices |
+| --- | --- |
+| `tug backup run --select` | Enabled backup targets; cannot combine with explicit targets |
+| `tug hold release --select` | Services held for a deployment move; excludes restore-owned holds |
+| `tug node replace --select`, `tug node repair --select` | Registered nodes with SSH targets in local config; cannot combine with a node, SSH target or `--all`; continues and reports per-node failures |
+| `tug restore run --select-snapshot` | Successful catalog backups for the chosen target, including explicitly labeled pre-restore safety backups; `--from` still accepts an external restic snapshot ID |
+| `tug restore run --select-writers` | Services to stop instead of automatic writer discovery; cannot combine with writer flags or scratch mode |
+| `tug credential create ci --role deploy --select-projects` | Existing projects, or “All projects, including future projects”; explicit repeatable `--project` also supports future project names |
+
+Selections are project-scoped where the operation is project-scoped and honor the credential's permissions. Service choices use current desired state rather than potentially stale reports. Deployment cancellation lists active deployments without the status history limit; snapshot selection reads the successful backup catalog rather than the recent-run display. Restore recovery lists only interrupted restores, including those outside the recent history display. Volume removal offers only unreferenced volumes and identifies persistent data that requires `--delete-data`. A choice does not bypass control's validation or existing confirmation requirements: `restore run` still previews unless given `--confirm`, and volume/node removal retains its existing guards.
+
+## Config file
+
+The CLI reads a YAML or JSON file, `tug.yml` by convention, so a repository can record how to reach its control plane and which servers it uses. [`examples/tug.yml`](../examples/tug.yml) is a complete sample.
+
+- **Lookup.** `--config` or `TUG_CONFIG` select a file, which must exist. Otherwise the working directory is searched for `tug.yml`, `tug.yaml`, `tug.json` and `.tug.json`; more than one is an error, and none is fine. The format follows the extension (`.json` is JSON, anything else YAML, which also reads JSON). Only the working directory is searched, not its parents.
+- **Overlay.** `tug.local.yml` next to `tug.yml` (`.tug.local.json` next to `.tug.json`, `<name>.local<ext>` in general) holds what must not be committed. It has the same keys plus `credential` (a path to a credential file), and its values win key by key, including inside `nodeDefaults` and each node. `credential` in the tracked file is an error. `tug init` writes the tracked file only when it is missing (an existing one keeps its comments), points the overlay at the imported credential file (written with mode `0600` under `~/.config/tug/credentials/`), and adds the overlay to `.gitignore` in a git checkout. Connection flags given to `init` for an existing tracked file are saved in the overlay.
+- **Strict keys.** Unknown keys, wrong types and out-of-range values fail with the file and key in the message. Paths in the file (`file`, `ca`) resolve relative to the file, not the working directory.
+
+Every key is a default for the flag it is named after, on every command that has that flag (a test enforces that the table below and the commands' real options agree), and a flag on the command line always wins. Each key is the camelCase of the long flag it supplies a default for, with the same value type. `nodeDefaults` keys that name `control setup` options (`installDocker`, `image`) are its defaults too. A repeatable flag takes a list (`label: [web, db]`; a single string is a one-item list), and a negated flag maps to the positive key set to `false` (`--no-wait` is `wait: false`). A flag on the command line always wins over the file; a list from the file is replaced by flags, not appended to. A key applies only to the commands listed for it.
+
+| Key | Flag | Commands |
+|---|---|---|
+| `controlUrl` | `--control-url` (connections), the address `control setup` advertises | all; `control setup` also writes it into control's environment |
+| `project` | `--project` | project-scoped commands |
+| `credential` (overlay only) | `--credential`, `TUG_CREDENTIAL_FILE` | all |
+| `file` | `-f, --file` | `deploy` |
+| `buildLocal` | `--build-local` | `deploy` |
+| `wait` | `--no-wait` | `deploy`, `rollback`, `backup run`, `node add`, `node replace`, `node repair` |
+| `nodes.<name>.ssh` | `<ssh-target>` | `node add`, `node replace`, `node repair` |
+| `nodes.<name>.address` | `--address` | `node add`, `node replace`, `node repair`, `node register`, `node enroll` |
+| `label` | `--label` | `node add`, `node replace`, `node repair`, `node register`, `node enroll` |
+| `installDocker` | `--install-docker` | `node add`, `node replace`, `node repair`, `node enroll`, `control setup` |
+| `image` | `--image` | `node add`, `node replace`, `node repair`, `node enroll`, `control setup` |
+| `agentControlUrl` | `--agent-control-url` | `node add`, `node replace`, `node repair`, `node enroll` |
+
+`control setup` reads `controlUrl` from the selected file (or the discovered default file), with the overlay taking precedence. `controlUrl` is the single name for where control is. For `control setup` the address is, in order, `--control-url`, `TUG_CONTROL_URL`, the config file's `controlUrl`, then the SSH target's host and `--port`; the command prints which one it used. For every other command it is the connect target, falling back to the URL stored in your credential, so a tunnel is just `--control-url https://127.0.0.1:18082`. The credential's URL is never used as an installation address. A stale exported `TUG_CONTROL_URL` therefore changes what `control setup` advertises; unset it when installing. A configured URL includes its own public port; `--port` controls the host port published by Docker and does not rewrite the configured URL.
+
+The rows from `label` down are node keys: set them under `nodes.<name>`, or once for every node under `nodeDefaults` (everything except `ssh` and `address`). Secrets and per-invocation choices have no key: `--value`, `--force`, `--confirm`, `--version`, `--ttl`, `--timeout`, `--build-local-image` and the like.
+
+The file is not desired state and Tug never reconciles against it. Only `node add`, `node replace`, `node repair`, `node register` and `node enroll` read `nodes`, as defaults for flags; every other command asks the control plane. See [Nodes](nodes.md#declaring-nodes-in-the-config-file).
+
+## Setup
+
+- `tug init [<credential-file>|-]` creates the config file with the resolved control URL and project when it is missing, imports the credential (a file, or `-` for stdin) into a `0600` file under `~/.config/tug/credentials/` and points the git-ignored overlay next to the config at it; see [Config file](#config-file). Use global `--control-url`, `--project` or `--config` to select them.
+- `tug setup <ssh-target>` is a shortcut for `tug control setup <ssh-target> [--extra-san <name>]... [--install-docker] [--port <port>] [--image <tag>] [--build-local-image] [--credential-out <file>] [--no-import] [--recover-root] [-y]`. It installs control over SSH (the advertised address defaults to `controlUrl` from the config, then the SSH target's host and `--port`), issues the root credential and, like `init`, imports it into your local config, so no separate `init` step follows. In a terminal it offers to install Docker when it is missing, prints the plan and asks to confirm (`-y` skips the question); without a terminal it never prompts. `--credential-out <file>` also saves the credential to a `0600` file, and `--no-import` skips the import (it requires `--credential-out`). The mTLS port defaults to `8443`. `--recover-root` replaces only root's key from the control-plane host. See [Control plane](control-plane.md).
+- `tug node address [node] [address] [--to <address>]` updates the registered private address used for published-port binding and cross-node routing.
+- `tug node disable [node]` blocks a node's key immediately without SSH, deleting or changing placement or stopping its containers: it withdraws the node's service endpoints and ingress routes but does not drain workloads. `node replace` brings it back.
+- `tug node add [<ssh-target>] [--name <node>] [--ingress] [-y] [--all] [--agent-control-url <url>] [--install-docker] [--address <ip>] [--label <label>]... [--image <tag>] [--build-local-image] [--no-wait]` creates a node and installs its agent over SSH. The name defaults to the first DNS label of the SSH host (`app1` for `root@app1.example.com`); with an IP address a terminal asks for the name, offering the host's own `hostname -s` as the default, and without a terminal `--name` is required. `node enroll` likewise asks for `--name` in a terminal (there is no host to suggest one). Without `<ssh-target>` the target is `nodes.<node>.ssh` from the [config file](#config-file). It checks for a control credential first, then always prints the plan (node, SSH target, Docker, address, image, ingress) to stderr before changing anything; without a terminal the plan says Docker was not checked. In a terminal it also checks for Docker and offers to install it, lists the host's global IPv4 addresses (private ones first, never defaulting to a public one) to pick the inter-node `--address` from, asks whether the node should handle ingress (`--ingress` answers yes) and asks to confirm (`-y` skips the question); without a terminal it never prompts, so scripts pass `--address`, `--ingress` and `--install-docker` explicitly. `--all` walks every node with an `ssh` entry, continues after failures and prints per-node results. Unless `--no-wait` is set the result carries `reporting`; a node that has not reported after about a minute prints `reporting: false` and the command exits nonzero. The agent uses host networking for TCP and HTTP health checks, without added capabilities. See [Nodes](nodes.md).
+- `tug node replace|repair [<node>] [<ssh-target>] [--all] [same options]` runs the same SSH-assisted flow for an existing node: `replace` issues a new identity (it also brings back a disabled node) and `repair` first moves corrupt agent credential documents aside. `<node>` is positional; the target defaults to `nodes.<node>.ssh`.
+- `tug node list [--json]` (alias `ls`) prints a table of the registered nodes with status, address, ingress role, labels and last contact, or the raw API result with `--json`.
+- `tug node register <node> --csr <file> [--mode create|replace]` signs a node-generated CSR and prints the identity response (node, certificate, CA and public URL). It installs nothing; use `agent install-cert` on the host.
+- `tug node enroll --name <node> [--address <addr>] [--agent-control-url <url>] [--install-docker] [--label <label>]... [--ttl <seconds>] [--image <tag>] [--json]` prepares a host to enroll itself: it mints a single-use, name-bound join key (an enrollment credential) and prints the Docker command that enrolls the host when run, or cloud-init with `--install-docker`. TTL defaults to `3600` seconds. A saved cloud-init document contains a credential. The printed image must be pullable by the new host; a development `tug:local` override must already be loaded there.
+- `tug node remove [node] [sshTarget] [--force] [--teardown [--delete-data] [-y]]` removes a retired node from the control plane: its identity (the node ID is retired and never reused), statuses and queued commands, and any unconsumed join key. It is refused while a current deployment still places a service or volume on the node (deploy without it first) and while its agent is still reporting, unless `--force`. Stop the agent and remove its containers on the host yourself. With `--teardown` it also cleans the host over SSH after the control record is gone: it stops the agent and ingress, removes the Tug containers and networks and deletes `/etc/tug` and `/var/lib/tug`. Project volumes and ingress certificates are kept unless you add `--delete-data`. It asks for confirmation (`-y` skips it, and is required without a terminal), refuses a host that runs the control plane, and can be repeated if the cleanup failed after the record was removed. The SSH target defaults to `nodes.<name>.ssh` in the config file.
+- `tug node ingress [node] [--disable]` enables the managed Caddy ingress role, or removes it with `--disable`.
+
+Every control URL, including overrides, must be `https://`. Agents can dial out to control without an inbound control-channel port; cross-node workloads still need reachable node `address` values. See [Nodes](nodes.md) and [Networking](networking.md).
+
+`--control-url` selects where the operator CLI connects and `--agent-control-url` where the agent connects; both are connect targets only, and the server is always verified by the control name derived from the pinned CA, so the address can be a DNS name, an IP, a tunnel or a moved host. Use them when the operator connects through a local SSH tunnel while nodes connect over a private network, for example `tug --control-url https://127.0.0.1:18082 node add root@app1.example.com --name app1 --agent-control-url https://10.110.0.3:8443`.
+
+## Secrets
+
+`tug deploy` also resolves secrets for you. It scans the Compose file for variables with no default (`${NAME}` or `$NAME`) that are not a control secret, in the environment or in `.env`. In a terminal it asks, for each one, whether to generate a random 192-bit value or to enter one (hidden), and stores it before deploying; existing secrets are never overwritten and generated values are never printed. Without a terminal, or when `CI` is set, it fails immediately naming the variables, so a pipeline never hangs or deploys an empty value.
+
+- `tug secret set <name> [--value <value>]` sets a secret, reading stdin if `--value` is omitted. After you enter the value and press Enter, it prints the result and exits without requiring EOF (Ctrl-D).
+- `tug secret list` lists names and versions, without values.
+- `tug secret remove [name]` deletes a secret.
+
+## Backups
+
+The backup commands are `tug backup ...` and `tug restore ...`. These are workload backups of service commands and volumes; `control backup` below is the separate control-plane backup. Destination and restore commands need an admin credential.
+
+- `tug backup destination set <url> [--env NAME ...]` sets where backups go, for example `s3:https://s3.example.com/bucket/tug-backups` with `--env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY`. Each value is read from the environment variable of that name, otherwise from a line on stdin; it is never a command-line argument. S3 needs both AWS variables. Values are stored encrypted on control and released only to a node running a backup. An absolute path is accepted for development, but it is read on each node and is not shared. Setting a destination replaces the previous one.
+- `tug backup run [targets...] [--no-wait] [--timeout <seconds>]` backs up the named services or volumes now, or every backup target of the project, and waits for the result. It exits nonzero if any backup fails or is missed. Needs the `deploy` role.
+- `tug backup list [--limit <n>] [--check]` shows each backup target with its node, schedule, last success and next run, then the most recent runs. With `--check` it also prints a `PROBLEM` line for each target that is stale (no success for two periods), whose last run failed or was missed, or when no destination is configured, and exits nonzero, so cron or a monitor can alert without parsing output. Needs the `read` role.
+- `tug restore run [target] [--from <id|latest>] [--scratch] [--no-safety-backup] [--writers <service>...] [--no-writers] [--confirm]` restores a service or volume. Without `--confirm` it prints the plan and changes nothing. `--from` takes a backup run id, a restic snapshot id or `latest`. See [Restore](backups.md#restore). It waits until the services it stopped report serving again (up to five minutes, then it finishes with a warning). Needs an admin credential.
+- `tug restore list` shows recent restores; `tug restore resume [id]` runs an interrupted one again; `tug restore release [id]` starts the services it stopped and unlocks the project.
+- `tug backup destination show` prints the URL and the variable names, never values.
+- `tug backup destination remove` removes the destination and its credentials. Existing backups stay in the repository.
+- `tug backup password` prints the project's repository and its password. Each project has its own restic repository at `<destination>/<project>` with a password derived from control's master key, so keep the printed password with the destination credentials: losing the master key loses every repository.
+
+These operations use the resolved project.
+
+## Deployments
+
+- `tug deploy [-f, --file <file>] [--version <version>] [--build-local] [--no-wait] [--timeout <seconds>]` deploys the first of `compose.yaml`, `compose.yml`, `docker-compose.yaml` or `docker-compose.yml` found in the working directory by default. Services with `build:` upload the current directory for remote builds, or build on the operator's Docker daemon with `--build-local` and upload saved images without source. Published-image-only applications need no context upload. Control rejects a built service with no available build input. The display version defaults to `git-<Compose hash>`; deployment image identities are immutable even when that display version is reused. Local builds are single-platform and must match the target node architecture. Deployment waits for convergence by default, with a `300` second timeout; `--no-wait` returns once desired state is created. Needs Docker Compose v2 on the operator machine to normalize the file. With `--move <service>` (repeatable) it acknowledges that a service with data placed on another node starts empty there: without the flag such a deploy is refused. The services that write to the moved one are held until you restore it (`tug restore run`) or run `tug hold release`. If the wait is interrupted (control unreachable, timeout) the error names the deployment and `tug deploy cancel [id]`; a deploy refused because another is still rolling out names that one the same way. It prints one summary line (`site git-ab12: deployment dep_1 succeeded, 3 services`); `--json` prints the full result with the desired state. `rollback` and `scale` print a summary the same way and also take `--json`. `deploy` warns when the file's `name:` differs from the Tug project, because secrets, backups and restores are scoped to the project.
+- `tug status [--json]` summarizes each project’s current deployment, node reports and errors, plus node status and last contact. Stale node health is identified as the last report. `--json` prints the raw API result (all projects, recent deployments, nodes and per-node status) for scripts. This is still a control-wide, credential-scoped view.
+- `tug events [--limit <n>] [--all]` lists events in the resolved project, or in every project with `--all` (including control events that belong to no project, such as credential changes); the limit defaults to `100`, with a maximum of `500`. Control actions show the `actor` credential name that caused them. Agents add a `service.*` event when a service changes, not on every poll: `pulling`, `starting`, `health_checking` and `serving` when a new revision, changed secrets or a changed rendered configuration are applied, one `failed` event when an apply fails, and one `serving` event when it recovers. Control deletes events older than 30 days and keeps at most 100,000; see [Configuration](configuration.md#control-plane).
+- `tug rollback [--force] [--no-wait] [--timeout <seconds>]` rolls the resolved project back; `--force` permits rollback while a deployment is active. Waiting defaults to `300` seconds. Projects with `x-tug: { rollback: auto }` roll back failed deployments themselves; see [Compose extensions](compose.md#automatic-rollback).
+- `tug deploy cancel <deployment-id>` cancels an active deployment.
+
+To build on a laptop and upload image archives, run `tug deploy --build-local -f docker-compose.yml`. To deploy CI-built images, push them to your registry and declare `image:`; control pins tags to digests before publishing the deployment. See [Private registry credentials](control-plane.md#published-images-and-private-registries).
+
+Local builds use generated temporary image tags and remove those tags after saving each archive, including cleanup attempts after build/save failure. They do not remove your published image tags or the Tug runtime image. A refused cleanup produces a warning without hiding the original build error.
+
+## Operator credentials
+
+Scoped credentials for CI and teammates; see [Access control](access-control.md) for roles, scoping and a GitHub Actions example. All three commands need an admin credential.
+
+- `tug credential create <name> --role read|deploy|admin [--project <name> ...] [--expires-in-days <n>]` creates an operator credential and prints it once on stdout; control keeps no copy of its private key. Repeat `--project` to allow several projects; without it the credential covers all projects (admin credentials cannot be limited). Here `--project` selects the credential's scope rather than the connection project. Names are 1 to 64 letters, numbers, dots, underscores or hyphens, and cannot be reused after revocation. Expiry is at most `365` days and defaults to `365`.
+- `tug credential list` lists operator credentials with role, projects, creator, last use, expiry and revocation, never secrets.
+- `tug credential revoke [name]` revokes a credential immediately and closes its live sessions.
+
+Use a created credential anywhere a credential works, for example `TUG_CREDENTIAL=<credential> tug deploy`.
+
+## Services and volumes
+
+- `tug restart [service] [--timeout <seconds>]` restarts a service and waits for the result; timeout defaults to `60` seconds.
+- `tug hold release [service...]` starts the services that `tug deploy --move` held, without restoring them (all of them when none is named). A restore releases its own holds.
+- `tug image prune [--keep <n>] [--dry-run]` (admin) deletes build archives kept on control that rollback no longer needs: all but the newest `n` deployments of each project (default 5), the current deployment and any version a node still reports. Rolling back to a pruned deployment is refused; deploy again instead. `tug control backup --no-images` leaves the archives out of a control backup.
+- `tug scale [service] [replicas] [--replicas <n>]` changes how many matching nodes run a service, spreading by count over nodes that carry the service's `placement.labels` (all nodes without labels), staying within an explicit `placement.nodes` when the service has no labels, and updates the current desired state without a new deployment. `global` services run on every matching node and are scaled to that set. The next `deploy` re-resolves placement from the Compose file.
+- `tug exec [service] [-i, --interactive] [-t, --tty] [--node <name>] [-- <command> ...]` executes a command in the current service container, defaulting to `sh`. Arguments are passed directly without a shell. `-i` forwards stdin. `-t` allocates a terminal (with resize) and requires a local terminal. Use `--node` when a service has more than one placement.
+- `tug run [service] [--node <name>] -- <command> ...` runs a one-off command in a new container instead of the running one: the service's image, environment (secrets included), user, working directory, network and volume mounts, no published ports, removed when it ends. It is for jobs that should not run inside the live container, such as `tug run web -- wp core update-db`. It is non-interactive, exits with the command's exit code, needs the `deploy` role and is recorded as a `job.run` event with its command line. Mounts are shared with the service, so a destructive command is the operator's responsibility; the one-off container does not take the project lock.
+- `tug logs [service] [--tail <n>] [-f, --follow] [--node <name>]` fetches recent service logs through the durable command queue, or follows them live with `-f`; the default is `200` lines. `--node` applies to followed logs and is required when the service has more than one eligible node.
+- `tug ps [--node <name>]` prints live per-container state as JSON. Without `--node`, the project must resolve to exactly one eligible connected node.
+- `tug inspect [service] [--node <name>]` prints the live Docker inspection object as JSON with environment variable values redacted (names are kept). It requires the `deploy` role, which grants access to the workload's secrets regardless. Use `--node` when a service has more than one placement.
+- `tug volume list` lists recorded volumes.
+- `tug volume remove [name] [--delete-data]` removes a volume record; `--delete-data` also removes the Docker volume on its node. See [Compose extensions](compose.md#volumes).
+
+These operations use the resolved project. Live operations use an authenticated WebSocket from the CLI through control to the agent's existing outbound connection. They are ephemeral: control does not queue or replay them, and disconnecting any hop closes the local session. Cancelling or disconnecting `exec` kills the agent's Docker client, but an already-started process inside the container may continue because the Docker exec stream does not expose a remote process identifier.
+
+## Operations
+
+- `tug doctor [--nodes] [--node [name]] [--timeout <seconds>]` runs diagnostics on this machine (Node.js, Docker and the Compose plugin that `deploy` needs) and on the control plane: SQLite, the CA and server certificate expiry, each node's certificate expiry and connectivity, each online node's observed clock skew (unknown for offline nodes; see [Nodes](nodes.md#doctor-certificate-expiry-and-clock-skew)), and ingress status. `--nodes` also runs host diagnostics through the agent on every node and `--node` on one; both wait for the agent result (timeout defaults to `60` seconds) instead of returning the queued command id. An unknown `--node` is an error. Node diagnostics include `ports.public`, a warning listing containers of Tug projects that listen on all interfaces or a public IP ([Networking](networking.md#published-ports-private-by-default)).
+- `tug control backup` creates a control-plane backup: control's state, PKI, master key and retained built-image archives. It does not capture workload volume contents; those are covered by [workload backups](backups.md).
+- `tug control restore <artifact-path> --confirm` restores a control-plane backup and requires a control restart. The archive path is on the control-plane host. See [Control plane](control-plane.md#backup-and-restore).
+
+## CLI architecture and naming
+
+The interface follows the relevant conventions used by `wp-release`: `tug <subject> <verb>` operations, with the everyday project commands at the top level; kebab-case command/flag names; primary targets as positional arguments; named qualifiers with consistent meanings; and positive camelCase internal options (`--no-wait` becomes `wait: false`, never `noWait`). New structured CLI results use camelCase; API responses printed unchanged may retain their existing database field names. Help, docs and tests must change together when the interface changes.
+
+CLI adapters resolve configuration, parse arguments and format results. Build, archive, installation and API primitives receive explicit values and injectable runners; they do not discover CLI config or read unrelated environment variables. Keep domain logic in ordinary module functions, and introduce stateful classes only when a multi-step workflow needs shared state/dependencies. Only operations that use Docker or SSH should require those tools. Validation must complete before side effects, and stdout intended as a runnable artifact or JSON must remain directly usable. These conventions do not imply GitHub Actions logging or additional output modes.