@wpelevator/tug 0.12.0 → 0.13.0

package/docs/access-control.md

@@ -0,0 +1,166 @@
+# Access control
+
+Control listens on one TLS port (default `8443`) and requires a client certificate from every peer. There is no plain HTTP listener, no unauthenticated route and no bearer token. Certificates are signed by control's private CA; clients pin that CA and still verify the server name. A missing or foreign certificate fails the TLS handshake before any HTTP is parsed. After the handshake, a gate and then every request and WebSocket upgrade check the principal in the database, so revocation and key replacement apply to the next request and close live sessions within a second.
+
+The certificate's URI SAN names the **identity class**, which decides which routes the peer can reach at all:
+
+| SAN | Class | May reach |
+| --- | --- | --- |
+| `tug:node:<id>` | node (`<id>` is the immutable node ID, not the node name) | `/agent/v1/*` (including `POST /agent/v1/renew` and the agent stream) |
+| `tug:operator:<id>` | operator or CI | `/api/v1/*` and `/api/v1/live`, subject to role and project scope |
+| `tug:enroll:<id>` | bootstrap | `POST /agent/v1/enroll` only |
+
+A principal outside its class gets `403`. Authorization follows the **public key**, not the certificate: control stores the SHA-256 of each principal's SubjectPublicKeyInfo (`key_hash`) and accepts any CA-signed certificate for that key, so renewal is just signing another certificate for the same key. Replacing a key makes every certificate for the old key unusable at once.
+
+Operator credentials come in two kinds, both with a unique name, a role and optionally allowed projects:
+
+- **`root`** is an ordinary named admin issued on the control-plane host by `tug control setup` (`control issue-credential --name root --role admin`). Upgrades never reissue it; `tug control setup --recover-root` replaces only root's key without the API.
+- **Named credentials** are for CI systems, teammates and automation. Create them with `tug credential create` (control generates the key and returns the credential once; it keeps no private-key copy) and revoke them with `tug credential revoke`. They last one year at most.
+
+Replace a credential by creating a new name, verifying it, updating consumers and then revoking the old name. Keep at least one verified admin throughout.
+
+## Roles
+
+Roles are cumulative.
+
+| Role | Can do |
+| --- | --- |
+| `read` | Status, events, project and deployment records, volume listings and secret names (never values). No mutations. |
+| `deploy` | Everything in `read`, plus create projects, deploy, roll back, cancel deployments, restart services, fetch service logs and read command results, for its allowed projects. |
+| `admin` | Everything an operator can do: secrets, volumes, nodes, enrollment, backups, restore, diagnostics and credential management. Admin credentials cannot be limited to projects. |
+
+A `deploy` credential must be trusted with its workloads' secrets: live `inspect` redacts environment values to avoid accidental disclosure, but `exec` can read them, and deployed code can disclose any secret it receives. Setting and deleting secrets is `admin` only; listing secret names is `read`. The deploy role can trigger project backups, while control backups and restores require `admin`.
+
+Project scope limits API access to project records; it does not isolate hosts or co-located projects. Current deployments can select shared nodes and request host bind mounts, external volumes, volume driver options and added capabilities. Treat deployment authors as administrators of the nodes they can select. Use separate hosts or control planes for mutually untrusted authors. See the [security audit](security.md) for the demonstrated boundaries and outstanding fixes.
+
+## Endpoint table
+
+This table is the contract. It mirrors `ROUTE_ROLES` in `src/credentials.ts`, `createApp` refuses to register a `/api/v1` route that is not listed there, and a test compares the table with the live router and with this page, so a new route cannot ship without a declared role.
+
+The project column says how a scoped credential's allowed projects are checked: the project comes from the request body, the query string (default `site`), the path, or the stored record for the given ID. Admin-only routes are control-wide.
+
+| Route | Role | Project check |
+| --- | --- | --- |
+| `GET /api/v1/projects` | `read` | Response is filtered to allowed projects |
+| `GET /api/v1/services` | `read` | Query; service names and eligible placements only |
+| `GET /api/v1/deployments` | `read` | Query; active deployments only |
+| `GET /api/v1/backup/snapshots` | `read` | Query; successful catalog backups for target |
+| `GET /api/v1/holds` | `admin` | Query; deployment move holds only |
+| `GET /api/v1/status` | `read` | Response is filtered to allowed projects |
+| `GET /api/v1/events` | `read` | `project` query, or filtered to allowed projects |
+| `GET /api/v1/projects/:project` | `read` | Path |
+| `GET /api/v1/deployments/:id` | `read` | Deployment's project |
+| `GET /api/v1/volumes` | `read` | Query; includes node name, persistence and referencing services |
+| `GET /api/v1/secrets` | `read` | Query (names and versions only) |
+| `POST /api/v1/projects` | `deploy` | Body `name` |
+| `POST /api/v1/deployments` | `deploy` | Body or upload payload `project` |
+| `POST /api/v1/deployments/:id/cancel` | `deploy` | Deployment's project |
+| `POST /api/v1/rollback` | `deploy` | Body |
+| `POST /api/v1/scale` | `deploy` | Body |
+| `POST /api/v1/restart` | `deploy` | Body |
+| `POST /api/v1/logs` | `deploy` | Body |
+| `GET /api/v1/live` | `deploy` | First WebSocket frame |
+| `GET /api/v1/commands/:id` | `deploy` | Command's project; commands without a project (diagnostics) need `admin` |
+| `POST /api/v1/backups/run` | `deploy` | Body `project` |
+| `GET /api/v1/backups` | `read` | Query `project` |
+| `POST /api/v1/restores` | `admin` | Body `project`; without `confirm` it only returns the plan |
+| `GET /api/v1/restores` | `read` | Query `project`; `state=interrupted` returns all interrupted restores |
+| `POST /api/v1/restores/:id/release` | `admin` | Drops the holds of an interrupted restore |
+| `POST /api/v1/restores/:id/resume` | `admin` | Runs an interrupted restore again |
+| `GET /api/v1/backup/destination` | `admin` | Control-wide; URL and variable names, never values |
+| `POST /api/v1/backup/destination` | `admin` | Control-wide |
+| `DELETE /api/v1/backup/destination` | `admin` | Control-wide |
+| `GET /api/v1/backup/repository-password` | `admin` | Query `project`; prints the repository password |
+| `POST /api/v1/secrets` | `admin` | Control-wide |
+| `DELETE /api/v1/secrets/:name` | `admin` | Control-wide |
+| `DELETE /api/v1/volumes/:name` | `admin` | Control-wide |
+| `GET /api/v1/nodes` | `admin` | Control-wide |
+| `POST /api/v1/nodes/:name/identity` | `admin` | Control-wide |
+| `POST /api/v1/nodes/:name/disable` | `admin` | Control-wide |
+| `POST /api/v1/nodes/:name/address` | `admin` | Control-wide |
+| `POST /api/v1/nodes/enrollment-credential` | `admin` | Control-wide |
+| `POST /api/v1/nodes/:name/ingress` | `admin` | Control-wide |
+| `DELETE /api/v1/nodes/:name` | `admin` | Control-wide |
+| `POST /api/v1/control/backups/run` | `admin` | Control-wide |
+| `POST /api/v1/control/restore` | `admin` | Control-wide |
+| `POST /api/v1/holds/release` | `admin` | Body `project`; drops the holds a deploy placed for a move |
+| `POST /api/v1/images/prune` | `admin` | Control-wide; deletes build archives older than the kept deployments |
+| `GET /api/v1/diagnostics` | `admin` | Control-wide |
+| `POST /api/v1/diagnostics/nodes` | `admin` | Control-wide |
+| `POST /api/v1/credentials` | `admin` | Control-wide |
+| `GET /api/v1/credentials` | `admin` | Control-wide |
+| `DELETE /api/v1/credentials/:name` | `admin` | Control-wide |
+
+## Failures
+
+- `401` means the certificate is unknown, revoked, expired, disabled or its key is not the stored one. A missing or foreign certificate never gets this far: the TLS handshake fails.
+- `403` means the credential is valid but its role is too low (`forbidden: this action requires the deploy role`) or the project is outside its scope (`forbidden: this credential may not access that project`). The project message is identical whether the other project exists, whether a deployment or command ID exists, or whether the project name is valid, so a scoped credential cannot probe for projects or IDs.
+
+## Projects and scoping
+
+A `read` or `deploy` credential created with `--project` may only act on those projects. Listings are filtered rather than refused: `status` returns only allowed projects, their deployments and their node status, and `events` without `--project` returns only events of allowed projects (events without a project, such as credential changes and node events, are hidden from scoped credentials). A credential without `--project` covers every project, including ones created later. `status` still lists node names and health to every credential, since placement names appear in Compose files.
+
+## Audit trail
+
+Every deployment and every control event records the acting credential name, `root` for the root credential. `tug events` and `GET /api/v1/events` return it as `actor`, and deployment records (`GET /api/v1/deployments/:id`, `tug status`) carry it as `actor`. Control actions write events of these types: `deploy`, `rollback`, `cancel`, `restart`, `secret.set`, `secret.delete`, `credential.create` and `credential.revoke`. Events never contain secret values or credential private keys. Rollbacks that control starts by itself (`x-tug: { rollback: auto }`, see [Compose extensions](compose.md#automatic-rollback)) and their `deployment.auto_rollback` events are attributed to the reserved actor `system:auto-rollback`; credential names cannot contain a colon, so no credential can use it. Events written by agents have no `actor`. Control events that belong to no project (credential changes) show up in `tug events --all`.
+
+## Managing operator credentials
+
+```bash
+tug credential create ci-site --role deploy --project site
+tug credential create ci-site --role deploy --project site --project blog --expires-in-days 90
+tug credential create alice --role read
+tug credential list
+tug credential revoke ci-site
+```
+
+- Names are 1 to 64 letters, numbers, dots, underscores or hyphens. A revoked credential keeps its name so audit entries stay unambiguous; choose a new name to replace it.
+- `credential create` prints the credential once, on stdout; a note on stderr reminds you to store it. Only the key hash is stored, so a lost credential cannot be recovered, only replaced.
+- `credential list` shows name, role, projects, creator, creation, last use, expiry and revocation time, never secrets.
+- Revocation applies to the next request. Expired credentials are refused with `401` the same way. `last_used_at` is refreshed at most once a minute in the background, so it is approximate and bookkeeping never delays or fails a request.
+- Credential records (name, role, projects, key hash, expiry, revocation) live in the control database, so they are part of `tug control backup` and come back with `tug control restore`; a restore returns the credential set (including revocations) to the backup's state. Private keys are never stored on control.
+- `root` is listed like any other admin credential. Use `--recover-root` if its credential is lost; prefer day-to-day use of named admin credentials.
+
+## Deploying from GitHub Actions
+
+Create a project-scoped `deploy` credential and store it as a repository secret (the credential already carries the control plane's default address and CA):
+
+```bash
+tug credential create github-site --role deploy --project site
+```
+
+Add `TUG_CREDENTIAL` (the printed credential) under the repository's Settings, Secrets and variables, Actions. A workflow:
+
+```yaml
+name: Deploy
+on:
+  push:
+    branches: [main]
+
+concurrency:
+  group: deploy-site
+  cancel-in-progress: false
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+      - name: Deploy
+        env:
+          TUG_CREDENTIAL: ${{ secrets.TUG_CREDENTIAL }}
+          TUG_PROJECT: site
+        run: npx --yes @wpelevator/tug deploy -f docker-compose.yml --version "git-${GITHUB_SHA::12}"
+```
+
+`deploy` waits for the rollout and fails the job when the deployment fails or times out (add `--no-wait` to return immediately). Services with `build:` upload the checkout for a remote build on control; add `--build-local` to build on the runner, which has Docker, and upload the images instead. Pin the CLI version (`@wpelevator/tug@<version>`) to match your control plane. Rolling back from CI needs only the same credential (`npx @wpelevator/tug rollback`).
+
+## Limits
+
+- A `deploy` credential can run any image or Compose configuration on the nodes of its projects, execute commands in those containers, and read their logs and Docker inspection data. Treat it as access to those nodes: inspection redacts environment values, but `exec` and deployed code can read them, and logs can contain whatever the application prints. Scope credentials to the projects they need and expire short-lived ones.
+- Control-wide state (nodes, backups, diagnostics, credentials, secrets and volume deletion) is admin only. There is no per-project secret management role.
+- Roles and project scopes are fixed at creation; to change them, create a new credential and revoke the old one.
+- A `deploy` event is written when a deployment is published. A build that fails before publishing appears only in its deployment record (with its `actor`), not as an event.

package/docs/architecture.md

@@ -0,0 +1,254 @@
+# Architecture
+
+Tug deploys Docker Compose applications to one or more Docker hosts through a small self-hosted control plane and a pull-based agent on each node. The agents drive each node's Docker Engine API directly; Compose is used only on the operator machine to normalize the file. SSH is used only to install the control plane and nodes; normal deploys go through the control API and the agents, which share one mutual-TLS port. [Terminology](terminology.md) defines the vocabulary used here.
+
+## Components
+
+The [README](../README.md#how-it-fits-together) has an overview diagram of how these fit together.
+
+- **Control plane** (`src/control.ts`): an Express API behind a single TLS 1.3 listener that requires a client certificate ([Control plane](control-plane.md#authentication-and-pki)), backed by SQLite (`tug.db`), an install-time master key (`master.key`) and the private CA (`pki/`) in `TUG_DATA_DIR`. It stores nodes, projects, deployments, desired states, encrypted secrets, volumes, queued commands, events, operator and enrollment credentials (public-key hashes, never private keys) and locks, and builds images from uploaded source tarballs or imports operator-built archives. It always stores and serves built images as compressed artifacts and resolves published image tags to immutable registry digests before publication.
+- **Access control** (`src/credentials.ts`, `src/pki.ts`): every peer presents a certificate signed by control's CA. The certificate names one principal (`tug:operator:<id>`, `tug:node:<id>` or `tug:enroll:<id>`) and control authorizes it by comparing the certificate's public-key hash with the key stored for that principal. Every operator route is declared once in a role table (`read`, `deploy`, `admin`) and passes through one guard that checks the role and the credential's allowed projects, and attaches the acting credential name that deployments and audit events record. Agent routes derive the node from the certificate and touch only that node's objects. See [Access control](access-control.md).
+- **Agent** (`src/agent.ts`): runs on every node, fetches authoritative work from `GET /agent/v1/work` and reconciles containers through the Docker Engine API (`src/docker-api.ts`) over the local socket. It builds one container definition per placed service (`src/container-spec.ts`), substitutes released secrets into the container environment, verifies/downloads/loads artifact-backed images, pulls published images by their pinned registry digest when the desired revision is new to the node, executes health checks, and reports status. Nodes need only the Docker Engine, not the Compose plugin.
+- **Reconciler** (`src/reconciler.ts`): one loop for every service. Containers carry `tug.project`, `tug.service`, `tug.revision` and `tug.config-hash` labels; the hash covers the resolved definition, the image identity and the versions of the secrets the service uses, and a container whose hash differs is replaced. `recreate` stops the old container and starts the new one; `rolling` (`x-tug.rollout.strategy: rolling`) starts the new one on a free host port beside the old one, health-checks it, lets the ingress switch and removes the old one after a drain. Tug names its own network `tug_<project>`, volumes `tug_v<project-length>_<project>_<volume>` and containers `tug-<project>-<service>-<hash8>`; service names are network aliases. Reconciliation never removes volumes; `volume.delete` is the only way one goes away.
+- **Compose normalization** (`src/normalize.ts`): `tug deploy` runs `docker compose config` on the operator machine, which does interpolation, `extends`, profiles and `env_file`. Control secrets are interpolated as `@@tug:secret:NAME@@` sentinels that the agent replaces with released values. The CLI reads Compose's YAML rendering because its JSON rendering drops service-level `x-tug`. Control validates the result against an allowlist of keys ([Compose](compose.md#supported-keys)); agents never parse Compose YAML.
+- **CLI** (`src/cli.ts`): the `tug` command for operators. It talks to the control API and uses SSH only for `control setup` and `node add`. See [CLI reference](cli.md).
+- **Networking** (`src/ingress.ts`, `src/node-address.ts`): health-aware service endpoints, private-address port binding and the managed Caddy edge. `ingress.ts` resolves service endpoints and ingress routes from each node's `address`, renders the Caddyfile (control) and reconciles the Caddy container (agent). Tug provides no overlay or tunnel; the operator owns the inter-node network. See [Networking](networking.md).
+- **Runtime entrypoint** (`src/main.ts`, installed as the `tug-runtime` binary and `dist/main.js` in the image): one image runs both control and agent; `node src/main.ts control|agent` or `TUG_ROLE`. It also provides the `agent identity` and `agent install-cert` subcommands that node installation uses.
+- **Protocol** (`src/protocol.ts`): the control↔agent contract. It holds the types, and a few shared constants such as the active and terminal deployment states, for everything crossing the boundary: `DesiredState`, `ServiceSpec` and `HealthCheck`; the `/agent/v1/*` payloads (`WorkResponse` with its optional `serviceEndpoints`, `ports` and `ingress` blocks, `Command` / `CommandResult`, `StatusReport` / `ProjectStatus` / `IngressStatus`, `StreamMessage`, `WakeResponse`, `SecretsResponse`); and identity and enrollment (`IdentityRequest` / `IdentityResponse`, `EnrollRequest` / `EnrollResponse`, `RenewRequest` / `RenewResponse`, `EnrollmentCredentialResponse`, `JoinKey`). `control.ts`, `agent.ts`, `compose.ts` and `cli.ts` all import from it, so change the wire format there first.
+- **Injectable agent runners** (`src/agent.ts`, `src/runners.ts`): `reconcile`, `executeCommand`, `waitForHealth`, `containerIp` and `pollOnce` take an `AgentContext` (`{ docker, http, nodeName, stateDir, apply, controlUrl, credential?, caddyImage?, rolloutPorts? }`). `docker` is a `DockerApi` (the real `DockerEngine` talks to the Engine API over the local socket, `tests/fake-docker.ts` implements it in memory), and `http` is a `fetch`-compatible client pinned to the node's credential. Production code uses `defaultContext()`; the agent tests pass the in-memory Docker fake to unit-test rollout ordering, secret-triggered recreation and the docker, tcp and http health-check branches without a Docker daemon. The privileged-Docker integration test remains the end-to-end check.
+
+## Desired state and reconciliation
+
+`tug deploy` sends the Compose file to control, which validates `x-tug` placement (see [Compose extensions](compose.md)), optionally builds images for services with `build:`, and stores an immutable desired-state record for the deployment. Each agent receives the desired states that place services on its node, applies them, and reports the observed display version, immutable desired-state revision and health per project. When every assigned node reports the current revision as `serving` (or `skipped`), control also waits for every enabled managed-ingress node to report the current routing configuration as applied when the project adds, changes or removes ingress routes. The acknowledged configuration must contain each current project route with a serving upstream; a failed, stale or older ingress report cannot complete deployment. With no enabled ingress node, convergence reports service health only. This acknowledgement confirms the agent applied its Caddy configuration, not external DNS, certificate issuance or a successful public request. A failed node report still fails the deployment with its apply and rollout errors; an unavailable ingress can leave the deployment `rolling_out` until it recovers or the CLI times out.
+
+Deploys, rollbacks, secret changes and data-deleting volume removals take a per-project operation lock, so concurrent operations on one project are rejected with `409`. Long builds renew the project lease and check ownership/cancellation before publication. Deploy publication, rollback and artifact backup/restore share an operation-lifetime writer queue. Rollback creates a new deployment and revision whose desired state copies the most recent earlier `succeeded` or `rolling_out` one. With `x-tug.rollback: auto` control creates that rollback itself when a deployment turns `failed`; see [Rolling rollouts and automatic rollback](#rolling-rollouts-and-automatic-rollback).
+
+Restarts, log fetches, diagnostics and volume deletions are queued as commands for the node that runs the service; control wakes that node, and the agent fetches the command and posts its result back.
+
+### Agent wake channel
+
+After each reconciliation, the agent waits on `GET /agent/v1/stream`, a WebSocket upgrade authenticated by the same node certificate as the HTTP agent API. Control sends a `work-available` hint for durable work; desired state and queued commands remain in SQLite and the agent always fetches them through `GET /agent/v1/work`. Deployments, rollbacks, secret changes and cluster-wide network changes wake every node because they can change placement, service endpoints or ingress. Node commands and ingress-role changes wake only their target node.
+
+The same socket carries ephemeral live operations. The CLI opens `GET /api/v1/live` with its operator credential, sends one versioned request, and control resolves the project, service and optional node before binding the session to that exact connected agent. For a rolling service, the agent addresses the active rollout container instead of the Compose service container. `exec` (with a PTY when asked), `tug run` one-off containers, `logs -f`, `ps` and `inspect` stream responses directly; input, cancellation and disconnect cleanup travel in the other direction. Live requests are never persisted, queued or replayed, all three hops enforce frame, buffer and session limits, and losing any socket closes the session and kills the agent's local Docker client process. Docker does not expose the remote exec process identity through this client stream, so an already-started process inside the container may continue after cancellation or disconnect; do not treat client cancellation as remote process termination. `inspect` is deliberately a `deploy` operation and redacts environment values; it does not limit what a deploy credential can read through `exec`.
+
+If a proxy or firewall rejects WebSockets, the agent uses `GET /agent/v1/wake?after=<cursor>&timeout=<milliseconds>` for durable work only. Each node has a monotonic in-memory wake cursor, so a wake issued just before the long poll begins is returned immediately instead of being lost. Cursors and sockets are intentionally ephemeral: control restart merely loses a hint, while periodic `TUG_POLL_INTERVAL_MS` reconciliation remains the correctness and drift-repair path. Live operations require the WebSocket path because replaying them through long-poll would be unsafe.
+
+### Events
+
+Agents report lifecycle events to control, and `tug events` shows them next to control's own audit events. An agent does not report that it is still reconciling: for the recreated services of a project it keeps a fingerprint of the desired revision, the Compose file rendered for its node and the secret versions in its own database (`event_state` in `agent.db`), and sends `service.pulling`, `starting`, `health_checking` and `serving` only when the fingerprint changes. A failed apply produces one `service.failed` event (level `error`, naming the service whose check failed and why) and retries of the same failure stay quiet. When the apply later succeeds the agent sends one `service.serving` event saying it recovered. The agent still applies and health-checks the project after every wake or poll, so drift is repaired the same way; only the events are quiet. Rolling services report their own events when a revision is started, switched or rejected. Control deletes events older than `TUG_EVENT_RETENTION_DAYS` (30) and keeps at most `TUG_EVENT_MAX_ROWS` (100,000); see [Configuration](configuration.md#control-plane).
+
+## SQLite storage
+
+Control metadata and agent applied/event state use Node's built-in synchronous SQLite driver. `src/db-driver.ts` owns the runtime-neutral `Db` contract, binding values and unique/busy error categories; `src/db-node.ts` alone imports `node:sqlite`. Queries return ordinary objects, missing rows are `undefined`, `all<Row>()` returns `Row[]`, and writes expose numeric change counts. Only anonymous `?` parameters are supported, with strict arity and value validation. Integer reads use numbers and reject unsafe values. Unexpected driver failures propagate with their original identity; classified errors preserve the driver error as their cause.
+
+`src/db.ts` selects one connection factory for control, agents and restore snapshots. Writable connections enable WAL, foreign keys and a 1,000 ms busy timeout; initialization closes the connection on failure. Read-only snapshots require an existing file and run no schema creation or journal-mode change. Schemas initialize in place without migrations. Deployment publication executes synchronously from `BEGIN IMMEDIATE` through commit or rollback; writer queues and project leases coordinate workflows that also await files, Docker or network operations.
+
+`src/db-locks.ts` owns control’s project-lock queries, and `src/db-backup-destination.ts` owns destination and encrypted credential storage shared by control and the backup scheduler. Keep HTTP responses, lease timers, encryption, audit events and wake notifications in their callers. These focused modules use parameterized SQL directly; row interfaces remain manually maintained and do not validate SQL projections. Other query families remain in their existing callers for later extraction.
+
+`src/db-transaction.ts` provides `immediateTransaction()` for synchronous database-only work: it begins an immediate transaction, returns the callback result after commit and rolls back on callback or commit failure without masking the original error. Callbacks must not start asynchronous work; promise-like results are rejected as a defensive check, which cannot cancel work already started. Nested transactions are rejected, and a failed begin never rolls back the existing transaction. Destination replacement and removal commit configuration and credentials together; deployment publication uses the same helper.
+
+SQL executes on the owning process's event loop. Busy waits, snapshots and integrity checks can pause APIs, WebSocket traffic and timers; backup blocking is described in [Control plane](control-plane.md#backup-and-restore). Database-only work does not return promises. Filesystem preparation and other asynchronous workflows retain their asynchronous interfaces.
+
+A future Bun bundle that includes control or agent code can supply a `bun:sqlite` adapter through the same factory. Select it at build time so the bundle never imports the Node adapter; normalize results, bindings, error categories and connection cleanup, and run the same factory-driven database contract tests. Bun implementation and build aliases are deferred. The proposed CLI-only executable does not access SQLite and needs no adapter.
+
+## Image artifacts
+
+The `builds` table records immutable image references, config-digest identity, platform, archive SHA-256/size and a relative artifact path. Image save/download/hash operations stream bytes; saved archives are validated before import. Build archives finalize before the desired-state transaction becomes current. Failed publication leaves the previous deployment current. `GET /agent/v1/images/:buildId` authenticates the node certificate and checks the retained service placement; `WorkResponse.retainedImages` protects rollback images from agent pruning.
+
+Docker image stores can expose either a config digest or an OCI index/manifest digest as their local image ID. Artifacts use the saved configuration digest as their canonical identity. After a verified import, agents cache the daemon's actual tag ID paired with that canonical ID and archive hash; they check the tag again before reusing it. This permits classic Docker and containerd-backed Docker to exchange saved images without confusing their different IDs.
+
+Archives are validated before Docker can import them. A containerd-backed `docker save` can list BuildKit build attestations of the image next to it, either nested in the image index or as extra top-level entries. These are accepted only when they carry no image name, their in-toto contents match their digests, and their subject is this exact image; any other extra manifest, extra tag, link, absolute or parent-relative path, or oversized entry rejects the archive.
+
+Local builds use a multipart `POST /api/v1/deployments`: `payload` contains the Compose/project/version and `localImages` descriptors; each descriptor names a corresponding `image-N` file. Each archive must contain exactly its declared Tug tag and actual configuration/platform/checksum. Source contexts and local images are mutually exclusive. Control retags imported images under its own deployment identity before publishing their saved archives.
+
+## Node installation (`tug node add`)
+
+The node makes its own key and certificate request, so no bootstrap secret travels and the private key never leaves the host. The operator's admin credential authorizes the identity at that moment.
+
+```mermaid
+sequenceDiagram
+    participant Op as Operator
+    participant CLI as tug CLI
+    participant Ctrl as Control plane
+    participant DB as SQLite
+    participant Host as Node host (SSH)
+    participant Agent as Agent container
+
+    Op->>CLI: node add root@host --name app1
+    CLI->>Host: SSH preflight (docker, paths, disk, control reachable)
+    CLI->>Host: docker save | scp | docker load (only with --build-local-image)
+    CLI->>Host: ssh: docker run image agent identity (key stays in the state volume)
+    Host-->>CLI: certificate request (CSR)
+    CLI->>Ctrl: POST /api/v1/nodes/app1/identity { csr, mode: create } (admin credential, mTLS)
+    Ctrl->>DB: insert node with a new immutable ID and the CSR key hash
+    Ctrl-->>CLI: { node, certificate, ca, url }
+    CLI->>Host: ssh: agent install-cert (response on stdin)
+    CLI->>Host: ssh node-install.sh (writes /etc/tug/agent.env, docker run agent)
+    Host->>Agent: start container
+    loop every TUG_POLL_INTERVAL_MS
+        Agent->>Ctrl: GET /agent/v1/work (node certificate, mTLS)
+        Ctrl->>DB: update last_seen_at, status=online
+    end
+    CLI-->>Op: { ok, node }
+```
+
+`node replace` and `node repair` run the same flow with `mode: replace` (and, for repair, move corrupt credential documents aside first). See [Nodes](nodes.md).
+
+## Self-service enrollment (`tug node enroll`)
+
+```mermaid
+sequenceDiagram
+    participant Op as Operator
+    participant CLI as tug CLI
+    participant Ctrl as Control plane
+    participant DB as SQLite
+    participant Host as Fresh host
+    participant Agent as Agent container
+
+    Op->>CLI: node enroll --name app1
+    CLI->>Ctrl: POST /api/v1/nodes/enrollment-credential { name, labels, ttlSeconds }
+    Ctrl->>DB: insert enrollment credential (key hash, name, reserved node ID, expiry)
+    Ctrl-->>CLI: bootstrap credential
+    CLI-->>Op: docker run ... -e TUG_JOIN_KEY=<join key> ... agent
+    Op->>Host: paste docker run (or cloud-init)
+    Host->>Agent: start container
+    Agent->>Agent: no identity on the volume -> decode join key, make key and CSR
+    Agent->>Ctrl: POST /agent/v1/enroll { csr, address } (bootstrap certificate, mTLS)
+    Ctrl->>DB: validate and consume the key, insert node with the reserved ID
+    Ctrl-->>Agent: { node, certificate, ca, url }
+    Agent->>Agent: persist agent.json to the state volume
+    loop every TUG_POLL_INTERVAL_MS
+        Agent->>Ctrl: GET /agent/v1/work (node certificate, mTLS)
+        Ctrl->>DB: update last_seen_at, status=online
+    end
+```
+
+The bootstrap certificate reaches only `POST /agent/v1/enroll`. A retry with the same key signs another certificate for the same node without changing stored values.
+
+## Deploy (`tug deploy`)
+
+```mermaid
+sequenceDiagram
+    participant Op as Operator
+    participant CLI as tug CLI
+    participant Ctrl as Control plane
+    participant DB as SQLite
+    participant Agent as Agent (per node)
+
+    Op->>CLI: deploy -f docker-compose.yml
+    CLI->>CLI: read & hash compose, tar build context
+    CLI->>Ctrl: POST /api/v1/deployments (compose + context)
+    Ctrl->>DB: lock project, insert deployment (state=building)
+    Ctrl->>Ctrl: build images; save archives; pin published image digests
+    Ctrl->>DB: insert desired_state, set current_deployment_id, state=rolling_out
+    Ctrl-->>CLI: { deployment, desiredState }
+    loop poll interval
+        Agent->>Ctrl: GET /agent/v1/work
+        Ctrl-->>Agent: matching desiredStates for this node
+        Agent->>Agent: fetch secrets, pull or load images, recreate containers whose config hash changed, health checks
+        Agent->>Ctrl: POST /agent/v1/status (observedVersion, observedRevision, health)
+        Ctrl->>DB: upsert node_statuses, check convergence
+    end
+    Ctrl->>DB: all nodes serving desired revision -> deployment state=succeeded
+    CLI->>Ctrl: GET /api/v1/deployments/:id (poll until terminal state)
+    CLI-->>Op: deployment result
+```
+
+## Rollback (`tug rollback`)
+
+```mermaid
+sequenceDiagram
+    participant Op as Operator
+    participant CLI as tug CLI
+    participant Ctrl as Control plane
+    participant DB as SQLite
+    participant Agent as Agent (per node)
+
+    Op->>CLI: rollback [--force]
+    CLI->>Ctrl: POST /api/v1/rollback { project, force }
+    Ctrl->>DB: lock project, reject 409 if active deployment & not --force
+    Ctrl->>DB: find most recent prior succeeded/rolling_out desired_state
+    Ctrl->>DB: insert new deployment (version=rollback-...), insert desired_state copy, set current_deployment_id
+    Ctrl-->>CLI: { deployment, desiredState: previousState }
+    loop poll interval
+        Agent->>Ctrl: GET /agent/v1/work
+        Ctrl-->>Agent: previous desiredState
+        Agent->>Agent: reconcile (pull/apply prior images, health checks)
+        Agent->>Ctrl: POST /agent/v1/status (observedVersion=previousState.version)
+        Ctrl->>DB: convergence check -> deployment state=succeeded
+    end
+    CLI-->>Op: rollback deployment result
+```
+
+## Rolling rollouts and automatic rollback
+
+Services are `recreate` by default: a container whose config hash changed is stopped and replaced in place. A service with `x-tug.rollout.strategy: rolling` is rolled instead, one changed service at a time:
+
+```mermaid
+sequenceDiagram
+    participant Op as Operator / CI
+    participant Ctrl as Control plane
+    participant A as Agent (app node)
+    participant E as Agent (ingress node) + Caddy
+
+    Op->>Ctrl: POST /api/v1/deployments (web v2, rolling, rollback: auto)
+    Ctrl->>Ctrl: validate, store desired state, deployment rolling_out
+    A->>Ctrl: GET /agent/v1/work (desired state v2)
+    A->>A: hash the rendered service + its secret versions (new revision?)
+    A->>A: allocate a free host port, persist candidate in rollout.json
+    A->>A: create and start container tug-proj-web-HASH via the Engine API (old revision untouched)
+    A->>A: health check the new container
+    alt healthy
+        A->>A: active = new, old revision scheduled to drain
+        A->>Ctrl: POST /agent/v1/status (web serving, port 20001)
+        Ctrl->>Ctrl: endpoint port 20001, re-render Caddyfile; deployment still rolling_out
+        E->>Ctrl: GET /agent/v1/work (new Caddyfile)
+        E->>E: caddy reload (graceful, no dropped connections)
+        E->>Ctrl: report applied configuration hash
+        Ctrl->>Ctrl: matching ingress acknowledged -> deployment succeeded
+        A->>A: a later poll, after the drain: remove the old revision
+    else health check fails or no free port
+        A->>A: remove the new revision, remember the failure for this desired revision
+        A->>Ctrl: POST /agent/v1/status (health failed, web still serving, old port)
+        Ctrl->>Ctrl: deployment failed, ingress keeps the old port
+        opt x-tug.rollback is auto and a previous succeeded deployment exists
+            Ctrl->>Ctrl: createRollback(), the same code as POST /api/v1/rollback, event deployment.auto_rollback
+            A->>A: desired state equals the revision still running, nothing restarts
+            A->>Ctrl: POST /agent/v1/status (rollback revision serving)
+            Ctrl->>Ctrl: rollback deployment succeeded
+        end
+    end
+```
+
+Rules the agent follows (`src/reconciler.ts`):
+
+- A revision is identified by a hash of the rendered service (image id for built images, environment, `extra_hosts`, ports shape, volumes) and the versions of the secrets it references. The same hash means the revision is already running and nothing is restarted; a secret change rolls the service instead of force-recreating it.
+- Each revision is a container `tug-<project>-<service>-<hash8>` on the project network, with the single published port remapped to a free port from `TUG_ROLLOUT_PORTS` (default 20000-20999) and the bind address kept. A running container that predates the rollout state is adopted as the active revision.
+- State lives in `<state dir>/projects/<project>/rollout.json`: the active revision, a candidate (written before `up`, so a restarted agent resumes the same project and port instead of starting a second one), revisions waiting to drain with their deadline, and the last rejected revision per service so it is not retried on every poll. Drain deadlines are checked on every poll, so a drain never blocks the poll loop.
+- A failed health check, an exhausted port range or a failed start removes the candidate and leaves the active revision untouched. The node reports `health: failed` and the rejection reason, while the service state stays `serving` with the old port so control keeps routing to it. If a recreated service of the same project fails, rolling services keep being reported serving the same way.
+- Control resolves each endpoint's `port` from the service state, so the ingress follows takeovers without any ingress-specific agent protocol. Convergence is unchanged: `failed` still comes from a node reporting failed health for the current revision.
+- Automatic rollback runs on control right after the status update that moved a deployment from `rolling_out` to `failed`. `scheduleAutoRollback` works in the background so a busy writer queue never blocks status reports, skips rollback deployments (their versions start with the reserved `rollback-`), requires the failed deployment to still be current, and takes the writer queue and project lock like a manual rollback. On a rolling node the previous revision is still running, so applying the rollback restarts nothing; other nodes recreate as usual.
+
+## Networking
+
+`GET /agent/v1/work` returns `serviceEndpoints` by default (serving endpoints per service for injecting `extra_hosts`), `ports` bindings based on each node's `address` and, for nodes marked as ingress nodes, an `ingress` block with the rendered Caddyfile. Set `TUG_SERVICE_NAMES=false` on control to disable service endpoints. Agents report ingress state in the status report. Cross-node clients and Caddy dial the node's sole inter-node `address` over the operator's network; there is no Tug-owned overlay or workload-traffic encryption. See [Networking](networking.md) for the model and sequence diagram.
+
+## Secrets
+
+Secrets are encrypted at rest with AES-256-GCM using the control master key, and versioned per project. An agent only receives the secrets referenced by the `environment` of services placed on its node, substituted into the container environment by the agent (no file on disk). When only a referenced secret's version changes, the agent force-recreates just the affected services.
+
+## Runtime paths
+
+Agents write generated state under `TUG_STATE_DIR` (default `/var/lib/tug`):
+
+```text
+/var/lib/tug/agent.json                                # the node identity document: private key, certificate and pinned CA (0600)
+/var/lib/tug/agent-pending.json                        # a new key waiting for its certificate during enrollment or renewal
+/var/lib/tug/agent.db                                  # last applied version/revision and secret versions per project
+/var/lib/tug/managed-images.json                       # only Tug-owned image references eligible for conservative pruning
+/var/lib/tug/image-identities.json                     # canonical artifact identities paired with daemon-local tag IDs
+/var/lib/tug/ingress/                                  # ingress nodes only: applied Caddy config and its hash
+/var/lib/tug/projects/<project>/rollout.json               # rolling services: active/candidate/draining revisions and the last rejected one
+```
+
+The state directory must be a persistent volume (`-v tug-agent:/var/lib/tug`; `node add` mounts the host directory), so `agent.json` survives container recreation and the agent does not enroll again after an image update or reboot.

package/docs/backups.md

@@ -0,0 +1,160 @@
+# Backups and recovery
+
+Tug schedules, verifies and restores backups itself. You mark a service or a volume with `x-tug.backup`, give control one S3-compatible destination, and agents run [restic](https://restic.net) on the node that holds the data. There is no backup container to build or keep alive.
+
+## Plan for recovery
+
+Workload snapshots contain application data; control-plane archives contain Tug state and keys. Store both away from the VPS, and keep their credentials and passwords somewhere you can reach if the server is lost.
+
+| What | How to protect it | What to keep for recovery |
+| --- | --- | --- |
+| Database | Back up a logical dump from the running service with `x-tug.backup.command`; copying a live database volume is generally not consistent. | Repository password, storage credentials, a suitable restore command and application configuration. |
+| Files in a named volume | Mark the volume with `x-tug.backup`; live files are crash-consistent, not necessarily application-consistent. | Repository password and storage credentials; test a scratch restore. |
+| External object storage | Configure separate protection with the storage provider. | Provider credentials and a way to restore objects to a compatible point in time with the database. |
+| Control state, secrets, private CA and master key | Run `tug control backup` and copy the archive off-host encrypted; it does not include workload data. | The archive and an original operator credential; see [control recovery](control-plane.md#recover-control-on-another-host). |
+| Compose file and source | Keep them in version control. | Access to the repository and any external dependencies. |
+
+Tug backs up each target separately. If your application needs a consistent point-in-time copy, plan how to stop writes and coordinate external storage. [The WordPress example](../examples/wordpress/README.md) illustrates a database and files on one node.
+
+## Set up your first backup
+
+Choose S3-compatible storage reachable from the workload node and credentials allowed to use the bucket. Configure one destination for control, then mark each target in your Compose file as shown in the [backup reference](compose.md#backups). A top-level schedule only supplies defaults; it does not enable any targets. For a complete database and volume configuration, use the [WordPress example](../examples/wordpress/README.md).
+
+```sh
+# Supply the values through your shell or secret manager; do not commit them.
+export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
+tug backup destination set s3:https://s3.example.com/bucket/tug-backups --env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY
+tug deploy
+tug backup run
+tug backup list --check
+tug backup password
+```
+
+Replace the endpoint and bucket with your own. `backup run` waits for the configured targets and reports failure; inspect `backup list --check` to confirm each target succeeded. Store the printed repository password and storage credentials securely off-host. The password is derived from control's master key; losing both the password and master key makes the repository unreadable. `tug backup destination show` shows the destination without credential values. Credentials are encrypted on control and released to a node running a backup. Each project has its own restic repository under `<destination>/<project>`; commands use the [resolved project](cli.md#arguments-options-and-configuration).
+
+Also [back up control](#control-plane-backup), copy the archive off-host, and practice a [restore drill](#restore-drill). No backup notification is pushed automatically; monitor `tug backup list --check`.
+
+## Backup destination
+
+The control plane uses one destination. Running `tug backup destination set` replaces it. Setting or removing a destination updates its configuration and stored credentials atomically; a database failure leaves the previous configuration and credentials intact. Each project uses its own repository. The complete command and credential options are in the [CLI reference](cli.md#backups).
+
+## Scheduled backups
+
+Mark a service or a volume with `x-tug.backup` ([Compose extensions](compose.md#backups)) and Tug backs it up on its node, on the schedule you give, to the project's repository. Control runs a scheduler that ticks every 30 seconds (`TUG_BACKUP_TICK_MS`); at each slot it queues a `backup.run` command with a deadline. The agent runs it as a background job, so reconciliation and health checks are never held up by a long backup, and one node runs one backup at a time unless `TUG_BACKUP_CONCURRENCY` says otherwise.
+
+- **Services** run your `command` through an exec in the running container and stream its output into `restic backup --stdin`, so no staging disk is needed. The command sees the container's own environment, including its secrets, so passwords are read from the environment (`$MARIADB_ROOT_PASSWORD`) and never appear in Tug's configuration, payloads or events. A command that fails, writes nothing or lacks the `verify` text is never kept: the partial snapshot is forgotten and the run fails. The service must be running on the node.
+- **Volumes** are mounted read-only into the restic container and archived, with your `exclude` patterns. A volume that does not exist on the node fails the run instead of backing up an empty one. Live volumes are crash-consistent, which suits uploads and static files.
+- **Retention** runs after each successful backup (`restic forget --prune` per node and target). Snapshots tagged `trigger:pre-restore` are kept regardless.
+
+- **Images** for restic come from `TUG_RESTIC_IMAGE` on control (default `restic/restic:0.17.3`); a digest reference is recommended. Nodes pull it outbound like any published image.
+- **Restic runs in a helper container** with a memory limit, labelled `tug.job`, removed when it finishes. An agent that restarts mid-job removes leftovers and reports the job failed.
+
+Before mounting a managed volume, backup and restore check its project and volume ownership labels; a scratch restore checks its scratch volume's project and restore labels. A conflicting or unlabelled volume is refused instead of being read or overwritten.
+
+## Recipes
+
+Tug has no database support of its own; each recipe is a command for the service container. In a Compose file use `$$` for variables that the container shell must expand: a single `$` is interpolated on the operator machine before deployment. Read credentials from the container environment, write the data to standard output, and give `verify` the line the tool prints on success.
+
+```yaml
+# MariaDB (use mysqldump and MYSQL_ROOT_PASSWORD for MySQL)
+backup:
+  command: mariadb-dump -u root -p"$$MARIADB_ROOT_PASSWORD" --single-transaction --quick --routines --events --triggers --add-drop-database --databases wordpress
+  filename: wordpress.sql
+  verify: "-- Dump completed"
+
+# PostgreSQL
+backup:
+  command: pg_dump -U "$$POSTGRES_USER" --clean --if-exists "$$POSTGRES_DB"
+  filename: app.sql
+  verify: "PostgreSQL database dump complete"
+
+# SQLite file in a volume mounted into the service
+backup:
+  command: sqlite3 /data/app.db .dump
+  filename: app.sql
+  verify: COMMIT
+
+# A directory as a tar stream
+backup:
+  command: tar -C /var/www/html -cf - wp-content/themes
+  filename: themes.tar
+```
+
+`mariadb-dump` and `mysqldump` do not dump users and grants; they come from the image's init environment. A service has one backup command, so to include PostgreSQL roles chain them: `sh -c 'pg_dumpall --globals-only; pg_dump ...'`. Restore with the same tool: `restic dump <snapshot> wordpress.sql | mariadb ...`.
+
+### Events and what to watch
+
+`tug events` shows `backup.completed`, `backup.failed`, `backup.missed`, `backup.stale` and `backup.no_destination`. A backup is `missed` when its node did not pick it up before its deadline (`node_offline`) or when control was not running at the slot: a week of downtime produces one missed event and one catch-up run per target, not a burst. A failed scheduled run is retried once after ten minutes. A target with no success for two periods raises `backup.stale` once a day, which catches a destination key that has quietly stopped working. While control is down no backup starts, which is the price of one scheduler; running jobs finish and report when control returns.
+
+`tug backup run` takes a backup now and `tug backup list` shows the catalog ([CLI](cli.md#backups)).
+
+### Restore
+
+In a terminal, `tug backup run --select` chooses a subset of enabled targets. `tug restore run` offers a target when omitted; add `--select-snapshot` to choose a successful catalog backup or `--select-writers` to override writer discovery. `tug restore resume` and `tug restore release` offer interrupted restores when their ID is omitted. Existing preview and confirmation requirements still apply. See [interactive selection](cli.md#interactive-selection).
+
+```bash
+tug restore run db                       # prints the plan, changes nothing
+tug restore run db --confirm             # latest backup of the db service
+tug restore run uploads --scratch --confirm   # files into a new volume, touching nothing live
+```
+
+A restore plan and its queued restore refresh backup placement from the latest successful deployment before choosing a node, so moving a project does not leave a restore aimed at the previous node until the next scheduler tick. Check the printed target before confirming.
+
+Restores are as generic as backups. A service is restored by running its `x-tug.backup.restore` command in the running container with the stored file on standard input (`restic dump` piped in), so a MariaDB service uses `restore: sh -c 'mariadb -uroot -p"$$MARIADB_ROOT_PASSWORD"'`. A volume is emptied and filled from the snapshot, or restored into a new volume with `--scratch`. A service without a `restore` command is refused rather than guessed at.
+
+`tug restore` returns when the data is back and the services it stopped report serving again, or after five minutes with a warning if they do not, so a script can probe the site as soon as it returns. An in-place restore is deliberately careful:
+
+1. It **locks the project**, so a deploy, rollback or secret change gets a clear refusal while it runs.
+2. It **stops the writers**: the `writers` you named, or discovered from `depends_on`, or host names in environment variables such as `WORDPRESS_DB_HOST=db` or `mysql://u@db:3306/app` (for a service), or volume mounts (for a volume), including everything that depends on them. If none are found it refuses unless you pass `--writers` or `--no-writers`, because an app that reaches the database without `depends_on` is the hole in discovery and the plan makes you look.
+3. It takes a **safety backup** of the current data, tagged `trigger:pre-restore` and exempt from retention (skip with `--no-safety-backup`). If that fails, nothing has changed: the writers start again and the lock is dropped.
+4. It restores, then starts the writers and unlocks.
+
+While a writer is held, a service that health-checks through it (an nginx check that requests a PHP page, for example) is unhealthy, so its node reports a failed apply for the duration. Tug still sees the writer as held and continues, but the deployment looks failed until the hold ends. Give such services a health check that does not depend on the writer (a static `/healthz`) to keep them serving through a restore. If a writer never reports held within five minutes, the restore fails and the error names the node, the writer's state and the apply error that node reported.
+
+Only a snapshot this target produced can be restored into it (its `project:` and `target:` tags are checked). If the restore itself fails, the writers stay stopped and the project locked, because a half-restored database serving traffic is worse than an outage. The restore is `interrupted`; `tug restore resume <id>` tries again and `tug restore release <id>` starts the services anyway. Scheduled backups of a target are skipped while it is being restored. A restore is recorded with `restore.started`, `restore.completed`, `restore.failed` and `restore.released` events.
+
+A scratch restore of a volume is also the restore drill: it proves a backup can be read without touching anything live. Databases are the same drill by hand: `restic dump <snapshot> wordpress.sql` into a throwaway container.
+
+## Recover a site on a replacement node
+
+This is a manual procedure, not automatic failover. A WordPress and MariaDB project was recovered this way with the published 0.9.0 release in the [real-host drill](real-host-evaluation.md#recovery-drill-with-the-published-090-release-2-november-2026). The original host was simulated as unavailable while control and the backup destination stayed available. Restore control first if it is also lost.
+
+1. Fence the lost node or otherwise ensure its old workloads cannot write to shared services. Enroll the replacement node and verify it is online. Keep the recovery site private or otherwise block public traffic while its new database and volumes are empty; do not expose a fresh WordPress installer.
+2. Change the project’s service and volume placement to the replacement node and deploy the same project using the retained images, configuration and secrets. The replacement uses fresh local volumes; deploying alone does not restore their contents. Keep the old backup repository and its password available.
+3. Check `tug backup list` and run `tug restore run db` and `tug restore run wp-content` without `--confirm`. Verify each plan names the replacement node and the intended pre-loss snapshot. If the latest snapshot is not the desired one, use `--from` with the intended backup run or snapshot ID. Older releases may briefly show the former node until the backup scheduler catches up; never confirm a plan aimed at the wrong host.
+4. Restore the database and uploads with the printed commands and `--confirm`. Tug normally takes a safety backup before each in-place restore and stops the writers; wait for each restore to finish and investigate any warning before continuing. Verify known database content and uploaded files, probe the application, then re-enable traffic and update DNS or routing as needed.
+
+This does not restore data that was never backed up, external services, DNS, or changes after the selected snapshots. A live workload can write new data during recovery unless traffic and other writers are fenced; control alone cannot fence an unreachable or compromised host.
+
+## Control plane backup
+
+`tug control backup` covers the control plane only, not workload data. Take it with an `admin` operator credential and copy the archive off the control-plane host. The archive holds the master key, every secret and the private CA key (which can issue credentials for the whole installation), so treat it like a password vault and store it encrypted. Create a dedicated credential for the job: `tug credential create backup --role admin` and store it at `/etc/tug/backup.credential.json` (mode `0600`). Example `/etc/cron.d/tug-control-backup` on the control-plane host (the runtime image runs the CLI; adjust to how you invoke `tug` there):
+
+```cron
+15 2 * * * root TUG_CONTROL_URL=https://127.0.0.1:8443 TUG_CREDENTIAL_FILE=/etc/tug/backup.credential.json tug control backup > /var/log/tug-control-backup.json 2>&1 && restic -r s3:https://s3.eu-central-1.amazonaws.com/acme-backups/control backup /var/lib/tug/control-backups --password-file /etc/tug/restic.pass
+```
+
+Put the S3 credentials in the crontab environment or a `restic.env` file with mode `0600`. Archives accumulate in `TUG_CONTROL_BACKUP_DIR`; prune old files with `find ... -mtime +14 -delete`. The cron line has not been run on a real control-plane host; verify that `tug` is on the cron `PATH` and that the data directory matches your install (`TUG_CONTROL_BACKUP_DIR`, see [Configuration](configuration.md)). `TUG_CONTROL_URL` is a connect target only; the certificate is still verified against the public URL in the credential. Restore with `tug control restore <path> --confirm` and restart control ([Control plane](control-plane.md#backup-and-restore)).
+
+## Restore drill
+
+A backup you have not restored is a hope. Start with an isolated restore to verify the backup without changing live data. Time the exercise and record what you needed.
+
+1. **Check the repository and the catalog.** `tug backup list` shows every target with its last success; the newest run should be recent and `succeeded`. `tug backup password` prints the repository and its password, so you can also inspect it directly with `restic snapshots` from any machine that can reach the bucket.
+2. **Restore files into a scratch volume.** `tug restore run wp-content --scratch --confirm` reads the latest snapshot into a new volume next to the live one and touches nothing live. List it with `docker run --rm -v <volume>:/data alpine ls /data` on the node, then remove the volume with `docker volume rm <volume>`.
+3. **Restore the database into an isolated throwaway database.** Use the original project repository and its escrowed password to read the snapshot with restic into a separate database container. Verify known data without connecting the test container to production services, mail or scheduled jobs. This manual end-to-end drill has not yet been validated on a fresh host.
+4. **Write down how long it took.** That is your real recovery time.
+
+Tug does not restore a snapshot into a different project: repositories and snapshot tags are project-specific. For an in-place restore, inspect `tug restore run db` first; confirmation stops writers, takes a safety backup and changes the live database. See [Restore](#restore).
+
+## Known limits
+
+- One failed night is visible as a `backup.failed` or `backup.stale` event, not a push notification, until [webhooks](roadmap.md#m8--central-api-advantages-partly-done) exist. Watch `tug events`, or point a monitor at `tug backup list`.
+- While control is down no backup starts. Running jobs finish and report when control returns, and a long gap yields one missed event and one catch-up run per target.
+- Live volumes are crash-consistent, not application-consistent. Use a command backup for anything with its own consistency (databases).
+- A volume is backed up from the one node that holds it. A volume you mount from a file server with `driver_opts` is read through that node's mount; back up the file server's own disk where you can.
+- Repository maintenance (`restic check`, a read-verify of the data) and SFTP destinations are not built yet.
+- A scratch restore leaves its volume (`tug_r<project-length>_<project>_<id>`) behind, and the safety snapshot of every in-place restore is kept for ever. There is no command to list or remove either yet: remove scratch volumes with `docker volume rm` on the node, and safety snapshots with `restic forget` using the repository from `tug backup password`.
+- The restic helper container defaults to a 512 MB memory limit, which is half of a 1 GB host; lower it with `memory` under `x-tug.backup`. Restic needs more memory for larger repositories, so test the limit against your data.
+- Failures are not pushed anywhere until webhooks exist. Run `tug backup list --check` from cron or a monitor: it exits nonzero when a target is stale, failed or missed.
+- Cancelling `tug exec` does not stop the remote process.