@wpelevator/tug 0.11.0 → 0.13.0

package/docs/roadmap.md

@@ -0,0 +1,234 @@
+# Roadmap
+
+This is the canonical statement of what Tug is for, the shape of deployment it targets, and the milestones that get it there. A milestone that is being worked on gets a `plan-{task-name}.md` working file (see [AGENTS.md](../AGENTS.md)); when it lands, its lasting decisions move into the docs and its item here is checked off.
+
+## Goal
+
+Run a few Docker Compose applications across a handful of ordinary Linux VPSes from one small self-hosted control plane, without Kubernetes, a hosted PaaS or a cloud provider's API.
+
+The operator this is for runs one to a few dozen sites on two to ten VPSes, deploys from a laptop or CI, and needs the day-2 jobs covered as well as the deploy: backups and restores, one-off commands such as `wp` CLI and database upgrades, logs, safe rollouts and rollbacks, and an audit of who changed what. Feature parity with other tools is not the goal; [uncloud](https://github.com/psviderski/uncloud) and Kamal are useful baselines for what operators expect (see [Positioning](#positioning)).
+
+## Principles
+
+1. **Compose stays the app format.** `docker compose up` works locally; `x-tug` adds deployment metadata, and Compose features that cannot work across nodes are rejected rather than half-supported ([Compose extensions](compose.md#not-supported)). Compose is the input, not the node runtime: nodes need only the Docker Engine ([M14](#m14--engine-api-agent)).
+2. **Production runs immutable images.** Application code, including WordPress core, themes and plugins, ships inside an image built on control or on the operator's machine. Every image a deployment uses is pinned to a digest when the deployment is created ([M13](#m13--pinned-images-and-one-delivery-path-done)). Nothing is built or synced into a running node during reconciliation, so an image rollback restores the earlier image and configuration when retained artifacts remain available. It does not reverse database changes, uploads or external side effects.
+3. **Agents reconcile desired state and only dial out.** Normal deploys need no SSH and no inbound port on nodes, so nodes behind NAT or a deny-all firewall work. Running containers keep serving while control is down.
+4. **One writer.** Desired state, locks, secrets and history live in one control plane. This keeps ordering, rollback and audit simple; availability comes from control backups and, if needed later, a warm standby.
+5. **Stateful things are pinned and backed up.** Databases and data volumes stay on their placement node unless the operator moves them, are never deleted by a deploy, and every first-class stateful service has a backup and restore path.
+6. **Private by default.** Bind ports to the configured node address or loopback unless the operator explicitly requests a public bind. Reachability, firewalling and workload-traffic encryption depend on the operator's network; a private address alone is not an isolation guarantee.
+7. **Start simple.** No distributed storage, no autoscaling, no bin-packing scheduler, no database failover. Single points of failure are allowed when they are documented. A single node needs nothing beyond Docker, the agent and control: ingress and other subsystems stay optional layers on top of that core, and multi-node workloads use the operator's existing network.
+
+Non-goals: Kubernetes compatibility, provisioning VPSes through provider APIs, autoscaling, distributed block or file storage, database HA, a multi-tenant hosted control plane.
+
+## Reference topology
+
+These are the shapes Tug is tested against and the ones the docs should steer people toward.
+
+**Small: one node per site, or several sites per node.** App and database on the same node, the managed Caddy ingress on that node, data in local named volumes. This is what most WordPress sites need and what was validated first ([Real-host evaluation](real-host-evaluation.md)). Many sites per node is the common case, so clear project boundaries, ingress routing for many hostnames and per-site backups matter more than multi-node features. Project boundaries are not isolation between mutually untrusted workloads.
+
+**Scaled WordPress (a possible future topology, not a validated support promise):**
+
+```text
+Internet -> ingress node (managed Caddy, TLS)
+              | HTTP, over the operator's private network
+              v
+         app1, app2, app3: nginx + php-fpm, image with core/themes/plugins
+              | uploads             | SQL / cache
+              v                     v
+         uploads store         db node (MariaDB), cache (Memcached)
+         (S3-compatible or a file server you run, wp-content/uploads only)
+```
+
+- WordPress code is in the image (Bedrock or Composer style), so every app node runs exactly the deployed version and a rolling rollout or rollback swaps images atomically per node.
+- Only `wp-content/uploads` is shared: preferably on S3-compatible object storage through an offload plugin, otherwise on a file server you run and mount with plain Compose `driver_opts`. PHP code is never served from a network file system, which avoids opcache stat latency, mixed versions during a sync and non-atomic rollbacks.
+- Each app node serves its own static files with nginx next to php-fpm, and the ingress balances HTTP across app nodes. There is no FastCGI across hosts and no front node that needs the shared filesystem.
+- The database, cache, uploads store and ingress are each single instances. Scaling the app tier adds PHP capacity, not availability; the docs say so, and backups are the recovery story.
+
+The earlier specification (a front nginx speaking FastCGI to app nodes over a shared NFS docroot, with WordPress files synced to NFS on deploy) was dropped for the reasons above.
+
+## Positioning
+
+Keep the single-writer control plane rather than decentralizing it like uncloud. Deploy locks, ordered deployments and rollbacks, secret versions and build-then-deploy sequencing are simple and correct with one writer and hard to get right with several. Invest in what a central API makes easy and distributed control makes hard: deploy safety, scheduled backups with a catalog, auditability, CI automation and multi-operator access.
+
+Alternatives considered:
+
+- **uncloud** (decentralized, WireGuard mesh, container-routable networking, unregistry image push). Closest in spirit. Tug differs by having one control plane, agents that only dial out and no Tug-owned workload overlay: multi-node traffic uses the operator's existing private network.
+- **Docker Swarm** overlaps Tug's placement, reconciliation, rolling updates, secrets and node mTLS, and adds continuous task rescheduling, service DNS/VIPs, overlay networking and replicated managers. A single manager works; three tolerate one manager failure. Manager and overlay connectivity must be available on the private network, which can be a suitable VPN behind NAT. Tug instead integrates source builds, registry-free application image delivery, managed HTTPS, actor-attributed deployment history, scoped operator access and workload backup/restore. Docker has events and supports authorization plugins, so those Tug advantages concern the built-in application workflow. Not adopted; it remains the yardstick for how much Tug should build itself. See the source-linked [Docker Swarm comparison](swarm-comparison.md) for implemented differences, failure behavior and workload fit.
+- **Kamal** pushes over SSH with no control plane. Simpler, but no reconciliation, no central audit or scoped CI credentials, and no state for scheduled jobs such as backups.
+
+Re-check uncloud's releases at the start of each milestone; baseline rechecked on 2026-10-31: [v0.21.0](https://github.com/psviderski/uncloud/releases/tag/v0.21.0), adding shared Caddy certificate storage, systemd socket activation and runtime templates to the existing mesh, DNS and deployment features.
+
+## Milestones
+
+Numbers are identifiers, not order; the order is in [Sequencing](#sequencing).
+
+### M1 — TypeScript adoption (done)
+
+Typed `src/protocol.ts` control↔agent contract and injectable agent runners. Source and tests run as `.ts` through Node type stripping; only the Docker image is compiled to `dist/`. See [Development](development.md) and [Architecture](architecture.md).
+
+### M2 — Cross-node networking, service names, ingress/TLS (done)
+
+Health-aware cross-node service names (hosts-file `extra_hosts` entries, not DNS) are on by default with a `TUG_SERVICE_NAMES=false` opt-out, and the managed Caddy ingress provides optional `x-tug.ingress` routes and automatic TLS. Names and upstreams resolve to each node's `address` on the operator's private network; service names need published host ports. Tug owns no overlay or workload-traffic encryption. See [Networking](networking.md). Stable rolling-service name/port and avoiding client recreation on endpoint changes remain in [Later](#later--only-if-needed).
+
+### M3 — Zero-downtime rolling deploys + automatic rollback (done)
+
+Rolling rollouts for ingress-fronted services (start the new revision on its own host port, health-check it, switch the Caddy upstream, drain the old one), `x-tug.rollout` options and opt-in `x-tug.rollback: auto`. See [Architecture](architecture.md#rolling-rollouts-and-automatic-rollback) and [Compose extensions](compose.md#rolling-rollouts). The numbers it is held to are in [Testing](testing.md#metrics). Rolling services have no stable cross-node name and port; see [Later](#later--only-if-needed).
+
+### M4 — Images without a registry, bare-host install (done)
+
+Control stores built images as saved image files and agents download them over their own authenticated connection, so no registry or daemon change is needed. `deploy --build-local` uploads an image built on the operator's machine. `--install-docker` bootstraps Docker on apt and dnf hosts and in `node enroll` cloud-init. See [Architecture](architecture.md#image-artifacts). Each deploy transfers the whole image; layer reuse is in [Later](#later--only-if-needed).
+
+### M5 — Streaming channel between agent and control (mostly done)
+
+Agent-initiated WebSocket with long-poll fallback that wakes agents when work is queued and carries live sessions: non-TTY `tug exec` with stdin, `logs -f`, `ps` and `inspect` ([Architecture](architecture.md#agent-wake-channel), [Observability](observability.md#live-commands)). Remaining, at low priority: dropping the long-poll fallback (the WebSocket plus the periodic poll cover a lost socket), agent and daemon log streams, and terminating the remote process when an exec session is cancelled. PTY allocation and resize for `exec -t` shipped with [M14](#m14--engine-api-agent) through the Engine API.
+
+### M6 — Placement and scheduling (done)
+
+Node labels, `placement.labels`, `replicas: N` spread across matching nodes, `global: true` and `tug scale`. Scheduling stays central and simple: spread by count, skip stale nodes, no bin-packing. See [Compose extensions](compose.md).
+
+### M7 — Shared volumes (removed)
+
+An `x-tug.type: nfs` volume type was built and verified on real hosts, then removed: it mounted a share but could not run the NFS server, so users still prepared the host, and installing and exporting from Tug would make it a host configuration tool. Shared storage is the operator's: S3-compatible object storage, or a file server mounted with plain Compose `driver: local` and `driver_opts` ([Volumes](compose.md#volumes)).
+
+### M8 — Central API advantages (partly done)
+
+Shipped: scoped operator credentials (`read`, `deploy`, `admin`, per project), a GitHub Actions deploy example and credential names in the audit trail ([Access control](access-control.md)). Remaining: outbound webhooks for deployment state changes, failed health checks and backup results (shared with M9 alerting). The read-only web dashboard is dropped to [Later](#later--only-if-needed): `tug status`, `/metrics` and webhooks cover the need.
+
+### M9 — Observability
+
+Make behavior visible without adding required services: event hygiene (done), structured logs, a Prometheus `/metrics` endpoint on control that also carries node and service stats reported by agents (Prometheus cannot scrape nodes behind NAT, which is why agents report through control), a readiness endpoint, sample alert rules and webhooks. Design and status are in [Observability](observability.md). Keep agent-reported stats small; point operators who want depth at cAdvisor or node_exporter deployed as ordinary services.
+
+### M10 — Private-by-default port exposure (done)
+
+A published port such as `3306:3306` listened on every interface of the node, and Docker's iptables rules bypass host firewalls such as ufw, so a database published for cross-node service names was reachable from the internet. Now published ports without a host IP are bound to `127.0.0.1` when only the local ingress dials them, and otherwise to the node's `address` with a `ports.public` warning event when it is public; an explicit host IP (including `0.0.0.0`) is left as written, and `tug doctor --nodes` reports containers listening on a public address. See [Networking](networking.md#published-ports-private-by-default). This was a prerequisite for recommending the scaled topology on an operator-provided private network.
+
+### M11 — Backups, restores and run-once jobs
+
+Working plan: `plan-backups.md`.
+
+The stateful half of principle 5, and something a central control plane does better than distributed schedulers: one schedule, one catalog, retention and restore.
+
+- **Done: generic backups.** `x-tug.backup` marks a service or a volume. Tug has no database support of its own: a service backup is a `command` run in the running container whose output is stored (database recipes are documentation), a volume backup archives its files. Control schedules them as agent commands on the placement node; the agent streams into restic in a helper container, uploading to a destination configured on control (S3-compatible, encrypted before leaving the node), one repository per project. Retention, a catalog, `backup run` and `backup list`, and `backup.*` events for completed, failed, missed and stale runs. See [Backups](backups.md).
+- **Done: restore.** `tug restore` locks the project, stops the services that write to the target, takes a safety backup, restores through the service's own `restore` command or the volume's files, and starts them again; an interrupted restore keeps the writers stopped until resumed or released. A scratch restore of a volume touches nothing live.
+- **Done: `tug run`.** `tug run SERVICE -- cmd` runs a one-off container with the service's image, environment, secrets, network and mounts, returns the exit code and is audited.
+- Not done: `x-tug.release` (a command that runs once per deployment before or after rollout, recorded on the deployment, a failure stopping the rollout), SFTP and postgres-free extras such as a weekly `restic check`/prune, a restore drill, `destination test`, and failed or missed backups as webhooks (M8).
+- Moving a stateful service to another node (`tug move SERVICE --to NODE`) as backup, stop, restore and re-place, since the pieces are the same.
+
+### M12 — Container networking over the mesh (cancelled)
+
+Cancelled when Tug stopped providing a workload overlay. Tug will not build the proposed per-node container subnets, managed resolver or routable container network. The two operator-facing limitations this milestone was meant to solve—stable cross-node name and port for rolling services, and client recreation when `extra_hosts` endpoints change—are tracked independently in [Later](#later--only-if-needed) and do not assume a Tug-owned overlay.
+
+### M13 — Pinned images and one delivery path (done)
+
+Pin image identity for reliable image rollback, and leave one way for Tug-built images to reach a node. Mutable data and external side effects are not rolled back.
+
+- **Digest pinning.** Control resolves each distinct published image tag once per deployment and stores `image@sha256:…` in desired state and canonical Compose before publication. Explicit pins need no lookup. Multi-platform indexes stay intact. Resolution failure leaves current desired state unchanged; rollback and scaling reuse stored pins. Nodes pull by digest from upstream. Private HTTPS registries use the encrypted project secret `TUG_REGISTRY_AUTH` and Engine API authentication, released only to assigned services. See [Control plane](control-plane.md#published-images-and-private-registries).
+- **One built-image delivery path.** Remote and local builds always travel as retained saved archives from control. Built-image registry configuration, `--no-build-upload`, registry pull tokens and their rotate/revoke commands, registry auth hooks and node `docker login` are removed. Each node has one agent credential. CI-built images can be pushed independently and deployed through `image:`.
+
+Each deploy still transfers whole Tug-built images; a control-hosted image store, mirroring of published images and layer reuse are in [Later](#later--only-if-needed) until image size hurts.
+
+### M14 — Engine API agent
+
+The agent stops running the `docker compose` CLI and reconciles containers through the Docker Engine API. The agent already works around Compose: rolling revisions are separate Compose projects with hand-made networks and volumes carrying copied Compose labels, `depends_on` is stripped across them, and secret changes need forced recreates.
+
+- The CLI normalizes the Compose file with `docker compose config` (Compose does interpolation, `extends`, profiles and `env_file`; secret variables are left as sentinels for the agent to substitute) and sends the canonical definition (read from Compose's YAML rendering, since its JSON rendering drops `x-tug`); control validates an explicit allowlist of supported keys, including resource limits; agents never parse Compose YAML. Nodes need only the Docker Engine, not the Compose plugin. The control-plane host keeps using Compose for its own install unless that moves to `docker run`.
+- One reconciler for every service: containers carry `tug.*` labels with a hash of their resolved configuration, including the image reference and the versions of the secrets they use, and are recreated when it changes. `recreate` and `rolling` become two strategies of the same loop. Tug names its own networks, volumes and containers; service names are network aliases.
+- Secrets are substituted into the container environment by the agent; no `.env` file on disk. Health checks stay agent-run, because images often lack the tools a Docker `HEALTHCHECK` would need.
+- `exec -t` with a real PTY and resize, and `logs`, `ps` and `inspect` from the API instead of CLI output.
+
+### M15 — Flexible node roles
+
+- **Control on a runtime node.** Installing control and a node on the same host is supported, so a small setup needs no dedicated control VPS. Control's API is then served through the node's managed Caddy, since both need ports 80 and 443. Keep a direct control port bound to loopback or a private node address as break-glass access, so a broken ingress or agent never locks out the fix path.
+- **Builder nodes.** Builds run on a node labelled `builder`: control queues a build command, the agent downloads the source context from control, builds it and uploads the saved image to control the way `deploy --build-local` does. Without a builder node, control builds, as today. This keeps untrusted Dockerfile builds and their CPU and memory use off the host that holds the secrets database.
+
+## Later / only if needed
+
+- Warm-standby control plane (WAL shipping or periodic backup plus promote), see [Positioning](#positioning).
+- A stable cross-node name and port for rolling services, so clients need not switch to an ingress hostname or discover the serving revision's port. Revisit only with a concrete multi-node need; a solution must work over the operator's network and must not assume a Tug-owned overlay.
+- Avoid recreating client containers when cross-node `extra_hosts` endpoints change, while preserving health-aware names and zero-downtime rollouts. This is an endpoint-update problem, not a reason to recreate a Tug-owned overlay.
+- A host-level agent helper or standalone agent binary, for work a container should not do, such as host mounts outside Docker. Shares the build with the standalone CLI binary ([Standalone binary](standalone-binary.md)).
+- A control-hosted OCI image store, mirroring of published images and an agent-local registry endpoint for layer reuse, once whole-image transfers or upstream rate limits hurt. Mirroring is opt-in; it makes control a bandwidth and disk bottleneck and a dependency for every pull.
+- Standalone Bun-compiled `tug` binaries and a Homebrew tap, so operators need no Node.js.
+- A minimal read-only web dashboard over `/api/v1/status` and `/api/v1/events`.
+- Load-balanced service names and multiple routes per service, wildcard certificates.
+- Managed DNS convenience domain for ingress, similar to uncloud's `*.uncloud.run`.
+
+## Product decisions and reasoning
+
+Tug serves a trusted operator or small trusted team running Compose applications on one or a few ordinary VPSes. The primary topology is an application and its state on one node, including several independent applications on one node; additional nodes first add capacity and placement for independent applications. A scaled single application is a later, demand-driven use case. Judge work by whether it makes installing, operating and recovering those applications easier and safer, not by feature parity or the number of orchestration milestones completed.
+
+The control plane earns its cost through understandable state, reliable deployments, backup visibility and recovery. Prefer a proven, repeatable recovery journey over additional scheduling, networking or infrastructure features. Restic is the backup repository mechanism; Tug's responsibility is to make backup status, credentials, off-host copies and actual restore outcomes understandable. Do not set recovery time or data-loss targets until a representative exercise measures them.
+
+Deployment authors are trusted infrastructure operators. Project-scoped credentials are operational least-privilege controls that reduce accidental access and limit a compromised credential's API authority; they do not isolate mutually untrusted code, shared kernels, application networks, builds or workload secrets. An operator who can deploy or execute code must be trusted with the workload's secrets and the nodes they can select. Use separate hosts or control planes for mutually untrusted authors. Continue fixing concrete authorization, confidentiality and data-safety defects, but do not add a complex tenant-isolation policy without an explicit user need and a coherent end-to-end threat model. An earlier project node/hostname grants and host-feature restrictions experiment remains separate from the recovery-first branch; evaluate it against this contract before adopting it. It is not a product decision to support hostile tenants.
+
+Every subsystem Tug owns carries maintenance and failure costs. Keep the outbound agent while its NAT and unattended reconciliation benefits justify it, but a bounded SSH-controller experiment is legitimate if it preserves the operational guarantees users need. Compare control-hosted image archives with an existing registry on deployment, recovery, storage, security and operational cost before expanding either path. Keep the Compose subset explicit and predictable, use the operator's private network for cross-node workloads, and prefer simple placement over a general scheduler. Remove or narrow an existing feature if its cost exceeds its demonstrated value.
+
+Claims must match verified behavior. Image rollback does not undo data, schema or external side effects; private port binding depends on the configured address and network; scoped credentials are not tenant isolation. Deployment success needs a documented definition that distinguishes node health from ingress applying the intended revision and, if relevant, an external probe. Each claim needs tests at the cheapest useful layer and real-host evidence for the end-to-end operating journey.
+
+## Sequencing
+
+1. Review the current security work against the trusted-operator contract. Preserve independently useful data-safety and confidentiality fixes, assess grants and restrictions on their own merits, and do not claim a tenant-isolation boundary.
+2. Prove recovery on disposable hosts using published artifacts: install WordPress, deploy a meaningful change, back up database, uploads, control state, credentials and image artifacts; recover a lost workload host and then a lost control plane onto fresh storage, reconnect the fleet and verify the site. Record elapsed time, data-loss window, manual actions and gaps; fix and retest failures. The control and workload backups are separate and both must be recoverable.
+3. Improve the ordinary operator experience: clear human-readable status and errors, backup freshness and actionable alerts, and explicit deployment progress and success semantics including ingress acknowledgement where promised.
+4. Harden the actual production path: bounded work and storage, safe build inputs, credential handling, and repeatable release, install and upgrade checks.
+5. Reconsider flexible node roles, release hooks, scaled WordPress, additional networking, dashboards and stronger isolation only for a demonstrated need. Co-locating control is attractive for small installations but requires an independently tested off-host recovery path and an explicit host trust model.
+
+M10 is done; M12 is cancelled and its residual limitations are deferred. M14 and much of M11 are implemented; the current priority is verifying the complete operating journey rather than repeating those milestones. The milestone descriptions and checklist record existing work and candidates, not commitments that override this sequence.
+
+Guided onboarding (`tug setup`, an inferred node name, and `node add` offering Docker, the inter-node address and ingress in a terminal) answers the first-run friction recorded in [real-host runs](real-host-evaluation.md). It prompts only in a terminal outside CI; scripts pass the same choices as flags. The prompts have unit and CLI tests but have not yet been exercised on real hosts, so the next real-host run should follow the README quick start as written.
+
+Historical validation: [real-host runs](real-host-evaluation.md) on two Ubuntu 24.04 hosts covered SSH install with the Docker bootstrap, the now-removed WireGuard mesh, an agent behind bridge NAT, a publicly trusted Let's Encrypt certificate from the managed Caddy, cross-node service names as then implemented, remote and `--build-local` deploys, rollback from a retained image, a rolling rollout through the public ingress with no failed requests (over 2,200 sampled), a failing version rejected while the old revision kept serving, automatic rollback, and on freshly wiped hosts label placement, `global`, `tug scale`, `ps`/`inspect`/`exec`/`logs -f` and a cloud-init enrolled node behind a deny-all-inbound firewall ([M6, M7 and enrollment](real-host-evaluation.md#m6-m7-and-enrollment-on-fresh-hosts)). The mesh and mesh-based name results are retained as history, not evidence for the current plain-address network path. Not yet validated on real hosts: the current plain-address cross-node service-name, ingress and private-port path, a node behind real NAT, certificate renewal, arm64 nodes and dnf-based hosts.
+
+The test layers, the metrics milestones are held to and the backlog of mocked tests are in [Testing](testing.md).
+
+## Checklist
+
+- [x] M1 TypeScript adoption merged
+- [x] M2 networking, service names, ingress/TLS: default-on cross-node service names over node `address` values, `x-tug.ingress`, managed Caddy with `auto`/`custom`/`off` TLS ([Networking](networking.md))
+- [x] M3 rolling deploys with ingress upstream switching, `x-tug.rollout` (`strategy`, `drain`; `maxSurge` not implemented), opt-in automatic rollback, run on real hosts through the public ingress
+- [x] M4 images stored on control, retention tied to rollback history, agent image pruning, `deploy --build-local`; M13 makes this the only built-image delivery path
+- [x] M4 opt-in Docker install for `node add`/`control setup` (verified on Ubuntu 24.04) and `node enroll --install-docker` cloud-init (verified on a cloud-init host)
+- [ ] M4 dnf bootstrap verified on real Fedora and Rocky/AlmaLinux hosts
+- [x] M5 wake channel, faster convergence, non-TTY `exec` with stdin, `logs -f`, `ps`, `inspect`
+- [ ] M5 agent/daemon log streams
+- [ ] M5 long-poll wake fallback removed; WebSocket plus periodic poll
+- [ ] M5 remote process termination when an exec session is cancelled or disconnected
+- [x] M6 node labels, label placement, replicas spread, `global` mode, `scale` command, verified on real hosts
+- [x] M7 `nfs` volume type, removed: shared storage is the operator's (S3-compatible storage or `driver_opts`)
+- [x] M8 scoped operator credentials, CI deploy example, credential names in the audit trail ([Access control](access-control.md))
+- [ ] M8 outbound webhooks
+- [x] M9 event hygiene ([Architecture](architecture.md#events)) and observability design ([Observability](observability.md))
+- [ ] M9 structured logs (`TUG_LOG_FORMAT`, `TUG_LOG_LEVEL`)
+- [ ] M9 `GET /metrics` on control (Prometheus text, `read` credential), with agent-reported node and service stats
+- [ ] M9 an authenticated `/api/v1/readyz`, sample alert rules, and the metrics documentation test gate
+- [x] M10 published ports bound to the node address or loopback by default; `doctor` reports public listeners
+- [x] M11 spec written
+- [x] M11 `x-tug.backup` (generic: a command in a service, or a volume's files), scheduled backups to S3-compatible storage with restic, catalog, `backup run`, `backup list`, `restore` (verified against real Docker and on two real hosts with an S3-compatible server and a scheduled run: [Real-host evaluation](real-host-evaluation.md#generic-backups-restore-and-tug-run-on-fresh-hosts-1-november-2026); not yet against AWS S3)
+- [x] M11 `tug run` one-off containers
+- [ ] M11 `x-tug.release` run-once hooks
+- [ ] M11 weekly maintenance (`restic check`, prune), restore drill, `backup-destination-test`, SFTP
+- [x] M11 `restore` waits for the restored writers to be serving again (five minutes, then it finishes with a warning)
+- [ ] M11 list and remove scratch-restore volumes, and expire safety snapshots (they accumulate for ever and only `restic forget` removes them)
+- [x] M11 `backup list --check`: exit nonzero when any target is stale, failed or missed, as an interim for cron and monitors until webhooks (M8) exist
+- [x] M11 `memory` under `x-tug.backup` for the restic helper's memory limit
+- [x] A failed health check says when the agent runs outside a container (`doctor` does not check it separately)
+- [x] `TUG_ROLE=agent` works in the runtime image: it no longer sets a default command
+- [ ] A `tug run` option to start without the service's volumes (`--no-volumes`), and a decision on whether it should take the project lock
+- [ ] M11 `tug move` for stateful services
+- [x] M12 cancelled with the Tug-owned overlay; stable rolling-service name/port and endpoint-change recreation moved to [Later](#later--only-if-needed)
+- [x] M13 stopgap: published images pulled only when a deployment revision is new to the node
+- [x] M13 digest pinning for every `image:` at deploy time, used by rollback
+- [x] M13 bring-your-own registry, registry pull tokens and `docker login` removed; one node credential
+- [x] M14 Compose normalized on the operator machine; allowlist validation on control
+- [x] M14 Engine API reconciler with config-hash recreation, `recreate` and `rolling` strategies, no `.env` on disk
+- [x] M14 `exec -t` with PTY and resize; `logs`, `ps`, `inspect` and ingress on the Engine API
+- [x] M14 Compose plugin no longer required on nodes (the runtime image no longer ships it; the control-plane host still uses Compose to run the control container)
+- [x] One mutual-TLS port for operators, CI and nodes, certificates from control's private CA, immutable node IDs, and `tug doctor` certificate expiry and clock skew ([Control plane](control-plane.md#authentication-and-pki), [Nodes](nodes.md))
+- [ ] M15 control on a runtime node behind the managed Caddy, with a break-glass control port
+- [ ] M15 builds on `builder`-labelled nodes, uploaded to control as saved images
+- [ ] Scaled WordPress reference example (code in image, shared uploads) validated on real hosts
+- [ ] Mocked tests from the [Testing](testing.md#backlog-of-mocked-tests-proposed) backlog: revisions started per deployment and old-schema database behavior
+- [x] M13 uncloud release baseline rechecked and updated in [Positioning](#positioning); repeat at the start of each milestone
+- [x] Guided onboarding: grouped `tug <subject> <verb>` commands, `tug setup`, name inference and `node add` prompts that never run without a terminal
+- [ ] Follow the README quick start, prompts included, on fresh hosts and record the result in [real-host evaluation](real-host-evaluation.md)
+- [ ] Each milestone: tests added, README.md and docs updated, its own plan file folded into docs and deleted when done

package/docs/security.md

@@ -0,0 +1,198 @@
+# Security audit
+
+Original audit: 1 November 2026, based on `a80b94236402e06138e61867c527cbf07540f60f` plus the then-uncommitted mTLS implementation. Rechecked on the same date against `19185259c32c816f8c3ab301efa28aa7b96c683b`. The recheck began at `da7ffb4`; the branch was squash-merged during review, and its tree is identical to the recorded recheck commit. Findings describe source code, not a deployed installation. This is a source review with isolated local reproductions, not a live infrastructure penetration test. Neither review changed runtime behavior.
+
+## Assessment
+
+The branch has fixed several original exploit paths: malformed status ports are rejected, foreign backup results are ignored, completed commands cannot reopen, disabled nodes are withdrawn from service/upstream lists, backup archives have private modes, default credential documents are excluded from source uploads, exact hostname conflicts are rejected and the originally identified dependency versions have been patched. A follow-up change then closed the enrollment-address injection, backup-credential release after placement withdrawal, explicit placement on disabled or unknown nodes, wildcard hostname overlap, custom credential files and character classes in source uploads, raw environment values in `inspect`, and unbounded child-process runtime and output. The critical deployment privilege issue (SEC-01), the global writer lifetime (SEC-09) and residual build-input and credential-location risks (SEC-06) remain. SEC-04's residual WireGuard revocation caveat is moot because the Tug-owned mesh has been removed; its historical evidence is retained below.
+
+Current disposition of the ten findings: one open, two partially fixed, five fixed for the reproduced finding, one moot after feature removal, and one with its original advisories patched. Original severities are retained below to identify the findings; they are not a count of ten still-unfixed vulnerabilities. Severity assumes the prerequisite described for each finding. None of the reproduced application findings establishes an unauthenticated remote takeover.
+
+| ID | Original severity | Current status | Recheck evidence |
+| --- | --- | --- | --- |
+| SEC-01 | Critical | Open | Scoped deployment still accepts a writable host Docker socket mount |
+| SEC-02 | High | Fixed | Status ports and the node `address` are validated; syntax-bearing addresses receive `400` |
+| SEC-03 | High | Fixed | Foreign/terminal report attacks are blocked; credentials are released only while the project is placed on the calling node |
+| SEC-04 | High | Moot after feature removal | The Tug-owned WireGuard mesh has been removed, so its certificate-expiry and peer-key replacement caveat no longer applies; API identity disable/replacement remains enforced |
+| SEC-05 | High | Fixed | Backup is `0600`, directory is `0700` under umask `0000`, and failed backup staging is cleaned |
+| SEC-06 | High | Partial | Remote and local builds exclude known credential paths; default credential placement and unknown secrets remain risks |
+| SEC-07 | High | Fixed | Exact and wildcard overlaps with another project's current hosts are rejected on deploy and rollback |
+| SEC-08 | High | Original advisories patched | Updated lockfile and a closed descriptor after the aborted-upload reproduction; a fresh complete advisory scan was unavailable |
+| SEC-09 | Medium | Partial | Child processes have a deadline and bounded output tail; the global writer still covers preparation and upload totals are unbounded |
+| SEC-10 | Medium | Fixed | Docs grant deploy access to workload secrets; `inspect` redacts environment values to avoid accidental disclosure |
+
+Tug does not encrypt node-to-node workload traffic. Reachability, firewalling and encryption for service connections and ingress upstreams are the operator network's responsibility; mutual TLS to control is unchanged. See [Bring your own network](networking.md#bring-your-own-network).
+
+The deployment trust model remains unchanged: every deployment author must be trusted with the nodes they can select, their co-located workloads and their secrets. Project scope is an API record boundary, not host isolation. The details below retain the original evidence as history, with current status and newly reproduced residuals stated before it.
+
+## Findings and required regression coverage
+
+### SEC-01: Project scopes do not confine deployment privileges
+
+**Recheck: open.** A credential scoped to project `alpha` still received `200` when deploying a service on `n2` with `/var/run/docker.sock:/var/run/docker.sock`. `containerSpecs` preserved that writable bind. No administrator-owned node/feature policy was found in the current admission path. The access-control document now accurately acknowledges this host-administrator trust requirement; documentation does not remove the capability.
+
+**Evidence:** `src/credentials.ts:35` grants deployment creation to `deploy`; `src/control.ts:2086` onward uses the shared node inventory without a per-actor node allowlist. `src/compose.ts:297` accepts any absolute bind source. `src/container-spec.ts:247` copies it into Engine `HostConfig.Binds`. External volumes, local-volume `driver_opts` and `cap_add` provide additional privileged surfaces.
+
+**Reproduction:** An operator credential limited to project `alpha` successfully submitted a deployment placed on `n2` with `volumes: [{ type: "bind", source: "/var/run/docker.sock", target: "/var/run/docker.sock" }]`. The resulting container plan retained `/var/run/docker.sock:/var/run/docker.sock` unchanged. No real host socket was mounted during the audit.
+
+**Impact:** The deployed process can command the host Docker daemon, create privileged containers and read or alter host files and other projects. Mounting `/` or a host-backed local volume provides other paths. If the node shares the control host, the control data directory is also exposed. API project checks and the absence of a `privileged` Compose key do not prevent this escalation.
+
+**Remediation:** Decide whether deployment authors are host administrators. For the current unrestricted model, state this explicitly and grant `deploy` only to equally trusted operators. To make project scopes a security boundary, enforce an administrator-owned policy for node placement, bind sources, external volumes, volume drivers/options and added capabilities before publishing desired state. Use separate hosts or control planes for mutually untrusted authors. A denylist of only the Docker socket is insufficient.
+
+**Regression coverage:** A scoped deployment must be refused when it requests an unauthorized node, host bind, foreign external volume, host-backed driver option or prohibited capability. Preserve explicitly authorized administrator use cases.
+
+### SEC-02: Node-reported ports become Caddyfile syntax
+
+**Current: fixed.** The remaining node `address` field is validated; the `publicAddress` field in the historical evidence below was removed with the Tug-owned mesh and is not a current API field. **Follow-up (history): fixed.** `insertNode` then rejected an `address` that was not an IP literal or hostname without a port and a `publicAddress` that was not a host with an optional port, so enrollment and identity creation could no longer store configuration syntax. Regression: `node addresses cannot carry configuration syntax`. **Recheck (history): partially fixed; configuration injection remained.** A malicious status port now receives `400` before any status write. `src/status-report.ts` enforces numeric port types, control filters unplaced services and `resolveServiceEndpoints` drops invalid stored ports. That historical residual is closed in the current code: enrollment and identity creation both call `insertNode`, which validates the node `address` before storage. The reproduction above remains historical evidence only. Real Caddy execution was not attempted.
+
+**Evidence:** `src/control.ts:3062` stores request project reports without runtime schema validation. `src/ingress.ts:71` accepts `serviceState.port` as supplied, and `src/ingress.ts:152` and `src/ingress.ts:193` interpolate it into shared Caddyfile text. The enrollment/identity address fields also lack strict IP/hostname validation before being used as upstream addresses.
+
+**Reproduction:** A node reported its placed service as serving with `port: "8080 {\n\t\theader_down X-Audit injected\n\t\t#"`. An ingress node's subsequent work response contained the injected `header_down` directive. The final `#` consumes the renderer's appended opening brace, leaving the intended block structure available to the attacker. Only configuration generation was exercised; a real Caddy process was not started.
+
+**Impact:** A compromised application node can affect the shared ingress agent beyond its own workload. Crafted configuration can disrupt all routes or introduce additional listeners, proxy handlers or file serving under the ingress container's permissions. The ingress container uses host networking and holds TLS material. Actual execution of those broader payloads remains an inference from the configuration injection, not an observed live exploit.
+
+**Remediation:** Validate the full status schema before any database write. Require reported ports to be integer values within `1..65535`, associated with a service currently placed on that node and an allowed active rollout. Validate node addresses at registration and enrollment and use structured upstream addresses. Prefer Caddy JSON configuration, retaining strict typed validation even with JSON.
+
+**Regression coverage:** Reject string, object, newline-bearing and out-of-range ports and malformed addresses; reject unplaced projects/services; demonstrate that valid numeric rolling ports still reach ingress.
+
+### SEC-03: Backup reports bypass ownership and command lifecycle checks
+
+**Follow-up: fixed.** `/agent/v1/backup/credentials` now also requires the project's current desired state to place a service on the calling node, so a running command left behind by a placement move receives `403`. A compromised node can still keep a job alive with heartbeats until a maximum execution lifetime is enforced. **Recheck (history): original result-forgery and terminal-reopening attacks fixed; credential lifecycle remained partial.** Foreign command results left the other node's catalog unchanged, and reporting `state: running` received `400` without reopening a terminal command. The handler now checks ownership and running state before scheduler calls. A residual was reproduced through normal API operations: publish project version 1 on `n1`, deliver its backup command, publish version 2 on `n2`, rotate the destination secret, then fetch that command's credentials as `n1`. The old node received the newly rotated destination secret with `200`, although the project was no longer placed there. `src/control.ts:3283` checks command owner/type/state but does not invalidate outstanding authorization on placement withdrawal. A compromised node can keep its running job alive through heartbeats. The existing command deadline is a pickup deadline; it should not be confused with an enforced maximum execution lifetime.
+
+**Evidence:** `src/control.ts:3075` passes job heartbeats to the scheduler without the calling node. `src/control.ts:3079` calls `backups.onResult` before the SQL update that checks `node_id`. `src/backup-scheduler.ts:577` and `src/backup-scheduler.ts:589` locate runs solely by command ID. The guarded command update accepts arbitrary reported states, including a transition from a terminal state back to `running`; the backup credential endpoint subsequently trusts that stored state without checking current placement.
+
+**Reproduction:** A running `backup.run` command and catalog row belonged to `n2` and project `beta`. Node `n1` posted a successful result for that command with a fabricated snapshot ID. The response was `200`; the foreign backup run became `succeeded`, while the correctly guarded command row remained `running`.
+
+In a second reproduction, an already-completed backup command belonged to `n1` and a project with no current placement on that node. Credential retrieval initially returned `409`. The node reported that historical command as `running`, then successfully fetched the currently configured dummy destination secret and the project's repository password. A former node can retain its own historical command IDs without discovering a foreign ID.
+
+**Impact:** A compromised node can forge another node's backup success/failure, poison snapshot selection and monitoring, or keep a foreign job alive through forged heartbeats. The cross-node variant requires a foreign command ID; IDs are random and are not shown to be enumerable through node-only APIs. The historical-command variant can disclose newly rotated backup-destination credentials after project placement has been removed. Restore planning trusts successful catalog entries, although the backup helper also checks snapshot project/target tags before restoring.
+
+**Remediation:** Resolve each command under the authenticated node before invoking any scheduler or restore-related side effect. Pass the node ID into scheduler methods and enforce ownership, command type, allowed transition and expected result schema there too. Nodes must not reopen terminal commands or mark commands as running through result reports. Verify current command authorization when releasing credentials. Ignore or reject foreign IDs consistently.
+
+**Regression coverage:** A foreign command result or heartbeat must leave commands, backup runs, schedules, timeout bookkeeping and audit events unchanged. A terminal command must remain terminal, and a historical command must not release newly configured credentials. Test valid same-node results and repeated terminal results as well.
+
+### SEC-04: Node disable revokes API identity but not network membership (historical mesh finding)
+
+**Current: moot after feature removal.** Tug no longer owns a WireGuard mesh or peer membership, so the residual certificate-expiry and key-replacement caveat has no current data path. This does not change the historical result below, and node API identities are still disabled or replaced as described in [Nodes](nodes.md#certificates-renewal-and-recovery). **Follow-up (history):** deployment now refuses any explicit `placement.nodes` entry that is not an enabled registered node. Certificate expiry in the mesh query and identity-replacement key handling remained open. **Recheck (history): partially fixed.** Disable correctly rejected subsequent node HTTP access and removed the node from healthy peers' desired network and ingress work. Label/global scheduling queries exclude it. However, a new deployment with explicit `placement.nodes: [n1]` still returned `200` after `n1` was disabled; filtering the node inventory does not validate the explicit selection. Existing placement may remain for recovery, but accepting new placement is a separate scheduling gap. Certificate expiry is still absent from the mesh query, and identity replacement retains the stored WireGuard peer key. Packet-level revocation and safe key replacement remain integration/recovery requirements.
+
+**Evidence (history):** `src/control.ts:1560` sets `disabled_at` and reconciles TLS sockets. `src/control.ts:2924` still returns every stored WireGuard peer; `src/control.ts:2904` constructs endpoint views without excluding disabled nodes. Deployment and scale inventory queries likewise omit the disabled flag.
+
+**Reproduction (history):** After disabling `n1`, its HTTP request was correctly denied. Node `n2` still received `n1` in `network.peers`, and the ingress node still received `n1` as an upstream while its last report was fresh. The actual WireGuard data path was not exercised.
+
+**Impact (history):** A quarantined host retaining its WireGuard private key remains authorized by healthy peers, and user traffic can continue reaching it. Staleness eventually withdraws ingress endpoints, but does not remove the WireGuard peer. Explicit placement and scaling can also continue considering disabled nodes.
+
+**Remediation proposed at the recheck (history):** Define disable as a consistent control, scheduling and network state. Exclude disabled nodes from peer lists, endpoint resolution, ingress and new placement; wake healthy peers immediately after a disable. Exclude expired node identities too if expiry is intended to revoke mesh access. Document that network removal cannot undo secrets already disclosed or stop a compromised host's local workloads.
+
+**Regression coverage proposed at the recheck (history):** Disabling a node must immediately remove it from healthy peers' desired network configurations and ingress endpoints and prevent new scheduling. Replacing an identity should restore only the intended memberships.
+
+### SEC-05: Control backup archives are readable beyond the control user
+
+**Recheck: fixed for the original permissions and cleanup finding.** Under umask `0000`, the backup archive was `0600` and its directory `0700`. Removing the test master key forced a backup failure, and no staging directory or partial archive remained. Source inspection confirms private staging and the installer directory chmod. Archives made before the fix remain sensitive, and existing installations retain the old data-directory mode until the installer is rerun or permissions are repaired. Backup export encryption remains an operational requirement rather than an implemented feature.
+
+**Evidence:** `src/control.ts:2584` copies the master key, and `src/control.ts:2585` copies PKI material into the backup. `src/control.ts:2595` writes the gzip archive without a restrictive creation mode. The control installer creates `/var/lib/tug-control` without a private directory mode.
+
+**Reproduction:** With the audit process's ordinary `0022` umask, the backup API produced a `0644` `.tar.gz` in a `0755` backup directory. Archive entries included `master.key` and `pki/ca.key`. The temporary audit root itself was private, so no other user actually read the test keys.
+
+**Impact:** On the installer-created directory hierarchy, another local account with directory traversal access can copy the entire backup. Encryption at rest no longer protects project, registry and backup-destination secrets because the decryption key is packaged beside the database. CA disclosure also enables forged control-server certificates when an attacker can redirect client connections. The backup does not contain operator private keys, and CA possession alone does not bypass the live server's stored principal key-hash checks.
+
+**Remediation:** Create control and backup directories as `0700`, staging directories as `0700` and archives as `0600` from their first write; harden already-created paths too. Encrypt exported backups with a separately held recovery key if they leave the protected host. Treat every existing archive as sensitive key material.
+
+**Regression coverage:** Assert initial directory/archive modes under a permissive umask, including the error path, and verify that cleanup removes staging material after failed backup creation.
+
+### SEC-06: Root credentials and ignored secrets enter source archives
+
+**Current:** Remote and local builds now use the same filtered, validated source snapshot. The resolved credential path and recognized credential documents are mandatory exclusions even when ignore rules re-include them; a real Docker `COPY .` test checks the resulting image. The default credential output remains in the working directory, and arbitrary secret names and full Docker ignore semantics are not covered. **Earlier follow-up:** `tug deploy` began passing the resolved credential file to the source archive filter, and `.dockerignore` character classes (`[ab]`, `[a-z]`, `[!c]`) were implemented. **Recheck (history): partially fixed.** A default `tug-root.credential.json` is now excluded and `.gitignore` covers that naming pattern. Custom paths remain exposed: `src/cli.ts:1217` passes only the tracked config and overlay to `createSourceArchive`, omitting `resolvedConfig.credentialFile`, so an in-tree credential named `admin.json` still appeared in the archive. The hand-written ignore matcher also escapes character classes instead of implementing them: `.dockerignore` entry `secrets/[ab].pem` did not exclude `secrets/a.pem`. Local image builds still receive the raw checkout through `prepareLocalImages`, without mandatory Tug credential exclusions. The default credential output remains in the working directory. Do not describe this as complete Docker ignore compatibility or complete credential protection.
+
+**Evidence (original audit):** `src/cli.ts:457` defaults control installation output to `tug-root.credential.json` in the working directory. It is not covered by the repository's credential ignore rules or `src/cli-build.ts:19`. `src/cli.ts:1198` excludes only the tracked config and local overlay, omitting the resolved credential file. `src/cli-build.ts:56` reads `.dockerignore` rules, but `sourceArchiveFilter` compares literal paths instead of Docker pattern semantics.
+
+**Reproduction (original audit):** A source directory with `.dockerignore` entries `*.pem`, `.env*` and `private/**` still uploaded `key.pem`, `.env.production` and `private/config.json`. A dummy `tug-root.credential.json` was also present in the resulting archive. Only dummy values were used.
+
+**Impact (original audit):** An ordinary subsequent remote build can upload the full administrator credential to control. A `COPY .` Dockerfile can include it in an application image when it is not correctly excluded at build time, giving a compromised application control-wide credentials. Matching Docker ignore rules still protect the actual Docker build context, but they do not currently protect the earlier Tug source upload. The installation output is also easy to commit accidentally.
+
+**Remediation:** Default credential output outside source trees, automatically protect explicitly selected output paths, exclude the resolved credential path and recognized credential documents as mandatory rules, and implement compatible `.dockerignore` semantics for source uploads. Keep mandatory credential exclusions effective even when a pattern attempts to re-include them. Explain how to dispose of the original imported credential file.
+
+**Regression coverage:** Exercise the installation-output-to-deploy workflow with a dummy credential, in-tree credential overrides, glob and recursive patterns, nested ignore paths and negation. Assert credentials are absent from uploaded archives and any resulting image context.
+
+### SEC-07: Ingress host ownership is not authorized across projects
+
+**Follow-up: fixed.** Publication and rollback now reject any host that equals or is covered by (or covers) a `*.` wildcard routed by another project, using `ingressHostsOverlap`. Wildcards do not cover their apex. **Recheck (history): partially fixed.** An existing exact hostname blocked a case-varied claim from a scoped credential with `409`; source inspection and regression coverage also show the rollback check. Wildcards are accepted: `src/compose.ts:136` begins its hostname regex with optional `*.`. Publishing `*.example.org` for one project followed by `login.example.org` for another succeeded twice, so overlapping ownership is not checked. The former statement that wildcard overlap could not arise was incorrect. The current global writer serializes deploy and rollback; preserve that consistency or recheck ownership at publication when shortening the writer lifetime. Per-project grants remain unimplemented. Real Caddy route precedence was not exercised.
+
+**Evidence:** `src/compose.ts:145` validates hostname duplication only within the submitted Compose definition. `src/ingress.ts:139` sorts all project/service routes; `src/ingress.ts:144` gives a duplicate host to the first route and only warns about the skipped project.
+
+**Reproduction:** Desired states for `victim` and the lexically earlier project `alpha` both claimed `alpha.example.com`. Route generation selected `alpha` and warned that `victim` was skipped. Input order did not preserve the earlier owner's route. This reproduction exercised route selection, not public traffic.
+
+**Impact:** A deployment credential limited to a lexically earlier project can take a hostname already serving another project on shared ingress. The attacker does not need that project's API scope or a host mount. Automatic TLS issuance validates domain control for the shared ingress address, not authorization between projects on that same ingress.
+
+**Remediation:** Reserve hostnames across current deployments at publication time and reject conflicting owners, including case normalization and applicable wildcard overlaps. Use an administrator-managed hostname allowlist per project if scoped deployment authors should not claim arbitrary domains. Rollbacks must obey the same ownership rules.
+
+**Regression coverage:** A scoped project cannot take an existing hostname through deploy, scale-related regeneration or rollback; the existing route remains selected. Test normalized names, wildcard policy and explicit release/reassignment.
+
+### SEC-08: Published dependency vulnerabilities remain in the lockfile
+
+**Recheck: original advisories patched; no fresh clean-scan claim.** The current lockfile resolves Multer `2.5.0`, js-yaml `5.5.0`, tar `7.5.22`, Express `5.3.0`, qs `6.17.1` and proxy-addr `2.0.8`. The aborted-upload reproduction now observes `closed: true`, `destroyed: true` and no open destination descriptor. Maintainer affected-version ranges for the original advisories were rechecked, including the js-yaml 5.x counterpart [GHSA-r3ph-w7gj-g6xm](https://github.com/nodeca/js-yaml/security/advisories/GHSA-r3ph-w7gj-g6xm). The new npm audit attempt failed with `ENOTFOUND registry.npmjs.org`, so a full current advisory or container scan was not verified. Direct dependency minimums for Multer and tar still permit old vulnerable versions if the lockfile is regenerated; raising them and adding parser/work bounds and CI scans remain hardening work.
+
+The attempted `npm audit --json --fetch-retries=0 --fetch-timeout=15000` could not reach `registry.npmjs.org` because DNS/network access is restricted. Maintainer advisories were checked through web retrieval instead. This is a targeted advisory review, not an exhaustive clean dependency scan.
+
+| Locked production package | Confirmed advisory | Applicability | Required action |
+| --- | --- | --- | --- |
+| `multer@2.2.0` | [GHSA-qfvm-cv95-jqjf](https://github.com/expressjs/multer/security/advisories/GHSA-qfvm-cv95-jqjf), CVE-2026-77037 | Tug uses the affected disk storage path for authenticated deployment uploads | Upgrade to a patched release, at least `2.3.0` |
+| `js-yaml@4.2.0` | [GHSA-2883-xcg3-v3hh](https://github.com/nodeca/js-yaml/security/advisories/GHSA-2883-xcg3-v3hh) | Untrusted local config/Compose YAML can cause excessive parsing work; the control deployment API parses JSON | Upgrade the 4.x line to at least `4.3.2` and bound input size/work |
+| `tar@7.5.17` | [GHSA-w8wr-v893-vjvp](https://github.com/isaacs/node-tar/security/advisories/GHSA-w8wr-v893-vjvp), [GHSA-23hp-3jrh-7fpw](https://github.com/isaacs/node-tar/security/advisories/GHSA-23hp-3jrh-7fpw), [GHSA-r292-9mhp-454m](https://github.com/isaacs/node-tar/security/advisories/GHSA-r292-9mhp-454m) | Version falls in affected ranges; Tug's prescan and entry limits mitigate some paths, and the member-selection recursion path was not established as reachable | Upgrade to a currently patched version, at least `7.5.21` for these advisories; retain and strengthen application bounds |
+
+**Reproduction:** A synthetic multipart file upload through the installed Multer middleware, with the same limits as control, was aborted after opening the destination. The callback completed while the destination stream remained `closed: false`, `destroyed: false` with an open file descriptor. The audit explicitly destroyed the stream afterward. No listening socket or repeated resource exhaustion was required.
+
+**Impact:** Repeated failed deployment uploads can exhaust descriptors and retain disk blocks in the control process. The role/class gates mean this specific Tug upload path requires a `deploy` or admin identity, unlike an unauthenticated generic Multer installation. Other Multer advisories requiring two text fields are not claimed as demonstrated because Tug limits text fields to one. The YAML and tar issues are dependency risks with the narrower reachability stated above.
+
+**Regression coverage:** Aborted and truncated uploads must close destination streams and remove files. Run malformed archive checks in subprocesses so unexpected process termination is observable. Run a complete production and development dependency scan and container OS-package scan in network-enabled CI.
+
+### SEC-09: Builds can hold a global writer indefinitely
+
+**Follow-up: partial.** `run` in `src/build.ts` now kills child processes after 30 minutes and retains only the last 1 MiB of each output stream. The global writer still covers preparation, and multipart limits still lack an aggregate budget or per-principal quotas. **Recheck (history): open.** `src/control.ts:2102` still holds the global writer through `deployRequest`, including preparation. `src/build.ts:23` still spawns without a cancellation signal/deadline and appends all stdout/stderr to strings. Multipart limits remain 64 files of up to 5 GiB each without an aggregate request budget. No destructive load test was run.
+
+**Evidence:** `src/control.ts:758` maintains a process-wide writer queue. Deployment handling enters it at `src/control.ts:2049` and holds it while awaiting build, save, import and resolution work. `src/build.ts:23` starts child processes without a timeout or cancellation signal and accumulates all stdout/stderr in memory. Cancelling the deployment marks its record but does not terminate the running build. Multipart limits apply per file, allowing up to 64 files of 5 GiB each by default before aggregate validation.
+
+**Impact:** One malicious or stalled Dockerfile, such as a build step that never finishes, can block every later deployment, rollback and control backup/restore waiting on that writer. Excessive build output can exhaust control memory. Uploads and queued operations also lack per-principal aggregate storage/concurrency quotas. A denial-of-service load test was deliberately not run on the shared Docker host.
+
+**Remediation:** Apply build and child-process deadlines, bounded output tails, cancellation that terminates the process tree, total upload byte limits and per-principal admission quotas. Perform long-running preparation outside the global writer and hold the publication lock only for the short validated state transition. Preserve project-level exclusion and recheck authorization/leases at publication.
+
+**Regression coverage:** A blocked build in project A must not block an unrelated project's deployment or administrative recovery. Cancellation/timeout must terminate work and release resources. Oversized combined uploads and output must be rejected without unbounded buffering.
+
+### SEC-10: Deploy-role operations expose project secret values
+
+**Follow-up: fixed.** Live `inspect` keeps `Config.Env` names but replaces values with `<redacted>`; this reduces accidental disclosure only. **Recheck (history): misleading promise corrected; inspect redaction was pending.** `docs/access-control.md:32` now explicitly states that deploy/exec grants workload-secret access. This resolves the original documentation mismatch under the chosen trusted-operator model. A fake Docker inspect result still forwarded a dummy secret through `src/live.ts:234`, so the planned safe inspect projection has not been implemented. Redaction would reduce accidental disclosure, not create a secrecy boundary against deployment authors.
+
+**Evidence:** `/api/v1/live` requires `deploy`. Its `inspect` operation reaches `src/live.ts:234`, which forwards the complete `docker.inspectContainer` result, including resolved `Config.Env`. Exec can also read the environment, and a deployment author can create a workload that sends referenced secrets elsewhere. This contradicts the former access-control statement that a deploy token can never read secret values.
+
+**Reproduction:** The live operation engine forwarded an inspect result containing a dummy `DATABASE_PASSWORD` value. This was a structured forwarding test with a fake Docker API, not a read of real container credentials.
+
+**Impact:** Anyone granted deploy access can read service secrets in that project. This is usually an inherent capability of application deployment and execution; the security defect is granting that capability under a contradictory promise. It remains relevant even after host access in SEC-01 is restricted.
+
+**Remediation:** Treat deployment/execution access as access to the workload's secrets and document that contract. Redact inspect environment values by default to reduce accidental disclosure, while making clear that redaction cannot constrain an actor allowed to deploy or execute arbitrary application code. Separating those privileges requires a constrained deployment model.
+
+**Regression coverage:** Verify the chosen inspect redaction behavior and role policy, and keep the documented secret-access contract consistent with exec, one-off runs, logs and deployment capabilities.
+
+## Reviewed controls that provide useful protection
+
+- The production listener requires CA-validated client certificates with TLS 1.3; identity class, URI and stored public-key hash are checked against current database state. The route class gate runs before body parsing. Real network handshakes could not be rerun in this sandbox, so these are source-reviewed controls rather than a new live-handshake certification.
+- Operator roles come from the route table, and the common guard enforces declared roles and project scope. Node-only routes do not accept operator identities. Revocation/disable closes tracked TLS connections, and ongoing requests recheck identity state.
+- CSRs are signature-checked, restricted to P-256 and signed with server-assigned identity/usage extensions. Credential and private-key writes use restrictive modes and durable replacement. Client transports pin the supplied CA and verify the public control hostname even when the TCP destination is overridden; redirects are refused.
+- Secrets use AES-256-GCM with random nonces. The normal node secret endpoint filters by service placement and referenced secret names; registry credentials are separately scoped by registry host. These controls do not remove deployment-author secret access or the outstanding backup credential lifecycle gap.
+- Uploaded source/image archives are prescanned, links and path traversal are rejected, image metadata and tags are checked before import, and agent image downloads verify byte count and SHA-256. Build context and Dockerfile resolution reject paths outside the uploaded root.
+- SQL values generally use bound parameters. Process execution generally uses argv rather than a shell; SSH command interpolation uses shell quoting, and rendered control Compose scalar values are quoted. No literal production private keys or common provider-token patterns were found in the reviewed repository scan; this was not a full Git-history or entropy scan.
+
+## Validation and limits
+
+The recheck full test run under Node `v26.13.0` reported 456 tests: 327 passed and 129 failed. Failure entries comprised 47 explicit listener `EPERM` errors, 81 attempts to read a null server address after failed listening, and one Docker API test-file failure associated with its listener errors. No assertion failure independent of those setup failures was observed. This is not a passing full-suite result. A targeted run of 97 existing tests covering status parsing, builds, archives, registry handling, encryption, Compose/container rendering, ingress, mesh, backup scheduling, restore and agent jobs passed.
+
+Thirteen isolated Node probes, without a listening socket or real Docker operations, confirmed current behavior: scoped host mounts, rejection of injected status ports, blocked foreign/terminal command reports, stale-placement backup credential release after rotation, exact-host conflict rejection, wildcard-overlap acceptance, enrollment-address injection, disabled-node withdrawal, acceptance of explicit disabled-node placement, backup modes and failure cleanup, default/custom credential archive behavior, raw inspect forwarding and patched Multer descriptor cleanup. Only dummy credentials and temporary private test state were used.
+
+`npm run typecheck`, `npm run lint` and `npm run build` passed. The live mTLS suite, WebSocket transport tests, Docker-backed integration tests, actual Caddy loading/route precedence, WireGuard packet revocation and container image/OS scanning were not validated here. Local TCP listeners are denied; no real Docker operations were attempted during the isolated reproductions. npm advisory access from the shell also failed, so the dependency result establishes fixes for the original advisories rather than a complete new clean scan. Repository branch protections, deployed firewall rules, cloud IAM, backup-bucket policies and running host/image patch levels were not inspected.
+
+At the time of the audit the README and several operational documents still described pre-mTLS bearer-token installation and authentication; the documentation has since been updated to match the code, which requires credential files and direct mTLS or TLS passthrough. A conventional TLS-terminating proxy cannot substitute for the end-to-end client certificate check implemented here.
+
+## Remediation order
+
+1. Move the default credential output outside source trees and address remaining build-input gaps (SEC-06). The trusted-operator decision and its limits are in [Roadmap](roadmap.md#product-decisions-and-reasoning); do not imply project scopes isolate untrusted workloads (SEC-01).
+2. Bound running-job lifetime (SEC-03). SEC-04's mesh-specific residual is closed by removal of the Tug-owned mesh, not by a claim that packet revocation was implemented.
+3. Preserve the verified backup permission and upload cleanup fixes (SEC-05/SEC-08), and rerun complete dependency/container scans in suitable CI.
+4. Move preparation outside the global writer and bound uploads and queue occupancy (SEC-09), and run the full transport/Docker suites outside the restricted sandbox.

package/docs/service-placement.md

@@ -0,0 +1,72 @@
+# Service placement and scaling
+
+With one registered node, a service with no placement runs there. Once multiple nodes are registered, each service must select explicit nodes, labels or `global: true`. Adding a node does not automatically move a deployed service, copy its data or provide failover. Placement is resolved at deploy or scale time, not continuously rescheduled when a node fails.
+
+## Spread the first service across two nodes
+
+This continues from [Deploy your first service](getting-started.md). The nginx service is stateless; both VPSes can run an independent copy. The example keeps the tutorial's public port, so restrict TCP 8080 to your computer on both VPSes with provider firewalls. For a public site, use [ingress and HTTPS](networking.md#layer-b-ingress-and-tls) instead.
+
+1. Give the new VPS a reachable address on your private network. Make sure it can connect to control on TCP 8443. To use services across nodes or ingress on one node for services on the other, both hosts must have mutually reachable private addresses and appropriate firewall rules. Tug does not create or encrypt this network. See [Networking](networking.md#bring-your-own-network).
+2. Add the new agent, replacing the SSH address and private address with yours. With an interactive terminal Tug offers Docker installation and shows the plan before acting.
+
+```sh
+tug node add root@198.51.100.7 --name app2 --address 10.0.0.7
+tug node list
+```
+
+3. In your existing `compose.yaml`, add placement to `web`. Keep its image, port and health check:
+
+```yaml
+services:
+  web:
+    image: nginx:1.27-alpine
+    ports:
+      - "0.0.0.0:8080:80"
+    x-tug:
+      placement:
+        nodes: [app1, app2]
+      health:
+        type: http
+        port: 80
+        path: /
+```
+
+4. Redeploy and check both nodes. Replace the example public IPs with the VPS addresses; the nginx welcome page should respond from each.
+
+```sh
+tug deploy
+tug status
+curl http://203.0.113.5:8080/
+curl http://198.51.100.7:8080/
+```
+
+This places one container on each named node; it does not load-balance the two public addresses. Managed ingress can route to healthy services on multiple nodes when their workload network is reachable. Its hostname needs DNS pointing to an ingress node; see [Ingress and TLS](networking.md#layer-b-ingress-and-tls).
+
+## Keep state on its server
+
+A named local volume is local to a node. For an application with a database, pin the database service and its volume to the node holding the data, while choosing placement for every other service. For example, add these blocks to your existing database service and volume definitions:
+
+```yaml
+services:
+  db:
+    x-tug:
+      placement:
+        nodes: [app1]
+volumes:
+  db-data:
+    x-tug:
+      placement:
+        nodes: [app1]
+```
+
+This is a placement fragment, not a complete database Compose file. If a web service on `app2` connects to the database on `app1`, the database must publish a port on its private node address, the client must use the published host port, and the network and firewall must allow that connection. Same-node services use Docker's project network. Cross-node service names are hosts-file entries and do not encrypt traffic; see [Cross-node service names](networking.md#layer-a-cross-node-service-names). Use application connection retries.
+
+To move a stateful service, back up and restore its data. Tug requires an explicit `tug deploy --move <service>` for guarded moves and holds detected writers until restore or release; see [Move the database](migration.md#move-the-database). Test recovery before moving data.
+
+## Other placement options
+
+- **Explicit nodes:** `placement.nodes: [app1, app2]` runs one copy on each listed node. Scaling stays within the named nodes.
+- **Labels and replicas:** add nodes with `--label web`, then select `placement.labels: [web]` and `replicas: 2`. Tug chooses up to two matching, recently seen nodes when placement is resolved. All listed labels must match. Nodes created without the label do not match; [node labels](nodes.md#node-labels) explains how they are set.
+- **Global:** `global: true` runs one copy on each matching node, or each node if no labels are specified.
+
+`tug scale web 2` changes the current desired placement within its allowed set; the next deploy re-resolves it from the Compose file. No mode guarantees multiple copies on one node or automatically replaces a failed host. Check placement with `tug status`. The [placement reference](compose.md#services) lists the complete rules.