@wpelevator/tug 0.11.0 → 0.12.0

package/README.md

@@ -1,8 +1,10 @@
 # Tug
 
-Deploy Docker Compose apps to a few ordinary Linux servers, from your laptop or CI, without Kubernetes or a hosted platform.
+Deploy Docker Compose apps to your own Linux VPS, from your laptop or CI, without Kubernetes or a hosted platform.
 
-You keep your `docker-compose.yml`. Tug installs a small control plane on one server and an agent on each of the others, then `tug deploy` puts your services where you say, waits until they are healthy and rolls back if they are not. It is built for running one to a few dozen sites, WordPress first, on two to ten VPSes, with backups and restores included.
+A VPS is a durable computer with enough resources to host multiple sites or a medium-size application and its database. Tug helps you use that capacity: keep your Compose file, deploy with one command, and manage rollouts, secrets, backups and restores. Start with one server and add more when you need them. Tug is built for a small number of servers and Compose applications, including Node.js, Laravel, Ruby and WordPress apps.
+
+Tug runs a small control plane and an agent, which can share the same VPS with your applications. It builds application images on the control server or your machine and delivers them to your nodes, so you do not need to run or configure an image registry for those builds. Published images are still pulled from their upstream registries.
 
 ## What you can do with it
 
@@ -14,48 +16,73 @@ You keep your `docker-compose.yml`. Tug installs a small control plane on one se
 - **Look inside running services.** `tug logs`, `tug exec`, `tug run` (for example `wp` CLI), `tug ps` and `tug status` work through the control plane, so you never SSH to a node to see what is going on.
 - **Give CI a limited key.** `tug credential create` makes read-only, deploy-only or admin credentials, and every change records who made it.
 
-Nodes only dial out to the control plane, so they work behind NAT or a firewall that allows nothing inbound. Tug does not need a cloud provider's API, a registry or an overlay network.
+Agents only dial out to the control plane, so nodes can work behind NAT or a firewall that allows nothing inbound. Tug does not need a cloud provider's API or an overlay network.
 
 ## Try it
 
-You need Node.js 22.18 or newer and Docker Compose v2 on your machine, and SSH access to one or more Linux servers (Debian, Ubuntu, Fedora or a RHEL-compatible distribution). Tug can install Docker on them for you.
+Follow [Deploy your first service](docs/getting-started.md) to install Tug, deploy nginx to one VPS and verify that it works. You need Node.js 22.18 or newer and Docker Compose v2 on your computer, plus SSH access to a Linux server. Tug can install Docker on supported servers.
 
 ```bash
 npm install -g @wpelevator/tug      # or run it with npx @wpelevator/tug
 
-tug setup root@203.0.113.5          # install the control plane on a server
-tug node add root@198.51.100.7      # add a server that will run your apps
-tug deploy                          # deploy ./docker-compose.yml
+tug setup root@203.0.113.5          # install the control plane
+tug node add root@203.0.113.5 --name app1  # install the agent on the same VPS
+tug deploy                          # deploy a Compose file in this directory
 ```
 
+The control plane and agent share the VPS with your application. Once your service is running, you can [add HTTPS](docs/networking.md#layer-b-ingress-and-tls), [back up your data](docs/backups.md) or [add another server](docs/service-placement.md).
+
 In a terminal Tug asks before it changes anything: whether to install Docker, which private address the node should use and whether it should handle ingress. It shows the plan and waits for you to confirm. `tug deploy` asks for any Compose variables, such as `${DB_PASSWORD}`, that no secret or `.env` provides. In CI or any other script nothing is ever asked, so pass the choices as flags (`--address`, `--ingress`, `--install-docker`, `-y`).
 
 ## Your Compose file, plus a little `x-tug`
 
-A normal Compose file already works. Add `x-tug` to say where services run, how to tell they are healthy and where traffic comes in:
+This example builds on the [quickstart](docs/getting-started.md). It pins the app, database and database volume to `app1`. Placement can be omitted with one node, but explicit placement keeps the database on its chosen server when you add more nodes. Enable ingress with `tug node ingress app1`, point the hostname's DNS at the VPS, and allow public ports 80 and 443. Save this as `compose.yaml` and replace the hostname with yours:
 
 ```yaml
 services:
   db:
     image: mariadb:11.4.8
-    volumes: [db-data:/var/lib/mysql]
+    volumes:
+      - db-data:/var/lib/mysql
     environment:
       MARIADB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
     x-tug:
-      placement: { nodes: [app1] }
-      health: { type: tcp, port: 3306 }
+      placement:
+        nodes:
+          - app1
+      health:
+        type: tcp
+        port: 3306
 
   wordpress:
     image: wordpress:6.8.3-apache
-    depends_on: [db]
+    depends_on:
+      - db
+    environment:
+      WORDPRESS_DB_HOST: db
+      WORDPRESS_DB_USER: root
+      WORDPRESS_DB_PASSWORD: ${DB_ROOT_PASSWORD}
+    ports:
+      - "8080:80"
     x-tug:
-      placement: { nodes: [app1] }
-      health: { type: http, port: 80, path: /wp-includes/version.php }
-      ingress: { host: blog.example.com, port: 80, tls: auto }
+      placement:
+        nodes:
+          - app1
+      health:
+        type: http
+        port: 80
+        path: /wp-includes/version.php
+      ingress:
+        host: blog.example.com
+        port: 80
+        tls: auto
 
 volumes:
   db-data:
-    x-tug: { placement: { nodes: [app1] } }
+    x-tug:
+      placement:
+        nodes:
+          - app1
 ```
 
 ```bash
@@ -65,7 +92,9 @@ tug status       # what is running where, and anything failing
 tug logs wordpress --follow
 ```
 
-A complete WordPress site with scheduled database and uploads backups is in the [WordPress example](examples/wordpress/README.md). Everything `x-tug` can express is in [Compose extensions](docs/compose.md).
+The named volume holds data on `app1`; it is not copied to another node automatically. Back it up and restore or move its data deliberately if you replace the server.
+
+See the [examples directory](examples/) for stack-specific applications. Everything `x-tug` can express is in [Compose extensions](docs/compose.md).
 
 ## Everyday commands
 
@@ -89,68 +118,77 @@ Every command and option is in the [CLI reference](docs/cli.md). `tug status --j
 
 ## Is Tug right for you?
 
-Tug fits if you run a handful of Compose apps on VPSes you already pay for, want deploys, rollbacks, secrets and backups handled in one tool, and are happy with a single control server that you back up.
+Tug fits if you run Compose apps on one or a few VPSes, want deploys, rollbacks, secrets and backups handled in one tool, and are happy with a single control server that you back up. A server can host multiple apps as long as it has enough capacity; Tug does not automatically balance resource usage or replace a failed server.
 
 It deliberately does not do Kubernetes-style scheduling, autoscaling, distributed storage or database failover, and it does not provision servers through a provider's API. Node-to-node traffic travels over your own private network (Tailscale, a VPC, a LAN), so you own its firewalling and encryption. If you want automatic rescheduling of containers across a cluster, see how it compares with [Docker Swarm](docs/swarm-comparison.md). The goals, principles and planned milestones are in the [Roadmap](docs/roadmap.md).
 
 ## How it works
 
 ```text
-  tug CLI / CI (operator credential)
-       |
-       |  mTLS: deploy, status, logs, exec
-       v
-+-------------------------------------+
-| Control plane (one mTLS port)       |
-|   Control API (role-checked)        |
-|   SQLite: desired states, secrets   |
-|   private CA, image builds          |
-+-------------------------------------+
-       ^                       ^
-       |  agents dial out: work, status,
-       |  WebSocket wake and live sessions
-       |                       |
-+------+------------+   +------+------------------+
-| Node app1         |   | Node app2 (ingress node)|
-|   Agent           |   |   Agent                 |
-|   Docker Engine   |   |   Docker Engine         |
-|   services        |   |   services              |
-|                   |   |   managed Caddy ingress |
-+-------------------+   +-------------------------+
-        \_________________________/
-          workload traffic between node addresses
-          over the operator's private network
-
-  SSH is used only by the CLI to install the control plane and nodes.
+                      tug CLI / CI
+                           |
+                           | mTLS: deploy, status, logs, exec
+                           v
++--------------------------------+     +--------------------------------+
+| VPS app1                       |     | VPS app2 (optional)            |
+|                                |     |                                |
+| Control plane                  |     |                                |
+| API, SQLite, image builds      |     |                                |
+|                                |     |                                |
+| Agent --> Docker Engine        |     | Agent --> Docker Engine        |
+|              |                 |     |              |                 |
+|              +-- service A     |     |              +-- service C     |
+|              +-- database      |     |              +-- service D     |
+|              +-- more...       |     |              +-- more...       |
+|     (zero or more services)    |     |     (zero or more services)    |
+|                                |     |                                |
+| Local volumes, optional Caddy  |     | Local volumes, optional Caddy  |
++--------------------------------+     +--------------------------------+
+             |                                         |
+             +------- private workload network --------+
+
+  Each agent connects to the control plane over mTLS. Services are
+  placed on either node; local volumes stay on their node. app1 can
+  run alone. SSH installs control and agents; normal operations use
+  the control API.
 ```
 
-`tug deploy` reads your Compose file on your machine and sends it to the control plane, which stores it as an immutable desired state, builds any images and pins published images to digests. Each agent fetches the work for its node, applies it through the Docker Engine API, runs the health checks and reports back. The control plane marks the deployment done when every node reports it serving. Compose runs only on your machine; nodes need just Docker.
+`tug deploy` reads your Compose file on your machine and sends it to the control plane, which stores it as an immutable desired state, builds application images when needed and pins published images to digests. Tug delivers its built images as archives; published images are pulled from their upstream registries. Each agent fetches the work for its node, applies it through the Docker Engine API, runs the health checks and reports back. The control plane marks the deployment done when every node reports it serving. Compose runs only on your machine for application normalization; nodes need Docker, and the control installation also uses Compose. Cross-node workload traffic uses your own private network.
 
 Control serves mutual TLS directly on one port (`8443`). Every operator, CI job and node authenticates with a certificate from a private CA created at install, so the control address can be a bare IP and can change later. Do not terminate TLS in front of it. A `controlUrl` in `tug.yml` tells the CLI where control is (see [CLI configuration](docs/cli.md#config-file)). [Architecture](docs/architecture.md) covers every flow in detail, and [Security](docs/security.md) the trust boundaries.
 
-The full walkthrough, including CI credentials, moving the control plane and recovery, is in [Getting started](docs/getting-started.md).
-
 ## Documentation
 
-- [Getting started](docs/getting-started.md): install control, add nodes, deploy, give CI a credential, move control, recover.
-- [Terminology](docs/terminology.md): core concepts, the naming review (which recommendations are implemented), industry comparisons and the remaining proposed interface improvements.
-- [Architecture](docs/architecture.md): components, reconciliation, workflow diagrams, secrets and runtime paths.
-- [Compose extensions](docs/compose.md): `x-tug` placement, health checks, volumes and limitations.
-- [Control plane](docs/control-plane.md): install, mutual TLS and the private CA, image delivery, control-plane backup and recovery on another host.
-- [Backups](docs/backups.md): scheduled workload backups with `x-tug.backup` (restic), restore, recipes for common databases, and a restore drill. See also the [WordPress example](examples/wordpress/README.md).
-- [Access control](docs/access-control.md): certificate identity classes, scoped operator credentials, roles, audit trail and a GitHub Actions deploy example.
-- [Security](docs/security.md): deployment trust boundaries, security audit findings and validation limits.
-- [Networking](docs/networking.md): bring-your-own private networking, cross-node service names, ingress and TLS, firewall notes.
-- [Nodes](docs/nodes.md): node names and IDs, SSH install, self-service enrollment, certificate renewal and recovery, registration and status.
-- [CLI reference](docs/cli.md): every `tug` command and option.
-- [Configuration](docs/configuration.md): CLI, control and agent environment variables.
-- [Development](docs/development.md): local setup, CI, Compose stack, host mode and the integration test.
-- [Testing](docs/testing.md): test layers, rollout metrics, mocked test backlog, and the [Caddy/PHP and static-file benchmark](docs/testing.md#caddy-proxy-overhead-benchmark).
-- [Observability](docs/observability.md): what can be observed today, and the proposed logs, metrics and alerting.
-- [Releases](docs/releases.md): changesets, npm and Docker image publishing.
-- [Standalone binary](docs/standalone-binary.md): proposed Bun-compiled CLI binaries and Homebrew distribution, with the mTLS dependency findings.
-- [Roadmap](docs/roadmap.md): goals, principles, the reference WordPress topology and planned milestones.
-- [Docker Swarm comparison](docs/swarm-comparison.md): overlapping capabilities, workflow advantages, scheduling and networking limits, failure behavior and workload fit.
+Start with the [single-VPS tutorial](docs/getting-started.md), then choose the topic that matches your next task:
+
+- **Deploy and operate**
+  - [Application examples](examples/README.md)
+  - [Health checks and rollouts](docs/healthchecks.md)
+  - [Troubleshooting](docs/troubleshooting.md)
+- **Networking and access**
+  - [Hostnames, HTTPS and service networking](docs/networking.md)
+  - [Credentials and CI](docs/access-control.md)
+  - [Security trust boundaries](docs/security.md)
+- **Servers and data**
+  - [Service placement and scaling](docs/service-placement.md)
+  - [Node management](docs/nodes.md)
+  - [Backups and recovery](docs/backups.md)
+  - [Moving to new servers](docs/migration.md)
+- **Reference**
+  - [CLI](docs/cli.md)
+  - [Compose extensions](docs/compose.md)
+  - [Configuration](docs/configuration.md)
+  - [Control plane](docs/control-plane.md)
+- **Contribute and design**
+  - [Development](docs/development.md)
+  - [Testing](docs/testing.md)
+  - [Releases](docs/releases.md)
+  - [Architecture](docs/architecture.md)
+  - [Roadmap](docs/roadmap.md)
+  - [Observability status and proposal](docs/observability.md)
+  - [Terminology](docs/terminology.md)
+  - [Standalone binary proposal](docs/standalone-binary.md)
+  - [Docker Swarm comparison](docs/swarm-comparison.md)
 
 ## Development
 

package/dist/agent.js

@@ -518,9 +518,11 @@ export async function waitForHealth(ctx, project, container, service, health) {
                     throw new Error(`${service}: docker health ${state.Health.Status}`);
             }
             else if (health.type === 'tcp') {
+                await assertRunning(ctx, container, service);
                 await tcpCheck(await containerIp(ctx, project, container), health.port, parseDuration(health.timeout || '2s'));
             }
             else if (health.type === 'http') {
+                await assertRunning(ctx, container, service);
                 const ip = await containerIp(ctx, project, container);
                 const res = await ctx.http(`http://${ip}:${health.port || 80}${health.path || '/'}`, {
                     signal: AbortSignal.timeout(parseDuration(health.timeout || '2s')),
@@ -537,6 +539,33 @@ export async function waitForHealth(ctx, project, container, service, health) {
     }
     throw new Error(`${service}: ${describeHealthCheck(health)} failed after ${retries} attempt${retries === 1 ? '' : 's'}: ${describeError(last)}${outsideContainerHint(health)}`);
 }
+/** Fails fast with the exit code and last log lines when the container is gone, instead of dialing a dead address. */
+async function assertRunning(ctx, container, service) {
+    const state = (await ctx.docker.inspectContainer(container)).State;
+    if (!state || state.Running)
+        return;
+    const tail = await lastLogLines(ctx, container);
+    throw new Error(`${service}: container exited with code ${state.ExitCode ?? '?'}${tail ? `; last log: ${tail}` : ''}`);
+}
+async function lastLogLines(ctx, container, lines = 3) {
+    try {
+        const proc = await ctx.docker.logs(container, { follow: false, tail: lines });
+        let out = '';
+        const collect = (chunk) => void (out += chunk.toString());
+        proc.stdout.on('data', collect);
+        proc.stderr.on('data', collect);
+        await new Promise((resolve) => {
+            const timer = setTimeout(resolve, 2000);
+            proc.stdout.once('close', () => (clearTimeout(timer), resolve()));
+            proc.stdout.once('end', () => (clearTimeout(timer), resolve()));
+        });
+        proc.kill();
+        return out.trim().split('\n').slice(-lines).join(' | ').slice(0, 400);
+    }
+    catch {
+        return '';
+    }
+}
 /** Node's fetch reports every network failure as "fetch failed"; the system error that explains it is on `cause`. */
 function describeError(error) {
     const e = error;
@@ -987,8 +1016,10 @@ export async function reconcile(desiredState, ctx = defaultContext(), endpoints
                 markEvents('failed');
             }
             // Rolling services are untouched by a failure here, so keep reporting their current revision as serving.
-            if (!rolling.length)
-                throw e;
+            if (!rolling.length) {
+                // Callers turn the throw into a failed status; carry the per-service states so a held writer is still seen.
+                throw Object.assign(e instanceof Error ? e : new Error(errMessage(e)), { serviceStates });
+            }
             Object.assign(serviceStates, await activeRollingStates(dir, rolling));
             return failedStatus(errMessage(e));
         }
@@ -1099,6 +1130,11 @@ export async function pollOnce(ctx = defaultContext()) {
                 ...(desired.revision ? { observedRevision: desired.revision } : {}),
                 health: 'failed',
                 error: errMessage(e),
+                ...(e.serviceStates
+                    ? {
+                        serviceStates: e.serviceStates,
+                    }
+                    : {}),
             });
         }
     }

package/dist/backup-config.js

@@ -145,7 +145,7 @@ export function resolveBackups(compose, project = '') {
                 ? defaultSchedule.trim()
                 : undefined;
         if (spec.schedule === undefined && schedule === undefined)
-            errors.push(`${label} needs a schedule: set ${label}.schedule or x-tug.backup.schedule`);
+            errors.push(`${label} needs a schedule: set ${label}.schedule or a project-level schedule at the top-level x-tug.backup.schedule`);
         const retention = spec.retention !== undefined
             ? validateRetention(label, spec.retention, errors)
             : defaultRetention;

package/dist/backup-restore.d.ts

@@ -31,7 +31,14 @@ export interface RestoreDeps {
     clock?: () => Date;
 }
 /**
- * The services that write to a target. For a service they are the services that depend on it, transitively;
+ * True when an environment entry points at `service` over the network: a host-like variable that is exactly the
+ * service name (`WORDPRESS_DB_HOST=db`), or a value that names it as a host (`db:3306`, `mysql://u:p@db/app`,
+ * `mysql:host=db;dbname=app`). A service reached across nodes has no `depends_on` to find it by.
+ */
+export declare function environmentReaches(service: string, key: string, value: string): boolean;
+/**
+ * The services that write to a target. For a service they are the services that depend on it, transitively,
+ * plus the services whose environment names it as a host, which is how a service on another node reaches it (only for the target itself, not for what depends on the writers);
  * for a volume, the services that mount it and everything depending on those. Stored `depends_on` is used,
  * because the per-node copies drop cross-node dependencies.
  */

package/dist/backup-restore.js

@@ -8,8 +8,36 @@ const VERIFY_TIMEOUT_MS = 5 * MINUTE;
 /** Restore states in which the project must stay locked. */
 const LOCKED_STATES = ['locking', 'holding', 'safety_backup', 'restoring', 'interrupted'];
 const TERMINAL = ['succeeded', 'failed', 'released'];
+const HOST_KEY = /host|server|addr|endpoint|url|uri|dsn/i;
 /**
- * The services that write to a target. For a service they are the services that depend on it, transitively;
+ * True when an environment entry points at `service` over the network: a host-like variable that is exactly the
+ * service name (`WORDPRESS_DB_HOST=db`), or a value that names it as a host (`db:3306`, `mysql://u:p@db/app`,
+ * `mysql:host=db;dbname=app`). A service reached across nodes has no `depends_on` to find it by.
+ */
+export function environmentReaches(service, key, value) {
+    const name = service.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+    if (value === service)
+        return HOST_KEY.test(key);
+    return [
+        new RegExp(`^${name}:\\d+$`),
+        new RegExp(`(?:://|@)(?:[^/@\\s]*@)?${name}(?::\\d+)?(?:[/?\\s]|$)`),
+        new RegExp(`(?:host|server)=${name}(?:[;:,\\s]|$)`, 'i'),
+    ].some((pattern) => pattern.test(value));
+}
+function environmentOf(service) {
+    const env = service.environment;
+    if (Array.isArray(env))
+        return env.map(String).map((entry) => {
+            const at = entry.indexOf('=');
+            return at < 0 ? [entry, ''] : [entry.slice(0, at), entry.slice(at + 1)];
+        });
+    return env && typeof env === 'object'
+        ? Object.entries(env).map(([k, v]) => [k, String(v ?? '')])
+        : [];
+}
+/**
+ * The services that write to a target. For a service they are the services that depend on it, transitively,
+ * plus the services whose environment names it as a host, which is how a service on another node reaches it (only for the target itself, not for what depends on the writers);
  * for a volume, the services that mount it and everything depending on those. Stored `depends_on` is used,
  * because the per-node copies drop cross-node dependencies.
  */
@@ -34,7 +62,12 @@ export function discoverWriters(compose, target) {
     while (queue.length) {
         const current = queue.pop();
         for (const service of Object.keys(compose.services))
-            if (!writers.has(service) && service !== name && dependsOn(service).includes(current)) {
+            if (!writers.has(service) &&
+                service !== name &&
+                (dependsOn(service).includes(current) ||
+                    (kind === 'service' &&
+                        current === name &&
+                        environmentOf(compose.services[service]).some(([k, v]) => environmentReaches(name, k, v))))) {
                 writers.add(service);
                 queue.push(service);
             }
@@ -106,7 +139,7 @@ export function createRestoreService(deps) {
                     return { error: `writer ${writer} is not a service of ${request.project}`, status: 400 };
             if (!writers.length && !request.noWriters && !request.writers)
                 return {
-                    error: `no services were found that write to ${request.target}; pass --writers to name them or --no-writers to restore without holding any`,
+                    error: `no services were found that write to ${request.target} (looked at depends_on, volume mounts and host names in environment variables); pass --writers to name them or --no-writers to restore without holding any`,
                     status: 409,
                 };
         }
@@ -136,8 +169,16 @@ export function createRestoreService(deps) {
     function unlock(restore) {
         db.run('DELETE FROM locks WHERE name=? AND owner=?', `project:${restore.project_id}`, `restore:${restore.id}`);
     }
-    function release(restore) {
+    /**
+     * Drops a restore's holds and lock. Holds a deploy placed for a move (no `restore_id`) are dropped with them,
+     * unless `keepMoveHolds`: a restore that failed before changing data must not start the writers against the
+     * empty database it was meant to fill.
+     */
+    function release(restore, keepMoveHolds = false) {
         db.run('DELETE FROM service_holds WHERE restore_id=?', restore.id);
+        if (!keepMoveHolds)
+            for (const service of JSON.parse(restore.writers_json))
+                db.run('DELETE FROM service_holds WHERE project_id=? AND service=? AND restore_id IS NULL', restore.project_id, service);
         unlock(restore);
         deps.wake(restore.project_id);
     }
@@ -166,8 +207,12 @@ export function createRestoreService(deps) {
         else
             move(restore, 'holding');
         if (restore.mode === 'in-place')
-            for (const service of restorePlan.writers)
+            for (const service of restorePlan.writers) {
+                // A hold a deploy placed for a move stays as it is; this restore drops it when it succeeds.
+                if (db.get('SELECT 1 FROM service_holds WHERE project_id=? AND service=? AND restore_id IS NULL', restore.project_id, service))
+                    continue;
                 db.run('INSERT OR REPLACE INTO service_holds(project_id,service,reason,restore_id,created_at) VALUES(?,?,?,?,?)', restore.project_id, service, `restore ${restoreId}`, restoreId, ts);
+            }
         deps.wake(restore.project_id);
         advance();
         return {
@@ -189,12 +234,29 @@ export function createRestoreService(deps) {
             });
         });
     };
+    /** Says which writers never reported `held` and why: the node's own apply error, or that it has not reported. */
+    const holdDiagnosis = (restore, writers) => {
+        const desired = deployed(restore.project_id);
+        const problems = [];
+        for (const service of writers)
+            for (const node of desired?.services[service]?.placement.nodes ?? []) {
+                const row = db.get('SELECT s.* FROM node_statuses s JOIN nodes n ON n.id=s.node_id WHERE n.name=? AND s.project_id=?', node, restore.project_id);
+                const status = row ? JSON.parse(row.status_json) : undefined;
+                const state = status?.serviceStates?.[service]?.rolloutState;
+                if (state === 'held')
+                    continue;
+                problems.push(!status
+                    ? `${service} on ${node}: the node has not reported`
+                    : `${service} on ${node}: ${state ?? 'no state'}${status.error ? `; the node's apply failed with: ${status.error}` : ''}`);
+            }
+        return problems.join('; ');
+    };
     /** Fails a restore. Before anything changed, holds and the lock are dropped; after, they stay until released. */
     function fail(restore, error, dataChanged) {
         if (dataChanged)
             move(restore, 'interrupted', { error });
         else {
-            release(restore);
+            release(restore, true);
             move(restore, 'failed', { error });
         }
         event('error', 'restore.failed', restore, `restore of ${restore.target} failed: ${error}${dataChanged ? '; the writers stay held until tug restore release' : ''}`, { error, held: dataChanged });
@@ -212,8 +274,10 @@ export function createRestoreService(deps) {
     }
     function advanceHolding(restore, writers, age) {
         if (!writersReport(restore, writers, 'held')) {
-            if (age > HOLD_TIMEOUT_MS)
-                fail(restore, 'the writers did not stop in time', false);
+            if (age > HOLD_TIMEOUT_MS) {
+                const why = holdDiagnosis(restore, writers);
+                fail(restore, `the writers did not stop in time${why ? ` (${why})` : ''}`, false);
+            }
             return;
         }
         if (!restore.safety_backup)

package/dist/cli-config.d.ts

@@ -66,7 +66,7 @@ export declare function applyConfigDefaults(command: OptionSource, values: objec
  * overlay at the credential file, git-ignoring it. An existing tracked file is never rewritten, so its comments survive;
  * connection flags passed to `init` are saved to the overlay instead.
  */
-export declare function writeInitConfig(config: ResolvedCliConfig, credentialFile: string, explicit: ConnectionOptions): Promise<string[]>;
+export declare function writeInitConfig(config: ResolvedCliConfig, credentialFile: string, explicit: ConnectionOptions, credentialUrl?: string): Promise<string[]>;
 export declare function normalizeControlUrl(value: string): string;
 export declare function integerOption(value: string, name: string, minimum?: number, maximum?: number): number;
 export {};

package/dist/cli-config.js

@@ -272,11 +272,16 @@ function serialize(file, data, notice) {
  * overlay at the credential file, git-ignoring it. An existing tracked file is never rewritten, so its comments survive;
  * connection flags passed to `init` are saved to the overlay instead.
  */
-export async function writeInitConfig(config, credentialFile, explicit) {
+export async function writeInitConfig(config, credentialFile, explicit, credentialUrl) {
     const written = [];
     if (!config.fileExists) {
         await fs.mkdir(path.dirname(config.file), { recursive: true });
-        await fs.writeFile(config.file, serialize(config.file, { controlUrl: config.controlUrl, project: config.project }, `Safe to commit: keep credentials out of this file (use TUG_CREDENTIAL_FILE or ${path.basename(config.localFile)}).`));
+        await fs.writeFile(config.file, serialize(config.file, {
+            controlUrl: explicit.controlUrl
+                ? config.controlUrl
+                : (credentialUrl ?? config.controlUrl),
+            project: config.project,
+        }, `Safe to commit: keep credentials out of this file (use TUG_CREDENTIAL_FILE or ${path.basename(config.localFile)}).`));
         written.push(config.file);
     }
     let local = {};

package/dist/cli.js

@@ -14,6 +14,8 @@ import { registerInit } from "./commands/init.js";
 import { registerNode } from "./commands/node.js";
 import { registerObserve } from "./commands/observe.js";
 import { registerRestore } from "./commands/restore.js";
+import { registerHold } from "./commands/hold.js";
+import { registerImage } from "./commands/image.js";
 import { registerSecret } from "./commands/secret.js";
 import { registerService } from "./commands/service.js";
 import { registerVolume } from "./commands/volume.js";
@@ -33,6 +35,8 @@ registerNode(program);
 registerDeploy(program);
 registerObserve(program);
 registerService(program);
+registerHold(program);
+registerImage(program);
 registerSecret(program);
 registerBackup(program);
 registerRestore(program);

package/dist/commands/context.d.ts

@@ -16,6 +16,10 @@ export interface DeployResponse {
     deployment: DeploymentRecord;
     desiredState?: unknown;
 }
+/** One line for a person: which project and version reached which state, and how many services it runs. */
+export declare function formatDeployment(response: DeployResponse): string;
+/** Prints the full API result with `--json`, otherwise the human summary. */
+export declare function printResult<T>(json: boolean | undefined, value: T, summary: (value: T) => string): void;
 export interface CommonOpts {
     project?: string;
 }

package/dist/commands/context.js

@@ -8,6 +8,21 @@ import { TERMINAL_DEPLOYMENT_STATES } from "../protocol.js";
 // Templates live next to the commands' parent: src/templates in dev (`node src/cli.ts`), dist/templates in the
 // built image (`node dist/cli.js`; tools/scripts/copy-templates.js puts them there).
 export const templatesDir = path.join(path.dirname(fileURLToPath(import.meta.url)), '..', 'templates');
+/** One line for a person: which project and version reached which state, and how many services it runs. */
+export function formatDeployment(response) {
+    const { deployment } = response;
+    const state = response.desiredState;
+    const services = Object.keys(state?.services ?? {}).length;
+    const label = [state?.project, state?.version].filter(Boolean).join(' ');
+    return `${label ? `${label}: ` : ''}deployment ${deployment.id} ${deployment.state}${services ? `, ${services} service${services === 1 ? '' : 's'}` : ''}${deployment.error_message ? ` (${deployment.error_message})` : ''}`;
+}
+/** Prints the full API result with `--json`, otherwise the human summary. */
+export function printResult(json, value, summary) {
+    if (json)
+        print(value);
+    else
+        console.log(summary(value));
+}
 export const CONFIG_HELP = 'CLI config file, YAML or JSON (default: TUG_CONFIG, else tug.yml, tug.yaml, tug.json or .tug.json)';
 let resolvedConfig;
 /** Set by the root `preAction` hook before any command runs. */

package/dist/commands/control.js

@@ -98,7 +98,8 @@ export function registerControl(program) {
     control
         .command('backup')
         .description('create a control-plane backup (state, PKI, master key and retained image archives; not workload volumes)')
-        .action(async () => print(await request('POST', '/api/v1/control/backups/run', {})));
+        .option('--no-images', 'leave the retained build archives out (deploy again to rebuild them)')
+        .action(async (o) => print(await request('POST', '/api/v1/control/backups/run', { images: o.images })));
     control
         .command('restore <artifactRef>')
         .description('restore a control-plane backup on the control-plane host; requires a control restart')

package/dist/commands/deploy.d.ts

@@ -1,2 +1,6 @@
 import type { Command } from 'commander';
+/** The first conventional Compose file name in the working directory; `docker-compose.yml` when none exists. */
+export declare function defaultComposeFile(cwd?: string): string;
+/** The top-level `name:` of a Compose file, which Compose would use as the project name. */
+export declare function composeName(file: string): Promise<string | undefined>;
 export declare function registerDeploy(program: Command): void;

package/dist/commands/deploy.js

@@ -1,4 +1,4 @@
-import { openAsBlob } from 'node:fs';
+import { existsSync, openAsBlob } from 'node:fs';
 import fs from 'node:fs/promises';
 import os from 'node:os';
 import path from 'node:path';
@@ -7,7 +7,7 @@ import { canPrompt } from "../cli-prompt.js";
 import { findMissingSecrets, promptForSecrets } from "../cli-secrets.js";
 import { composeHash } from "../compose.js";
 import { normalizeCompose } from "../normalize.js";
-import { loadCfg, print, projectOf, request, waitForCommand, waitForDeployment, } from "./context.js";
+import { loadCfg, collect, formatDeployment, print, printResult, projectOf, request, waitForCommand, waitForDeployment, } from "./context.js";
 async function makeContext() {
     const cfg = await loadCfg();
     const tmp = path.join(os.tmpdir(), `tug-${Date.now()}.tgz`);
@@ -24,15 +24,27 @@ async function makeContext() {
     }
     return tmp;
 }
+/** The first conventional Compose file name in the working directory; `docker-compose.yml` when none exists. */
+export function defaultComposeFile(cwd = process.cwd()) {
+    const names = ['compose.yaml', 'compose.yml', 'docker-compose.yaml', 'docker-compose.yml'];
+    return names.find((name) => existsSync(path.join(cwd, name))) ?? 'docker-compose.yml';
+}
+/** The top-level `name:` of a Compose file, which Compose would use as the project name. */
+export async function composeName(file) {
+    const text = await fs.readFile(file, 'utf8').catch(() => '');
+    return /^name:\s*["']?([A-Za-z0-9][\w.-]*)["']?\s*(?:#.*)?$/m.exec(text)?.[1];
+}
 export function registerDeploy(program) {
     const deploy = program
         .command('deploy')
         .description('deploy the Compose project, or cancel a deployment with `deploy cancel <id>`')
-        .option('-f, --file <file>', 'compose file', 'docker-compose.yml')
+        .option('-f, --file <file>', 'compose file', defaultComposeFile())
         .option('--version <version>', 'display version (default: git-<Compose hash>)')
         .option('--build-local', 'build application images locally and upload them without source')
         .option('--no-wait', 'return after desired state is created')
         .option('--timeout <seconds>', 'wait timeout in seconds', '300')
+        .option('--json', 'print the full result, including the desired state, as JSON')
+        .option('--move <service>', 'acknowledge that a service placed on another node starts empty (its data does not move); the services that write to it are held until you restore it or run `tug hold release` (repeatable)', collect, [])
         .action(async (o) => {
         const cfg = await loadCfg();
         const project = o.project || cfg.project || 'site';
@@ -50,6 +62,10 @@ export function registerDeploy(program) {
                 secretNames.push(name);
             }
         }
+        const movePayload = o.move.length ? { moves: o.move } : {};
+        const declared = await composeName(o.file);
+        if (declared && declared !== project)
+            console.error(`warning: ${o.file} is named ${declared} but the Tug project is ${project}; secrets, backups and restores are scoped to the Tug project (set project: ${declared} in tug.yml, or pass --project)`);
         const compose = await normalizeCompose(o.file, secretNames, { project });
         const version = o.version || `git-${composeHash(compose)}`;
         let response;
@@ -67,7 +83,7 @@ export function registerDeploy(program) {
                 const localImages = await prepareLocalImages(compose, source, directory);
                 for (const { field } of localImages)
                     form.append(field, await openAsBlob(path.join(directory, `${field}.tar.gz`)), `${field}.tar.gz`);
-                form.append('payload', JSON.stringify({ project, version, compose, localImages }));
+                form.append('payload', JSON.stringify({ project, version, compose, localImages, ...movePayload }));
                 response = await request('POST', '/api/v1/deployments', form, true);
             }
             finally {
@@ -78,7 +94,7 @@ export function registerDeploy(program) {
             const tarball = await makeContext();
             try {
                 const form = new FormData();
-                form.append('payload', JSON.stringify({ project, version, compose }));
+                form.append('payload', JSON.stringify({ project, version, compose, ...movePayload }));
                 form.append('context', await openAsBlob(tarball), 'context.tgz');
                 response = await request('POST', '/api/v1/deployments', form, true);
             }
@@ -91,10 +107,11 @@ export function registerDeploy(program) {
                 project,
                 version,
                 compose,
+                ...movePayload,
             });
         if (o.wait)
             response.deployment = await waitForDeployment(response.deployment.id, Number(o.timeout));
-        print(response);
+        printResult(o.json, response, formatDeployment);
     });
     deploy
         .command('cancel <id>')
@@ -106,6 +123,7 @@ export function registerDeploy(program) {
         .option('--force')
         .option('--no-wait', 'return immediately after activating rollback')
         .option('--timeout <seconds>', 'wait timeout', '300')
+        .option('--json', 'print the full result, including the desired state, as JSON')
         .action(async (o) => {
         const response = await request('POST', '/api/v1/rollback', {
             project: await projectOf(o),
@@ -113,16 +131,18 @@ export function registerDeploy(program) {
         });
         if (o.wait)
             response.deployment = await waitForDeployment(response.deployment.id, Number(o.timeout));
-        print(response);
+        printResult(o.json, response, formatDeployment);
     });
     program
         .command('scale <service> <replicas>')
         .description('scale a service to N replicas across matching nodes')
+        .option('--json', 'print the full result as JSON')
         .action(async (service, replicas, o) => {
         const project = await projectOf(o);
         if (!/^[1-9]\d*$/.test(replicas))
             throw new Error('replicas must be a positive integer');
-        print(await request('POST', '/api/v1/scale', { project, service, replicas: Number(replicas) }));
+        const result = await request('POST', '/api/v1/scale', { project, service, replicas: Number(replicas) });
+        printResult(o.json, result, (r) => `${r.service} scaled to ${r.replicas} replica${r.replicas === 1 ? '' : 's'} on ${r.nodes.join(', ')}`);
     });
     program
         .command('restart <service>')

package/dist/commands/hold.d.ts

@@ -0,0 +1,2 @@
+import type { Command } from 'commander';
+export declare function registerHold(program: Command): void;

package/dist/commands/hold.js

@@ -0,0 +1,13 @@
+import { print, projectOf, request } from "./context.js";
+export function registerHold(program) {
+    const hold = program
+        .command('hold')
+        .description('services a deploy stopped because their data moved to another node');
+    hold
+        .command('release [services...]')
+        .description('start the services that `tug deploy --move` held, without restoring (all of them when none is named); a restore releases its own holds')
+        .action(async (services, o) => print(await request('POST', '/api/v1/holds/release', {
+        project: await projectOf(o),
+        services,
+    })));
+}

package/dist/commands/host.d.ts

@@ -16,3 +16,9 @@ export declare function hostHasDocker(target: string, compose?: boolean): boolea
 export declare function remoteHostname(target: string): string | undefined;
 /** The host's global IPv4 addresses, private ones first; empty when the host cannot say. */
 export declare function discoverAddresses(target: string): AddressCandidate[];
+/**
+ * The script that retires a node host: stops the agent and ingress, removes the project containers, networks and
+ * agent state. Project volumes (your data) and the ingress certificates go only with `deleteData`. A host that runs
+ * the control plane is refused, because removing it would destroy the master key and the private CA.
+ */
+export declare function teardownScript(deleteData: boolean): string;

package/dist/commands/host.js

@@ -97,3 +97,26 @@ export function discoverAddresses(target) {
     const result = spawnSync('ssh', ['--', target, LIST_ADDRESSES_COMMAND], { encoding: 'utf8' });
     return result.status === 0 ? sortCandidates(parseAddressCandidates(String(result.stdout))) : [];
 }
+/**
+ * The script that retires a node host: stops the agent and ingress, removes the project containers, networks and
+ * agent state. Project volumes (your data) and the ingress certificates go only with `deleteData`. A host that runs
+ * the control plane is refused, because removing it would destroy the master key and the private CA.
+ */
+export function teardownScript(deleteData) {
+    return [
+        'set -eu',
+        "if docker ps -a --format '{{.Names}}' | grep -q '^tug-control'; then echo 'this host runs the control plane; retire it by hand (see docs/migration.md)' >&2; exit 3; fi",
+        'docker rm -f tug-agent tug-caddy >/dev/null 2>&1 || true',
+        'ids=$(docker ps -aq --filter label=tug.project --filter label=tug.job; docker ps -aq --filter label=tug.project)',
+        '[ -z "$ids" ] || docker rm -fv $ids >/dev/null',
+        "docker network ls -q --filter 'name=^tug_' | xargs -r docker network rm >/dev/null 2>&1 || true",
+        'rm -rf /etc/tug /var/lib/tug',
+        ...(deleteData
+            ? [
+                'docker volume ls -q --filter label=tug.project | xargs -r docker volume rm >/dev/null',
+                'docker volume rm tug-caddy-data tug-caddy-config tug-caddy-etc tug_restic_cache >/dev/null 2>&1 || true',
+            ]
+            : []),
+        'echo "node host cleaned"',
+    ].join('\n');
+}

package/dist/commands/image.d.ts

@@ -0,0 +1,2 @@
+import type { Command } from 'commander';
+export declare function registerImage(program: Command): void;

package/dist/commands/image.js

@@ -0,0 +1,13 @@
+import { print, request } from "./context.js";
+export function registerImage(program) {
+    const image = program.command('image').description('manage the build archives kept on control');
+    image
+        .command('prune')
+        .description('delete build archives that rollback no longer needs: all but the newest N deployments of each project, the current one and any version a node still runs')
+        .option('--keep <n>', 'deployments per project whose archives are kept', '5')
+        .option('--dry-run', 'list what would be deleted without deleting it')
+        .action(async (o) => print(await request('POST', '/api/v1/images/prune', {
+        keep: Number(o.keep),
+        dryRun: Boolean(o.dryRun),
+    })));
+}

package/dist/commands/init.js

@@ -15,7 +15,7 @@ export async function importCredential(cfg, text, connection) {
         path.join(process.env.XDG_CONFIG_HOME || path.join(os.homedir(), '.config'), 'tug', 'credentials');
     const source = path.join(dir, `${new URL(credential.url).host.replace(/[^A-Za-z0-9.-]/g, '_')}.json`);
     await writeFileDurable(source, `${JSON.stringify(credential, null, 2)}\n`, 0o600);
-    for (const file of await writeInitConfig(cfg, source, connection))
+    for (const file of await writeInitConfig(cfg, source, connection, credential.url))
         console.log(`wrote ${file}`);
     console.log(`imported credential to ${source}`);
 }

package/dist/commands/node.js

@@ -7,7 +7,7 @@ import { isNodeHost } from "../node-address.js";
 import { resolveImage } from "../image.js";
 import { errMessage, loadTemplate } from "../util.js";
 import { collect, loadCfg, print, readStdin, request, runCapture, runLocal, templatesDir, transportFor, } from "./context.js";
-import { bootstrapDocker, discoverAddresses, remoteHostname, dockerBootstrap, hostHasDocker, preflightRemote, transferTugImage, } from "./host.js";
+import { bootstrapDocker, discoverAddresses, remoteHostname, dockerBootstrap, hostHasDocker, preflightRemote, teardownScript, transferTugImage, } from "./host.js";
 import fs from 'node:fs/promises';
 const AGENT_STATE = '/var/lib/tug';
 function addNodeIdentityOptions(command) {
@@ -306,10 +306,44 @@ export function registerNode(program) {
         .description("block a node's key immediately without deleting its metadata or placement (no SSH needed); bring it back with node replace")
         .action(async (name) => print(await request('POST', `/api/v1/nodes/${encodeURIComponent(name)}/disable`, {})));
     node
-        .command('remove <name>')
-        .description('remove a retired node from the control plane (its identity, statuses and queued commands); stop its agent and remove its containers on the host yourself')
+        .command('remove <name> [sshTarget]')
+        .description('remove a retired node from the control plane (its identity, statuses and queued commands); with --teardown also stop its agent and delete its containers and agent state over SSH')
         .option('--force', 'remove even if the node is still reporting')
-        .action(async (name, o) => print(await request('DELETE', `/api/v1/nodes/${encodeURIComponent(name)}${o.force ? '?force=true' : ''}`)));
+        .option('--teardown', 'also clean the host over SSH (the SSH target defaults to nodes.<name>.ssh in the config file)')
+        .option('--delete-data', 'with --teardown, also delete project volumes and ingress certificates')
+        .option('-y, --yes', 'with --teardown, do not ask for confirmation')
+        .action(async (name, ssh, o) => {
+        if (!o.teardown) {
+            if (ssh || o.deleteData || o.yes)
+                throw new Error('an SSH target, --delete-data and --yes need --teardown');
+            return print(await request('DELETE', `/api/v1/nodes/${encodeURIComponent(name)}${o.force ? '?force=true' : ''}`));
+        }
+        const cfg = await loadCfg();
+        const sshTarget = ssh ?? nodeSettings(cfg, name).ssh;
+        if (!sshTarget)
+            throw new Error(`--teardown needs an SSH target: pass <ssh-target> or set nodes.${name}.ssh in ${cfg.file}`);
+        if (!o.yes) {
+            if (!canPrompt())
+                throw new Error('--teardown deletes data on the host; pass -y to confirm without a terminal');
+            console.error(`This will stop the Tug agent on ${sshTarget}, remove its Tug containers and networks and delete /etc/tug and /var/lib/tug${o.deleteData
+                ? ', and DELETE every project volume and the ingress certificates'
+                : ' (project volumes are kept)'}.`);
+            if (!(await confirm('Continue?', false)))
+                throw new Error('cancelled');
+        }
+        // The control record goes first: it refuses a node that is still placed in a current deployment before any
+        // host is touched. A node control no longer knows is cleaned anyway, so a failed cleanup can be retried.
+        let removed = { ok: true, node: name, alreadyRemoved: true };
+        try {
+            removed = await request('DELETE', `/api/v1/nodes/${encodeURIComponent(name)}?force=true`);
+        }
+        catch (error) {
+            if (!/node not found/.test(errMessage(error)))
+                throw error;
+        }
+        runLocal('ssh', ['--', sshTarget, teardownScript(Boolean(o.deleteData))]);
+        print({ ...removed, teardown: sshTarget, deleteData: Boolean(o.deleteData) });
+    });
     node
         .command('ingress <name>')
         .description('mark a node as an ingress node (managed Caddy on ports 80/443), or remove the role with --disable')

package/dist/control.js

@@ -29,7 +29,7 @@ import { parseStatusReport } from "./status-report.js";
 import { ACTIVE_DEPLOYMENT_STATES, LIVE_MAX_FRAME_BYTES, LIVE_MAX_BUFFERED_BYTES, LIVE_MAX_SESSIONS, LIVE_START_TIMEOUT_MS, BACKUP_COMMAND_TYPES, } from "./protocol.js";
 import { BACKUP_SECRET_PROJECT, isReservedProject, parseBackupDestination, projectRepository, repositoryPassword, redactSecrets, } from "./backup-destination.js";
 import { createBackupService } from "./backup-scheduler.js";
-import { createRestoreService } from "./backup-restore.js";
+import { createRestoreService, discoverWriters } from "./backup-restore.js";
 import { AUTO_ROLLBACK_ACTOR, ROUTE_ROLES, PROJECT_NAME_PATTERN, authenticate, hasRole, assertProject, bodyProject, createGuard, createCredential, listCredentials, parseCredentialRequest, queryProject, recordEvent, revokeCredential, } from "./credentials.js";
 class WakeHub {
     cursors = new Map();
@@ -758,6 +758,7 @@ export async function createApp(options = {}) {
         WHERE ds.project_id=? AND ds.deployment_id != ? AND d.state IN (${states}) ORDER BY ds.created_at DESC LIMIT 1`, project, current?.current_deployment_id || '');
             if (!previousDs)
                 throw httpError(404, 'no previous deployment found');
+            assertArtifactsAvailable(JSON.parse(previousDs.state_json), `roll back to ${previousDs.version}`);
             const ts = now();
             const depId = id('dep');
             const previousState = JSON.parse(previousDs.state_json);
@@ -934,10 +935,17 @@ export async function createApp(options = {}) {
         const global = ext.global === true;
         const labels = Array.isArray(placement.labels) ? placement.labels : [];
         const matched = nodeInfos.filter((node) => labels.every((label) => node.labels.includes(label)));
-        const pool = labels.length ? matched : nodeInfos;
+        // Explicit `placement.nodes` (without labels or global, which deploy resolves into nodes) bound the pool; scaling never re-homes a service off them.
+        const pinned = !labels.length && !global && Array.isArray(placement.nodes)
+            ? placement.nodes
+            : [];
+        const candidates = pinned.length
+            ? matched.filter((node) => pinned.includes(node.name))
+            : matched;
+        const pool = candidates;
         if (!pool.length)
-            throw httpError(400, labels.length
-                ? 'no healthy nodes match the service placement labels'
+            throw httpError(400, labels.length || pinned.length
+                ? 'no healthy nodes match the service placement'
                 : 'no nodes available for scaling');
         const selected = global ? pool : pool.slice(0, Math.min(replicas, pool.length));
         state.services[service].placement = {
@@ -1536,7 +1544,70 @@ export async function createApp(options = {}) {
     }
     // Previous-process builds cannot finish after a restart; clear their unpublished files.
     db.run("UPDATE deployments SET state='build_failed',error_message='control restarted during build' WHERE state IN ('uploading','building')");
+    /** Archives deleted by `image prune`, or missing from a backup taken without images, can no longer be deployed. */
+    async function markMissingArtifacts() {
+        for (const row of db.all("SELECT * FROM builds WHERE state='complete' AND artifact_path IS NOT NULL"))
+            if (!(await fs.stat(path.join(DATA_DIR, row.artifact_path)).catch(() => null)))
+                db.run("UPDATE builds SET state='pruned',artifact_path=NULL,updated_at=? WHERE id=?", now(), row.id);
+    }
+    /** Throws a 409 when a desired state needs a build archive that was pruned. */
+    function assertArtifactsAvailable(state, action) {
+        for (const [name, service] of Object.entries(state.services)) {
+            if (!service.artifact)
+                continue;
+            const build = db.get('SELECT * FROM builds WHERE id=?', service.artifact.buildId);
+            if (build?.state === 'pruned')
+                throw httpError(409, `cannot ${action}: the build archive of ${name} was pruned (tug image prune); deploy it again instead`);
+        }
+    }
+    /**
+     * Deletes build archives no longer needed: everything except the newest `keep` desired states of each project,
+     * the current deployment and any version a node still reports. The desired states stay, so history lists them,
+     * but rolling back to a pruned one is refused.
+     */
+    async function pruneArtifacts(keep, dryRun) {
+        const kept = new Set();
+        const reported = db
+            .all('SELECT project_id,status_json FROM node_statuses')
+            .map((row) => ({
+            project: row.project_id,
+            ...JSON.parse(row.status_json),
+        }));
+        const current = new Set(db
+            .all('SELECT current_deployment_id FROM projects')
+            .map((row) => row.current_deployment_id));
+        const rows = db.all('SELECT * FROM desired_states ORDER BY created_at DESC');
+        const seen = new Map();
+        for (const row of rows) {
+            const index = seen.get(row.project_id) ?? 0;
+            seen.set(row.project_id, index + 1);
+            const state = JSON.parse(row.state_json);
+            const lagging = reported.some((r) => r.project === row.project_id &&
+                r.observedVersion === state.version &&
+                (!r.observedRevision || r.observedRevision === state.revision));
+            if (index >= keep && !lagging && !current.has(row.deployment_id))
+                continue;
+            for (const service of Object.values(state.services))
+                if (service.artifact)
+                    kept.add(service.artifact.buildId);
+        }
+        const pruned = [];
+        for (const build of db.all("SELECT * FROM builds WHERE state='complete' AND artifact_path IS NOT NULL")) {
+            if (kept.has(build.id))
+                continue;
+            pruned.push({ id: build.id, service: build.service, sizeBytes: build.size_bytes });
+            if (dryRun)
+                continue;
+            await fs.rm(path.join(DATA_DIR, build.artifact_path), { force: true });
+            db.run("UPDATE builds SET state='pruned',artifact_path=NULL,updated_at=? WHERE id=?", now(), build.id);
+            const shared = db.get("SELECT 1 FROM builds WHERE image_ref=? AND state='complete'", build.image_ref);
+            if (!shared)
+                await images.run('docker', ['image', 'rm', build.image_ref]).catch(() => { });
+        }
+        return { pruned, freedBytes: pruned.reduce((sum, build) => sum + build.sizeBytes, 0) };
+    }
     await collectArtifacts();
+    await markMissingArtifacts();
     const receiveDeployment = upload.any();
     api('post', '/api/v1/deployments', null, (req, res) => {
         receiveDeployment(req, res, (error) => {
@@ -1547,6 +1618,25 @@ export async function createApp(options = {}) {
             });
         });
     });
+    /**
+     * Services that keep data on a node and are placed on different nodes by `compose` than by the current
+     * deployment. Deploying such a change starts the service empty on the new node: the data does not move.
+     */
+    function movedServices(project, compose) {
+        const current = currentDesired(project);
+        if (!current)
+            return [];
+        const before = JSON.parse(current.state_json);
+        return Object.entries(compose.services || {})
+            .filter(([name, svc]) => {
+            const old = before.services?.[name]?.placement?.nodes ?? [];
+            const next = svc['x-tug']?.placement?.nodes ?? [];
+            const keepsData = Boolean(svc['x-tug']?.backup) ||
+                (svc.volumes ?? []).some((mount) => typeof mount === 'object' && mount.type === 'volume');
+            return keepsData && old.length > 0 && next.length > 0 && old.some((n) => !next.includes(n));
+        })
+            .map(([name]) => name);
+    }
     async function deployRequest(req, res, uploadError) {
         const files = (Array.isArray(req.files) ? req.files : []);
         let responseBody;
@@ -1620,6 +1710,10 @@ export async function createApp(options = {}) {
                 throw httpError(400, errMessage(e));
             }
             assertIngressHostsFree(project, compose);
+            const acknowledgedMoves = Array.isArray(body.moves) ? body.moves.map(String) : [];
+            const unacknowledged = movedServices(project, compose).filter((name) => !acknowledgedMoves.includes(name));
+            if (unacknowledged.length)
+                throw httpError(409, `${unacknowledged.join(', ')} would start empty on a different node: its data does not move. Back it up, then deploy with --move ${unacknowledged.join(' --move ')} to hold the services that write to it, and run tug restore run <target> --confirm`);
             // The CLI interpolates control secrets as sentinels; a reference to a secret that is gone would deploy an empty value.
             const referenced = sentinelNames(compose);
             if (referenced.includes(REGISTRY_AUTH_SECRET))
@@ -1770,6 +1864,11 @@ export async function createApp(options = {}) {
                         !db.get('SELECT 1 FROM locks WHERE name=? AND owner=? AND expires_at>?', buildLock.name, buildLock.owner, now()))
                         throw httpError(409, 'deployment was cancelled or its build lease was lost');
                     db.run('INSERT INTO desired_states(id,project_id,deployment_id,version,state_json,created_at) VALUES(?,?,?,?,?,?)', id('ds'), project, publishingId, version, JSON.stringify(desired), ts);
+                    // The moved service starts empty: keep what writes to it stopped until its data is restored.
+                    for (const moved of acknowledgedMoves.filter((name) => compose.services?.[name]))
+                        for (const writer of discoverWriters(compose, `service:${moved}`))
+                            if (!acknowledgedMoves.includes(writer))
+                                db.run('INSERT OR REPLACE INTO service_holds(project_id,service,reason,restore_id,created_at) VALUES(?,?,?,?,?)', project, writer, `moved ${moved}`, null, ts);
                     for (const [volumeName, volumeSpec] of Object.entries(compose.volumes || {})) {
                         const lp = volumeSpec?.['x-tug'] || {};
                         const nodeName = lp.placement?.nodes?.[0] || null;
@@ -1933,6 +2032,31 @@ export async function createApp(options = {}) {
             releaseLock(lock);
         }
     });
+    api('post', '/api/v1/holds/release', bodyProject, async (req, res) => withWriter(async () => {
+        const project = String(req.body?.project || 'site');
+        const services = Array.isArray(req.body?.services)
+            ? req.body.services.map(String)
+            : [];
+        // Only holds a deploy placed for a move; a restore's holds are dropped by `restore release`.
+        const rows = db.all('SELECT service FROM service_holds WHERE project_id=? AND restore_id IS NULL', project);
+        const released = rows
+            .map((row) => row.service)
+            .filter((service) => !services.length || services.includes(service));
+        for (const service of released)
+            db.run('DELETE FROM service_holds WHERE project_id=? AND service=? AND restore_id IS NULL', project, service);
+        wakeAll(project);
+        res.json({ ok: true, project, released });
+    }));
+    api('post', '/api/v1/images/prune', null, async (req, res) => withWriter(async () => {
+        const keep = req.body?.keep === undefined ? 5 : Number(req.body.keep);
+        if (!Number.isInteger(keep) || keep < 1)
+            return res.status(400).json({ error: 'keep must be a positive integer' });
+        res.json({
+            dryRun: Boolean(req.body?.dryRun),
+            keep,
+            ...(await pruneArtifacts(keep, Boolean(req.body?.dryRun))),
+        });
+    }));
     api('post', '/api/v1/control/backups/run', null, async (req, res) => withWriter(async () => {
         const backupDir = process.env.TUG_CONTROL_BACKUP_DIR || path.join(DATA_DIR, 'control-backups');
         // The archive holds the master key and the CA key, so nothing here is ever readable beyond the control user.
@@ -1954,14 +2078,16 @@ export async function createApp(options = {}) {
                 recursive: true,
                 filter: (src) => path.basename(src) !== 'server.json',
             });
-            await fs.cp(artifactDir, path.join(staging, 'images'), { recursive: true });
-            await fs.writeFile(path.join(staging, 'metadata.json'), JSON.stringify({ backupId, createdAt: ts, type: 'control' }, null, 2), { mode: 0o600 });
+            const withImages = req.body?.images !== false;
+            if (withImages)
+                await fs.cp(artifactDir, path.join(staging, 'images'), { recursive: true });
+            await fs.writeFile(path.join(staging, 'metadata.json'), JSON.stringify({ backupId, createdAt: ts, type: 'control', images: withImages }, null, 2), { mode: 0o600 });
             await tar.c({ gzip: true, file: artifactRef, cwd: staging, mode: 0o600 }, [
                 'tug.db',
                 'master.key',
                 'metadata.json',
                 'pki',
-                'images',
+                ...(withImages ? ['images'] : []),
             ]);
         }
         catch (error) {
@@ -2007,11 +2133,16 @@ export async function createApp(options = {}) {
             for (const secret of snapshot.all('SELECT * FROM secrets'))
                 decrypt(secret.encrypted_value, key);
             const states = snapshot.all('SELECT * FROM desired_states');
-            for (const stateRow of states) {
+            const meta = JSON.parse(await fs.readFile(path.join(restoreDir, 'metadata.json'), 'utf8').catch(() => '{}'));
+            // A backup taken without images restores the archive-less builds as pruned on the next start.
+            for (const stateRow of meta.images === false ? [] : states) {
                 const state = JSON.parse(stateRow.state_json);
                 for (const service of Object.values(state.services)) {
                     if (!service.artifact)
                         continue;
+                    // `image prune` already removed this one before the backup.
+                    if (snapshot.get("SELECT 1 FROM builds WHERE id=? AND state='pruned'", service.artifact.buildId))
+                        continue;
                     const build = snapshot.get('SELECT * FROM builds WHERE id=? AND state=?', service.artifact.buildId, 'complete');
                     if (!build?.artifact_path ||
                         !/^images\/build_[a-zA-Z0-9_-]+\.tar\.gz$/.test(build.artifact_path))

package/dist/credentials.js

@@ -53,6 +53,8 @@ export const ROUTE_ROLES = {
     'DELETE /api/v1/backup/destination': 'admin',
     'GET /api/v1/backup/repository-password': 'admin',
     'POST /api/v1/control/backups/run': 'admin',
+    'POST /api/v1/images/prune': 'admin',
+    'POST /api/v1/holds/release': 'admin',
     'POST /api/v1/control/restore': 'admin',
     'GET /api/v1/diagnostics': 'admin',
     'POST /api/v1/diagnostics/nodes': 'admin',

package/dist/docker-api.d.ts

@@ -40,6 +40,7 @@ export interface ContainerInspect {
         NetworkMode?: string;
         Binds?: string[];
         Mounts?: unknown[];
+        ExtraHosts?: string[];
     };
     State?: {
         Status?: string;

package/dist/live.js

@@ -312,6 +312,7 @@ export async function startOneOff(docker, template, command) {
             NetworkMode: host.NetworkMode,
             ...(host.Binds?.length ? { Binds: host.Binds } : {}),
             ...(host.Mounts?.length ? { Mounts: host.Mounts } : {}),
+            ...(host.ExtraHosts?.length ? { ExtraHosts: host.ExtraHosts } : {}),
         },
     });
     const remove = () => docker.removeContainer(id).catch(() => { });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wpelevator/tug",
-  "version": "0.11.0",
+  "version": "0.12.0",
   "description": "Portable VPS deployments with Docker Compose",
   "repository": {
     "type": "git",