@wpelevator/tug 0.11.0 → 0.13.0

package/README.md

@@ -1,12 +1,16 @@
 # Tug
 
-Deploy Docker Compose apps to a few ordinary Linux servers, from your laptop or CI, without Kubernetes or a hosted platform.
+Deploy Docker Compose apps to your own Linux VPS, from your laptop or CI, without Kubernetes or a hosted platform.
 
-You keep your `docker-compose.yml`. Tug installs a small control plane on one server and an agent on each of the others, then `tug deploy` puts your services where you say, waits until they are healthy and rolls back if they are not. It is built for running one to a few dozen sites, WordPress first, on two to ten VPSes, with backups and restores included.
+A VPS is a durable computer with enough resources to host multiple sites or a medium-size application and its database. Tug helps you use that capacity: keep your Compose file, deploy with one command, and manage rollouts, secrets, backups and restores. Start with one server and add more when you need them. Tug is built for a small number of servers and Compose applications, including Node.js, Laravel, Ruby and WordPress apps.
+
+Tug runs a small control plane and an agent, which can share the same VPS with your applications. It builds application images on the control server or your machine and delivers them to your nodes, so you do not need to run or configure an image registry for those builds. Published images are still pulled from their upstream registries.
+
+Existing service and resource targets can be chosen directly in an interactive terminal: try `tug logs`, `tug deploy cancel` or `tug restore run`. Use `--select` for backup and move-hold subsets; see [interactive CLI selection](docs/cli.md#interactive-selection) for all supported commands and scripting behavior.
 
 ## What you can do with it
 
-- **Deploy a Compose file to your servers.** `tug deploy` builds or pulls your images, starts them on the right nodes, waits for health checks and tells you why if something fails. `tug rollback` returns to the previous version, and `x-tug: { rollback: auto }` does it for you.
+- **Deploy a Compose file to your servers.** `tug deploy` builds or pulls your images, starts them on the right nodes, waits for health checks and tells you why if something fails. It reports deployment stages and elapsed time while work is pending. `tug rollback` returns to the previous version, and `x-tug: { rollback: auto }` does it for you.
 - **Choose where things run.** Pin a database and its volume to one node, spread web containers over several, and let Tug's managed Caddy ingress serve them over HTTPS with certificates it obtains for you.
 - **Update without downtime.** Opt-in rolling rollouts start the new version, wait until it is healthy and only then switch traffic.
 - **Keep secrets out of your repository.** `tug secret set DB_PASSWORD` stores them encrypted and sends each one only to the nodes whose services use it.
@@ -14,48 +18,75 @@ You keep your `docker-compose.yml`. Tug installs a small control plane on one se
 - **Look inside running services.** `tug logs`, `tug exec`, `tug run` (for example `wp` CLI), `tug ps` and `tug status` work through the control plane, so you never SSH to a node to see what is going on.
 - **Give CI a limited key.** `tug credential create` makes read-only, deploy-only or admin credentials, and every change records who made it.
 
-Nodes only dial out to the control plane, so they work behind NAT or a firewall that allows nothing inbound. Tug does not need a cloud provider's API, a registry or an overlay network.
+Agents only dial out to the control plane, so nodes can work behind NAT or a firewall that allows nothing inbound. Tug does not need a cloud provider's API or an overlay network.
 
 ## Try it
 
-You need Node.js 22.18 or newer and Docker Compose v2 on your machine, and SSH access to one or more Linux servers (Debian, Ubuntu, Fedora or a RHEL-compatible distribution). Tug can install Docker on them for you.
+Follow [Deploy your first service](docs/getting-started.md) to install Tug, deploy nginx to one VPS and verify that it works. You need Node.js 22.18 or newer and Docker Compose v2 on your computer, plus SSH access to a Linux server. Tug can install Docker on supported servers.
 
 ```bash
 npm install -g @wpelevator/tug      # or run it with npx @wpelevator/tug
 
-tug setup root@203.0.113.5          # install the control plane on a server
-tug node add root@198.51.100.7      # add a server that will run your apps
-tug deploy                          # deploy ./docker-compose.yml
+tug setup root@203.0.113.5          # install the control plane
+tug node add root@203.0.113.5 --name app1  # install the agent on the same VPS
+tug deploy                          # deploy a Compose file in this directory
 ```
 
+The control plane and agent share the VPS with your application. Once your service is running, you can [add HTTPS](docs/networking.md#layer-b-ingress-and-tls), [back up your data](docs/backups.md) or [add another server](docs/service-placement.md).
+
 In a terminal Tug asks before it changes anything: whether to install Docker, which private address the node should use and whether it should handle ingress. It shows the plan and waits for you to confirm. `tug deploy` asks for any Compose variables, such as `${DB_PASSWORD}`, that no secret or `.env` provides. In CI or any other script nothing is ever asked, so pass the choices as flags (`--address`, `--ingress`, `--install-docker`, `-y`).
 
+Long-running deployments, rollbacks, setup, image transfers, backups, restores and agent command waits report their current stage and elapsed time to stderr every ten seconds. Progress also works in CI and keeps JSON results separate. See [CLI progress reporting](docs/cli.md).
+
 ## Your Compose file, plus a little `x-tug`
 
-A normal Compose file already works. Add `x-tug` to say where services run, how to tell they are healthy and where traffic comes in:
+This example builds on the [quickstart](docs/getting-started.md). It pins the app, database and database volume to `app1`. Placement can be omitted with one node, but explicit placement keeps the database on its chosen server when you add more nodes. Enable ingress with `tug node ingress app1`, point the hostname's DNS at the VPS, and allow public ports 80 and 443. Save this as `compose.yaml` and replace the hostname with yours:
 
 ```yaml
 services:
   db:
     image: mariadb:11.4.8
-    volumes: [db-data:/var/lib/mysql]
+    volumes:
+      - db-data:/var/lib/mysql
     environment:
       MARIADB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
     x-tug:
-      placement: { nodes: [app1] }
-      health: { type: tcp, port: 3306 }
+      placement:
+        nodes:
+          - app1
+      health:
+        type: tcp
+        port: 3306
 
   wordpress:
     image: wordpress:6.8.3-apache
-    depends_on: [db]
+    depends_on:
+      - db
+    environment:
+      WORDPRESS_DB_HOST: db
+      WORDPRESS_DB_USER: root
+      WORDPRESS_DB_PASSWORD: ${DB_ROOT_PASSWORD}
+    ports:
+      - "8080:80"
     x-tug:
-      placement: { nodes: [app1] }
-      health: { type: http, port: 80, path: /wp-includes/version.php }
-      ingress: { host: blog.example.com, port: 80, tls: auto }
+      placement:
+        nodes:
+          - app1
+      health:
+        type: http
+        port: 80
+        path: /wp-includes/version.php
+      ingress:
+        host: blog.example.com
+        port: 80
+        tls: auto
 
 volumes:
   db-data:
-    x-tug: { placement: { nodes: [app1] } }
+    x-tug:
+      placement:
+        nodes:
+          - app1
 ```
 
 ```bash
@@ -65,7 +96,9 @@ tug status       # what is running where, and anything failing
 tug logs wordpress --follow
 ```
 
-A complete WordPress site with scheduled database and uploads backups is in the [WordPress example](examples/wordpress/README.md). Everything `x-tug` can express is in [Compose extensions](docs/compose.md).
+The named volume holds data on `app1`; it is not copied to another node automatically. Back it up and restore or move its data deliberately if you replace the server.
+
+See the [examples directory](examples/) for stack-specific applications and [local stack benchmarks](docs/testing.md#example-stack-benchmarks) for measured behavior of those examples. Everything `x-tug` can express is in [Compose extensions](docs/compose.md).
 
 ## Everyday commands
 
@@ -89,68 +122,77 @@ Every command and option is in the [CLI reference](docs/cli.md). `tug status --j
 
 ## Is Tug right for you?
 
-Tug fits if you run a handful of Compose apps on VPSes you already pay for, want deploys, rollbacks, secrets and backups handled in one tool, and are happy with a single control server that you back up.
+Tug fits if you run Compose apps on one or a few VPSes, want deploys, rollbacks, secrets and backups handled in one tool, and are happy with a single control server that you back up. A server can host multiple apps as long as it has enough capacity; Tug does not automatically balance resource usage or replace a failed server.
 
 It deliberately does not do Kubernetes-style scheduling, autoscaling, distributed storage or database failover, and it does not provision servers through a provider's API. Node-to-node traffic travels over your own private network (Tailscale, a VPC, a LAN), so you own its firewalling and encryption. If you want automatic rescheduling of containers across a cluster, see how it compares with [Docker Swarm](docs/swarm-comparison.md). The goals, principles and planned milestones are in the [Roadmap](docs/roadmap.md).
 
 ## How it works
 
 ```text
-  tug CLI / CI (operator credential)
-       |
-       |  mTLS: deploy, status, logs, exec
-       v
-+-------------------------------------+
-| Control plane (one mTLS port)       |
-|   Control API (role-checked)        |
-|   SQLite: desired states, secrets   |
-|   private CA, image builds          |
-+-------------------------------------+
-       ^                       ^
-       |  agents dial out: work, status,
-       |  WebSocket wake and live sessions
-       |                       |
-+------+------------+   +------+------------------+
-| Node app1         |   | Node app2 (ingress node)|
-|   Agent           |   |   Agent                 |
-|   Docker Engine   |   |   Docker Engine         |
-|   services        |   |   services              |
-|                   |   |   managed Caddy ingress |
-+-------------------+   +-------------------------+
-        \_________________________/
-          workload traffic between node addresses
-          over the operator's private network
-
-  SSH is used only by the CLI to install the control plane and nodes.
+                      tug CLI / CI
+                           |
+                           | mTLS: deploy, status, logs, exec
+                           v
++--------------------------------+     +--------------------------------+
+| VPS app1                       |     | VPS app2 (optional)            |
+|                                |     |                                |
+| Control plane                  |     |                                |
+| API, SQLite, image builds      |     |                                |
+|                                |     |                                |
+| Agent --> Docker Engine        |     | Agent --> Docker Engine        |
+|              |                 |     |              |                 |
+|              +-- service A     |     |              +-- service C     |
+|              +-- database      |     |              +-- service D     |
+|              +-- more...       |     |              +-- more...       |
+|     (zero or more services)    |     |     (zero or more services)    |
+|                                |     |                                |
+| Local volumes, optional Caddy  |     | Local volumes, optional Caddy  |
++--------------------------------+     +--------------------------------+
+             |                                         |
+             +------- private workload network --------+
+
+  Each agent connects to the control plane over mTLS. Services are
+  placed on either node; local volumes stay on their node. app1 can
+  run alone. SSH installs control and agents; normal operations use
+  the control API.
 ```
 
-`tug deploy` reads your Compose file on your machine and sends it to the control plane, which stores it as an immutable desired state, builds any images and pins published images to digests. Each agent fetches the work for its node, applies it through the Docker Engine API, runs the health checks and reports back. The control plane marks the deployment done when every node reports it serving. Compose runs only on your machine; nodes need just Docker.
+`tug deploy` reads your Compose file on your machine and sends it to the control plane, which stores it as an immutable desired state, builds application images when needed and pins published images to digests. Tug delivers its built images as archives; published images are pulled from their upstream registries. Each agent fetches the work for its node, applies it through the Docker Engine API, runs the health checks and reports back. The control plane marks the deployment done when every node reports it serving. Compose runs only on your machine for application normalization; nodes need Docker, and the control installation also uses Compose. Cross-node workload traffic uses your own private network.
 
 Control serves mutual TLS directly on one port (`8443`). Every operator, CI job and node authenticates with a certificate from a private CA created at install, so the control address can be a bare IP and can change later. Do not terminate TLS in front of it. A `controlUrl` in `tug.yml` tells the CLI where control is (see [CLI configuration](docs/cli.md#config-file)). [Architecture](docs/architecture.md) covers every flow in detail, and [Security](docs/security.md) the trust boundaries.
 
-The full walkthrough, including CI credentials, moving the control plane and recovery, is in [Getting started](docs/getting-started.md).
-
 ## Documentation
 
-- [Getting started](docs/getting-started.md): install control, add nodes, deploy, give CI a credential, move control, recover.
-- [Terminology](docs/terminology.md): core concepts, the naming review (which recommendations are implemented), industry comparisons and the remaining proposed interface improvements.
-- [Architecture](docs/architecture.md): components, reconciliation, workflow diagrams, secrets and runtime paths.
-- [Compose extensions](docs/compose.md): `x-tug` placement, health checks, volumes and limitations.
-- [Control plane](docs/control-plane.md): install, mutual TLS and the private CA, image delivery, control-plane backup and recovery on another host.
-- [Backups](docs/backups.md): scheduled workload backups with `x-tug.backup` (restic), restore, recipes for common databases, and a restore drill. See also the [WordPress example](examples/wordpress/README.md).
-- [Access control](docs/access-control.md): certificate identity classes, scoped operator credentials, roles, audit trail and a GitHub Actions deploy example.
-- [Security](docs/security.md): deployment trust boundaries, security audit findings and validation limits.
-- [Networking](docs/networking.md): bring-your-own private networking, cross-node service names, ingress and TLS, firewall notes.
-- [Nodes](docs/nodes.md): node names and IDs, SSH install, self-service enrollment, certificate renewal and recovery, registration and status.
-- [CLI reference](docs/cli.md): every `tug` command and option.
-- [Configuration](docs/configuration.md): CLI, control and agent environment variables.
-- [Development](docs/development.md): local setup, CI, Compose stack, host mode and the integration test.
-- [Testing](docs/testing.md): test layers, rollout metrics, mocked test backlog, and the [Caddy/PHP and static-file benchmark](docs/testing.md#caddy-proxy-overhead-benchmark).
-- [Observability](docs/observability.md): what can be observed today, and the proposed logs, metrics and alerting.
-- [Releases](docs/releases.md): changesets, npm and Docker image publishing.
-- [Standalone binary](docs/standalone-binary.md): proposed Bun-compiled CLI binaries and Homebrew distribution, with the mTLS dependency findings.
-- [Roadmap](docs/roadmap.md): goals, principles, the reference WordPress topology and planned milestones.
-- [Docker Swarm comparison](docs/swarm-comparison.md): overlapping capabilities, workflow advantages, scheduling and networking limits, failure behavior and workload fit.
+Start with the [single-VPS tutorial](docs/getting-started.md), then choose the topic that matches your next task:
+
+- **Deploy and operate**
+  - [Application examples](examples/README.md)
+  - [Health checks and rollouts](docs/healthchecks.md)
+  - [Troubleshooting](docs/troubleshooting.md)
+- **Networking and access**
+  - [Hostnames, HTTPS and service networking](docs/networking.md)
+  - [Credentials and CI](docs/access-control.md)
+  - [Security trust boundaries](docs/security.md)
+- **Servers and data**
+  - [Service placement and scaling](docs/service-placement.md)
+  - [Node management](docs/nodes.md)
+  - [Backups and recovery](docs/backups.md)
+  - [Moving to new servers](docs/migration.md)
+- **Reference**
+  - [CLI](docs/cli.md)
+  - [Compose extensions](docs/compose.md)
+  - [Configuration](docs/configuration.md)
+  - [Control plane](docs/control-plane.md)
+- **Contribute and design**
+  - [Development](docs/development.md)
+  - [Testing](docs/testing.md)
+  - [Releases](docs/releases.md)
+  - [Architecture](docs/architecture.md)
+  - [Roadmap](docs/roadmap.md)
+  - [Observability status and proposal](docs/observability.md)
+  - [Terminology](docs/terminology.md)
+  - [Standalone binary proposal](docs/standalone-binary.md)
+  - [Docker Swarm comparison](docs/swarm-comparison.md)
 
 ## Development
 
@@ -161,4 +203,4 @@ npm run lint
 npm run test:integration
 ```
 
-See [Development](docs/development.md) for details.
+See [Development](docs/development.md) for details. Database code uses direct SQLite queries; [SQLite storage](docs/architecture.md#sqlite-storage) describes query boundaries and synchronous transactions.

package/dist/agent.js

@@ -518,9 +518,11 @@ export async function waitForHealth(ctx, project, container, service, health) {
                     throw new Error(`${service}: docker health ${state.Health.Status}`);
             }
             else if (health.type === 'tcp') {
+                await assertRunning(ctx, container, service);
                 await tcpCheck(await containerIp(ctx, project, container), health.port, parseDuration(health.timeout || '2s'));
             }
             else if (health.type === 'http') {
+                await assertRunning(ctx, container, service);
                 const ip = await containerIp(ctx, project, container);
                 const res = await ctx.http(`http://${ip}:${health.port || 80}${health.path || '/'}`, {
                     signal: AbortSignal.timeout(parseDuration(health.timeout || '2s')),
@@ -537,6 +539,33 @@ export async function waitForHealth(ctx, project, container, service, health) {
     }
     throw new Error(`${service}: ${describeHealthCheck(health)} failed after ${retries} attempt${retries === 1 ? '' : 's'}: ${describeError(last)}${outsideContainerHint(health)}`);
 }
+/** Fails fast with the exit code and last log lines when the container is gone, instead of dialing a dead address. */
+async function assertRunning(ctx, container, service) {
+    const state = (await ctx.docker.inspectContainer(container)).State;
+    if (!state || state.Running)
+        return;
+    const tail = await lastLogLines(ctx, container);
+    throw new Error(`${service}: container exited with code ${state.ExitCode ?? '?'}${tail ? `; last log: ${tail}` : ''}`);
+}
+async function lastLogLines(ctx, container, lines = 3) {
+    try {
+        const proc = await ctx.docker.logs(container, { follow: false, tail: lines });
+        let out = '';
+        const collect = (chunk) => void (out += chunk.toString());
+        proc.stdout.on('data', collect);
+        proc.stderr.on('data', collect);
+        await new Promise((resolve) => {
+            const timer = setTimeout(resolve, 2000);
+            proc.stdout.once('close', () => (clearTimeout(timer), resolve()));
+            proc.stdout.once('end', () => (clearTimeout(timer), resolve()));
+        });
+        proc.kill();
+        return out.trim().split('\n').slice(-lines).join(' | ').slice(0, 400);
+    }
+    catch {
+        return '';
+    }
+}
 /** Node's fetch reports every network failure as "fetch failed"; the system error that explains it is on `cause`. */
 function describeError(error) {
     const e = error;
@@ -987,8 +1016,10 @@ export async function reconcile(desiredState, ctx = defaultContext(), endpoints
                 markEvents('failed');
             }
             // Rolling services are untouched by a failure here, so keep reporting their current revision as serving.
-            if (!rolling.length)
-                throw e;
+            if (!rolling.length) {
+                // Callers turn the throw into a failed status; carry the per-service states so a held writer is still seen.
+                throw Object.assign(e instanceof Error ? e : new Error(errMessage(e)), { serviceStates });
+            }
             Object.assign(serviceStates, await activeRollingStates(dir, rolling));
             return failedStatus(errMessage(e));
         }
@@ -1099,6 +1130,11 @@ export async function pollOnce(ctx = defaultContext()) {
                 ...(desired.revision ? { observedRevision: desired.revision } : {}),
                 health: 'failed',
                 error: errMessage(e),
+                ...(e.serviceStates
+                    ? {
+                        serviceStates: e.serviceStates,
+                    }
+                    : {}),
             });
         }
     }

package/dist/backup-config.js

@@ -145,7 +145,7 @@ export function resolveBackups(compose, project = '') {
                 ? defaultSchedule.trim()
                 : undefined;
         if (spec.schedule === undefined && schedule === undefined)
-            errors.push(`${label} needs a schedule: set ${label}.schedule or x-tug.backup.schedule`);
+            errors.push(`${label} needs a schedule: set ${label}.schedule or a project-level schedule at the top-level x-tug.backup.schedule`);
         const retention = spec.retention !== undefined
             ? validateRetention(label, spec.retention, errors)
             : defaultRetention;

package/dist/backup-restore.d.ts

@@ -31,7 +31,14 @@ export interface RestoreDeps {
     clock?: () => Date;
 }
 /**
- * The services that write to a target. For a service they are the services that depend on it, transitively;
+ * True when an environment entry points at `service` over the network: a host-like variable that is exactly the
+ * service name (`WORDPRESS_DB_HOST=db`), or a value that names it as a host (`db:3306`, `mysql://u:p@db/app`,
+ * `mysql:host=db;dbname=app`). A service reached across nodes has no `depends_on` to find it by.
+ */
+export declare function environmentReaches(service: string, key: string, value: string): boolean;
+/**
+ * The services that write to a target. For a service they are the services that depend on it, transitively,
+ * plus the services whose environment names it as a host, which is how a service on another node reaches it (only for the target itself, not for what depends on the writers);
  * for a volume, the services that mount it and everything depending on those. Stored `depends_on` is used,
  * because the per-node copies drop cross-node dependencies.
  */
@@ -62,6 +69,6 @@ export declare function createRestoreService(deps: RestoreDeps): {
         status: number;
     };
     holds: () => Record<string, string[]>;
-    list: (project: string, limit?: number) => RestoreRunRow[];
+    list: (project: string, limit?: number, interruptedOnly?: boolean) => RestoreRunRow[];
 };
 export type RestoreService = ReturnType<typeof createRestoreService>;

package/dist/backup-restore.js

@@ -8,8 +8,36 @@ const VERIFY_TIMEOUT_MS = 5 * MINUTE;
 /** Restore states in which the project must stay locked. */
 const LOCKED_STATES = ['locking', 'holding', 'safety_backup', 'restoring', 'interrupted'];
 const TERMINAL = ['succeeded', 'failed', 'released'];
+const HOST_KEY = /host|server|addr|endpoint|url|uri|dsn/i;
 /**
- * The services that write to a target. For a service they are the services that depend on it, transitively;
+ * True when an environment entry points at `service` over the network: a host-like variable that is exactly the
+ * service name (`WORDPRESS_DB_HOST=db`), or a value that names it as a host (`db:3306`, `mysql://u:p@db/app`,
+ * `mysql:host=db;dbname=app`). A service reached across nodes has no `depends_on` to find it by.
+ */
+export function environmentReaches(service, key, value) {
+    const name = service.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+    if (value === service)
+        return HOST_KEY.test(key);
+    return [
+        new RegExp(`^${name}:\\d+$`),
+        new RegExp(`(?:://|@)(?:[^/@\\s]*@)?${name}(?::\\d+)?(?:[/?\\s]|$)`),
+        new RegExp(`(?:host|server)=${name}(?:[;:,\\s]|$)`, 'i'),
+    ].some((pattern) => pattern.test(value));
+}
+function environmentOf(service) {
+    const env = service.environment;
+    if (Array.isArray(env))
+        return env.map(String).map((entry) => {
+            const at = entry.indexOf('=');
+            return at < 0 ? [entry, ''] : [entry.slice(0, at), entry.slice(at + 1)];
+        });
+    return env && typeof env === 'object'
+        ? Object.entries(env).map(([k, v]) => [k, String(v ?? '')])
+        : [];
+}
+/**
+ * The services that write to a target. For a service they are the services that depend on it, transitively,
+ * plus the services whose environment names it as a host, which is how a service on another node reaches it (only for the target itself, not for what depends on the writers);
  * for a volume, the services that mount it and everything depending on those. Stored `depends_on` is used,
  * because the per-node copies drop cross-node dependencies.
  */
@@ -34,7 +62,12 @@ export function discoverWriters(compose, target) {
     while (queue.length) {
         const current = queue.pop();
         for (const service of Object.keys(compose.services))
-            if (!writers.has(service) && service !== name && dependsOn(service).includes(current)) {
+            if (!writers.has(service) &&
+                service !== name &&
+                (dependsOn(service).includes(current) ||
+                    (kind === 'service' &&
+                        current === name &&
+                        environmentOf(compose.services[service]).some(([k, v]) => environmentReaches(name, k, v))))) {
                 writers.add(service);
                 queue.push(service);
             }
@@ -106,7 +139,7 @@ export function createRestoreService(deps) {
                     return { error: `writer ${writer} is not a service of ${request.project}`, status: 400 };
             if (!writers.length && !request.noWriters && !request.writers)
                 return {
-                    error: `no services were found that write to ${request.target}; pass --writers to name them or --no-writers to restore without holding any`,
+                    error: `no services were found that write to ${request.target} (looked at depends_on, volume mounts and host names in environment variables); pass --writers to name them or --no-writers to restore without holding any`,
                     status: 409,
                 };
         }
@@ -136,8 +169,16 @@ export function createRestoreService(deps) {
     function unlock(restore) {
         db.run('DELETE FROM locks WHERE name=? AND owner=?', `project:${restore.project_id}`, `restore:${restore.id}`);
     }
-    function release(restore) {
+    /**
+     * Drops a restore's holds and lock. Holds a deploy placed for a move (no `restore_id`) are dropped with them,
+     * unless `keepMoveHolds`: a restore that failed before changing data must not start the writers against the
+     * empty database it was meant to fill.
+     */
+    function release(restore, keepMoveHolds = false) {
         db.run('DELETE FROM service_holds WHERE restore_id=?', restore.id);
+        if (!keepMoveHolds)
+            for (const service of JSON.parse(restore.writers_json))
+                db.run('DELETE FROM service_holds WHERE project_id=? AND service=? AND restore_id IS NULL', restore.project_id, service);
         unlock(restore);
         deps.wake(restore.project_id);
     }
@@ -166,8 +207,12 @@ export function createRestoreService(deps) {
         else
             move(restore, 'holding');
         if (restore.mode === 'in-place')
-            for (const service of restorePlan.writers)
+            for (const service of restorePlan.writers) {
+                // A hold a deploy placed for a move stays as it is; this restore drops it when it succeeds.
+                if (db.get('SELECT 1 FROM service_holds WHERE project_id=? AND service=? AND restore_id IS NULL', restore.project_id, service))
+                    continue;
                 db.run('INSERT OR REPLACE INTO service_holds(project_id,service,reason,restore_id,created_at) VALUES(?,?,?,?,?)', restore.project_id, service, `restore ${restoreId}`, restoreId, ts);
+            }
         deps.wake(restore.project_id);
         advance();
         return {
@@ -189,12 +234,29 @@ export function createRestoreService(deps) {
             });
         });
     };
+    /** Says which writers never reported `held` and why: the node's own apply error, or that it has not reported. */
+    const holdDiagnosis = (restore, writers) => {
+        const desired = deployed(restore.project_id);
+        const problems = [];
+        for (const service of writers)
+            for (const node of desired?.services[service]?.placement.nodes ?? []) {
+                const row = db.get('SELECT s.* FROM node_statuses s JOIN nodes n ON n.id=s.node_id WHERE n.name=? AND s.project_id=?', node, restore.project_id);
+                const status = row ? JSON.parse(row.status_json) : undefined;
+                const state = status?.serviceStates?.[service]?.rolloutState;
+                if (state === 'held')
+                    continue;
+                problems.push(!status
+                    ? `${service} on ${node}: the node has not reported`
+                    : `${service} on ${node}: ${state ?? 'no state'}${status.error ? `; the node's apply failed with: ${status.error}` : ''}`);
+            }
+        return problems.join('; ');
+    };
     /** Fails a restore. Before anything changed, holds and the lock are dropped; after, they stay until released. */
     function fail(restore, error, dataChanged) {
         if (dataChanged)
             move(restore, 'interrupted', { error });
         else {
-            release(restore);
+            release(restore, true);
             move(restore, 'failed', { error });
         }
         event('error', 'restore.failed', restore, `restore of ${restore.target} failed: ${error}${dataChanged ? '; the writers stay held until tug restore release' : ''}`, { error, held: dataChanged });
@@ -212,8 +274,10 @@ export function createRestoreService(deps) {
     }
     function advanceHolding(restore, writers, age) {
         if (!writersReport(restore, writers, 'held')) {
-            if (age > HOLD_TIMEOUT_MS)
-                fail(restore, 'the writers did not stop in time', false);
+            if (age > HOLD_TIMEOUT_MS) {
+                const why = holdDiagnosis(restore, writers);
+                fail(restore, `the writers did not stop in time${why ? ` (${why})` : ''}`, false);
+            }
             return;
         }
         if (!restore.safety_backup)
@@ -310,7 +374,9 @@ export function createRestoreService(deps) {
             (result[row.project_id] ??= []).push(row.service);
         return result;
     }
-    function list(project, limit = 20) {
+    function list(project, limit = 20, interruptedOnly = false) {
+        if (interruptedOnly)
+            return db.all("SELECT * FROM restore_runs WHERE project_id=? AND state='interrupted' ORDER BY started_at DESC,id", project);
         return db.all('SELECT * FROM restore_runs WHERE project_id=? ORDER BY started_at DESC LIMIT ?', project, limit);
     }
     return { plan, start, advance, releaseInterrupted, resume, holds, list };

package/dist/backup-scheduler.js

@@ -4,6 +4,7 @@ import { projectRepository } from "./backup-destination.js";
 import { volumeName } from "./container-spec.js";
 import { id } from "./util.js";
 import { isUniqueViolation } from "./db-driver.js";
+import { backupDestinationQueries } from "./db-backup-destination.js";
 /** Pinned by tag; override on control with `TUG_RESTIC_IMAGE` (a digest reference is recommended for production). */
 export const DEFAULT_RESTIC_IMAGE = 'restic/restic:0.17.3';
 const MINUTE = 60_000;
@@ -63,7 +64,8 @@ export function createBackupService(deps) {
     const started = clock().getTime();
     const stamp = () => clock().toISOString();
     const heartbeats = new Map();
-    const destination = () => db.get("SELECT * FROM backup_destination WHERE id='default'");
+    const destinationQueries = backupDestinationQueries(db);
+    const destination = () => destinationQueries.find();
     const nodeId = (name) => db.get('SELECT id FROM nodes WHERE name=?', name)?.id;
     const nodeName = (nodeIdValue) => nodeIdValue
         ? (db.get('SELECT name FROM nodes WHERE id=?', nodeIdValue)?.name ?? null)

package/dist/cli-build.d.ts

@@ -14,4 +14,4 @@ export declare function createSourceArchive(contextDir: string, archive: string,
 /** Build from the same filtered, validated source snapshot used for remote builds. */
 export declare function createLocalBuildSnapshot(contextDir: string, directory: string, configFiles: string[]): Promise<string>;
 /** Build only application build services; callers own temporary files and upload transport. */
-export declare function prepareLocalImages(compose: ComposeFile, contextDir: string, directory: string, dependencies?: LocalBuildDependencies): Promise<LocalImageUpload[]>;
+export declare function prepareLocalImages(compose: ComposeFile, contextDir: string, directory: string, dependencies?: LocalBuildDependencies, onProgress?: (message: string) => void): Promise<LocalImageUpload[]>;

package/dist/cli-build.js

@@ -143,7 +143,7 @@ export async function prepareLocalImages(compose, contextDir, directory, depende
     remove: async (imageRef) => {
         await run('docker', ['image', 'rm', imageRef]);
     },
-}) {
+}, onProgress) {
     const images = [];
     const nonce = dependencies.nonce();
     for (const [service, spec] of Object.entries(compose.services)) {
@@ -152,16 +152,19 @@ export async function prepareLocalImages(compose, contextDir, directory, depende
         const field = `image-${images.length}`;
         const imageRef = `tug-build/${nonce}/${field}:local`;
         try {
+            onProgress?.(`Building image for ${service} locally`);
             await dependencies.build({
                 contextDir,
                 service,
                 buildSpec: spec.build,
                 imageRef,
             });
+            onProgress?.(`Exporting image for ${service}`);
             const archive = await dependencies.save(imageRef, path.join(directory, `${field}.tar.gz`));
             images.push({ service, imageRef, ...archive, field });
         }
         finally {
+            onProgress?.(`Cleaning up temporary image for ${service}`);
             try {
                 await dependencies.remove(imageRef);
             }

package/dist/cli-config.d.ts

@@ -66,7 +66,7 @@ export declare function applyConfigDefaults(command: OptionSource, values: objec
  * overlay at the credential file, git-ignoring it. An existing tracked file is never rewritten, so its comments survive;
  * connection flags passed to `init` are saved to the overlay instead.
  */
-export declare function writeInitConfig(config: ResolvedCliConfig, credentialFile: string, explicit: ConnectionOptions): Promise<string[]>;
+export declare function writeInitConfig(config: ResolvedCliConfig, credentialFile: string, explicit: ConnectionOptions, credentialUrl?: string): Promise<string[]>;
 export declare function normalizeControlUrl(value: string): string;
 export declare function integerOption(value: string, name: string, minimum?: number, maximum?: number): number;
 export {};

package/dist/cli-config.js

@@ -272,11 +272,16 @@ function serialize(file, data, notice) {
  * overlay at the credential file, git-ignoring it. An existing tracked file is never rewritten, so its comments survive;
  * connection flags passed to `init` are saved to the overlay instead.
  */
-export async function writeInitConfig(config, credentialFile, explicit) {
+export async function writeInitConfig(config, credentialFile, explicit, credentialUrl) {
     const written = [];
     if (!config.fileExists) {
         await fs.mkdir(path.dirname(config.file), { recursive: true });
-        await fs.writeFile(config.file, serialize(config.file, { controlUrl: config.controlUrl, project: config.project }, `Safe to commit: keep credentials out of this file (use TUG_CREDENTIAL_FILE or ${path.basename(config.localFile)}).`));
+        await fs.writeFile(config.file, serialize(config.file, {
+            controlUrl: explicit.controlUrl
+                ? config.controlUrl
+                : (credentialUrl ?? config.controlUrl),
+            project: config.project,
+        }, `Safe to commit: keep credentials out of this file (use TUG_CREDENTIAL_FILE or ${path.basename(config.localFile)}).`));
         written.push(config.file);
     }
     let local = {};

package/dist/cli-process.d.ts

@@ -0,0 +1,13 @@
+/** Keep the event loop free for progress timers during SSH, Docker and file transfers. */
+export declare function runProcess(cmd: string, args: string[], { capture, input }?: {
+    capture?: boolean;
+    input?: string;
+}): Promise<{
+    status: number | null;
+    stdout: string;
+    stderr: string;
+}>;
+export declare function runLocal(cmd: string, args: string[]): Promise<void>;
+export declare function runCapture(cmd: string, args: string[], opts?: {
+    input?: string;
+}): Promise<string>;